Travesal Through The Go Compiler, Part 5

irgen.generate Check each file’s pragma list and DeclList. Generate ir.Decl for type declarations. Generate ir.Decl for other declarations. Process later functions, this step is for the same purpose as type check. Type-check CallExpr again. Check missing function bodies. Build generic instantiations. It scans calls and generated needed methods. Remove all generic Decl’s. After irgen.generate, g.target.Decls will be the final Decl’s to generate. enqueueFunc and compileFunctions This step is the compile step of cmd/compile, it happens after type checking and IR generation. ...

September 19, 2022

Travesal Through The Go Compiler, Part 4

noder.LoadPackage Parsing noder.LoadPackage will call syntax.Parse to generate the syntax tree of all package files. Initialize a compile/internal/syntax.scanner Advance scanner to the next token. Start parsing. Check the package declaration first, and take the package name from the parsed pragma. Note that the parser is expecting a ; token, but this is automatically generated at the scanner, so users don’t have to write the ;. And parsing errors will not always abort the parser loop. Parse top-level declarations: const type var func Each type has its function to parse the whole syntax tree. Type Checking After syntax.Parse, check2 is called to do the IR generation. ...

September 18, 2022

Travesal Through The Go Compiler, Part 3

BuildToolchain BuildToolchain is initialized as noToolchain{} by default, then it’s set dynamically to gccgo or gc. Both implementations are wrappers around the binary toolchains installed on the machine. For gc, the delegations are as follows: BuildToolchain.gc - base.Tool("compile") BuildToolchain.cc - base.Tool("cgo"). Actually cgo has been executed before the compile and cfiles has been cleared, so BuildToolchain.cc should never be called and it always returns an error. BuildToolchain.asm - base.Tool("asm") BuildToolchain.pack - base.Tool("pack") BuildToolchain.ld - base.Tool("link") cmd/compile aka base.Tool(“compile”) Entry point: internal/gc.Main It’s a standalone binary which means all packages path assumes a relative path to go/src/cmd/compile. gc.Main will get an archInit function as the parameter, this function populates the ssagen.ArchInfo which contains necessary architecture information. ...

September 17, 2022

Travesal Through The Go Compiler, Part 2

Builder.build Builder.build generate an archive for a single package. Builder.build gets building information from two sources: internal/cfg package. It contains some global variables saving build flags. load.Package. It contains information for the current building package. Builder.build will build a need flag, finish the step and uncheck the finished bits one by one. Set up caching information by action, its package, and package’s target. Set up objects output directory. Setup cgo cache & vet cache. Set up building target directory. Set up non-Go files overlay. This step is to copy the non-Go files to the object directory. Preprocess coverage files and replace the file path of original *.go files. It runs go tool cover -mode=b.coverMode -var="varName" -o dst.go src.go to generate the coverage files. Build cgo files. After the build, the cfiles will be cleared and generated go files will be added to gofiles Cache all source files. It’s all *.go files now Generate import configuration and package dependencies information. Generate embedded files configuration. Run BuildToolchain.gc. This is the actual go build stage. Run BuildToolchain.cc. This is the actual cgo build stage. Run BuildToolchain.asm. This is the actual *.s build stage. Run BuildToolchain.pack. This will generate the object archive (*.a). Update the BuildID accordingly. Builder.link Builder.link links a package and generates the final binary. It’s very simple compared with the build stage. ...

September 16, 2022

Traversal Through The Go Compiler, Part 1

Set Up the Environment vim-go doesn’t support travesal through the go source code, so I switch to vscode and it works out of box (with Go plugin && gopls installed). Start Point (go run) File: src/cmd/go/main.go Declaration: run.CmdRun Function: run.runRun Key data structures: build.Context work.Builder load.Package work.Action The runRun function will go through several stages: Check shouldUseOutsideModuleMode. This procedure is used by go run cmd@version work.BuildInit. This will setup the build context: Initialize modload module. It will check go module flags and set up the flags, no actual module download. instrumentInit and buildModeInit, and update the default build.Context accordingly. Get a work.Builder. work.NewBuilder will check the environment and make sure it’s ready to do the actual build. Inits a load.Package from all *.go files passed to go run. load.Package has a public struct for definitions, and an internal struct for running state. Setup builder.LinkAction. LinkAction will call cacheAction (for looking up action cache, not build cache), CompileAction, and installAction (not for go run). work.Action is a DAG, the action cache depends on mode and package info. Initialize a work.Action, use the link action initialized at the above stage as dependencies. Build the work.Action with builder. Build Stage (Builder.Do) Set up the cache trim. There’s a trim.txt inside go-build cache folder, it’s used to track the timestamp of last trim action. Build the action list, visit the DAG as “depth-first post-order travelsal”. The priority will set by the list order, which means deepest will run first. writeActionGraph. Internal feature, this will dump the DAG as JSON. Set up triggers. Triggers are the inverse of dependencies, which means when the dependency ready, it will trigger its root instead of its dependencies. The handle function will do the actual jobs. Actions are run in parallel, the actual job is defined by action.Func. After actions done, update the global state. There’s a lock to make sure there’s no data races. If all dependencies finished, push the action node to ready queue and signal the readySema. Ready queue are protected with the same global state lock. Run all actions from the DAG in parallel. What’s Next Travesal through the action.Func definitions: ...

September 15, 2022

Why You Shouldn't Save Sessions in Redis Anymore

Sessions are important Security is an important factor of a reliable system, and from a user’s perspective, sessions are where the secure process starts. Sessions are no longer a volatile string saved in Redis/memcached, but a key to everything owned by users. Expiration time, login IP, login device, and many other properties of sessions should be tracked and managed properly. The importance of sessions makes the cache an improper choice: the nature of caching means you shouldn’t put anything important in it. Saving a business-related data in cache will make the cache a critical path in your business logic, make the logic fragile and increase the SLA number of cache systems, then make it more expensive to operate the caching system. ...

March 12, 2022

A Glide Helper for Jetpack Compose

The design of Jetpack Compose makes an important assumption on the rendering process: the developer should not assume whether or not a composable will render, and how it will render. So any code inside a composable can be called by [0-N] times. That’s why Compose introduced several helpers to control the code execution of composable (instead of controlling the rendering process): remember makes a state persisted among recomposition. The state will be destroyed if the composable is removed from Composition. LaunchedEffect controls how suspend functions will be executed and canceled. It has a key to determine whether to re-execute the code. LocalContext.current will get the current context of Composable. Most Android libraries will detect the context lifecycle. The Implementation import android.graphics.Bitmap import android.graphics.drawable.Drawable import android.net.Uri import android.util.Log import androidx.compose.runtime.* import androidx.compose.ui.platform.LocalContext import com.bumptech.glide.Glide import com.bumptech.glide.request.target.CustomTarget import com.bumptech.glide.request.transition.Transition @Composable fun loadPicture(picUri: Uri): MutableState<Bitmap?> { // the persisted state of helper val bitmapState: MutableState<Bitmap?> = remember { mutableStateOf(null) } // current context val context = LocalContext.current // glide execution control LaunchedEffect(picUri) { Log.d("utils", "loading image: $picUri") // Thanks for the original idea from https://www.youtube.com/channel/UCoNZZLhPuuRteu02rh7bzsw // Code copied from https://www.youtube.com/watch?v=ktOWiLx83bQ Glide.with(context).asBitmap().load(picUri) .into(object : CustomTarget<Bitmap>() { override fun onResourceReady( resource: Bitmap, transition: Transition<in Bitmap>? ) { Log.d("utils", "setting bitmap") bitmapState.value = resource } override fun onLoadCleared(placeholder: Drawable?) { bitmapState.value = null } }) } return bitmapState }

January 3, 2022

Introducing Tailwind CSS to My React Project

Why Tailwind CSS is good? Recently, I picked up the frontend project at the current company. Compared with traditional MVC-style native development, the web development feels more natural to me: declarative, MVVM, state flow, etc. But if we compare the react/vue with SwiftUI/Jetpack Compose, there’s a key pain point: styling. With SwiftUI/Jetpack Compose, the styling is part of view declaration, which makes it easy to read and maintain. Traditionally, CSS & HTML are separated from each other, and there’re several practices to organize them. All of these practices have some defects, more or less. In my opinion, all of these issues come from the separation of CSS and HTML. The author of Tailwind CSS wrote a great blog on this. ...

December 25, 2021

Fix GCC 11 Missing Headers on Mac OS Monterey

Encountered this issue trying to compile a simple “Hello World” program with gcc-11 from homebrew. The compiler complained about fatal error: _stdio.h: No such file or directory. After some investigation, this was caused by wrong building configuration for gcc: $ gcc -v Using built-in specs. COLLECT_GCC=gcc-11 COLLECT_LTO_WRAPPER=/opt/homebrew/Cellar/gcc/11.2.0_3/libexec/gcc/aarch64-apple-darwin21/11/lto-wrapper Target: aarch64-apple-darwin21 Configured with: ../configure --prefix=/opt/homebrew/Cellar/gcc/11.2.0_3 --libdir=/opt/homebrew/Cellar/gcc/11.2.0_3/lib/gcc/11 --disable-nls --enable-checking=release --with-gcc-major-version-only --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-11 --with-gmp=/opt/homebrew/opt/gmp --with-mpfr=/opt/homebrew/opt/mpfr --with-mpc=/opt/homebrew/opt/libmpc --with-isl=/opt/homebrew/opt/isl --with-zstd=/opt/homebrew/opt/zstd --with-pkgversion='Homebrew GCC 11.2.0_3' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --build=aarch64-apple-darwin21 --with-system-zlib --disable-multilib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk Thread model: posix Supported LTO compression algorithms: zlib zstd gcc version 11.2.0 (Homebrew GCC 11.2.0_3) The gcc from homebrew is assuming --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk, but it doesn’t exists on Mac OS Monterey. Instead the path is /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk. As we now know the root cause of this issue, fixing it will be easy: ...

December 8, 2021

Quick Start Guide with create-react-app (TypeScript + Eslint + Prettier)

Why use create-react-app Originally I was using a manually written webpack script to build my react project. The reason was to reduce the length of package-lock.json. But as I’m diving deeper into the frontend ecosystem, it doesn’t look like a good choice to wait for frontend guys fixing the dependency hell. So instead of waiting for a neat solution, I need to find a quick solution. As part of the lessons I learned from handwriting webpack scripts, it’s hard to build a complete and reusable scaffold to build a React.js + TypeScript + ESLint project. create-react-app is good enough with only minimal extra configurations. This blog is mainly a note on how to quickly start another project later. ...

December 2, 2021