Go Runtime Implementation: CGO

Code Path: src/runtime/cgocall.go The key part here is to switch the stack between Go and CGO Go -> C cgocall entersyscall: this will ensure that CGO codes will not block the Go runtime. mark m as incgo asmcgocall KeepAlive CGO-related fn, arg, and mp asmcgocall Set-up the stack. C -> Go cgocallback Set-up the stack for cgocallbackg, after cgocallbackg, the sp will point to the previous code called cgocallback cgocallbackg callbackUpdateSystemStack lockOSThread to pin the g on the current m exitsyscall cgocallbackg1 undo the previous steps reentersyscall

July 16, 2024

Go Runtime Implementations: Defer, Panic and Recover

Code Path: src/runtime/panic.go Ref: Defer, Panic, and Recover Defer Two methods of defer Copied from the comment: The older way involves creating a defer record at the time that a defer statement is executing and adding it to a defer chain. This chain is inspected by the deferreturn call at all function exits in order to run the appropriate defer calls. A cheaper way (which we call open-coded defers) is used for functions in which no defer statements occur in loops. In that case, we simply store the defer function/arg information into specific stack slots at the point of each defer statement, as well as setting a bit in a bitmask. At each function exit, we add inline code to directly make the appropriate defer calls based on the bitmask and fn/arg information stored on the stack. During panic/Goexit processing, the appropriate defer calls are made using extra funcdata info that indicates the exact stack slots that contain the bitmask and defer fn/args. ...

June 25, 2023

Set Up OpenLDAP and phpldapadmin on Ubuntu 22.04

Install LDAP Before install ldap, set-up a valid FQDN for your hostname: Edit /etc/hostname, for example void.kassiansun.com Restart the host If you’ve installed ldap before, purge them all: sudo apt-get remove --purge slapd ldap-utils -y Now we can install the ldap packages: sudo apt-get install slapd ldap-utils -y During the installation, it will prompt to set the default password. Test that you now have a valid LDAP tree: # Output: # dn: # namingContexts: dc=kassiansun,dc=com ldapsearch -H ldap://localhost -x -LLL -s base -b "" namingContexts Clean-Up Old apache2 and php installation sudo apt-get remove --purge apache2 phpldapadmin php* If you’re not using apache or php on your machine, clean them all so we can get started from the scratch. ...

June 16, 2023

Redis Internal: List Types

The Generic Set Implementation Code Path: src/t_set.c It’s a wrapper around the actual set type. The set can be one of the following types: An intset. Even if the member is a string, Redis will try to convert it to an integer with string2ll. A listpack. Used for small sets. A dict. Used for large sets. Listpack Implementation The same implementation of hash type. Dict Implementation The same implementation of hash type. ...

January 18, 2023

Redis Internal: List Types

The Generic List Implementation Code Path: src/t_list.c It’s a Redis command wrapper of listpack and quicklist. Listpack Implementation The same implementation of hash type. Quicklist Implementation Code Path: src/quicklist.c If the size of the list exceeds a certain number, Redis will try to convert a listpack to quicklist. A quicklist is a double-linked list of listpack: If the data is too large, it will store the data as a “plain” node, without encoding it as listpack. If the data size exceeds the size limit of listpack, it will create a new list node to store it. If it’s in the middle of the node, the node will split into two nodes. If the list node is empty (after deletion), it will get removed from the list. The list node can be compressed, but not the head or the tail. The pop operation is a combination of get & delete.

January 3, 2023

Redis Internal: Hash Types

The Generic Hash Implementation Code Path: src/t_hash.c The generic hash type is a wrapper around listpack and dict, it will switch between two different implementation and call the routines accordingly. Listpack Implementation Code Path: src/listpack.c Listpack is a densed storage of a series of keys and values, it supports storing both integers and strings. If the hash size is lower than hash_max_listpack_value, it will be saved as a listpack. If the hash size is higher than hash_max_listpack_value, it will create a dict and copy the data from listpack to the dict. ...

November 26, 2022

Go Runtime Implementations: Map

Code Path: src/runtime/map.go makemap Each hmap has a hash0 as seed hash. When the compiler generates the typing information for a map, the hasher function will be calculated with genhash function (code path: src/cmd/compile/internal/reflectdata/alg.go). If the map is not escaped, the hmap will be allocated by the compiler on the stack. Then makemap will allocate an array of bucket, the type of bucket is generated at compiling stage by MapBucketType (code path: src/cmd/compile/internal/reflectdata/reflect.go). ...

November 23, 2022

Redis Architecture

Start-Up Procedures spt_init: This procedure initializes the process’s name and deep copy command-line arguments & environments. tzset. zmalloc_set_oom_handler init_genrand64 with timestamp and pid number. Code Path: mt19937-64.c crc64_init. Code Path: crc64.c umask dictSetHashFunctionSeed with a random 16-byte seed. Code Path: dict.c initServerConfig ACLInit. Code Path: acl.c moduleInitModulesSystem. Code Path: module.c connTypeInitialize. Code Path: connection.c initSentinelConfig and initSentinel if the server was started in sentinel mode. Code Path: sentinel.c Run redis_check_aof_main or redis_check_rdb_main and exit. Code Path: redis-check-aof.c, redis-check-rdb.c Parse command-line options and loadServerConfig. Code Path: config.c. sentinelCheckConfigFile if it’s in sentinel mode. Code Path: sentinel.c Check whether the server is supervised by upstart or systemd. daemonize if required. initServer. It initializes the global redisServer structure. createPidFile if required. redisSetProcTitle if required. checkTcpBacklogSettings. clusterInit if it’s in cluster mode. Code Path: cluster.c moduleLoadFromQueue. Code Path: module.c. ACLLoadUsersAtStartup, Code path: acl.c. initListeners. Redis supports multiple types of listeners, and each one has its own accept handling logic. The listeners will be registered by aeCreateFileEvent clusterInitListeners if it’s in cluster mode. Code Path: cluster.c bioInit initializes the redis’s background I/O. Code Path: bio.c initThreadedIO initializes the threaded I/O. Code Path: networking.c set_jemalloc_bg_thread. If it’s not sentinel mode: aofLoadManifestFromDisk. Code Path: aof.c loadDataFromDisk. It will either load the AOF file with loadAppendOnlyFiles, or load the RDB file with rdbLoad and initialize the replication backlog with createReplicationBacklog. RDB Code Path: rdb.c aofOpenIfNeededOnServerStart to open the AOF file on disk. Code Path: aof.c aofDelHistoryFiles to clear the history_aof_list. Code Path: aof.c verifyClusterConfigWithData check the cluster slots. Note that cluster mode only allows data in db 0. Code Path: cluster.c Else, do sentinelIsRunning. redisSetCpuAffinity setOOMScoreAdj will update the value in /proc/self/oom_score_adj aeMain starts the event loop. aeDeleteEventLoop after the event loop is finished. Code Path: ae.c Event Loop server.el got initialized during initServer. The main event loop is created, serverCron and module_pipe events are registered. beforeSleep and afterSleep are also configured on the event loop. ...

November 10, 2022

Go Runtime Implementations: Timer Scheduling

Code Path: src/runtime/time.go time.startTimer (implemented as addtimer) Each p has a timers list. When a user is adding a new timer, it will first try to clean up the timers list, then adds the new timer. Each timers list is a heap sorted by timer.when, and will be updated during add/delete timers. cleantimers cleantimers only checks the head of the timers list. Check if it’s timerDeleted, delete it from the timers list, and update p’s timer0When. Check if it’s timerModifiedEarlier or timerModifiedLater, delete it from the timers list and add it back to put it at the right position in the list. doaddtimer doaddtimer link the time to the p’s timers list, and sift up the heap to put it in the right position. ...

November 7, 2022

Go Runtime Implementations: Network Polling

Code Path: src/runtime/netpoll.go Network polling is platform-dependent, the following is of BSD family operating systems (based on kqueue). netpollinit Initialize a kqueue descriptor, and set CLOSEXEC flag on it. Initialize a non-blocking pipe, and use the reader/writer for netpollBreak. netpollBreak will try to write to the pipe and interrupt the kevent. netpoll netpoll is used to check the connections and return a list of g ready to run. It can block for delay ns at most. The delay parameter will be passed to kevent syscall. ...

November 5, 2022