# SchIFT — the kernel Long mode, 4-level paging, preemptive multitasking, and more than one processor. What follows is the parts of the kernel that are not a filesystem, a driver or a protocol stack: memory, processes, scheduling, traps and signals. --- ## Address space One page table per process; the kernel is mapped into all of them at `KERNBASE = 0xFFFF800000000000`, so a trap does not switch `CR3`. `kvmalloc()` maps as many 1 GB pages of physical memory as the machine has RAM (up to `PHYSTOP_MAX`, 256 GB), always at least the low four. The 3–4 GB slot stays uncached because that is where the PCI hole is; everything above 4 GB is RAM, so those are cached, and a device BAR up there still gets its own uncached window from `ioremap()`, which takes the entries the direct map did not. On a CPU without 1 GB pages (Intel's desktop parts before Haswell), each gigabyte is a page directory of 2 MB pages instead — the same reach, one extra page of table per gigabyte. A direct map with a fixed offset makes `P2V`/`V2P` arithmetic rather than a lookup. How much RAM there is comes from the firmware's E820/UEFI map, as a *list* of usable regions, not one run ending at a top address (OVMF drops an ACPI NVS region at 8 MB, which used to hide everything above it). The kernel itself is linked at 16 MB, clear of what firmware keeps low. `kalloc` holds never-used memory as ranges, so a large machine costs nothing until its memory is used. `kalloc()` hands out one 4 KB page at a time from a free list. Objects smaller than a page come from the slab caches below; objects larger than one do not have a home yet, so the biggest tables (`NPROC` processes, `NSOCK` sockets, `NDEVICE` devices, `NSHM` segments) are still fixed-size arrays, which is why their limits are compile-time constants rather than a design decision about each one. The free list carries a canary, checked at the top of both `kalloc()` and `kfree()`, so a wild write into it panics at the point it is touched rather than handing back a bad pointer that fails somewhere else entirely. ### DMA `kernel/core/dma.c` owns an arena carved off the top of physical memory before `kalloc` is given the rest. Devices need physically contiguous memory; `kalloc` cannot promise it, so the region has to be reserved before there is any competition for it. ## Per-CPU state The per-CPU block's base is in **both** `FS_BASE` and `GS_BASE`, set by `seginit()`: ``` +0 self-pointer required by the x86-64 TLS ABI (%fs:0) +8 syscall scratch trapasm.S parks %rsp here across the stack switch +16 struct cpu * +24 struct proc * +32 syscall rax the three registers wrmsr destroys, parked across +40 syscall rcx the FS_BASE swap at the top of syscall_entry +48 syscall rdx ``` **These offsets are shared with assembly and must not be reordered.** Offset 8 is not free: `syscall_entry` uses `%fs:(8)`, and because FS and GS base point at the same address, that is the same memory as `%gs:8`. A `_Static_assert` in `vm.c` pins all seven, because assembly spells them as literals and a slot that moves leaves a register quietly holding somebody else's value. Reached through `mycpu()` and `myproc()`, which compile to a single `movq %gs:16` / `%gs:24` with no cached base. ### The three words at +32, and what they cost before they were here `syscall_entry` has to swap `FS_BASE` from the user's TLS to the kernel's before it can touch any `%fs:` data, and `wrmsr` destroys `rax`, `rcx` and `rdx`. Those three have to be parked somewhere first — not on the user stack, which is untrusted and has not been switched away from yet. They were parked in **three globals**, `_sc_rax`, `_sc_rcx` and `_sc_rdx`, under a comment reading: > Single-CPU scratch for syscall_entry FS swap (safe while IF cleared by > SYSCALL/SFMASK) `IF` being clear answers for an interrupt arriving on *this* CPU. It says nothing about a second CPU executing the same three stores at the same instant, and with more than one processor entering syscalls that is not a rare interleaving — it is the ordinary one. **`%rcx` is the user return address.** `SYSCALL` puts the user's `rip` there and `SYSRET` takes it back from there. So the CPU that lost the race returned its process into *another process's* instruction stream, and `%rax` — the syscall number on the way in, the return value on the way out — was swapped with it. The symptom was one to four processes per boot dying at `rip 0x0` (an instruction fetch at zero, which a process does not branch to; it is handed a zero and follows it) or dereferencing `0xffffffffffffffea`, which is `-EINVAL`: a syscall return value used as a pointer. It hit shells and `schinitd` alike, and it had been written off as a mysterious "zsh sometimes jumps into .data". Measured, at four CPUs, with the same pristine disk image each run: | | boots | faults | |---|---|---| | three globals | 10 | **10**, in five of the ten boots | | three per-CPU words | 18 | **0** | This is the *second* bug of exactly this shape in this file. The first was `kernel_fs_base`, a single global that every CPU wrote during its own `seginit()`, so the last writer won and every CPU restored one CPU's block. `RESTORE_KERNEL_FS` reads `%gs:0` now for that reason. The lesson generalises and is worth stating plainly: **on the entry path, a global is a bug.** There is one per-CPU block and everything that changes per CPU belongs in it. > **`__thread` is wrong for per-CPU kernel data**, and this is the single most > expensive lesson in the tree. `__thread` means per *thread*, and the compiler > is entitled to read the thread pointer once and keep it in a register across > calls. But `%fs` here is per *CPU*, and a kernel thread changes CPU at every > `sched()`. So the assumption is false for exactly the variables that most need > to be right, and the failure is a null dereference deep in the scheduler on any > machine with more than one core. `include/core/proc.h` carries the disassembly that > proves it. ## Memory `kalloc()` hands out one 4096-byte page and `kfree()` takes it back. That is the whole page allocator: a free list threaded through the free pages themselves, no buddy system, no zones, no page-frame database. ### Small objects `kernel/core/kmem.c`, `include/core/kmem.h`. A slab is one page from `kalloc()` with a header at the front and objects packed after it; free objects hold the free list in their own first eight bytes. That last detail is the whole design. **An object needs no header and freeing needs no search**: the page base of any object *is* its slab, and the slab names its cache. It is also why an object cannot exceed a page. | | | |---|---| | `kmem_cache_create/alloc/free` | a named cache of one object size, for a subsystem with one kind of thing | | `kmalloc` / `kmfree` | power-of-two classes from 16 to 2048, for code with no single type to build a cache around | Each cache keeps `partial` and `full` lists; a slab that empties completely goes straight back to `kalloc()` rather than being kept, so `/proc/slab` never overstates what the kernel is holding. `make kmem-test` asserts the bookkeeping on the build machine — the same arrangement as `schfs-objtest`, and for the same reason: slab lists, free-list threading and whether pages actually come back are all things that can be wrong without a kernel to be wrong in. ### Shared memory objects `shmget`/`shmat`/`shmdt` are System V shared memory: segments of `kalloc`'d pages, reference counted, mapped through a VMA with `no_free` set so that `freevm()` does not hand another process's pages to `kfree`. `memfd_create(2)` is the same machinery reached by **descriptor** instead of by key. That difference is the whole point and the reason it exists on Linux at all: two processes share memory by passing the fd over a socket, so neither has to agree on a path or race to invent a key. It is how a display server and its client share a buffer, and it is why descriptor passing (`SCM_RIGHTS`) had to be built first. | | | |---|---| | `memfd_create(name, flags)` | a new segment of size 0, and a descriptor owning one reference | | `ftruncate(fd, n)` | gives it a size — growing allocates zeroed pages, shrinking frees from the end | | `mmap(…, MAP_SHARED, fd, off)` | maps the segment's existing pages, so both mappers get the same physical memory | | `read`/`write`/`lseek`/`fstat` | work, so it is a file and not a token only `mmap` understands | Three things are refused rather than faked, and each refusal is the honest version of something this kernel cannot do: - **`MAP_PRIVATE` on a memfd.** Every caller is sharing on purpose; a private mapping of an anonymous object is an expensive `MAP_ANONYMOUS`. - **Mapping past the end.** Linux lets it through and faults the pages in. There is no fault handler here that could, so the range is rejected. - **Shrinking while mapped.** Linux truncates the pages out from under the mapper. Here a freed page would simply stay mapped to memory somebody else now owns. Writable `MAP_SHARED` of a **real file** is still refused. That needs a page cache, it is a separate project, and nothing that runs today asks for it — an anonymous object has no such contract to break, which is exactly why `memfd` is the narrow answer. ### Demand paging Anonymous memory can be recorded without being backed: the page appears, zeroed, the first time it is touched — by the program, through `trap()`, or by the kernel on its behalf, through `uva2ka()`. `mmap` does this for mappings of 1 MB or more, for `MAP_NORESERVE`, and for `PROT_NONE`. The reason is WebKit, which reserves 4 GB of address space at start for its JIT and heaps and touches a sliver of it; backing all of it at `mmap` time failed with `ENOMEM` and the browser aborted. Go, Java and Firefox reserve the same way. `vm_fault()` takes a spinlock and nothing that sleeps, so `uva2ka()` may call it from anywhere; the lock keeps two threads of one process from both mapping the same page. `fork` and `munmap` skip the holes, so a page nobody touched is one they never see, and a child that touches it reads zeroes — what it would have read in the parent. File-backed mappings are **not** lazy: `mmap` of a file still reads the range in at map time. ### TLB shootdowns Unmapping from an address space that threads on other processors are running in interrupts those processors and waits until each has reloaded `CR3` (`tlb_shootdown()` in `vm.c`; the vector is one of the MSI lines, which `trap()` already dispatches with an EOI). Without it, when one thread unmapped a range and `mmap` handed the same addresses straight back, a thread on another CPU went on reading and writing the *old* pages through its TLB: stores vanished and loads were stale. musl's allocator noticed first — a group's metadata no longer pointing back at it — and Epiphany, forty threads on four processors, died at a different address on every run. Only one shootdown runs at a time, and it is taken without disabling interrupts: two processors each waiting with interrupts off for the other to answer would wait for ever. On a single processor, or if no vector could be allocated, it does nothing (and says so at boot). ### What the fixed tables actually cost The standing claim was that every fixed-size table in the kernel exists to avoid asking `kalloc` for something smaller than a page. Measured, that is only partly true: | | size | table | | |---|---|---|---| | `struct proc` | 1984 | `NPROC` 128 | 254 KB in all; the address space, descriptors and signal handlers live in their own objects now, shared by threads | | `struct mm` | 8232 | one per process, 128 | 1 MB in all, almost all of it the 256-entry VMA table | | `struct socket` | about 4900 | `NSOCK` 128 | larger than a page | | `struct device` | 120 | `NDEVICE` 128 | 15 KB, never hit | | `struct buf` | 200 | `NBUF` 256 to start | grows from a slab, data from `kalloc`, up to RAM/32 | | `struct inode` | about 460 | none | grows from a slab; `NINODE` 200 is only how many are made up front | | `struct file` | about 88 | none | grows from a slab; `NFILE` 512 is only the static pool | (Sizes from `nm -S` on a build of the tree, not from a header comment.) `struct proc` shrank by a factor of six when threads arrived, because the page tables, VMAs, descriptor table and signal dispositions moved out of it into objects a thread group shares. `struct socket` is still too large for a page-backed slab; freeing it needs a contiguous multi-page allocator, which is a separate thing and is not written. The buffer header shrank to 200 bytes when its 4 KB of data moved out to a page of its own, which is what let the cache grow at all. ## Processes `struct proc` in [`include/core/proc.h`](../../include/core/proc.h). A fixed table of `NPROC` entries, allocated by linear scan. A thread is a `struct proc` too. **The field order at the top of the struct is fixed by assembly.** `trapasm.S` hard-codes `fs_base` at offset `0x18`, and there is a `_Static_assert` in `proc.c` that fails the build if it moves. Anything new goes at the end. Each process carries: a kernel stack, cwd inode *and* cwd path string, signal masks, POSIX identity (pgid, sid, and all six of real/effective/saved uid and gid), the Linux-ABI syscall map, and its x87/SSE registers (saved on a switch; the kernel itself stays off them). Three things it *points to* rather than owns, because a thread group shares them: | | | |---|---| | `struct mm` | the address space: page tables, heap top, the VMA table, which file each span came from, and a lock so one thread at a time changes the mappings | | `struct files` | the descriptor table and its `FD_CLOEXEC` flags | | `struct sighand` | signal dispositions — what `sigaction()` sets | Each is reference counted. `clone()` shares what its flags say and `fork()` copies all three. ### Threads `clone()` takes `pthread_create`'s flags (`CLONE_VM`, `FILES`, `SIGHAND`, `THREAD`, `SETTLS`, `PARENT_SETTID`, `CHILD_CLEARTID`) and `posix_spawn`'s (`VM`, `VFORK`, a stack). What follows from that: - **futex** — `WAIT`, `WAIT_BITSET` with timeouts, `WAKE`, `REQUEUE`, `CMP_REQUEUE`, keyed by the word's *kernel* address so processes that map the same page meet. `set_tid_address` keeps the word that exit zeroes and wakes, which is what `pthread_join` waits on. - **Exit.** `exit` ends the calling thread; `exit_group`, a fatal signal, or the leader exiting ends them all. `reapd` frees exited threads, and the leader is reaped by its parent once the rest are gone. `getpid` is the group, `gettid` the thread. - **Signals.** Stops and resumes reach every thread; other process-directed signals go to the leader. - **`exec`** kills the other threads and unshares the dispositions. - **`/proc`** lists processes, `task/` lists threads; `comm`, `Threads:` and `Tgid` are there. The first program to need all of this was GTK's, then WebKit's. ### Slot reuse The table recycles slots, and a recycled slot that inherits the previous occupant's state is a whole class of bug that has bitten here twice. `allocproc()` therefore explicitly clears the signal fields, the exit status, the kill signal, and the identity fields. The failure mode when it did not: `deliver_signals()` jumping to a totally unrelated earlier process's handler address on this process's very first syscall return. ### Kernel threads `kthread_create(fn, name)` makes a process that runs `fn` in kernel mode and never returns to userland. It gets a page directory with only the kernel mapped, no files, no cwd, no parent — it is not in the process tree, cannot be waited for, and must never exit. The point is that it is a *process* and not a timer callback: it shares the scheduler's ordinary rotation, so it can hold sleeplocks and block on the disk exactly as a syscall would. The network thread is the current user. ## Scheduling Fair share by virtual runtime, one scheduler per CPU, each with its own context — the rule, the guards and `nice` are in [scheduler.md](scheduler.md). `swtch()` saves callee-saved registers and switches stacks; `sched()` returns to the scheduler and asserts four invariants on the way — holding `ptable.lock`, `ncli == 1`, not `RUNNING`, interrupts off. When nothing is runnable the CPU goes idle rather than spinning: into ACPI's `_CST` C2 state (CC6 on Zen), by port read or `MWAIT`, where the APIC timer keeps running through it, and otherwise a plain `hlt`. **Locking is a single `ptable.lock` over the whole table.** Not per-process: a scheduling operation touches two processes (the one leaving and the one arriving) and two-object locking needs an ordering rule that every path obeys. The same reasoning produced a single lock over the socket table. ### sleep / wakeup xv6's discipline, unchanged because it is correct: `sleep(chan, lk)` atomically releases `lk` and blocks; `wakeup(chan)` makes every sleeper on that channel runnable. The atomicity is what matters — a sleeper that tests its condition under one lock while the waker sets it under another has no interlock and sleeps through the wakeup meant for it. ## Traps and syscalls Interrupts and exceptions enter through `vectors.S` → `trapasm.S` → `trap()`. Syscalls enter through `SYSCALL`/`SYSRET` and `syscall_entry`, which is a separate and faster path. Every entry from user mode checks a **canary at the bottom of the kernel stack page**. A deep or nested trap that overflows the 4 KB stack clobbers that word first, so the failure is reported as `kstack overflow` at the point it happened rather than as arbitrary corruption later. One configuration detail with an expensive history: `MSR_STAR`'s `SYSRET` selector field must hold the **compat 32-bit** code selector, because `SYSRETQ` computes the returned `cs`/`ss` as `STAR[63:48]+16`/`+8`. Putting the real 64-bit user selector there makes every `sysretq` load an invalid pair — which QEMU's TCG does not validate, so it stayed hidden until a hardware interrupt forced a strictly-validated `IRETQ`. ## Signals Real delivery, not stubs. `sys_rt_sigaction()` records the handler; `deliver_signals()` runs in the tail of `syscall()`, just before returning to user mode, and if a pending unblocked signal has a handler it rewrites the trapframe to enter the handler instead of resuming — stashing the original in `proc->saved_sigtf`. `sys_rt_sigreturn()` puts it back. The return address is **not** `sa_restorer`. `SA_RESTORER` is not defined in this musl build, so `sigaction()` never fills one in, and pushing 0 as a return address crashes with `rip = 0` the moment the handler returns. Instead a 7-byte trampoline (`mov eax,15; syscall`) is injected onto the user's own stack below `rsp`, clear of the SysV red zone — the standard technique when the C library supplies no restorer. Signals without a handler fall back to a default action: termination for the fatal set, dropped otherwise. ### PID 1 does not die from a signal it never asked to handle `init_ignores()` discards any signal whose disposition is the default action when the target is PID 1 — **including `SIGKILL`**. This is what POSIX requires and what Linux does in `sig_task_ignored()`, and without it the system has a single keystroke that kills it: Ctrl-C signals the foreground process group, init is pgid 1, and any moment where the terminal's foreground group is still 1 lands SIGINT on init. It has no handler, so `exit()` runs, and `exit()` panics outright on initproc. Spamming Ctrl-C at a frozen program is exactly the input that finds that window. ## Process groups and sessions `setsid`, `setpgid`, `getpgid`, `getsid`, and signal delivery to a whole group. Job control is defined entirely in terms of these — a shell puts each pipeline in its own process group and signals the group, never individual pids — and mksh's job control could not make progress until they existed. ## What userspace gets `fork`, `clone` (threads), `exec`, `wait`/`waitpid`/`wait4` with `WNOHANG` and real pid selection, pipes, signals, `mmap` (anonymous and file-backed), POSIX shared memory, `memfd`, Unix domain sockets, `poll`/`select`/`epoll`, `eventfd`, `timerfd`, `signalfd`, `inotify`, `futex`. ### Dynamic linking `exec` handles `ET_EXEC` and `ET_DYN`. A dynamically linked program names an interpreter in `PT_INTERP` — on musl, `/lib/ld-musl-x86_64.so.1`, which is libc under another name — and the kernel's whole part is to load that as well and start it instead of the program. The interpreter finds the program through the auxiliary vector (`AT_PHDR`, `AT_ENTRY`), maps libc and whatever else it needs with `mmap()`, relocates, and jumps to `AT_ENTRY` itself. A position-independent program goes where a fixed one would (`0x400000`); the interpreter goes at the start of the `mmap` area (`0x40000000`) and `mmap` carries on above it, so the two never meet. This is what lets Alpine's binaries run unmodified, and `dlopen` works, so Xorg loads its drivers as modules. `mmap` uses a bump allocator (`proc->mmap_next`, starting at `0x40000000`) so every mapping gets its own disjoint, exactly-sized range. It previously assigned a fixed 8 MB window per VMA slot regardless of the requested length, so a single allocation over 8 MB mapped straight into a neighbouring live mapping. --- ## What it does not do - **`ru_stime` is effectively always zero.** CPU time is sampled at the timer tick and split on the interrupted privilege level, but `MSR_SFMASK` includes `FL_IF`, so `SYSCALL` clears interrupts and no tick can land while a process is inside a syscall. User time is real and discriminates properly — a busy loop reports 97%, `sleep 2` reports 0% — but kernel time needs TSC deltas closed and reopened around every `sched()`, which is scheduler accounting rather than sampling, and is not written. - **`struct socket` is too big for a slab**, and every subsystem's fixed-size table (`NPROC`, `NSOCK`, `NDEVICE`, `NSHM`) is still a compile-time constant. Slabs exist for the objects that fit in a page. - **No page cache.** `mmap` of a file *reads the range in* at map time, and writable `MAP_SHARED` of a file is refused outright rather than faked. Demand paging covers anonymous memory only. - **Permissions are tracked but not enforced.** uid/gid are maintained honestly so `getuid()`/`setuid()` stop lying, and file access does not consult them; `setpriority` is the one place the uids are checked. - **No swap**, and no memory pressure handling of any kind. `kalloc` failing is reported up as `-ENOMEM` and that is the whole policy. - **Nested signal delivery is not supported.** `in_signal` guards against re-entering a handler while one is running. - **One global run queue.** Fair share and `nice` exist; per-CPU run queues, work stealing and CPU affinity do not, and `sched_setaffinity` is accepted and ignored. - **No IOMMU**, so DMA-capable devices (and the driver processes that program them) are trusted. - **256 GB is the most RAM** the kernel uses; the rest is ignored, with a line saying so.