c98bdf0019
Gate 0's last structural row. kernel/core/kmem.c: slabs are pages from
kalloc() with a header at the front, and free objects hold the free list in
their own first eight bytes.
That last detail is the whole design. **An object carries no header and
freeing performs no search**: the page base of any object *is* its slab, and
the slab names its cache. It is also exactly why an object cannot exceed a
page, which matters below.
Two interfaces. kmem_cache_create/alloc/free for a subsystem with one kind of
thing -- it gets its own line in /proc/slab, which is most of why naming it is
worth the argument -- and kmalloc/kmfree over power-of-two classes from 16 to
2048 for code with no single type to build a cache around. Each cache keeps
partial and full lists; a slab that empties goes straight back to kalloc()
rather than being kept, so /proc/slab never overstates what is held.
`make kmem-test` asserts the bookkeeping on the build machine. Same arrangement
as schfs-objtest, and for the same reason: slab lists, free-list threading, the
page-to-cache lookup and whether pages actually come back are all things that
can be wrong without a kernel to be wrong in. 26 checks, including one that a
permissive allocator cannot pass -- write a distinct pattern through 512
objects and read them all back, because two objects sharing storage shows up
there and nowhere else.
One bug the test found immediately: kmem_max_obj() returned the raw page
remainder, which kmem_cache_create() then rounded *up* past its own limit. The
largest legal size was reported as illegal.
**It does not do what the register said it would.** The claim was that every
fixed table exists to work around kalloc. Measured:
struct proc 10656 NPROC 128 larger than a page
struct socket 4928 NSOCK 32 larger than a page
struct buf 4272 NBUF 256 larger than a page
struct inode 280 NINODE 200 a cache, bounded on purpose
struct device 120 NDEVICE 32 3.8 KB, never hit
struct file 48 NFILE 512 24 KB, never hit
The three largest cannot go in a page-backed slab whatever it does. NINODE is
a cache and is meant to be bounded. NFILE and NDEVICE are small enough that
their tables are rounding errors and their ceilings have never been reached.
Freeing the big ones needs a contiguous multi-page allocator, which is a
different thing and is not written. That is now in the register and in
docs/architecture/kernel.md rather than left as a claim nobody checked.
What it does fix is real, and was measured rather than assumed:
- **A pipe is 640 bytes and came from a whole 4096-byte page.** Every `|` in
a shell command burned three and a half kilobytes it never touched. Six
pipes per page now, and /proc/slab shows the page going back when they
close.
- **procread() built its answer in a 2048-byte array on the kernel stack**,
under a comment that had warned about exactly that since it was written --
it stayed a local because a whole page for 2 KB felt worse, and kmalloc is
the answer that was missing. Its frame is 64 bytes now instead of ~2100.
- procfs's other generator took a page from kalloc for the same 2 KB.
Verified on a boot: /proc/slab tracks pipe and kmalloc-2048 through real
pipelines, allocs and frees balance, and $(), ps, time, readlink, sigtest and
rename are all still green with no traps.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
313 lines
15 KiB
Markdown
313 lines
15 KiB
Markdown
# SchIFT — the kernel
|
||
|
||
Long mode, 4-level paging, preemptive multitasking, and more than one
|
||
processor. What follows is the parts of the kernel that are not a filesystem, a
|
||
driver or a protocol stack: memory, processes, scheduling, traps and signals.
|
||
|
||
---
|
||
|
||
## Address space
|
||
|
||
One page table per process; the kernel is mapped into all of them at
|
||
`KERNBASE = 0xFFFF800000000000`, so a trap does not switch `CR3`.
|
||
|
||
`kvmalloc()` maps physical 0–4 GB as four 1 GB pages, with the top one marked
|
||
uncached — that is where PCI BARs and the framebuffer live. This is why 3 GB is
|
||
the most memory the kernel can treat as RAM, and it is a deliberate simplicity:
|
||
a direct map with a fixed offset makes `P2V`/`V2P` arithmetic rather than a
|
||
lookup.
|
||
|
||
`kalloc()` hands out one 4 KB page at a time from a free list. **There is no
|
||
slab or pool allocator**, and the consequence is visible throughout the tree:
|
||
every subsystem that needs many small objects has invented its own fixed-size
|
||
table — `NPROC` processes, `NSOCK` sockets, `NDEVICE` devices, `NSHM` segments.
|
||
That is why the limits are compile-time constants rather than a design decision
|
||
about each one.
|
||
|
||
The free list carries a canary, checked at the top of both `kalloc()` and
|
||
`kfree()`, so a wild write into it panics at the point it is touched rather than
|
||
handing back a bad pointer that fails somewhere else entirely.
|
||
|
||
### DMA
|
||
|
||
`kernel/core/dma.c` owns an arena carved off the top of physical memory before
|
||
`kalloc` is given the rest. Devices need physically contiguous memory; `kalloc`
|
||
cannot promise it, so the region has to be reserved before there is any
|
||
competition for it.
|
||
|
||
## Per-CPU state
|
||
|
||
The per-CPU block's base is in **both** `FS_BASE` and `GS_BASE`, set by
|
||
`seginit()`:
|
||
|
||
```
|
||
+0 self-pointer required by the x86-64 TLS ABI (%fs:0)
|
||
+8 syscall scratch trapasm.S parks %rsp here across the stack switch
|
||
+16 struct cpu *
|
||
+24 struct proc *
|
||
+32 syscall rax the three registers wrmsr destroys, parked across
|
||
+40 syscall rcx the FS_BASE swap at the top of syscall_entry
|
||
+48 syscall rdx
|
||
```
|
||
|
||
**These offsets are shared with assembly and must not be reordered.** Offset 8
|
||
is not free: `syscall_entry` uses `%fs:(8)`, and because FS and GS base point at
|
||
the same address, that is the same memory as `%gs:8`. A `_Static_assert` in
|
||
`vm.c` pins all seven, because assembly spells them as literals and a slot that
|
||
moves leaves a register quietly holding somebody else's value.
|
||
|
||
Reached through `mycpu()` and `myproc()`, which compile to a single
|
||
`movq %gs:16` / `%gs:24` with no cached base.
|
||
|
||
### The three words at +32, and what they cost before they were here
|
||
|
||
`syscall_entry` has to swap `FS_BASE` from the user's TLS to the kernel's
|
||
before it can touch any `%fs:` data, and `wrmsr` destroys `rax`, `rcx` and
|
||
`rdx`. Those three have to be parked somewhere first — not on the user stack,
|
||
which is untrusted and has not been switched away from yet.
|
||
|
||
They were parked in **three globals**, `_sc_rax`, `_sc_rcx` and `_sc_rdx`,
|
||
under a comment reading:
|
||
|
||
> Single-CPU scratch for syscall_entry FS swap (safe while IF cleared by
|
||
> SYSCALL/SFMASK)
|
||
|
||
`IF` being clear answers for an interrupt arriving on *this* CPU. It says
|
||
nothing about a second CPU executing the same three stores at the same instant,
|
||
and with more than one processor entering syscalls that is not a rare
|
||
interleaving — it is the ordinary one.
|
||
|
||
**`%rcx` is the user return address.** `SYSCALL` puts the user's `rip` there and
|
||
`SYSRET` takes it back from there. So the CPU that lost the race returned its
|
||
process into *another process's* instruction stream, and `%rax` — the syscall
|
||
number on the way in, the return value on the way out — was swapped with it.
|
||
|
||
The symptom was one to four processes per boot dying at `rip 0x0` (an
|
||
instruction fetch at zero, which a process does not branch to; it is handed a
|
||
zero and follows it) or dereferencing `0xffffffffffffffea`, which is `-EINVAL`:
|
||
a syscall return value used as a pointer. It hit shells and `schinitd` alike,
|
||
and it had been written off as a mysterious "zsh sometimes jumps into .data".
|
||
|
||
Measured, at four CPUs, with the same pristine disk image each run:
|
||
|
||
| | boots | faults |
|
||
|---|---|---|
|
||
| three globals | 10 | **10**, in five of the ten boots |
|
||
| three per-CPU words | 18 | **0** |
|
||
|
||
This is the *second* bug of exactly this shape in this file. The first was
|
||
`kernel_fs_base`, a single global that every CPU wrote during its own
|
||
`seginit()`, so the last writer won and every CPU restored one CPU's block.
|
||
`RESTORE_KERNEL_FS` reads `%gs:0` now for that reason. The lesson generalises
|
||
and is worth stating plainly: **on the entry path, a global is a bug.** There is
|
||
one per-CPU block and everything that changes per CPU belongs in it.
|
||
|
||
> **`__thread` is wrong for per-CPU kernel data**, and this is the single most
|
||
> expensive lesson in the tree. `__thread` means per *thread*, and the compiler
|
||
> is entitled to read the thread pointer once and keep it in a register across
|
||
> calls. But `%fs` here is per *CPU*, and a kernel thread changes CPU at every
|
||
> `sched()`. So the assumption is false for exactly the variables that most need
|
||
> to be right, and the failure is a null dereference deep in the scheduler on any
|
||
> machine with more than one core. `include/proc.h` carries the disassembly that
|
||
> proves it.
|
||
|
||
## Memory
|
||
|
||
`kalloc()` hands out one 4096-byte page and `kfree()` takes it back. That is the
|
||
whole page allocator: a free list threaded through the free pages themselves,
|
||
no buddy system, no zones, no page-frame database.
|
||
|
||
### Small objects
|
||
|
||
`kernel/core/kmem.c`, `include/kmem.h`. A slab is one page from `kalloc()` with
|
||
a header at the front and objects packed after it; free objects hold the free
|
||
list in their own first eight bytes.
|
||
|
||
That last detail is the whole design. **An object needs no header and freeing
|
||
needs no search**: the page base of any object *is* its slab, and the slab
|
||
names its cache. It is also why an object cannot exceed a page.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `kmem_cache_create/alloc/free` | a named cache of one object size, for a subsystem with one kind of thing |
|
||
| `kmalloc` / `kmfree` | power-of-two classes from 16 to 2048, for code with no single type to build a cache around |
|
||
|
||
Each cache keeps `partial` and `full` lists; a slab that empties completely
|
||
goes straight back to `kalloc()` rather than being kept, so `/proc/slab` never
|
||
overstates what the kernel is holding.
|
||
|
||
`make kmem-test` asserts the bookkeeping on the build machine — the same
|
||
arrangement as `schfs-objtest`, and for the same reason: slab lists, free-list
|
||
threading and whether pages actually come back are all things that can be wrong
|
||
without a kernel to be wrong in.
|
||
|
||
### What the fixed tables actually cost
|
||
|
||
The standing claim was that every fixed-size table in the kernel exists to
|
||
avoid asking `kalloc` for something smaller than a page. Measured, that is only
|
||
partly true:
|
||
|
||
| | size | table | |
|
||
|---|---|---|---|
|
||
| `struct proc` | 10656 | `NPROC` 128 | larger than a page |
|
||
| `struct socket` | 4928 | `NSOCK` 32 | larger than a page |
|
||
| `struct buf` | 4272 | `NBUF` 256 | larger than a page |
|
||
| `struct inode` | 280 | `NINODE` 200 | a cache, bounded on purpose |
|
||
| `struct device` | 120 | `NDEVICE` 32 | 3.8 KB, never hit |
|
||
| `struct file` | 48 | `NFILE` 512 | 24 KB, never hit |
|
||
|
||
The three largest cannot go in a page-backed slab at all. Freeing them needs a
|
||
contiguous multi-page allocator, which is a separate thing and is not written.
|
||
|
||
## Processes
|
||
|
||
`struct proc` in [`include/proc.h`](../../include/proc.h). A fixed table of
|
||
`NPROC` entries, allocated by linear scan.
|
||
|
||
**The field order at the top of the struct is fixed by assembly.** `trapasm.S`
|
||
hard-codes `fs_base` at offset `0x18`, and there is a `_Static_assert` in
|
||
`proc.c` that fails the build if it moves. Anything new goes at the end.
|
||
|
||
Each process carries: page directory and size, kernel stack, open file table,
|
||
cwd inode *and* cwd path string, signal handlers and masks, VMA table for
|
||
`mmap`, POSIX identity (pgid, sid, and all six of real/effective/saved uid and
|
||
gid), and the Linux-ABI syscall map.
|
||
|
||
### Slot reuse
|
||
|
||
The table recycles slots, and a recycled slot that inherits the previous
|
||
occupant's state is a whole class of bug that has bitten here twice.
|
||
`allocproc()` therefore explicitly clears the signal fields, the exit status,
|
||
the kill signal, and the identity fields. The failure mode when it did not:
|
||
`deliver_signals()` jumping to a totally unrelated earlier process's handler
|
||
address on this process's very first syscall return.
|
||
|
||
### Kernel threads
|
||
|
||
`kthread_create(fn, name)` makes a process that runs `fn` in kernel mode and
|
||
never returns to userland. It gets a page directory with only the kernel mapped,
|
||
no files, no cwd, no parent — it is not in the process tree, cannot be waited
|
||
for, and must never exit.
|
||
|
||
The point is that it is a *process* and not a timer callback: it shares the
|
||
scheduler's ordinary rotation, so it can hold sleeplocks and block on the disk
|
||
exactly as a syscall would. The network thread is the current user.
|
||
|
||
## Scheduling
|
||
|
||
Round-robin over the process table, one scheduler per CPU, each with its own
|
||
context. `swtch()` saves callee-saved registers and switches stacks; `sched()`
|
||
returns to the scheduler and asserts four invariants on the way — holding
|
||
`ptable.lock`, `ncli == 1`, not `RUNNING`, interrupts off.
|
||
|
||
When a full pass finds nothing runnable the CPU halts rather than spinning.
|
||
|
||
**Locking is a single `ptable.lock` over the whole table.** Not per-process: a
|
||
scheduling operation touches two processes (the one leaving and the one
|
||
arriving) and two-object locking needs an ordering rule that every path obeys.
|
||
The same reasoning produced a single lock over the socket table.
|
||
|
||
### sleep / wakeup
|
||
|
||
xv6's discipline, unchanged because it is correct: `sleep(chan, lk)` atomically
|
||
releases `lk` and blocks; `wakeup(chan)` makes every sleeper on that channel
|
||
runnable. The atomicity is what matters — a sleeper that tests its condition
|
||
under one lock while the waker sets it under another has no interlock and sleeps
|
||
through the wakeup meant for it.
|
||
|
||
## Traps and syscalls
|
||
|
||
Interrupts and exceptions enter through `vectors.S` → `trapasm.S` → `trap()`.
|
||
Syscalls enter through `SYSCALL`/`SYSRET` and `syscall_entry`, which is a
|
||
separate and faster path.
|
||
|
||
Every entry from user mode checks a **canary at the bottom of the kernel stack
|
||
page**. A deep or nested trap that overflows the 4 KB stack clobbers that word
|
||
first, so the failure is reported as `kstack overflow` at the point it happened
|
||
rather than as arbitrary corruption later.
|
||
|
||
One configuration detail with an expensive history: `MSR_STAR`'s `SYSRET`
|
||
selector field must hold the **compat 32-bit** code selector, because `SYSRETQ`
|
||
computes the returned `cs`/`ss` as `STAR[63:48]+16`/`+8`. Putting the real
|
||
64-bit user selector there makes every `sysretq` load an invalid pair — which
|
||
QEMU's TCG does not validate, so it stayed hidden until a hardware interrupt
|
||
forced a strictly-validated `IRETQ`.
|
||
|
||
## Signals
|
||
|
||
Real delivery, not stubs. `sys_rt_sigaction()` records the handler;
|
||
`deliver_signals()` runs in the tail of `syscall()`, just before returning to
|
||
user mode, and if a pending unblocked signal has a handler it rewrites the
|
||
trapframe to enter the handler instead of resuming — stashing the original in
|
||
`proc->saved_sigtf`. `sys_rt_sigreturn()` puts it back.
|
||
|
||
The return address is **not** `sa_restorer`. `SA_RESTORER` is not defined in
|
||
this musl build, so `sigaction()` never fills one in, and pushing 0 as a return
|
||
address crashes with `rip = 0` the moment the handler returns. Instead a 7-byte
|
||
trampoline (`mov eax,15; syscall`) is injected onto the user's own stack below
|
||
`rsp`, clear of the SysV red zone — the standard technique when the C library
|
||
supplies no restorer.
|
||
|
||
Signals without a handler fall back to a default action: termination for the
|
||
fatal set, dropped otherwise.
|
||
|
||
### PID 1 does not die from a signal it never asked to handle
|
||
|
||
`init_ignores()` discards any signal whose disposition is the default action
|
||
when the target is PID 1 — **including `SIGKILL`**. This is what POSIX requires
|
||
and what Linux does in `sig_task_ignored()`, and without it the system has a
|
||
single keystroke that kills it: Ctrl-C signals the foreground process group,
|
||
init is pgid 1, and any moment where the terminal's foreground group is still 1
|
||
lands SIGINT on init. It has no handler, so `exit()` runs, and `exit()` panics
|
||
outright on initproc.
|
||
|
||
Spamming Ctrl-C at a frozen program is exactly the input that finds that window.
|
||
|
||
## Process groups and sessions
|
||
|
||
`setsid`, `setpgid`, `getpgid`, `getsid`, and signal delivery to a whole group.
|
||
Job control is defined entirely in terms of these — a shell puts each pipeline
|
||
in its own process group and signals the group, never individual pids — and
|
||
mksh's job control could not make progress until they existed.
|
||
|
||
## What userspace gets
|
||
|
||
`fork`, `exec`, `wait`/`waitpid`/`wait4` with `WNOHANG` and real pid selection,
|
||
pipes, signals, `mmap` (anonymous and file-backed), POSIX shared memory, Unix
|
||
domain sockets, `poll`/`select`/`epoll`, `eventfd`, `timerfd`, `signalfd`.
|
||
|
||
`mmap` uses a bump allocator (`proc->mmap_next`, starting at `0x40000000`) so
|
||
every mapping gets its own disjoint, exactly-sized range. It previously assigned
|
||
a fixed 8 MB window per VMA slot regardless of the requested length, so a single
|
||
allocation over 8 MB mapped straight into a neighbouring live mapping.
|
||
|
||
---
|
||
|
||
## What it does not do
|
||
|
||
- **`ru_stime` is effectively always zero.** CPU time is sampled at the timer
|
||
tick and split on the interrupted privilege level, but `MSR_SFMASK` includes
|
||
`FL_IF`, so `SYSCALL` clears interrupts and no tick can land while a process
|
||
is inside a syscall. User time is real and discriminates properly — a busy
|
||
loop reports 97%, `sleep 2` reports 0% — but kernel time needs TSC deltas
|
||
closed and reopened around every `sched()`, which is scheduler accounting
|
||
rather than sampling, and is not written.
|
||
|
||
- **No slab allocator.** One page at a time, and every subsystem's fixed-size
|
||
table is a consequence. This is the single change that would most improve the
|
||
rest of the kernel.
|
||
- **No demand paging and no page cache.** `mmap` of a file *reads the whole
|
||
thing in* at map time. Writable `MAP_SHARED` of a file is therefore refused
|
||
outright rather than faked — see [../plan/foundations.md](../plan/foundations.md).
|
||
- **No dynamic linking.** `exec` handles `ET_EXEC` only: no `PT_INTERP`, no
|
||
`ET_DYN`, no shared libraries, no `dlopen`.
|
||
- **No threads.** `clone` exists but does not share an address space;
|
||
`gettid` returns the pid.
|
||
- **Permissions are tracked but not enforced.** uid/gid are maintained honestly
|
||
so `getuid()`/`setuid()` stop lying, and nothing consults them on access.
|
||
- **No swap**, and no memory pressure handling of any kind. `kalloc` failing is
|
||
reported up as `-ENOMEM` and that is the whole policy.
|
||
- **Nested signal delivery is not supported.** `in_signal` guards against
|
||
re-entering a handler while one is running.
|
||
- **Scheduling is round-robin with no priorities**, no fairness accounting, and
|
||
no CPU affinity despite `sched_setaffinity` existing.
|