3185685997
One to four processes died per boot with a fault they could not have caused:
`rip 0x0` -- an instruction fetch at zero, which a process does not branch to
-- or a dereference of `0xffffffffffffffea`, which is `-EINVAL`. It hit shells
and schinitd alike and had been carried for a long time as "zsh sometimes jumps
into .data".
`syscall_entry` has to swap FS_BASE from the user's TLS to the kernel's before
it can touch any `%fs:` data, and `wrmsr` destroys rax, rcx and rdx. Those have
to be parked somewhere first -- not on the user stack, which is untrusted and
has not been switched away from yet. They were parked in three globals:
// Single-CPU scratch for syscall_entry FS swap
// (safe while IF cleared by SYSCALL/SFMASK)
uint64 _sc_rax, _sc_rcx, _sc_rdx;
IF being clear answers for an interrupt arriving on *this* CPU. It says nothing
whatever about a second CPU executing the same three stores at the same
instant, and with more than one processor entering syscalls that is not a rare
interleaving -- it is the ordinary one.
**%rcx is the user return address.** SYSCALL puts the user's rip there and
SYSRET takes it back from there. So the CPU that lost the race returned its
process into *another process's* instruction stream, with %rax -- the syscall
number on the way in, the return value on the way out -- swapped alongside it.
Both symptoms fall straight out of that: a zero handed over and followed, and
an errno used as a pointer.
Per-CPU words at %gs:32..48 now. %gs is usable there, before the FS swap, for
exactly the reason RESTORE_KERNEL_FS can already read %gs:0: GS base is per-CPU
in hardware, seginit() sets it once, and userspace cannot change it --
sys_arch_prctl() implements ARCH_SET_FS and ARCH_GET_FS and rejects the rest.
Measured rather than argued. Both variants built once, `build/esp` saved for
each, the same pristine schfs.img restored before every boot, -smp 4:
three globals 10 boots 10 faults, in five of the ten
three per-CPU 18 boots 0 faults
A _Static_assert pins all seven per-CPU offsets, because trapasm.S spells them
as literals and cannot include a header -- a slot that moves would leave a
register quietly holding somebody else's value, which is the hardest possible
thing to find.
**This is the second bug of this exact shape in this file.** The first was
`kernel_fs_base`, a single global every CPU wrote during its own seginit(), so
the last writer won and every CPU restored one CPU's per-CPU block;
RESTORE_KERNEL_FS reads %gs:0 for that reason. The lesson generalises and is
now written down in both places: **on the entry path, a global is a bug.**
Closes the row added to Gate 0 four commits ago. It was the right row to put
first -- a compositor built on this would have failed in ways indistinguishable
from its own bugs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
257 lines
12 KiB
Markdown
257 lines
12 KiB
Markdown
# SchIFT — the kernel
|
||
|
||
Long mode, 4-level paging, preemptive multitasking, and more than one
|
||
processor. What follows is the parts of the kernel that are not a filesystem, a
|
||
driver or a protocol stack: memory, processes, scheduling, traps and signals.
|
||
|
||
---
|
||
|
||
## Address space
|
||
|
||
One page table per process; the kernel is mapped into all of them at
|
||
`KERNBASE = 0xFFFF800000000000`, so a trap does not switch `CR3`.
|
||
|
||
`kvmalloc()` maps physical 0–4 GB as four 1 GB pages, with the top one marked
|
||
uncached — that is where PCI BARs and the framebuffer live. This is why 3 GB is
|
||
the most memory the kernel can treat as RAM, and it is a deliberate simplicity:
|
||
a direct map with a fixed offset makes `P2V`/`V2P` arithmetic rather than a
|
||
lookup.
|
||
|
||
`kalloc()` hands out one 4 KB page at a time from a free list. **There is no
|
||
slab or pool allocator**, and the consequence is visible throughout the tree:
|
||
every subsystem that needs many small objects has invented its own fixed-size
|
||
table — `NPROC` processes, `NSOCK` sockets, `NDEVICE` devices, `NSHM` segments.
|
||
That is why the limits are compile-time constants rather than a design decision
|
||
about each one.
|
||
|
||
The free list carries a canary, checked at the top of both `kalloc()` and
|
||
`kfree()`, so a wild write into it panics at the point it is touched rather than
|
||
handing back a bad pointer that fails somewhere else entirely.
|
||
|
||
### DMA
|
||
|
||
`kernel/core/dma.c` owns an arena carved off the top of physical memory before
|
||
`kalloc` is given the rest. Devices need physically contiguous memory; `kalloc`
|
||
cannot promise it, so the region has to be reserved before there is any
|
||
competition for it.
|
||
|
||
## Per-CPU state
|
||
|
||
The per-CPU block's base is in **both** `FS_BASE` and `GS_BASE`, set by
|
||
`seginit()`:
|
||
|
||
```
|
||
+0 self-pointer required by the x86-64 TLS ABI (%fs:0)
|
||
+8 syscall scratch trapasm.S parks %rsp here across the stack switch
|
||
+16 struct cpu *
|
||
+24 struct proc *
|
||
+32 syscall rax the three registers wrmsr destroys, parked across
|
||
+40 syscall rcx the FS_BASE swap at the top of syscall_entry
|
||
+48 syscall rdx
|
||
```
|
||
|
||
**These offsets are shared with assembly and must not be reordered.** Offset 8
|
||
is not free: `syscall_entry` uses `%fs:(8)`, and because FS and GS base point at
|
||
the same address, that is the same memory as `%gs:8`. A `_Static_assert` in
|
||
`vm.c` pins all seven, because assembly spells them as literals and a slot that
|
||
moves leaves a register quietly holding somebody else's value.
|
||
|
||
Reached through `mycpu()` and `myproc()`, which compile to a single
|
||
`movq %gs:16` / `%gs:24` with no cached base.
|
||
|
||
### The three words at +32, and what they cost before they were here
|
||
|
||
`syscall_entry` has to swap `FS_BASE` from the user's TLS to the kernel's
|
||
before it can touch any `%fs:` data, and `wrmsr` destroys `rax`, `rcx` and
|
||
`rdx`. Those three have to be parked somewhere first — not on the user stack,
|
||
which is untrusted and has not been switched away from yet.
|
||
|
||
They were parked in **three globals**, `_sc_rax`, `_sc_rcx` and `_sc_rdx`,
|
||
under a comment reading:
|
||
|
||
> Single-CPU scratch for syscall_entry FS swap (safe while IF cleared by
|
||
> SYSCALL/SFMASK)
|
||
|
||
`IF` being clear answers for an interrupt arriving on *this* CPU. It says
|
||
nothing about a second CPU executing the same three stores at the same instant,
|
||
and with more than one processor entering syscalls that is not a rare
|
||
interleaving — it is the ordinary one.
|
||
|
||
**`%rcx` is the user return address.** `SYSCALL` puts the user's `rip` there and
|
||
`SYSRET` takes it back from there. So the CPU that lost the race returned its
|
||
process into *another process's* instruction stream, and `%rax` — the syscall
|
||
number on the way in, the return value on the way out — was swapped with it.
|
||
|
||
The symptom was one to four processes per boot dying at `rip 0x0` (an
|
||
instruction fetch at zero, which a process does not branch to; it is handed a
|
||
zero and follows it) or dereferencing `0xffffffffffffffea`, which is `-EINVAL`:
|
||
a syscall return value used as a pointer. It hit shells and `schinitd` alike,
|
||
and it had been written off as a mysterious "zsh sometimes jumps into .data".
|
||
|
||
Measured, at four CPUs, with the same pristine disk image each run:
|
||
|
||
| | boots | faults |
|
||
|---|---|---|
|
||
| three globals | 10 | **10**, in five of the ten boots |
|
||
| three per-CPU words | 18 | **0** |
|
||
|
||
This is the *second* bug of exactly this shape in this file. The first was
|
||
`kernel_fs_base`, a single global that every CPU wrote during its own
|
||
`seginit()`, so the last writer won and every CPU restored one CPU's block.
|
||
`RESTORE_KERNEL_FS` reads `%gs:0` now for that reason. The lesson generalises
|
||
and is worth stating plainly: **on the entry path, a global is a bug.** There is
|
||
one per-CPU block and everything that changes per CPU belongs in it.
|
||
|
||
> **`__thread` is wrong for per-CPU kernel data**, and this is the single most
|
||
> expensive lesson in the tree. `__thread` means per *thread*, and the compiler
|
||
> is entitled to read the thread pointer once and keep it in a register across
|
||
> calls. But `%fs` here is per *CPU*, and a kernel thread changes CPU at every
|
||
> `sched()`. So the assumption is false for exactly the variables that most need
|
||
> to be right, and the failure is a null dereference deep in the scheduler on any
|
||
> machine with more than one core. `include/proc.h` carries the disassembly that
|
||
> proves it.
|
||
|
||
## Processes
|
||
|
||
`struct proc` in [`include/proc.h`](../../include/proc.h). A fixed table of
|
||
`NPROC` entries, allocated by linear scan.
|
||
|
||
**The field order at the top of the struct is fixed by assembly.** `trapasm.S`
|
||
hard-codes `fs_base` at offset `0x18`, and there is a `_Static_assert` in
|
||
`proc.c` that fails the build if it moves. Anything new goes at the end.
|
||
|
||
Each process carries: page directory and size, kernel stack, open file table,
|
||
cwd inode *and* cwd path string, signal handlers and masks, VMA table for
|
||
`mmap`, POSIX identity (pgid, sid, and all six of real/effective/saved uid and
|
||
gid), and the Linux-ABI syscall map.
|
||
|
||
### Slot reuse
|
||
|
||
The table recycles slots, and a recycled slot that inherits the previous
|
||
occupant's state is a whole class of bug that has bitten here twice.
|
||
`allocproc()` therefore explicitly clears the signal fields, the exit status,
|
||
the kill signal, and the identity fields. The failure mode when it did not:
|
||
`deliver_signals()` jumping to a totally unrelated earlier process's handler
|
||
address on this process's very first syscall return.
|
||
|
||
### Kernel threads
|
||
|
||
`kthread_create(fn, name)` makes a process that runs `fn` in kernel mode and
|
||
never returns to userland. It gets a page directory with only the kernel mapped,
|
||
no files, no cwd, no parent — it is not in the process tree, cannot be waited
|
||
for, and must never exit.
|
||
|
||
The point is that it is a *process* and not a timer callback: it shares the
|
||
scheduler's ordinary rotation, so it can hold sleeplocks and block on the disk
|
||
exactly as a syscall would. The network thread is the current user.
|
||
|
||
## Scheduling
|
||
|
||
Round-robin over the process table, one scheduler per CPU, each with its own
|
||
context. `swtch()` saves callee-saved registers and switches stacks; `sched()`
|
||
returns to the scheduler and asserts four invariants on the way — holding
|
||
`ptable.lock`, `ncli == 1`, not `RUNNING`, interrupts off.
|
||
|
||
When a full pass finds nothing runnable the CPU halts rather than spinning.
|
||
|
||
**Locking is a single `ptable.lock` over the whole table.** Not per-process: a
|
||
scheduling operation touches two processes (the one leaving and the one
|
||
arriving) and two-object locking needs an ordering rule that every path obeys.
|
||
The same reasoning produced a single lock over the socket table.
|
||
|
||
### sleep / wakeup
|
||
|
||
xv6's discipline, unchanged because it is correct: `sleep(chan, lk)` atomically
|
||
releases `lk` and blocks; `wakeup(chan)` makes every sleeper on that channel
|
||
runnable. The atomicity is what matters — a sleeper that tests its condition
|
||
under one lock while the waker sets it under another has no interlock and sleeps
|
||
through the wakeup meant for it.
|
||
|
||
## Traps and syscalls
|
||
|
||
Interrupts and exceptions enter through `vectors.S` → `trapasm.S` → `trap()`.
|
||
Syscalls enter through `SYSCALL`/`SYSRET` and `syscall_entry`, which is a
|
||
separate and faster path.
|
||
|
||
Every entry from user mode checks a **canary at the bottom of the kernel stack
|
||
page**. A deep or nested trap that overflows the 4 KB stack clobbers that word
|
||
first, so the failure is reported as `kstack overflow` at the point it happened
|
||
rather than as arbitrary corruption later.
|
||
|
||
One configuration detail with an expensive history: `MSR_STAR`'s `SYSRET`
|
||
selector field must hold the **compat 32-bit** code selector, because `SYSRETQ`
|
||
computes the returned `cs`/`ss` as `STAR[63:48]+16`/`+8`. Putting the real
|
||
64-bit user selector there makes every `sysretq` load an invalid pair — which
|
||
QEMU's TCG does not validate, so it stayed hidden until a hardware interrupt
|
||
forced a strictly-validated `IRETQ`.
|
||
|
||
## Signals
|
||
|
||
Real delivery, not stubs. `sys_rt_sigaction()` records the handler;
|
||
`deliver_signals()` runs in the tail of `syscall()`, just before returning to
|
||
user mode, and if a pending unblocked signal has a handler it rewrites the
|
||
trapframe to enter the handler instead of resuming — stashing the original in
|
||
`proc->saved_sigtf`. `sys_rt_sigreturn()` puts it back.
|
||
|
||
The return address is **not** `sa_restorer`. `SA_RESTORER` is not defined in
|
||
this musl build, so `sigaction()` never fills one in, and pushing 0 as a return
|
||
address crashes with `rip = 0` the moment the handler returns. Instead a 7-byte
|
||
trampoline (`mov eax,15; syscall`) is injected onto the user's own stack below
|
||
`rsp`, clear of the SysV red zone — the standard technique when the C library
|
||
supplies no restorer.
|
||
|
||
Signals without a handler fall back to a default action: termination for the
|
||
fatal set, dropped otherwise.
|
||
|
||
### PID 1 does not die from a signal it never asked to handle
|
||
|
||
`init_ignores()` discards any signal whose disposition is the default action
|
||
when the target is PID 1 — **including `SIGKILL`**. This is what POSIX requires
|
||
and what Linux does in `sig_task_ignored()`, and without it the system has a
|
||
single keystroke that kills it: Ctrl-C signals the foreground process group,
|
||
init is pgid 1, and any moment where the terminal's foreground group is still 1
|
||
lands SIGINT on init. It has no handler, so `exit()` runs, and `exit()` panics
|
||
outright on initproc.
|
||
|
||
Spamming Ctrl-C at a frozen program is exactly the input that finds that window.
|
||
|
||
## Process groups and sessions
|
||
|
||
`setsid`, `setpgid`, `getpgid`, `getsid`, and signal delivery to a whole group.
|
||
Job control is defined entirely in terms of these — a shell puts each pipeline
|
||
in its own process group and signals the group, never individual pids — and
|
||
mksh's job control could not make progress until they existed.
|
||
|
||
## What userspace gets
|
||
|
||
`fork`, `exec`, `wait`/`waitpid`/`wait4` with `WNOHANG` and real pid selection,
|
||
pipes, signals, `mmap` (anonymous and file-backed), POSIX shared memory, Unix
|
||
domain sockets, `poll`/`select`/`epoll`, `eventfd`, `timerfd`, `signalfd`.
|
||
|
||
`mmap` uses a bump allocator (`proc->mmap_next`, starting at `0x40000000`) so
|
||
every mapping gets its own disjoint, exactly-sized range. It previously assigned
|
||
a fixed 8 MB window per VMA slot regardless of the requested length, so a single
|
||
allocation over 8 MB mapped straight into a neighbouring live mapping.
|
||
|
||
---
|
||
|
||
## What it does not do
|
||
|
||
- **No slab allocator.** One page at a time, and every subsystem's fixed-size
|
||
table is a consequence. This is the single change that would most improve the
|
||
rest of the kernel.
|
||
- **No demand paging and no page cache.** `mmap` of a file *reads the whole
|
||
thing in* at map time. Writable `MAP_SHARED` of a file is therefore refused
|
||
outright rather than faked — see [../plan/foundations.md](../plan/foundations.md).
|
||
- **No dynamic linking.** `exec` handles `ET_EXEC` only: no `PT_INTERP`, no
|
||
`ET_DYN`, no shared libraries, no `dlopen`.
|
||
- **No threads.** `clone` exists but does not share an address space;
|
||
`gettid` returns the pid.
|
||
- **Permissions are tracked but not enforced.** uid/gid are maintained honestly
|
||
so `getuid()`/`setuid()` stop lying, and nothing consults them on access.
|
||
- **No swap**, and no memory pressure handling of any kind. `kalloc` failing is
|
||
reported up as `-ENOMEM` and that is the whole policy.
|
||
- **Nested signal delivery is not supported.** `in_signal` guards against
|
||
re-entering a handler while one is running.
|
||
- **Scheduling is round-robin with no priorities**, no fairness accounting, and
|
||
no CPU affinity despite `sched_setaffinity` existing.
|