3185685997
One to four processes died per boot with a fault they could not have caused:
`rip 0x0` -- an instruction fetch at zero, which a process does not branch to
-- or a dereference of `0xffffffffffffffea`, which is `-EINVAL`. It hit shells
and schinitd alike and had been carried for a long time as "zsh sometimes jumps
into .data".
`syscall_entry` has to swap FS_BASE from the user's TLS to the kernel's before
it can touch any `%fs:` data, and `wrmsr` destroys rax, rcx and rdx. Those have
to be parked somewhere first -- not on the user stack, which is untrusted and
has not been switched away from yet. They were parked in three globals:
// Single-CPU scratch for syscall_entry FS swap
// (safe while IF cleared by SYSCALL/SFMASK)
uint64 _sc_rax, _sc_rcx, _sc_rdx;
IF being clear answers for an interrupt arriving on *this* CPU. It says nothing
whatever about a second CPU executing the same three stores at the same
instant, and with more than one processor entering syscalls that is not a rare
interleaving -- it is the ordinary one.
**%rcx is the user return address.** SYSCALL puts the user's rip there and
SYSRET takes it back from there. So the CPU that lost the race returned its
process into *another process's* instruction stream, with %rax -- the syscall
number on the way in, the return value on the way out -- swapped alongside it.
Both symptoms fall straight out of that: a zero handed over and followed, and
an errno used as a pointer.
Per-CPU words at %gs:32..48 now. %gs is usable there, before the FS swap, for
exactly the reason RESTORE_KERNEL_FS can already read %gs:0: GS base is per-CPU
in hardware, seginit() sets it once, and userspace cannot change it --
sys_arch_prctl() implements ARCH_SET_FS and ARCH_GET_FS and rejects the rest.
Measured rather than argued. Both variants built once, `build/esp` saved for
each, the same pristine schfs.img restored before every boot, -smp 4:
three globals 10 boots 10 faults, in five of the ten
three per-CPU 18 boots 0 faults
A _Static_assert pins all seven per-CPU offsets, because trapasm.S spells them
as literals and cannot include a header -- a slot that moves would leave a
register quietly holding somebody else's value, which is the hardest possible
thing to find.
**This is the second bug of this exact shape in this file.** The first was
`kernel_fs_base`, a single global every CPU wrote during its own seginit(), so
the last writer won and every CPU restored one CPU's per-CPU block;
RESTORE_KERNEL_FS reads %gs:0 for that reason. The lesson generalises and is
now written down in both places: **on the entry path, a global is a bug.**
Closes the row added to Gate 0 four commits ago. It was the right row to put
first -- a compositor built on this would have failed in ways indistinguishable
from its own bugs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4.1 KiB
| name | description | metadata | ||
|---|---|---|---|---|
| schos-smp-userspace-corruption | The intermittent "zsh jumps to rip 0" crash was three shared globals on the syscall entry path — found, fixed, and the rule it generalises to. |
|
Fixed 2026-09-21. Kept because the shape of the bug generalises.
Was: Roughly one to three processes per boot die with a user-mode fault they could not have caused themselves. It has been written off as "one zsh in ~5 boots makes a wild jump" for a while; it is more than that.
It is SMP-only. Three boots at -smp 1: zero faults. Three at -smp 2:
one, three, one. This is the single most useful fact about it and it was not
known before.
It is not only zsh. schinitd has taken one too, so it is not something
about the shell.
The two signatures seen, both from kernel/arch/x86_64/trap.c's extended dump
(which now fires for rip below a page or a faulting address near zero, not
only for GP faults):
trap 14 err 20 rip 0x0 addr 0x0— an instruction fetch at zero. A process does not branch there; it was handed a zero and followed it.schinitd: trap 14 err 4 rip 0x5d7b addr 0xffffffffffffffea—-22, which is-EINVAL. Something returned an error and the caller used it as a pointer.
Ruled out: kernel stack depth. This was the leading hypothesis and it is
wrong. Raising KSTACKSIZE to 8192 with static per-slot stacks did not change
the rate. The search was still worth it — the high-water mark really was 3736
of 4096 bytes in netd, and the AP scheduler stack really was being handed a
page it did not own — but neither was this.
Not yet examined: TLB coherence when a page table is modified on one CPU while
the process runs on another, kalloc's free list under contention, and the
buffer cache. switchuvm() and the p->tf handling were both read and look
correct — trap() never overwrites p->tf, only syscall() does.
Reproduce: boot -smp 2 and count trap 1 lines in the serial log. Six tty
shells start at once at boot, which is when it almost always happens.
Related: schunix-smp-and-memory-bugs, schunix-console-tty.
It was three globals on the syscall entry path
syscall_entry in kernel/arch/x86_64/trapasm.S has to swap FS_BASE from
the user's TLS to the kernel's before touching any %fs: data, and wrmsr
destroys rax, rcx and rdx. Those were parked in three globals —
_sc_rax, _sc_rcx, _sc_rdx in vm.c — under a comment reading "safe while
IF cleared by SYSCALL/SFMASK".
IF answers for an interrupt on this CPU. It says nothing about a second CPU
executing the same three stores at the same instant.
%rcx is the user return address — SYSCALL puts the user's rip there,
SYSRET takes it back. So the CPU that lost the race returned its process into
another process's instruction stream, with %rax (syscall number in, return
value out) swapped alongside it. rip 0x0 was a process handed a zero and
following it; 0xffffffffffffffea was -EINVAL in a stolen %rax, used as a
pointer.
Per-CPU words at %gs:32..48 now, with a _Static_assert pinning all seven
offsets because the assembly spells them as literals.
Measured at four CPUs against the same pristine disk image each run: 10 faults across 10 boots before, 0 across 18 after.
The rule
This is the second bug of this exact shape in that file — kernel_fs_base was
the first, a single global every CPU wrote in its own seginit(). On the
entry path, a global is a bug. There is one per-CPU block; everything that
varies per CPU belongs in it. %gs is usable before the FS swap because GS base
is per-CPU in hardware and userspace cannot change it (sys_arch_prctl
implements ARCH_SET_FS/ARCH_GET_FS only).
How to A/B a probabilistic kernel bug here
Rebuilding between runs dominates the clock. Build both variants once, save
build/esp for each, keep a pristine schfs.img, then each trial is just
restore-and-boot. Faults land during boot (six tty shells exec'ing at once), so
a 40-second timeout is enough — no typed input needed. -smp 4 roughly doubles
the rate over -smp 2.