syscall: three globals on the entry path, shared by every CPU

One to four processes died per boot with a fault they could not have caused:
`rip 0x0` -- an instruction fetch at zero, which a process does not branch to
-- or a dereference of `0xffffffffffffffea`, which is `-EINVAL`. It hit shells
and schinitd alike and had been carried for a long time as "zsh sometimes jumps
into .data".

`syscall_entry` has to swap FS_BASE from the user's TLS to the kernel's before
it can touch any `%fs:` data, and `wrmsr` destroys rax, rcx and rdx. Those have
to be parked somewhere first -- not on the user stack, which is untrusted and
has not been switched away from yet. They were parked in three globals:

    // Single-CPU scratch for syscall_entry FS swap
    // (safe while IF cleared by SYSCALL/SFMASK)
    uint64 _sc_rax, _sc_rcx, _sc_rdx;

IF being clear answers for an interrupt arriving on *this* CPU. It says nothing
whatever about a second CPU executing the same three stores at the same
instant, and with more than one processor entering syscalls that is not a rare
interleaving -- it is the ordinary one.

**%rcx is the user return address.** SYSCALL puts the user's rip there and
SYSRET takes it back from there. So the CPU that lost the race returned its
process into *another process's* instruction stream, with %rax -- the syscall
number on the way in, the return value on the way out -- swapped alongside it.
Both symptoms fall straight out of that: a zero handed over and followed, and
an errno used as a pointer.

Per-CPU words at %gs:32..48 now. %gs is usable there, before the FS swap, for
exactly the reason RESTORE_KERNEL_FS can already read %gs:0: GS base is per-CPU
in hardware, seginit() sets it once, and userspace cannot change it --
sys_arch_prctl() implements ARCH_SET_FS and ARCH_GET_FS and rejects the rest.

Measured rather than argued. Both variants built once, `build/esp` saved for
each, the same pristine schfs.img restored before every boot, -smp 4:

    three globals     10 boots    10 faults, in five of the ten
    three per-CPU     18 boots     0 faults

A _Static_assert pins all seven per-CPU offsets, because trapasm.S spells them
as literals and cannot include a header -- a slot that moves would leave a
register quietly holding somebody else's value, which is the hardest possible
thing to find.

**This is the second bug of this exact shape in this file.** The first was
`kernel_fs_base`, a single global every CPU wrote during its own seginit(), so
the last writer won and every CPU restored one CPU's per-CPU block;
RESTORE_KERNEL_FS reads %gs:0 for that reason. The lesson generalises and is
now written down in both places: **on the entry path, a global is a bug.**

Closes the row added to Gate 0 four commits ago. It was the right row to put
first -- a compositor built on this would have failed in ways indistinguishable
from its own bugs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-09-21 22:56:45 +02:00
commit 3185685997
6 changed files with 173 additions and 36 deletions

View file

@ -1,11 +1,13 @@
---
name: schos-smp-userspace-corruption
description: The intermittent "zsh jumps to rip 0" crash is SMP-only and still open — what has been ruled out, and the two facts worth starting from.
description: The intermittent "zsh jumps to rip 0" crash was three shared globals on the syscall entry path — found, fixed, and the rule it generalises to.
metadata:
type: project
---
Open as of 2026-09-21. Roughly one to three processes per boot die with a
**Fixed 2026-09-21.** Kept because the shape of the bug generalises.
Was: Roughly one to three processes per boot die with a
user-mode fault they could not have caused themselves. It has been written off
as "one zsh in ~5 boots makes a wild jump" for a while; it is more than that.
@ -41,3 +43,45 @@ Reproduce: boot `-smp 2` and count `trap 1` lines in the serial log. Six tty
shells start at once at boot, which is when it almost always happens.
Related: [[schunix-smp-and-memory-bugs]], [[schunix-console-tty]].
## It was three globals on the syscall entry path
`syscall_entry` in `kernel/arch/x86_64/trapasm.S` has to swap `FS_BASE` from
the user's TLS to the kernel's before touching any `%fs:` data, and `wrmsr`
destroys `rax`, `rcx` and `rdx`. Those were parked in three **globals** —
`_sc_rax`, `_sc_rcx`, `_sc_rdx` in `vm.c` — under a comment reading "safe while
IF cleared by SYSCALL/SFMASK".
`IF` answers for an interrupt on *this* CPU. It says nothing about a second CPU
executing the same three stores at the same instant.
**`%rcx` is the user return address** — `SYSCALL` puts the user's `rip` there,
`SYSRET` takes it back. So the CPU that lost the race returned its process into
another process's instruction stream, with `%rax` (syscall number in, return
value out) swapped alongside it. `rip 0x0` was a process handed a zero and
following it; `0xffffffffffffffea` was `-EINVAL` in a stolen `%rax`, used as a
pointer.
Per-CPU words at `%gs:32..48` now, with a `_Static_assert` pinning all seven
offsets because the assembly spells them as literals.
Measured at four CPUs against the same pristine disk image each run: **10
faults across 10 boots before, 0 across 18 after.**
## The rule
This is the second bug of this exact shape in that file — `kernel_fs_base` was
the first, a single global every CPU wrote in its own `seginit()`. **On the
entry path, a global is a bug.** There is one per-CPU block; everything that
varies per CPU belongs in it. `%gs` is usable before the FS swap because GS base
is per-CPU in hardware and userspace cannot change it (`sys_arch_prctl`
implements `ARCH_SET_FS`/`ARCH_GET_FS` only).
## How to A/B a probabilistic kernel bug here
Rebuilding between runs dominates the clock. Build both variants once, save
`build/esp` for each, keep a pristine `schfs.img`, then each trial is just
restore-and-boot. Faults land during boot (six tty shells exec'ing at once), so
a 40-second timeout is enough — no typed input needed. `-smp 4` roughly doubles
the rate over `-smp 2`.