---
name: schos-smp-userspace-corruption
description: The intermittent "zsh jumps to rip 0" crash was three shared globals on the syscall entry path — found, fixed, and the rule it generalises to.
metadata:
  type: project
---

**Fixed 2026-09-21.** Kept because the shape of the bug generalises.

Was: Roughly one to three processes per boot die with a
user-mode fault they could not have caused themselves. It has been written off
as "one zsh in ~5 boots makes a wild jump" for a while; it is more than that.

**It is SMP-only.** Three boots at `-smp 1`: zero faults. Three at `-smp 2`:
one, three, one. This is the single most useful fact about it and it was not
known before.

**It is not only zsh.** `schinitd` has taken one too, so it is not something
about the shell.

The two signatures seen, both from `kernel/arch/x86_64/trap.c`'s extended dump
(which now fires for `rip` below a page or a faulting address near zero, not
only for GP faults):

- `trap 14 err 20 rip 0x0 addr 0x0` — an instruction fetch at zero. A process
  does not branch there; it was handed a zero and followed it.
- `schinitd: trap 14 err 4 rip 0x5d7b addr 0xffffffffffffffea` — `-22`, which
  is `-EINVAL`. Something returned an error and the caller used it as a
  pointer.

**Ruled out: kernel stack depth.** This was the leading hypothesis and it is
wrong. Raising `KSTACKSIZE` to 8192 with static per-slot stacks did not change
the rate. The search was still worth it — the high-water mark really was 3736
of 4096 bytes in netd, and the AP scheduler stack really was being handed a
page it did not own — but neither was this.

Not yet examined: TLB coherence when a page table is modified on one CPU while
the process runs on another, `kalloc`'s free list under contention, and the
buffer cache. `switchuvm()` and the `p->tf` handling were both read and look
correct — `trap()` never overwrites `p->tf`, only `syscall()` does.

Reproduce: boot `-smp 2` and count `trap 1` lines in the serial log. Six tty
shells start at once at boot, which is when it almost always happens.

Related: [[schunix-smp-and-memory-bugs]], [[schunix-console-tty]].


## It was three globals on the syscall entry path

`syscall_entry` in `kernel/arch/x86_64/trapasm.S` has to swap `FS_BASE` from
the user's TLS to the kernel's before touching any `%fs:` data, and `wrmsr`
destroys `rax`, `rcx` and `rdx`. Those were parked in three **globals** —
`_sc_rax`, `_sc_rcx`, `_sc_rdx` in `vm.c` — under a comment reading "safe while
IF cleared by SYSCALL/SFMASK".

`IF` answers for an interrupt on *this* CPU. It says nothing about a second CPU
executing the same three stores at the same instant.

**`%rcx` is the user return address** — `SYSCALL` puts the user's `rip` there,
`SYSRET` takes it back. So the CPU that lost the race returned its process into
another process's instruction stream, with `%rax` (syscall number in, return
value out) swapped alongside it. `rip 0x0` was a process handed a zero and
following it; `0xffffffffffffffea` was `-EINVAL` in a stolen `%rax`, used as a
pointer.

Per-CPU words at `%gs:32..48` now, with a `_Static_assert` pinning all seven
offsets because the assembly spells them as literals.

Measured at four CPUs against the same pristine disk image each run: **10
faults across 10 boots before, 0 across 18 after.**

## The rule

This is the second bug of this exact shape in that file — `kernel_fs_base` was
the first, a single global every CPU wrote in its own `seginit()`. **On the
entry path, a global is a bug.** There is one per-CPU block; everything that
varies per CPU belongs in it. `%gs` is usable before the FS swap because GS base
is per-CPU in hardware and userspace cannot change it (`sys_arch_prctl`
implements `ARCH_SET_FS`/`ARCH_GET_FS` only).

## How to A/B a probabilistic kernel bug here

Rebuilding between runs dominates the clock. Build both variants once, save
`build/esp` for each, keep a pristine `schfs.img`, then each trial is just
restore-and-boot. Faults land during boot (six tty shells exec'ing at once), so
a 40-second timeout is enough — no typed input needed. `-smp 4` roughly doubles
the rate over `-smp 2`.


## Found while here: `$(command substitution)` hung — fixed 2026-09-22

**It was never the kernel.** `ports/zsh/Makefile` answered configure's
"does `kill(pid, 0)` return ESRCH correctly?" with `zsh_cv_sys_killesrch=no`,
which makes zsh `#define ESRCH EINVAL`. `waitforpid()` ends its wait on
`errno != ESRCH`; with the kernel correctly returning ESRCH (3) and zsh
comparing against EINVAL (22), the shell waited for ever on a child it had
already reaped.

Two things to carry forward, both from getting it wrong first:

- **`zsh_cv_*` names answer the check's *question*, not a property.** `no` on
  a check called "killesrch" reads as reassuring and means "broken".
  Autoconf's cross-compile default there is `yes` — the line should not have
  existed.
- **mksh installs as `/bin/sh`, not `mksh`.** A test invoking `mksh -c` is a
  `command not found`, and reading that as "mksh hangs too" is what kept the
  search in the kernel for an extra day.

The general lesson: when a probe reproducing a program's exact syscall
sequence behaves correctly and the program does not, stop looking under the
syscalls and look at how each was compiled.

Full write-up: `docs/history/kernel.md` section 7.