---
name: schos-smp-userspace-corruption
description: The intermittent "zsh jumps to rip 0" crash was three shared globals on the syscall entry path — found, fixed, and the rule it generalises to.
metadata:
  type: project
---

**Fixed 2026-09-21.** Kept because the shape of the bug generalises.

Was: Roughly one to three processes per boot die with a
user-mode fault they could not have caused themselves. It has been written off
as "one zsh in ~5 boots makes a wild jump" for a while; it is more than that.

**It is SMP-only.** Three boots at `-smp 1`: zero faults. Three at `-smp 2`:
one, three, one. This is the single most useful fact about it and it was not
known before.

**It is not only zsh.** `schinitd` has taken one too, so it is not something
about the shell.

The two signatures seen, both from `kernel/arch/x86_64/trap.c`'s extended dump
(which now fires for `rip` below a page or a faulting address near zero, not
only for GP faults):

- `trap 14 err 20 rip 0x0 addr 0x0` — an instruction fetch at zero. A process
  does not branch there; it was handed a zero and followed it.
- `schinitd: trap 14 err 4 rip 0x5d7b addr 0xffffffffffffffea` — `-22`, which
  is `-EINVAL`. Something returned an error and the caller used it as a
  pointer.

**Ruled out: kernel stack depth.** This was the leading hypothesis and it is
wrong. Raising `KSTACKSIZE` to 8192 with static per-slot stacks did not change
the rate. The search was still worth it — the high-water mark really was 3736
of 4096 bytes in netd, and the AP scheduler stack really was being handed a
page it did not own — but neither was this.

Not yet examined: TLB coherence when a page table is modified on one CPU while
the process runs on another, `kalloc`'s free list under contention, and the
buffer cache. `switchuvm()` and the `p->tf` handling were both read and look
correct — `trap()` never overwrites `p->tf`, only `syscall()` does.

Reproduce: boot `-smp 2` and count `trap 1` lines in the serial log. Six tty
shells start at once at boot, which is when it almost always happens.

Related: [[schunix-smp-and-memory-bugs]], [[schunix-console-tty]].


## It was three globals on the syscall entry path

`syscall_entry` in `kernel/arch/x86_64/trapasm.S` has to swap `FS_BASE` from
the user's TLS to the kernel's before touching any `%fs:` data, and `wrmsr`
destroys `rax`, `rcx` and `rdx`. Those were parked in three **globals** —
`_sc_rax`, `_sc_rcx`, `_sc_rdx` in `vm.c` — under a comment reading "safe while
IF cleared by SYSCALL/SFMASK".

`IF` answers for an interrupt on *this* CPU. It says nothing about a second CPU
executing the same three stores at the same instant.

**`%rcx` is the user return address** — `SYSCALL` puts the user's `rip` there,
`SYSRET` takes it back. So the CPU that lost the race returned its process into
another process's instruction stream, with `%rax` (syscall number in, return
value out) swapped alongside it. `rip 0x0` was a process handed a zero and
following it; `0xffffffffffffffea` was `-EINVAL` in a stolen `%rax`, used as a
pointer.

Per-CPU words at `%gs:32..48` now, with a `_Static_assert` pinning all seven
offsets because the assembly spells them as literals.

Measured at four CPUs against the same pristine disk image each run: **10
faults across 10 boots before, 0 across 18 after.**

## The rule

This is the second bug of this exact shape in that file — `kernel_fs_base` was
the first, a single global every CPU wrote in its own `seginit()`. **On the
entry path, a global is a bug.** There is one per-CPU block; everything that
varies per CPU belongs in it. `%gs` is usable before the FS swap because GS base
is per-CPU in hardware and userspace cannot change it (`sys_arch_prctl`
implements `ARCH_SET_FS`/`ARCH_GET_FS` only).

## How to A/B a probabilistic kernel bug here

Rebuilding between runs dominates the clock. Build both variants once, save
`build/esp` for each, keep a pristine `schfs.img`, then each trial is just
restore-and-boot. Faults land during boot (six tty shells exec'ing at once), so
a 40-second timeout is enough — no typed input needed. `-smp 4` roughly doubles
the rate over `-smp 2`.


## Found while here: `$(command substitution)` hangs every shell

Unrelated to the above and **still open**. `X=$(echo hi)` never returns, in zsh
*and* mksh, on 1 CPU and on 4, interactive and not, with `set +m`. Pipelines,
subshells, background jobs and foreground externals are all fine.

Ruled out: `setpgid` (real now), SMP, job control, zsh specifically, and the
kernel's SIGCHLD/sigsuspend contract — `cmd/sigtest` proves that one and passes
all eight checks. A full syscall trace of a hang has every return value
correct.

The contradiction is at `Src/jobs.c:1569`: zsh's `waitforpid()` ends its loop
on `ESRCH` from `kill(pid, 0)`, and the trace shows that `kill` returning ESRCH
with the loop entered anyway. Next step is a musl-linked probe that does
`fork; let the handler reap; kill(pid,0); print errno` — everything upstream of
that line is known good.

Documented in full at `docs/history/kernel.md` section 7, which is where to
start.