a053a96add
Went looking for why `X=$(echo hi)` hangs every shell. Did not find it. Found
two other things, and built the instrument that rules out the largest suspect.
**The signal return trampoline was ABI-blind.** deliver_signals() manufactures
a return address for the handler -- seven bytes on the user stack reading
`mov eax, 15 ; syscall`. Fifteen is *Linux's* rt_sigreturn, which is right for
anything musl-linked, and that is every port and every shell. For a first-party
binary it is native syscall 15, which is `open`. So a handler in any cmd/
program ran, returned into the trampoline, called open() with whatever the
handler had left in its registers, got -ENOENT, and carried on executing the
stack bytes after the trampoline.
pid 14 sigtest: trap 14 err 4 rip 0x46ca7 addr 0xfffffffffffffffe
rax=fffffffffffffffe ... rsp=46ca0
ustack: 46e30 46ca0 50f0000000fb8 0 0 0
rip is rsp+7 -- one past a seven-byte trampoline -- and the third word of the
stack is the trampoline's own bytes. It emits the number for the process's ABI
now; syscall_map is the thing that says which numbering this process speaks.
**SIG_BLOCK, SIG_UNBLOCK and SIG_SETMASK were not defined anywhere.**
sys_sigprocmask branched on bare 0, 1 and 2 with the names in comments, so
anything outside the kernel had to spell the number and hope it had the right
one. Defined in posix.h, used in the kernel.
**cmd/sigtest** asserts the SIGCHLD/sigsuspend contract every shell is built
on, and it is written so a permissive stub cannot pass: a blocked SIGCHLD stays
pending, sigsuspend delivers an *already pending* signal without spinning, the
handler runs exactly once, the mask is restored, the child is still reapable,
and a signal arriving mid-wait also wakes it. Eight checks, all green. The
kernel's half of that contract is now proven rather than assumed, which is what
lets the remaining bug be pursued somewhere else.
Along with it, `rt_sigsuspend` in lib/usys.S -- the syscall existed and the
first-party library simply never exposed it.
On the hang itself: it is not job control, not SMP, not zsh (mksh does it too),
and not any value the kernel returns -- a full syscall trace is in
docs/history/kernel.md section 7, which also records what the contradiction
narrows to. That section had said "root cause not yet isolated" with three
suspects; all three are now eliminated, and the note says what to do next
instead of what to guess.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5 KiB
| name | description | metadata | ||
|---|---|---|---|---|
| schos-smp-userspace-corruption | The intermittent "zsh jumps to rip 0" crash was three shared globals on the syscall entry path — found, fixed, and the rule it generalises to. |
|
Fixed 2026-09-21. Kept because the shape of the bug generalises.
Was: Roughly one to three processes per boot die with a user-mode fault they could not have caused themselves. It has been written off as "one zsh in ~5 boots makes a wild jump" for a while; it is more than that.
It is SMP-only. Three boots at -smp 1: zero faults. Three at -smp 2:
one, three, one. This is the single most useful fact about it and it was not
known before.
It is not only zsh. schinitd has taken one too, so it is not something
about the shell.
The two signatures seen, both from kernel/arch/x86_64/trap.c's extended dump
(which now fires for rip below a page or a faulting address near zero, not
only for GP faults):
trap 14 err 20 rip 0x0 addr 0x0— an instruction fetch at zero. A process does not branch there; it was handed a zero and followed it.schinitd: trap 14 err 4 rip 0x5d7b addr 0xffffffffffffffea—-22, which is-EINVAL. Something returned an error and the caller used it as a pointer.
Ruled out: kernel stack depth. This was the leading hypothesis and it is
wrong. Raising KSTACKSIZE to 8192 with static per-slot stacks did not change
the rate. The search was still worth it — the high-water mark really was 3736
of 4096 bytes in netd, and the AP scheduler stack really was being handed a
page it did not own — but neither was this.
Not yet examined: TLB coherence when a page table is modified on one CPU while
the process runs on another, kalloc's free list under contention, and the
buffer cache. switchuvm() and the p->tf handling were both read and look
correct — trap() never overwrites p->tf, only syscall() does.
Reproduce: boot -smp 2 and count trap 1 lines in the serial log. Six tty
shells start at once at boot, which is when it almost always happens.
Related: schunix-smp-and-memory-bugs, schunix-console-tty.
It was three globals on the syscall entry path
syscall_entry in kernel/arch/x86_64/trapasm.S has to swap FS_BASE from
the user's TLS to the kernel's before touching any %fs: data, and wrmsr
destroys rax, rcx and rdx. Those were parked in three globals —
_sc_rax, _sc_rcx, _sc_rdx in vm.c — under a comment reading "safe while
IF cleared by SYSCALL/SFMASK".
IF answers for an interrupt on this CPU. It says nothing about a second CPU
executing the same three stores at the same instant.
%rcx is the user return address — SYSCALL puts the user's rip there,
SYSRET takes it back. So the CPU that lost the race returned its process into
another process's instruction stream, with %rax (syscall number in, return
value out) swapped alongside it. rip 0x0 was a process handed a zero and
following it; 0xffffffffffffffea was -EINVAL in a stolen %rax, used as a
pointer.
Per-CPU words at %gs:32..48 now, with a _Static_assert pinning all seven
offsets because the assembly spells them as literals.
Measured at four CPUs against the same pristine disk image each run: 10 faults across 10 boots before, 0 across 18 after.
The rule
This is the second bug of this exact shape in that file — kernel_fs_base was
the first, a single global every CPU wrote in its own seginit(). On the
entry path, a global is a bug. There is one per-CPU block; everything that
varies per CPU belongs in it. %gs is usable before the FS swap because GS base
is per-CPU in hardware and userspace cannot change it (sys_arch_prctl
implements ARCH_SET_FS/ARCH_GET_FS only).
How to A/B a probabilistic kernel bug here
Rebuilding between runs dominates the clock. Build both variants once, save
build/esp for each, keep a pristine schfs.img, then each trial is just
restore-and-boot. Faults land during boot (six tty shells exec'ing at once), so
a 40-second timeout is enough — no typed input needed. -smp 4 roughly doubles
the rate over -smp 2.
Found while here: $(command substitution) hangs every shell
Unrelated to the above and still open. X=$(echo hi) never returns, in zsh
and mksh, on 1 CPU and on 4, interactive and not, with set +m. Pipelines,
subshells, background jobs and foreground externals are all fine.
Ruled out: setpgid (real now), SMP, job control, zsh specifically, and the
kernel's SIGCHLD/sigsuspend contract — cmd/sigtest proves that one and passes
all eight checks. A full syscall trace of a hang has every return value
correct.
The contradiction is at Src/jobs.c:1569: zsh's waitforpid() ends its loop
on ESRCH from kill(pid, 0), and the trace shows that kill returning ESRCH
with the loop entered anyway. Next step is a musl-linked probe that does
fork; let the handler reap; kill(pid,0); print errno — everything upstream of
that line is known good.
Documented in full at docs/history/kernel.md section 7, which is where to
start.