b012060699
`X=$(echo hi)` never returned. It has been in docs/history/kernel.md section 7
as "mksh's job control still doesn't reliably progress ... root cause not yet
isolated" with three suspects, and it was none of them, and it was not the
kernel at all.
ports/zsh/Makefile carries a block of `zsh_cv_*` answers for the checks
autoconf cannot run when cross-compiling. One of them was
zsh_cv_sys_killesrch=no
which reads like a reassuring property and is in fact the answer to the
question configure asks: *does kill(pid, 0) return ESRCH correctly?* `no` makes
configure emit BROKEN_KILL_ESRCH, and Src/zsh_system.h answers that with
#ifdef BROKEN_KILL_ESRCH
#undef ESRCH
#define ESRCH EINVAL
#endif
zsh's waitforpid() ends its wait on `errno != ESRCH` (Src/jobs.c:1569). With
ESRCH redefined to EINVAL (22) and the kernel correctly returning ESRCH (3),
that condition was true for ever: the shell sat in sigsuspend waiting for a
child it had already reaped. An instrumented zsh says the whole thing in one
line -- kill returned -1, errno is 3, and the constant it is compared against
is 22:
WFPkill -1 3 22
**Autoconf's own cross-compilation default for that check is `yes`.** Deleting
the line would have been correct. Writing it down as "a fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.
Verified: `X=$(echo inline)`, `$(/bin/echo external)`, backticks, nested
`$(echo $(echo deep))`, a slow child, and `$(ls /bin | wc -l)` all return.
mksh -- as /bin/sh -- was never affected.
Two corrections to yesterday's note, both confident and both wrong:
- "Not zsh, mksh hangs the same way" was false. mksh installs as /bin/sh,
so that test was a `command not found` whose output scrolled past. Had the
binary been named right, this would have localised to zsh's build in one
step instead of a day.
- cmd/sigtest had already proven the kernel's SIGCHLD/sigsuspend contract,
and a musl-linked probe had already shown kill(pid,0) returning -1 with
errno == ESRCH == 3 and zsh's own loop condition evaluating false. Both
results were right; both were read as "so it must be subtler in the
kernel". What they actually pointed at was the one place the probe and the
shell differed -- how each was compiled.
The lesson is narrow enough to state: when a probe that reproduces a program's
exact syscall sequence behaves correctly and the program does not, stop looking
under the syscalls and look at the build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5.2 KiB
| name | description | metadata | ||
|---|---|---|---|---|
| schos-smp-userspace-corruption | The intermittent "zsh jumps to rip 0" crash was three shared globals on the syscall entry path — found, fixed, and the rule it generalises to. |
|
Fixed 2026-09-21. Kept because the shape of the bug generalises.
Was: Roughly one to three processes per boot die with a user-mode fault they could not have caused themselves. It has been written off as "one zsh in ~5 boots makes a wild jump" for a while; it is more than that.
It is SMP-only. Three boots at -smp 1: zero faults. Three at -smp 2:
one, three, one. This is the single most useful fact about it and it was not
known before.
It is not only zsh. schinitd has taken one too, so it is not something
about the shell.
The two signatures seen, both from kernel/arch/x86_64/trap.c's extended dump
(which now fires for rip below a page or a faulting address near zero, not
only for GP faults):
trap 14 err 20 rip 0x0 addr 0x0— an instruction fetch at zero. A process does not branch there; it was handed a zero and followed it.schinitd: trap 14 err 4 rip 0x5d7b addr 0xffffffffffffffea—-22, which is-EINVAL. Something returned an error and the caller used it as a pointer.
Ruled out: kernel stack depth. This was the leading hypothesis and it is
wrong. Raising KSTACKSIZE to 8192 with static per-slot stacks did not change
the rate. The search was still worth it — the high-water mark really was 3736
of 4096 bytes in netd, and the AP scheduler stack really was being handed a
page it did not own — but neither was this.
Not yet examined: TLB coherence when a page table is modified on one CPU while
the process runs on another, kalloc's free list under contention, and the
buffer cache. switchuvm() and the p->tf handling were both read and look
correct — trap() never overwrites p->tf, only syscall() does.
Reproduce: boot -smp 2 and count trap 1 lines in the serial log. Six tty
shells start at once at boot, which is when it almost always happens.
Related: schunix-smp-and-memory-bugs, schunix-console-tty.
It was three globals on the syscall entry path
syscall_entry in kernel/arch/x86_64/trapasm.S has to swap FS_BASE from
the user's TLS to the kernel's before touching any %fs: data, and wrmsr
destroys rax, rcx and rdx. Those were parked in three globals —
_sc_rax, _sc_rcx, _sc_rdx in vm.c — under a comment reading "safe while
IF cleared by SYSCALL/SFMASK".
IF answers for an interrupt on this CPU. It says nothing about a second CPU
executing the same three stores at the same instant.
%rcx is the user return address — SYSCALL puts the user's rip there,
SYSRET takes it back. So the CPU that lost the race returned its process into
another process's instruction stream, with %rax (syscall number in, return
value out) swapped alongside it. rip 0x0 was a process handed a zero and
following it; 0xffffffffffffffea was -EINVAL in a stolen %rax, used as a
pointer.
Per-CPU words at %gs:32..48 now, with a _Static_assert pinning all seven
offsets because the assembly spells them as literals.
Measured at four CPUs against the same pristine disk image each run: 10 faults across 10 boots before, 0 across 18 after.
The rule
This is the second bug of this exact shape in that file — kernel_fs_base was
the first, a single global every CPU wrote in its own seginit(). On the
entry path, a global is a bug. There is one per-CPU block; everything that
varies per CPU belongs in it. %gs is usable before the FS swap because GS base
is per-CPU in hardware and userspace cannot change it (sys_arch_prctl
implements ARCH_SET_FS/ARCH_GET_FS only).
How to A/B a probabilistic kernel bug here
Rebuilding between runs dominates the clock. Build both variants once, save
build/esp for each, keep a pristine schfs.img, then each trial is just
restore-and-boot. Faults land during boot (six tty shells exec'ing at once), so
a 40-second timeout is enough — no typed input needed. -smp 4 roughly doubles
the rate over -smp 2.
Found while here: $(command substitution) hung — fixed 2026-09-22
It was never the kernel. ports/zsh/Makefile answered configure's
"does kill(pid, 0) return ESRCH correctly?" with zsh_cv_sys_killesrch=no,
which makes zsh #define ESRCH EINVAL. waitforpid() ends its wait on
errno != ESRCH; with the kernel correctly returning ESRCH (3) and zsh
comparing against EINVAL (22), the shell waited for ever on a child it had
already reaped.
Two things to carry forward, both from getting it wrong first:
zsh_cv_*names answer the check's question, not a property.noon a check called "killesrch" reads as reassuring and means "broken". Autoconf's cross-compile default there isyes— the line should not have existed.- mksh installs as
/bin/sh, notmksh. A test invokingmksh -cis acommand not found, and reading that as "mksh hangs too" is what kept the search in the kernel for an extra day.
The general lesson: when a probe reproducing a program's exact syscall sequence behaves correctly and the program does not, stop looking under the syscalls and look at how each was compiled.
Full write-up: docs/history/kernel.md section 7.