diff --git a/.claude/memory/schos-smp-userspace-corruption.md b/.claude/memory/schos-smp-userspace-corruption.md index 7979f0d..4b5821e 100644 --- a/.claude/memory/schos-smp-userspace-corruption.md +++ b/.claude/memory/schos-smp-userspace-corruption.md @@ -87,22 +87,27 @@ a 40-second timeout is enough — no typed input needed. `-smp 4` roughly double the rate over `-smp 2`. -## Found while here: `$(command substitution)` hangs every shell +## Found while here: `$(command substitution)` hung — fixed 2026-09-22 -Unrelated to the above and **still open**. `X=$(echo hi)` never returns, in zsh -*and* mksh, on 1 CPU and on 4, interactive and not, with `set +m`. Pipelines, -subshells, background jobs and foreground externals are all fine. +**It was never the kernel.** `ports/zsh/Makefile` answered configure's +"does `kill(pid, 0)` return ESRCH correctly?" with `zsh_cv_sys_killesrch=no`, +which makes zsh `#define ESRCH EINVAL`. `waitforpid()` ends its wait on +`errno != ESRCH`; with the kernel correctly returning ESRCH (3) and zsh +comparing against EINVAL (22), the shell waited for ever on a child it had +already reaped. -Ruled out: `setpgid` (real now), SMP, job control, zsh specifically, and the -kernel's SIGCHLD/sigsuspend contract — `cmd/sigtest` proves that one and passes -all eight checks. A full syscall trace of a hang has every return value -correct. +Two things to carry forward, both from getting it wrong first: -The contradiction is at `Src/jobs.c:1569`: zsh's `waitforpid()` ends its loop -on `ESRCH` from `kill(pid, 0)`, and the trace shows that `kill` returning ESRCH -with the loop entered anyway. Next step is a musl-linked probe that does -`fork; let the handler reap; kill(pid,0); print errno` — everything upstream of -that line is known good. +- **`zsh_cv_*` names answer the check's *question*, not a property.** `no` on + a check called "killesrch" reads as reassuring and means "broken". + Autoconf's cross-compile default there is `yes` — the line should not have + existed. +- **mksh installs as `/bin/sh`, not `mksh`.** A test invoking `mksh -c` is a + `command not found`, and reading that as "mksh hangs too" is what kept the + search in the kernel for an extra day. -Documented in full at `docs/history/kernel.md` section 7, which is where to -start. +The general lesson: when a probe reproducing a program's exact syscall +sequence behaves correctly and the program does not, stop looking under the +syscalls and look at how each was compiled. + +Full write-up: `docs/history/kernel.md` section 7. diff --git a/docs/history/kernel.md b/docs/history/kernel.md index 0604597..0b51c5d 100644 --- a/docs/history/kernel.md +++ b/docs/history/kernel.md @@ -432,85 +432,78 @@ likelihood: its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails to find the job for reasons not yet traced. -### 2026-09-21: narrowed a long way, still open +### Resolved 2026-09-22: it was never the kernel -Picked up again with the debugger that did not exist when the above was -written. What is now *ruled out* is most of what that note suspected: +**`X=$(...)` hung because zsh was built with `ESRCH` defined as `EINVAL`.** -- **Not `setpgid`/`getpgid`.** Both are real now — `proc_setpgid`, `p->pgid`, - `p->sid` — and `waitpid()` honours the POSIX pid selection rules. -- **Not signal delivery, and not the sigsuspend contract.** `cmd/sigtest` - asserts the whole pattern every shell uses and passes all eight checks: a - blocked SIGCHLD stays pending, `sigsuspend` delivers an *already pending* - signal without spinning, the handler runs exactly once, the mask is restored, - and the child is still reapable afterwards. A permissive stub cannot pass it. -- **Not SMP.** It hangs identically on `-smp 1` and `-smp 4`. -- **Not job control.** `set +m` makes no difference, and a non-interactive - `zsh -c` hangs the same way. -- **Not zsh.** `mksh -c 'X=$(echo hi)'` hangs in the same place. -- **Not the values the kernel returns.** The full syscall trace of a hang is - below and every one of them is correct. +`ports/zsh/Makefile` carries a block of `zsh_cv_*` cache answers, for the +checks autoconf cannot run when cross-compiling. One of them was -The symptom is narrower than "job control": it is **`$(command -substitution)`**, and only that. Pipelines work. Subshells work. Background -jobs and `wait` work. Foreground external commands work. - -A complete trace of `X=$(echo inline)` in zsh, from the fork to the hang: - -``` -fork() -> 14 -sigprocmask x2 (child_block) -close(pipe write end) -sigprocmask(SIG_UNBLOCK, SIGCHLD) (child_unblock) - -> SIGCHLD delivered to zsh's handler - wait4(-1, WNOHANG|WUNTRACED|WCONTINUED) -> 14, status 0 - getrusage() -> 0 - wait4(same) -> -ECHILD - rt_sigreturn -read(pipe) -> 7 ("inline\n") -read(pipe) -> 0 (EOF) -sigprocmask, close -kill(14, 0) -> -ESRCH -rt_sigsuspend(mask=SIGINT) -> never returns +```make +zsh_cv_sys_killesrch=no ``` -Everything there is right. The child is reaped with status 0, the pipe carries -its output and closes, and `kill` on the reaped pid correctly says ESRCH. - -**And that last pair is the contradiction.** zsh's `waitforpid()` -(`Src/jobs.c:1569`) is +Read as a property that looks reassuring. Read as the answer to the question +configure actually asks — *"does `kill(pid, 0)` return ESRCH correctly?"* — it +says the opposite. `no` makes configure emit `BROKEN_KILL_ESRCH`, and +`Src/zsh_system.h` answers that with: ```c -while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) { - ... - signal_suspend(SIGCHLD, wait_cmd); +#ifdef BROKEN_KILL_ESRCH +#undef ESRCH +#define ESRCH EINVAL +#endif ``` -with a comment above it saying, in as many words, that ESRCH from that `kill` -is how the loop ends. The trace shows the `kill` returning ESRCH **and the -loop being entered anyway**, which leaves only two possibilities: `errno` is -not `ESRCH` when zsh reads it, or `kill()` did not return negative. errno is -demonstrably fine elsewhere — `kill -0 31000` prints "no such process" from -both zsh's builtin and toybox's `kill`, and `cat` of a missing file prints "No -such file or directory". +zsh's `waitforpid()` (`Src/jobs.c:1569`) ends its wait on -That is where it stands. The next step is to distinguish those two: a musl- -linked probe that does exactly `fork; wait for the handler to reap; kill(pid, -0); print errno`, because everything upstream of that line is now known good. +```c +while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) +``` -Two real bugs were found on the way and fixed, neither of them this one: +so with `ESRCH` redefined to `EINVAL` (22) and the kernel correctly returning +`ESRCH` (3), that condition was true for ever. The shell waited, in +`sigsuspend`, for a child it had already reaped. Every command substitution in +the shell hung. -- **The signal return trampoline was ABI-blind.** `deliver_signals()` wrote - `mov eax, 15 ; syscall` as the handler's return address. Fifteen is *Linux's* - `rt_sigreturn`, correct for anything musl-linked; for a first-party binary it - is native syscall 15, which is `open`. So a handler in any `cmd/` program - ran, returned into the trampoline, called `open()` with the handler's - leftover registers, got `-ENOENT`, and carried on executing the stack bytes - after the trampoline. It emits the right number for the process's ABI now. - This is why `cmd/sigtest` had to exist before anything else could be trusted. -- **`SIG_BLOCK`, `SIG_UNBLOCK` and `SIG_SETMASK` were not defined anywhere.** - The kernel branched on bare `0`, `1` and `2` with the names in comments, so - anything outside the kernel had to spell the number and hope. +The instrumented build says it in one line — `kill` returned −1, `errno` is 3, +and the constant it is compared against is 22: + +``` +WFPkill -1 3 22 +``` + +**Autoconf's own cross-compilation default for this check is `yes`.** Deleting +the line would have been correct. Writing it down as a "fact about SchOS" is +what got it wrong, and the block's comment now says to read each name as the +question rather than as a property. + +#### What this cost, and what to take from it + +This sat in section 7 as "mksh's job control still doesn't reliably progress +... root cause not yet isolated" with three suspects, none of which were it. +All three have since been eliminated on their own merits — `setpgid` and +`getpgid` are real, `getrusage` is not consulted on the path, and the pid +matching was never reached. + +Two corrections to the note written on 2026-09-21, both worth keeping because +they were confident and wrong: + +- **"Not zsh — `mksh -c` hangs the same way" was false.** mksh installs as + `/bin/sh`, not `mksh`, so that test was a `command not found` whose output + scrolled past. mksh was never affected. Had the binary been named correctly + the bug would have been localised to zsh's build in one step. +- **The kernel was cleared and then still suspected.** `cmd/sigtest` had + already proven the SIGCHLD/sigsuspend contract, and a musl-linked probe had + already shown `kill(pid, 0)` returning −1 with `errno == ESRCH == 3` and + zsh's own loop condition evaluating false. Both results were correct and + both were read as "so it must be something subtler in the kernel". The thing + they actually pointed at was the one place the probe and the shell differed: + **how each was compiled.** + +The lesson is narrow and worth stating: when a probe that reproduces a +program's exact syscall sequence behaves correctly and the program does not, +stop looking under the syscalls and start looking at the build. --- diff --git a/docs/plan/1.0.md b/docs/plan/1.0.md index 66dfc54..1b03076 100644 --- a/docs/plan/1.0.md +++ b/docs/plan/1.0.md @@ -206,24 +206,15 @@ right then. in 18 after.** See [../architecture/kernel.md](../architecture/kernel.md), "Per-CPU state". -- [ ] **`$(command substitution)` hangs every shell.** Not job control, not - SMP, not zsh — `mksh -c 'X=$(echo hi)'` hangs the same way, on one CPU, - with `set +m`. Pipelines, subshells, background jobs and foreground - externals all work; only `$( )` does not. - - Narrowed a long way this session and still open. The kernel's half of - the SIGCHLD/sigsuspend contract is now *proven* rather than assumed — - `cmd/sigtest` asserts the whole pattern and passes — and a full syscall - trace of a hang shows every return value correct. The contradiction is - at one line of zsh's `waitforpid()`, which ends its loop on `ESRCH` from - `kill(pid, 0)`; the trace shows that `kill` returning ESRCH and the loop - being entered regardless. See +- [x] ~~**`$(command substitution)` hangs the shell.**~~ **Not a kernel bug.** + `ports/zsh/Makefile` answered configure's "does `kill(pid, 0)` return + ESRCH correctly?" with `no`, which makes zsh `#define ESRCH EINVAL`. + `waitforpid()` ends its wait on `errno != ESRCH`, so with the kernel + correctly returning ESRCH (3) and zsh comparing against EINVAL (22), the + shell waited for ever on a child it had already reaped. One word in a + cross-compile cache. Autoconf's own default for that check is `yes`; + omitting the line would have been right. See [../history/kernel.md](../history/kernel.md), section 7. - - Two real bugs were found and fixed on the way: the signal-return - trampoline was hardcoded to Linux's `rt_sigreturn` number, so handlers - in first-party binaries called `open()` and then executed stack bytes; - and `SIG_BLOCK`/`SIG_UNBLOCK`/`SIG_SETMASK` were not defined anywhere. - [ ] **`getrusage` does not distinguish `RUSAGE_CHILDREN`** from `RUSAGE_SELF`. - [ ] **`/proc//exe` is a regular file, not a symlink**, so `readlink()` fails and `ps -o comm` prints `[init]`. Symlinks themselves now work — diff --git a/ports/zsh/Makefile b/ports/zsh/Makefile index 8e799be..713eddc 100644 --- a/ports/zsh/Makefile +++ b/ports/zsh/Makefile @@ -98,6 +98,21 @@ ZSH_LIBS := -lncursesw $(MUSL)/lib/libc.a $(LIBGCC) $(MUSL)/lib/libc.a \ # Answers configure cannot work out without running a program on the target. # Each one is a fact about SchOS, so each one is written down here rather than # guessed by autoconf's cross-compilation defaults. +# +# **Read each name as the question configure asks, not as a property.** These +# are `zsh_cv_*` cache entries for AC_CACHE_CHECK, and the value answers the +# *check's* wording. `zsh_cv_sys_killesrch` is "does kill(pid, 0) return ESRCH +# correctly", so `yes` is the healthy answer and `no` means broken. +# +# It was `no` here, and that single word cost every `$(command substitution)` +# in the shell. `no` makes configure emit BROKEN_KILL_ESRCH, and zsh_system.h +# answers that with `#undef ESRCH / #define ESRCH EINVAL`. zsh's waitforpid() +# ends its wait on `errno != ESRCH`; with ESRCH redefined to EINVAL (22) and +# the kernel correctly returning ESRCH (3), that test was true for ever and +# the shell waited on a child it had already reaped. Every `X=$(...)` hung. +# +# The cruel part: autoconf's own cross-compilation default for this one is +# `yes`. Leaving the line out entirely would have been correct. ZSH_CACHE := \ zsh_cv_shared_environ=yes \ zsh_cv_shared_tgetent=no \ @@ -107,7 +122,7 @@ ZSH_CACHE := \ zsh_cv_func_realpath_accepts_null=yes \ zsh_cv_sys_path_dev_fd=/dev/fd \ zsh_cv_sys_fifo=yes \ - zsh_cv_sys_killesrch=no \ + zsh_cv_sys_killesrch=yes \ zsh_cv_sys_signal_handler=void \ zsh_cv_prog_tcsetpgrp=yes \ ac_cv_func_mmap_fixed_mapped=yes \