From b01206069961bee23dcd929c2adeffa73e3ef8a2 Mon Sep 17 00:00:00 2001 From: Nexoniarz Date: Tue, 22 Sep 2026 00:29:39 +0200 Subject: [PATCH] zsh: one word in a cross-compile cache, and every $(...) hung `X=$(echo hi)` never returned. It has been in docs/history/kernel.md section 7 as "mksh's job control still doesn't reliably progress ... root cause not yet isolated" with three suspects, and it was none of them, and it was not the kernel at all. ports/zsh/Makefile carries a block of `zsh_cv_*` answers for the checks autoconf cannot run when cross-compiling. One of them was zsh_cv_sys_killesrch=no which reads like a reassuring property and is in fact the answer to the question configure asks: *does kill(pid, 0) return ESRCH correctly?* `no` makes configure emit BROKEN_KILL_ESRCH, and Src/zsh_system.h answers that with #ifdef BROKEN_KILL_ESRCH #undef ESRCH #define ESRCH EINVAL #endif zsh's waitforpid() ends its wait on `errno != ESRCH` (Src/jobs.c:1569). With ESRCH redefined to EINVAL (22) and the kernel correctly returning ESRCH (3), that condition was true for ever: the shell sat in sigsuspend waiting for a child it had already reaped. An instrumented zsh says the whole thing in one line -- kill returned -1, errno is 3, and the constant it is compared against is 22: WFPkill -1 3 22 **Autoconf's own cross-compilation default for that check is `yes`.** Deleting the line would have been correct. Writing it down as "a fact about SchOS" is what got it wrong, and the block's comment now says to read each name as the question rather than as a property. Verified: `X=$(echo inline)`, `$(/bin/echo external)`, backticks, nested `$(echo $(echo deep))`, a slow child, and `$(ls /bin | wc -l)` all return. mksh -- as /bin/sh -- was never affected. Two corrections to yesterday's note, both confident and both wrong: - "Not zsh, mksh hangs the same way" was false. mksh installs as /bin/sh, so that test was a `command not found` whose output scrolled past. Had the binary been named right, this would have localised to zsh's build in one step instead of a day. - cmd/sigtest had already proven the kernel's SIGCHLD/sigsuspend contract, and a musl-linked probe had already shown kill(pid,0) returning -1 with errno == ESRCH == 3 and zsh's own loop condition evaluating false. Both results were right; both were read as "so it must be subtler in the kernel". What they actually pointed at was the one place the probe and the shell differed -- how each was compiled. The lesson is narrow enough to state: when a probe that reproduces a program's exact syscall sequence behaves correctly and the program does not, stop looking under the syscalls and look at the build. Co-Authored-By: Claude Opus 5 --- .../memory/schos-smp-userspace-corruption.md | 35 ++--- docs/history/kernel.md | 127 +++++++++--------- docs/plan/1.0.md | 25 ++-- ports/zsh/Makefile | 17 ++- 4 files changed, 104 insertions(+), 100 deletions(-) diff --git a/.claude/memory/schos-smp-userspace-corruption.md b/.claude/memory/schos-smp-userspace-corruption.md index 7979f0d..4b5821e 100644 --- a/.claude/memory/schos-smp-userspace-corruption.md +++ b/.claude/memory/schos-smp-userspace-corruption.md @@ -87,22 +87,27 @@ a 40-second timeout is enough — no typed input needed. `-smp 4` roughly double the rate over `-smp 2`. -## Found while here: `$(command substitution)` hangs every shell +## Found while here: `$(command substitution)` hung — fixed 2026-09-22 -Unrelated to the above and **still open**. `X=$(echo hi)` never returns, in zsh -*and* mksh, on 1 CPU and on 4, interactive and not, with `set +m`. Pipelines, -subshells, background jobs and foreground externals are all fine. +**It was never the kernel.** `ports/zsh/Makefile` answered configure's +"does `kill(pid, 0)` return ESRCH correctly?" with `zsh_cv_sys_killesrch=no`, +which makes zsh `#define ESRCH EINVAL`. `waitforpid()` ends its wait on +`errno != ESRCH`; with the kernel correctly returning ESRCH (3) and zsh +comparing against EINVAL (22), the shell waited for ever on a child it had +already reaped. -Ruled out: `setpgid` (real now), SMP, job control, zsh specifically, and the -kernel's SIGCHLD/sigsuspend contract — `cmd/sigtest` proves that one and passes -all eight checks. A full syscall trace of a hang has every return value -correct. +Two things to carry forward, both from getting it wrong first: -The contradiction is at `Src/jobs.c:1569`: zsh's `waitforpid()` ends its loop -on `ESRCH` from `kill(pid, 0)`, and the trace shows that `kill` returning ESRCH -with the loop entered anyway. Next step is a musl-linked probe that does -`fork; let the handler reap; kill(pid,0); print errno` — everything upstream of -that line is known good. +- **`zsh_cv_*` names answer the check's *question*, not a property.** `no` on + a check called "killesrch" reads as reassuring and means "broken". + Autoconf's cross-compile default there is `yes` — the line should not have + existed. +- **mksh installs as `/bin/sh`, not `mksh`.** A test invoking `mksh -c` is a + `command not found`, and reading that as "mksh hangs too" is what kept the + search in the kernel for an extra day. -Documented in full at `docs/history/kernel.md` section 7, which is where to -start. +The general lesson: when a probe reproducing a program's exact syscall +sequence behaves correctly and the program does not, stop looking under the +syscalls and look at how each was compiled. + +Full write-up: `docs/history/kernel.md` section 7. diff --git a/docs/history/kernel.md b/docs/history/kernel.md index 0604597..0b51c5d 100644 --- a/docs/history/kernel.md +++ b/docs/history/kernel.md @@ -432,85 +432,78 @@ likelihood: its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails to find the job for reasons not yet traced. -### 2026-09-21: narrowed a long way, still open +### Resolved 2026-09-22: it was never the kernel -Picked up again with the debugger that did not exist when the above was -written. What is now *ruled out* is most of what that note suspected: +**`X=$(...)` hung because zsh was built with `ESRCH` defined as `EINVAL`.** -- **Not `setpgid`/`getpgid`.** Both are real now — `proc_setpgid`, `p->pgid`, - `p->sid` — and `waitpid()` honours the POSIX pid selection rules. -- **Not signal delivery, and not the sigsuspend contract.** `cmd/sigtest` - asserts the whole pattern every shell uses and passes all eight checks: a - blocked SIGCHLD stays pending, `sigsuspend` delivers an *already pending* - signal without spinning, the handler runs exactly once, the mask is restored, - and the child is still reapable afterwards. A permissive stub cannot pass it. -- **Not SMP.** It hangs identically on `-smp 1` and `-smp 4`. -- **Not job control.** `set +m` makes no difference, and a non-interactive - `zsh -c` hangs the same way. -- **Not zsh.** `mksh -c 'X=$(echo hi)'` hangs in the same place. -- **Not the values the kernel returns.** The full syscall trace of a hang is - below and every one of them is correct. +`ports/zsh/Makefile` carries a block of `zsh_cv_*` cache answers, for the +checks autoconf cannot run when cross-compiling. One of them was -The symptom is narrower than "job control": it is **`$(command -substitution)`**, and only that. Pipelines work. Subshells work. Background -jobs and `wait` work. Foreground external commands work. - -A complete trace of `X=$(echo inline)` in zsh, from the fork to the hang: - -``` -fork() -> 14 -sigprocmask x2 (child_block) -close(pipe write end) -sigprocmask(SIG_UNBLOCK, SIGCHLD) (child_unblock) - -> SIGCHLD delivered to zsh's handler - wait4(-1, WNOHANG|WUNTRACED|WCONTINUED) -> 14, status 0 - getrusage() -> 0 - wait4(same) -> -ECHILD - rt_sigreturn -read(pipe) -> 7 ("inline\n") -read(pipe) -> 0 (EOF) -sigprocmask, close -kill(14, 0) -> -ESRCH -rt_sigsuspend(mask=SIGINT) -> never returns +```make +zsh_cv_sys_killesrch=no ``` -Everything there is right. The child is reaped with status 0, the pipe carries -its output and closes, and `kill` on the reaped pid correctly says ESRCH. - -**And that last pair is the contradiction.** zsh's `waitforpid()` -(`Src/jobs.c:1569`) is +Read as a property that looks reassuring. Read as the answer to the question +configure actually asks — *"does `kill(pid, 0)` return ESRCH correctly?"* — it +says the opposite. `no` makes configure emit `BROKEN_KILL_ESRCH`, and +`Src/zsh_system.h` answers that with: ```c -while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) { - ... - signal_suspend(SIGCHLD, wait_cmd); +#ifdef BROKEN_KILL_ESRCH +#undef ESRCH +#define ESRCH EINVAL +#endif ``` -with a comment above it saying, in as many words, that ESRCH from that `kill` -is how the loop ends. The trace shows the `kill` returning ESRCH **and the -loop being entered anyway**, which leaves only two possibilities: `errno` is -not `ESRCH` when zsh reads it, or `kill()` did not return negative. errno is -demonstrably fine elsewhere — `kill -0 31000` prints "no such process" from -both zsh's builtin and toybox's `kill`, and `cat` of a missing file prints "No -such file or directory". +zsh's `waitforpid()` (`Src/jobs.c:1569`) ends its wait on -That is where it stands. The next step is to distinguish those two: a musl- -linked probe that does exactly `fork; wait for the handler to reap; kill(pid, -0); print errno`, because everything upstream of that line is now known good. +```c +while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) +``` -Two real bugs were found on the way and fixed, neither of them this one: +so with `ESRCH` redefined to `EINVAL` (22) and the kernel correctly returning +`ESRCH` (3), that condition was true for ever. The shell waited, in +`sigsuspend`, for a child it had already reaped. Every command substitution in +the shell hung. -- **The signal return trampoline was ABI-blind.** `deliver_signals()` wrote - `mov eax, 15 ; syscall` as the handler's return address. Fifteen is *Linux's* - `rt_sigreturn`, correct for anything musl-linked; for a first-party binary it - is native syscall 15, which is `open`. So a handler in any `cmd/` program - ran, returned into the trampoline, called `open()` with the handler's - leftover registers, got `-ENOENT`, and carried on executing the stack bytes - after the trampoline. It emits the right number for the process's ABI now. - This is why `cmd/sigtest` had to exist before anything else could be trusted. -- **`SIG_BLOCK`, `SIG_UNBLOCK` and `SIG_SETMASK` were not defined anywhere.** - The kernel branched on bare `0`, `1` and `2` with the names in comments, so - anything outside the kernel had to spell the number and hope. +The instrumented build says it in one line — `kill` returned −1, `errno` is 3, +and the constant it is compared against is 22: + +``` +WFPkill -1 3 22 +``` + +**Autoconf's own cross-compilation default for this check is `yes`.** Deleting +the line would have been correct. Writing it down as a "fact about SchOS" is +what got it wrong, and the block's comment now says to read each name as the +question rather than as a property. + +#### What this cost, and what to take from it + +This sat in section 7 as "mksh's job control still doesn't reliably progress +... root cause not yet isolated" with three suspects, none of which were it. +All three have since been eliminated on their own merits — `setpgid` and +`getpgid` are real, `getrusage` is not consulted on the path, and the pid +matching was never reached. + +Two corrections to the note written on 2026-09-21, both worth keeping because +they were confident and wrong: + +- **"Not zsh — `mksh -c` hangs the same way" was false.** mksh installs as + `/bin/sh`, not `mksh`, so that test was a `command not found` whose output + scrolled past. mksh was never affected. Had the binary been named correctly + the bug would have been localised to zsh's build in one step. +- **The kernel was cleared and then still suspected.** `cmd/sigtest` had + already proven the SIGCHLD/sigsuspend contract, and a musl-linked probe had + already shown `kill(pid, 0)` returning −1 with `errno == ESRCH == 3` and + zsh's own loop condition evaluating false. Both results were correct and + both were read as "so it must be something subtler in the kernel". The thing + they actually pointed at was the one place the probe and the shell differed: + **how each was compiled.** + +The lesson is narrow and worth stating: when a probe that reproduces a +program's exact syscall sequence behaves correctly and the program does not, +stop looking under the syscalls and start looking at the build. --- diff --git a/docs/plan/1.0.md b/docs/plan/1.0.md index 66dfc54..1b03076 100644 --- a/docs/plan/1.0.md +++ b/docs/plan/1.0.md @@ -206,24 +206,15 @@ right then. in 18 after.** See [../architecture/kernel.md](../architecture/kernel.md), "Per-CPU state". -- [ ] **`$(command substitution)` hangs every shell.** Not job control, not - SMP, not zsh — `mksh -c 'X=$(echo hi)'` hangs the same way, on one CPU, - with `set +m`. Pipelines, subshells, background jobs and foreground - externals all work; only `$( )` does not. - - Narrowed a long way this session and still open. The kernel's half of - the SIGCHLD/sigsuspend contract is now *proven* rather than assumed — - `cmd/sigtest` asserts the whole pattern and passes — and a full syscall - trace of a hang shows every return value correct. The contradiction is - at one line of zsh's `waitforpid()`, which ends its loop on `ESRCH` from - `kill(pid, 0)`; the trace shows that `kill` returning ESRCH and the loop - being entered regardless. See +- [x] ~~**`$(command substitution)` hangs the shell.**~~ **Not a kernel bug.** + `ports/zsh/Makefile` answered configure's "does `kill(pid, 0)` return + ESRCH correctly?" with `no`, which makes zsh `#define ESRCH EINVAL`. + `waitforpid()` ends its wait on `errno != ESRCH`, so with the kernel + correctly returning ESRCH (3) and zsh comparing against EINVAL (22), the + shell waited for ever on a child it had already reaped. One word in a + cross-compile cache. Autoconf's own default for that check is `yes`; + omitting the line would have been right. See [../history/kernel.md](../history/kernel.md), section 7. - - Two real bugs were found and fixed on the way: the signal-return - trampoline was hardcoded to Linux's `rt_sigreturn` number, so handlers - in first-party binaries called `open()` and then executed stack bytes; - and `SIG_BLOCK`/`SIG_UNBLOCK`/`SIG_SETMASK` were not defined anywhere. - [ ] **`getrusage` does not distinguish `RUSAGE_CHILDREN`** from `RUSAGE_SELF`. - [ ] **`/proc//exe` is a regular file, not a symlink**, so `readlink()` fails and `ps -o comm` prints `[init]`. Symlinks themselves now work — diff --git a/ports/zsh/Makefile b/ports/zsh/Makefile index 8e799be..713eddc 100644 --- a/ports/zsh/Makefile +++ b/ports/zsh/Makefile @@ -98,6 +98,21 @@ ZSH_LIBS := -lncursesw $(MUSL)/lib/libc.a $(LIBGCC) $(MUSL)/lib/libc.a \ # Answers configure cannot work out without running a program on the target. # Each one is a fact about SchOS, so each one is written down here rather than # guessed by autoconf's cross-compilation defaults. +# +# **Read each name as the question configure asks, not as a property.** These +# are `zsh_cv_*` cache entries for AC_CACHE_CHECK, and the value answers the +# *check's* wording. `zsh_cv_sys_killesrch` is "does kill(pid, 0) return ESRCH +# correctly", so `yes` is the healthy answer and `no` means broken. +# +# It was `no` here, and that single word cost every `$(command substitution)` +# in the shell. `no` makes configure emit BROKEN_KILL_ESRCH, and zsh_system.h +# answers that with `#undef ESRCH / #define ESRCH EINVAL`. zsh's waitforpid() +# ends its wait on `errno != ESRCH`; with ESRCH redefined to EINVAL (22) and +# the kernel correctly returning ESRCH (3), that test was true for ever and +# the shell waited on a child it had already reaped. Every `X=$(...)` hung. +# +# The cruel part: autoconf's own cross-compilation default for this one is +# `yes`. Leaving the line out entirely would have been correct. ZSH_CACHE := \ zsh_cv_shared_environ=yes \ zsh_cv_shared_tgetent=no \ @@ -107,7 +122,7 @@ ZSH_CACHE := \ zsh_cv_func_realpath_accepts_null=yes \ zsh_cv_sys_path_dev_fd=/dev/fd \ zsh_cv_sys_fifo=yes \ - zsh_cv_sys_killesrch=no \ + zsh_cv_sys_killesrch=yes \ zsh_cv_sys_signal_handler=void \ zsh_cv_prog_tcsetpgrp=yes \ ac_cv_func_mmap_fixed_mapped=yes \