zsh: one word in a cross-compile cache, and every $(...) hung

`X=$(echo hi)` never returned. It has been in docs/history/kernel.md section 7
as "mksh's job control still doesn't reliably progress ... root cause not yet
isolated" with three suspects, and it was none of them, and it was not the
kernel at all.

ports/zsh/Makefile carries a block of `zsh_cv_*` answers for the checks
autoconf cannot run when cross-compiling. One of them was

    zsh_cv_sys_killesrch=no

which reads like a reassuring property and is in fact the answer to the
question configure asks: *does kill(pid, 0) return ESRCH correctly?* `no` makes
configure emit BROKEN_KILL_ESRCH, and Src/zsh_system.h answers that with

    #ifdef BROKEN_KILL_ESRCH
    #undef ESRCH
    #define ESRCH EINVAL
    #endif

zsh's waitforpid() ends its wait on `errno != ESRCH` (Src/jobs.c:1569). With
ESRCH redefined to EINVAL (22) and the kernel correctly returning ESRCH (3),
that condition was true for ever: the shell sat in sigsuspend waiting for a
child it had already reaped. An instrumented zsh says the whole thing in one
line -- kill returned -1, errno is 3, and the constant it is compared against
is 22:

    WFPkill -1 3 22

**Autoconf's own cross-compilation default for that check is `yes`.** Deleting
the line would have been correct. Writing it down as "a fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.

Verified: `X=$(echo inline)`, `$(/bin/echo external)`, backticks, nested
`$(echo $(echo deep))`, a slow child, and `$(ls /bin | wc -l)` all return.
mksh -- as /bin/sh -- was never affected.

Two corrections to yesterday's note, both confident and both wrong:

  - "Not zsh, mksh hangs the same way" was false. mksh installs as /bin/sh,
    so that test was a `command not found` whose output scrolled past. Had the
    binary been named right, this would have localised to zsh's build in one
    step instead of a day.
  - cmd/sigtest had already proven the kernel's SIGCHLD/sigsuspend contract,
    and a musl-linked probe had already shown kill(pid,0) returning -1 with
    errno == ESRCH == 3 and zsh's own loop condition evaluating false. Both
    results were right; both were read as "so it must be subtler in the
    kernel". What they actually pointed at was the one place the probe and the
    shell differed -- how each was compiled.

The lesson is narrow enough to state: when a probe that reproduces a program's
exact syscall sequence behaves correctly and the program does not, stop looking
under the syscalls and look at the build.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.claude/memory/schos-smp-userspace-corruption.md

@@ -87,22 +87,27 @@ a 40-second timeout is enough — no typed input needed. `-smp 4` roughly double
 the rate over `-smp 2`.
 
 
-## Found while here: `$(command substitution)` hangs every shell
+## Found while here: `$(command substitution)` hung — fixed 2026-09-22
 
-Unrelated to the above and **still open**. `X=$(echo hi)` never returns, in zsh
+**It was never the kernel.** `ports/zsh/Makefile` answered configure's
-*and* mksh, on 1 CPU and on 4, interactive and not, with `set +m`. Pipelines,
+"does `kill(pid, 0)` return ESRCH correctly?" with `zsh_cv_sys_killesrch=no`,
-subshells, background jobs and foreground externals are all fine.
+which makes zsh `#define ESRCH EINVAL`. `waitforpid()` ends its wait on
+`errno != ESRCH`; with the kernel correctly returning ESRCH (3) and zsh
+comparing against EINVAL (22), the shell waited for ever on a child it had
+already reaped.
 
-Ruled out: `setpgid` (real now), SMP, job control, zsh specifically, and the
+Two things to carry forward, both from getting it wrong first:
-kernel's SIGCHLD/sigsuspend contract — `cmd/sigtest` proves that one and passes
-all eight checks. A full syscall trace of a hang has every return value
-correct.
 
-The contradiction is at `Src/jobs.c:1569`: zsh's `waitforpid()` ends its loop
+- **`zsh_cv_*` names answer the check's *question*, not a property.** `no` on
-on `ESRCH` from `kill(pid, 0)`, and the trace shows that `kill` returning ESRCH
+  a check called "killesrch" reads as reassuring and means "broken".
-with the loop entered anyway. Next step is a musl-linked probe that does
+  Autoconf's cross-compile default there is `yes` — the line should not have
-`fork; let the handler reap; kill(pid,0); print errno` — everything upstream of
+  existed.
-that line is known good.
+- **mksh installs as `/bin/sh`, not `mksh`.** A test invoking `mksh -c` is a
+  `command not found`, and reading that as "mksh hangs too" is what kept the
+  search in the kernel for an extra day.
 
-Documented in full at `docs/history/kernel.md` section 7, which is where to
+The general lesson: when a probe reproducing a program's exact syscall
-start.
+sequence behaves correctly and the program does not, stop looking under the
+syscalls and look at how each was compiled.
+
+Full write-up: `docs/history/kernel.md` section 7.

docs/history/kernel.md

@@ -432,85 +432,78 @@ likelihood:
   its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails
   to find the job for reasons not yet traced.
 
-### 2026-09-21: narrowed a long way, still open
+### Resolved 2026-09-22: it was never the kernel
 
-Picked up again with the debugger that did not exist when the above was
+**`X=$(...)` hung because zsh was built with `ESRCH` defined as `EINVAL`.**
-written. What is now *ruled out* is most of what that note suspected:
 
-- **Not `setpgid`/`getpgid`.** Both are real now — `proc_setpgid`, `p->pgid`,
+`ports/zsh/Makefile` carries a block of `zsh_cv_*` cache answers, for the
-  `p->sid` — and `waitpid()` honours the POSIX pid selection rules.
+checks autoconf cannot run when cross-compiling. One of them was
-- **Not signal delivery, and not the sigsuspend contract.** `cmd/sigtest`
-  asserts the whole pattern every shell uses and passes all eight checks: a
-  blocked SIGCHLD stays pending, `sigsuspend` delivers an *already pending*
-  signal without spinning, the handler runs exactly once, the mask is restored,
-  and the child is still reapable afterwards. A permissive stub cannot pass it.
-- **Not SMP.** It hangs identically on `-smp 1` and `-smp 4`.
-- **Not job control.** `set +m` makes no difference, and a non-interactive
-  `zsh -c` hangs the same way.
-- **Not zsh.** `mksh -c 'X=$(echo hi)'` hangs in the same place.
-- **Not the values the kernel returns.** The full syscall trace of a hang is
-  below and every one of them is correct.
 
-The symptom is narrower than "job control": it is **`$(command
+```make
-substitution)`**, and only that. Pipelines work. Subshells work. Background
+zsh_cv_sys_killesrch=no
-jobs and `wait` work. Foreground external commands work.
-
-A complete trace of `X=$(echo inline)` in zsh, from the fork to the hang:
-
-```
-fork()                        -> 14
-sigprocmask x2                            (child_block)
-close(pipe write end)
-sigprocmask(SIG_UNBLOCK, SIGCHLD)         (child_unblock)
-  -> SIGCHLD delivered to zsh's handler
-     wait4(-1, WNOHANG|WUNTRACED|WCONTINUED) -> 14, status 0
-     getrusage()                             -> 0
-     wait4(same)                             -> -ECHILD
-     rt_sigreturn
-read(pipe)                    -> 7          ("inline\n")
-read(pipe)                    -> 0          (EOF)
-sigprocmask, close
-kill(14, 0)                   -> -ESRCH
-rt_sigsuspend(mask=SIGINT)    -> never returns
 ```
 
-Everything there is right. The child is reaped with status 0, the pipe carries
+Read as a property that looks reassuring. Read as the answer to the question
-its output and closes, and `kill` on the reaped pid correctly says ESRCH.
+configure actually asks — *"does `kill(pid, 0)` return ESRCH correctly?"* — it
-
+says the opposite. `no` makes configure emit `BROKEN_KILL_ESRCH`, and
-**And that last pair is the contradiction.** zsh's `waitforpid()`
+`Src/zsh_system.h` answers that with:
-(`Src/jobs.c:1569`) is
 
 ```c
-while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) {
+#ifdef BROKEN_KILL_ESRCH
-    ...
+#undef ESRCH
-    signal_suspend(SIGCHLD, wait_cmd);
+#define ESRCH EINVAL
+#endif
 ```
 
-with a comment above it saying, in as many words, that ESRCH from that `kill`
+zsh's `waitforpid()` (`Src/jobs.c:1569`) ends its wait on
-is how the loop ends. The trace shows the `kill` returning ESRCH **and the
-loop being entered anyway**, which leaves only two possibilities: `errno` is
-not `ESRCH` when zsh reads it, or `kill()` did not return negative. errno is
-demonstrably fine elsewhere — `kill -0 31000` prints "no such process" from
-both zsh's builtin and toybox's `kill`, and `cat` of a missing file prints "No
-such file or directory".
 
-That is where it stands. The next step is to distinguish those two: a musl-
+```c
-linked probe that does exactly `fork; wait for the handler to reap; kill(pid,
+while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH))
-0); print errno`, because everything upstream of that line is now known good.
+```
 
-Two real bugs were found on the way and fixed, neither of them this one:
+so with `ESRCH` redefined to `EINVAL` (22) and the kernel correctly returning
+`ESRCH` (3), that condition was true for ever. The shell waited, in
+`sigsuspend`, for a child it had already reaped. Every command substitution in
+the shell hung.
 
-- **The signal return trampoline was ABI-blind.** `deliver_signals()` wrote
+The instrumented build says it in one line — `kill` returned −1, `errno` is 3,
-  `mov eax, 15 ; syscall` as the handler's return address. Fifteen is *Linux's*
+and the constant it is compared against is 22:
-  `rt_sigreturn`, correct for anything musl-linked; for a first-party binary it
+
-  is native syscall 15, which is `open`. So a handler in any `cmd/` program
+```
-  ran, returned into the trampoline, called `open()` with the handler's
+WFPkill -1 3 22
-  leftover registers, got `-ENOENT`, and carried on executing the stack bytes
+```
-  after the trampoline. It emits the right number for the process's ABI now.
+
-  This is why `cmd/sigtest` had to exist before anything else could be trusted.
+**Autoconf's own cross-compilation default for this check is `yes`.** Deleting
-- **`SIG_BLOCK`, `SIG_UNBLOCK` and `SIG_SETMASK` were not defined anywhere.**
+the line would have been correct. Writing it down as a "fact about SchOS" is
-  The kernel branched on bare `0`, `1` and `2` with the names in comments, so
+what got it wrong, and the block's comment now says to read each name as the
-  anything outside the kernel had to spell the number and hope.
+question rather than as a property.
+
+#### What this cost, and what to take from it
+
+This sat in section 7 as "mksh's job control still doesn't reliably progress
+... root cause not yet isolated" with three suspects, none of which were it.
+All three have since been eliminated on their own merits — `setpgid` and
+`getpgid` are real, `getrusage` is not consulted on the path, and the pid
+matching was never reached.
+
+Two corrections to the note written on 2026-09-21, both worth keeping because
+they were confident and wrong:
+
+- **"Not zsh — `mksh -c` hangs the same way" was false.** mksh installs as
+  `/bin/sh`, not `mksh`, so that test was a `command not found` whose output
+  scrolled past. mksh was never affected. Had the binary been named correctly
+  the bug would have been localised to zsh's build in one step.
+- **The kernel was cleared and then still suspected.** `cmd/sigtest` had
+  already proven the SIGCHLD/sigsuspend contract, and a musl-linked probe had
+  already shown `kill(pid, 0)` returning −1 with `errno == ESRCH == 3` and
+  zsh's own loop condition evaluating false. Both results were correct and
+  both were read as "so it must be something subtler in the kernel". The thing
+  they actually pointed at was the one place the probe and the shell differed:
+  **how each was compiled.**
+
+The lesson is narrow and worth stating: when a probe that reproduces a
+program's exact syscall sequence behaves correctly and the program does not,
+stop looking under the syscalls and start looking at the build.
 
 ---
 

docs/plan/1.0.md

@@ -206,24 +206,15 @@ right then.
       in 18 after.** See [../architecture/kernel.md](../architecture/kernel.md),
       "Per-CPU state".
 
-- [ ] **`$(command substitution)` hangs every shell.** Not job control, not
+- [x] ~~**`$(command substitution)` hangs the shell.**~~ **Not a kernel bug.**
-      SMP, not zsh — `mksh -c 'X=$(echo hi)'` hangs the same way, on one CPU,
+      `ports/zsh/Makefile` answered configure's "does `kill(pid, 0)` return
-      with `set +m`. Pipelines, subshells, background jobs and foreground
+      ESRCH correctly?" with `no`, which makes zsh `#define ESRCH EINVAL`.
-      externals all work; only `$( )` does not.
+      `waitforpid()` ends its wait on `errno != ESRCH`, so with the kernel
-
+      correctly returning ESRCH (3) and zsh comparing against EINVAL (22), the
-      Narrowed a long way this session and still open. The kernel's half of
+      shell waited for ever on a child it had already reaped. One word in a
-      the SIGCHLD/sigsuspend contract is now *proven* rather than assumed —
+      cross-compile cache. Autoconf's own default for that check is `yes`;
-      `cmd/sigtest` asserts the whole pattern and passes — and a full syscall
+      omitting the line would have been right. See
-      trace of a hang shows every return value correct. The contradiction is
-      at one line of zsh's `waitforpid()`, which ends its loop on `ESRCH` from
-      `kill(pid, 0)`; the trace shows that `kill` returning ESRCH and the loop
-      being entered regardless. See
       [../history/kernel.md](../history/kernel.md), section 7.
-
-      Two real bugs were found and fixed on the way: the signal-return
-      trampoline was hardcoded to Linux's `rt_sigreturn` number, so handlers
-      in first-party binaries called `open()` and then executed stack bytes;
-      and `SIG_BLOCK`/`SIG_UNBLOCK`/`SIG_SETMASK` were not defined anywhere.
 - [ ] **`getrusage` does not distinguish `RUSAGE_CHILDREN`** from `RUSAGE_SELF`.
 - [ ] **`/proc/<pid>/exe` is a regular file, not a symlink**, so `readlink()`
       fails and `ps -o comm` prints `[init]`. Symlinks themselves now work —

ports/zsh/Makefile

@@ -98,6 +98,21 @@ ZSH_LIBS     := -lncursesw $(MUSL)/lib/libc.a $(LIBGCC) $(MUSL)/lib/libc.a \
 # Answers configure cannot work out without running a program on the target.
 # Each one is a fact about SchOS, so each one is written down here rather than
 # guessed by autoconf's cross-compilation defaults.
+#
+# **Read each name as the question configure asks, not as a property.** These
+# are `zsh_cv_*` cache entries for AC_CACHE_CHECK, and the value answers the
+# *check's* wording. `zsh_cv_sys_killesrch` is "does kill(pid, 0) return ESRCH
+# correctly", so `yes` is the healthy answer and `no` means broken.
+#
+# It was `no` here, and that single word cost every `$(command substitution)`
+# in the shell. `no` makes configure emit BROKEN_KILL_ESRCH, and zsh_system.h
+# answers that with `#undef ESRCH / #define ESRCH EINVAL`. zsh's waitforpid()
+# ends its wait on `errno != ESRCH`; with ESRCH redefined to EINVAL (22) and
+# the kernel correctly returning ESRCH (3), that test was true for ever and
+# the shell waited on a child it had already reaped. Every `X=$(...)` hung.
+#
+# The cruel part: autoconf's own cross-compilation default for this one is
+# `yes`. Leaving the line out entirely would have been correct.
 ZSH_CACHE := \
   zsh_cv_shared_environ=yes \
   zsh_cv_shared_tgetent=no \
@@ -107,7 +122,7 @@ ZSH_CACHE := \
   zsh_cv_func_realpath_accepts_null=yes \
   zsh_cv_sys_path_dev_fd=/dev/fd \
   zsh_cv_sys_fifo=yes \
-  zsh_cv_sys_killesrch=no \
+  zsh_cv_sys_killesrch=yes \
   zsh_cv_sys_signal_handler=void \
   zsh_cv_prog_tcsetpgrp=yes \
   ac_cv_func_mmap_fixed_mapped=yes \