zsh: one word in a cross-compile cache, and every $(...) hung

`X=$(echo hi)` never returned. It has been in docs/history/kernel.md section 7
as "mksh's job control still doesn't reliably progress ... root cause not yet
isolated" with three suspects, and it was none of them, and it was not the
kernel at all.

ports/zsh/Makefile carries a block of `zsh_cv_*` answers for the checks
autoconf cannot run when cross-compiling. One of them was

    zsh_cv_sys_killesrch=no

which reads like a reassuring property and is in fact the answer to the
question configure asks: *does kill(pid, 0) return ESRCH correctly?* `no` makes
configure emit BROKEN_KILL_ESRCH, and Src/zsh_system.h answers that with

    #ifdef BROKEN_KILL_ESRCH
    #undef ESRCH
    #define ESRCH EINVAL
    #endif

zsh's waitforpid() ends its wait on `errno != ESRCH` (Src/jobs.c:1569). With
ESRCH redefined to EINVAL (22) and the kernel correctly returning ESRCH (3),
that condition was true for ever: the shell sat in sigsuspend waiting for a
child it had already reaped. An instrumented zsh says the whole thing in one
line -- kill returned -1, errno is 3, and the constant it is compared against
is 22:

    WFPkill -1 3 22

**Autoconf's own cross-compilation default for that check is `yes`.** Deleting
the line would have been correct. Writing it down as "a fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.

Verified: `X=$(echo inline)`, `$(/bin/echo external)`, backticks, nested
`$(echo $(echo deep))`, a slow child, and `$(ls /bin | wc -l)` all return.
mksh -- as /bin/sh -- was never affected.

Two corrections to yesterday's note, both confident and both wrong:

  - "Not zsh, mksh hangs the same way" was false. mksh installs as /bin/sh,
    so that test was a `command not found` whose output scrolled past. Had the
    binary been named right, this would have localised to zsh's build in one
    step instead of a day.
  - cmd/sigtest had already proven the kernel's SIGCHLD/sigsuspend contract,
    and a musl-linked probe had already shown kill(pid,0) returning -1 with
    errno == ESRCH == 3 and zsh's own loop condition evaluating false. Both
    results were right; both were read as "so it must be subtler in the
    kernel". What they actually pointed at was the one place the probe and the
    shell differed -- how each was compiled.

The lesson is narrow enough to state: when a probe that reproduces a program's
exact syscall sequence behaves correctly and the program does not, stop looking
under the syscalls and look at the build.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.claude/memory/schos-smp-userspace-corruption.md

@@ -87,22 +87,27 @@ a 40-second timeout is enough — no typed input needed. `-smp 4` roughly double
 the rate over `-smp 2`.
 
 
-## Found while here: `$(command substitution)` hangs every shell
+## Found while here: `$(command substitution)` hung — fixed 2026-09-22
 
-Unrelated to the above and **still open**. `X=$(echo hi)` never returns, in zsh
-*and* mksh, on 1 CPU and on 4, interactive and not, with `set +m`. Pipelines,
-subshells, background jobs and foreground externals are all fine.
+**It was never the kernel.** `ports/zsh/Makefile` answered configure's
+"does `kill(pid, 0)` return ESRCH correctly?" with `zsh_cv_sys_killesrch=no`,
+which makes zsh `#define ESRCH EINVAL`. `waitforpid()` ends its wait on
+`errno != ESRCH`; with the kernel correctly returning ESRCH (3) and zsh
+comparing against EINVAL (22), the shell waited for ever on a child it had
+already reaped.
 
-Ruled out: `setpgid` (real now), SMP, job control, zsh specifically, and the
-kernel's SIGCHLD/sigsuspend contract — `cmd/sigtest` proves that one and passes
-all eight checks. A full syscall trace of a hang has every return value
-correct.
+Two things to carry forward, both from getting it wrong first:
 
-The contradiction is at `Src/jobs.c:1569`: zsh's `waitforpid()` ends its loop
-on `ESRCH` from `kill(pid, 0)`, and the trace shows that `kill` returning ESRCH
-with the loop entered anyway. Next step is a musl-linked probe that does
-`fork; let the handler reap; kill(pid,0); print errno` — everything upstream of
-that line is known good.
+- **`zsh_cv_*` names answer the check's *question*, not a property.** `no` on
+  a check called "killesrch" reads as reassuring and means "broken".
+  Autoconf's cross-compile default there is `yes` — the line should not have
+  existed.
+- **mksh installs as `/bin/sh`, not `mksh`.** A test invoking `mksh -c` is a
+  `command not found`, and reading that as "mksh hangs too" is what kept the
+  search in the kernel for an extra day.
 
-Documented in full at `docs/history/kernel.md` section 7, which is where to
-start.
+The general lesson: when a probe reproducing a program's exact syscall
+sequence behaves correctly and the program does not, stop looking under the
+syscalls and look at how each was compiled.
+
+Full write-up: `docs/history/kernel.md` section 7.