signal: the return trampoline was Linux-only, and a test that proves the rest

Went looking for why `X=$(echo hi)` hangs every shell. Did not find it. Found
two other things, and built the instrument that rules out the largest suspect.

**The signal return trampoline was ABI-blind.** deliver_signals() manufactures
a return address for the handler -- seven bytes on the user stack reading
`mov eax, 15 ; syscall`. Fifteen is *Linux's* rt_sigreturn, which is right for
anything musl-linked, and that is every port and every shell. For a first-party
binary it is native syscall 15, which is `open`. So a handler in any cmd/
program ran, returned into the trampoline, called open() with whatever the
handler had left in its registers, got -ENOENT, and carried on executing the
stack bytes after the trampoline.

    pid 14 sigtest: trap 14 err 4 rip 0x46ca7 addr 0xfffffffffffffffe
      rax=fffffffffffffffe ... rsp=46ca0
      ustack: 46e30 46ca0 50f0000000fb8 0 0 0

rip is rsp+7 -- one past a seven-byte trampoline -- and the third word of the
stack is the trampoline's own bytes. It emits the number for the process's ABI
now; syscall_map is the thing that says which numbering this process speaks.

**SIG_BLOCK, SIG_UNBLOCK and SIG_SETMASK were not defined anywhere.**
sys_sigprocmask branched on bare 0, 1 and 2 with the names in comments, so
anything outside the kernel had to spell the number and hope it had the right
one. Defined in posix.h, used in the kernel.

**cmd/sigtest** asserts the SIGCHLD/sigsuspend contract every shell is built
on, and it is written so a permissive stub cannot pass: a blocked SIGCHLD stays
pending, sigsuspend delivers an *already pending* signal without spinning, the
handler runs exactly once, the mask is restored, the child is still reapable,
and a signal arriving mid-wait also wakes it. Eight checks, all green. The
kernel's half of that contract is now proven rather than assumed, which is what
lets the remaining bug be pursued somewhere else.

Along with it, `rt_sigsuspend` in lib/usys.S -- the syscall existed and the
first-party library simply never exposed it.

On the hang itself: it is not job control, not SMP, not zsh (mksh does it too),
and not any value the kernel returns -- a full syscall trace is in
docs/history/kernel.md section 7, which also records what the contradiction
narrows to. That section had said "root cause not yet isolated" with three
suspects; all three are now eliminated, and the note says what to do next
instead of what to guess.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/history/kernel.md

@@ -432,6 +432,88 @@ likelihood:
   its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails
   to find the job for reasons not yet traced.
 
+### 2026-09-21: narrowed a long way, still open
+
+Picked up again with the debugger that did not exist when the above was
+written. What is now *ruled out* is most of what that note suspected:
+
+- **Not `setpgid`/`getpgid`.** Both are real now — `proc_setpgid`, `p->pgid`,
+  `p->sid` — and `waitpid()` honours the POSIX pid selection rules.
+- **Not signal delivery, and not the sigsuspend contract.** `cmd/sigtest`
+  asserts the whole pattern every shell uses and passes all eight checks: a
+  blocked SIGCHLD stays pending, `sigsuspend` delivers an *already pending*
+  signal without spinning, the handler runs exactly once, the mask is restored,
+  and the child is still reapable afterwards. A permissive stub cannot pass it.
+- **Not SMP.** It hangs identically on `-smp 1` and `-smp 4`.
+- **Not job control.** `set +m` makes no difference, and a non-interactive
+  `zsh -c` hangs the same way.
+- **Not zsh.** `mksh -c 'X=$(echo hi)'` hangs in the same place.
+- **Not the values the kernel returns.** The full syscall trace of a hang is
+  below and every one of them is correct.
+
+The symptom is narrower than "job control": it is **`$(command
+substitution)`**, and only that. Pipelines work. Subshells work. Background
+jobs and `wait` work. Foreground external commands work.
+
+A complete trace of `X=$(echo inline)` in zsh, from the fork to the hang:
+
+```
+fork()                        -> 14
+sigprocmask x2                            (child_block)
+close(pipe write end)
+sigprocmask(SIG_UNBLOCK, SIGCHLD)         (child_unblock)
+  -> SIGCHLD delivered to zsh's handler
+     wait4(-1, WNOHANG|WUNTRACED|WCONTINUED) -> 14, status 0
+     getrusage()                             -> 0
+     wait4(same)                             -> -ECHILD
+     rt_sigreturn
+read(pipe)                    -> 7          ("inline\n")
+read(pipe)                    -> 0          (EOF)
+sigprocmask, close
+kill(14, 0)                   -> -ESRCH
+rt_sigsuspend(mask=SIGINT)    -> never returns
+```
+
+Everything there is right. The child is reaped with status 0, the pipe carries
+its output and closes, and `kill` on the reaped pid correctly says ESRCH.
+
+**And that last pair is the contradiction.** zsh's `waitforpid()`
+(`Src/jobs.c:1569`) is
+
+```c
+while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) {
+    ...
+    signal_suspend(SIGCHLD, wait_cmd);
+```
+
+with a comment above it saying, in as many words, that ESRCH from that `kill`
+is how the loop ends. The trace shows the `kill` returning ESRCH **and the
+loop being entered anyway**, which leaves only two possibilities: `errno` is
+not `ESRCH` when zsh reads it, or `kill()` did not return negative. errno is
+demonstrably fine elsewhere — `kill -0 31000` prints "no such process" from
+both zsh's builtin and toybox's `kill`, and `cat` of a missing file prints "No
+such file or directory".
+
+That is where it stands. The next step is to distinguish those two: a musl-
+linked probe that does exactly `fork; wait for the handler to reap; kill(pid,
+0); print errno`, because everything upstream of that line is now known good.
+
+Two real bugs were found on the way and fixed, neither of them this one:
+
+- **The signal return trampoline was ABI-blind.** `deliver_signals()` wrote
+  `mov eax, 15 ; syscall` as the handler's return address. Fifteen is *Linux's*
+  `rt_sigreturn`, correct for anything musl-linked; for a first-party binary it
+  is native syscall 15, which is `open`. So a handler in any `cmd/` program
+  ran, returned into the trampoline, called `open()` with the handler's
+  leftover registers, got `-ENOENT`, and carried on executing the stack bytes
+  after the trampoline. It emits the right number for the process's ABI now.
+  This is why `cmd/sigtest` had to exist before anything else could be trusted.
+- **`SIG_BLOCK`, `SIG_UNBLOCK` and `SIG_SETMASK` were not defined anywhere.**
+  The kernel branched on bare `0`, `1` and `2` with the names in comments, so
+  anything outside the kernel had to spell the number and hope.
+
+---
+
 This was left here deliberately rather than continuing to iterate blind:
 SIGCHLD delivery is left **enabled** (`kernel/proc/proc.c`'s `exit()`)
 since disabling it guarantees zero chance of mksh's wait loop ever exiting,

docs/plan/1.0.md

@@ -206,6 +206,24 @@ right then.
       in 18 after.** See [../architecture/kernel.md](../architecture/kernel.md),
       "Per-CPU state".
 
+- [ ] **`$(command substitution)` hangs every shell.** Not job control, not
+      SMP, not zsh — `mksh -c 'X=$(echo hi)'` hangs the same way, on one CPU,
+      with `set +m`. Pipelines, subshells, background jobs and foreground
+      externals all work; only `$( )` does not.
+
+      Narrowed a long way this session and still open. The kernel's half of
+      the SIGCHLD/sigsuspend contract is now *proven* rather than assumed —
+      `cmd/sigtest` asserts the whole pattern and passes — and a full syscall
+      trace of a hang shows every return value correct. The contradiction is
+      at one line of zsh's `waitforpid()`, which ends its loop on `ESRCH` from
+      `kill(pid, 0)`; the trace shows that `kill` returning ESRCH and the loop
+      being entered regardless. See
+      [../history/kernel.md](../history/kernel.md), section 7.
+
+      Two real bugs were found and fixed on the way: the signal-return
+      trampoline was hardcoded to Linux's `rt_sigreturn` number, so handlers
+      in first-party binaries called `open()` and then executed stack bytes;
+      and `SIG_BLOCK`/`SIG_UNBLOCK`/`SIG_SETMASK` were not defined anywhere.
 - [ ] **`getrusage` does not distinguish `RUSAGE_CHILDREN`** from `RUSAGE_SELF`.
 - [ ] **`/proc/<pid>/exe` is a regular file, not a symlink**, so `readlink()`
       fails and `ps -o comm` prints `[init]`. Symlinks themselves now work —