nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

zsh: one word in a cross-compile cache, and every $(...) hung

Browse source 
`X=$(echo hi)` never returned. It has been in docs/history/kernel.md section 7
as "mksh's job control still doesn't reliably progress ... root cause not yet
isolated" with three suspects, and it was none of them, and it was not the
kernel at all.

ports/zsh/Makefile carries a block of `zsh_cv_*` answers for the checks
autoconf cannot run when cross-compiling. One of them was

    zsh_cv_sys_killesrch=no

which reads like a reassuring property and is in fact the answer to the
question configure asks: *does kill(pid, 0) return ESRCH correctly?* `no` makes
configure emit BROKEN_KILL_ESRCH, and Src/zsh_system.h answers that with

    #ifdef BROKEN_KILL_ESRCH
    #undef ESRCH
    #define ESRCH EINVAL
    #endif

zsh's waitforpid() ends its wait on `errno != ESRCH` (Src/jobs.c:1569). With
ESRCH redefined to EINVAL (22) and the kernel correctly returning ESRCH (3),
that condition was true for ever: the shell sat in sigsuspend waiting for a
child it had already reaped. An instrumented zsh says the whole thing in one
line -- kill returned -1, errno is 3, and the constant it is compared against
is 22:

    WFPkill -1 3 22

**Autoconf's own cross-compilation default for that check is `yes`.** Deleting
the line would have been correct. Writing it down as "a fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.

Verified: `X=$(echo inline)`, `$(/bin/echo external)`, backticks, nested
`$(echo $(echo deep))`, a slow child, and `$(ls /bin | wc -l)` all return.
mksh -- as /bin/sh -- was never affected.

Two corrections to yesterday's note, both confident and both wrong:

  - "Not zsh, mksh hangs the same way" was false. mksh installs as /bin/sh,
    so that test was a `command not found` whose output scrolled past. Had the
    binary been named right, this would have localised to zsh's build in one
    step instead of a day.
  - cmd/sigtest had already proven the kernel's SIGCHLD/sigsuspend contract,
    and a musl-linked probe had already shown kill(pid,0) returning -1 with
    errno == ESRCH == 3 and zsh's own loop condition evaluating false. Both
    results were right; both were read as "so it must be subtler in the
    kernel". What they actually pointed at was the one place the probe and the
    shell differed -- how each was compiled.

The lesson is narrow enough to state: when a probe that reproduces a program's
exact syscall sequence behaves correctly and the program does not, stop looking
under the syscalls and look at the build.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-09-22 00:29:39 +02:00
commit b012060699
4 changed files with 112 additions and 108 deletions
Download patch file Download diff file Expand all files Collapse all files

127
docs/history/kernel.md
View file

@ -432,85 +432,78 @@ likelihood:
its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails
to find the job for reasons not yet traced.
### 2026-09-21: narrowed a long way, still open
### Resolved 2026-09-22: it was never the kernel
Picked up again with the debugger that did not exist when the above was
written. What is now *ruled out* is most of what that note suspected:
**`X=$(...)` hung because zsh was built with `ESRCH` defined as `EINVAL`.**
- **Not `setpgid`/`getpgid`.** Both are real now — `proc_setpgid`, `p->pgid`,
`p->sid` — and `waitpid()` honours the POSIX pid selection rules.
- **Not signal delivery, and not the sigsuspend contract.** `cmd/sigtest`
asserts the whole pattern every shell uses and passes all eight checks: a
blocked SIGCHLD stays pending, `sigsuspend` delivers an *already pending*
signal without spinning, the handler runs exactly once, the mask is restored,
and the child is still reapable afterwards. A permissive stub cannot pass it.
- **Not SMP.** It hangs identically on `-smp 1` and `-smp 4`.
- **Not job control.** `set +m` makes no difference, and a non-interactive
`zsh -c` hangs the same way.
- **Not zsh.** `mksh -c 'X=$(echo hi)'` hangs in the same place.
- **Not the values the kernel returns.** The full syscall trace of a hang is
below and every one of them is correct.
`ports/zsh/Makefile` carries a block of `zsh_cv_*` cache answers, for the
checks autoconf cannot run when cross-compiling. One of them was
The symptom is narrower than "job control": it is **`$(command
substitution)`**, and only that. Pipelines work. Subshells work. Background
jobs and `wait` work. Foreground external commands work.
A complete trace of `X=$(echo inline)` in zsh, from the fork to the hang:
```
fork() -> 14
sigprocmask x2 (child_block)
close(pipe write end)
sigprocmask(SIG_UNBLOCK, SIGCHLD) (child_unblock)
-> SIGCHLD delivered to zsh's handler
wait4(-1, WNOHANG|WUNTRACED|WCONTINUED) -> 14, status 0
getrusage() -> 0
wait4(same) -> -ECHILD
rt_sigreturn
read(pipe) -> 7 ("inline\n")
read(pipe) -> 0 (EOF)
sigprocmask, close
kill(14, 0) -> -ESRCH
rt_sigsuspend(mask=SIGINT) -> never returns
```make
zsh_cv_sys_killesrch=no
```
Everything there is right. The child is reaped with status 0, the pipe carries
its output and closes, and `kill` on the reaped pid correctly says ESRCH.
**And that last pair is the contradiction.** zsh's `waitforpid()`
(`Src/jobs.c:1569`) is
Read as a property that looks reassuring. Read as the answer to the question
configure actually asks — *"does `kill(pid, 0)` return ESRCH correctly?"* — it
says the opposite. `no` makes configure emit `BROKEN_KILL_ESRCH`, and
`Src/zsh_system.h` answers that with:
```c
while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) {
...
signal_suspend(SIGCHLD, wait_cmd);
#ifdef BROKEN_KILL_ESRCH
#undef ESRCH
#define ESRCH EINVAL
#endif
```
with a comment above it saying, in as many words, that ESRCH from that `kill`
is how the loop ends. The trace shows the `kill` returning ESRCH **and the
loop being entered anyway**, which leaves only two possibilities: `errno` is
not `ESRCH` when zsh reads it, or `kill()` did not return negative. errno is
demonstrably fine elsewhere — `kill -0 31000` prints "no such process" from
both zsh's builtin and toybox's `kill`, and `cat` of a missing file prints "No
such file or directory".
zsh's `waitforpid()` (`Src/jobs.c:1569`) ends its wait on
That is where it stands. The next step is to distinguish those two: a musl-
linked probe that does exactly `fork; wait for the handler to reap; kill(pid,
0); print errno`, because everything upstream of that line is now known good.
```c
while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH))
```
Two real bugs were found on the way and fixed, neither of them this one:
so with `ESRCH` redefined to `EINVAL` (22) and the kernel correctly returning
`ESRCH` (3), that condition was true for ever. The shell waited, in
`sigsuspend`, for a child it had already reaped. Every command substitution in
the shell hung.
- **The signal return trampoline was ABI-blind.** `deliver_signals()` wrote
`mov eax, 15 ; syscall` as the handler's return address. Fifteen is *Linux's*
`rt_sigreturn`, correct for anything musl-linked; for a first-party binary it
is native syscall 15, which is `open`. So a handler in any `cmd/` program
ran, returned into the trampoline, called `open()` with the handler's
leftover registers, got `-ENOENT`, and carried on executing the stack bytes
after the trampoline. It emits the right number for the process's ABI now.
This is why `cmd/sigtest` had to exist before anything else could be trusted.
- **`SIG_BLOCK`, `SIG_UNBLOCK` and `SIG_SETMASK` were not defined anywhere.**
The kernel branched on bare `0`, `1` and `2` with the names in comments, so
anything outside the kernel had to spell the number and hope.
The instrumented build says it in one line — `kill` returned −1, `errno` is 3,
and the constant it is compared against is 22:
```
WFPkill -1 3 22
```
**Autoconf's own cross-compilation default for this check is `yes`.** Deleting
the line would have been correct. Writing it down as a "fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.
#### What this cost, and what to take from it
This sat in section 7 as "mksh's job control still doesn't reliably progress
... root cause not yet isolated" with three suspects, none of which were it.
All three have since been eliminated on their own merits — `setpgid` and
`getpgid` are real, `getrusage` is not consulted on the path, and the pid
matching was never reached.
Two corrections to the note written on 2026-09-21, both worth keeping because
they were confident and wrong:
- **"Not zsh — `mksh -c` hangs the same way" was false.** mksh installs as
`/bin/sh`, not `mksh`, so that test was a `command not found` whose output
scrolled past. mksh was never affected. Had the binary been named correctly
the bug would have been localised to zsh's build in one step.
- **The kernel was cleared and then still suspected.** `cmd/sigtest` had
already proven the SIGCHLD/sigsuspend contract, and a musl-linked probe had
already shown `kill(pid, 0)` returning −1 with `errno == ESRCH == 3` and
zsh's own loop condition evaluating false. Both results were correct and
both were read as "so it must be something subtler in the kernel". The thing
they actually pointed at was the one place the probe and the shell differed:
**how each was compiled.**
The lesson is narrow and worth stating: when a probe that reproduces a
program's exact syscall sequence behaves correctly and the program does not,
stop looking under the syscalls and start looking at the build.
---