`X=$(echo hi)` never returned. It has been in docs/history/kernel.md section 7
as "mksh's job control still doesn't reliably progress ... root cause not yet
isolated" with three suspects, and it was none of them, and it was not the
kernel at all.
ports/zsh/Makefile carries a block of `zsh_cv_*` answers for the checks
autoconf cannot run when cross-compiling. One of them was
zsh_cv_sys_killesrch=no
which reads like a reassuring property and is in fact the answer to the
question configure asks: *does kill(pid, 0) return ESRCH correctly?* `no` makes
configure emit BROKEN_KILL_ESRCH, and Src/zsh_system.h answers that with
#ifdef BROKEN_KILL_ESRCH
#undef ESRCH
#define ESRCH EINVAL
#endif
zsh's waitforpid() ends its wait on `errno != ESRCH` (Src/jobs.c:1569). With
ESRCH redefined to EINVAL (22) and the kernel correctly returning ESRCH (3),
that condition was true for ever: the shell sat in sigsuspend waiting for a
child it had already reaped. An instrumented zsh says the whole thing in one
line -- kill returned -1, errno is 3, and the constant it is compared against
is 22:
WFPkill -1 3 22
**Autoconf's own cross-compilation default for that check is `yes`.** Deleting
the line would have been correct. Writing it down as "a fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.
Verified: `X=$(echo inline)`, `$(/bin/echo external)`, backticks, nested
`$(echo $(echo deep))`, a slow child, and `$(ls /bin | wc -l)` all return.
mksh -- as /bin/sh -- was never affected.
Two corrections to yesterday's note, both confident and both wrong:
- "Not zsh, mksh hangs the same way" was false. mksh installs as /bin/sh,
so that test was a `command not found` whose output scrolled past. Had the
binary been named right, this would have localised to zsh's build in one
step instead of a day.
- cmd/sigtest had already proven the kernel's SIGCHLD/sigsuspend contract,
and a musl-linked probe had already shown kill(pid,0) returning -1 with
errno == ESRCH == 3 and zsh's own loop condition evaluating false. Both
results were right; both were read as "so it must be subtler in the
kernel". What they actually pointed at was the one place the probe and the
shell differed -- how each was compiled.
The lesson is narrow enough to state: when a probe that reproduces a program's
exact syscall sequence behaves correctly and the program does not, stop looking
under the syscalls and look at the build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Went looking for why `X=$(echo hi)` hangs every shell. Did not find it. Found
two other things, and built the instrument that rules out the largest suspect.
**The signal return trampoline was ABI-blind.** deliver_signals() manufactures
a return address for the handler -- seven bytes on the user stack reading
`mov eax, 15 ; syscall`. Fifteen is *Linux's* rt_sigreturn, which is right for
anything musl-linked, and that is every port and every shell. For a first-party
binary it is native syscall 15, which is `open`. So a handler in any cmd/
program ran, returned into the trampoline, called open() with whatever the
handler had left in its registers, got -ENOENT, and carried on executing the
stack bytes after the trampoline.
pid 14 sigtest: trap 14 err 4 rip 0x46ca7 addr 0xfffffffffffffffe
rax=fffffffffffffffe ... rsp=46ca0
ustack: 46e30 46ca0 50f0000000fb8 0 0 0
rip is rsp+7 -- one past a seven-byte trampoline -- and the third word of the
stack is the trampoline's own bytes. It emits the number for the process's ABI
now; syscall_map is the thing that says which numbering this process speaks.
**SIG_BLOCK, SIG_UNBLOCK and SIG_SETMASK were not defined anywhere.**
sys_sigprocmask branched on bare 0, 1 and 2 with the names in comments, so
anything outside the kernel had to spell the number and hope it had the right
one. Defined in posix.h, used in the kernel.
**cmd/sigtest** asserts the SIGCHLD/sigsuspend contract every shell is built
on, and it is written so a permissive stub cannot pass: a blocked SIGCHLD stays
pending, sigsuspend delivers an *already pending* signal without spinning, the
handler runs exactly once, the mask is restored, the child is still reapable,
and a signal arriving mid-wait also wakes it. Eight checks, all green. The
kernel's half of that contract is now proven rather than assumed, which is what
lets the remaining bug be pursued somewhere else.
Along with it, `rt_sigsuspend` in lib/usys.S -- the syscall existed and the
first-party library simply never exposed it.
On the hang itself: it is not job control, not SMP, not zsh (mksh does it too),
and not any value the kernel returns -- a full syscall trace is in
docs/history/kernel.md section 7, which also records what the contradiction
narrows to. That section had said "root cause not yet isolated" with three
suspects; all three are now eliminated, and the note says what to do next
instead of what to guess.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One to four processes died per boot with a fault they could not have caused:
`rip 0x0` -- an instruction fetch at zero, which a process does not branch to
-- or a dereference of `0xffffffffffffffea`, which is `-EINVAL`. It hit shells
and schinitd alike and had been carried for a long time as "zsh sometimes jumps
into .data".
`syscall_entry` has to swap FS_BASE from the user's TLS to the kernel's before
it can touch any `%fs:` data, and `wrmsr` destroys rax, rcx and rdx. Those have
to be parked somewhere first -- not on the user stack, which is untrusted and
has not been switched away from yet. They were parked in three globals:
// Single-CPU scratch for syscall_entry FS swap
// (safe while IF cleared by SYSCALL/SFMASK)
uint64 _sc_rax, _sc_rcx, _sc_rdx;
IF being clear answers for an interrupt arriving on *this* CPU. It says nothing
whatever about a second CPU executing the same three stores at the same
instant, and with more than one processor entering syscalls that is not a rare
interleaving -- it is the ordinary one.
**%rcx is the user return address.** SYSCALL puts the user's rip there and
SYSRET takes it back from there. So the CPU that lost the race returned its
process into *another process's* instruction stream, with %rax -- the syscall
number on the way in, the return value on the way out -- swapped alongside it.
Both symptoms fall straight out of that: a zero handed over and followed, and
an errno used as a pointer.
Per-CPU words at %gs:32..48 now. %gs is usable there, before the FS swap, for
exactly the reason RESTORE_KERNEL_FS can already read %gs:0: GS base is per-CPU
in hardware, seginit() sets it once, and userspace cannot change it --
sys_arch_prctl() implements ARCH_SET_FS and ARCH_GET_FS and rejects the rest.
Measured rather than argued. Both variants built once, `build/esp` saved for
each, the same pristine schfs.img restored before every boot, -smp 4:
three globals 10 boots 10 faults, in five of the ten
three per-CPU 18 boots 0 faults
A _Static_assert pins all seven per-CPU offsets, because trapasm.S spells them
as literals and cannot include a header -- a slot that moves would leave a
register quietly holding somebody else's value, which is the hardest possible
thing to find.
**This is the second bug of this exact shape in this file.** The first was
`kernel_fs_base`, a single global every CPU wrote during its own seginit(), so
the last writer won and every CPU restored one CPU's per-CPU block;
RESTORE_KERNEL_FS reads %gs:0 for that reason. The lesson generalises and is
now written down in both places: **on the entry path, a global is a bug.**
Closes the row added to Gate 0 four commits ago. It was the right row to put
first -- a compositor built on this would have failed in ways indistinguishable
from its own bugs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two notes worth keeping out of the commit log because they are about where to
start next time rather than what changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nexoniarz