# Kernel history Bugs that were expensive to find and whose causes are not visible from the code that fixed them, plus a few subsystem write-ups that have nowhere better to live. **Every entry here is a thing that looked like a different problem than it was.** What each subsystem *is* today is [../architecture/](../architecture/); this is only how it got that way. | | | |---|---| | [1](#1-fixed-boot-didnt-reliably-reach-a-working-shell) | Boot did not reliably reach a working shell — ten bugs behind one symptom | | [2](#2-naming-standardization) · [3](#3-coreutils) | Naming, and where the userland comes from. Not bugs. | | [4](#4-two-stage-bootloader--psogm-splash) | The two-stage BIOS bootloader. **Since removed** — kept for the bugs. | | [5](#5-at-syscalls-ignoredrejected-real-dirfds) | `*at()` ignored real dirfds, and `ls -l` hung | | [6](#6-boot-speed) | Boot speed, in the loader | | [7](#7-at-syscalls-real-signal-delivery-and-mksh-job-control) | Real signal delivery, and mksh's job control | | [7b](#7b-the-loader-jumped-into-the-framebuffer-and-executed-pixels) | The loader jumped into the framebuffer and executed pixels | | [8](#8-two-abi-assumptions-that-only-gcc-was-keeping-true) | Two ABI assumptions only GCC was keeping true | | [9](#9-the-boot-looked-like-a-hobby-os-and-two-things-were-why) | The boot looked like a hobby OS | | [10](#10-a-full-screen-program-typed-the-whole-session-into-the-shell) | A full-screen program typed the session into the shell | | [11](#11-ps-and-top-and-why-they-never-worked) | `ps` and `top`, and why they never worked | | [12](#12-a-test-program-crashed-at-random-and-had-been-typing-its-own-launch-command) | A program crashed at random, typing its own launch command | The SMP bring-up bugs are not here — they are summarised in [../architecture/boot.md](../architecture/boot.md#4-smp), because the lesson (`__thread` is wrong for per-CPU data) is load-bearing on the current code rather than merely historical. > **Where the old design plan went.** Sections 1-6 of this file used to hold a > "Segregated Kernel" design: a Ring 0 Alpha / Ring 0 Beta split with driver > modules isolated by CR3 switching, from an `IDEA.txt` that no longer exists. > None of it was ever implemented, and it is superseded by > [../plan/schapi.md](../plan/schapi.md) — a macOS-shaped system answers driver isolation with > userspace drivers and a personality boundary, not a bespoke in-kernel ring > split. It also referenced a Zig toolchain that is now clang/LLVM. Removed > rather than left to be mistaken for a plan. --- ## 1. Fixed: boot didn't reliably reach a working shell Investigated 2026-07-05, before phase 1 started (predates this design, not caused by it). Originally manifested as `ksh` (mksh) crashing on every launch, apparently inside musl's mallocng allocator (`get_meta`, `src/malloc/mallocng/meta.h`, deliberately executing `hlt` on detecting "corrupted" heap metadata — a `trap 13` GPF at a fixed rip). It wasn't a musl bug. Six real kernel bugs were found and fixed across three investigation passes, and mksh boots and runs cleanly with all of them in place — the calloc assertion was a symptom of real memory/pointer-handling bugs downstream in the kernel, not anything musl or mallocng did wrong. Fixed: - `copyuvm()` (`kernel/arch/x86_64/vm.c`, used by `fork()`) never copied `mmap()`-backed `proc->vma[]` regions into the child — only `[0, sz)`. Fixed via new `copy_vma_range()`, called from `fork()` (`kernel/proc/proc.c`). - `kmem.freelist` (`kernel/core/kalloc.c`) could get corrupted by a wild write elsewhere and would silently hand back a bad pointer, panicking far from the actual cause. Added a canary check (`check_run()`) at the top of both `kalloc()`/`kfree()` so corruption panics immediately at the point it's touched, with a clear message. - No TLB invalidation existed anywhere except the full `lcr3` reload on context switch — `mappages()`/`vm_unmap_range()` never called `invlpg`. Fixed by adding `invlpg()` (`include/x86.h`) to both functions. - **`MSR_STAR` was misconfigured** (`syscallinit()`, `kernel/arch/x86_64/vm.c`): `SYSRETQ` computes the returned `cs`/`ss` as `STAR[63:48]+16`/`+8`, which requires the *compat* 32-bit code selector (`SEG_UCODE32`) in that field — the kernel was putting the real 64-bit `USER_CS` there instead. Every `sysretq` silently loaded an invalid `cs` (`0x3b`, pointing at the TSS descriptor)/`ss` pair; QEMU's TCG doesn't validate segment descriptors on `SYSRET`, so this went unnoticed until a hardware interrupt forced a strictly-validated `IRETQ` with that bogus `cs` still loaded, which correctly `#GP`'d. - `sys_mmap`'s non-`MAP_FIXED` path (`kernel/sys/sys_mem.c`) assigned a fixed 8MB virtual-address window per `vma[]` slot regardless of the requested length. A single allocation over 8MB (e.g. `realloc()` growth past that size) mapped straight into a neighboring live mapping and hit `mappages()`'s `panic("remap")`. Replaced with a real bump allocator (`proc->mmap_next`, `include/proc.h`), so every mapping gets its own disjoint, exactly-sized range regardless of size. - **`uva2ka()`** (`kernel/arch/x86_64/vm.c`) returned only the page-aligned base address of a user pointer's page, silently dropping the in-page byte offset. Direct callers (`sys_ioctl`'s `TIOCGPGRP`/`TCGETS`, `sys_getcwd`, signal masks, `times()`, `sysinfo()`, ...) read/wrote near the wrong address whenever the pointer wasn't page-aligned — i.e. almost always. Fixed at the source so every caller benefits. - **`fetchint`/`fetchaddr`/`fetchstr`/`argptr`** (`kernel/sys/syscall.c`) rejected any address `>= proc->sz`, wrongly assuming every valid user pointer lives in the heap. Stack addresses and `mmap()`'d buffers (used constantly by real programs, e.g. path-resolution buffers) live above `proc->sz` and were being rejected outright. Fixed to rely solely on `uva2ka()`'s page-table validation. Bringing in toybox (see "Coreutils" below) surfaced four more, all fixed: - Linux syscall 58 (`vfork`) wasn't in the syscall-ABI translation table at all (`kernel/sys/linux_abi.c`) — every external command spawn from a program that prefers `vfork()` over `fork()` failed outright. Mapped to the existing `fork()` (a fully correct, if slightly slower, substitute). - `sys_stat`/`fstat`/`fstatat` (`kernel/sys/sys_fs.c`) filled in SchOS's own native `struct stat` layout regardless of the caller — musl/toybox parse a fixed Linux ABI byte layout instead, so `st_mode` was garbage and tools like `ls` couldn't tell files from directories, or executables from non-executables. Added `struct linux_stat` + `stati_linux()`, gated on Linux-ABI mode (`proc->syscall_map`). - `sys_open`/`sys_openat` required `omode == O_RDONLY` *exactly* to open a directory, but `opendir()`-style calls legitimately OR in `O_DIRECTORY`/ `O_CLOEXEC` alongside it. Fixed to check just the access-mode bits (`O_ACCMODE`, added to `include/fcntl.h`). - `sys_getdents64` read one `struct dirent` (16 bytes) at a time — a full `bmap()`+`bread()` per slot. Directories here are hash-indexed (see `tree_search()`/`tree_expand()` below), so a directory with a few dozen real entries can have a size in the hundreds of KB, nearly all empty slots — this made listing such a directory look hung (it wasn't a deadlock, just thousands of redundant single-slot disk reads). Rewrote it to read a full block at a time and resolve each entry's type via `iget(inum)` (exported from `fs.c`, it was `static`) instead of re-searching the whole directory by name with `dirlookup()` for every entry. That rewrite's first version introduced a **new** self-deadlock: the "." entry's inode number equals the directory's own, and `iget()` correctly hands back the *same, already-locked* `struct inode*` — calling `ilock()` on it again sleeps forever on a lock the process already holds. Fixed by special-casing `xde->inum == ip->inum` (we already know it's a directory, no need to re-lock at all). A synthetic stress test (interleaved malloc/free, realloc growth crossing `MMAP_THRESHOLD`, calloc, large allocations) runs to completion with zero crashes, and mksh now boots straight to an interactive prompt, runs external programs, pipes, and `ls`/`mkdir`/`grep` without issue. ## 2. Naming standardization SchOS standardizes on exactly one name per role, regardless of the underlying implementation: the shell is always installed and exec'd as `/bin/sh` (currently mksh, built from `ports/mksh/`), and PID 1 is always `/bin/init`. `cmd/init/main.c`'s exec loop only ever tries `/bin/sh` — no ksh/mksh/dash/zsh/other-name fallbacks. dash was tried first (BSD-licensed, minimal) while the boot crash above was still open, then replaced with mksh once the underlying kernel bugs were fixed and dash's lack of interactive features (history, completion, line editing) made it feel unusably bare for day-to-day use; the dash port's source tree was left in place rather than deleted, in case it's useful again (e.g. as a `/bin/sh` POSIX-strict target distinct from an interactive login shell). ## 3. Coreutils `ports/toybox/` (0BSD-licensed, https://landley.net/toybox/) provides `ls`, `cat`, `cp`, `mv`, `rm`, `mkdir`, `grep`, `sed`, `find`, `chmod`, `date`, `sort`, and more — see the `APPLETS` list in `ports/toybox/Makefile`. It builds as a single multicall binary; `tools/mkfs.c` doesn't understand symlinks or hardlink dedup, so `ports/toybox/Makefile`'s `install` target copies the built binary once per applet name into `sysroot/bin/` rather than symlinking (costs disk space per applet — currently ~44MB total — instead of one shared inode, but needed no mkfs changes). ## 4. Two-stage bootloader + PSOGM splash > **This describes the BIOS loader, which no longer exists** — see > [`boot/README.md`](../../boot/README.md) for why it was removed and what > replaced it. The section is kept because four of the bugs in it are about > things that are still true: extended scancodes, register hand-off across a > mode switch, and layout constants that several artifacts must agree on. `boot/` used to be a single BIOS boot sector doing everything (VBE mode set, A20, GDT, protected-mode switch, multiboot kernel scan/load) — silent, and packed to within 4 bytes of the hard 510-byte MBR limit, so there was no room to add so much as a print statement. Split into two stages, the standard way real bootloaders solve this: - **Stage 1** (`boot/bios/bootasm.S`, still the literal MBR sector, still ≤510 bytes) does exactly one thing: raw IDE PIO (LBA28) reads `STAGE2_SECTORS` (16) sectors starting at disk sector 1 into memory at `0x7E00`, then jumps there. Nothing else fits, nothing else is needed. (Caught a real bug writing this: `outb $0x1F7,%al`-style immediates silently truncate to 8 bits — ports 0x1F2-0x1F7 are all > 0xFF and need the `%dx` register form, or you're hitting the wrong I/O port entirely.) - **Stage 2** (`boot/bios/stage2.S` + `boot/bios/stage2main.c`, loaded at `0x7E00`, occupies disk sectors 1-16) does everything stage 1 used to skip straight to, plus a boot banner (BIOS teletype, real mode, before the VBE mode switch — note this only reaches the video device, not serial, so it won't show up in a `-serial file:...`-captured log), a splash screen, a real-time loading bar, and an F8 boot menu (all described below). Not size-constrained the same way as stage 1 — `tools/padstage2.pl` zero-pads it out to a fixed `STAGE2_SECTORS*512` (8192) bytes so the kernel's on-disk starting sector stays predictable regardless of stage2's actual compiled size, and errors out (like `tools/sign.pl` does for stage 1) if it ever grows past that budget instead of silently corrupting the layout. The kernel now starts at disk sector 17 instead of 1 (`schos.img`'s `dd ... seek=17` in the root `Makefile`, and the equivalent `+(1+STAGE2_SECTORS)` in `stage2main.c`'s `readseg()` — all three places that encode `STAGE2_SECTORS` have to agree, or the kernel won't be found on disk at all). `boot/bootmain.c` no longer exists; its logic was split between `stage2.S` (real-mode/protected-mode setup) and `stage2main.c` (splash + the unchanged multiboot-header-scan-and-load logic). The VBE mode info block itself lives at `0x30000`, not the classic single-stage bootloader's `0x9000` — stage2's own code now runs from `0x7E00` up to nearly `0x9000` once the boot-menu/progress-bar logic is included, and a block at `0x9000` gets partially overwritten by stage2's own tail end (found this the hard way: the kernel's `LFB:` cprintf started reporting a different wrong value on every boot, because it was reading whatever stage2 code/data now happened to occupy that address). `kernel/dev/vga.c`'s `vga_init()` has to agree on `0x30000` too, since it independently re-reads mode fields from the same block. Splash/progress bar/boot menu rendering is a small standalone pixel renderer in `stage2main.c` (not the kernel's `vga.c` — paging isn't set up yet, so the LFB physical address from the VBE block is used as a plain pointer) reusing the kernel's own font data (`include/font.h`) directly. It fills the framebuffer with PSOGM's purple (matching `kernel/core/console.c`'s panic screen) and draws "SCHOS" centered, then: - **Real-time loading bar**: `readsect()` (in `stage2main.c`, not the kernel) calls `progress_tick()` after every 512-byte disk read, updating an actual pixel-filled bar against the real total byte count — first a small one while scanning for the multiboot header, then a second, properly-sized one once the header's found and the true kernel size is known, labeled "Loading kernel...". Not simulated/timed — it reflects real bytes read off disk. - **F8 boot menu**: `wait_for_f8()` busy-polls the keyboard controller (`inb(0x64)`/`inb(0x60)`, raw scancodes — real-mode BIOS keyboard calls aren't available once we're in protected mode) for about a second after the splash appears. F8 (scancode `0x42`) drops into `boot_menu()`: three options (boot normal, boot verbose, reboot to firmware setup) navigated with up/down arrows and Enter, plus a "detected drives" panel from `ata_present()` — a standard floating-bus presence check (select the drive, write a pattern to two registers, read it back) on the primary IDE channel's master/slave (the only channel this project ever actually puts drives on; QEMU's IDE emulation answers the same test "successfully" on the unpopulated secondary channel too, so that channel isn't shown — it'd just be confidently wrong). "Reboot to firmware setup" pulses the keyboard controller's reset line (`outb(0x64, 0xFE)`, same technique as `kernel/sys/sys_proc.c`'s `sys_reboot()`); a real bootloader can't jump into BIOS/UEFI setup directly, so this is the closest honest equivalent — it gives the user a fresh POST during which their firmware's own setup-menu key works. - **Verbose boot**: picking "Boot SchOS (verbose)" writes a flag byte to `BOOT_VERBOSE_FLAG_ADDR` (`VBE_INFO_ADDR + 0x100`, i.e. just past the VBE block) before jumping to the kernel. `kernel/core/main.c`'s `kmain()` reads it into a global `boot_verbose` as its very first action and, if set, prints a line after every subsystem init call (`uartearlyinit`, `kinit1`, ..., `sinit`) — real per-step boot logging, not a canned message. Two bugs worth remembering if this code gets touched again: - **Keyboard extended-scancode handling**: `0xE0` (the prefix byte for arrow keys and friends) also has bit 7 set, the same bit used to detect key-*release* ("break") codes. A naive "skip anything with bit 7 set, then check for 0xE0" loop skips the 0xE0 byte itself before ever testing for it, so extended keys silently never register. The 0xE0 check has to come before the break-code check, not after. - **The kernel entry call is not a real C function call.** Both the original single-stage `bootmain()` and the new `stage2main()` do `((void(*)(uint32))entry)(fbaddr)` — normal cdecl, which pushes `fbaddr` on the stack. But `kernel/arch/x86_64/entry.S`'s `mboot_entry` reads its framebuffer argument out of **EAX** (a multiboot-style raw-register handoff, then `mov %eax,%ebp` / `mov %ebp,%ebx` / `mov %rbx,%rdi` carries it through the 32→64-bit transition into `kmain()`'s first argument). The cdecl call never puts anything in EAX — it "worked" for years purely because simple bootmain.c code happened, as a side effect of its own codegen, to leave the right value sitting in EAX right up until the `call` instruction. Adding the splash/menu/progress-bar code changed that codegen and broke it silently: the kernel booted fine and reported a plausible-looking but wrong `LFB:` address (garbage left over in EAX from unrelated prior code, different on every boot). Fixed by pinning the argument down explicitly with `asm volatile("jmp *%0" : : "r"(entry), "a"(fbaddr))` instead of hoping the compiler's register allocation cooperates. If this code moves around again, re-verify with a screendump taken *immediately* before the jump compared against the kernel's own `LFB:` log line — they silently diverging is exactly this bug. ## 5. `*at()` syscalls ignored/rejected real dirfds `ls -l`/`ls -lR` (and, by the same code path, `find`/`du`/anything using toybox's `lib/dirtree.c`) would hang on `ls -l` and either hang or silently misbehave on recursive listings. Root cause: `kernel/sys/sys_fs.c`'s `*at()` syscalls (`openat`, `fstatat`, `unlinkat`, `renameat`, `linkat`, `symlinkat`, `mkdirat`, `mknodat`, `readlinkat`, `faccessat`) only handled two cases for their `dirfd` argument — `AT_FDCWD` (resolve relative to `proc->cwd`) or an absolute path (resolve from root, ignoring `dirfd` entirely). A **real** dirfd with a **relative** path — e.g. `fstatat(open("/bin", O_RDONLY), "ls", &st, ...)`, the standard, efficient way any directory-tree walker avoids rebuilding full path strings for every entry — was either rejected outright (`fstatat`, `unlinkat`, ...) or silently ignored in favor of `proc->cwd` (`openat`, which never checked at all — worse than rejecting, since it can silently resolve the *wrong* file if cwd happens to contain a same-named entry). Fixed by adding a real dirfd-relative path resolver: `kernel/fs/fs.c`'s `namex()` was refactored into `namex_from(struct inode *start, ...)`, parameterizing what used to be a hardcoded `idup(proc->cwd)`, with two new exported entry points, `nameifrom()`/`nameiparentfrom()`. `sys_fs.c` gained `at_dir(dirfd)` (resolves `AT_FDCWD` or a real fd to a starting inode) and `at_namei()`/`at_nameiparent()` built on top of it, plus a `createat()` (parameterizing `create()`'s hardcoded `nameiparent()` call the same way). Every `*at()` syscall now uses these instead of its own inline absolute-path/`AT_FDCWD`-only check. **Known follow-on issue, not yet root-caused**: heavy `ls`/`ls -lR` use across many commands in one long-running shell session can eventually make a *later*, unrelated directory hang (reproduced with `ls -lR /var` after ~10 prior commands in the same session; the identical command succeeds immediately on a fresh boot). Not an inode-table or file-table exhaustion — both `iget()` and `filealloc()` fail cleanly (panic / return 0 respectively) when full, and neither fired. Smells like a lock or reference-count leak somewhere in the *at() paths above, or in toybox's own fd handling, that only manifests after enough accumulated opens/closes. Next step: instrument `itable`/`ftable` occupancy counts and watch them across a long `ls`-heavy session to see what's actually climbing. ## 6. Boot speed Two cheap, real wins in `boot/bios/stage2main.c`, no functional changes: - `progress_tick()` (called once per 512-byte disk sector) used to trigger a full progress-bar redraw — border plus fill, a few thousand pixel writes — on *every* sector. For the ~330KB kernel image that's ~650 full redraws (the multiboot-header-scan phase adds another ~32) for a bar that only has 101 visually distinct states (0-100%). Split into `progress_border()` (drawn once, in `progress_start()`, since the border never changes) and `progress_fill()`, and `progress_tick()` now skips the redraw entirely unless the percentage actually changed. This was the single biggest chunk of the bootloader's own wall-clock time. - `fill_rect()` (the full-screen splash fill, 480,000 pixels at 800x600, run at least twice per boot) used to call `draw_pixel()` per pixel, which recomputes `g_fb + y*stride + x*4` from scratch every time. Now computes each row's base pointer once and indexes into it. - The F8 boot-menu wait was shortened from ~1s (`5000000` busy-loop iterations) to ~0.6s (`3000000`) — flat overhead on every boot whether or not the menu is wanted, so it's the most direct lever available without giving up the feature entirely. ## 7. `*at()` syscalls, real signal delivery, and mksh job control > **Resolved since this was written.** Process groups and sessions are real > (`pgid`/`sid` in `struct proc`, `setpgid`/`setsid`/`getpgid`/`getsid`, and > signal delivery to a whole group), which was the first of the three suspects > listed at the end. mksh's job control works. The investigation is kept in full > because the signal-delivery mechanism described below is still exactly how the > kernel does it, and because the reasoning about *why SIGCHLD was left enabled > while broken* is the transferable part. `ls -l`/`ls -la` hanging outright turned out to be the same root cause described in §5 above (dirfd-relative path resolution) — fixed and verified extensively (plain `ls`, `ls -l`, `ls -la`, `ls -lR`, piped output, all confirmed working repeatedly). A *second*, deeper issue was found investigating a follow-up report (`cd` with no `HOME` set, then `cd var`, then `ls` hanging): mksh's interactive job control (`ports/mksh (mksh R59c source): jobs.c`) blocks in `sigsuspend()` waiting for `SIGCHLD`, expecting its own handler (`j_sigchld()`) to run, reap the child via `waitpid(-1, &status, WNOHANG)`, and update its internal job-state tracking. This kernel had **no real signal delivery at all** — `sys_rt_sigaction()` was a no-op stub, so no handler was ever invoked for anything, and `sigsuspend()` just slept out one tick and returned `-EINTR` unconditionally. mksh's wait loop had no way to ever learn the child had exited, hanging forever. Real signal delivery was implemented from scratch: - `struct proc` (`include/proc.h`) gained `sig_handler[32]`/ `sig_restorer[32]` (populated by `sys_rt_sigaction()`, `kernel/sys/sys_sig.c`), `saved_sigtf` (stashed pre-signal trapframe), and `in_signal` (re-entrancy guard). `allocproc()` zeroes these for a freshly claimed ptable slot (they weren't before — a reused slot silently inherited a previous, unrelated process's handler addresses, which crashed the *next* process to occupy that slot the moment anything tried to delivver a signal to it). `fork()` copies handler/ restorer/mask from parent to child (POSIX semantics) but not pending signals. - `kernel/proc/proc.c`'s `exit()` sets `SIGCHLD` pending on the parent. - `kernel/sys/syscall.c`'s `syscall()` calls `deliver_signals(tf)` in its tail, right before returning to user mode: if there's a pending, unblocked signal with a registered handler, it redirects the trapframe to jump to the handler (`rdi`=signal number per SysV ABI, `rip`=handler) instead of resuming the interrupted code, having first stashed the original trapframe in `proc->saved_sigtf`. - The return address is **not** `sig_restorer[sig]` — checked, `SA_RESTORER` is not defined anywhere in this musl build, so `sigaction()` never fills in a restorer; it's always 0, and pushing 0 as a return address crashes with `rip=0` the instant the handler function returns. Instead, `deliver_signals()` injects a tiny 7-byte trampoline (`mov eax,15; syscall` — `rt_sigreturn`) directly onto the user's own stack, below the current `rsp` (clear of the SysV red zone), and points the return address at that — the standard technique for when the C library doesn't supply its own restorer. - `sys_rt_sigreturn()` restores `*proc->tf` from `proc->saved_sigtf`, clearing `in_signal`. **A real bug found and fixed along the way**: the first version of `sys_rt_sigaction()` translated the user's `struct sigaction` pointer *once* via `uva2ka()` and then did a bulk `memset`+copy across the whole ~152-byte structure. `uva2ka()` only guarantees the *page containing that address* is mapped, not everything up to some further offset — when the user's pointer landed close enough to a page boundary, the write ran past the mapped page into whatever physical page happened to be next in kernel address space (P2V is a flat offset, so "next page" there has nothing to do with the user's actual next page), corrupting unrelated kernel memory. This crashed every freshly `exec`'d `/bin/sh` almost immediately, since mksh calls `sigaction()` during its own startup. Fixed by only touching the two individual 8-byte fields actually needed (`sa_handler` at offset 0, `sa_restorer` at offset 144), each through its own `uva2ka()` call. `sys_waitpid()`/`sys_wait4()` (`kernel/sys/sys_proc.c`) were also rewritten on top of a new `waitpid(pid, status, options)` in `kernel/proc/proc.c` (refactored from the old any-child-only, always-blocking `wait()`, which is now just `waitpid(-1, 0, 0)`): a specific `pid` (not just "any child"), `WNOHANG` (return 0 immediately instead of blocking), and — importantly — a genuinely optional status pointer. The old code ran a NULL status pointer through `argptr()`, which rejects NULL/unmapped addresses outright (page 0 is deliberately unmapped) — but `waitpid(pid, NULL, WNOHANG)` is completely standard, and is exactly what mksh's job control does. **Status: signal delivery mechanically works** — verified directly (a debug build logged the handler being invoked with the correct trapframe and `sys_rt_sigreturn()` restoring cleanly, no crash, repeatable). **mksh's job control still doesn't reliably progress afterward**, including for plain non-piped foreground commands, not just the original piped-into-head case. Root cause not yet isolated. Leading suspects, in order of likelihood: - `sys_setpgid()`/`sys_getpgid()` are no-op stubs (`kernel/sys/sys_proc.c`) — `j_sigchld()` (`ports/mksh (mksh R59c source): jobs.c:1344-1348`) has an early-out keyed on `j->ppid == procpid`, which may never resolve the way mksh expects without real process-group tracking. - `getrusage(RUSAGE_CHILDREN, ...)` — called at the top of `j_sigchld()` — isn't verified to behave sanely under this kernel's implementation (`kernel/sys/sys_proc.c`'s `sys_getrusage()` only fills in `ru_utime` from the *calling* process's own tick count, not accumulated child time; `RUSAGE_CHILDREN` isn't distinguished from `RUSAGE_SELF` at all). - Possibly something in how `j_sigchld()` matches a reaped `pid` against its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails to find the job for reasons not yet traced. ### Resolved 2026-09-22: it was never the kernel **`X=$(...)` hung because zsh was built with `ESRCH` defined as `EINVAL`.** `ports/zsh/Makefile` carries a block of `zsh_cv_*` cache answers, for the checks autoconf cannot run when cross-compiling. One of them was ```make zsh_cv_sys_killesrch=no ``` Read as a property that looks reassuring. Read as the answer to the question configure actually asks — *"does `kill(pid, 0)` return ESRCH correctly?"* — it says the opposite. `no` makes configure emit `BROKEN_KILL_ESRCH`, and `Src/zsh_system.h` answers that with: ```c #ifdef BROKEN_KILL_ESRCH #undef ESRCH #define ESRCH EINVAL #endif ``` zsh's `waitforpid()` (`Src/jobs.c:1569`) ends its wait on ```c while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH)) ``` so with `ESRCH` redefined to `EINVAL` (22) and the kernel correctly returning `ESRCH` (3), that condition was true for ever. The shell waited, in `sigsuspend`, for a child it had already reaped. Every command substitution in the shell hung. The instrumented build says it in one line — `kill` returned −1, `errno` is 3, and the constant it is compared against is 22: ``` WFPkill -1 3 22 ``` **Autoconf's own cross-compilation default for this check is `yes`.** Deleting the line would have been correct. Writing it down as a "fact about SchOS" is what got it wrong, and the block's comment now says to read each name as the question rather than as a property. #### What this cost, and what to take from it This sat in section 7 as "mksh's job control still doesn't reliably progress ... root cause not yet isolated" with three suspects, none of which were it. All three have since been eliminated on their own merits — `setpgid` and `getpgid` are real, `getrusage` is not consulted on the path, and the pid matching was never reached. Two corrections to the note written on 2026-09-21, both worth keeping because they were confident and wrong: - **"Not zsh — `mksh -c` hangs the same way" was false.** mksh installs as `/bin/sh`, not `mksh`, so that test was a `command not found` whose output scrolled past. mksh was never affected. Had the binary been named correctly the bug would have been localised to zsh's build in one step. - **The kernel was cleared and then still suspected.** `cmd/sigtest` had already proven the SIGCHLD/sigsuspend contract, and a musl-linked probe had already shown `kill(pid, 0)` returning −1 with `errno == ESRCH == 3` and zsh's own loop condition evaluating false. Both results were correct and both were read as "so it must be something subtler in the kernel". The thing they actually pointed at was the one place the probe and the shell differed: **how each was compiled.** The lesson is narrow and worth stating: when a probe that reproduces a program's exact syscall sequence behaves correctly and the program does not, stop looking under the syscalls and start looking at the build. --- This was left here deliberately rather than continuing to iterate blind: SIGCHLD delivery is left **enabled** (`kernel/proc/proc.c`'s `exit()`) since disabling it guarantees zero chance of mksh's wait loop ever exiting, whereas leaving it on at least gives the cases that don't hit the still-unknown remaining gap a chance to work. Next session should start by instrumenting `j_sigchld()`'s own control flow (or strace-equivalent logging in the kernel around `sys_setpgid`/`sys_getrusage`/`sys_waitpid` calls specifically while reproducing `cd; cd var; ls` or even simpler `ls -l /bin` followed by a second command) rather than re-deriving the signal-delivery mechanism, which is now confirmed solid. --- ## 7b. The loader jumped into the framebuffer and executed pixels The same bug as section 4, in the other loader, found five years later by a compiler upgrade. Worth its own entry because it is the *third* time a register the code depends on but never asks for has broken this project. **Symptom.** Boot died immediately after `schos: exiting boot services` with a KVM internal error and a register dump: RIP=00000000801a5dc5 RBX=0000000080000000 Code=13 0e 0c 00 13 0e 0c 00 ... dc d5 d1 00 dc d1 00 `0x80000000` is the framebuffer. `0x0C0E13` is `SCH_BG` and `0xD1D5DC` is a grey from the splash's metal gradient. So `RIP` was *inside video memory* and the bytes being executed were the boot logo. The machine was executing the picture of itself starting up. **Cause.** `boot/uefi/main.c`'s hand-off to the kernel: "mov %0, %%cr3\n\t" "mov %1, %%rbx\n\t" <- writes rbx "jmp *%2\n\t" : : "r"(PML4_ADDR), "r"(fb), "r"(entry) : "memory"); <- rbx NOT declared clobbered Nothing told the compiler rbx was written, so it was free to allocate an *input* there. clang 21 allocated `entry`: movq %rax, %cr3 movq %rcx, %rbx <- the framebuffer, into rbx jmpq *%rbx <- %2 was rbx; it now holds the framebuffer The fix is `"rbx"` in the clobber list, which forbids the allocation. Confirmed by reading the generated assembly before and after -- `jmpq *%rbx` became `jmpq *%r14`. **Why it appeared now.** It did not. It had been latent since the loader was written and worked only because every compiler that had built it happened to choose a different register. It surfaced the first time the binary was relinked with a different clang. That is the whole lesson, and it is identical to section 4's: **a register the code depends on, that nothing in the source actually asks for, is not a dependency -- it is a coincidence with a long half-life.** Section 4 was the BIOS loader passing the framebuffer in EAX by luck; section 8 was `%fs:0` used as scratch when the ABI reserves it; this is an asm block writing a register it did not declare. Reading the *generated* assembly is what settles all three, and it takes a minute: clang $(UEFI_CFLAGS) -S -o - boot/uefi/main.c | grep -B3 'jmpq.*\*' **A second failure sat underneath it and is worth its own note.** The loader had been stale for half a day -- the Makefile rule for `BOOTX64.EFI` did not list `logo.h`, `theme.h` or `font.h`, all of which `boot/common/ui.c` includes, so a new logo never rebuilt it. The splash kept drawing an old mask while `make` reported nothing to do. Fixing that dependency is what caused the relink, which is what exposed the register bug. A stale artifact hides the bug that the fresh one reveals. --- ## 8. Two ABI assumptions that only GCC was keeping true Found when the build moved from GCC-in-WSL to clang-on-Windows. Neither is a clang bug; both are places where the code relied on GCC doing something it never promised, and both had been latent for years. ### `%fs:0` was being used as scratch — it is the TCB self-pointer `syscall_entry` (`kernel/arch/x86_64/trapasm.S`) stashed `%rax` at `%fs:(0)` while it fetched `proc` to find the kernel stack. The comment called it a "per-CPU scratch slot". It is not: the x86-64 TLS ABI reserves the word *at* the thread pointer to hold the thread pointer, and a compiler may read it to materialise the TP before applying a variable's offset. GCC compiles `__thread` access to a direct `%fs:-8` segment reference and never reads that word, so the clobber was invisible. clang at `-O0` emits `movq %fs:0, %rax; addq $-8, %rax; (%rax)` — reading back the syscall number this code had just stored. The first process's first syscall is `exec` (7), so `proc` resolved to address `7 - 8 = -1` and the kernel page-faulted recursively until the stack was gone. Fixed by moving the scratch to `%fs:(8)`. Positive offsets are safe: the TLS block lives at negative offsets and `seginit` places the thread pointer 2048 bytes into a 4096-byte page. **How it was found** is the part worth copying next time. Three rounds of `cprintf` narrowed it to "the word is right at `mpmain`, wrong by the first trap" and no further — the instrumentation perturbed the timing enough to move the symptom. A hardware watchpoint through QEMU's gdb stub named the instruction on the first run: ``` qemu ... -S -gdb tcp::26000 gdb -batch -x tcp.gdb # hbreak mpmain; watch *(unsigned long*)kernel_fs_base ``` `seginit` also now writes the self-pointer, which it never did. That is ABI -correct regardless and costs one store per CPU. ### `_start` read incoming registers with `register ... asm("rdi")` `lib/glue.c` captured `argc`/`argv` from `%rdi`/`%rsi` using GCC local register variables. Both compilers only guarantee that binding for inline-asm operands; using it to pick up an incoming argument is undefined. clang ignored it and read two uninitialised stack slots: ```asm movl -0x4(%rbp), %eax # should have been %edi movq -0x10(%rbp), %rax # should have been %rsi ``` Every program still ran. `init`, `shmtest` and `socktest` take no arguments and passed their entire suites; the only symptom in the whole system was `inputtest -n 6` ignoring its limit. Moved to `lib/start.S`, where reading incoming registers is expressible and correct — `exec` passes argc/argv in the SysV argument registers already, so `_start` calls `main` without touching them. --- ## 9. The boot looked like a hobby OS, and two things were why Not a bug — a design problem, fixed by looking at a screenshot instead of the code. **The display was the kernel log.** Boot printed `pci: 6 devices` and `vga: display controller 1234:1111, aperture fd000000 (16 MB)` to the screen the user is looking at. That single fact is most of what separates a hobby project from a product; every shipping system puts that behind a flag. `cons_serial_only` already existed for panic, so the fix was to start it *set*. Everything still goes out the UART — which is where `tools/windows/bootcheck.ps1` reads it from, so all five suites kept passing untouched — and `console_takeover()` clears it on the first `exec()`, the moment userspace genuinely starts. `vga_init` no longer clears the screen, so the splash survives the whole of boot rather than being wiped a few hundred milliseconds in. **There were three unrelated designs.** The splash was deep purple with gold text, deliberately matching the *panic* screen. The console was cream with near-black text. So a healthy boot was dressed as a crash, then abruptly became an amber terminal. `include/theme.h` is now the one palette, and the rule is: **normal states are calm and dark; only failure is loud.** PSOGM keeps the purple entirely to itself, which is what makes it mean something. The splash copy went from "SchOS Boot Manager / version 1.0 / Memory: 1023 MB / Press F8 for boot menu..." to the product name, a rule in the accent colour, and the one hint that is actionable — moved to the bottom, where a hint belongs. No shipping OS names its bootloader on the splash. ### Three latent bugs it flushed out Retheming meant `stage2main.c` included `types.h` properly for the first time, and the 32-bit build stopped compiling: - `_Static_assert(sizeof(uint64) == 8)` fired. `uint64` was `unsigned long`, which is **4 bytes** in the `-m32` loader — the file even carried a local `e820_u64` workaround with a comment saying so. Now conditioned on `__SIZEOF_LONG__` rather than on the platform, which covers LP64, LLP64 *and* ILP32 correctly. - `x86.h`'s `stosq` and `xchg` use `rep stosq` and `xchgq`, which do not exist in 32-bit mode. They only ever compiled because `uint64` was silently narrowed. Now behind `#ifdef __x86_64__`. - The progress bar computed `(uint64)done * 100 / total`, a 64-bit divide that needs libgcc's `__udivdi3` — which a freestanding loader does not link. Now scaled to KiB and done in 32 bits. All three had been sitting in a bootloader that worked fine, waiting for someone to make `uint64` mean what it says. --- ## 10. A full-screen program typed the whole session into the shell Keys reached `/dev/input/event0` **and** the tty line discipline at the same time. That is correct for two consumers of one keyboard and completely wrong for a full-screen program: while it had the display, the shell behind it was quietly accumulating every keystroke, and the first Enter afterwards ran whatever had piled up. Arrow keys arrived as CP437 glyphs on the console for good measure. The fix is the standard one and it is a *primitive*, not a patch: an exclusive grab, `EVIOCGRAB`, with Linux's ioctl number. While a device is grabbed its events still flow to `/dev/input/eventN` but both keyboard drivers withhold the decoded character from the console. Three details that matter more than the feature: - **The grab is released by the devsw `close` hook**, not by the program. A program that grabs the keyboard and then crashes would otherwise leave the console permanently deaf, with no way to type the command that fixes it. - **Modifier state is still tracked while grabbed** (`kbdgetc` keeps updating shift/caps and only skips the character). Otherwise releasing the grab would hand the line discipline a stuck Shift. - **The USB path updates `prev[]` before returning**, or the next report would look like every held key had just been pressed again. It is also exactly what Schcom will need, which is why it lives in the input layer rather than in any one client. ### The near-miss worth remembering The first attempt appeared to work: the harness passed, every assertion green. It had not worked at all — the ioctl call was never inserted into the client source, because a scripted string replacement silently failed to match. The `grep` that should have caught it printed three hits (a comment, a `#define`, and the *release* call) and was read as confirmation without counting what they were. What actually caught it was reading the transcript rather than the verdict: four bare `#` prompts after quitting, which is precisely the symptom the change was supposed to remove. `input_grab()` now announces itself, so "did the grab happen" is never again a thing anyone has to infer. --- ## 11. `ps` and `top`, and why they never worked /proc was a set of character devices: init ran `mknod("/proc/uptime", 6, 2)` and friends, so every file was a real inode on the root filesystem with a major/minor behind it. That is fine for a fixed list and cannot work at all for the thing /proc is mostly for -- one directory per process. You cannot mknod a directory for a process that does not exist yet and will be gone in a minute. `ps` calls `dirtree_flagread("/proc")`, got a listing with no numbers in it, and reported nothing. It was never a `ps` problem. So /proc is now a real filesystem behind the same `struct fs_ops` vtable SchFS uses (`kernel/fs/procfs.c`, `PROCFS_DEV`), mounted by init with `mount("proc", "/proc", "proc", ...)`. Nothing is stored: directory listings are generated from the process table when asked, file contents when read. Three things worth keeping in mind: - **The split is deliberate.** `kernel/fs/procfs.c` owns the *namespace* (what exists); `kernel/dev/procfs.c` keeps the *generators* (what the global files say). They change for different reasons, so `proc_gen_global()` is the only thing that crosses between them. - **procfs never touches ptable.** `proc_snapshot()` copies what is needed out from under the lock, so a process that exits between the readdir that listed it and the open that reads it produces a stale line rather than a use-after-free. - **Sizes are reported as zero**, exactly as Linux does. A synthetic file has no length until it is generated, and generating every one on every `stat()` to fill in a number almost nothing reads would be absurd. ### What the suite caught Marking every procfs inode read-only broke `echo s > /proc/sysrq`, and three USB checks failed on it within a minute of the change -- they drive the disk through sysrq. /proc is read-only *except* sysrq, which is the one deliberate write target, and the code now says so in one place. ### Still not right `ps -o comm` prints `[init]` rather than `init`. That field comes from `/proc//exe`, which Linux makes a **symlink** and toybox `readlink()`s; ours is a regular file, so the readlink fails and toybox brackets the name as it would for a kernel thread. Default `ps` and `top` output are unaffected. Fixing it needs symlink support in procfs, which belongs with the rest of the VFS work rather than as a one-off. --- ## 12. A test program crashed at random, and had been typing its own launch command Symptom: a full-screen framebuffer program, launched with arguments, worked four times and then died with pid 9 : trap 14 err 4 on cpu 0 rip 0x2269be addr 0x18--kill proc `err 4` is a user-mode read of a non-present page and `addr 0x18` is a null pointer at offset 24. `llvm-nm` put `rip` inside the routine that opens the program's overlay view, which dereferences a pointer the program only fills in once a level is loaded -- at offset 0x18 on x86-64, because the three 64-bit words of a base struct sit in front of it. So the program had been asked to open that view before there was anything to show. **The technique is the transferable part:** the fault address alone said "null pointer plus 24", which is nearly useless; `llvm-nm` on the faulting `rip` turned it into a function name, and the offset then identified the exact field. Reach for that before adding printfs to a program you did not write. The cause was ours, and it was visible in the log the whole time: every run printed a burst of garbage bytes *before* the grab was taken, and the amount differed each run. **`/dev/input/eventN` had no notion of being opened.** Events were queued from boot onwards whether or not anything was listening, into one 256-entry ring with one read cursor. When the program opened the device it inherited the entire backlog -- including the keystrokes of the command line that had launched it, which the shell had already consumed -- and replayed them as input. Whether that crashed depended on which keys happened to be sitting there, which is exactly the "sometimes" in the report. Fixed by giving `struct devsw` an `open` hook to match `close`, and having `/dev/input`: - drop the backlog on open (`r = w`) -- a reader starts from *now*; - only queue events while `opens > 0`, which is what Linux does: an evdev device with no clients does not accumulate. Linux gets both for free by giving every open its own client queue. One shared ring plus "start from now" is the same answer for a single reader, which is the only case that exists today; per-client queues become necessary the moment Schcom and a client both want the keyboard. ### Also fixed `cd` into a file reported "Operation not permitted". It was right to fail, but `sys_chdir` returned a bare `-1`, so the shell printed whatever errno happened to be lying around. It returns `-ENOTDIR` now, and `-ENOENT`/`-EFAULT` on the other two failure paths -- "Not a directory" sends you to the right place, "Operation not permitted" sends you looking at permissions. ## 13. The clock ran seven times slow, and the graphics card was why The report was about time. fastfetch said the machine had been up for 3 seconds after 15 real ones. `time ping 127.0.0.1` claimed `0m03.12s real` for a command that visibly took 20. ping's one-second interval was closer to six. Three programs, three wrong numbers, all wrong by roughly the same factor and all in the same direction -- which is the shape of one shared cause, not three bugs. The shared cause was obviously the tick. Everything above is derived from `ticks`, and `ticks` comes from the timer interrupt. So the hunt started at the bottom of that chain, and the first thing it found was real: ```c // If xv6 cared more about precise timekeeping, // TICR would be calibrated using an external time source. lapicw(TICR, 10000000); ``` That is xv6's own comment, still in the tree, admitting that the LAPIC timer's countdown value was never measured against anything. 10000000 yields the intended 100 Hz only if the APIC bus happens to run at exactly 1 GHz. It was a genuine gap and it is now fixed -- `calibrate_timers()` measures the LAPIC and the TSC against the 8254 PIT, which is fixed at 1193182 Hz by the hardware and is exactly the external time source the comment asked for. But it was not the bug. The measurement came back at 999.9 MHz. The bus really was 1 GHz, `TICR = 10000000` really was 100 Hz, and the clock was still wrong. ### What the instruments said Every clock agreed with every other clock, which is the confusing part: | measured against | TSC | LAPIC timer | |---|---|---| | the PIT | 3792 MHz | 999 MHz | | one CMOS RTC second | 3791 MHz | 999 MHz | Three independent clocks, mutually consistent, and 3792 MHz is exactly the 5700G's base clock. So the timer was programmed for 10 ms and the registers read back correct at runtime: `TICR=10000314`, `TDCR=11` (divide by one), `LVT=0x20020` (periodic, vector 32). Interrupts nonetheless arrived every 68 ms. Two hypotheses died next. Interrupts were not being masked -- the CPU had `IF=1` for 99.98% of the measured window. And it was not the idle `hlt()`: deleting it changed nothing, and neither did dropping to `-smp 1`. What broke it open was measuring the gaps between consecutive timer interrupts instead of averaging them: ``` timer gaps us: 2851 137614 2438 136293 3805 136889 3019 138420 6950 137248 ``` Bimodal, strictly alternating. Not a slow clock -- a clock being interrupted. In one-shot mode the short gaps sharpened to 10031, 10078, 10064, 10011 µs, which is a perfect 10 ms: the LAPIC was doing its job exactly as programmed, and then something was eating the next 130 ms. Timing every trap by vector found it in one boot: ``` trapno 43: 5828 ms total, 510000 entries ``` Vector 43 is `T_IRQ0 + 11`. IRQ 11 was taking **72,000 interrupts a second** and consuming 83% of CPU0. ### The actual bug IRQ 11 was the e1000. Its handler reads ICR, and reading ICR clears it, so the driver looked correct -- and it was. The cause register read back **zero** on essentially every one of those 500,000 interrupts: the e1000 was being handed somebody else's interrupt and correctly answering "not mine". The somebody else was `virtio-gpu-pci`, sharing IRQ 11, **with no interrupt handler registered at all**. PCI INTx is level-triggered. The device asserted the line, the kernel signalled EOI, nothing had cleared the device's interrupt status, so the line was still asserted and it fired again immediately. Forever. The fix is one read. Reading virtio's ISR byte clears it and drops the line: ```c static void vgpu_intr(void *arg) { struct vgpu *g = arg; if (g && g->isr) (void)*g->isr; } ``` This driver polls for command completion and wants no notifications. "Wants none" and "gets none" are different things, and only the second one is free. Booting without `-device virtio-gpu-pci` had already proved it: 100 ticks in 999 ms, and the e1000 taking 2 interrupts instead of half a million. With the handler registered, uptime advances 5.01 seconds per 5 real seconds. ### What to take from it **A shared level-triggered line is only as good as its worst driver.** Every other device on IRQ 11 was correct. One driver that never acknowledged its device was enough to starve the whole machine. **The symptom appears where the missing CPU time hurts first, which is nowhere near the fault.** This surfaced as a timekeeping bug, in userspace, in three unrelated programs. It sent the investigation into the timer code, where there was a real defect sitting in plain sight with a comment explaining itself -- and fixing that defect changed nothing, because it was not this one. A plausible bug found early is the most expensive thing in a debugging session. **Averages hide alternation.** "68 ms per interrupt" suggested a wrong frequency and implied the timer. "10 ms, then 130 ms, repeating" says the timer is fine and something else is stealing the machine. Same data, one of them divided. `irq_dispatch` now measures the rate on every line against the TSC -- not against `ticks`, which is the first casualty of a storm -- and says so once: ``` irq 11: 75894 interrupts/second -- a device on this line is not acknowledging it. irq 11: handler 'virtio-gpu' is attached irq 11: handler 'e1000' is attached ``` That is the message this cost an afternoon for. The threshold is 20000/s, which no real device on this machine sustains. ### Also fixed `microdelay()` was an empty function body -- xv6's stub, with a comment saying real hardware would want to tune it. Every `microdelay(1000)` in the xHCI reset path and the USB hub power-on sequence returned instantly. Those waits are in the specifications because the hardware needs the time; a port that "works anyway" is racing and winning. It now spins on the calibrated TSC. The tick rate itself stopped being a magic number. `HZ` lives in `param.h`, and the dozen places that wrote `ticks / 100` or `sec * 100` use it. `USER_HZ` is separate and deliberately does not follow it: musl answers `sysconf(_SC_CLK_TCK)` with 100 no matter what, so `times()` and `/proc/stat` are scaled at the syscall boundary the way Linux does it. Before, `sys_times` returned raw ticks -- correct only while `HZ` happened to be 100, and silently wrong the day anyone changed it. ## 14. A shell that would not start, and four bugs underneath it Porting zsh was supposed to be a port. It found four defects in the kernel and one in the filesystem, none of which had anything to do with zsh, and every one of which had been there the whole time. ### The environment did not exist `kernel/fs/exec.c` built the initial stack for every new program and wrote this: ```c stack_slots[slot++] = 0; // end of envp (no environment) ``` Not "no environment yet" — no environment, ever, for any process. musl calls `execve(path, argv, envp)`, which is Linux syscall 59, which SchOS answers; the third argument was simply never read. So `$TERM` was unset, `$PATH` was whatever each program invented for itself, and a variable exported in a shell did not survive the exec that was supposed to carry it. That is fatal for zsh specifically. A line editor with no `TERM` cannot ask terminfo how to move a cursor, so ZLE is exactly as useless as it would be with no terminal library linked at all. `exec()` takes an `envp` now, `sys_exec` reads the third argument, and schinit manifests grew an `Environment` key: ``` Environment = TERM=linux Environment = PATH=/bin:/usr/bin ``` The userland `exec(path, argv)` spelling stays, as a wrapper in `lib/glue.c` that passes a null environment explicitly. The raw stub is `execve` now — otherwise a two-argument caller would leave `%rdx` holding whatever happened to be in it, and the kernel would read that as a pointer. ### Which immediately overflowed the kernel stack The three pointer arrays exec needs came to about 1.8 KB, and `KSTACKSIZE` is 4096, under a call chain that already runs exec → loaduvm → readi → SchFS → btree. The first exec after the change took pid 1 down. The canary caught it cleanly, named the process, and printed a stop code, which is the whole reason it exists. The arrays live in one `kalloc`'d page now. ### `wait` said "operation not permitted" ``` compdef:80: wait failed: operation not permitted ``` `waitpid` returned a bare `-1` for two different failures: no children to wait for, and interrupted by a signal. musl reads any return in [-4095, -1] as `-errno`, so `-1` arrived in userspace as `EPERM`. zsh saw it, printed that message, and abandoned the rest of `/etc/zsh/zshrc` — so the shell came up with its default prompt and no completion, which reads as a broken config file rather than as a `wait()` that could not say what happened. It returns `-ECHILD` and `-EINTR` now. This is the third time a bare `-1` has cost real time (see §12, `sys_chdir`). **EPERM on SchOS should be read as "a syscall returned -1 and meant something else."** There is no permission model here for it to legitimately come from. ### mmap ran out of slots and blamed memory ``` compdef:80: fatal error: out of memory ``` On a machine with 1000 MB free, of which zsh had used 1.5. `NVMA` was 64 — the per-process mmap table — and musl's allocator takes a fresh mapping per size class and does not release them eagerly. compinit reads twelve hundred completion functions, ran out of slots, and got `-1` from `sys_mmap`, which is `EPERM` again, which `malloc` turns into NULL, which zsh reports as out of memory. Three translations from the truth. `NVMA` is 256 and the failure returns `-ENOMEM`. ### And underneath all of it: a small write costs 58 ms With everything above fixed, `compinit` still appeared to hang. It was not hanging. Measured on this machine, 200 writes: | | time | per write | |---|---|---| | to one open file on SchFS | 11.75 s | **58 ms** | | open + write + close each time | 21.39 s | **107 ms** | | to `/dev/null` | 0.14 s | 0.7 ms | `compdump` writes its digest with a few thousand separate `print` calls. At 58 ms each that is minutes, and it looks exactly like a hang. `compinit -D`, which skips the dump, finishes in 3.6 seconds — the scan was never the problem. The cause is not a bug; it is the design, stated plainly at the top of `kernel/fs/schfs/kmount.c`: > Every entry point below that changes anything is one transaction group: it > takes the volume lock, does its work, and commits before returning. That makes > an operation durable the moment it returns. A commit is two cache flushes and an uberblock write, and a flush is a barrier. `schfs_ktx_open` already lets one `write(2)` of 1 MB be a single group — 267 block writes instead of 1410 — but nothing brackets *across* syscalls, so two thousand small writes are two thousand barriers. `cp` does not notice, because it uses large buffers. Anything that writes a line at a time notices enormously. **This is the open question, and it is a design decision rather than a fix.** Deferring commits would mean giving up durable-on-return, which is a property SchFS chose deliberately. The options are a writeback window like every other filesystem has, committing on `fsync`/`close`/timer, or leaving it and accepting that line-at-a-time writers are slow. Until it is decided, `/etc/zsh/zshrc` leaves `compinit` switched off and says why. ### What zsh actually got It runs, as `/bin/zsh`, and it is the console shell. ZLE works: arrow keys recall history through terminfo, `$COLUMNS` reads 160 because `TIOCGWINSZ` now answers with the real console geometry (§13), prompt expansion and colour work, and startup is 0.14 s. mksh keeps `/bin/sh`, so schinit's no-jobs fallback is a different program from the one that might have failed. Completion is the piece that waits on the filesystem.