SchOS/docs/history/kernel.md

# Kernel history

Bugs that were expensive to find and whose causes are not visible from the code
that fixed them, plus a few subsystem write-ups that have nowhere better to live.

**Every entry here is a thing that looked like a different problem than it was.**
What each subsystem *is* today is [../architecture/](../architecture/); this is
only how it got that way.

| | |
|---|---|
| [1](#1-fixed-boot-didnt-reliably-reach-a-working-shell) | Boot did not reliably reach a working shell — ten bugs behind one symptom |
| [2](#2-naming-standardization) · [3](#3-coreutils) | Naming, and where the userland comes from. Not bugs. |
| [4](#4-two-stage-bootloader--psogm-splash) | The two-stage BIOS bootloader. **Since removed** — kept for the bugs. |
| [5](#5-at-syscalls-ignoredrejected-real-dirfds) | `*at()` ignored real dirfds, and `ls -l` hung |
| [6](#6-boot-speed) | Boot speed, in the loader |
| [7](#7-at-syscalls-real-signal-delivery-and-mksh-job-control) | Real signal delivery, and mksh's job control |
| [7b](#7b-the-loader-jumped-into-the-framebuffer-and-executed-pixels) | The loader jumped into the framebuffer and executed pixels |
| [8](#8-two-abi-assumptions-that-only-gcc-was-keeping-true) | Two ABI assumptions only GCC was keeping true |
| [9](#9-the-boot-looked-like-a-hobby-os-and-two-things-were-why) | The boot looked like a hobby OS |
| [10](#10-a-full-screen-program-typed-the-whole-session-into-the-shell) | A full-screen program typed the session into the shell |
| [11](#11-ps-and-top-and-why-they-never-worked) | `ps` and `top`, and why they never worked |
| [12](#12-a-test-program-crashed-at-random-and-had-been-typing-its-own-launch-command) | A program crashed at random, typing its own launch command |

The SMP bring-up bugs are not here — they are summarised in
[../architecture/boot.md](../architecture/boot.md#4-smp), because the lesson
(`__thread` is wrong for per-CPU data) is load-bearing on the current code
rather than merely historical.

> **Where the old design plan went.** Sections 1-6 of this file used to hold a
> "Segregated Kernel" design: a Ring 0 Alpha / Ring 0 Beta split with driver
> modules isolated by CR3 switching, from an `IDEA.txt` that no longer exists.
> None of it was ever implemented, and it is superseded by
> [../plan/schapi.md](../plan/schapi.md) — a macOS-shaped system answers driver isolation with
> userspace drivers and a personality boundary, not a bespoke in-kernel ring
> split. It also referenced a Zig toolchain that is now clang/LLVM. Removed
> rather than left to be mistaken for a plan.

---

## 1. Fixed: boot didn't reliably reach a working shell

Investigated 2026-07-05, before phase 1 started (predates this design, not
caused by it). Originally manifested as `ksh` (mksh) crashing on every
launch, apparently inside musl's mallocng allocator (`get_meta`,
`src/malloc/mallocng/meta.h`, deliberately executing `hlt` on detecting
"corrupted" heap metadata — a `trap 13` GPF at a fixed rip). It wasn't a
musl bug. Six real kernel bugs were found and fixed across three
investigation passes, and mksh boots and runs cleanly with all of them in
place — the calloc assertion was a symptom of real memory/pointer-handling
bugs downstream in the kernel, not anything musl or mallocng did wrong.

Fixed:
- `copyuvm()` (`kernel/arch/x86_64/vm.c`, used by `fork()`) never copied
  `mmap()`-backed `proc->vma[]` regions into the child — only `[0, sz)`.
  Fixed via new `copy_vma_range()`, called from `fork()` (`kernel/proc/proc.c`).
- `kmem.freelist` (`kernel/core/kalloc.c`) could get corrupted by a wild
  write elsewhere and would silently hand back a bad pointer, panicking far
  from the actual cause. Added a canary check (`check_run()`) at the top of
  both `kalloc()`/`kfree()` so corruption panics immediately at the point
  it's touched, with a clear message.
- No TLB invalidation existed anywhere except the full `lcr3` reload on
  context switch — `mappages()`/`vm_unmap_range()` never called `invlpg`.
  Fixed by adding `invlpg()` (`include/arch/x86.h`) to both functions.
- **`MSR_STAR` was misconfigured** (`syscallinit()`, `kernel/arch/x86_64/vm.c`):
  `SYSRETQ` computes the returned `cs`/`ss` as `STAR[63:48]+16`/`+8`, which
  requires the *compat* 32-bit code selector (`SEG_UCODE32`) in that field —
  the kernel was putting the real 64-bit `USER_CS` there instead. Every
  `sysretq` silently loaded an invalid `cs` (`0x3b`, pointing at the TSS
  descriptor)/`ss` pair; QEMU's TCG doesn't validate segment descriptors on
  `SYSRET`, so this went unnoticed until a hardware interrupt forced a
  strictly-validated `IRETQ` with that bogus `cs` still loaded, which
  correctly `#GP`'d.
- `sys_mmap`'s non-`MAP_FIXED` path (`kernel/sys/sys_mem.c`) assigned a fixed
  8MB virtual-address window per `vma[]` slot regardless of the requested
  length. A single allocation over 8MB (e.g. `realloc()` growth past that
  size) mapped straight into a neighboring live mapping and hit
  `mappages()`'s `panic("remap")`. Replaced with a real bump allocator
  (`proc->mmap_next`, `include/core/proc.h`), so every mapping gets its own
  disjoint, exactly-sized range regardless of size.
- **`uva2ka()`** (`kernel/arch/x86_64/vm.c`) returned only the page-aligned
  base address of a user pointer's page, silently dropping the in-page byte
  offset. Direct callers (`sys_ioctl`'s `TIOCGPGRP`/`TCGETS`, `sys_getcwd`,
  signal masks, `times()`, `sysinfo()`, ...) read/wrote near the wrong
  address whenever the pointer wasn't page-aligned — i.e. almost always.
  Fixed at the source so every caller benefits.
- **`fetchint`/`fetchaddr`/`fetchstr`/`argptr`** (`kernel/sys/syscall.c`)
  rejected any address `>= proc->sz`, wrongly assuming every valid user
  pointer lives in the heap. Stack addresses and `mmap()`'d buffers (used
  constantly by real programs, e.g. path-resolution buffers) live above
  `proc->sz` and were being rejected outright. Fixed to rely solely on
  `uva2ka()`'s page-table validation.

Bringing in toybox (see "Coreutils" below) surfaced four more, all fixed:
- Linux syscall 58 (`vfork`) wasn't in the syscall-ABI translation table at
  all (`kernel/sys/linux_abi.c`) — every external command spawn from a
  program that prefers `vfork()` over `fork()` failed outright. Mapped to
  the existing `fork()` (a fully correct, if slightly slower, substitute).
- `sys_stat`/`fstat`/`fstatat` (`kernel/sys/sys_fs.c`) filled in SchOS's
  own native `struct stat` layout regardless of the caller — musl/toybox
  parse a fixed Linux ABI byte layout instead, so `st_mode` was garbage and
  tools like `ls` couldn't tell files from directories, or executables from
  non-executables. Added `struct linux_stat` + `stati_linux()`, gated on
  Linux-ABI mode (`proc->syscall_map`).
- `sys_open`/`sys_openat` required `omode == O_RDONLY` *exactly* to open a
  directory, but `opendir()`-style calls legitimately OR in `O_DIRECTORY`/
  `O_CLOEXEC` alongside it. Fixed to check just the access-mode bits
  (`O_ACCMODE`, added to `include/uapi/fcntl.h`).
- `sys_getdents64` read one `struct dirent` (16 bytes) at a time — a full
  `bmap()`+`bread()` per slot. Directories here are hash-indexed (see
  `tree_search()`/`tree_expand()` below), so a directory with a few dozen
  real entries can have a size in the hundreds of KB, nearly all empty
  slots — this made listing such a directory look hung (it wasn't a
  deadlock, just thousands of redundant single-slot disk reads). Rewrote it
  to read a full block at a time and resolve each entry's type via
  `iget(inum)` (exported from `fs.c`, it was `static`) instead of
  re-searching the whole directory by name with `dirlookup()` for every
  entry. That rewrite's first version introduced a **new** self-deadlock:
  the "." entry's inode number equals the directory's own, and `iget()`
  correctly hands back the *same, already-locked* `struct inode*` — calling
  `ilock()` on it again sleeps forever on a lock the process already holds.
  Fixed by special-casing `xde->inum == ip->inum` (we already know it's a
  directory, no need to re-lock at all).

A synthetic stress test (interleaved malloc/free, realloc growth crossing
`MMAP_THRESHOLD`, calloc, large allocations) runs to completion with zero
crashes, and mksh now boots straight to an interactive prompt, runs external
programs, pipes, and `ls`/`mkdir`/`grep` without issue.

## 2. Naming standardization

SchOS standardizes on exactly one name per role, regardless of the
underlying implementation: the shell is always installed and exec'd as
`/bin/sh` (currently mksh, built from `ports/mksh/`), and PID 1 is always
`/bin/init`. `cmd/init/main.c`'s exec loop only ever tries `/bin/sh` — no
ksh/mksh/dash/zsh/other-name fallbacks. dash was tried first
(BSD-licensed, minimal) while the boot crash above was still open, then
replaced with mksh once the underlying kernel bugs were fixed and dash's
lack of interactive features (history, completion, line editing) made it
feel unusably bare for day-to-day use; the dash port's source tree was left in
place rather than deleted, in case it's useful again (e.g. as a `/bin/sh`
POSIX-strict target distinct from an interactive login shell).

## 3. Coreutils

`ports/toybox/` (0BSD-licensed, https://landley.net/toybox/) provides `ls`,
`cat`, `cp`, `mv`, `rm`, `mkdir`, `grep`, `sed`, `find`, `chmod`, `date`,
`sort`, and more — see the `APPLETS` list in `ports/toybox/Makefile`. It builds
as a single multicall binary; `tools/mkfs.c` doesn't understand symlinks or
hardlink dedup, so `ports/toybox/Makefile`'s `install` target copies the built
binary once per applet name into `sysroot/bin/` rather than symlinking
(costs disk space per applet — currently ~44MB total — instead of one
shared inode, but needed no mkfs changes).

## 4. Two-stage bootloader + PSOGM splash

> **This describes the BIOS loader, which no longer exists** — see
> [`boot/README.md`](../../boot/README.md) for why it was removed and what
> replaced it. The section is kept because four of the bugs in it are about
> things that are still true: extended scancodes, register hand-off across a
> mode switch, and layout constants that several artifacts must agree on.

`boot/` used to be a single BIOS boot sector doing everything (VBE mode
set, A20, GDT, protected-mode switch, multiboot kernel scan/load) — silent,
and packed to within 4 bytes of the hard 510-byte MBR limit, so there was
no room to add so much as a print statement.

Split into two stages, the standard way real bootloaders solve this:

- **Stage 1** (`boot/bios/bootasm.S`, still the literal MBR sector, still
  ≤510 bytes) does exactly one thing: raw IDE PIO (LBA28) reads
  `STAGE2_SECTORS` (16) sectors starting at disk sector 1 into memory at
  `0x7E00`, then jumps there. Nothing else fits, nothing else is needed.
  (Caught a real bug writing this: `outb $0x1F7,%al`-style immediates
  silently truncate to 8 bits — ports 0x1F2-0x1F7 are all > 0xFF and need
  the `%dx` register form, or you're hitting the wrong I/O port entirely.)
- **Stage 2** (`boot/bios/stage2.S` + `boot/bios/stage2main.c`, loaded at `0x7E00`,
  occupies disk sectors 1-16) does everything stage 1 used to skip straight
  to, plus a boot banner (BIOS teletype, real mode, before the VBE mode
  switch — note this only reaches the video device, not serial, so it
  won't show up in a `-serial file:...`-captured log), a splash screen, a
  real-time loading bar, and an F8 boot menu (all described below). Not
  size-constrained the same way as stage 1 — `tools/padstage2.pl` zero-pads
  it out to a fixed `STAGE2_SECTORS*512` (8192) bytes so the kernel's
  on-disk starting sector stays predictable regardless of stage2's actual
  compiled size, and errors out (like `tools/sign.pl` does for stage 1) if
  it ever grows past that budget instead of silently corrupting the layout.

The kernel now starts at disk sector 17 instead of 1 (`schos.img`'s `dd
... seek=17` in the root `Makefile`, and the equivalent `+(1+STAGE2_SECTORS)`
in `stage2main.c`'s `readseg()` — all three places that encode
`STAGE2_SECTORS` have to agree, or the kernel won't be found on disk at
all). `boot/bootmain.c` no longer exists; its logic was split between
`stage2.S` (real-mode/protected-mode setup) and `stage2main.c` (splash +
the unchanged multiboot-header-scan-and-load logic).

The VBE mode info block itself lives at `0x30000`, not the classic
single-stage bootloader's `0x9000` — stage2's own code now runs from
`0x7E00` up to nearly `0x9000` once the boot-menu/progress-bar logic is
included, and a block at `0x9000` gets partially overwritten by stage2's
own tail end (found this the hard way: the kernel's `LFB:` cprintf started
reporting a different wrong value on every boot, because it was reading
whatever stage2 code/data now happened to occupy that address).
`kernel/dev/vga.c`'s `vga_init()` has to agree on `0x30000` too, since it
independently re-reads mode fields from the same block.

Splash/progress bar/boot menu rendering is a small standalone pixel
renderer in `stage2main.c` (not the kernel's `vga.c` — paging isn't set up
yet, so the LFB physical address from the VBE block is used as a plain
pointer) reusing the kernel's own font data (`include/gfx/font.h`) directly.
It fills the framebuffer with PSOGM's purple (matching
`kernel/core/console.c`'s panic screen) and draws "SCHOS" centered, then:

- **Real-time loading bar**: `readsect()` (in `stage2main.c`, not the
  kernel) calls `progress_tick()` after every 512-byte disk read, updating
  an actual pixel-filled bar against the real total byte count — first a
  small one while scanning for the multiboot header, then a second,
  properly-sized one once the header's found and the true kernel size is
  known, labeled "Loading kernel...". Not simulated/timed — it reflects
  real bytes read off disk.
- **F8 boot menu**: `wait_for_f8()` busy-polls the keyboard controller
  (`inb(0x64)`/`inb(0x60)`, raw scancodes — real-mode BIOS keyboard calls
  aren't available once we're in protected mode) for about a second after
  the splash appears. F8 (scancode `0x42`) drops into `boot_menu()`: three
  options (boot normal, boot verbose, reboot to firmware setup) navigated
  with up/down arrows and Enter, plus a "detected drives" panel from
  `ata_present()` — a standard floating-bus presence check (select the
  drive, write a pattern to two registers, read it back) on the primary
  IDE channel's master/slave (the only channel this project ever actually
  puts drives on; QEMU's IDE emulation answers the same test
  "successfully" on the unpopulated secondary channel too, so that channel
  isn't shown — it'd just be confidently wrong). "Reboot to firmware
  setup" pulses the keyboard controller's reset line
  (`outb(0x64, 0xFE)`, same technique as `kernel/sys/sys_proc.c`'s
  `sys_reboot()`); a real bootloader can't jump into BIOS/UEFI setup
  directly, so this is the closest honest equivalent — it gives the user
  a fresh POST during which their firmware's own setup-menu key works.
- **Verbose boot**: picking "Boot SchOS (verbose)" writes a flag byte to
  `BOOT_VERBOSE_FLAG_ADDR` (`VBE_INFO_ADDR + 0x100`, i.e. just past the VBE
  block) before jumping to the kernel. `kernel/core/main.c`'s `kmain()`
  reads it into a global `boot_verbose` as its very first action and, if
  set, prints a line after every subsystem init call
  (`uartearlyinit`, `kinit1`, ..., `sinit`) — real per-step boot logging,
  not a canned message.

Two bugs worth remembering if this code gets touched again:

- **Keyboard extended-scancode handling**: `0xE0` (the prefix byte for
  arrow keys and friends) also has bit 7 set, the same bit used to detect
  key-*release* ("break") codes. A naive "skip anything with bit 7 set,
  then check for 0xE0" loop skips the 0xE0 byte itself before ever testing
  for it, so extended keys silently never register. The 0xE0 check has to
  come before the break-code check, not after.
- **The kernel entry call is not a real C function call.** Both the
  original single-stage `bootmain()` and the new `stage2main()` do
  `((void(*)(uint32))entry)(fbaddr)` — normal cdecl, which pushes `fbaddr`
  on the stack. But `kernel/arch/x86_64/entry.S`'s `mboot_entry` reads its
  framebuffer argument out of **EAX** (a multiboot-style raw-register
  handoff, then `mov %eax,%ebp` / `mov %ebp,%ebx` / `mov %rbx,%rdi` carries
  it through the 32→64-bit transition into `kmain()`'s first argument).
  The cdecl call never puts anything in EAX — it "worked" for years purely
  because simple bootmain.c code happened, as a side effect of its own
  codegen, to leave the right value sitting in EAX right up until the
  `call` instruction. Adding the splash/menu/progress-bar code changed
  that codegen and broke it silently: the kernel booted fine and reported
  a plausible-looking but wrong `LFB:` address (garbage left over in EAX
  from unrelated prior code, different on every boot). Fixed by pinning
  the argument down explicitly with `asm volatile("jmp *%0" : : "r"(entry),
  "a"(fbaddr))` instead of hoping the compiler's register allocation
  cooperates. If this code moves around again, re-verify with a screendump
  taken *immediately* before the jump compared against the kernel's own
  `LFB:` log line — they silently diverging is exactly this bug.

## 5. `*at()` syscalls ignored/rejected real dirfds

`ls -l`/`ls -lR` (and, by the same code path, `find`/`du`/anything using
toybox's `lib/dirtree.c`) would hang on `ls -l` and either hang or silently
misbehave on recursive listings. Root cause: `kernel/sys/sys_fs.c`'s
`*at()` syscalls (`openat`, `fstatat`, `unlinkat`, `renameat`, `linkat`,
`symlinkat`, `mkdirat`, `mknodat`, `readlinkat`, `faccessat`) only handled
two cases for their `dirfd` argument — `AT_FDCWD` (resolve relative to
`proc->cwd`) or an absolute path (resolve from root, ignoring `dirfd`
entirely). A **real** dirfd with a **relative** path — e.g.
`fstatat(open("/bin", O_RDONLY), "ls", &st, ...)`, the standard, efficient
way any directory-tree walker avoids rebuilding full path strings for every
entry — was either rejected outright (`fstatat`, `unlinkat`, ...) or
silently ignored in favor of `proc->cwd` (`openat`, which never checked at
all — worse than rejecting, since it can silently resolve the *wrong*
file if cwd happens to contain a same-named entry).

Fixed by adding a real dirfd-relative path resolver: `kernel/fs/fs.c`'s
`namex()` was refactored into `namex_from(struct inode *start, ...)`,
parameterizing what used to be a hardcoded `idup(proc->cwd)`, with two new
exported entry points, `nameifrom()`/`nameiparentfrom()`. `sys_fs.c` gained
`at_dir(dirfd)` (resolves `AT_FDCWD` or a real fd to a starting inode) and
`at_namei()`/`at_nameiparent()` built on top of it, plus a `createat()`
(parameterizing `create()`'s hardcoded `nameiparent()` call the same way).
Every `*at()` syscall now uses these instead of its own inline
absolute-path/`AT_FDCWD`-only check.

**Known follow-on issue, not yet root-caused**: heavy `ls`/`ls -lR` use
across many commands in one long-running shell session can eventually make
a *later*, unrelated directory hang (reproduced with `ls -lR /var` after
~10 prior commands in the same session; the identical command succeeds
immediately on a fresh boot). Not an inode-table or file-table exhaustion
— both `iget()` and `filealloc()` fail cleanly (panic / return 0
respectively) when full, and neither fired. Smells like a lock or
reference-count leak somewhere in the *at() paths above, or in toybox's
own fd handling, that only manifests after enough accumulated opens/closes.
Next step: instrument `itable`/`ftable` occupancy counts and watch them
across a long `ls`-heavy session to see what's actually climbing.

## 6. Boot speed

Two cheap, real wins in `boot/bios/stage2main.c`, no functional changes:

- `progress_tick()` (called once per 512-byte disk sector) used to trigger
  a full progress-bar redraw — border plus fill, a few thousand pixel
  writes — on *every* sector. For the ~330KB kernel image that's ~650 full
  redraws (the multiboot-header-scan phase adds another ~32) for a bar
  that only has 101 visually distinct states (0-100%). Split into
  `progress_border()` (drawn once, in `progress_start()`, since the border
  never changes) and `progress_fill()`, and `progress_tick()` now skips the
  redraw entirely unless the percentage actually changed. This was the
  single biggest chunk of the bootloader's own wall-clock time.
- `fill_rect()` (the full-screen splash fill, 480,000 pixels at 800x600,
  run at least twice per boot) used to call `draw_pixel()` per pixel, which
  recomputes `g_fb + y*stride + x*4` from scratch every time. Now computes
  each row's base pointer once and indexes into it.
- The F8 boot-menu wait was shortened from ~1s (`5000000` busy-loop
  iterations) to ~0.6s (`3000000`) — flat overhead on every boot whether
  or not the menu is wanted, so it's the most direct lever available
  without giving up the feature entirely.

## 7. `*at()` syscalls, real signal delivery, and mksh job control

> **Resolved since this was written.** Process groups and sessions are real
> (`pgid`/`sid` in `struct proc`, `setpgid`/`setsid`/`getpgid`/`getsid`, and
> signal delivery to a whole group), which was the first of the three suspects
> listed at the end. mksh's job control works. The investigation is kept in full
> because the signal-delivery mechanism described below is still exactly how the
> kernel does it, and because the reasoning about *why SIGCHLD was left enabled
> while broken* is the transferable part.

`ls -l`/`ls -la` hanging outright turned out to be the same root cause
described in §5 above (dirfd-relative path resolution) — fixed
and verified extensively (plain `ls`, `ls -l`, `ls -la`, `ls -lR`, piped
output, all confirmed working repeatedly).

A *second*, deeper issue was found investigating a follow-up report
(`cd` with no `HOME` set, then `cd var`, then `ls` hanging): mksh's
interactive job control (`ports/mksh (mksh R59c source): jobs.c`) blocks in
`sigsuspend()` waiting for `SIGCHLD`, expecting its own handler
(`j_sigchld()`) to run, reap the child via `waitpid(-1, &status, WNOHANG)`,
and update its internal job-state tracking. This kernel had **no real
signal delivery at all** — `sys_rt_sigaction()` was a no-op stub, so no
handler was ever invoked for anything, and `sigsuspend()` just slept out
one tick and returned `-EINTR` unconditionally. mksh's wait loop had no
way to ever learn the child had exited, hanging forever.

Real signal delivery was implemented from scratch:
- `struct proc` (`include/core/proc.h`) gained `sig_handler[32]`/
  `sig_restorer[32]` (populated by `sys_rt_sigaction()`,
  `kernel/sys/sys_sig.c`), `saved_sigtf` (stashed pre-signal trapframe),
  and `in_signal` (re-entrancy guard). `allocproc()` zeroes these for a
  freshly claimed ptable slot (they weren't before — a reused slot
  silently inherited a previous, unrelated process's handler addresses,
  which crashed the *next* process to occupy that slot the moment
  anything tried to delivver a signal to it). `fork()` copies handler/
  restorer/mask from parent to child (POSIX semantics) but not pending
  signals.
- `kernel/proc/proc.c`'s `exit()` sets `SIGCHLD` pending on the parent.
- `kernel/sys/syscall.c`'s `syscall()` calls `deliver_signals(tf)` in its
  tail, right before returning to user mode: if there's a pending,
  unblocked signal with a registered handler, it redirects the trapframe
  to jump to the handler (`rdi`=signal number per SysV ABI, `rip`=handler)
  instead of resuming the interrupted code, having first stashed the
  original trapframe in `proc->saved_sigtf`.
- The return address is **not** `sig_restorer[sig]` — checked, `SA_RESTORER`
  is not defined anywhere in this musl build, so `sigaction()` never fills
  in a restorer; it's always 0, and pushing 0 as a return address crashes
  with `rip=0` the instant the handler function returns. Instead,
  `deliver_signals()` injects a tiny 7-byte trampoline (`mov eax,15;
  syscall` — `rt_sigreturn`) directly onto the user's own stack, below the
  current `rsp` (clear of the SysV red zone), and points the return
  address at that — the standard technique for when the C library doesn't
  supply its own restorer.
- `sys_rt_sigreturn()` restores `*proc->tf` from `proc->saved_sigtf`,
  clearing `in_signal`.

**A real bug found and fixed along the way**: the first version of
`sys_rt_sigaction()` translated the user's `struct sigaction` pointer
*once* via `uva2ka()` and then did a bulk `memset`+copy across the whole
~152-byte structure. `uva2ka()` only guarantees the *page containing that
address* is mapped, not everything up to some further offset — when the
user's pointer landed close enough to a page boundary, the write ran past
the mapped page into whatever physical page happened to be next in kernel
address space (P2V is a flat offset, so "next page" there has nothing to
do with the user's actual next page), corrupting unrelated kernel memory.
This crashed every freshly `exec`'d `/bin/sh` almost immediately, since
mksh calls `sigaction()` during its own startup. Fixed by only touching
the two individual 8-byte fields actually needed (`sa_handler` at offset
0, `sa_restorer` at offset 144), each through its own `uva2ka()` call.

`sys_waitpid()`/`sys_wait4()` (`kernel/sys/sys_proc.c`) were also rewritten
on top of a new `waitpid(pid, status, options)` in `kernel/proc/proc.c`
(refactored from the old any-child-only, always-blocking `wait()`, which
is now just `waitpid(-1, 0, 0)`): a specific `pid` (not just "any child"),
`WNOHANG` (return 0 immediately instead of blocking), and — importantly —
a genuinely optional status pointer. The old code ran a NULL status
pointer through `argptr()`, which rejects NULL/unmapped addresses outright
(page 0 is deliberately unmapped) — but `waitpid(pid, NULL, WNOHANG)` is
completely standard, and is exactly what mksh's job control does.

**Status: signal delivery mechanically works** — verified directly (a
debug build logged the handler being invoked with the correct trapframe
and `sys_rt_sigreturn()` restoring cleanly, no crash, repeatable). **mksh's
job control still doesn't reliably progress afterward**, including for
plain non-piped foreground commands, not just the original piped-into-head
case. Root cause not yet isolated. Leading suspects, in order of
likelihood:
- `sys_setpgid()`/`sys_getpgid()` are no-op stubs (`kernel/sys/sys_proc.c`)
  — `j_sigchld()` (`ports/mksh (mksh R59c source): jobs.c:1344-1348`) has an early-out
  keyed on `j->ppid == procpid`, which may never resolve the way mksh
  expects without real process-group tracking.
- `getrusage(RUSAGE_CHILDREN, ...)` — called at the top of `j_sigchld()` —
  isn't verified to behave sanely under this kernel's implementation
  (`kernel/sys/sys_proc.c`'s `sys_getrusage()` only fills in `ru_utime`
  from the *calling* process's own tick count, not accumulated child
  time; `RUSAGE_CHILDREN` isn't distinguished from `RUSAGE_SELF` at all).
- Possibly something in how `j_sigchld()` matches a reaped `pid` against
  its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails
  to find the job for reasons not yet traced.

### Resolved 2026-09-22: it was never the kernel

**`X=$(...)` hung because zsh was built with `ESRCH` defined as `EINVAL`.**

`ports/zsh/Makefile` carries a block of `zsh_cv_*` cache answers, for the
checks autoconf cannot run when cross-compiling. One of them was

```make
zsh_cv_sys_killesrch=no
```

Read as a property that looks reassuring. Read as the answer to the question
configure actually asks — *"does `kill(pid, 0)` return ESRCH correctly?"* — it
says the opposite. `no` makes configure emit `BROKEN_KILL_ESRCH`, and
`Src/zsh_system.h` answers that with:

```c
#ifdef BROKEN_KILL_ESRCH
#undef ESRCH
#define ESRCH EINVAL
#endif
```

zsh's `waitforpid()` (`Src/jobs.c:1569`) ends its wait on

```c
while (!errflag && (kill(pid, 0) >= 0 || errno != ESRCH))
```

so with `ESRCH` redefined to `EINVAL` (22) and the kernel correctly returning
`ESRCH` (3), that condition was true for ever. The shell waited, in
`sigsuspend`, for a child it had already reaped. Every command substitution in
the shell hung.

The instrumented build says it in one line — `kill` returned −1, `errno` is 3,
and the constant it is compared against is 22:

```
WFPkill -1 3 22
```

**Autoconf's own cross-compilation default for this check is `yes`.** Deleting
the line would have been correct. Writing it down as a "fact about SchOS" is
what got it wrong, and the block's comment now says to read each name as the
question rather than as a property.

#### What this cost, and what to take from it

This sat in section 7 as "mksh's job control still doesn't reliably progress
... root cause not yet isolated" with three suspects, none of which were it.
All three have since been eliminated on their own merits — `setpgid` and
`getpgid` are real, `getrusage` is not consulted on the path, and the pid
matching was never reached.

Two corrections to the note written on 2026-09-21, both worth keeping because
they were confident and wrong:

- **"Not zsh — `mksh -c` hangs the same way" was false.** mksh installs as
  `/bin/sh`, not `mksh`, so that test was a `command not found` whose output
  scrolled past. mksh was never affected. Had the binary been named correctly
  the bug would have been localised to zsh's build in one step.
- **The kernel was cleared and then still suspected.** `cmd/sigtest` had
  already proven the SIGCHLD/sigsuspend contract, and a musl-linked probe had
  already shown `kill(pid, 0)` returning −1 with `errno == ESRCH == 3` and
  zsh's own loop condition evaluating false. Both results were correct and
  both were read as "so it must be something subtler in the kernel". The thing
  they actually pointed at was the one place the probe and the shell differed:
  **how each was compiled.**

The lesson is narrow and worth stating: when a probe that reproduces a
program's exact syscall sequence behaves correctly and the program does not,
stop looking under the syscalls and start looking at the build.

#### Two more in the same block

Checking the rest of `ZSH_CACHE` after fixing that one found two more answers
claiming capabilities SchOS does not have, both `yes`, both measurable on the
running system in one line each:

- `zsh_cv_sys_path_dev_fd=/dev/fd` — neither `/dev/fd` nor `/proc/self/fd`
  exists. `<(cmd)` therefore failed at the point of use with `cat:
  /dev/fd/11: No such file or directory`, which reads as a broken command
  rather than an unimplemented feature.
- `zsh_cv_sys_fifo=yes` — `mkfifo()` does not fail, it *succeeds* and produces
  a character device: `mknod()` turns `S_IFIFO` into a `T_DEV`, so `ls -l`
  shows `crw-rw-rw- 438, 0` and nothing written comes back out.

Both are `no` now, and `<(cmd)` says *"doesn't look like your system supports
FIFOs"* — which names the missing feature instead of blaming `cat`. The gaps
themselves are recorded in
[../architecture/filesystem.md](../architecture/filesystem.md).

---

This was left here deliberately rather than continuing to iterate blind:
SIGCHLD delivery is left **enabled** (`kernel/proc/proc.c`'s `exit()`)
since disabling it guarantees zero chance of mksh's wait loop ever exiting,
whereas leaving it on at least gives the cases that don't hit the
still-unknown remaining gap a chance to work. Next session should start
by instrumenting `j_sigchld()`'s own control flow (or strace-equivalent
logging in the kernel around `sys_setpgid`/`sys_getrusage`/`sys_waitpid`
calls specifically while reproducing `cd; cd var; ls` or even simpler
`ls -l /bin` followed by a second command) rather than re-deriving the
signal-delivery mechanism, which is now confirmed solid.

---

## 7b. The loader jumped into the framebuffer and executed pixels

The same bug as section 4, in the other loader, found five years later by a
compiler upgrade. Worth its own entry because it is the *third* time a register
the code depends on but never asks for has broken this project.

**Symptom.** Boot died immediately after `schos: exiting boot services` with a
KVM internal error and a register dump:

    RIP=00000000801a5dc5  RBX=0000000080000000
    Code=13 0e 0c 00 13 0e 0c 00 ... dc d5 d1 00 dc <d5> d1 00

`0x80000000` is the framebuffer. `0x0C0E13` is `SCH_BG` and `0xD1D5DC` is a grey
from the splash's metal gradient. So `RIP` was *inside video memory* and the
bytes being executed were the boot logo. The machine was executing the picture
of itself starting up.

**Cause.** `boot/uefi/main.c`'s hand-off to the kernel:

    "mov %0, %%cr3\n\t"
    "mov %1, %%rbx\n\t"          <- writes rbx
    "jmp *%2\n\t"
    : : "r"(PML4_ADDR), "r"(fb), "r"(entry)
    : "memory");                 <- rbx NOT declared clobbered

Nothing told the compiler rbx was written, so it was free to allocate an
*input* there. clang 21 allocated `entry`:

    movq %rax, %cr3
    movq %rcx, %rbx      <- the framebuffer, into rbx
    jmpq *%rbx           <- %2 was rbx; it now holds the framebuffer

The fix is `"rbx"` in the clobber list, which forbids the allocation. Confirmed
by reading the generated assembly before and after -- `jmpq *%rbx` became
`jmpq *%r14`.

**Why it appeared now.** It did not. It had been latent since the loader was
written and worked only because every compiler that had built it happened to
choose a different register. It surfaced the first time the binary was relinked
with a different clang.

That is the whole lesson, and it is identical to section 4's: **a register the
code depends on, that nothing in the source actually asks for, is not a
dependency -- it is a coincidence with a long half-life.** Section 4 was the
BIOS loader passing the framebuffer in EAX by luck; section 8 was `%fs:0` used
as scratch when the ABI reserves it; this is an asm block writing a register it
did not declare.

Reading the *generated* assembly is what settles all three, and it takes a
minute:

    clang $(UEFI_CFLAGS) -S -o - boot/uefi/main.c | grep -B3 'jmpq.*\*'

**A second failure sat underneath it and is worth its own note.** The loader had
been stale for half a day -- the Makefile rule for `BOOTX64.EFI` did not list
`logo.h`, `theme.h` or `font.h`, all of which `boot/common/ui.c` includes, so a
new logo never rebuilt it. The splash kept drawing an old mask while `make`
reported nothing to do. Fixing that dependency is what caused the relink, which
is what exposed the register bug. A stale artifact hides the bug that the fresh
one reveals.

---

## 8. Two ABI assumptions that only GCC was keeping true

Found when the build moved from GCC-in-WSL to clang-on-Windows. Neither is a
clang bug; both are places where the code relied on GCC doing something it
never promised, and both had been latent for years.

### `%fs:0` was being used as scratch — it is the TCB self-pointer

`syscall_entry` (`kernel/arch/x86_64/trapasm.S`) stashed `%rax` at `%fs:(0)`
while it fetched `proc` to find the kernel stack. The comment called it a
"per-CPU scratch slot". It is not: the x86-64 TLS ABI reserves the word *at*
the thread pointer to hold the thread pointer, and a compiler may read it to
materialise the TP before applying a variable's offset.

GCC compiles `__thread` access to a direct `%fs:-8` segment reference and never
reads that word, so the clobber was invisible. clang at `-O0` emits
`movq %fs:0, %rax; addq $-8, %rax; (%rax)` — reading back the syscall number
this code had just stored. The first process's first syscall is
`exec` (7), so `proc` resolved to address `7 - 8 = -1` and the kernel
page-faulted recursively until the stack was gone.

Fixed by moving the scratch to `%fs:(8)`. Positive offsets are safe: the TLS
block lives at negative offsets and `seginit` places the thread pointer 2048
bytes into a 4096-byte page.

**How it was found** is the part worth copying next time. Three rounds of
`cprintf` narrowed it to "the word is right at `mpmain`, wrong by the first
trap" and no further — the instrumentation perturbed the timing enough to move
the symptom. A hardware watchpoint through QEMU's gdb stub named the
instruction on the first run:

```
qemu ... -S -gdb tcp::26000
gdb -batch -x tcp.gdb        # hbreak mpmain; watch *(unsigned long*)kernel_fs_base
```

`seginit` also now writes the self-pointer, which it never did. That is ABI
-correct regardless and costs one store per CPU.

### `_start` read incoming registers with `register ... asm("rdi")`

`lib/glue.c` captured `argc`/`argv` from `%rdi`/`%rsi` using GCC local register
variables. Both compilers only guarantee that binding for inline-asm operands;
using it to pick up an incoming argument is undefined. clang ignored it and
read two uninitialised stack slots:

```asm
movl  -0x4(%rbp), %eax      # should have been %edi
movq  -0x10(%rbp), %rax     # should have been %rsi
```

Every program still ran. `init`, `shmtest` and `socktest` take no arguments and
passed their entire suites; the only symptom in the whole system was
`inputtest -n 6` ignoring its limit. Moved to `lib/start.S`, where reading
incoming registers is expressible and correct — `exec` passes argc/argv in the
SysV argument registers already, so `_start` calls `main` without touching
them.

---

## 9. The boot looked like a hobby OS, and two things were why

Not a bug — a design problem, fixed by looking at a screenshot instead of the
code.

**The display was the kernel log.** Boot printed `pci: 6 devices` and
`vga: display controller 1234:1111, aperture fd000000 (16 MB)` to the screen
the user is looking at. That single fact is most of what separates a hobby
project from a product; every shipping system puts that behind a flag.

`cons_serial_only` already existed for panic, so the fix was to start it *set*.
Everything still goes out the UART — which is where `tools/windows/bootcheck.ps1` reads
it from, so all five suites kept passing untouched — and `console_takeover()`
clears it on the first `exec()`, the moment userspace genuinely starts.
`vga_init` no longer clears the screen, so the splash survives the whole of
boot rather than being wiped a few hundred milliseconds in.

**There were three unrelated designs.** The splash was deep purple with gold
text, deliberately matching the *panic* screen. The console was cream with
near-black text. So a healthy boot was dressed as a crash, then abruptly
became an amber terminal. `include/gfx/theme.h` is now the one palette, and the
rule is: **normal states are calm and dark; only failure is loud.** PSOGM keeps
the purple entirely to itself, which is what makes it mean something.

The splash copy went from "SchOS Boot Manager / version 1.0 / Memory: 1023 MB
/ Press F8 for boot menu..." to the product name, a rule in the accent colour,
and the one hint that is actionable — moved to the bottom, where a hint
belongs. No shipping OS names its bootloader on the splash.

### Three latent bugs it flushed out

Retheming meant `stage2main.c` included `types.h` properly for the first time,
and the 32-bit build stopped compiling:

- `_Static_assert(sizeof(uint64) == 8)` fired. `uint64` was `unsigned long`,
  which is **4 bytes** in the `-m32` loader — the file even carried a local
  `e820_u64` workaround with a comment saying so. Now conditioned on
  `__SIZEOF_LONG__` rather than on the platform, which covers LP64, LLP64 *and*
  ILP32 correctly.
- `x86.h`'s `stosq` and `xchg` use `rep stosq` and `xchgq`, which do not exist
  in 32-bit mode. They only ever compiled because `uint64` was silently
  narrowed. Now behind `#ifdef __x86_64__`.
- The progress bar computed `(uint64)done * 100 / total`, a 64-bit divide that
  needs libgcc's `__udivdi3` — which a freestanding loader does not link. Now
  scaled to KiB and done in 32 bits.

All three had been sitting in a bootloader that worked fine, waiting for
someone to make `uint64` mean what it says.

---

## 10. A full-screen program typed the whole session into the shell

Keys reached `/dev/input/event0` **and** the tty line discipline at the same
time. That is correct for two consumers of one keyboard and completely wrong
for a full-screen program: while it had the display, the shell behind it was
quietly accumulating every keystroke, and the first Enter afterwards ran
whatever had piled up. Arrow keys arrived as CP437 glyphs on the console for
good measure.

The fix is the standard one and it is a *primitive*, not a patch: an exclusive
grab, `EVIOCGRAB`, with Linux's ioctl number. While a device is grabbed its
events still flow to `/dev/input/eventN` but both keyboard drivers withhold the
decoded character from the console.

Three details that matter more than the feature:

- **The grab is released by the devsw `close` hook**, not by the program. A
  program that grabs the keyboard and then crashes would otherwise leave the
  console permanently deaf, with no way to type the command that fixes it.
- **Modifier state is still tracked while grabbed** (`kbdgetc` keeps updating
  shift/caps and only skips the character). Otherwise releasing the grab would
  hand the line discipline a stuck Shift.
- **The USB path updates `prev[]` before returning**, or the next report would
  look like every held key had just been pressed again.

It is also exactly what Schcom will need, which is why it lives in the input
layer rather than in any one client.

### The near-miss worth remembering

The first attempt appeared to work: the harness passed, every assertion green.
It had not worked at all — the ioctl call was never inserted into the client
source, because a scripted string replacement silently failed to match. The
`grep` that should have caught it printed three hits (a comment, a `#define`,
and the *release* call) and was read as confirmation without counting what
they were.

What actually caught it was reading the transcript rather than the verdict:
four bare `#` prompts after quitting, which is precisely the symptom the change
was supposed to remove. `input_grab()` now announces itself, so "did the grab
happen" is never again a thing anyone has to infer.

---

## 11. `ps` and `top`, and why they never worked

/proc was a set of character devices: init ran `mknod("/proc/uptime", 6, 2)` and
friends, so every file was a real inode on the root filesystem with a
major/minor behind it. That is fine for a fixed list and cannot work at all for
the thing /proc is mostly for -- one directory per process. You cannot mknod a
directory for a process that does not exist yet and will be gone in a minute.
`ps` calls `dirtree_flagread("/proc")`, got a listing with no numbers in it, and
reported nothing. It was never a `ps` problem.

So /proc is now a real filesystem behind the same `struct fs_ops` vtable SchFS
uses (`kernel/fs/procfs.c`, `PROCFS_DEV`), mounted by init with
`mount("proc", "/proc", "proc", ...)`. Nothing is stored: directory listings
are generated from the process table when asked, file contents when read.

Three things worth keeping in mind:

- **The split is deliberate.** `kernel/fs/procfs.c` owns the *namespace* (what
  exists); `kernel/dev/procfs.c` keeps the *generators* (what the global files
  say). They change for different reasons, so `proc_gen_global()` is the only
  thing that crosses between them.
- **procfs never touches ptable.** `proc_snapshot()` copies what is needed out
  from under the lock, so a process that exits between the readdir that listed
  it and the open that reads it produces a stale line rather than a
  use-after-free.
- **Sizes are reported as zero**, exactly as Linux does. A synthetic file has
  no length until it is generated, and generating every one on every `stat()`
  to fill in a number almost nothing reads would be absurd.

### What the suite caught

Marking every procfs inode read-only broke `echo s > /proc/sysrq`, and three
USB checks failed on it within a minute of the change -- they drive the disk
through sysrq. /proc is read-only *except* sysrq, which is the one deliberate
write target, and the code now says so in one place.

### Still not right

`ps -o comm` prints `[init]` rather than `init`. That field comes from
`/proc/<pid>/exe`, which Linux makes a **symlink** and toybox `readlink()`s;
ours is a regular file, so the readlink fails and toybox brackets the name as
it would for a kernel thread. Default `ps` and `top` output are unaffected.
Fixing it needs symlink support in procfs, which belongs with the rest of the
VFS work rather than as a one-off.

---

## 12. A test program crashed at random, and had been typing its own launch command

Symptom: a full-screen framebuffer program, launched with arguments, worked
four times and then died with

    pid 9 <prog>: trap 14 err 4 on cpu 0 rip 0x2269be addr 0x18--kill proc

`err 4` is a user-mode read of a non-present page and `addr 0x18` is a null
pointer at offset 24. `llvm-nm` put `rip` inside the routine that opens the
program's overlay view, which dereferences a pointer the program only fills in
once a level is loaded -- at offset 0x18 on x86-64, because the three 64-bit
words of a base struct sit in front of it. So the program had been asked to open
that view before there was anything to show.

**The technique is the transferable part:** the fault address alone said
"null pointer plus 24", which is nearly useless; `llvm-nm` on the faulting `rip`
turned it into a function name, and the offset then identified the exact field.
Reach for that before adding printfs to a program you did not write.

The cause was ours, and it was visible in the log the whole time: every run
printed a burst of garbage bytes *before* the grab was taken, and the amount
differed each run.

**`/dev/input/eventN` had no notion of being opened.** Events were queued from
boot onwards whether or not anything was listening, into one 256-entry ring
with one read cursor. When the program opened the device it inherited the entire
backlog -- including the keystrokes of the command line that had launched it,
which the shell had already consumed -- and replayed them as input. Whether that
crashed depended on which keys happened to be sitting there, which is exactly
the "sometimes" in the report.

Fixed by giving `struct devsw` an `open` hook to match `close`, and having
`/dev/input`:

- drop the backlog on open (`r = w`) -- a reader starts from *now*;
- only queue events while `opens > 0`, which is what Linux does: an evdev
  device with no clients does not accumulate.

Linux gets both for free by giving every open its own client queue. One shared
ring plus "start from now" is the same answer for a single reader, which is the
only case that exists today; per-client queues become necessary the moment
Schcom and a client both want the keyboard.

### Also fixed

`cd` into a file reported "Operation not permitted". It was right to fail, but
`sys_chdir` returned a bare `-1`, so the shell printed whatever errno happened
to be lying around. It returns `-ENOTDIR` now, and `-ENOENT`/`-EFAULT` on the
other two failure paths -- "Not a directory" sends you to the right place,
"Operation not permitted" sends you looking at permissions.

## 13. The clock ran seven times slow, and the graphics card was why

The report was about time. fastfetch said the machine had been up for 3 seconds
after 15 real ones. `time ping 127.0.0.1` claimed `0m03.12s real` for a command
that visibly took 20. ping's one-second interval was closer to six. Three
programs, three wrong numbers, all wrong by roughly the same factor and all in
the same direction -- which is the shape of one shared cause, not three bugs.

The shared cause was obviously the tick. Everything above is derived from
`ticks`, and `ticks` comes from the timer interrupt. So the hunt started at the
bottom of that chain, and the first thing it found was real:

```c
// If xv6 cared more about precise timekeeping,
// TICR would be calibrated using an external time source.
lapicw(TICR, 10000000);
```

That is xv6's own comment, still in the tree, admitting that the LAPIC timer's
countdown value was never measured against anything. 10000000 yields the
intended 100 Hz only if the APIC bus happens to run at exactly 1 GHz.

It was a genuine gap and it is now fixed -- `calibrate_timers()` measures the
LAPIC and the TSC against the 8254 PIT, which is fixed at 1193182 Hz by the
hardware and is exactly the external time source the comment asked for. But it
was not the bug. The measurement came back at 999.9 MHz. The bus really was
1 GHz, `TICR = 10000000` really was 100 Hz, and the clock was still wrong.

### What the instruments said

Every clock agreed with every other clock, which is the confusing part:

| measured against | TSC | LAPIC timer |
|---|---|---|
| the PIT | 3792 MHz | 999 MHz |
| one CMOS RTC second | 3791 MHz | 999 MHz |

Three independent clocks, mutually consistent, and 3792 MHz is exactly the
5700G's base clock. So the timer was programmed for 10 ms and the registers
read back correct at runtime: `TICR=10000314`, `TDCR=11` (divide by one),
`LVT=0x20020` (periodic, vector 32). Interrupts nonetheless arrived every 68 ms.

Two hypotheses died next. Interrupts were not being masked -- the CPU had
`IF=1` for 99.98% of the measured window. And it was not the idle `hlt()`:
deleting it changed nothing, and neither did dropping to `-smp 1`.

What broke it open was measuring the gaps between consecutive timer interrupts
instead of averaging them:

```
timer gaps us: 2851 137614 2438 136293 3805 136889 3019 138420 6950 137248
```

Bimodal, strictly alternating. Not a slow clock -- a clock being interrupted.
In one-shot mode the short gaps sharpened to 10031, 10078, 10064, 10011 µs,
which is a perfect 10 ms: the LAPIC was doing its job exactly as programmed,
and then something was eating the next 130 ms.

Timing every trap by vector found it in one boot:

```
trapno 43: 5828 ms total, 510000 entries
```

Vector 43 is `T_IRQ0 + 11`. IRQ 11 was taking **72,000 interrupts a second** and
consuming 83% of CPU0.

### The actual bug

IRQ 11 was the e1000. Its handler reads ICR, and reading ICR clears it, so the
driver looked correct -- and it was. The cause register read back **zero** on
essentially every one of those 500,000 interrupts: the e1000 was being handed
somebody else's interrupt and correctly answering "not mine".

The somebody else was `virtio-gpu-pci`, sharing IRQ 11, **with no interrupt
handler registered at all**. PCI INTx is level-triggered. The device asserted
the line, the kernel signalled EOI, nothing had cleared the device's interrupt
status, so the line was still asserted and it fired again immediately. Forever.

The fix is one read. Reading virtio's ISR byte clears it and drops the line:

```c
static void
vgpu_intr(void *arg)
{
  struct vgpu *g = arg;
  if (g && g->isr)
    (void)*g->isr;
}
```

This driver polls for command completion and wants no notifications. "Wants
none" and "gets none" are different things, and only the second one is free.

Booting without `-device virtio-gpu-pci` had already proved it: 100 ticks in
999 ms, and the e1000 taking 2 interrupts instead of half a million. With the
handler registered, uptime advances 5.01 seconds per 5 real seconds.

### What to take from it

**A shared level-triggered line is only as good as its worst driver.** Every
other device on IRQ 11 was correct. One driver that never acknowledged its
device was enough to starve the whole machine.

**The symptom appears where the missing CPU time hurts first, which is nowhere
near the fault.** This surfaced as a timekeeping bug, in userspace, in three
unrelated programs. It sent the investigation into the timer code, where there
was a real defect sitting in plain sight with a comment explaining itself --
and fixing that defect changed nothing, because it was not this one. A
plausible bug found early is the most expensive thing in a debugging session.

**Averages hide alternation.** "68 ms per interrupt" suggested a wrong
frequency and implied the timer. "10 ms, then 130 ms, repeating" says the timer
is fine and something else is stealing the machine. Same data, one of them
divided.

`irq_dispatch` now measures the rate on every line against the TSC -- not
against `ticks`, which is the first casualty of a storm -- and says so once:

```
irq 11: 75894 interrupts/second -- a device on this line is not acknowledging it.
irq 11:   handler 'virtio-gpu' is attached
irq 11:   handler 'e1000' is attached
```

That is the message this cost an afternoon for. The threshold is 20000/s, which
no real device on this machine sustains.

### Also fixed

`microdelay()` was an empty function body -- xv6's stub, with a comment saying
real hardware would want to tune it. Every `microdelay(1000)` in the xHCI reset
path and the USB hub power-on sequence returned instantly. Those waits are in
the specifications because the hardware needs the time; a port that "works
anyway" is racing and winning. It now spins on the calibrated TSC.

The tick rate itself stopped being a magic number. `HZ` lives in `param.h`, and
the dozen places that wrote `ticks / 100` or `sec * 100` use it. `USER_HZ` is
separate and deliberately does not follow it: musl answers
`sysconf(_SC_CLK_TCK)` with 100 no matter what, so `times()` and `/proc/stat`
are scaled at the syscall boundary the way Linux does it. Before, `sys_times`
returned raw ticks -- correct only while `HZ` happened to be 100, and silently
wrong the day anyone changed it.

## 14. A shell that would not start, and four bugs underneath it

Porting zsh was supposed to be a port. It found four defects in the kernel and
one in the filesystem, none of which had anything to do with zsh, and every one
of which had been there the whole time.

### The environment did not exist

`kernel/fs/exec.c` built the initial stack for every new program and wrote this:

```c
stack_slots[slot++] = 0;    // end of envp (no environment)
```

Not "no environment yet" — no environment, ever, for any process. musl calls
`execve(path, argv, envp)`, which is Linux syscall 59, which SchOS answers; the
third argument was simply never read. So `$TERM` was unset, `$PATH` was whatever
each program invented for itself, and a variable exported in a shell did not
survive the exec that was supposed to carry it.

That is fatal for zsh specifically. A line editor with no `TERM` cannot ask
terminfo how to move a cursor, so ZLE is exactly as useless as it would be with
no terminal library linked at all.

`exec()` takes an `envp` now, `sys_exec` reads the third argument, and schinit
manifests grew an `Environment` key:

```
Environment = TERM=linux
Environment = PATH=/bin:/usr/bin
```

The userland `exec(path, argv)` spelling stays, as a wrapper in `lib/glue.c`
that passes a null environment explicitly. The raw stub is `execve` now —
otherwise a two-argument caller would leave `%rdx` holding whatever happened to
be in it, and the kernel would read that as a pointer.

### Which immediately overflowed the kernel stack

The three pointer arrays exec needs came to about 1.8 KB, and `KSTACKSIZE` is
4096, under a call chain that already runs exec → loaduvm → readi → SchFS →
btree. The first exec after the change took pid 1 down.

The canary caught it cleanly, named the process, and printed a stop code, which
is the whole reason it exists. The arrays live in one `kalloc`'d page now.

### `wait` said "operation not permitted"

```
compdef:80: wait failed: operation not permitted
```

`waitpid` returned a bare `-1` for two different failures: no children to wait
for, and interrupted by a signal. musl reads any return in [-4095, -1] as
`-errno`, so `-1` arrived in userspace as `EPERM`.

zsh saw it, printed that message, and abandoned the rest of `/etc/zsh/zshrc` —
so the shell came up with its default prompt and no completion, which reads as
a broken config file rather than as a `wait()` that could not say what happened.
It returns `-ECHILD` and `-EINTR` now.

This is the third time a bare `-1` has cost real time (see §12, `sys_chdir`).
**EPERM on SchOS should be read as "a syscall returned -1 and meant something
else."** There is no permission model here for it to legitimately come from.

### mmap ran out of slots and blamed memory

```
compdef:80: fatal error: out of memory
```

On a machine with 1000 MB free, of which zsh had used 1.5. `NVMA` was 64 — the
per-process mmap table — and musl's allocator takes a fresh mapping per size
class and does not release them eagerly. compinit reads twelve hundred
completion functions, ran out of slots, and got `-1` from `sys_mmap`, which is
`EPERM` again, which `malloc` turns into NULL, which zsh reports as out of
memory. Three translations from the truth.

`NVMA` is 256 and the failure returns `-ENOMEM`.

256 ran out too, later, and silently: `x=($(<file))` over 30000 words exited
zsh with status 1 and no message. zsh takes one 16 KB mapping per heap block,
so 256 slots was 4 MB of heap. Neighbouring anonymous mappings are merged into
one record now, as Linux merges them. Addresses come from a bump allocator, so
a new mapping nearly always starts where the last one ended. `munmap` trims or
splits the record. The slots now bound how many *separate* regions a process
holds, not how many `mmap` calls it has made, and 100000 words split fine.

### exec took 32 arguments

`MAXARG` was 32 and `MAXENV` 64, in fixed arrays, so `rm *.o` in any build
directory failed with `E2BIG`. exec now reads the caller's arrays where they
are, through the uaccess helpers and a bounce page, so a second thread
rewriting them cannot slip a kernel address in after a check. It sizes the new
stack to fit them. The limits are Linux's: 2 MB of strings and pointers
(`ARG_MAX`), and 128 KB per string.

### And underneath all of it: a small write costs 58 ms

With everything above fixed, `compinit` still appeared to hang. It was not
hanging. Measured on this machine, 200 writes:

| | time | per write |
|---|---|---|
| to one open file on SchFS | 11.75 s | **58 ms** |
| open + write + close each time | 21.39 s | **107 ms** |
| to `/dev/null` | 0.14 s | 0.7 ms |

`compdump` writes its digest with a few thousand separate `print` calls. At
58 ms each that is minutes, and it looks exactly like a hang. `compinit -D`,
which skips the dump, finishes in 3.6 seconds — the scan was never the problem.

The cause is not a bug; it is the design, stated plainly at the top of
`kernel/fs/schfs/kmount.c`:

> Every entry point below that changes anything is one transaction group: it
> takes the volume lock, does its work, and commits before returning. That makes
> an operation durable the moment it returns.

A commit is two cache flushes and an uberblock write, and a flush is a barrier.
`schfs_ktx_open` already lets one `write(2)` of 1 MB be a single group — 267
block writes instead of 1410 — but nothing brackets *across* syscalls, so two
thousand small writes are two thousand barriers.

`cp` does not notice, because it uses large buffers. Anything that writes a line
at a time notices enormously.

**This is the open question, and it is a design decision rather than a fix.**
Deferring commits would mean giving up durable-on-return, which is a property
SchFS chose deliberately. The options are a writeback window like every other
filesystem has, committing on `fsync`/`close`/timer, or leaving it and accepting
that line-at-a-time writers are slow. Until it is decided, `/etc/zsh/zshrc`
leaves `compinit` switched off and says why.

### What zsh actually got

It runs, as `/bin/zsh`, and it is the console shell. ZLE works: arrow keys
recall history through terminfo, `$COLUMNS` reads 160 because `TIOCGWINSZ` now
answers with the real console geometry (§13), prompt expansion and colour work,
and startup is 0.14 s. mksh keeps `/bin/sh`, so schinit's no-jobs fallback is a
different program from the one that might have failed.

Completion is the piece that waits on the filesystem.

## 22. Every stack on every application processor was misaligned by eight

`jmp mpenter`.

x86-64 requires `rsp` to be 16-byte aligned immediately before a `call`, so a
function begins with `rsp` ending in 8 — the return address — and its
`push %rbp; mov %rsp,%rbp` prologue leaves `%rbp` 16-byte aligned. Compilers
build on that: an aligned vector store to a frame slot is `movaps`, and
`movaps` to a misaligned address is `#GP`.

`entry64mp` loaded the AP's stack pointer, which is page-aligned, and *jumped*.
No return address was pushed, so `%rbp` came out eight bytes off — on every
frame, on every application processor, for the life of the machine.

It cost nothing for as long as no function an AP ran needed an aligned vector
store. Then the console's glyph path started returning a small struct by value,
clang spilled it with `movaps`, and CPU 1 took a `#GP` the instant it announced
itself:

```
unexpected trap 13 from cpu 1 rip ffff800000120a3b <draw_glyph_px+0x1cb>
```

```
ffff800000120a37: 0f 10 45 90    movups -0x70(%rbp), %xmm0
ffff800000120a3b: 0f 29 45 b0    movaps %xmm0, -0x50(%rbp)   <- #GP
```

`movups` — *unaligned* — loads fine on the line above; `movaps` stores and
faults. The displacement is a multiple of 16, so the misalignment is `%rbp`
itself, which is a fact about how the function was entered rather than
anything it did.

`call mpenter` instead of `jmp`. Eight bytes for an ABI the whole toolchain
assumes.

### It hid behind a second bug

The first report of this was not a `#GP`, it was:

```
Stop code     ACQUIRE
      #2  cprintf+0xae
      #3  trap+0x2c4
```

`trap()` reported the fault with `cprintf()`, `cprintf()` takes the console
lock, and the faulting code was already holding it — every character drawn on
a verbose boot is drawn under that lock. So `acquire()` panicked about a lock
its own call chain owned, and the trap number, the faulting address and the
register state were all lost.

A kernel-mode trap ends in `panic()` a dozen lines later regardless, so the
console is put into panic mode *before* the report rather than during it. The
same fault then prints what it actually was, which is the entire purpose of
the report.

### And a third, in the same boot

The console reported `JetBrains Mono, 8x16 cells` — a name from one font and
metrics from another. The built-in bitmap had become a provider like any other,
so `draw_glyph_px()` installed it lazily on the first glyph; on a verbose boot
that happens before `gfx_font_init()` runs, and `gfx_font_init()` returned early
because a font was already installed. The console spent the whole boot in the
fallback while claiming not to be.

"A font is installed" and "a typeface was chosen" had been the same test
(`cfont == 0`) and quietly stopped being. They are two flags now, and
`gfx_font_ready()` answers the second one.

---

## 23. Every resolution but 800x600, on two real machines

Two reports in one afternoon. A desktop with an NVIDIA card booted fine at
800x600 and at anything else showed the kernel log as dotted lines across the
screen, and the loader's splash as four squashed copies of the logo side by
side. A laptop stopped dead after the kernel's `vga:` line. Three separate
defects were found under them; the third is a guard rather than a cure, and
the desktop has not yet been retested.

**The framebuffer address was cut in half.** The loader hands the kernel all
sixty-four bits of the aperture address in RBX, and `kmain()` took it as a
`uint32`. Below 4 GB that is harmless. Above it -- where a laptop's GPU is
entitled to put its aperture -- the low half is a plausible address pointing
at nothing. `kmain()` itself reads the loader's display block and puts the
early console on the right memory, but `vga_fb_init()` then re-published the
scanout from the truncated argument and the BIOS-era VBE block (32-bit
address, 16-bit sizes). The second publication won, and every line after
`vga:` went somewhere else. Both are fixed: `kmain(uint64)`, and
`vga_fb_init()` reads the same 64-bit block `kmain()` does and maps it with
`ioremap()` instead of `P2V()`.

**The loader mirrored the screen onto itself.** It copies its picture to
every other GOP it finds, for machines with two displays. But the firmware's
console splitter is itself a GOP, on a virtual handle, and its `Blt` forwards
to every real adapter -- the primary included. A laptop that reaches one panel
through two GOPs is the same thing with a shared aperture. Once the resolution
menu changed the primary's mode, the splitter still described the old one, so
each mirror pass wrote the primary's own pixels back over themselves at an
offset and with another mode's row length. Handles without a device path (the
splitter) and GOPs whose aperture overlaps the primary's are no longer mirror
targets, and a mode chosen in the menu re-matches the mirrors that remain.
QEMU's OVMF has the splitter too: `schos: 2 display adapter(s)`, both at the
same address.

**A mode the firmware cannot describe is refused.** After `SetMode`, the
loader now checks that the mode it got is the size it asked for, that the row
is at least as long as the width, and that the aperture holds that many rows.
If not, it goes back to the previous mode and says so in the menu. This proves
nothing about a mode that passes, but it stops a lying one from reaching the
kernel.