nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

docs: four directories, and a rule for when a thing leaves the roadmap

Browse source 
docs/ was ten files where a plan, a description, a guide and a bug report all
sat next to each other. It is now architecture/ (what SchOS is), plan/ (what it
will be), guides/ (how to work on it) and history/ (why it is like this), with
an index that states the one rule holding it together: when something is
finished it moves from plan/ to architecture/ and is rewritten as a description
of the thing rather than a record of building it.

A ticked checkbox says the work happened. It does not say what the work
*produced*, and six months later the checkbox is all that is left. The network
stack proved the point: finished, verified end to end, and for a month the only
description of it anywhere was a checkbox four levels of indentation deep.

So roadmap.md went from 534 lines to 181 -- nothing deleted, the done work
graduated. Seven subsystems that were finished and undescribed now have
descriptions: boot, kernel, syscalls, filesystem, networking, graphics, usb,
init. Every one ends with "What it does not do", which is the section that pays;
a document listing only strengths is marketing, and the reader who needs it most
is the one about to trip over a limitation nobody wrote down.

schfs.md still opened with "phases 1 through 3 1/2 are done, phase 4 is next"
long after all five had landed. It is a description now, with the phase record
and the hard-won bugs moved to history/.

Three corrections found on the way, all load-bearing:

  * projects/schcom/README.md claimed SCM_RIGHTS was done. There is no
    sendmsg/recvmsg in the kernel at all, and an entire GUI plan rested on it.
  * The root README said "No TCP/IP" months after the stack landed.
  * history/kernel.md sec.7 was marked "in progress" on a problem that process
    groups solved.

Also adds plan/1.0.md, which draws a finish line so the open list stops being
infinite, and marks schwm/ and schde/ superseded: in Wayland the compositor is
the window manager, so that split dissolves rather than being abandoned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
Nexoniarz 2026-09-12 17:13:35 +02:00
commit 57aff40228
59 changed files with 3981 additions and 1124 deletions
Download patch file Download diff file Expand all files Collapse all files

54
docs/history/README.md Normal file
View file

@ -0,0 +1,54 @@
# History
**Why SchOS is the way it is.** Bugs whose cause was nowhere near their symptom,
and the few write-ups that have nowhere better to live.
| | |
|---|---|
| [kernel.md](kernel.md) | The expensive ones. Boot, SMP, signals, ABI assumptions, `/proc`, input. |
| [schfs.md](schfs.md) | How the filesystem was built, and what each phase cost. |
| [moving-to-linux.md](moving-to-linux.md) | The Windows-to-Fedora move. |
---
## What belongs in here
> **A bug whose cause is not visible from the code that fixed it.**
That is the whole test. If reading the fix explains the problem, the fix is the
documentation and nothing needs writing down. If the fix looks arbitrary — a
constant moved by eight, a field reordered, a check inverted — then the
strangeness is load-bearing and the next person to touch it deserves to know
why before they change it back.
Every entry here is a thing that looked like a different problem than it was.
## What does not belong
- **A changelog.** `git log` is the changelog.
- **A bug that was found and fixed in an hour.** Cost is the filter.
- **How something works.** That is [../architecture/](../architecture/), written
in the present tense for somebody who does not care what it used to be.
- **Anything still open.** An unfinished investigation goes in
[../plan/](../plan/) or in the code as a comment. This directory is past tense.
## How to write one
**Symptom first, then diagnosis, then the fix, then the transferable part.**
The transferable part is the reason the entry exists. `__thread` being wrong for
per-CPU data is worth more than the specific null dereference it caused;
"a layout constant that several artifacts must agree on wants a check that stops
the build, not a comment asking people to remember" is worth more than the
bootloader offset that taught it.
**Say how it was found**, when the technique was the hard part. A hardware
watchpoint through QEMU's gdb stub named an instruction on the first run after
three rounds of `cprintf` had narrowed it to nothing — that is reusable, and the
bug is not.
**Do not clean up the embarrassing part.** The near-miss where a change passed
every assertion green *without having been applied at all* is one of the most
useful entries in this directory, and it only works because it admits that a
`grep` returning three hits was read as confirmation without counting what they
were.

698
docs/history/kernel.md Normal file
View file

@ -0,0 +1,698 @@
# Kernel history
Bugs that were expensive to find and whose causes are not visible from the code
that fixed them, plus a few subsystem write-ups that have nowhere better to live.
**Every entry here is a thing that looked like a different problem than it was.**
What each subsystem *is* today is [../architecture/](../architecture/); this is
only how it got that way.
| | |
|---|---|
| [1](#1-fixed-boot-didnt-reliably-reach-a-working-shell) | Boot did not reliably reach a working shell — ten bugs behind one symptom |
| [2](#2-naming-standardization) · [3](#3-coreutils) | Naming, and where the userland comes from. Not bugs. |
| [4](#4-two-stage-bootloader--psogm-splash) | The two-stage BIOS bootloader. **Since removed** — kept for the bugs. |
| [5](#5-at-syscalls-ignoredrejected-real-dirfds) | `*at()` ignored real dirfds, and `ls -l` hung |
| [6](#6-boot-speed) | Boot speed, in the loader |
| [7](#7-at-syscalls-real-signal-delivery-and-mksh-job-control) | Real signal delivery, and mksh's job control |
| [8](#8-two-abi-assumptions-that-only-gcc-was-keeping-true) | Two ABI assumptions only GCC was keeping true |
| [9](#9-the-boot-looked-like-a-hobby-os-and-two-things-were-why) | The boot looked like a hobby OS |
| [10](#10-a-full-screen-program-typed-the-whole-session-into-the-shell) | A full-screen program typed the session into the shell |
| [11](#11-ps-and-top-and-why-they-never-worked) | `ps` and `top`, and why they never worked |
| [12](#12-a-test-program-crashed-at-random-and-had-been-typing-its-own-launch-command) | A program crashed at random, typing its own launch command |
The SMP bring-up bugs are not here — they are summarised in
[../architecture/boot.md](../architecture/boot.md#4-smp), because the lesson
(`__thread` is wrong for per-CPU data) is load-bearing on the current code
rather than merely historical.
> **Where the old design plan went.** Sections 1-6 of this file used to hold a
> "Segregated Kernel" design: a Ring 0 Alpha / Ring 0 Beta split with driver
> modules isolated by CR3 switching, from an `IDEA.txt` that no longer exists.
> None of it was ever implemented, and it is superseded by
> [../plan/schapi.md](../plan/schapi.md) — a macOS-shaped system answers driver isolation with
> userspace drivers and a personality boundary, not a bespoke in-kernel ring
> split. It also referenced a Zig toolchain that is now clang/LLVM. Removed
> rather than left to be mistaken for a plan.
---
## 1. Fixed: boot didn't reliably reach a working shell
Investigated 2026-07-05, before phase 1 started (predates this design, not
caused by it). Originally manifested as `ksh` (mksh) crashing on every
launch, apparently inside musl's mallocng allocator (`get_meta`,
`src/malloc/mallocng/meta.h`, deliberately executing `hlt` on detecting
"corrupted" heap metadata — a `trap 13` GPF at a fixed rip). It wasn't a
musl bug. Six real kernel bugs were found and fixed across three
investigation passes, and mksh boots and runs cleanly with all of them in
place — the calloc assertion was a symptom of real memory/pointer-handling
bugs downstream in the kernel, not anything musl or mallocng did wrong.
Fixed:
- `copyuvm()` (`kernel/arch/x86_64/vm.c`, used by `fork()`) never copied
`mmap()`-backed `proc->vma[]` regions into the child — only `[0, sz)`.
Fixed via new `copy_vma_range()`, called from `fork()` (`kernel/proc/proc.c`).
- `kmem.freelist` (`kernel/core/kalloc.c`) could get corrupted by a wild
write elsewhere and would silently hand back a bad pointer, panicking far
from the actual cause. Added a canary check (`check_run()`) at the top of
both `kalloc()`/`kfree()` so corruption panics immediately at the point
it's touched, with a clear message.
- No TLB invalidation existed anywhere except the full `lcr3` reload on
context switch — `mappages()`/`vm_unmap_range()` never called `invlpg`.
Fixed by adding `invlpg()` (`include/x86.h`) to both functions.
- **`MSR_STAR` was misconfigured** (`syscallinit()`, `kernel/arch/x86_64/vm.c`):
`SYSRETQ` computes the returned `cs`/`ss` as `STAR[63:48]+16`/`+8`, which
requires the *compat* 32-bit code selector (`SEG_UCODE32`) in that field —
the kernel was putting the real 64-bit `USER_CS` there instead. Every
`sysretq` silently loaded an invalid `cs` (`0x3b`, pointing at the TSS
descriptor)/`ss` pair; QEMU's TCG doesn't validate segment descriptors on
`SYSRET`, so this went unnoticed until a hardware interrupt forced a
strictly-validated `IRETQ` with that bogus `cs` still loaded, which
correctly `#GP`'d.
- `sys_mmap`'s non-`MAP_FIXED` path (`kernel/sys/sys_mem.c`) assigned a fixed
8MB virtual-address window per `vma[]` slot regardless of the requested
length. A single allocation over 8MB (e.g. `realloc()` growth past that
size) mapped straight into a neighboring live mapping and hit
`mappages()`'s `panic("remap")`. Replaced with a real bump allocator
(`proc->mmap_next`, `include/proc.h`), so every mapping gets its own
disjoint, exactly-sized range regardless of size.
- **`uva2ka()`** (`kernel/arch/x86_64/vm.c`) returned only the page-aligned
base address of a user pointer's page, silently dropping the in-page byte
offset. Direct callers (`sys_ioctl`'s `TIOCGPGRP`/`TCGETS`, `sys_getcwd`,
signal masks, `times()`, `sysinfo()`, ...) read/wrote near the wrong
address whenever the pointer wasn't page-aligned — i.e. almost always.
Fixed at the source so every caller benefits.
- **`fetchint`/`fetchaddr`/`fetchstr`/`argptr`** (`kernel/sys/syscall.c`)
rejected any address `>= proc->sz`, wrongly assuming every valid user
pointer lives in the heap. Stack addresses and `mmap()`'d buffers (used
constantly by real programs, e.g. path-resolution buffers) live above
`proc->sz` and were being rejected outright. Fixed to rely solely on
`uva2ka()`'s page-table validation.
Bringing in toybox (see "Coreutils" below) surfaced four more, all fixed:
- Linux syscall 58 (`vfork`) wasn't in the syscall-ABI translation table at
all (`kernel/sys/linux_abi.c`) — every external command spawn from a
program that prefers `vfork()` over `fork()` failed outright. Mapped to
the existing `fork()` (a fully correct, if slightly slower, substitute).
- `sys_stat`/`fstat`/`fstatat` (`kernel/sys/sys_fs.c`) filled in SchOS's
own native `struct stat` layout regardless of the caller — musl/toybox
parse a fixed Linux ABI byte layout instead, so `st_mode` was garbage and
tools like `ls` couldn't tell files from directories, or executables from
non-executables. Added `struct linux_stat` + `stati_linux()`, gated on
Linux-ABI mode (`proc->syscall_map`).
- `sys_open`/`sys_openat` required `omode == O_RDONLY` *exactly* to open a
directory, but `opendir()`-style calls legitimately OR in `O_DIRECTORY`/
`O_CLOEXEC` alongside it. Fixed to check just the access-mode bits
(`O_ACCMODE`, added to `include/fcntl.h`).
- `sys_getdents64` read one `struct dirent` (16 bytes) at a time — a full
`bmap()`+`bread()` per slot. Directories here are hash-indexed (see
`tree_search()`/`tree_expand()` below), so a directory with a few dozen
real entries can have a size in the hundreds of KB, nearly all empty
slots — this made listing such a directory look hung (it wasn't a
deadlock, just thousands of redundant single-slot disk reads). Rewrote it
to read a full block at a time and resolve each entry's type via
`iget(inum)` (exported from `fs.c`, it was `static`) instead of
re-searching the whole directory by name with `dirlookup()` for every
entry. That rewrite's first version introduced a **new** self-deadlock:
the "." entry's inode number equals the directory's own, and `iget()`
correctly hands back the *same, already-locked* `struct inode*` — calling
`ilock()` on it again sleeps forever on a lock the process already holds.
Fixed by special-casing `xde->inum == ip->inum` (we already know it's a
directory, no need to re-lock at all).
A synthetic stress test (interleaved malloc/free, realloc growth crossing
`MMAP_THRESHOLD`, calloc, large allocations) runs to completion with zero
crashes, and mksh now boots straight to an interactive prompt, runs external
programs, pipes, and `ls`/`mkdir`/`grep` without issue.
## 2. Naming standardization
SchOS standardizes on exactly one name per role, regardless of the
underlying implementation: the shell is always installed and exec'd as
`/bin/sh` (currently mksh, built from `ports/mksh/`), and PID 1 is always
`/bin/init`. `cmd/init/main.c`'s exec loop only ever tries `/bin/sh` — no
ksh/mksh/dash/zsh/other-name fallbacks. dash was tried first
(BSD-licensed, minimal) while the boot crash above was still open, then
replaced with mksh once the underlying kernel bugs were fixed and dash's
lack of interactive features (history, completion, line editing) made it
feel unusably bare for day-to-day use; the dash port's source tree was left in
place rather than deleted, in case it's useful again (e.g. as a `/bin/sh`
POSIX-strict target distinct from an interactive login shell).
## 3. Coreutils
`ports/toybox/` (0BSD-licensed, https://landley.net/toybox/) provides `ls`,
`cat`, `cp`, `mv`, `rm`, `mkdir`, `grep`, `sed`, `find`, `chmod`, `date`,
`sort`, and more — see the `APPLETS` list in `ports/toybox/Makefile`. It builds
as a single multicall binary; `tools/mkfs.c` doesn't understand symlinks or
hardlink dedup, so `ports/toybox/Makefile`'s `install` target copies the built
binary once per applet name into `sysroot/bin/` rather than symlinking
(costs disk space per applet — currently ~44MB total — instead of one
shared inode, but needed no mkfs changes).
## 4. Two-stage bootloader + PSOGM splash
> **This describes the BIOS loader, which no longer exists** — see
> [`boot/README.md`](../../boot/README.md) for why it was removed and what
> replaced it. The section is kept because four of the bugs in it are about
> things that are still true: extended scancodes, register hand-off across a
> mode switch, and layout constants that several artifacts must agree on.
`boot/` used to be a single BIOS boot sector doing everything (VBE mode
set, A20, GDT, protected-mode switch, multiboot kernel scan/load) — silent,
and packed to within 4 bytes of the hard 510-byte MBR limit, so there was
no room to add so much as a print statement.
Split into two stages, the standard way real bootloaders solve this:
- **Stage 1** (`boot/bios/bootasm.S`, still the literal MBR sector, still
≤510 bytes) does exactly one thing: raw IDE PIO (LBA28) reads
`STAGE2_SECTORS` (16) sectors starting at disk sector 1 into memory at
`0x7E00`, then jumps there. Nothing else fits, nothing else is needed.
(Caught a real bug writing this: `outb $0x1F7,%al`-style immediates
silently truncate to 8 bits — ports 0x1F2-0x1F7 are all > 0xFF and need
the `%dx` register form, or you're hitting the wrong I/O port entirely.)
- **Stage 2** (`boot/bios/stage2.S` + `boot/bios/stage2main.c`, loaded at `0x7E00`,
occupies disk sectors 1-16) does everything stage 1 used to skip straight
to, plus a boot banner (BIOS teletype, real mode, before the VBE mode
switch — note this only reaches the video device, not serial, so it
won't show up in a `-serial file:...`-captured log), a splash screen, a
real-time loading bar, and an F8 boot menu (all described below). Not
size-constrained the same way as stage 1 — `tools/padstage2.pl` zero-pads
it out to a fixed `STAGE2_SECTORS*512` (8192) bytes so the kernel's
on-disk starting sector stays predictable regardless of stage2's actual
compiled size, and errors out (like `tools/sign.pl` does for stage 1) if
it ever grows past that budget instead of silently corrupting the layout.
The kernel now starts at disk sector 17 instead of 1 (`schos.img`'s `dd
... seek=17` in the root `Makefile`, and the equivalent `+(1+STAGE2_SECTORS)`
in `stage2main.c`'s `readseg()` — all three places that encode
`STAGE2_SECTORS` have to agree, or the kernel won't be found on disk at
all). `boot/bootmain.c` no longer exists; its logic was split between
`stage2.S` (real-mode/protected-mode setup) and `stage2main.c` (splash +
the unchanged multiboot-header-scan-and-load logic).
The VBE mode info block itself lives at `0x30000`, not the classic
single-stage bootloader's `0x9000` — stage2's own code now runs from
`0x7E00` up to nearly `0x9000` once the boot-menu/progress-bar logic is
included, and a block at `0x9000` gets partially overwritten by stage2's
own tail end (found this the hard way: the kernel's `LFB:` cprintf started
reporting a different wrong value on every boot, because it was reading
whatever stage2 code/data now happened to occupy that address).
`kernel/dev/vga.c`'s `vga_init()` has to agree on `0x30000` too, since it
independently re-reads mode fields from the same block.
Splash/progress bar/boot menu rendering is a small standalone pixel
renderer in `stage2main.c` (not the kernel's `vga.c` — paging isn't set up
yet, so the LFB physical address from the VBE block is used as a plain
pointer) reusing the kernel's own font data (`include/font.h`) directly.
It fills the framebuffer with PSOGM's purple (matching
`kernel/core/console.c`'s panic screen) and draws "SCHOS" centered, then:
- **Real-time loading bar**: `readsect()` (in `stage2main.c`, not the
kernel) calls `progress_tick()` after every 512-byte disk read, updating
an actual pixel-filled bar against the real total byte count — first a
small one while scanning for the multiboot header, then a second,
properly-sized one once the header's found and the true kernel size is
known, labeled "Loading kernel...". Not simulated/timed — it reflects
real bytes read off disk.
- **F8 boot menu**: `wait_for_f8()` busy-polls the keyboard controller
(`inb(0x64)`/`inb(0x60)`, raw scancodes — real-mode BIOS keyboard calls
aren't available once we're in protected mode) for about a second after
the splash appears. F8 (scancode `0x42`) drops into `boot_menu()`: three
options (boot normal, boot verbose, reboot to firmware setup) navigated
with up/down arrows and Enter, plus a "detected drives" panel from
`ata_present()` — a standard floating-bus presence check (select the
drive, write a pattern to two registers, read it back) on the primary
IDE channel's master/slave (the only channel this project ever actually
puts drives on; QEMU's IDE emulation answers the same test
"successfully" on the unpopulated secondary channel too, so that channel
isn't shown — it'd just be confidently wrong). "Reboot to firmware
setup" pulses the keyboard controller's reset line
(`outb(0x64, 0xFE)`, same technique as `kernel/sys/sys_proc.c`'s
`sys_reboot()`); a real bootloader can't jump into BIOS/UEFI setup
directly, so this is the closest honest equivalent — it gives the user
a fresh POST during which their firmware's own setup-menu key works.
- **Verbose boot**: picking "Boot SchOS (verbose)" writes a flag byte to
`BOOT_VERBOSE_FLAG_ADDR` (`VBE_INFO_ADDR + 0x100`, i.e. just past the VBE
block) before jumping to the kernel. `kernel/core/main.c`'s `kmain()`
reads it into a global `boot_verbose` as its very first action and, if
set, prints a line after every subsystem init call
(`uartearlyinit`, `kinit1`, ..., `sinit`) — real per-step boot logging,
not a canned message.
Two bugs worth remembering if this code gets touched again:
- **Keyboard extended-scancode handling**: `0xE0` (the prefix byte for
arrow keys and friends) also has bit 7 set, the same bit used to detect
key-*release* ("break") codes. A naive "skip anything with bit 7 set,
then check for 0xE0" loop skips the 0xE0 byte itself before ever testing
for it, so extended keys silently never register. The 0xE0 check has to
come before the break-code check, not after.
- **The kernel entry call is not a real C function call.** Both the
original single-stage `bootmain()` and the new `stage2main()` do
`((void(*)(uint32))entry)(fbaddr)` — normal cdecl, which pushes `fbaddr`
on the stack. But `kernel/arch/x86_64/entry.S`'s `mboot_entry` reads its
framebuffer argument out of **EAX** (a multiboot-style raw-register
handoff, then `mov %eax,%ebp` / `mov %ebp,%ebx` / `mov %rbx,%rdi` carries
it through the 32→64-bit transition into `kmain()`'s first argument).
The cdecl call never puts anything in EAX — it "worked" for years purely
because simple bootmain.c code happened, as a side effect of its own
codegen, to leave the right value sitting in EAX right up until the
`call` instruction. Adding the splash/menu/progress-bar code changed
that codegen and broke it silently: the kernel booted fine and reported
a plausible-looking but wrong `LFB:` address (garbage left over in EAX
from unrelated prior code, different on every boot). Fixed by pinning
the argument down explicitly with `asm volatile("jmp *%0" : : "r"(entry),
"a"(fbaddr))` instead of hoping the compiler's register allocation
cooperates. If this code moves around again, re-verify with a screendump
taken *immediately* before the jump compared against the kernel's own
`LFB:` log line — they silently diverging is exactly this bug.
## 5. `*at()` syscalls ignored/rejected real dirfds
`ls -l`/`ls -lR` (and, by the same code path, `find`/`du`/anything using
toybox's `lib/dirtree.c`) would hang on `ls -l` and either hang or silently
misbehave on recursive listings. Root cause: `kernel/sys/sys_fs.c`'s
`*at()` syscalls (`openat`, `fstatat`, `unlinkat`, `renameat`, `linkat`,
`symlinkat`, `mkdirat`, `mknodat`, `readlinkat`, `faccessat`) only handled
two cases for their `dirfd` argument — `AT_FDCWD` (resolve relative to
`proc->cwd`) or an absolute path (resolve from root, ignoring `dirfd`
entirely). A **real** dirfd with a **relative** path — e.g.
`fstatat(open("/bin", O_RDONLY), "ls", &st, ...)`, the standard, efficient
way any directory-tree walker avoids rebuilding full path strings for every
entry — was either rejected outright (`fstatat`, `unlinkat`, ...) or
silently ignored in favor of `proc->cwd` (`openat`, which never checked at
all — worse than rejecting, since it can silently resolve the *wrong*
file if cwd happens to contain a same-named entry).
Fixed by adding a real dirfd-relative path resolver: `kernel/fs/fs.c`'s
`namex()` was refactored into `namex_from(struct inode *start, ...)`,
parameterizing what used to be a hardcoded `idup(proc->cwd)`, with two new
exported entry points, `nameifrom()`/`nameiparentfrom()`. `sys_fs.c` gained
`at_dir(dirfd)` (resolves `AT_FDCWD` or a real fd to a starting inode) and
`at_namei()`/`at_nameiparent()` built on top of it, plus a `createat()`
(parameterizing `create()`'s hardcoded `nameiparent()` call the same way).
Every `*at()` syscall now uses these instead of its own inline
absolute-path/`AT_FDCWD`-only check.
**Known follow-on issue, not yet root-caused**: heavy `ls`/`ls -lR` use
across many commands in one long-running shell session can eventually make
a *later*, unrelated directory hang (reproduced with `ls -lR /var` after
~10 prior commands in the same session; the identical command succeeds
immediately on a fresh boot). Not an inode-table or file-table exhaustion
— both `iget()` and `filealloc()` fail cleanly (panic / return 0
respectively) when full, and neither fired. Smells like a lock or
reference-count leak somewhere in the *at() paths above, or in toybox's
own fd handling, that only manifests after enough accumulated opens/closes.
Next step: instrument `itable`/`ftable` occupancy counts and watch them
across a long `ls`-heavy session to see what's actually climbing.
## 6. Boot speed
Two cheap, real wins in `boot/bios/stage2main.c`, no functional changes:
- `progress_tick()` (called once per 512-byte disk sector) used to trigger
a full progress-bar redraw — border plus fill, a few thousand pixel
writes — on *every* sector. For the ~330KB kernel image that's ~650 full
redraws (the multiboot-header-scan phase adds another ~32) for a bar
that only has 101 visually distinct states (0-100%). Split into
`progress_border()` (drawn once, in `progress_start()`, since the border
never changes) and `progress_fill()`, and `progress_tick()` now skips the
redraw entirely unless the percentage actually changed. This was the
single biggest chunk of the bootloader's own wall-clock time.
- `fill_rect()` (the full-screen splash fill, 480,000 pixels at 800x600,
run at least twice per boot) used to call `draw_pixel()` per pixel, which
recomputes `g_fb + y*stride + x*4` from scratch every time. Now computes
each row's base pointer once and indexes into it.
- The F8 boot-menu wait was shortened from ~1s (`5000000` busy-loop
iterations) to ~0.6s (`3000000`) — flat overhead on every boot whether
or not the menu is wanted, so it's the most direct lever available
without giving up the feature entirely.
## 7. `*at()` syscalls, real signal delivery, and mksh job control
> **Resolved since this was written.** Process groups and sessions are real
> (`pgid`/`sid` in `struct proc`, `setpgid`/`setsid`/`getpgid`/`getsid`, and
> signal delivery to a whole group), which was the first of the three suspects
> listed at the end. mksh's job control works. The investigation is kept in full
> because the signal-delivery mechanism described below is still exactly how the
> kernel does it, and because the reasoning about *why SIGCHLD was left enabled
> while broken* is the transferable part.
`ls -l`/`ls -la` hanging outright turned out to be the same root cause
described in §5 above (dirfd-relative path resolution) — fixed
and verified extensively (plain `ls`, `ls -l`, `ls -la`, `ls -lR`, piped
output, all confirmed working repeatedly).
A *second*, deeper issue was found investigating a follow-up report
(`cd` with no `HOME` set, then `cd var`, then `ls` hanging): mksh's
interactive job control (`ports/mksh (mksh R59c source): jobs.c`) blocks in
`sigsuspend()` waiting for `SIGCHLD`, expecting its own handler
(`j_sigchld()`) to run, reap the child via `waitpid(-1, &status, WNOHANG)`,
and update its internal job-state tracking. This kernel had **no real
signal delivery at all** — `sys_rt_sigaction()` was a no-op stub, so no
handler was ever invoked for anything, and `sigsuspend()` just slept out
one tick and returned `-EINTR` unconditionally. mksh's wait loop had no
way to ever learn the child had exited, hanging forever.
Real signal delivery was implemented from scratch:
- `struct proc` (`include/proc.h`) gained `sig_handler[32]`/
`sig_restorer[32]` (populated by `sys_rt_sigaction()`,
`kernel/sys/sys_sig.c`), `saved_sigtf` (stashed pre-signal trapframe),
and `in_signal` (re-entrancy guard). `allocproc()` zeroes these for a
freshly claimed ptable slot (they weren't before — a reused slot
silently inherited a previous, unrelated process's handler addresses,
which crashed the *next* process to occupy that slot the moment
anything tried to delivver a signal to it). `fork()` copies handler/
restorer/mask from parent to child (POSIX semantics) but not pending
signals.
- `kernel/proc/proc.c`'s `exit()` sets `SIGCHLD` pending on the parent.
- `kernel/sys/syscall.c`'s `syscall()` calls `deliver_signals(tf)` in its
tail, right before returning to user mode: if there's a pending,
unblocked signal with a registered handler, it redirects the trapframe
to jump to the handler (`rdi`=signal number per SysV ABI, `rip`=handler)
instead of resuming the interrupted code, having first stashed the
original trapframe in `proc->saved_sigtf`.
- The return address is **not** `sig_restorer[sig]` — checked, `SA_RESTORER`
is not defined anywhere in this musl build, so `sigaction()` never fills
in a restorer; it's always 0, and pushing 0 as a return address crashes
with `rip=0` the instant the handler function returns. Instead,
`deliver_signals()` injects a tiny 7-byte trampoline (`mov eax,15;
syscall` — `rt_sigreturn`) directly onto the user's own stack, below the
current `rsp` (clear of the SysV red zone), and points the return
address at that — the standard technique for when the C library doesn't
supply its own restorer.
- `sys_rt_sigreturn()` restores `*proc->tf` from `proc->saved_sigtf`,
clearing `in_signal`.
**A real bug found and fixed along the way**: the first version of
`sys_rt_sigaction()` translated the user's `struct sigaction` pointer
*once* via `uva2ka()` and then did a bulk `memset`+copy across the whole
~152-byte structure. `uva2ka()` only guarantees the *page containing that
address* is mapped, not everything up to some further offset — when the
user's pointer landed close enough to a page boundary, the write ran past
the mapped page into whatever physical page happened to be next in kernel
address space (P2V is a flat offset, so "next page" there has nothing to
do with the user's actual next page), corrupting unrelated kernel memory.
This crashed every freshly `exec`'d `/bin/sh` almost immediately, since
mksh calls `sigaction()` during its own startup. Fixed by only touching
the two individual 8-byte fields actually needed (`sa_handler` at offset
0, `sa_restorer` at offset 144), each through its own `uva2ka()` call.
`sys_waitpid()`/`sys_wait4()` (`kernel/sys/sys_proc.c`) were also rewritten
on top of a new `waitpid(pid, status, options)` in `kernel/proc/proc.c`
(refactored from the old any-child-only, always-blocking `wait()`, which
is now just `waitpid(-1, 0, 0)`): a specific `pid` (not just "any child"),
`WNOHANG` (return 0 immediately instead of blocking), and — importantly —
a genuinely optional status pointer. The old code ran a NULL status
pointer through `argptr()`, which rejects NULL/unmapped addresses outright
(page 0 is deliberately unmapped) — but `waitpid(pid, NULL, WNOHANG)` is
completely standard, and is exactly what mksh's job control does.
**Status: signal delivery mechanically works** — verified directly (a
debug build logged the handler being invoked with the correct trapframe
and `sys_rt_sigreturn()` restoring cleanly, no crash, repeatable). **mksh's
job control still doesn't reliably progress afterward**, including for
plain non-piped foreground commands, not just the original piped-into-head
case. Root cause not yet isolated. Leading suspects, in order of
likelihood:
- `sys_setpgid()`/`sys_getpgid()` are no-op stubs (`kernel/sys/sys_proc.c`)
— `j_sigchld()` (`ports/mksh (mksh R59c source): jobs.c:1344-1348`) has an early-out
keyed on `j->ppid == procpid`, which may never resolve the way mksh
expects without real process-group tracking.
- `getrusage(RUSAGE_CHILDREN, ...)` — called at the top of `j_sigchld()` —
isn't verified to behave sanely under this kernel's implementation
(`kernel/sys/sys_proc.c`'s `sys_getrusage()` only fills in `ru_utime`
from the *calling* process's own tick count, not accumulated child
time; `RUSAGE_CHILDREN` isn't distinguished from `RUSAGE_SELF` at all).
- Possibly something in how `j_sigchld()` matches a reaped `pid` against
its own `job_list` (`ports/mksh (mksh R59c source): jobs.c:1372-1385`) silently fails
to find the job for reasons not yet traced.
This was left here deliberately rather than continuing to iterate blind:
SIGCHLD delivery is left **enabled** (`kernel/proc/proc.c`'s `exit()`)
since disabling it guarantees zero chance of mksh's wait loop ever exiting,
whereas leaving it on at least gives the cases that don't hit the
still-unknown remaining gap a chance to work. Next session should start
by instrumenting `j_sigchld()`'s own control flow (or strace-equivalent
logging in the kernel around `sys_setpgid`/`sys_getrusage`/`sys_waitpid`
calls specifically while reproducing `cd; cd var; ls` or even simpler
`ls -l /bin` followed by a second command) rather than re-deriving the
signal-delivery mechanism, which is now confirmed solid.
---
## 8. Two ABI assumptions that only GCC was keeping true
Found when the build moved from GCC-in-WSL to clang-on-Windows. Neither is a
clang bug; both are places where the code relied on GCC doing something it
never promised, and both had been latent for years.
### `%fs:0` was being used as scratch — it is the TCB self-pointer
`syscall_entry` (`kernel/arch/x86_64/trapasm.S`) stashed `%rax` at `%fs:(0)`
while it fetched `proc` to find the kernel stack. The comment called it a
"per-CPU scratch slot". It is not: the x86-64 TLS ABI reserves the word *at*
the thread pointer to hold the thread pointer, and a compiler may read it to
materialise the TP before applying a variable's offset.
GCC compiles `__thread` access to a direct `%fs:-8` segment reference and never
reads that word, so the clobber was invisible. clang at `-O0` emits
`movq %fs:0, %rax; addq $-8, %rax; (%rax)` — reading back the syscall number
this code had just stored. The first process's first syscall is
`exec` (7), so `proc` resolved to address `7 - 8 = -1` and the kernel
page-faulted recursively until the stack was gone.
Fixed by moving the scratch to `%fs:(8)`. Positive offsets are safe: the TLS
block lives at negative offsets and `seginit` places the thread pointer 2048
bytes into a 4096-byte page.
**How it was found** is the part worth copying next time. Three rounds of
`cprintf` narrowed it to "the word is right at `mpmain`, wrong by the first
trap" and no further — the instrumentation perturbed the timing enough to move
the symptom. A hardware watchpoint through QEMU's gdb stub named the
instruction on the first run:
```
qemu ... -S -gdb tcp::26000
gdb -batch -x tcp.gdb # hbreak mpmain; watch *(unsigned long*)kernel_fs_base
```
`seginit` also now writes the self-pointer, which it never did. That is ABI
-correct regardless and costs one store per CPU.
### `_start` read incoming registers with `register ... asm("rdi")`
`lib/glue.c` captured `argc`/`argv` from `%rdi`/`%rsi` using GCC local register
variables. Both compilers only guarantee that binding for inline-asm operands;
using it to pick up an incoming argument is undefined. clang ignored it and
read two uninitialised stack slots:
```asm
movl -0x4(%rbp), %eax # should have been %edi
movq -0x10(%rbp), %rax # should have been %rsi
```
Every program still ran. `init`, `shmtest` and `socktest` take no arguments and
passed their entire suites; the only symptom in the whole system was
`inputtest -n 6` ignoring its limit. Moved to `lib/start.S`, where reading
incoming registers is expressible and correct — `exec` passes argc/argv in the
SysV argument registers already, so `_start` calls `main` without touching
them.
---
## 9. The boot looked like a hobby OS, and two things were why
Not a bug — a design problem, fixed by looking at a screenshot instead of the
code.
**The display was the kernel log.** Boot printed `pci: 6 devices` and
`vga: display controller 1234:1111, aperture fd000000 (16 MB)` to the screen
the user is looking at. That single fact is most of what separates a hobby
project from a product; every shipping system puts that behind a flag.
`cons_serial_only` already existed for panic, so the fix was to start it *set*.
Everything still goes out the UART — which is where `tools/windows/bootcheck.ps1` reads
it from, so all five suites kept passing untouched — and `console_takeover()`
clears it on the first `exec()`, the moment userspace genuinely starts.
`vga_init` no longer clears the screen, so the splash survives the whole of
boot rather than being wiped a few hundred milliseconds in.
**There were three unrelated designs.** The splash was deep purple with gold
text, deliberately matching the *panic* screen. The console was cream with
near-black text. So a healthy boot was dressed as a crash, then abruptly
became an amber terminal. `include/theme.h` is now the one palette, and the
rule is: **normal states are calm and dark; only failure is loud.** PSOGM keeps
the purple entirely to itself, which is what makes it mean something.
The splash copy went from "SchOS Boot Manager / version 1.0 / Memory: 1023 MB
/ Press F8 for boot menu..." to the product name, a rule in the accent colour,
and the one hint that is actionable — moved to the bottom, where a hint
belongs. No shipping OS names its bootloader on the splash.
### Three latent bugs it flushed out
Retheming meant `stage2main.c` included `types.h` properly for the first time,
and the 32-bit build stopped compiling:
- `_Static_assert(sizeof(uint64) == 8)` fired. `uint64` was `unsigned long`,
which is **4 bytes** in the `-m32` loader — the file even carried a local
`e820_u64` workaround with a comment saying so. Now conditioned on
`__SIZEOF_LONG__` rather than on the platform, which covers LP64, LLP64 *and*
ILP32 correctly.
- `x86.h`'s `stosq` and `xchg` use `rep stosq` and `xchgq`, which do not exist
in 32-bit mode. They only ever compiled because `uint64` was silently
narrowed. Now behind `#ifdef __x86_64__`.
- The progress bar computed `(uint64)done * 100 / total`, a 64-bit divide that
needs libgcc's `__udivdi3` — which a freestanding loader does not link. Now
scaled to KiB and done in 32 bits.
All three had been sitting in a bootloader that worked fine, waiting for
someone to make `uint64` mean what it says.
---
## 10. A full-screen program typed the whole session into the shell
Keys reached `/dev/input/event0` **and** the tty line discipline at the same
time. That is correct for two consumers of one keyboard and completely wrong
for a full-screen program: while it had the display, the shell behind it was
quietly accumulating every keystroke, and the first Enter afterwards ran
whatever had piled up. Arrow keys arrived as CP437 glyphs on the console for
good measure.
The fix is the standard one and it is a *primitive*, not a patch: an exclusive
grab, `EVIOCGRAB`, with Linux's ioctl number. While a device is grabbed its
events still flow to `/dev/input/eventN` but both keyboard drivers withhold the
decoded character from the console.
Three details that matter more than the feature:
- **The grab is released by the devsw `close` hook**, not by the program. A
program that grabs the keyboard and then crashes would otherwise leave the
console permanently deaf, with no way to type the command that fixes it.
- **Modifier state is still tracked while grabbed** (`kbdgetc` keeps updating
shift/caps and only skips the character). Otherwise releasing the grab would
hand the line discipline a stuck Shift.
- **The USB path updates `prev[]` before returning**, or the next report would
look like every held key had just been pressed again.
It is also exactly what Schcom will need, which is why it lives in the input
layer rather than in any one client.
### The near-miss worth remembering
The first attempt appeared to work: the harness passed, every assertion green.
It had not worked at all — the ioctl call was never inserted into the client
source, because a scripted string replacement silently failed to match. The
`grep` that should have caught it printed three hits (a comment, a `#define`,
and the *release* call) and was read as confirmation without counting what
they were.
What actually caught it was reading the transcript rather than the verdict:
four bare `#` prompts after quitting, which is precisely the symptom the change
was supposed to remove. `input_grab()` now announces itself, so "did the grab
happen" is never again a thing anyone has to infer.
---
## 11. `ps` and `top`, and why they never worked
/proc was a set of character devices: init ran `mknod("/proc/uptime", 6, 2)` and
friends, so every file was a real inode on the root filesystem with a
major/minor behind it. That is fine for a fixed list and cannot work at all for
the thing /proc is mostly for -- one directory per process. You cannot mknod a
directory for a process that does not exist yet and will be gone in a minute.
`ps` calls `dirtree_flagread("/proc")`, got a listing with no numbers in it, and
reported nothing. It was never a `ps` problem.
So /proc is now a real filesystem behind the same `struct fs_ops` vtable SchFS
uses (`kernel/fs/procfs.c`, `PROCFS_DEV`), mounted by init with
`mount("proc", "/proc", "proc", ...)`. Nothing is stored: directory listings
are generated from the process table when asked, file contents when read.
Three things worth keeping in mind:
- **The split is deliberate.** `kernel/fs/procfs.c` owns the *namespace* (what
exists); `kernel/dev/procfs.c` keeps the *generators* (what the global files
say). They change for different reasons, so `proc_gen_global()` is the only
thing that crosses between them.
- **procfs never touches ptable.** `proc_snapshot()` copies what is needed out
from under the lock, so a process that exits between the readdir that listed
it and the open that reads it produces a stale line rather than a
use-after-free.
- **Sizes are reported as zero**, exactly as Linux does. A synthetic file has
no length until it is generated, and generating every one on every `stat()`
to fill in a number almost nothing reads would be absurd.
### What the suite caught
Marking every procfs inode read-only broke `echo s > /proc/sysrq`, and three
USB checks failed on it within a minute of the change -- they drive the disk
through sysrq. /proc is read-only *except* sysrq, which is the one deliberate
write target, and the code now says so in one place.
### Still not right
`ps -o comm` prints `[init]` rather than `init`. That field comes from
`/proc/<pid>/exe`, which Linux makes a **symlink** and toybox `readlink()`s;
ours is a regular file, so the readlink fails and toybox brackets the name as
it would for a kernel thread. Default `ps` and `top` output are unaffected.
Fixing it needs symlink support in procfs, which belongs with the rest of the
VFS work rather than as a one-off.
---
## 12. A test program crashed at random, and had been typing its own launch command
Symptom: a full-screen framebuffer program, launched with arguments, worked
four times and then died with
pid 9 <prog>: trap 14 err 4 on cpu 0 rip 0x2269be addr 0x18--kill proc
`err 4` is a user-mode read of a non-present page and `addr 0x18` is a null
pointer at offset 24. `llvm-nm` put `rip` inside the routine that opens the
program's overlay view, which dereferences a pointer the program only fills in
once a level is loaded -- at offset 0x18 on x86-64, because the three 64-bit
words of a base struct sit in front of it. So the program had been asked to open
that view before there was anything to show.
**The technique is the transferable part:** the fault address alone said
"null pointer plus 24", which is nearly useless; `llvm-nm` on the faulting `rip`
turned it into a function name, and the offset then identified the exact field.
Reach for that before adding printfs to a program you did not write.
The cause was ours, and it was visible in the log the whole time: every run
printed a burst of garbage bytes *before* the grab was taken, and the amount
differed each run.
**`/dev/input/eventN` had no notion of being opened.** Events were queued from
boot onwards whether or not anything was listening, into one 256-entry ring
with one read cursor. When the program opened the device it inherited the entire
backlog -- including the keystrokes of the command line that had launched it,
which the shell had already consumed -- and replayed them as input. Whether that
crashed depended on which keys happened to be sitting there, which is exactly
the "sometimes" in the report.
Fixed by giving `struct devsw` an `open` hook to match `close`, and having
`/dev/input`:
- drop the backlog on open (`r = w`) -- a reader starts from *now*;
- only queue events while `opens > 0`, which is what Linux does: an evdev
device with no clients does not accumulate.
Linux gets both for free by giving every open its own client queue. One shared
ring plus "start from now" is the same answer for a single reader, which is the
only case that exists today; per-client queues become necessary the moment
Schcom and a client both want the keyboard.
### Also fixed
`cd` into a file reported "Operation not permitted". It was right to fail, but
`sys_chdir` returned a bare `-1`, so the shell printed whatever errno happened
to be lying around. It returns `-ENOTDIR` now, and `-ENOENT`/`-EFAULT` on the
other two failure paths -- "Not a directory" sends you to the right place,
"Operation not permitted" sends you looking at permissions.

146
docs/history/moving-to-linux.md Normal file
View file

@ -0,0 +1,146 @@
<!-- SPDX-License-Identifier: Apache-2.0 -->
# Moving the project to a Linux machine
**This move happened.** The tree lives on Fedora now, and the checkout is at
`/home/nexoniarz/Dokumenty/Projects/SchOS` rather than the path written below.
It is kept in `history/` rather than `guides/` because most of it is a
one-time procedure that has been performed — but three parts recur and are
worth knowing before the next machine:
- **what is and is not in the repo** (§1), which decides what a `git clone`
gives you versus an `rsync`
- **the memory key is derived from the absolute path** (§2), so it differs per
machine and `memory-sync.sh` exists for exactly that
- **the two things that behave differently on Linux** (§4) — `make ports` works,
and the SchFS round-trip test actually runs, because it needs symlinks and
hard links that Git Bash cannot create
Nothing here is Fedora-specific except the package names. For building on any
host, see [../guides/building.md](../guides/building.md).
## 1. Copy the repo
Copy the whole directory, `.git` included. Everything that matters is committed
— 17 commits, no remote — and that includes the project memory, which lives in
`.claude/memory/` precisely so it survives this.
```sh
# whatever transport suits: a USB stick, rsync, scp
rsync -a /path/to/SchOS/ /home/nexoniarz/Dokumenty/Projects/SchOS/
cd /home/nexoniarz/Dokumenty/Projects/SchOS
git log --oneline | head -3 # should show the same commits
git status # should be clean
```
### What is not in the repo
`build/`, `sysroot/` and `dist/` are all gitignored, so `rsync -a` of the
working tree carries them but a `git clone` would not. Two are worth thinking
about:
- **`dist/`** holds the upstream tarballs — musl, toybox, mksh, the DOOM WADs,
`usb.ids`. 61 MB of other people's release artefacts. Copy it to save the
download, or let the build fetch them again; `make ports` calls `curl` and
verifies each against a known SHA-256, so a re-download is safe either way.
Without a network connection, copy it.
- **`sysroot/`** is assembled by the build, so it does not need copying — and
on this machine it should not be. Its `bin/` holds musl, toybox and mksh,
staged once because Windows cannot build them. Linux can, so let `make ports`
produce them rather than carrying stale binaries across: the staged copies
predate the rename, and `sysroot/bin/sh` still contains `__schunix_tp`.
- **`build/`** is pure output. Delete it and forget it.
## 2. Restore the memory
Claude Code keeps per-project memory under a key derived from the project's
absolute path, so it differs per machine:
| | |
|---|---|
| `D:\DEV\SchOS` | `D--DEV-SchOS` |
| `/home/nexoniarz/Dokumenty/Projects/SchOS` | `-home-nexoniarz-Dokumenty-Projects-SchOS` |
One command puts it where this machine looks:
```sh
tools/posix/memory-sync.sh status # shows both sides, changes nothing
tools/posix/memory-sync.sh import # repo -> ~/.claude/projects/<key>/memory/
```
That writes to
`/home/nexoniarz/.claude/projects/-home-nexoniarz-Documents-SchOS/memory/`.
After a session that learned something, `export` and commit, so the knowledge
travels back:
```sh
tools/posix/memory-sync.sh export && git add -A && git commit -m "memory: ..."
```
## 3. Install the toolchain
```sh
tools/posix/setup-fedora.sh --check # report what is missing
tools/posix/setup-fedora.sh # install it
```
That covers clang, lld, llvm, make, gcc, perl, python3, qemu-system-x86,
edk2-ovmf, xorriso and mtools. See [../guides/building.md](../guides/building.md) for what each is
for and for the other supported hosts.
**PowerShell is the one thing dnf cannot supply.** `tools/windows/bootcheck.ps1`
is the boot test suite, and `pwsh` runs it on Linux perfectly well — there is
simply no Fedora package. The script prints the two ways to get it. Nothing
else needs it: `make`, `make schfs-check` and `make qemu` all work without.
## 4. Build and check
```sh
make # kernel + root filesystem
make schfs-check # B-tree, object and image suites -- all host-native
make qemu # boot it
```
Two things will behave differently here, both improvements:
- **`make ports` works.** musl, toybox and mksh build from source. On Windows
the build skips them and uses whatever is staged.
- **The SchFS round-trip test runs.** It needs symlinks and hard links, which
Git Bash cannot create, so on Windows it skips with a message. This is the
first machine that can actually exercise that path — worth running early,
since it is the one part of SchFS never yet tested.
## 5. KVM
A Ryzen 7000 has SVM, so QEMU can run at native speed instead of emulating:
```sh
ls -l /dev/kvm # should exist
sudo usermod -aG kvm $(id -un) # if it exists but is not readable
```
Then add `-enable-kvm` to a QEMU command line. It is deliberately **not** the
default in the Makefile: the test suite's timings were established under TCG,
and changing the execution engine underneath them is a variable worth
introducing on purpose rather than by accident.
## 6. UEFI
`make qemu-uefi` boots the UEFI path under OVMF. The Makefile searches the
usual locations for the firmware and Fedora's `edk2-ovmf`
(`/usr/share/edk2/ovmf/OVMF_CODE.fd`) is among them.
Note the open bug before spending time on it: under OVMF the kernel reports
`e820: only 8 MB usable`, because `e820_init` takes the usable run *containing
the kernel* and OVMF puts an ACPI NVS region at 0x800000, genuinely fragmenting
it. The loader's map is correct; the kernel's one-contiguous-run assumption is
BIOS-shaped. The fix is to free each usable region rather than one range, which
touches `kalloc`.
## The hardware, for reference
Fedora 44, Ryzen 5 7000-series, RTX 4050, 16 GB DDR5. Nothing in the build cares
about the GPU — QEMU runs headless for tests and uses its own emulated display
otherwise. The Nvidia driver matters only if you want a QEMU window, and the
`nouveau`/proprietary choice makes no difference to SchOS either way.

370
docs/history/schfs.md Normal file
View file

@ -0,0 +1,370 @@
# SchFS — how it was built
The filesystem as it is now is [../architecture/schfs.md](../architecture/schfs.md).
This is the record of building it: the phase plan, what each phase actually
produced, and the bugs that cost real time.
**All five phases are complete.** The plan is kept because the reasoning in it
is still the reasoning behind the format, and because the ordering argument —
*do not start phase 2 before phase 1 is solid, the B-tree is the load-bearing
piece* — is the transferable part.
---
## The phases, as they were planned and as they went
Each phase leaves a bootable system. Do not start phase 2 before phase 1 is
solid — the B-tree is the load-bearing piece and everything else assumes it.
**Phase 1 — the B-tree, in userland first.** Write the CoW B-tree as a plain C
library with a block-device abstraction, and test it on a *host* file with a
fuzz/torture harness (random insert/delete/lookup, simulated crash at every
write point, verify the tree always mounts). This is the one component where a
subtle bug is unfixable later, and testing it in the kernel is far harder. It is
also portable straight into `tools/mkfs`.
**Phase 2 — read-only mount. DONE.** New magic, uberblock scan, tree walk, inode
and dirent key types. `tools/mkschfs` writes images; the kernel only reads them.
The system boots from a ramdisk root (see [../guides/layout.md](../guides/layout.md) and the ramdisk work)
so a broken disk cannot stop you from booting.
Went further than planned in two places, both cheaply: data blocks carry
checksums too (a `CSUM` key type, as btrfs does — without it "end-to-end
checksums" would cover only metadata, which is the part least likely to rot),
and extents landed early because reserving a file's blocks in one go was
simpler than not doing it.
Not yet done: SchFS files are not reachable through `open(2)`. The mount is
observable through `/proc/schfs` and through the kernel API in `schfs_int.h`,
but wiring it into `namei`/`readi` means the VFS work that phase 3 needs anyway.
**Phase 3 — writes, transaction groups. DONE.** Every `schfs_k*` call that
changes anything is one transaction group, committed by writing a new uberblock
before it returns. The full write API — create, link, unlink, rename, write at
an arbitrary offset, truncate — lives in `objw.c` and is exercised by
`tools/schfs_objtest.c`.
Two things went differently from the plan above, both for reasons worth
recording.
*Free space is not tracked as tree keys.* Allocating from a free map that lives
in the tree mutates the tree, which allocates, which changes the map; btrfs
resolves that with a fixpoint loop and ZFS with log-structured space maps. SchFS
sidesteps it: the committed tree already says exactly which blocks are live, so
the map is **derived** by walking the tree once at mount and held only in RAM.
It cannot disagree with the filesystem, because it is computed from it, and a
crash raises no question at all. The cost is mount time proportional to metadata
size, and a ceiling on volume size set by how much bitmap fits in memory. If
mount time ever hurts, the map can be cached on disk purely as a hint,
validated against the generation and discarded on a mismatch — an addition, not
a format change. See `kernel/fs/schfs/space.c`.
*Data blocks needed a generation stamp too.* The B-tree already rewrites its own
nodes in place within a group, since a block that no committed root can reach is
nobody's fallback. Data blocks had no such marker, so every hundred-byte write
copied a whole 4 KB block and split an extent record around it. `schfs_extent`
gained a 64-bit `gen`, and the same rule now applies to file data.
**Phase 3½ — reachable through `open(2)`. DONE.** SchFS is a mounted filesystem:
```
mkdir /mnt
mount -t schfs /dev/hdb /mnt
ls /mnt ; echo hi > /mnt/tmp/w ; cat /mnt/tmp/w ; mkdir /mnt/tmp/d
```
The VFS grew a real dispatch layer rather than a set of `if (dev == SCHFS_DEV)`
checks. `struct fs_ops` (include/file.h) is one table per filesystem, chosen by
`iget()` from the device number and hung off the inode; v1's implementations
keep their bodies and are simply named `v1_*`. Adding a third filesystem is a
new table and nothing else. `kernel/fs/schfs/vfs.c` is SchFS's, and it is pure
translation — object ids in, inodes out — because all the filesystem logic
already lives below it.
Three things had to change on the way, none of them SchFS's fault:
- `struct inode.inum` and `.size` were 32-bit, capping objects at 2³² and files
at 4 GiB on a filesystem that promises far more. Both are `uint64` now, as are
`readi`/`writei` offsets and `struct file.off`.
- `struct dirent` carries a 16-bit inum and a 14-byte name, so it cannot
represent a SchFS entry at all. `getdents64` is now an fs_op: each filesystem
emits the record stream itself, and `f->off` is whatever position it needs to
resume. v1 walks its hash buckets; SchFS walks a key range and synthesises
"." and "..", which it does not store.
- Mounting is a substitution made during path resolution (`mount_add` /
`mount_cross` in `fs.c`). Nothing on either disk records it. The reverse
direction matters as much as the forward one: `..` at a mounted root has to
leave the mount, and the filesystem underneath has no idea it is mounted
anywhere.
Three long-standing bugs surfaced the moment real commands ran against it, all
in the shared VFS rather than in SchFS:
- `include/fcntl.h` used xv6's open flags while every program in the userland
sends Linux's. `O_APPEND` (0x400) landed on `O_TRUNC`, so `cmd >> file`
*truncated* it; `O_TRUNC` (0x200) landed on `O_CREATE`, so `cmd > file` left
whatever was past the new end in place. Nothing about either pointed at a
header.
- `sys_openat` silently dropped `O_TRUNC` and `O_APPEND` while `sys_open`
honoured both, so the same redirection behaved differently depending on which
entry point libc happened to call.
- `rmdir` had no Linux syscall number, so every Linux-ABI program got ENOSYS.
**Phase 3¾ — durability, and the cost of it. DONE.** Three things that the
earlier phases had assumed rather than arranged.
*The commit barrier.* Copy-on-write promises that a crash lands on either the
old tree or the new one, but only if the blocks a root points at reach the
medium *before* the block naming that root. Nothing enforced that: `bwrite`
order is not platter order on any drive with a write cache. `struct schfs_bdev`
gained a `bflush` hook, `schfs_vol_commit` calls it either side of the uberblock
write, and the kernel implements it as "drain this device's dirty buffers, then
ATA FLUSH CACHE" (`ideflush`, and SCSI SYNCHRONIZE CACHE for a USB stick).
*Falling back through the ring.* Mount took the highest-generation uberblock
that validated, checked the tree it named, and gave up if that failed — with up
to 127 older, entirely intact roots sitting on the disk. It now walks the ring
newest-first and adopts the first root it can actually read, counting what it
skipped into `vol->recovered` so the rollback is reported rather than silent.
*Deferred writes, and brackets.* The B-tree seals and writes a node on every
record it inserts, so a leaf collecting one file's data checksums went to the
disk once per 4 KB of that file. The kernel's block hook marks buffers dirty
instead (`bdirty`), and the commit barrier is what makes that safe. On its own
that changes little, because a commit *contains* a flush and every `schfs_k*`
call was one commit — so `schfs_ktx_open`/`schfs_ktx_close` bracket a run of
calls into one group, and `schfs_vfs_write` wraps a whole `write(2)` in one.
Measured over 1 MB of file data, in block writes reaching the device:
| | before | after |
|---|---|---|
| committing every 4 KB | 1947 | 1410 |
| one commit for the megabyte | 1798 | **267** |
267 device writes for 256 blocks of data. Note that the two changes are worth
almost nothing separately — deferral cannot help while every commit flushes,
and batching cannot help while every write goes straight to the disk — which is
why the first attempt at this measured an 8% difference and concluded batching
was not the problem.
**Phase 3⅞ — where the time actually went. DONE.** The filesystem was correct
and durable and the system still felt slow: `ls /bin` took the better part of a
second. Two causes, neither of them the disk.
Measured first, through `/proc/fsstat` (counters added for exactly this). The
decisive number was **398,999 block reads against 445 actual disk misses** —
99.9% of all block reads were re-reading a block already in the buffer cache. So
the disk was never the problem, and every plan that started with "cache more" or
"read ahead" would have been aimed at 1% of the cost.
*The cursor re-read its own buffer.* `cur_load` called `node_read`
unconditionally, and `schfs_cursor_val` calls `cur_load` on **every access to
the current record** — deliberately, so that a callback disturbing the shared
block buffer cannot corrupt a directory walk. But `node_read` copies 4096 bytes
out of the buffer cache and then runs CRC32c over all 4096. Listing a 243-entry
directory paid both once per entry. The cursor now remembers which block its
buffer holds and skips the reload, cross-checked against the block header (so a
second cursor sharing the page cannot fool it) and against a global epoch that
every mutation bumps (so a block rewritten in place cannot be served stale).
*The checksum was bit-at-a-time.* `schfs_crc32c` did eight shifts per byte —
32,768 iterations to checksum one block — to stay usable before any
initialisation. It builds a 1 KB table on first use now, which keeps that
property and is eight times faster. This only became the top cost *after* the
cursor fix, which is the usual shape: removing the largest item promotes
whatever was beneath it.
| cycles, one session | before | after |
|---|---|---|
| `getdents` | 26.8 G | **0.14 G** (196×) |
| `read` | 26.4 G | **5.5 G** (4.8×) |
| `exec` | 18.7 G | **4.0 G** (4.7×) |
| block reads | 398,999 | **127,713** |
`ls /bin` went from ~400 ms to ~20 ms. Verified by reading an 802 KB binary
through the whole path and checking its md5 against the host's copy.
There was also a third of the problem that was not the kernel at all: `make
qemu` ran QEMU without `-enable-kvm`, so every instruction was emulated. The run
targets probe `/dev/kvm` now. Measure against the KVM number or you are
measuring TCG.
**Phase 4 — extents with refcounts. DONE.** `cp` that costs nothing.
A reference count per run of blocks, in its own tree (`SCHFS_KEY_EXTREF` under
`SCHFS_EXTREF_OBJECTID`) rather than in the file's own extent record — because a
count kept per file cannot be decremented by the *other* file sharing the run.
`schfs_file_reflink` copies one extent record per run and bumps the count; both
files stay ordinary and writable, and the first write to either copies only the
run it touches. Reachable from userland as `FICLONE`, Linux's ioctl, so
`cp --reflink` works unmodified; `cmd/reflink` is a two-line caller of it.
Measured on the running system: cloning a 411 KB file moved `df` by 24 KB.
Three things had to be right, and each was wrong first:
- **The count is "how many extent records point into this run", not "how many
files".** Splitting a shared extent around a written block leaves a head *and*
a tail, which is one more reference than before. Missing that leaked four
blocks per split.
- **A file's extent may point into the middle of a run.** Records are keyed by
the block a run was *allocated* at, so every lookup is a range query
(`extref_find`), not an exact one. An exact lookup finds nothing for a split
tail, "nothing" means unshared, and the tail is then freed while another file
still reaches it.
- **The space scan has to walk the reference records too.** A file truncated
inside a shared run describes fewer blocks than exist; the rest would be
derived as free while still referenced, and the allocator would hand out a
block that a later release frees a second time.
None of those is visible until every holder of a run is deleted, which is why
the test that matters is the least glamorous one: after deleting every file that
shared a run, free space must be back to exactly what it was.
**Phase 5 — attribute index and query. DONE.** The reason the design was one
B-tree with many key types from the start: the "database filesystem" feature
turned out to be `kernel/fs/schfs/attr.c` and nothing else.
An attribute is a named value on an object, stored under `(ino, ATTR,
hash-of-name)` and packed exactly as directory entries are — same collision
argument, same shape. An attribute may be *indexed*, and an indexed attribute
gets a second set of records keyed by value: `(hash-of-name, INDEX,
hash-of-value)` holding the object ids that carry it. So "every object whose
`tag` is `holiday`" is one tree descent instead of a walk of every object.
There is no query language, no parser and no planner, and §6 above explains why:
the useful half of "query the filesystem" is the index, and WinFS spent three
years and a large team on the other interpretation.
```
schattr index tag # declare the index
schattr set /tmp/a tag holiday
schattr find tag holiday # one descent, whatever the volume holds
```
Four decisions worth keeping:
- **Which attributes are indexed is cached in the volume**, read once at mount.
Every `setxattr` has to ask, and the answer is almost always no; a tree
descent to say so would make every attribute write pay for a feature it is
not using.
- **The value is stored in the index entry, not just its hash.** Two values can
hash alike, and a query that returned the wrong file would be far worse than a
slow one.
- **An index covers values written after it is declared.** There is no
reindexing pass. Saying so is better than a half-truth that makes a query
answer confidently and wrongly.
- **A query on an unindexed attribute is refused, not answered by scanning.** A
query that silently degrades into a full walk teaches you to use it where it
will not scale.
Reachable as `setxattr`/`getxattr`/`removexattr` at Linux's syscall numbers, so
anything that already knows how to attach metadata needs no changes, plus two
native calls (`schfs_index`, `schfs_query`) that have no Linux equivalent
because asking a filesystem a question does not.
**Phase 5 — attribute index and query.** See below.
---
## What was reused from v1
Not starting from zero:
- **`struct schfs_extent`** is already defined in `include/fs.h` and unused.
The extent concept is already sketched; v2 gives it refcounts and 64-bit
fields.
- **CRC32c checksum fields** already exist in `dinode` and `superblock`, and
mkfs already computes them. Same algorithm, relocated into parent pointers.
- **`kernel/fs/log.c`** is a working transaction log with `begin_op`/`end_op`
call sites throughout the FS. Under CoW the log is not needed for consistency,
but `begin_op`/`end_op` are exactly the right places to bracket a transaction
group, so the *call sites* survive even though the implementation is replaced.
- **The buffer cache** (`bio.c`) is format-agnostic and needs no change beyond
the block-size constant.
---
## What it was estimated to cost
Phases 1-3 are roughly 3,000-4,000 lines and are the bulk of the work. Phase 1
alone (B-tree + torture harness) is worth doing carefully and unhurriedly; every
later phase compounds on it.
This is the largest single piece of work in SchOS. It is also entirely
ordinary filesystem engineering with excellent published references
(Giampaolo for BeFS; the btrfs design papers; the ZFS uberblock/checksum-in-
parent scheme) — there is no research risk here, only volume.
**Do this before the installer.** The installer's entire job is to create and
populate this format. Building it against v1 means building it twice.
---
## Things learned the hard way
Recorded because each one cost real time and none is obvious from the code.
- **The root's object id must be reserved.** `next_objectid` started at
`SCHFS_FIRST_OBJECTID`, which *is* the root's id, so the first directory
created silently became the root. It then contained itself, and every tree
walk recursed until the stack ran out. `schfs_fsck --check` now detects
directory loops explicitly rather than crashing.
- **Cursor buffers alias.** A range callback gets a pointer into the cursor's
block buffer, and almost every readdir caller immediately looks the entry up,
which overwrites it. Directory iteration copies each entry out before calling
back. Extracting a tree with directories reparented under their own children
is what this looks like when it goes wrong.
- **A 4096-byte block does not fit on a 4096-byte kernel stack.** The cursor
exists precisely to avoid one block buffer per tree level, but
`schfs_vol_open` still kept a whole uberblock as a local and blew the stack on
the first boot. The kstack canary caught it cleanly.
- **Extents only merge if you allocate the run first.** Writing an extent record
allocates blocks for its own copy-on-write path, so allocating data one block
at a time guarantees the blocks are never adjacent and every block needs its
own extent. `schfs_file_reserve` claims the whole run up front; an 822 KB
binary is one extent instead of 201.
- **Freed blocks are reusable during a build.** Copy-on-write forbids reusing a
released block because the previously committed tree still points at it — but
a build has committed nothing. Recycling them cut the test image from 3596
blocks to 374.
- **Reproducibility dies from atime.** mkschfs reads every source file, which
updates its atime, so copying atime into the image made two builds of the same
tree differ. All three timestamps come from mtime.
- **An optional hook must still be initialised.** `struct schfs_bdev` gained
`bflush`, and every host tool built one on the stack field by field. The tools
that did not set the new field handed `schfs_vol_commit` a garbage pointer to
test and call, which segfaulted at the first commit. Each of those helpers
memsets the struct first now — the fix is not "remember to set it".
- **Deferring writes makes an all-dirty buffer cache normal.** `bget` panicked
with "no buffers" when it could find nothing clean to recycle, which was
correct while a dirty buffer meant "the log has not committed". It now writes
the least recently used victim out and takes it; a transaction touching more
than `NBUF` blocks reaches that state long before it commits.
- **The two performance fixes are worthless apart.** Deferring tree writes buys
nothing while every commit flushes, and batching commits buys nothing while
every write goes straight to the disk. Measuring either alone says "this is
not where the time goes", and both measurements are correct.
- **`fsid_t` is eight bytes.** `struct statfs` declared `f_fsid` as `int64[2]`,
which is one slot too long, so everything after it landed in the wrong place:
userland read `f_frsize` out of `f_namelen` and got 255. df then computed in
units of 255 bytes and called a 32 MB root filesystem 1992 KB. It survived
because every field in that struct used to be an invented constant, so no
number in it had ever been checked against reality.
---
## The one about the tests themselves
`tools/posix/mutate.sh` injects six deliberate bugs and fails if any goes
undetected.
It exists because a suite that passes is not evidence until something has
checked that it can fail. The most valuable single assertion in the whole SchFS
suite is also the least glamorous: **after deleting everything, free space must
be back to exactly what it was on an empty volume.** Every other test still
passes on a filesystem that leaks a block on every delete.