25530e7eed
Started as "fix the ASAN leak in schfs_objtest" and the test turned out to be
the messenger. The allocation map was taken through a palloc callback and there
was no way to return it: no schfs_space_release, and a comment in
schfs_space_init reading "caller frees what was taken" for an API the caller
did not have.
Three separate leaks from that one gap:
* The kernel leaked the entire map on every unmount -- up to 2 MB, at 8 KiB
per GiB of volume -- because kmount_unmount flushed, cleared `mounted` and
walked away. This is the one that matters; the others are test-only.
* schfs_space_init stranded a page on its own error path. npages was raised
only after *both* of a pair of allocations succeeded, so a failure on the
second left the first outside the count -- unreachable to any cleanup that
could later exist.
* schfs_objtest leaked 2121728 bytes in 518 allocations, which is what made
the whole SchFS suite exit non-zero regardless of its assertions.
The map now carries its own pfree rather than taking one at release time, so it
can only ever be freed by the allocator that filled it -- the kernel's pages
come from kalloc and a host tool's from calloc, and the two are not
interchangeable. schfs_vol_open and schfs_vol_format release any map the volume
is already holding before building another, which is what stops a re-open
leaking; struct schfs_vol must therefore be zeroed before first use, and the
header now says so.
The test's own leak was the last one found and the simplest: it does
memset(&VOL, 0, sizeof(VOL)) before reopening, which wiped the map's pointers
before vol_open could release them. Eight of those, each now releasing first.
schfs_objtest asserts the balance -- 520 pages taken, 520 given back -- because
a suite that leaks cannot detect a leak: the filesystem's own allocations would
arrive as noise inside the harness's. The next one fails the run instead.
make schfs-check now exits 0. The 31 UBSAN misalignment reports are still
there and are still a Gate 0 item.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
150 lines
7 KiB
Markdown
150 lines
7 KiB
Markdown
# Testing
|
|
|
|
> **Booting to a prompt proves nothing.** A filesystem can mount and still fail
|
|
> the first real command, and that has happened here.
|
|
|
|
So the harnesses drive the system and assert on what comes back, rather than
|
|
reporting that it started.
|
|
|
|
---
|
|
|
|
## The three levels
|
|
|
|
| | What it runs | What it proves |
|
|
|---|---|---|
|
|
| `make schfs-check` | Host-native unit suites | The filesystem's logic, in about a second |
|
|
| `tools/windows/bootcheck.ps1` | A real boot, driven over the serial console | That the system answers correctly |
|
|
| `make crashqr`, `make schcom-test` | A boot, then pixel assertions from the host | That the *screen* is right |
|
|
|
|
The levels exist because each catches what the one below cannot. A unit test
|
|
cannot tell you the kernel fails to boot; a boot test cannot tell you the screen
|
|
is blank.
|
|
|
|
---
|
|
|
|
## `make schfs-check` — the fast one
|
|
|
|
Run this first and run it often. It is host-native, so a test takes a second
|
|
instead of a boot.
|
|
|
|
```sh
|
|
make schfs-check # B-tree, object and image suites
|
|
make schfs-test # the B-tree alone
|
|
make schfs-objtest # the object layer alone
|
|
make schfs-image-test # build an image with mkschfs, check it with fsck
|
|
```
|
|
|
|
This works because **the kernel and the host tools compile the same SchFS
|
|
source**. `mkschfs` cannot disagree with the kernel about the on-disk format,
|
|
because there is only one implementation — so format bugs are findable on the
|
|
host, where the loop is fast.
|
|
|
|
## `bootcheck` — the boot suite
|
|
|
|
```sh
|
|
tools/windows/bootcheck.ps1 # the default run
|
|
tools/windows/bootcheck.ps1 -Tty -Ps -Net # switches compose
|
|
```
|
|
|
|
It boots SchOS under QEMU, types commands at the serial console, and greps the
|
|
transcript for the answers they should produce.
|
|
|
|
**It is PowerShell because `pwsh` runs on every supported host** — Linux, macOS
|
|
and the BSDs as well as Windows. `brew install powershell`, or your
|
|
distribution's package. It finds QEMU on `PATH`.
|
|
|
|
### The switches, and why each exists
|
|
|
|
Each of these turns on something the default run cannot reach. That is the point:
|
|
a subsystem nothing exercises looks identical whether it works or not.
|
|
|
|
| Switch | Why it is not in the default run |
|
|
|---|---|
|
|
| `-Tty` | Pipes, `isatty`, `SIGPIPE`, Ctrl-C. **None can be checked from a screenshot**, and all were once broken in ways that looked like broken *commands*. |
|
|
| `-Usb` | Attaches an xHCI controller with a keyboard, mouse and mass-storage device. Nothing on the default machine has a USB port, so without it a broken driver looks exactly like a working one. |
|
|
| `-Sock` | Unix sockets. A socket layer that accepted every call and dropped every byte would look identical from outside. `socktest` has one process write and another read. |
|
|
| `-Evdev` | Raw input. Needs the QEMU monitor, because a key has to be *pressed* — characters typed at the serial port never touch the keyboard controller. |
|
|
| `-Ps` | Process enumeration. `/proc`'s whole reason for existing is one directory per process, and nothing else in the suite touches it. |
|
|
| `-Net` | TCP/IP. A listener runs **on the host** and the guest connects out. |
|
|
| `-Iso` | Boots `schos.iso` from an emulated optical drive with **no hard disk attached**, which is the only way to prove the ISO is self-contained. Needs `make iso` first. |
|
|
| `-Fb` | Framebuffer checks. |
|
|
|
|
`-Net` deserves its own note, because it is the shape every integration test
|
|
should copy: **both ends are asserted.** The guest reports what it received and
|
|
the listener reports what arrived. A stack that completed a handshake and then
|
|
dropped every byte would pass any check made from one side alone.
|
|
|
|
Under QEMU's user-mode networking the host is `10.0.2.2` seen from inside, and
|
|
SLIRP connects to the host's loopback — so the listener binds there.
|
|
|
|
## Pixel assertions
|
|
|
|
```sh
|
|
make crashshot && make crashqr # crash the kernel, decode the QR out of the framebuffer
|
|
make schcom-test # boot, run the compositor and a client, check the pixels
|
|
```
|
|
|
|
`tools/windows/splashshot.ps1` counts colours. **That exists because a log-only
|
|
harness once passed on a blank screen.**
|
|
|
|
`schcom-test` asserts **from the host** — the compositor's own report that it
|
|
composited something proves nothing. It runs two overlapping clients and checks
|
|
that the newer one wins in the overlap, because a compositor that cannot order
|
|
two surfaces cannot host a window manager.
|
|
|
|
Other capture tools, all in `tools/windows/`: `dispshot`, `fbshot`,
|
|
`splashshot`, `textshot`, `ttyshot`.
|
|
|
|
---
|
|
|
|
## Two lessons worth keeping
|
|
|
|
**Read the transcript, not the verdict.** A change to the input grab once passed
|
|
every assertion green while not having been applied at all — a scripted string
|
|
replacement had silently failed to match, and the `grep` that should have caught
|
|
it returned three hits (a comment, a `#define`, and the *release* call) that were
|
|
read as confirmation without counting what they were. What caught it was four
|
|
bare `#` prompts in the transcript, which is precisely the symptom the change was
|
|
meant to remove.
|
|
|
|
**A test that stops finding bugs has a standing cost.** A ported 1993 shooter
|
|
served as the full-stack acceptance test for a while and was a genuinely harsh
|
|
one, because it was not written against our bugs — it found three real defects
|
|
(`O_NONBLOCK` being a documented no-op, evdev replaying its backlog, and no
|
|
`EVIOCGRAB`). It was removed once it stopped finding anything, because carrying
|
|
it meant 33 MB of non-redistributable data staged into `sysroot/`, which is
|
|
linked into the kernel binary for a `ramroot` build.
|
|
|
|
> **The shape of that test is worth copying even though the program is gone:**
|
|
> something large that we did not write, driven to a real pixel-level assertion
|
|
> rather than a log line.
|
|
|
|
---
|
|
|
|
## One known failure left
|
|
|
|
`make schfs-check` passes. UndefinedBehaviorSanitizer still reports **31
|
|
misaligned reads** from casting pointers into tree leaves where records are not
|
|
8-aligned — itemised in [../plan/1.0.md](../plan/1.0.md), and printed on stderr
|
|
without failing the run.
|
|
|
|
The leak half of this is closed, and how it closed is the useful part: the test
|
|
leaked 518 pages because the allocation map had no release path *at all*, which
|
|
meant the kernel leaked the same map on every unmount. **A suite that leaks
|
|
cannot detect a leak** — the filesystem's own allocations would have arrived as
|
|
noise inside the harness's. `schfs_objtest` now asserts that every page it
|
|
handed out came back, so the next one fails the suite rather than joining the
|
|
background.
|
|
|
|
## What is not tested
|
|
|
|
- **No automated test runs on real hardware.** Everything above is QEMU.
|
|
- **SMP is exercised but not stressed** — boots on 1, 2 and 4 processors are
|
|
checked; nothing hunts for races.
|
|
- **No fuzzing anywhere**, and no `fsx`/`xfstests` against SchFS. Getting SchFS
|
|
under those is the main argument for the FUSE implementation in
|
|
[../plan/roadmap.md](../plan/roadmap.md).
|
|
- **The SchFS round-trip test needs symlinks and hard links**, so it skips on a
|
|
Git Bash host and only runs on Unix.
|
|
- **`make ports` does not work on Windows.** musl, toybox and mksh must be built
|
|
once on a Unix host and staged.
|