SchOS/docs/guides/testing.md

Testing

Booting to a prompt proves nothing. A filesystem can mount and still fail the first real command, and that has happened here.

So the harnesses drive the system and assert on what comes back, rather than reporting that it started.

---

The three levels

	What it runs	What it proves
make schfs-check	Host-native unit suites	The filesystem's logic, in about a second
tools/windows/bootcheck.ps1	A real boot, driven over the serial console	That the system answers correctly
make crashqr, make schcom-test	A boot, then pixel assertions from the host	That the screen is right

The levels exist because each catches what the one below cannot. A unit test cannot tell you the kernel fails to boot; a boot test cannot tell you the screen is blank.

---

make schfs-check — the fast one

Run this first and run it often. It is host-native, so a test takes a second instead of a boot.

make schfs-check          # B-tree, object and image suites
make schfs-test           # the B-tree alone
make schfs-objtest        # the object layer alone
make schfs-image-test     # build an image with mkschfs, check it with fsck

This works because the kernel and the host tools compile the same SchFS source. mkschfs cannot disagree with the kernel about the on-disk format, because there is only one implementation — so format bugs are findable on the host, where the loop is fast.

bootcheck — the boot suite

tools/windows/bootcheck.ps1                 # the default run
tools/windows/bootcheck.ps1 -Tty -Ps -Net   # switches compose

It boots SchOS under QEMU, types commands at the serial console, and greps the transcript for the answers they should produce.

It is PowerShell because pwsh runs on every supported host — Linux, macOS and the BSDs as well as Windows. brew install powershell, or your distribution's package. It finds QEMU on PATH.

The switches, and why each exists

Each of these turns on something the default run cannot reach. That is the point: a subsystem nothing exercises looks identical whether it works or not.

Switch	Why it is not in the default run
-Tty	Pipes, isatty, SIGPIPE, Ctrl-C. None can be checked from a screenshot, and all were once broken in ways that looked like broken commands.
-Usb	Attaches an xHCI controller with a keyboard, mouse and mass-storage device. Nothing on the default machine has a USB port, so without it a broken driver looks exactly like a working one.
-Sock	Unix sockets. A socket layer that accepted every call and dropped every byte would look identical from outside. socktest has one process write and another read.
-Evdev	Raw input. Needs the QEMU monitor, because a key has to be pressed — characters typed at the serial port never touch the keyboard controller.
-Ps	Process enumeration. /proc's whole reason for existing is one directory per process, and nothing else in the suite touches it.
-Net	TCP/IP. A listener runs on the host and the guest connects out.
-Iso	Boots schos.iso from an emulated optical drive with no hard disk attached, which is the only way to prove the ISO is self-contained. Needs make iso first.
-Fb	Framebuffer checks.

-Net deserves its own note, because it is the shape every integration test should copy: both ends are asserted. The guest reports what it received and the listener reports what arrived. A stack that completed a handshake and then dropped every byte would pass any check made from one side alone.

Under QEMU's user-mode networking the host is 10.0.2.2 seen from inside, and SLIRP connects to the host's loopback — so the listener binds there.

Pixel assertions

make crashshot && make crashqr   # crash the kernel, decode the QR out of the framebuffer
make schcom-test                 # boot, run the compositor and a client, check the pixels

tools/windows/splashshot.ps1 counts colours. That exists because a log-only harness once passed on a blank screen.

schcom-test asserts from the host — the compositor's own report that it composited something proves nothing. It runs two overlapping clients and checks that the newer one wins in the overlap, because a compositor that cannot order two surfaces cannot host a window manager.

Other capture tools, all in tools/windows/: dispshot, fbshot, splashshot, textshot, ttyshot.

---

Two lessons worth keeping

Read the transcript, not the verdict. A change to the input grab once passed every assertion green while not having been applied at all — a scripted string replacement had silently failed to match, and the grep that should have caught it returned three hits (a comment, a #define, and the release call) that were read as confirmation without counting what they were. What caught it was four bare # prompts in the transcript, which is precisely the symptom the change was meant to remove.

A test that stops finding bugs has a standing cost. A ported 1993 shooter served as the full-stack acceptance test for a while and was a genuinely harsh one, because it was not written against our bugs — it found three real defects (O_NONBLOCK being a documented no-op, evdev replaying its backlog, and no EVIOCGRAB). It was removed once it stopped finding anything, because carrying it meant 33 MB of non-redistributable data staged into sysroot/, which is linked into the kernel binary for a ramroot build.

The shape of that test is worth copying even though the program is gone: something large that we did not write, driven to a real pixel-level assertion rather than a log line.

---

The suite is currently red

make schfs-check exits non-zero at baseline, and has for some time. Two separate causes, both in the harness rather than the filesystem, both itemised in ../plan/1.0.md:

• AddressSanitizer reports 2121728 bytes leaked in 518 allocations, all from the test's own page allocator, which never frees.
• UndefinedBehaviorSanitizer reports 31 misaligned reads, from casting pointers into tree leaves where records are not 8-aligned.

Worth knowing before you run it, and worth fixing rather than living with: the assertions still pass, so it is tempting to read the [ok] lines and ignore the exit code — which is exactly how a suite stops being a check and becomes a decoration.

What is not tested

• No automated test runs on real hardware. Everything above is QEMU.
• SMP is exercised but not stressed — boots on 1, 2 and 4 processors are checked; nothing hunts for races.
• No fuzzing anywhere, and no fsx/xfstests against SchFS. Getting SchFS under those is the main argument for the FUSE implementation in ../plan/roadmap.md.
• The SchFS round-trip test needs symlinks and hard links, so it skips on a Git Bash host and only runs on Unix.
• make ports does not work on Windows. musl, toybox and mksh must be built once on a Unix host and staged.