SchOS/docs/guides/testing.md

Testing

Booting to a prompt proves nothing. A filesystem can mount and still fail the first real command, and that has happened here.

So the harnesses drive the system and assert on what comes back, rather than reporting that it started.

---

The three levels

	What it runs	What it proves
make schfs-check, make kmem-test, ldls-inspect/ldls-run	Host-native unit suites and the Linux-module loader run on the build machine	The filesystem's logic, the slab allocator's bookkeeping, and a .ko's layout, in about a second
tools/windows/bootcheck.ps1	A real boot, driven over the serial console	That the system answers correctly
make crashqr, make schcom-test	A boot, then pixel assertions from the host	That the screen is right

The levels exist because each catches what the one below cannot. A unit test cannot tell you the kernel fails to boot; a boot test cannot tell you the screen is blank.

---

make schfs-check — the fast one

Run this first and run it often. It is host-native, so a test takes a second instead of a boot.

make schfs-check          # B-tree, object and image suites
make schfs-test           # the B-tree alone
make schfs-objtest        # the object layer alone
make schfs-image-test     # build an image with mkschfs, check it with fsck

This works because the kernel and the host tools compile the same SchFS source. mkschfs cannot disagree with the kernel about the on-disk format, because there is only one implementation — so format bugs are findable on the host, where the loop is fast.

bootcheck — the boot suite

tools/windows/bootcheck.ps1                 # the default run
tools/windows/bootcheck.ps1 -Tty -Ps -Net   # switches compose

It boots SchOS under QEMU, types commands at the serial console, and greps the transcript for the answers they should produce.

It is PowerShell because pwsh runs on every supported host — Linux, macOS and the BSDs as well as Windows. brew install powershell, or your distribution's package. It finds QEMU on PATH.

The switches, and why each exists

Each of these turns on something the default run cannot reach. That is the point: a subsystem nothing exercises looks identical whether it works or not.

Switch	Why it is not in the default run
-Tty	Pipes, isatty, SIGPIPE, Ctrl-C. None can be checked from a screenshot, and all were once broken in ways that looked like broken commands.
-Usb	Attaches an xHCI controller with a keyboard, mouse and mass-storage device. Nothing on the default machine has a USB port, so without it a broken driver looks exactly like a working one.
-Sock	Unix sockets. A socket layer that accepted every call and dropped every byte would look identical from outside. socktest has one process write and another read — including the SCM_RIGHTS checks, where the descriptor passed is always a pipe end, because a pipe announces its own lifetime and that is what makes "was the kernel's reference released" a value rather than a belief.
-Evdev	Raw input. Needs the QEMU monitor, because a key has to be pressed — characters typed at the serial port never touch the keyboard controller.
-Ps	Process enumeration. /proc's whole reason for existing is one directory per process, and nothing else in the suite touches it.
-Net	TCP/IP. A listener runs on the host and the guest connects out.
-Iso	Boots schos.iso from an emulated optical drive with no hard disk attached, which is the only way to prove the ISO is self-contained. Needs make iso first.
-Fb	Framebuffer checks.

-Net deserves its own note, because it is the shape every integration test should copy: both ends are asserted. The guest reports what it received and the listener reports what arrived. A stack that completed a handshake and then dropped every byte would pass any check made from one side alone.

Under QEMU's user-mode networking the host is 10.0.2.2 seen from inside, and SLIRP connects to the host's loopback — so the listener binds there.

Test programs that run on SchOS

Several first-party programs in cmd/ exist to be run on the system and print which of their checks failed, because only the system can say whether its own interfaces behave. bootcheck runs the ones it can; the rest are run by hand or by a script:

	Checks
socktest, nettest, netecho	Unix and INET sockets, SCM_RIGHTS and credentials, setsockopt's refusals
shmtest, memfdtest	System V shared memory; memfd mapped by two processes, and one sent over a socket
pcitest	/dev/pci: claim, map, DMA, interrupts, release
evtest, locktest, sigtest	the event descriptors; flock and record locks; signals
dynhello	dynamic linking: TLS, the heap, a forked child that still has its libraries
xready	every kernel call Xorg makes, before Xorg is started — the evdev queries, the fbdev ioctls, the console calls, the /sys files
tools/python/ldls_check.py	what LDLS needs from a kernel's BTF

Pixel assertions

make crashshot && make crashqr   # crash the kernel, decode the QR out of the framebuffer
make schcom-test                 # boot, run the compositor and a client, check the pixels

tools/windows/splashshot.ps1 counts colours. That exists because a log-only harness once passed on a blank screen.

schcom-test asserts from the host — the compositor's own report that it composited something proves nothing. It runs two overlapping clients and checks that the newer one wins in the overlap, because a compositor that cannot order two surfaces cannot host a window manager.

Other capture tools, all in tools/windows/: dispshot, fbshot, splashshot, textshot, ttyshot.

---

Two lessons worth keeping

Read the transcript, not the verdict. A change to the input grab once passed every assertion green while not having been applied at all — a scripted string replacement had silently failed to match, and the grep that should have caught it returned three hits (a comment, a #define, and the release call) that were read as confirmation without counting what they were. What caught it was four bare # prompts in the transcript, which is precisely the symptom the change was meant to remove.

A test that stops finding bugs has a standing cost. A ported 1993 shooter served as the full-stack acceptance test for a while and was a genuinely harsh one, because it was not written against our bugs — it found three real defects (O_NONBLOCK being a documented no-op, evdev replaying its backlog, and no EVIOCGRAB). It was removed once it stopped finding anything, because carrying it meant 33 MB of non-redistributable data staged into sysroot/, which is linked into the kernel binary for a ramroot build.

The shape of that test is worth copying even though the program is gone: something large that we did not write, driven to a real pixel-level assertion rather than a log line.

---

The suite is green

make schfs-check passes with no sanitizer output at all. It was not, until recently, and both failures are worth knowing about because neither was in the filesystem's logic:

A leak with no API to fix it. The test leaked 518 pages because the allocation map had no release path at all — which meant the kernel leaked the same map on every unmount. A suite that leaks cannot detect a leak: the filesystem's own allocations would have arrived as noise inside the harness's. schfs_objtest now asserts that every page it handed out came back.

31 misaligned reads. Records are read by casting a pointer into a tree leaf, where they sit at arbitrary offsets. That works on x86 and is undefined in C, so it had never shown as a bug and would have arrived as apparent filesystem corruption at some future optimisation level.

The shared lesson is the one this directory keeps relearning: a sanitizer that is reliably noisy is a sanitizer nobody reads. Both were dismissible as "known and harmless" for as long as they were there, and one of them was a real kernel leak.

What is not tested

• No automated test runs on real hardware. Everything above is QEMU.
• SMP is exercised but not stressed — boots on 1, 2 and 4 processors are checked; nothing hunts for races.
• Nothing asserts the desktop by pixel. xready checks the calls Xorg makes; whether Window Maker and an xterm appear on a real screen is looked at in screenshots, not asserted.
• No fuzzing anywhere, and no fsx/xfstests against SchFS. Getting SchFS under those is the main argument for the FUSE implementation in ../plan/roadmap.md.
• The SchFS round-trip test needs symlinks and hard links, so it skips on a Git Bash host and only runs on Unix.
• make ports does not work on Windows. musl, toybox, mksh and the rest must be built once on a Unix host and staged.