SchOS/docs/guides/testing.md

# Testing

> **Booting to a prompt proves nothing.** A filesystem can mount and still fail
> the first real command, and that has happened here.

So the harnesses drive the system and assert on what comes back, rather than
reporting that it started.

---

## The three levels

| | What it runs | What it proves |
|---|---|---|
| `make schfs-check`, `make kmem-test`, `ldls-inspect`/`ldls-run` | Host-native unit suites and the Linux-module loader run on the build machine | The filesystem's logic, the slab allocator's bookkeeping, and a `.ko`'s layout, in about a second |
| `tools/windows/bootcheck.ps1` | A real boot, driven over the serial console | That the system answers correctly |
| `make crashqr`, `make schcom-test` | A boot, then pixel assertions from the host | That the *screen* is right |

The levels exist because each catches what the one below cannot. A unit test
cannot tell you the kernel fails to boot; a boot test cannot tell you the screen
is blank.

---

## `make schfs-check` — the fast one

Run this first and run it often. It is host-native, so a test takes a second
instead of a boot.

```sh
make schfs-check          # B-tree, object and image suites
make schfs-test           # the B-tree alone
make schfs-objtest        # the object layer alone
make schfs-image-test     # build an image with mkschfs, check it with fsck
```

This works because **the kernel and the host tools compile the same SchFS
source**. `mkschfs` cannot disagree with the kernel about the on-disk format,
because there is only one implementation — so format bugs are findable on the
host, where the loop is fast.

## `bootcheck` — the boot suite

```sh
tools/windows/bootcheck.ps1                 # the default run
tools/windows/bootcheck.ps1 -Tty -Ps -Net   # switches compose
```

It boots SchOS under QEMU, types commands at the serial console, and greps the
transcript for the answers they should produce.

**It is PowerShell because `pwsh` runs on every supported host** — Linux, macOS
and the BSDs as well as Windows. `brew install powershell`, or your
distribution's package. It finds QEMU on `PATH`.

### The switches, and why each exists

Each of these turns on something the default run cannot reach. That is the point:
a subsystem nothing exercises looks identical whether it works or not.

| Switch | Why it is not in the default run |
|---|---|
| `-Tty` | Pipes, `isatty`, `SIGPIPE`, Ctrl-C. **None can be checked from a screenshot**, and all were once broken in ways that looked like broken *commands*. |
| `-Usb` | Attaches an xHCI controller with a keyboard, mouse and mass-storage device. Nothing on the default machine has a USB port, so without it a broken driver looks exactly like a working one. |
| `-Sock` | Unix sockets. A socket layer that accepted every call and dropped every byte would look identical from outside. `socktest` has one process write and another read — including the `SCM_RIGHTS` checks, where the descriptor passed is always a pipe end, because a pipe announces its own lifetime and that is what makes "was the kernel's reference released" a value rather than a belief. |
| `-Evdev` | Raw input. Needs the QEMU monitor, because a key has to be *pressed* — characters typed at the serial port never touch the keyboard controller. |
| `-Ps` | Process enumeration. `/proc`'s whole reason for existing is one directory per process, and nothing else in the suite touches it. |
| `-Net` | TCP/IP. A listener runs **on the host** and the guest connects out. |
| `-Iso` | Boots `schos.iso` from an emulated optical drive with **no hard disk attached**, which is the only way to prove the ISO is self-contained. Needs `make iso` first. |
| `-Fb` | Framebuffer checks. |

`-Net` deserves its own note, because it is the shape every integration test
should copy: **both ends are asserted.** The guest reports what it received and
the listener reports what arrived. A stack that completed a handshake and then
dropped every byte would pass any check made from one side alone.

Under QEMU's user-mode networking the host is `10.0.2.2` seen from inside, and
SLIRP connects to the host's loopback — so the listener binds there.

## Test programs that run on SchOS

Several first-party programs in `cmd/` exist to be run **on** the system and print which of their checks
failed, because only the system can say whether its own interfaces behave. `bootcheck` runs the ones it
can; the rest are run by hand or by a script:

| | Checks |
|---|---|
| `socktest`, `nettest`, `netecho` | Unix and INET sockets, `SCM_RIGHTS` and credentials, `setsockopt`'s refusals |
| `shmtest`, `memfdtest` | System V shared memory; `memfd` mapped by two processes, and one sent over a socket |
| `pcitest` | `/dev/pci`: claim, map, DMA, interrupts, release |
| `evtest`, `locktest`, `sigtest` | the event descriptors; `flock` and record locks; signals |
| `dynhello` | dynamic linking: TLS, the heap, a forked child that still has its libraries |
| `xready` | every kernel call Xorg makes, before Xorg is started — the evdev queries, the fbdev ioctls, the console calls, the `/sys` files |
| `tools/python/ldls_check.py` | what LDLS needs from a kernel's BTF |

## Pixel assertions

```sh
make crashshot && make crashqr   # crash the kernel, decode the QR out of the framebuffer
make schcom-test                 # boot, run the compositor and a client, check the pixels
```

`tools/windows/splashshot.ps1` counts colours. **That exists because a log-only
harness once passed on a blank screen.**

`schcom-test` asserts **from the host** — the compositor's own report that it
composited something proves nothing. It runs two overlapping clients and checks
that the newer one wins in the overlap, because a compositor that cannot order
two surfaces cannot host a window manager.

Other capture tools, all in `tools/windows/`: `dispshot`, `fbshot`,
`splashshot`, `textshot`, `ttyshot`.

---

## Two lessons worth keeping

**Read the transcript, not the verdict.** A change to the input grab once passed
every assertion green while not having been applied at all — a scripted string
replacement had silently failed to match, and the `grep` that should have caught
it returned three hits (a comment, a `#define`, and the *release* call) that were
read as confirmation without counting what they were. What caught it was four
bare `#` prompts in the transcript, which is precisely the symptom the change was
meant to remove.

**A test that stops finding bugs has a standing cost.** A ported 1993 shooter
served as the full-stack acceptance test for a while and was a genuinely harsh
one, because it was not written against our bugs — it found three real defects
(`O_NONBLOCK` being a documented no-op, evdev replaying its backlog, and no
`EVIOCGRAB`). It was removed once it stopped finding anything, because carrying
it meant 33 MB of non-redistributable data staged into `sysroot/`, which is
linked into the kernel binary for a `ramroot` build.

> **The shape of that test is worth copying even though the program is gone:**
> something large that we did not write, driven to a real pixel-level assertion
> rather than a log line.

---

## The suite is green

`make schfs-check` passes with no sanitizer output at all. It was not, until
recently, and both failures are worth knowing about because neither was in the
filesystem's logic:

**A leak with no API to fix it.** The test leaked 518 pages because the
allocation map had no release path *at all* — which meant the kernel leaked the
same map on every unmount. **A suite that leaks cannot detect a leak**: the
filesystem's own allocations would have arrived as noise inside the harness's.
`schfs_objtest` now asserts that every page it handed out came back.

**31 misaligned reads.** Records are read by casting a pointer into a tree leaf,
where they sit at arbitrary offsets. That works on x86 and is undefined in C, so
it had never shown as a bug and would have arrived as apparent filesystem
corruption at some future optimisation level.

The shared lesson is the one this directory keeps relearning: **a sanitizer that
is reliably noisy is a sanitizer nobody reads.** Both were dismissible as "known
and harmless" for as long as they were there, and one of them was a real kernel
leak.

## What is not tested

- **No automated test runs on real hardware.** Everything above is QEMU.
- **SMP is exercised but not stressed** — boots on 1, 2 and 4 processors are
  checked; nothing hunts for races.
- **Nothing asserts the desktop by pixel.** `xready` checks the calls Xorg makes; whether Window Maker
  and an `xterm` appear on a real screen is looked at in screenshots, not asserted.
- **No fuzzing anywhere**, and no `fsx`/`xfstests` against SchFS. Getting SchFS
  under those is the main argument for the FUSE implementation in
  [../plan/roadmap.md](../plan/roadmap.md).
- **The SchFS round-trip test needs symlinks and hard links**, so it skips on a
  Git Bash host and only runs on Unix.
- **`make ports` does not work on Windows.** musl, toybox, mksh and the rest must be built
  once on a Unix host and staged.