3e2f3cd1ec
Two failures found by running `make schfs-check` rather than by reading it, both
predating the current work -- confirmed by stashing the session's changes and
re-running to byte-identical output.
* AddressSanitizer: 2121728 bytes leaked in 518 allocations, every one from
md_palloc in tools/schfs_objtest.c, the test's own page allocator. The
target exits non-zero, so the whole suite fails whatever the assertions say.
The cost is not the memory -- it is that a real leak in the filesystem would
now arrive as noise inside 2 MB of expected noise.
* UndefinedBehaviorSanitizer: 31 misaligned accesses across obj.c, objw.c,
attr.c and space.c, one root cause. Records are read by casting a pointer
into a tree leaf, and a record's offset within a leaf is not a multiple of
8. This works on x86, which is why it has never shown as a bug, and it is
undefined in C: a compiler entitled to assume 8-byte alignment may emit an
instruction that requires it, and the failure would then look like
filesystem corruption rather than codegen.
Recorded as Gate 0 items rather than as nice-to-haves for one reason: a suite
that is red for known reasons stops being read, and then it is red for unknown
reasons too. guides/testing.md now says so up front, so nobody reads the [ok]
lines and ignores the exit code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
151 lines
7 KiB
Markdown
151 lines
7 KiB
Markdown
# Testing
|
|
|
|
> **Booting to a prompt proves nothing.** A filesystem can mount and still fail
|
|
> the first real command, and that has happened here.
|
|
|
|
So the harnesses drive the system and assert on what comes back, rather than
|
|
reporting that it started.
|
|
|
|
---
|
|
|
|
## The three levels
|
|
|
|
| | What it runs | What it proves |
|
|
|---|---|---|
|
|
| `make schfs-check` | Host-native unit suites | The filesystem's logic, in about a second |
|
|
| `tools/windows/bootcheck.ps1` | A real boot, driven over the serial console | That the system answers correctly |
|
|
| `make crashqr`, `make schcom-test` | A boot, then pixel assertions from the host | That the *screen* is right |
|
|
|
|
The levels exist because each catches what the one below cannot. A unit test
|
|
cannot tell you the kernel fails to boot; a boot test cannot tell you the screen
|
|
is blank.
|
|
|
|
---
|
|
|
|
## `make schfs-check` — the fast one
|
|
|
|
Run this first and run it often. It is host-native, so a test takes a second
|
|
instead of a boot.
|
|
|
|
```sh
|
|
make schfs-check # B-tree, object and image suites
|
|
make schfs-test # the B-tree alone
|
|
make schfs-objtest # the object layer alone
|
|
make schfs-image-test # build an image with mkschfs, check it with fsck
|
|
```
|
|
|
|
This works because **the kernel and the host tools compile the same SchFS
|
|
source**. `mkschfs` cannot disagree with the kernel about the on-disk format,
|
|
because there is only one implementation — so format bugs are findable on the
|
|
host, where the loop is fast.
|
|
|
|
## `bootcheck` — the boot suite
|
|
|
|
```sh
|
|
tools/windows/bootcheck.ps1 # the default run
|
|
tools/windows/bootcheck.ps1 -Tty -Ps -Net # switches compose
|
|
```
|
|
|
|
It boots SchOS under QEMU, types commands at the serial console, and greps the
|
|
transcript for the answers they should produce.
|
|
|
|
**It is PowerShell because `pwsh` runs on every supported host** — Linux, macOS
|
|
and the BSDs as well as Windows. `brew install powershell`, or your
|
|
distribution's package. It finds QEMU on `PATH`.
|
|
|
|
### The switches, and why each exists
|
|
|
|
Each of these turns on something the default run cannot reach. That is the point:
|
|
a subsystem nothing exercises looks identical whether it works or not.
|
|
|
|
| Switch | Why it is not in the default run |
|
|
|---|---|
|
|
| `-Tty` | Pipes, `isatty`, `SIGPIPE`, Ctrl-C. **None can be checked from a screenshot**, and all were once broken in ways that looked like broken *commands*. |
|
|
| `-Usb` | Attaches an xHCI controller with a keyboard, mouse and mass-storage device. Nothing on the default machine has a USB port, so without it a broken driver looks exactly like a working one. |
|
|
| `-Sock` | Unix sockets. A socket layer that accepted every call and dropped every byte would look identical from outside. `socktest` has one process write and another read. |
|
|
| `-Evdev` | Raw input. Needs the QEMU monitor, because a key has to be *pressed* — characters typed at the serial port never touch the keyboard controller. |
|
|
| `-Ps` | Process enumeration. `/proc`'s whole reason for existing is one directory per process, and nothing else in the suite touches it. |
|
|
| `-Net` | TCP/IP. A listener runs **on the host** and the guest connects out. |
|
|
| `-Iso` | Boots `schos.iso` from an emulated optical drive with **no hard disk attached**, which is the only way to prove the ISO is self-contained. Needs `make iso` first. |
|
|
| `-Fb` | Framebuffer checks. |
|
|
|
|
`-Net` deserves its own note, because it is the shape every integration test
|
|
should copy: **both ends are asserted.** The guest reports what it received and
|
|
the listener reports what arrived. A stack that completed a handshake and then
|
|
dropped every byte would pass any check made from one side alone.
|
|
|
|
Under QEMU's user-mode networking the host is `10.0.2.2` seen from inside, and
|
|
SLIRP connects to the host's loopback — so the listener binds there.
|
|
|
|
## Pixel assertions
|
|
|
|
```sh
|
|
make crashshot && make crashqr # crash the kernel, decode the QR out of the framebuffer
|
|
make schcom-test # boot, run the compositor and a client, check the pixels
|
|
```
|
|
|
|
`tools/windows/splashshot.ps1` counts colours. **That exists because a log-only
|
|
harness once passed on a blank screen.**
|
|
|
|
`schcom-test` asserts **from the host** — the compositor's own report that it
|
|
composited something proves nothing. It runs two overlapping clients and checks
|
|
that the newer one wins in the overlap, because a compositor that cannot order
|
|
two surfaces cannot host a window manager.
|
|
|
|
Other capture tools, all in `tools/windows/`: `dispshot`, `fbshot`,
|
|
`splashshot`, `textshot`, `ttyshot`.
|
|
|
|
---
|
|
|
|
## Two lessons worth keeping
|
|
|
|
**Read the transcript, not the verdict.** A change to the input grab once passed
|
|
every assertion green while not having been applied at all — a scripted string
|
|
replacement had silently failed to match, and the `grep` that should have caught
|
|
it returned three hits (a comment, a `#define`, and the *release* call) that were
|
|
read as confirmation without counting what they were. What caught it was four
|
|
bare `#` prompts in the transcript, which is precisely the symptom the change was
|
|
meant to remove.
|
|
|
|
**A test that stops finding bugs has a standing cost.** A ported 1993 shooter
|
|
served as the full-stack acceptance test for a while and was a genuinely harsh
|
|
one, because it was not written against our bugs — it found three real defects
|
|
(`O_NONBLOCK` being a documented no-op, evdev replaying its backlog, and no
|
|
`EVIOCGRAB`). It was removed once it stopped finding anything, because carrying
|
|
it meant 33 MB of non-redistributable data staged into `sysroot/`, which is
|
|
linked into the kernel binary for a `ramroot` build.
|
|
|
|
> **The shape of that test is worth copying even though the program is gone:**
|
|
> something large that we did not write, driven to a real pixel-level assertion
|
|
> rather than a log line.
|
|
|
|
---
|
|
|
|
## The suite is currently red
|
|
|
|
`make schfs-check` **exits non-zero at baseline**, and has for some time. Two
|
|
separate causes, both in the harness rather than the filesystem, both itemised
|
|
in [../plan/1.0.md](../plan/1.0.md):
|
|
|
|
- AddressSanitizer reports 2121728 bytes leaked in 518 allocations, all from the
|
|
test's own page allocator, which never frees.
|
|
- UndefinedBehaviorSanitizer reports 31 misaligned reads, from casting pointers
|
|
into tree leaves where records are not 8-aligned.
|
|
|
|
Worth knowing before you run it, and worth fixing rather than living with: the
|
|
assertions still pass, so it is tempting to read the `[ok]` lines and ignore the
|
|
exit code — which is exactly how a suite stops being a check and becomes a
|
|
decoration.
|
|
|
|
## What is not tested
|
|
|
|
- **No automated test runs on real hardware.** Everything above is QEMU.
|
|
- **SMP is exercised but not stressed** — boots on 1, 2 and 4 processors are
|
|
checked; nothing hunts for races.
|
|
- **No fuzzing anywhere**, and no `fsx`/`xfstests` against SchFS. Getting SchFS
|
|
under those is the main argument for the FUSE implementation in
|
|
[../plan/roadmap.md](../plan/roadmap.md).
|
|
- **The SchFS round-trip test needs symlinks and hard links**, so it skips on a
|
|
Git Bash host and only runs on Unix.
|
|
- **`make ports` does not work on Windows.** musl, toybox and mksh must be built
|
|
once on a Unix host and staged.
|