57aff40228
docs/ was ten files where a plan, a description, a guide and a bug report all
sat next to each other. It is now architecture/ (what SchOS is), plan/ (what it
will be), guides/ (how to work on it) and history/ (why it is like this), with
an index that states the one rule holding it together: when something is
finished it moves from plan/ to architecture/ and is rewritten as a description
of the thing rather than a record of building it.
A ticked checkbox says the work happened. It does not say what the work
*produced*, and six months later the checkbox is all that is left. The network
stack proved the point: finished, verified end to end, and for a month the only
description of it anywhere was a checkbox four levels of indentation deep.
So roadmap.md went from 534 lines to 181 -- nothing deleted, the done work
graduated. Seven subsystems that were finished and undescribed now have
descriptions: boot, kernel, syscalls, filesystem, networking, graphics, usb,
init. Every one ends with "What it does not do", which is the section that pays;
a document listing only strengths is marketing, and the reader who needs it most
is the one about to trip over a limitation nobody wrote down.
schfs.md still opened with "phases 1 through 3 1/2 are done, phase 4 is next"
long after all five had landed. It is a description now, with the phase record
and the hard-won bugs moved to history/.
Three corrections found on the way, all load-bearing:
* projects/schcom/README.md claimed SCM_RIGHTS was done. There is no
sendmsg/recvmsg in the kernel at all, and an entire GUI plan rested on it.
* The root README said "No TCP/IP" months after the stack landed.
* history/kernel.md sec.7 was marked "in progress" on a problem that process
groups solved.
Also adds plan/1.0.md, which draws a finish line so the open list stops being
infinite, and marks schwm/ and schde/ superseded: in Wayland the compositor is
the window manager, so that split dissolves rather than being abandoned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
135 lines
6.3 KiB
Markdown
135 lines
6.3 KiB
Markdown
# Testing
|
|
|
|
> **Booting to a prompt proves nothing.** A filesystem can mount and still fail
|
|
> the first real command, and that has happened here.
|
|
|
|
So the harnesses drive the system and assert on what comes back, rather than
|
|
reporting that it started.
|
|
|
|
---
|
|
|
|
## The three levels
|
|
|
|
| | What it runs | What it proves |
|
|
|---|---|---|
|
|
| `make schfs-check` | Host-native unit suites | The filesystem's logic, in about a second |
|
|
| `tools/windows/bootcheck.ps1` | A real boot, driven over the serial console | That the system answers correctly |
|
|
| `make crashqr`, `make schcom-test` | A boot, then pixel assertions from the host | That the *screen* is right |
|
|
|
|
The levels exist because each catches what the one below cannot. A unit test
|
|
cannot tell you the kernel fails to boot; a boot test cannot tell you the screen
|
|
is blank.
|
|
|
|
---
|
|
|
|
## `make schfs-check` — the fast one
|
|
|
|
Run this first and run it often. It is host-native, so a test takes a second
|
|
instead of a boot.
|
|
|
|
```sh
|
|
make schfs-check # B-tree, object and image suites
|
|
make schfs-test # the B-tree alone
|
|
make schfs-objtest # the object layer alone
|
|
make schfs-image-test # build an image with mkschfs, check it with fsck
|
|
```
|
|
|
|
This works because **the kernel and the host tools compile the same SchFS
|
|
source**. `mkschfs` cannot disagree with the kernel about the on-disk format,
|
|
because there is only one implementation — so format bugs are findable on the
|
|
host, where the loop is fast.
|
|
|
|
## `bootcheck` — the boot suite
|
|
|
|
```sh
|
|
tools/windows/bootcheck.ps1 # the default run
|
|
tools/windows/bootcheck.ps1 -Tty -Ps -Net # switches compose
|
|
```
|
|
|
|
It boots SchOS under QEMU, types commands at the serial console, and greps the
|
|
transcript for the answers they should produce.
|
|
|
|
**It is PowerShell because `pwsh` runs on every supported host** — Linux, macOS
|
|
and the BSDs as well as Windows. `brew install powershell`, or your
|
|
distribution's package. It finds QEMU on `PATH`.
|
|
|
|
### The switches, and why each exists
|
|
|
|
Each of these turns on something the default run cannot reach. That is the point:
|
|
a subsystem nothing exercises looks identical whether it works or not.
|
|
|
|
| Switch | Why it is not in the default run |
|
|
|---|---|
|
|
| `-Tty` | Pipes, `isatty`, `SIGPIPE`, Ctrl-C. **None can be checked from a screenshot**, and all were once broken in ways that looked like broken *commands*. |
|
|
| `-Usb` | Attaches an xHCI controller with a keyboard, mouse and mass-storage device. Nothing on the default machine has a USB port, so without it a broken driver looks exactly like a working one. |
|
|
| `-Sock` | Unix sockets. A socket layer that accepted every call and dropped every byte would look identical from outside. `socktest` has one process write and another read. |
|
|
| `-Evdev` | Raw input. Needs the QEMU monitor, because a key has to be *pressed* — characters typed at the serial port never touch the keyboard controller. |
|
|
| `-Ps` | Process enumeration. `/proc`'s whole reason for existing is one directory per process, and nothing else in the suite touches it. |
|
|
| `-Net` | TCP/IP. A listener runs **on the host** and the guest connects out. |
|
|
| `-Iso` | Boots `schos.iso` from an emulated optical drive with **no hard disk attached**, which is the only way to prove the ISO is self-contained. Needs `make iso` first. |
|
|
| `-Fb` | Framebuffer checks. |
|
|
|
|
`-Net` deserves its own note, because it is the shape every integration test
|
|
should copy: **both ends are asserted.** The guest reports what it received and
|
|
the listener reports what arrived. A stack that completed a handshake and then
|
|
dropped every byte would pass any check made from one side alone.
|
|
|
|
Under QEMU's user-mode networking the host is `10.0.2.2` seen from inside, and
|
|
SLIRP connects to the host's loopback — so the listener binds there.
|
|
|
|
## Pixel assertions
|
|
|
|
```sh
|
|
make crashshot && make crashqr # crash the kernel, decode the QR out of the framebuffer
|
|
make schcom-test # boot, run the compositor and a client, check the pixels
|
|
```
|
|
|
|
`tools/windows/splashshot.ps1` counts colours. **That exists because a log-only
|
|
harness once passed on a blank screen.**
|
|
|
|
`schcom-test` asserts **from the host** — the compositor's own report that it
|
|
composited something proves nothing. It runs two overlapping clients and checks
|
|
that the newer one wins in the overlap, because a compositor that cannot order
|
|
two surfaces cannot host a window manager.
|
|
|
|
Other capture tools, all in `tools/windows/`: `dispshot`, `fbshot`,
|
|
`splashshot`, `textshot`, `ttyshot`.
|
|
|
|
---
|
|
|
|
## Two lessons worth keeping
|
|
|
|
**Read the transcript, not the verdict.** A change to the input grab once passed
|
|
every assertion green while not having been applied at all — a scripted string
|
|
replacement had silently failed to match, and the `grep` that should have caught
|
|
it returned three hits (a comment, a `#define`, and the *release* call) that were
|
|
read as confirmation without counting what they were. What caught it was four
|
|
bare `#` prompts in the transcript, which is precisely the symptom the change was
|
|
meant to remove.
|
|
|
|
**A test that stops finding bugs has a standing cost.** A ported 1993 shooter
|
|
served as the full-stack acceptance test for a while and was a genuinely harsh
|
|
one, because it was not written against our bugs — it found three real defects
|
|
(`O_NONBLOCK` being a documented no-op, evdev replaying its backlog, and no
|
|
`EVIOCGRAB`). It was removed once it stopped finding anything, because carrying
|
|
it meant 33 MB of non-redistributable data staged into `sysroot/`, which is
|
|
linked into the kernel binary for a `ramroot` build.
|
|
|
|
> **The shape of that test is worth copying even though the program is gone:**
|
|
> something large that we did not write, driven to a real pixel-level assertion
|
|
> rather than a log line.
|
|
|
|
---
|
|
|
|
## What is not tested
|
|
|
|
- **No automated test runs on real hardware.** Everything above is QEMU.
|
|
- **SMP is exercised but not stressed** — boots on 1, 2 and 4 processors are
|
|
checked; nothing hunts for races.
|
|
- **No fuzzing anywhere**, and no `fsx`/`xfstests` against SchFS. Getting SchFS
|
|
under those is the main argument for the FUSE implementation in
|
|
[../plan/roadmap.md](../plan/roadmap.md).
|
|
- **The SchFS round-trip test needs symlinks and hard links**, so it skips on a
|
|
Git Bash host and only runs on Unix.
|
|
- **`make ports` does not work on Windows.** musl, toybox and mksh must be built
|
|
once on a Unix host and staged.
|