SchOS/docs/history/schfs.md

SchFS — how it was built

The filesystem as it is now is ../architecture/schfs.md. This is the record of building it: the phase plan, what each phase actually produced, and the bugs that cost real time.

All five phases are complete. The plan is kept because the reasoning in it is still the reasoning behind the format, and because the ordering argument — do not start phase 2 before phase 1 is solid, the B-tree is the load-bearing piece — is the transferable part.

---

The phases, as they were planned and as they went

Each phase leaves a bootable system. Do not start phase 2 before phase 1 is solid — the B-tree is the load-bearing piece and everything else assumes it.

Phase 1 — the B-tree, in userland first. Write the CoW B-tree as a plain C library with a block-device abstraction, and test it on a host file with a fuzz/torture harness (random insert/delete/lookup, simulated crash at every write point, verify the tree always mounts). This is the one component where a subtle bug is unfixable later, and testing it in the kernel is far harder. It is also portable straight into tools/mkfs.

Phase 2 — read-only mount. DONE. New magic, uberblock scan, tree walk, inode and dirent key types. tools/mkschfs writes images; the kernel only reads them. The system boots from a ramdisk root (see ../guides/layout.md and the ramdisk work) so a broken disk cannot stop you from booting.

Went further than planned in two places, both cheaply: data blocks carry checksums too (a CSUM key type, as btrfs does — without it "end-to-end checksums" would cover only metadata, which is the part least likely to rot), and extents landed early because reserving a file's blocks in one go was simpler than not doing it.

Not yet done: SchFS files are not reachable through open(2). The mount is observable through /proc/schfs and through the kernel API in schfs_int.h, but wiring it into namei/readi means the VFS work that phase 3 needs anyway.

Phase 3 — writes, transaction groups. DONE. Every schfs_k* call that changes anything is one transaction group, committed by writing a new uberblock before it returns. The full write API — create, link, unlink, rename, write at an arbitrary offset, truncate — lives in objw.c and is exercised by tools/schfs_objtest.c.

Two things went differently from the plan above, both for reasons worth recording.

Free space is not tracked as tree keys. Allocating from a free map that lives in the tree mutates the tree, which allocates, which changes the map; btrfs resolves that with a fixpoint loop and ZFS with log-structured space maps. SchFS sidesteps it: the committed tree already says exactly which blocks are live, so the map is derived by walking the tree once at mount and held only in RAM. It cannot disagree with the filesystem, because it is computed from it, and a crash raises no question at all. The cost is mount time proportional to metadata size, and a ceiling on volume size set by how much bitmap fits in memory. If mount time ever hurts, the map can be cached on disk purely as a hint, validated against the generation and discarded on a mismatch — an addition, not a format change. See kernel/fs/schfs/space.c.

Data blocks needed a generation stamp too. The B-tree already rewrites its own nodes in place within a group, since a block that no committed root can reach is nobody's fallback. Data blocks had no such marker, so every hundred-byte write copied a whole 4 KB block and split an extent record around it. schfs_extent gained a 64-bit gen, and the same rule now applies to file data.

Phase 3½ — reachable through open(2). DONE. SchFS is a mounted filesystem:

mkdir /mnt
mount -t schfs /dev/hdb /mnt
ls /mnt ; echo hi > /mnt/tmp/w ; cat /mnt/tmp/w ; mkdir /mnt/tmp/d

The VFS grew a real dispatch layer rather than a set of if (dev == SCHFS_DEV) checks. struct fs_ops (include/fs/file.h) is one table per filesystem, chosen by iget() from the device number and hung off the inode; v1's implementations keep their bodies and are simply named v1_*. Adding a third filesystem is a new table and nothing else. kernel/fs/schfs/vfs.c is SchFS's, and it is pure translation — object ids in, inodes out — because all the filesystem logic already lives below it.

Three things had to change on the way, none of them SchFS's fault:

• struct inode.inum and .size were 32-bit, capping objects at 2³² and files at 4 GiB on a filesystem that promises far more. Both are uint64 now, as are readi/writei offsets and struct file.off.
• struct dirent carries a 16-bit inum and a 14-byte name, so it cannot represent a SchFS entry at all. getdents64 is now an fs_op: each filesystem emits the record stream itself, and f->off is whatever position it needs to resume. v1 walks its hash buckets; SchFS walks a key range and synthesises "." and "..", which it does not store.
• Mounting is a substitution made during path resolution (mount_add / mount_cross in fs.c). Nothing on either disk records it. The reverse direction matters as much as the forward one: .. at a mounted root has to leave the mount, and the filesystem underneath has no idea it is mounted anywhere.

Three long-standing bugs surfaced the moment real commands ran against it, all in the shared VFS rather than in SchFS:

• include/uapi/fcntl.h used xv6's open flags while every program in the userland sends Linux's. O_APPEND (0x400) landed on O_TRUNC, so cmd >> file truncated it; O_TRUNC (0x200) landed on O_CREATE, so cmd > file left whatever was past the new end in place. Nothing about either pointed at a header.
• sys_openat silently dropped O_TRUNC and O_APPEND while sys_open honoured both, so the same redirection behaved differently depending on which entry point libc happened to call.
• rmdir had no Linux syscall number, so every Linux-ABI program got ENOSYS.

Phase 3¾ — durability, and the cost of it. DONE. Three things that the earlier phases had assumed rather than arranged.

The commit barrier. Copy-on-write promises that a crash lands on either the old tree or the new one, but only if the blocks a root points at reach the medium before the block naming that root. Nothing enforced that: bwrite order is not platter order on any drive with a write cache. struct schfs_bdev gained a bflush hook, schfs_vol_commit calls it either side of the uberblock write, and the kernel implements it as "drain this device's dirty buffers, then ATA FLUSH CACHE" (ideflush, and SCSI SYNCHRONIZE CACHE for a USB stick).

Falling back through the ring. Mount took the highest-generation uberblock that validated, checked the tree it named, and gave up if that failed — with up to 127 older, entirely intact roots sitting on the disk. It now walks the ring newest-first and adopts the first root it can actually read, counting what it skipped into vol->recovered so the rollback is reported rather than silent.

Deferred writes, and brackets. The B-tree seals and writes a node on every record it inserts, so a leaf collecting one file's data checksums went to the disk once per 4 KB of that file. The kernel's block hook marks buffers dirty instead (bdirty), and the commit barrier is what makes that safe. On its own that changes little, because a commit contains a flush and every schfs_k* call was one commit — so schfs_ktx_open/schfs_ktx_close bracket a run of calls into one group, and schfs_vfs_write wraps a whole write(2) in one.

Measured over 1 MB of file data, in block writes reaching the device:

	before	after
committing every 4 KB	1947	1410
one commit for the megabyte	1798	267

267 device writes for 256 blocks of data. Note that the two changes are worth almost nothing separately — deferral cannot help while every commit flushes, and batching cannot help while every write goes straight to the disk — which is why the first attempt at this measured an 8% difference and concluded batching was not the problem.

Phase 3⅞ — where the time actually went. DONE. The filesystem was correct and durable and the system still felt slow: ls /bin took the better part of a second. Two causes, neither of them the disk.

Measured first, through /proc/fsstat (counters added for exactly this). The decisive number was 398,999 block reads against 445 actual disk misses — 99.9% of all block reads were re-reading a block already in the buffer cache. So the disk was never the problem, and every plan that started with "cache more" or "read ahead" would have been aimed at 1% of the cost.

The cursor re-read its own buffer. cur_load called node_read unconditionally, and schfs_cursor_val calls cur_load on every access to the current record — deliberately, so that a callback disturbing the shared block buffer cannot corrupt a directory walk. But node_read copies 4096 bytes out of the buffer cache and then runs CRC32c over all 4096. Listing a 243-entry directory paid both once per entry. The cursor now remembers which block its buffer holds and skips the reload, cross-checked against the block header (so a second cursor sharing the page cannot fool it) and against a global epoch that every mutation bumps (so a block rewritten in place cannot be served stale).

The checksum was bit-at-a-time. schfs_crc32c did eight shifts per byte — 32,768 iterations to checksum one block — to stay usable before any initialisation. It builds a 1 KB table on first use now, which keeps that property and is eight times faster. This only became the top cost after the cursor fix, which is the usual shape: removing the largest item promotes whatever was beneath it.

cycles, one session	before	after
getdents	26.8 G	0.14 G (196×)
read	26.4 G	5.5 G (4.8×)
exec	18.7 G	4.0 G (4.7×)
block reads	398,999	127,713

ls /bin went from ~400 ms to ~20 ms. Verified by reading an 802 KB binary through the whole path and checking its md5 against the host's copy.

There was also a third of the problem that was not the kernel at all: make qemu ran QEMU without -enable-kvm, so every instruction was emulated. The run targets probe /dev/kvm now. Measure against the KVM number or you are measuring TCG.

Phase 4 — extents with refcounts. DONE. cp that costs nothing.

A reference count per run of blocks, in its own tree (SCHFS_KEY_EXTREF under SCHFS_EXTREF_OBJECTID) rather than in the file's own extent record — because a count kept per file cannot be decremented by the other file sharing the run. schfs_file_reflink copies one extent record per run and bumps the count; both files stay ordinary and writable, and the first write to either copies only the run it touches. Reachable from userland as FICLONE, Linux's ioctl, so cp --reflink works unmodified; cmd/reflink is a two-line caller of it.

Measured on the running system: cloning a 411 KB file moved df by 24 KB.

Three things had to be right, and each was wrong first:

• The count is "how many extent records point into this run", not "how many files". Splitting a shared extent around a written block leaves a head and a tail, which is one more reference than before. Missing that leaked four blocks per split.
• A file's extent may point into the middle of a run. Records are keyed by the block a run was allocated at, so every lookup is a range query (extref_find), not an exact one. An exact lookup finds nothing for a split tail, "nothing" means unshared, and the tail is then freed while another file still reaches it.
• The space scan has to walk the reference records too. A file truncated inside a shared run describes fewer blocks than exist; the rest would be derived as free while still referenced, and the allocator would hand out a block that a later release frees a second time.

None of those is visible until every holder of a run is deleted, which is why the test that matters is the least glamorous one: after deleting every file that shared a run, free space must be back to exactly what it was.

Phase 5 — attribute index and query. DONE. The reason the design was one B-tree with many key types from the start: the "database filesystem" feature turned out to be kernel/fs/schfs/attr.c and nothing else.

An attribute is a named value on an object, stored under (ino, ATTR, hash-of-name) and packed exactly as directory entries are — same collision argument, same shape. An attribute may be indexed, and an indexed attribute gets a second set of records keyed by value: (hash-of-name, INDEX, hash-of-value) holding the object ids that carry it. So "every object whose tag is holiday" is one tree descent instead of a walk of every object.

There is no query language, no parser and no planner, and §6 above explains why: the useful half of "query the filesystem" is the index, and WinFS spent three years and a large team on the other interpretation.

schattr index tag                 # declare the index
schattr set /tmp/a tag holiday
schattr find tag holiday          # one descent, whatever the volume holds

Four decisions worth keeping:

• Which attributes are indexed is cached in the volume, read once at mount. Every setxattr has to ask, and the answer is almost always no; a tree descent to say so would make every attribute write pay for a feature it is not using.
• The value is stored in the index entry, not just its hash. Two values can hash alike, and a query that returned the wrong file would be far worse than a slow one.
• An index covers values written after it is declared. There is no reindexing pass. Saying so is better than a half-truth that makes a query answer confidently and wrongly.
• A query on an unindexed attribute is refused, not answered by scanning. A query that silently degrades into a full walk teaches you to use it where it will not scale.

Reachable as setxattr/getxattr/removexattr at Linux's syscall numbers, so anything that already knows how to attach metadata needs no changes, plus two native calls (schfs_index, schfs_query) that have no Linux equivalent because asking a filesystem a question does not.

Phase 5 — attribute index and query. See below.

---

What was reused from v1

Not starting from zero:

• struct schfs_extent is already defined in include/fs/fs.h and unused. The extent concept is already sketched; v2 gives it refcounts and 64-bit fields.
• CRC32c checksum fields already exist in dinode and superblock, and mkfs already computes them. Same algorithm, relocated into parent pointers.
• kernel/fs/log.c is a working transaction log with begin_op/end_op call sites throughout the FS. Under CoW the log is not needed for consistency, but begin_op/end_op are exactly the right places to bracket a transaction group, so the call sites survive even though the implementation is replaced.
• The buffer cache (bio.c) is format-agnostic and needs no change beyond the block-size constant.

---

What it was estimated to cost

Phases 1-3 are roughly 3,000-4,000 lines and are the bulk of the work. Phase 1 alone (B-tree + torture harness) is worth doing carefully and unhurriedly; every later phase compounds on it.

This is the largest single piece of work in SchOS. It is also entirely ordinary filesystem engineering with excellent published references (Giampaolo for BeFS; the btrfs design papers; the ZFS uberblock/checksum-in- parent scheme) — there is no research risk here, only volume.

Do this before the installer. The installer's entire job is to create and populate this format. Building it against v1 means building it twice.

---

Things learned the hard way

Recorded because each one cost real time and none is obvious from the code.

• The root's object id must be reserved. next_objectid started at SCHFS_FIRST_OBJECTID, which is the root's id, so the first directory created silently became the root. It then contained itself, and every tree walk recursed until the stack ran out. schfs_fsck --check now detects directory loops explicitly rather than crashing.
• Cursor buffers alias. A range callback gets a pointer into the cursor's block buffer, and almost every readdir caller immediately looks the entry up, which overwrites it. Directory iteration copies each entry out before calling back. Extracting a tree with directories reparented under their own children is what this looks like when it goes wrong.
• A 4096-byte block does not fit on a 4096-byte kernel stack. The cursor exists precisely to avoid one block buffer per tree level, but schfs_vol_open still kept a whole uberblock as a local and blew the stack on the first boot. The kstack canary caught it cleanly.
• Extents only merge if you allocate the run first. Writing an extent record allocates blocks for its own copy-on-write path, so allocating data one block at a time guarantees the blocks are never adjacent and every block needs its own extent. schfs_file_reserve claims the whole run up front; an 822 KB binary is one extent instead of 201.
• Freed blocks are reusable during a build. Copy-on-write forbids reusing a released block because the previously committed tree still points at it — but a build has committed nothing. Recycling them cut the test image from 3596 blocks to 374.
• Reproducibility dies from atime. mkschfs reads every source file, which updates its atime, so copying atime into the image made two builds of the same tree differ. All three timestamps come from mtime.
• An optional hook must still be initialised. struct schfs_bdev gained bflush, and every host tool built one on the stack field by field. The tools that did not set the new field handed schfs_vol_commit a garbage pointer to test and call, which segfaulted at the first commit. Each of those helpers memsets the struct first now — the fix is not "remember to set it".
• Deferring writes makes an all-dirty buffer cache normal. bget panicked with "no buffers" when it could find nothing clean to recycle, which was correct while a dirty buffer meant "the log has not committed". It now writes the least recently used victim out and takes it; a transaction touching more than NBUF blocks reaches that state long before it commits.
• The two performance fixes are worthless apart. Deferring tree writes buys nothing while every commit flushes, and batching commits buys nothing while every write goes straight to the disk. Measuring either alone says "this is not where the time goes", and both measurements are correct.
• fsid_t is eight bytes. struct statfs declared f_fsid as int64[2], which is one slot too long, so everything after it landed in the wrong place: userland read f_frsize out of f_namelen and got 255. df then computed in units of 255 bytes and called a 32 MB root filesystem 1992 KB. It survived because every field in that struct used to be an invented constant, so no number in it had ever been checked against reality.

---

The one about the tests themselves

tools/posix/mutate.sh injects six deliberate bugs and fails if any goes undetected.

It exists because a suite that passes is not evidence until something has checked that it can fail. The most valuable single assertion in the whole SchFS suite is also the least glamorous: after deleting everything, free space must be back to exactly what it was on an empty volume. Every other test still passes on a filesystem that leaks a block on every delete.