SchOS/docs/architecture/schfs.md

# SchFS — design

**Status: phases 1 and 2 are done.** The copy-on-write B-tree, the volume layer,
the object layer, a working image writer and a checker all exist, and the kernel
mounts a SchFS volume read-only at boot. Phase 3 (writes and transaction groups)
is next. See §8 for exactly what is built and how to run its tests.

Supersedes the on-disk format described in `include/fs.h`. Keeps the name SchFS;
the on-disk magic changes so v1 images are detected and rejected rather than
silently misread.

The brief was "the best aspects of BeFS, ZFS and btrfs." Those three disagree
with each other in places, so this document says explicitly what is taken from
each and what is deliberately left behind.

## 1. What each one is actually good at

**ZFS — never be inconsistent.** Copy-on-write: no block that is part of the
live tree is ever overwritten. An update writes new blocks, then atomically
flips a single root pointer. A crash therefore lands on either the old tree or
the new one — never a half-written mix. Consequence: **no fsck**, ever. Paired
with end-to-end checksums where a block's checksum is stored in its *parent*
pointer, not in the block itself, which makes the whole tree self-validating and
catches misdirected writes and bit rot that a self-stored checksum cannot.

**btrfs — one data structure, reused everywhere.** btrfs has essentially one
engine: a copy-on-write B-tree. Inodes, directory entries, extent maps, free
space, and snapshots are all just different key types in that same tree. For a
small team this is the single most important idea in the whole design: you write
and debug *one* tricky data structure, then get five subsystems from it. Also
takes extents with **reference counts**, which is what makes snapshots and
reflink copies fall out almost for free.

**BeFS — the part that feels like a database, at a size one person can build.**
BeFS is a 64-bit journaled filesystem whose distinguishing feature is B+tree
**indexes over file attributes**, with a query interface on top. That is the
"SQL-like" capability, and unlike WinFS it actually shipped. Critically, BeFS is
also the right *scale* model: a small team, a readable codebase, and a book
(Giampaolo, *Practical File System Design*, free online) that describes it
end to end.

## 2. What is deliberately not taken

- **No ZFS storage pools / RAID-Z / vdevs.** Multi-disk redundancy is a large
  subsystem that solves a problem SchOS does not have yet.
- **No btrfs-style rebalancing or online format conversion.** Enormous
  complexity, and it is the historical source of btrfs's worst bugs.
- **No actual SQL.** No parser, no query planner, no relational engine. See §6 —
  the useful part of "database filesystem" is the *index*, not the language.
  WinFS spent 2003-2006 and a large team on the other interpretation and
  shipped nothing.
- **No deduplication.** ZFS's dedup is famous for needing more RAM than anyone
  expects. Extent refcounts give reflink/snapshot benefits without it.

## 3. On-disk shape

```
+--------------------+  block 0        boot block (untouched by the FS)
+--------------------+  blocks 1..N    uberblock array (128 slots, rotated)
+--------------------+
|  CoW B-tree        |  one tree, many key types:
|                    |    (ino, INODE,  0)      -> inode record
|                    |    (ino, DIRENT, hash)   -> directory entry
|                    |    (ino, EXTENT, offset) -> extent mapping
|                    |    (ino, ATTR,   nameid) -> extended attribute
|                    |    (idx, INDEX,  key)    -> attribute index entry
+--------------------+
|  free space        |  extent-based, tracked as tree keys, not a bitmap
+--------------------+
```

**Uberblock array (from ZFS).** 128 slots written round-robin, each carrying a
transaction number and a checksum. Mount scans all slots and takes the
highest-numbered one that validates. This is what makes crash recovery
"pick the newest good root" instead of "replay a journal."

**Everything is one B-tree (from btrfs).** Keyed by
`(objectid, type, offset)`, 24 bytes. Adding a feature usually means adding a
key type, not a new on-disk structure.

**Checksums live in the parent (from ZFS).** Each internal node stores, for
every child pointer, that child's CRC32c. A block that is read is verified
against the pointer that led to it. v1 already chose CRC32c and already has
checksum fields in `dinode` and `superblock` — same algorithm, better placement.

### Widths — every v1 limit that has to die

| v1 | limit | v2 |
|---|---|---|
| `dirent.inum` is `ushort` | 65,535 files, total | `uint64` object ids |
| `dinode.size` is `uint` | 4 GB max file | `uint64` |
| `uint` block numbers, BSIZE 1024 | 4 TB volume | `uint64` blocks, 4096-byte |
| `DIRSIZ` 14 | 14-char filenames | variable, up to 255 |
| fixed hash table per dir | every dir ≥113 KB, capped at 119 KB | B+tree, pay per entry |
| `addrs[NDIRECT+3]` | block-at-a-time | extents with refcounts |

Block size moves 1024 → 4096 to match modern media and make extents worthwhile.

## 4. What already exists and gets reused

Not starting from zero:

- **`struct schfs_extent`** is already defined in `include/fs.h` and unused.
  The extent concept is already sketched; v2 gives it refcounts and 64-bit
  fields.
- **CRC32c checksum fields** already exist in `dinode` and `superblock`, and
  mkfs already computes them. Same algorithm, relocated into parent pointers.
- **`kernel/fs/log.c`** is a working transaction log with `begin_op`/`end_op`
  call sites throughout the FS. Under CoW the log is not needed for consistency,
  but `begin_op`/`end_op` are exactly the right places to bracket a transaction
  group, so the *call sites* survive even though the implementation is replaced.
- **The buffer cache** (`bio.c`) is format-agnostic and needs no change beyond
  the block-size constant.

## 5. Phasing

Each phase leaves a bootable system. Do not start phase 2 before phase 1 is
solid — the B-tree is the load-bearing piece and everything else assumes it.

**Phase 1 — the B-tree, in userland first.** Write the CoW B-tree as a plain C
library with a block-device abstraction, and test it on a *host* file with a
fuzz/torture harness (random insert/delete/lookup, simulated crash at every
write point, verify the tree always mounts). This is the one component where a
subtle bug is unfixable later, and testing it in the kernel is far harder. It is
also portable straight into `tools/mkfs`.

**Phase 2 — read-only mount. DONE.** New magic, uberblock scan, tree walk, inode
and dirent key types. `tools/mkschfs` writes images; the kernel only reads them.
The system boots from a ramdisk root (see `docs/layout.md` and the ramdisk work)
so a broken disk cannot stop you from booting.

Went further than planned in two places, both cheaply: data blocks carry
checksums too (a `CSUM` key type, as btrfs does — without it "end-to-end
checksums" would cover only metadata, which is the part least likely to rot),
and extents landed early because reserving a file's blocks in one go was
simpler than not doing it.

Not yet done: SchFS files are not reachable through `open(2)`. The mount is
observable through `/proc/schfs` and through the kernel API in `schfs_int.h`,
but wiring it into `namei`/`readi` means the VFS work that phase 3 needs anyway.

**Phase 3 — writes, transaction groups. DONE.** Every `schfs_k*` call that
changes anything is one transaction group, committed by writing a new uberblock
before it returns. The full write API — create, link, unlink, rename, write at
an arbitrary offset, truncate — lives in `objw.c` and is exercised by
`tools/schfs_objtest.c`.

Two things went differently from the plan above, both for reasons worth
recording.

*Free space is not tracked as tree keys.* Allocating from a free map that lives
in the tree mutates the tree, which allocates, which changes the map; btrfs
resolves that with a fixpoint loop and ZFS with log-structured space maps. SchFS
sidesteps it: the committed tree already says exactly which blocks are live, so
the map is **derived** by walking the tree once at mount and held only in RAM.
It cannot disagree with the filesystem, because it is computed from it, and a
crash raises no question at all. The cost is mount time proportional to metadata
size, and a ceiling on volume size set by how much bitmap fits in memory. If
mount time ever hurts, the map can be cached on disk purely as a hint,
validated against the generation and discarded on a mismatch — an addition, not
a format change. See `kernel/fs/schfs/space.c`.

*Data blocks needed a generation stamp too.* The B-tree already rewrites its own
nodes in place within a group, since a block that no committed root can reach is
nobody's fallback. Data blocks had no such marker, so every hundred-byte write
copied a whole 4 KB block and split an extent record around it. `schfs_extent`
gained a 64-bit `gen`, and the same rule now applies to file data.

**Phase 3½ — reachable through `open(2)`. DONE.** SchFS is a mounted filesystem:

```
mkdir /mnt
mount -t schfs /dev/hdb /mnt
ls /mnt ; echo hi > /mnt/tmp/w ; cat /mnt/tmp/w ; mkdir /mnt/tmp/d
```

The VFS grew a real dispatch layer rather than a set of `if (dev == SCHFS_DEV)`
checks. `struct fs_ops` (include/file.h) is one table per filesystem, chosen by
`iget()` from the device number and hung off the inode; v1's implementations
keep their bodies and are simply named `v1_*`. Adding a third filesystem is a
new table and nothing else. `kernel/fs/schfs/vfs.c` is SchFS's, and it is pure
translation — object ids in, inodes out — because all the filesystem logic
already lives below it.

Three things had to change on the way, none of them SchFS's fault:

- `struct inode.inum` and `.size` were 32-bit, capping objects at 2³² and files
  at 4 GiB on a filesystem that promises far more. Both are `uint64` now, as are
  `readi`/`writei` offsets and `struct file.off`.
- `struct dirent` carries a 16-bit inum and a 14-byte name, so it cannot
  represent a SchFS entry at all. `getdents64` is now an fs_op: each filesystem
  emits the record stream itself, and `f->off` is whatever position it needs to
  resume. v1 walks its hash buckets; SchFS walks a key range and synthesises
  "." and "..", which it does not store.
- Mounting is a substitution made during path resolution (`mount_add` /
  `mount_cross` in `fs.c`). Nothing on either disk records it. The reverse
  direction matters as much as the forward one: `..` at a mounted root has to
  leave the mount, and the filesystem underneath has no idea it is mounted
  anywhere.

Three long-standing bugs surfaced the moment real commands ran against it, all
in the shared VFS rather than in SchFS:

- `include/fcntl.h` used xv6's open flags while every program in the userland
  sends Linux's. `O_APPEND` (0x400) landed on `O_TRUNC`, so `cmd >> file`
  *truncated* it; `O_TRUNC` (0x200) landed on `O_CREATE`, so `cmd > file` left
  whatever was past the new end in place. Nothing about either pointed at a
  header.
- `sys_openat` silently dropped `O_TRUNC` and `O_APPEND` while `sys_open`
  honoured both, so the same redirection behaved differently depending on which
  entry point libc happened to call.
- `rmdir` had no Linux syscall number, so every Linux-ABI program got ENOSYS.

**Phase 4 — extents with refcounts.** Enables reflink copies and snapshots
nearly for free.

**Phase 5 — attribute index and query.** See below.

## 6. The "SQL" part, honestly

What you want from "works like SQL" is: *ask a question about a lot of files and
get an answer fast, without walking every directory*. That is an **index**, not
a query language. BeFS's design:

- Files carry typed attributes (`string`, `int`, `time`, …).
- An attribute can be *indexed*: a dedicated B+tree keyed by attribute value,
  mapping value → object ids.
- A query is a predicate over indexed attributes, evaluated by intersecting
  index ranges.

In SchFS this is just another key type — `(idx, INDEX, key)` — in the tree
that already exists. That is the whole reason to build phases 1-4 the btrfs way:
the "database filesystem" feature becomes a few hundred lines on top of
machinery already written and debugged, rather than a separate engine.

A concrete `query()` syscall and a `find`-like tool come after that, and are
small by comparison.

## 7. Honest cost

Phases 1-3 are roughly 3,000-4,000 lines and are the bulk of the work. Phase 1
alone (B-tree + torture harness) is worth doing carefully and unhurriedly; every
later phase compounds on it.

This is the largest single piece of work in SchOS. It is also entirely
ordinary filesystem engineering with excellent published references
(Giampaolo for BeFS; the btrfs design papers; the ZFS uberblock/checksum-in-
parent scheme) — there is no research risk here, only volume.

**Do this before the installer.** The installer's entire job is to create and
populate this format. Building it against v1 means building it twice.

## 7½. Limits

Three different things get called "the limit", and they are worth keeping apart:
what the **on-disk format** can express, what **this implementation** can
currently mount, and what the **rest of SchOS** can carry. The smallest one is
what you actually hit.

### The format

| | limit | where it comes from |
|---|---|---|
| Volume | **64 ZiB** (2⁶⁴ × 4 KiB blocks) | `uint64` block numbers |
| File | **16 EiB** (2⁶⁴ bytes) | `schfs_inode.size` |
| Files per volume | **1.8 × 10¹⁹** | `uint64` object ids |
| Hard links per file | **4.3 × 10⁹** | `uint32 nlink` |
| Filename | **255 bytes** | `SCHFS_NAME_MAX` |
| Entries per directory | no fixed limit | one tree key per name hash |
| Extent | **64 ZiB** in one record | `uint64 nblocks` |
| Tree depth | **8 levels** | `SCHFS_MAX_LEVEL` |
| Block size | **4096 bytes**, fixed | `SCHFS_BLKSIZE`; a mismatch is refused at mount |
| Transaction groups | **1.8 × 10¹⁹** | `uint64` generation, never reused |

Directory *depth* is unbounded on disk; path resolution is what limits it.

### This implementation

| | limit | where it comes from |
|---|---|---|
| Volume, mounted | **64 GiB** | the in-RAM allocation bitmap: `SCHFS_SPACE_PAGES` (512) pages × 32768 blocks |
| Volume, via the buffer cache | **4 TiB** | `bio.c` addresses blocks with a `uint`, and one SchFS block spans four of them |
| RAM per mount | **8 KiB per GiB** of volume | two bitmaps, one bit per block each |
| Inline value in a leaf | **1016 bytes** | `SCHFS_MAX_INLINE` = a quarter of a node |
| Colliding names in one directory | ~**60**, sharing a hash | the packed list for one hash must fit in one inline value |
| Directory entries at 8-byte names | ~**42** per key, unlimited keys | as above |
| Records per leaf / children per node | **127 / 84** | 4 KiB minus a 32-byte header |
| Mount time | one pass over the metadata | the allocation map is derived, not stored |

The 64 GiB mount ceiling is the one that bites first, and it is one constant:
raising `SCHFS_SPACE_PAGES` to 4096 makes it 512 GiB at 64 KiB of pointers per
mount. It is set where it is because the page-pointer vector lives inside
`struct schfs_vol`, which the kernel holds as static storage.

### The rest of SchOS

These are VFS and driver limits rather than SchFS ones:

- `MAXPATH` caps a path at **128 bytes**.
- `NMOUNT` allows **8** mounts.
- The IDE driver is 28-bit LBA: **128 GiB** per disk.
- A single `read`/`write` call is capped at **4 GiB** (`n` is a `uint`); the
  file itself is not.

The two that used to be here — a 16-bit inode number in `struct dirent` and a
32-bit `inum`/`size` in `struct inode` — are gone. `getdents64` is a per-
filesystem operation now, so SchFS emits its own records with 64-bit inode
numbers and names up to 255 bytes, and the inode carries 64-bit numbers and
sizes throughout.

## 8. What exists now

```
include/schfs.h              on-disk format + compile-time layout assertions
kernel/fs/schfs/btree.{c,h}  the copy-on-write B-tree and its cursor
kernel/fs/schfs/schfs_int.h  volume + object layer interface
kernel/fs/schfs/space.c      allocation bitmap, derived from the tree at mount
kernel/fs/schfs/vol.c        uberblock ring: open, format, commit
kernel/fs/schfs/obj.c        read side: inodes, dirents, extents, path lookup
kernel/fs/schfs/objw.c       write side: create, link, unlink, rename, write
kernel/fs/schfs/kmount.c     kernel: block device, mount, transaction bracket
kernel/fs/schfs/vfs.c        the fs_ops table: SchFS behind open/read/write
tools/mkschfs.c              build an image from a directory tree
tools/schfs_fsck.c           verify, list, extract
tools/schfs_test.c           B-tree torture harness
tools/schfs_objtest.c        object layer: files, directories, space accounting
tools/posix/schfs_roundtrip.sh     end-to-end filesystem test
tools/posix/mutate.sh              mutation-tests the suites themselves
tools/windows/bootcheck.ps1          boots under QEMU and drives the shell
```

Everything except `kmount.c` is compiled *both* into the kernel and into the
host tools, so the writer and the reader cannot drift apart: there is one
implementation of the format and nothing to disagree with.

### Running the tests

```
make schfs-check         all three suites
make schfs-test          B-tree only
make schfs-objtest       object layer only
make schfs-image-test    round-trip only
```

`make schfs-check` after touching anything under `kernel/fs/schfs`.

The round-trip's oracle is the host's own filesystem: an image is built from a
directory tree, extracted with a separate program, and `diff -r`'d against the
original. It also asserts hard-link counts (which `diff` cannot see), checks
that rebuilding produces byte-identical output, and runs the whole thing a
second time with the name hash cut to 6 bits so the packed-dirent collision path
— which a 64-bit hash would never reach — is actually exercised.

`schfs_objtest` runs against a memory-backed volume, so a full
create/write/delete/remount cycle takes milliseconds. Its most important check
is the least glamorous: after deleting everything, free space must be back to
exactly what it was on an empty volume. Every other test in the suite still
passes on a filesystem that leaks a block on every delete.

`tools/posix/mutate.sh` injects six deliberate bugs and fails if any of them goes
undetected. It is how the suites are kept honest; see the note in §9.

### Verifying on real hardware

```
make ramroot             root from RAM, so device 1 is free
make build/schfs.img
make qemu-schfs          then, at the shell:  cat /proc/schfs
```

`.\run.ps1 -SchFS` does the same on Windows, and `tools\bootcheck.ps1 -SchFS`
does it unattended and asserts on the transcript.

## 9. Things learned the hard way

Recorded because each one cost real time and none is obvious from the code.

- **The root's object id must be reserved.** `next_objectid` started at
  `SCHFS_FIRST_OBJECTID`, which *is* the root's id, so the first directory
  created silently became the root. It then contained itself, and every tree
  walk recursed until the stack ran out. `schfs_fsck --check` now detects
  directory loops explicitly rather than crashing.
- **Cursor buffers alias.** A range callback gets a pointer into the cursor's
  block buffer, and almost every readdir caller immediately looks the entry up,
  which overwrites it. Directory iteration copies each entry out before calling
  back. Extracting a tree with directories reparented under their own children
  is what this looks like when it goes wrong.
- **A 4096-byte block does not fit on a 4096-byte kernel stack.** The cursor
  exists precisely to avoid one block buffer per tree level, but
  `schfs_vol_open` still kept a whole uberblock as a local and blew the stack on
  the first boot. The kstack canary caught it cleanly.
- **Extents only merge if you allocate the run first.** Writing an extent record
  allocates blocks for its own copy-on-write path, so allocating data one block
  at a time guarantees the blocks are never adjacent and every block needs its
  own extent. `schfs_file_reserve` claims the whole run up front; an 822 KB
  binary is one extent instead of 201.
- **Freed blocks are reusable during a build.** Copy-on-write forbids reusing a
  released block because the previously committed tree still points at it — but
  a build has committed nothing. Recycling them cut the test image from 3596
  blocks to 374.
- **Reproducibility dies from atime.** mkschfs reads every source file, which
  updates its atime, so copying atime into the image made two builds of the same
  tree differ. All three timestamps come from mtime.