SchOS/docs/architecture/schfs.md

# SchFS

The root filesystem. A copy-on-write B-tree with an uberblock ring, 64-bit
object ids, checksummed metadata and extents.

**It is the root.** `forkret()` mounts it before PID 1 has a working directory
and panics if it is not there. All five design phases are complete: the B-tree,
read-only mount, the write path and transaction groups, the commit barrier and
uberblock fallback, extent refcounts and reflink, and attributes with an index
and a query.

The phases themselves, and the bugs they produced, are in
[../history/schfs.md](../history/schfs.md). The VFS it plugs into is
[filesystem.md](filesystem.md).

---

## The argument in one line

A crash leaves either the old tree or the new one, so **there is nothing to
repair** — and free space is derived from the tree rather than stored beside it,
so **there is no second structure to disagree with the first**.

`/proc/schfs` shows the generation counter advancing as writes commit.

---

## What was taken, from where

**ZFS — never be inconsistent.** Copy-on-write: no block that is part of the
live tree is ever overwritten. An update writes new blocks, then atomically
flips a single root pointer. A crash therefore lands on either the old tree or
the new one — never a half-written mix. Consequence: **no fsck**, ever. Paired
with end-to-end checksums where a block's checksum is stored in its *parent*
pointer, not in the block itself, which makes the whole tree self-validating and
catches misdirected writes and bit rot that a self-stored checksum cannot.

**btrfs — one data structure, reused everywhere.** btrfs has essentially one
engine: a copy-on-write B-tree. Inodes, directory entries, extent maps, free
space, and snapshots are all just different key types in that same tree. For a
small team this is the single most important idea in the whole design: you write
and debug *one* tricky data structure, then get five subsystems from it. Also
takes extents with **reference counts**, which is what makes snapshots and
reflink copies fall out almost for free.

**BeFS — the part that feels like a database, at a size one person can build.**
BeFS is a 64-bit journaled filesystem whose distinguishing feature is B+tree
**indexes over file attributes**, with a query interface on top. That is the
"SQL-like" capability, and unlike WinFS it actually shipped. Critically, BeFS is
also the right *scale* model: a small team, a readable codebase, and a book
(Giampaolo, *Practical File System Design*, free online) that describes it
end to end.

## What was deliberately not taken

- **No ZFS storage pools / RAID-Z / vdevs.** Multi-disk redundancy is a large
  subsystem that solves a problem SchOS does not have yet.
- **No btrfs-style rebalancing or online format conversion.** Enormous
  complexity, and it is the historical source of btrfs's worst bugs.
- **No actual SQL.** No parser, no query planner, no relational engine. See *Attributes, indexes and the query* below —
  the useful part of "database filesystem" is the *index*, not the language.
  WinFS spent 2003-2006 and a large team on the other interpretation and
  shipped nothing.
- **No deduplication.** ZFS's dedup is famous for needing more RAM than anyone
  expects. Extent refcounts give reflink/snapshot benefits without it.

## On-disk shape

```
+--------------------+  block 0        boot block (untouched by the FS)
+--------------------+  blocks 1..N    uberblock array (128 slots, rotated)
+--------------------+
|  CoW B-tree        |  one tree, many key types:
|                    |    (ino, INODE,  0)      -> inode record
|                    |    (ino, DIRENT, hash)   -> directory entry
|                    |    (ino, EXTENT, offset) -> extent mapping
|                    |    (ino, ATTR,   nameid) -> extended attribute
|                    |    (idx, INDEX,  key)    -> attribute index entry
+--------------------+
|  free space        |  extent-based, tracked as tree keys, not a bitmap
+--------------------+
```

**Uberblock array (from ZFS).** 128 slots written round-robin, each carrying a
transaction number and a checksum. Mount scans all slots and takes the
highest-numbered one that validates. This is what makes crash recovery
"pick the newest good root" instead of "replay a journal."

**Everything is one B-tree (from btrfs).** Keyed by
`(objectid, type, offset)`, 24 bytes. Adding a feature usually means adding a
key type, not a new on-disk structure.

**Checksums live in the parent (from ZFS).** Each internal node stores, for
every child pointer, that child's CRC32c. A block that is read is verified
against the pointer that led to it. v1 already chose CRC32c and already has
checksum fields in `dinode` and `superblock` — same algorithm, better placement.

### Widths — every v1 limit that has to die

| v1 | limit | v2 |
|---|---|---|
| `dirent.inum` is `ushort` | 65,535 files, total | `uint64` object ids |
| `dinode.size` is `uint` | 4 GB max file | `uint64` |
| `uint` block numbers, BSIZE 1024 | 4 TB volume | `uint64` blocks, 4096-byte |
| `DIRSIZ` 14 | 14-char filenames | variable, up to 255 |
| fixed hash table per dir | every dir ≥113 KB, capped at 119 KB | B+tree, pay per entry |
| `addrs[NDIRECT+3]` | block-at-a-time | extents with refcounts |

Block size moves 1024 → 4096 to match modern media and make extents worthwhile.

## Attributes, indexes, and the query

What you want from "works like SQL" is: *ask a question about a lot of files and
get an answer fast, without walking every directory*. That is an **index**, not
a query language. BeFS's design:

- Files carry typed attributes (`string`, `int`, `time`, …).
- An attribute can be *indexed*: a dedicated B+tree keyed by attribute value,
  mapping value → object ids.
- A query is a predicate over indexed attributes, evaluated by intersecting
  index ranges.

In SchFS this is just another key type — `(idx, INDEX, key)` — in the tree
that already exists. That is the whole reason the tree was built the btrfs way:
the "database filesystem" feature becomes a few hundred lines on top of
machinery already written and debugged, rather than a separate engine.

A concrete `query()` syscall and a `find`-like tool come after that, and are
small by comparison.

## Limits

Three different things get called "the limit", and they are worth keeping apart:
what the **on-disk format** can express, what **this implementation** can
currently mount, and what the **rest of SchOS** can carry. The smallest one is
what you actually hit.

### The format

| | limit | where it comes from |
|---|---|---|
| Volume | **64 ZiB** (2⁶⁴ × 4 KiB blocks) | `uint64` block numbers |
| File | **16 EiB** (2⁶⁴ bytes) | `schfs_inode.size` |
| Files per volume | **1.8 × 10¹⁹** | `uint64` object ids |
| Hard links per file | **4.3 × 10⁹** | `uint32 nlink` |
| Filename | **255 bytes** | `SCHFS_NAME_MAX` |
| Entries per directory | no fixed limit | one tree key per name hash |
| Extent | **64 ZiB** in one record | `uint64 nblocks` |
| Tree depth | **8 levels** | `SCHFS_MAX_LEVEL` |
| Block size | **4096 bytes**, fixed | `SCHFS_BLKSIZE`; a mismatch is refused at mount |
| Transaction groups | **1.8 × 10¹⁹** | `uint64` generation, never reused |

Directory *depth* is unbounded on disk; path resolution is what limits it.

### This implementation

| | limit | where it comes from |
|---|---|---|
| Volume, mounted | **64 TiB** | the in-RAM allocation bitmap: `SCHFS_SPACE_DIRS` (1024) pointer pages × 512 bitmap pages × 32768 blocks |
| Volumes mounted at once | **16** | `SCHFS_VOL_SLOTS`; slot 0 is the root |
| Snapshots mounted at once | **8** | `SCHFS_SNAP_SLOTS` |
| RAM per mount | **32 KiB per GiB** of volume | one bit per block, plus a page per 128 MiB that has seen a free since the last commit; 32 MiB for 1 TiB |
| Metadata reserve | **1%** of the volume, 1–64 MiB | data and new names stop there with `ENOSPC`; see below |
| Changes lost in a crash | at most **5 s** of them | `SCHFS_COMMIT_TICKS`; `sync` and `fsync` commit at once |
| Inline value in a leaf | **1016 bytes** | `SCHFS_MAX_INLINE` = a quarter of a node |
| Colliding names in one directory | ~**60**, sharing a hash | the packed list for one hash must fit in one inline value |
| Directory entries at 8-byte names | ~**42** per key, unlimited keys | as above |
| Records per leaf / children per node | **127 / 84** | 4 KiB minus a 32-byte header |
| Mount time | one pass over the metadata | the allocation map is derived, not stored |

The mount ceiling was 64 GiB, which refused most laptop disks: the page
vector was one level of 512 pointers inside `struct schfs_vol`. It is two
levels now, the way a page table is, so the struct holds 1024 pointers to
pointer pages and the ceiling is 64 TiB for the same 8 KiB. Raising
`SCHFS_SPACE_DIRS` raises it further.

## Commits, full disks and deletes

**Transaction groups commit every 5 seconds**, not after every operation:
from the `schfsd` kernel thread, on `sync(2)` and `fsync(2)`, on unmount and
shutdown, and early when space is short (below). This is the contract ext4
(5 s), ZFS (5 s) and btrfs (30 s) have, and the crash argument is unchanged: a
commit is one uberblock write, so a crash lands on the last group whole. What
it changes is how much can be lost: at most the last few seconds, and nothing
an `fsync` returned for.

It used to commit before every `write(2)` returned, two cache flushes and an
uberblock each time. Inside one group a block the group already wrote is
changed in place, so a tree path rewritten a hundred times between commits
goes to the disk once; committing per call threw that away. 1 MB in 4 KB
writes went from 2.8 to 6.6 MB/s on AHCI under QEMU, and from 66 KB/s to
3.9 MB/s on the emulated IDE disk.

**A full disk stays mountable.** A copy-on-write tree needs new blocks to
change anything, deleting included, and an update that runs out halfway
leaves a half-built tree the next commit publishes. That is what filling the
disk did: the root volume would not mount again. So 1% of the volume (at least
1 MiB, at most 64 MiB) is kept for metadata, as btrfs's global reserve and
ZFS's slop space are. File data stops at it with `ENOSPC`, as a short write
of what fitted. New names stop at half of it. Tree updates in progress and
deletes may use the rest.

Blocks a group frees stay unusable until it commits, because the committed
tree still points at them. So a group commits early once those are a quarter
of what is free, rather than report `ENOSPC` on a disk with room on it.

**Deleting a large file goes in 4 MiB steps** from the end, each one a whole
operation the group may commit after, as ext4 restarts its transaction through
a long truncate. One 80 MB delete was 20000 checksum-record deletions in a
single operation, which on a full disk ran out of room halfway.

**A file is freed on its last close**, not when its last name goes. The VFS
unlinks the name, drops `nlink`, and frees the object when nothing holds it,
so a file removed while open still reads. One that was still open when the
machine went down has no names and no holder: the mount finds such objects
during the scan it already makes, and frees them, as ext4 does with its orphan
list.

**Snapshots no longer stop space being reused.** A snapshot holds the tree
committed before the group it was made in, so it can only hold blocks written
before that group. A freed block written in or after the newest snapshot's
group goes back to the free map at once (ZFS's birth-time rule). The older
ones are kept until a mount works out which snapshots still reach them. It used
to keep every freed block while any snapshot existed.

### The rest of SchOS

These are VFS and driver limits rather than SchFS ones:

- A path is at most **4096 bytes** and a name **255**, Linux's `PATH_MAX` and
  `NAME_MAX` (`VFS_PATH_MAX`, `VFS_NAME_MAX`). Longer is `ENAMETOOLONG`. Both
  used to be cut short without a word, at 128 and 29.
- `NMOUNT` allows **32** mounts, and `NBLKDEV` **32** disks and partitions.
- The IDE driver handles the primary channel only: **2** ATA devices.
- The IDE driver uses LBA48 past 128 GiB, so the disk is not the limit there.
- A single `read`/`write` call is capped at **4 GiB** (`n` is a `uint`); the
  file itself is not.

The two that used to be here — a 16-bit inode number in `struct dirent` and a
32-bit `inum`/`size` in `struct inode` — are gone. `getdents64` is a per-
filesystem operation now, so SchFS emits its own records with 64-bit inode
numbers and names up to 255 bytes, and the inode carries 64-bit numbers and
sizes throughout.

---

## Format version, and what a bump means

Two mechanisms, because one is not enough. `SCHFS_VERSION` changes only when
the **frame** changes — the uberblock, the node header, the key, the things any
reader must understand before it can parse anything at all. Everything else is
a **feature flag**, in one of three classes:

| Class | An unknown bit means |
|---|---|
| `compat` | Ignore it. Mount read-write. |
| `ro_compat` | Mount **read-only**. A writer would maintain a structure this driver does not know about, and writing would leave it stale. |
| `incompat` | **Refuse.** The volume cannot be parsed correctly without it. |

Choosing the class is the whole design decision, and the test is one question:
*what happens if a driver that does not know this feature writes to the volume?*
Nothing → `compat`. Something on disk goes stale → `ro_compat`. Something
becomes unreadable → `incompat`.

The flags occupy what was `reserved0`, so every field after them keeps its
offset and existing images — which have zeros there — read identically.
`schfs_vol_commit` memsets the uberblock before filling it in, which is why
volumes written before the field existed already say "no features". **That trick
is available exactly once**; the next addition costs a version bump.

Four `_Static_assert`s pin the uberblock to one block and pin `version`,
the feature words and the field after them to fixed offsets, because those are
on-disk facts rather than C conveniences.

### Refusing is not falling back

The uberblock ring exists so that a damaged newest uberblock can be stepped
over — that is crash recovery, and the older root is a consistent filesystem.
A newest uberblock that is *sound but too new* is the opposite case, and
treating the two alike is how a mount quietly loses data:

> Falling back mounts a **stale** filesystem. Every commit made since the volume
> was upgraded is discarded, the mount reports success, and nobody finds out.

So `uber_check()` answers with a state rather than a yes/no —
`BAD` / `OK` / `RO` / `TOO_NEW` — and the scan stops on `TOO_NEW` instead of
continuing to the next candidate. The order of its checks is the order of the
questions: is it ours (magic), is it intact (crc), can we read it (version,
features), does it make sense (root in range). Compatibility is asked before
sanity deliberately — the fields a sanity check reads are only meaningfully
placed if the format is one we know.

Demonstrated rather than asserted, on a forged image whose two uberblocks are
`(version 1, generation 1, root 129)` and `(version 2, generation 2, root 130)`:
the old code mounted root 129 and exited 0, discarding the live filesystem; the
current code refuses and says why.

---

## Records are packed, and the compiler is told

A leaf's value area holds variable-length records end to end, so a record starts
wherever the previous one finished — 8-aligned only by luck. The seven types
that live there carry `SCHFS_PACKED`, because casting a pointer into a block and
calling it a `struct schfs_dirent_rec *` is otherwise a lie to the compiler: the
type promises alignment the address does not have.

**The annotation changes no on-disk layout.** Every one of those structs already
carries explicit `pad` fields and so has no implicit padding; sizes and offsets
are identical with and without it, and seven `_Static_assert`s hold them there.
What changes is the generated code — byte-wise access where alignment cannot be
proven, which is the truth of the situation.

The block-level structures are deliberately excluded. `schfs_node_hdr` is
exactly 32 bytes, so the items and key pointers after it are already aligned,
and packing them would cost byte-wise access on the tree's hottest paths for
nothing.

---

## One source, two compilers

Everything except `kmount.c` is compiled **both** into the kernel and into the
host tools.

> The writer and the reader cannot drift apart, because there is one
> implementation of the format and nothing to disagree with.

That is why `mkschfs` cannot produce an image the kernel misreads, and why
format bugs are found on the host where a test takes a second instead of a boot.

---

## The source

```
include/fs/schfs.h              on-disk format + compile-time layout assertions
kernel/fs/schfs/btree.{c,h}  the copy-on-write B-tree and its cursor
kernel/fs/schfs/schfs_int.h  volume + object layer interface
kernel/fs/schfs/space.c      allocation bitmap, derived from the tree at mount
kernel/fs/schfs/vol.c        uberblock ring: open, format, commit
kernel/fs/schfs/obj.c        read side: inodes, dirents, extents, path lookup
kernel/fs/schfs/objw.c       write side: create, link, unlink, rename, write
kernel/fs/schfs/kmount.c     kernel: block device, mount slots, transaction bracket
kernel/fs/schfs/vfs.c        the fs_ops table: SchFS behind open/read/write
tools/mkschfs.c              build an image from a directory tree
tools/schfs_fsck.c           verify, list, extract
include/fs/fsstat.h             counters behind /proc/fsstat: where the time goes
tools/schfs_test.c           B-tree torture harness
tools/schfs_objtest.c        object layer: files, directories, space accounting
tools/posix/schfs_roundtrip.sh     end-to-end filesystem test
tools/posix/mutate.sh              mutation-tests the suites themselves
tools/windows/bootcheck.ps1          boots under QEMU and drives the shell
```

Everything except `kmount.c` is compiled *both* into the kernel and into the
host tools, so the writer and the reader cannot drift apart: there is one
implementation of the format and nothing to disagree with.

### The test suites

```
make schfs-check         all three suites
make schfs-test          B-tree only
make schfs-objtest       object layer only
make schfs-image-test    round-trip only
```

`make schfs-check` after touching anything under `kernel/fs/schfs`.

The round-trip's oracle is the host's own filesystem: an image is built from a
directory tree, extracted with a separate program, and `diff -r`'d against the
original. It also asserts hard-link counts (which `diff` cannot see), checks
that rebuilding produces byte-identical output, and runs the whole thing a
second time with the name hash cut to 6 bits so the packed-dirent collision path
— which a 64-bit hash would never reach — is actually exercised.

`schfs_objtest` runs against a memory-backed volume, so a full
create/write/delete/remount cycle takes milliseconds. Its most important check
is the least glamorous: after deleting everything, free space must be back to
exactly what it was on an empty volume. Every other test in the suite still
passes on a filesystem that leaks a block on every delete.

Its section 10 is crash recovery, and it exists because the whole consistency
argument for this format — "mount takes the newest uberblock that validates, so
a crash lands on either the old tree or the new one" — had no end-to-end test at
all. `schfs_test` injected crashes into the B-tree; nothing did it to a volume.
The crash is a `longjmp` out of a block write, which is the only faithful model:
returning an error tests error handling instead, and letting execution continue
with the write swallowed has the filesystem reading back blocks it believes it
just wrote, which no real failure does. It runs the same transaction 119 times,
cutting the power one write later each time, and after every one the volume must
mount, check out structurally, and still hold the data committed before it.

`tools/posix/mutate.sh` injects six deliberate bugs and fails if any of them goes
undetected. It is how the suites are kept honest; see [../history/schfs.md](../history/schfs.md).

### Verifying on real hardware

```
make ramroot             root from RAM, so device 1 is free
make build/schfs.img
make qemu-schfs          then, at the shell:  cat /proc/schfs
```

`.\run.ps1 -SchFS` does the same on Windows, and `tools\bootcheck.ps1 -SchFS`
does it unattended and asserts on the transcript.

---

## What it does not do

- **No snapshots.** The B-tree is copy-on-write, which is most of the
  machinery — nothing takes or names a snapshot. This is the missing feature the
  existing design already pays for.
- **No journal replay** beyond what copy-on-write gives for free.
- **No sparse files.** A hole is not representable; writing at a large offset
  allocates everything before it.
- **No online `fsck`.** `schfs_fsck` is a host tool and the volume must not be
  mounted.
- **No upgrade path.** The version policy below refuses a volume it is too old
  for; nothing converts one format to another, because there is only one format.
  The first bump is when that has to be written.
- **No Linux driver.** FUSE first, then possibly a kernel module, and not before
  the format is pinned.
- **No deduplication**, deliberately — ZFS's is famous for needing more RAM than
  anyone expects, and extent refcounts give the reflink and snapshot benefits
  without it.
- **No multi-disk anything** — no pools, no RAID-Z, no vdevs.
- **No rebalancing or online format conversion**, deliberately: enormous
  complexity, and historically the source of btrfs's worst bugs.