nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

schfs: the commit barrier, reflink, and attributes with an index over them

Browse source 
The last three phases of the filesystem, which were written together and
touch the same files throughout -- kmount.c, objw.c and vfs.c are each
rewritten by all three -- so they land as one commit rather than three
that could not be built separately.

Durability. A transaction that reaches the platter out of order is worse
than one that never reached it: the uberblock can name a tree whose
blocks are still in the drive's write cache. bdirty/bflush give the
buffer layer an explicit barrier, ideflush issues ATA FLUSH CACHE and
usb_storage_flush the SCSI SYNCHRONIZE CACHE that means the same thing,
and a commit now waits for the data before it writes the root that
points at it. The uberblock groups are read newest-first with fallback,
so a volume whose newest roots are torn mounts at an older one and says
which generations were lost.

Reflink. Extents carry a refcount, so two objects can name the same
blocks and neither frees them while the other holds them. Exposed as
FICLONE -- Linux's ioctl value, so `cp --reflink` works unmodified --
and as cmd/reflink for the case where there is no cp to hand.

Attributes, and the one genuinely new idea here. attr.c stores named
values on an object, reachable through setxattr/getxattr/removexattr at
Linux's syscall numbers. On top of that, schfs_index declares an index
over an attribute name and schfs_query answers "which objects carry this
value" from the B-tree rather than by walking. A query against an
unindexed attribute is refused with an error, never silently answered by
a full scan: a query that quietly becomes a scan is worse than one that
admits it cannot help. cmd/schattr drives both.

Three bugs fixed on the way, each found by the numbers being wrong
rather than by inspection:

  - statfs's f_fsid was declared int64[2]. Linux's fsid_t is two *ints*,
    so every field after it landed one slot late: userland read f_frsize
    out of f_namelen, got 255, and df reported a 32 MB root filesystem
    as 1992 KB. It hid for as long as it did because the fields used to
    be invented constants, so wrong numbers looked like the usual ones.
  - struct inode's nlink was a short against SchFS's uint32, so a file
    past 32767 names came back negative -- and iput frees an object
    whose count reaches zero. It is an int now, signed on purpose, so an
    underflow lands somewhere a check can see rather than wrapping.
  - filewrite split every write to fit v1's log. That limit belongs to
    the journal, not to writing, and SchFS makes one transaction of
    whatever it is handed: a 1 MB write was becoming 170 commits and
    170 flushes for nothing. fs_ops carries max_write now, and SchFS
    sets no limit.

Also here: a real mount table, so `mount -t schfs /dev/hdb /mnt` shows
up in /proc/mounts -- df looks a path up there before it calls statfs at
all, so a mount missing from that list was invisible to it -- several
volumes mounted at once, snapshots mounted by name, and /proc/fsstat
with an rdtsc-based counter, which is the only instrument fine enough to
see inside a single syscall.

Verified: `make schfs-check` passes in full, including the sweep that
crashes the volume at each of 119 points mid-transaction and requires
every one to mount with exactly its committed data, and the case where
both newest uberblocks are rubbish and the volume still comes back. The
headless boot check passes with 27 [ok] and no failures, on a SchFS root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-08-15 16:01:22 +02:00
commit 03cc316001
39 changed files with 3942 additions and 550 deletions
Download patch file Download diff file Expand all files Collapse all files

204
docs/architecture/schfs.md
View file

@ -1,9 +1,11 @@
# SchFS — design
**Status: phases 1 and 2 are done.** The copy-on-write B-tree, the volume layer,
the object layer, a working image writer and a checker all exist, and the kernel
mounts a SchFS volume read-only at boot. Phase 3 (writes and transaction groups)
is next. See §8 for exactly what is built and how to run its tests.
**Status: phases 1 through 3½ are done, and SchFS is the root filesystem.**
`kernel/proc/proc.c` mounts it before pid 1 has a working directory and panics
if it is not there. The copy-on-write B-tree, the volume layer, the object
layer, the write path, snapshots, an image writer and a checker all exist, and
several volumes can be mounted at once. Phase 4 (extent refcounts) is next.
See §8 for exactly what is built and how to run its tests.
Supersedes the on-disk format described in `include/fs.h`. Keeps the name SchFS;
the on-disk magic changes so v1 images are detected and rejected rather than
@ -211,8 +213,159 @@ in the shared VFS rather than in SchFS:
entry point libc happened to call.
- `rmdir` had no Linux syscall number, so every Linux-ABI program got ENOSYS.
**Phase 4 — extents with refcounts.** Enables reflink copies and snapshots
nearly for free.
**Phase 3¾ — durability, and the cost of it. DONE.** Three things that the
earlier phases had assumed rather than arranged.
*The commit barrier.* Copy-on-write promises that a crash lands on either the
old tree or the new one, but only if the blocks a root points at reach the
medium *before* the block naming that root. Nothing enforced that: `bwrite`
order is not platter order on any drive with a write cache. `struct schfs_bdev`
gained a `bflush` hook, `schfs_vol_commit` calls it either side of the uberblock
write, and the kernel implements it as "drain this device's dirty buffers, then
ATA FLUSH CACHE" (`ideflush`, and SCSI SYNCHRONIZE CACHE for a USB stick).
*Falling back through the ring.* Mount took the highest-generation uberblock
that validated, checked the tree it named, and gave up if that failed — with up
to 127 older, entirely intact roots sitting on the disk. It now walks the ring
newest-first and adopts the first root it can actually read, counting what it
skipped into `vol->recovered` so the rollback is reported rather than silent.
*Deferred writes, and brackets.* The B-tree seals and writes a node on every
record it inserts, so a leaf collecting one file's data checksums went to the
disk once per 4 KB of that file. The kernel's block hook marks buffers dirty
instead (`bdirty`), and the commit barrier is what makes that safe. On its own
that changes little, because a commit *contains* a flush and every `schfs_k*`
call was one commit — so `schfs_ktx_open`/`schfs_ktx_close` bracket a run of
calls into one group, and `schfs_vfs_write` wraps a whole `write(2)` in one.
Measured over 1 MB of file data, in block writes reaching the device:
| | before | after |
|---|---|---|
| committing every 4 KB | 1947 | 1410 |
| one commit for the megabyte | 1798 | **267** |
267 device writes for 256 blocks of data. Note that the two changes are worth
almost nothing separately — deferral cannot help while every commit flushes,
and batching cannot help while every write goes straight to the disk — which is
why the first attempt at this measured an 8% difference and concluded batching
was not the problem.
**Phase 3⅞ — where the time actually went. DONE.** The filesystem was correct
and durable and the system still felt slow: `ls /bin` took the better part of a
second. Two causes, neither of them the disk.
Measured first, through `/proc/fsstat` (counters added for exactly this). The
decisive number was **398,999 block reads against 445 actual disk misses** —
99.9% of all block reads were re-reading a block already in the buffer cache. So
the disk was never the problem, and every plan that started with "cache more" or
"read ahead" would have been aimed at 1% of the cost.
*The cursor re-read its own buffer.* `cur_load` called `node_read`
unconditionally, and `schfs_cursor_val` calls `cur_load` on **every access to
the current record** — deliberately, so that a callback disturbing the shared
block buffer cannot corrupt a directory walk. But `node_read` copies 4096 bytes
out of the buffer cache and then runs CRC32c over all 4096. Listing a 243-entry
directory paid both once per entry. The cursor now remembers which block its
buffer holds and skips the reload, cross-checked against the block header (so a
second cursor sharing the page cannot fool it) and against a global epoch that
every mutation bumps (so a block rewritten in place cannot be served stale).
*The checksum was bit-at-a-time.* `schfs_crc32c` did eight shifts per byte —
32,768 iterations to checksum one block — to stay usable before any
initialisation. It builds a 1 KB table on first use now, which keeps that
property and is eight times faster. This only became the top cost *after* the
cursor fix, which is the usual shape: removing the largest item promotes
whatever was beneath it.
| cycles, one session | before | after |
|---|---|---|
| `getdents` | 26.8 G | **0.14 G** (196×) |
| `read` | 26.4 G | **5.5 G** (4.8×) |
| `exec` | 18.7 G | **4.0 G** (4.7×) |
| block reads | 398,999 | **127,713** |
`ls /bin` went from ~400 ms to ~20 ms. Verified by reading an 802 KB binary
through the whole path and checking its md5 against the host's copy.
There was also a third of the problem that was not the kernel at all: `make
qemu` ran QEMU without `-enable-kvm`, so every instruction was emulated. The run
targets probe `/dev/kvm` now. Measure against the KVM number or you are
measuring TCG.
**Phase 4 — extents with refcounts. DONE.** `cp` that costs nothing.
A reference count per run of blocks, in its own tree (`SCHFS_KEY_EXTREF` under
`SCHFS_EXTREF_OBJECTID`) rather than in the file's own extent record — because a
count kept per file cannot be decremented by the *other* file sharing the run.
`schfs_file_reflink` copies one extent record per run and bumps the count; both
files stay ordinary and writable, and the first write to either copies only the
run it touches. Reachable from userland as `FICLONE`, Linux's ioctl, so
`cp --reflink` works unmodified; `cmd/reflink` is a two-line caller of it.
Measured on the running system: cloning a 411 KB file moved `df` by 24 KB.
Three things had to be right, and each was wrong first:
- **The count is "how many extent records point into this run", not "how many
files".** Splitting a shared extent around a written block leaves a head *and*
a tail, which is one more reference than before. Missing that leaked four
blocks per split.
- **A file's extent may point into the middle of a run.** Records are keyed by
the block a run was *allocated* at, so every lookup is a range query
(`extref_find`), not an exact one. An exact lookup finds nothing for a split
tail, "nothing" means unshared, and the tail is then freed while another file
still reaches it.
- **The space scan has to walk the reference records too.** A file truncated
inside a shared run describes fewer blocks than exist; the rest would be
derived as free while still referenced, and the allocator would hand out a
block that a later release frees a second time.
None of those is visible until every holder of a run is deleted, which is why
the test that matters is the least glamorous one: after deleting every file that
shared a run, free space must be back to exactly what it was.
**Phase 5 — attribute index and query. DONE.** The reason the design was one
B-tree with many key types from the start: the "database filesystem" feature
turned out to be `kernel/fs/schfs/attr.c` and nothing else.
An attribute is a named value on an object, stored under `(ino, ATTR,
hash-of-name)` and packed exactly as directory entries are — same collision
argument, same shape. An attribute may be *indexed*, and an indexed attribute
gets a second set of records keyed by value: `(hash-of-name, INDEX,
hash-of-value)` holding the object ids that carry it. So "every object whose
`tag` is `holiday`" is one tree descent instead of a walk of every object.
There is no query language, no parser and no planner, and §6 above explains why:
the useful half of "query the filesystem" is the index, and WinFS spent three
years and a large team on the other interpretation.
```
schattr index tag # declare the index
schattr set /tmp/a tag holiday
schattr find tag holiday # one descent, whatever the volume holds
```
Four decisions worth keeping:
- **Which attributes are indexed is cached in the volume**, read once at mount.
Every `setxattr` has to ask, and the answer is almost always no; a tree
descent to say so would make every attribute write pay for a feature it is
not using.
- **The value is stored in the index entry, not just its hash.** Two values can
hash alike, and a query that returned the wrong file would be far worse than a
slow one.
- **An index covers values written after it is declared.** There is no
reindexing pass. Saying so is better than a half-truth that makes a query
answer confidently and wrongly.
- **A query on an unindexed attribute is refused, not answered by scanning.** A
query that silently degrades into a full walk teaches you to use it where it
will not scale.
Reachable as `setxattr`/`getxattr`/`removexattr` at Linux's syscall numbers, so
anything that already knows how to attach metadata needs no changes, plus two
native calls (`schfs_index`, `schfs_query`) that have no Linux equivalent
because asking a filesystem a question does not.
**Phase 5 — attribute index and query.** See below.
@ -279,6 +432,8 @@ Directory *depth* is unbounded on disk; path resolution is what limits it.
| | limit | where it comes from |
|---|---|---|
| Volume, mounted | **64 GiB** | the in-RAM allocation bitmap: `SCHFS_SPACE_PAGES` (512) pages × 32768 blocks |
| Volumes mounted at once | **4** | `SCHFS_VOL_SLOTS`; slot 0 is the root |
| Snapshots mounted at once | **4** | `SCHFS_SNAP_SLOTS` |
| Volume, via the buffer cache | **4 TiB** | `bio.c` addresses blocks with a `uint`, and one SchFS block spans four of them |
| RAM per mount | **8 KiB per GiB** of volume | two bitmaps, one bit per block each |
| Inline value in a leaf | **1016 bytes** | `SCHFS_MAX_INLINE` = a quarter of a node |
@ -298,7 +453,8 @@ These are VFS and driver limits rather than SchFS ones:
- `MAXPATH` caps a path at **128 bytes**.
- `NMOUNT` allows **8** mounts.
- The IDE driver is 28-bit LBA: **128 GiB** per disk.
- The IDE driver handles the primary channel only: **2** ATA devices.
- The IDE driver uses LBA48 past 128 GiB, so the disk is not the limit there.
- A single `read`/`write` call is capped at **4 GiB** (`n` is a `uint`); the
file itself is not.
@ -318,10 +474,11 @@ kernel/fs/schfs/space.c allocation bitmap, derived from the tree at mount
kernel/fs/schfs/vol.c uberblock ring: open, format, commit
kernel/fs/schfs/obj.c read side: inodes, dirents, extents, path lookup
kernel/fs/schfs/objw.c write side: create, link, unlink, rename, write
kernel/fs/schfs/kmount.c kernel: block device, mount, transaction bracket
kernel/fs/schfs/kmount.c kernel: block device, mount slots, transaction bracket
kernel/fs/schfs/vfs.c the fs_ops table: SchFS behind open/read/write
tools/mkschfs.c build an image from a directory tree
tools/schfs_fsck.c verify, list, extract
include/fsstat.h counters behind /proc/fsstat: where the time goes
tools/schfs_test.c B-tree torture harness
tools/schfs_objtest.c object layer: files, directories, space accounting
tools/posix/schfs_roundtrip.sh end-to-end filesystem test
@ -357,6 +514,17 @@ is the least glamorous: after deleting everything, free space must be back to
exactly what it was on an empty volume. Every other test in the suite still
passes on a filesystem that leaks a block on every delete.
Its section 10 is crash recovery, and it exists because the whole consistency
argument for this format — "mount takes the newest uberblock that validates, so
a crash lands on either the old tree or the new one" — had no end-to-end test at
all. `schfs_test` injected crashes into the B-tree; nothing did it to a volume.
The crash is a `longjmp` out of a block write, which is the only faithful model:
returning an error tests error handling instead, and letting execution continue
with the write swallowed has the filesystem reading back blocks it believes it
just wrote, which no real failure does. It runs the same transaction 119 times,
cutting the power one write later each time, and after every one the volume must
mount, check out structurally, and still hold the data committed before it.
`tools/posix/mutate.sh` injects six deliberate bugs and fails if any of them goes
undetected. It is how the suites are kept honest; see the note in §9.
@ -401,3 +569,23 @@ Recorded because each one cost real time and none is obvious from the code.
- **Reproducibility dies from atime.** mkschfs reads every source file, which
updates its atime, so copying atime into the image made two builds of the same
tree differ. All three timestamps come from mtime.
- **An optional hook must still be initialised.** `struct schfs_bdev` gained
`bflush`, and every host tool built one on the stack field by field. The tools
that did not set the new field handed `schfs_vol_commit` a garbage pointer to
test and call, which segfaulted at the first commit. Each of those helpers
memsets the struct first now — the fix is not "remember to set it".
- **Deferring writes makes an all-dirty buffer cache normal.** `bget` panicked
with "no buffers" when it could find nothing clean to recycle, which was
correct while a dirty buffer meant "the log has not committed". It now writes
the least recently used victim out and takes it; a transaction touching more
than `NBUF` blocks reaches that state long before it commits.
- **The two performance fixes are worthless apart.** Deferring tree writes buys
nothing while every commit flushes, and batching commits buys nothing while
every write goes straight to the disk. Measuring either alone says "this is
not where the time goes", and both measurements are correct.
- **`fsid_t` is eight bytes.** `struct statfs` declared `f_fsid` as `int64[2]`,
which is one slot too long, so everything after it landed in the wrong place:
userland read `f_frsize` out of `f_namelen` and got 255. df then computed in
units of 255 bytes and called a 32 MB root filesystem 1992 KB. It survived
because every field in that struct used to be an invented constant, so no
number in it had ever been checked against reality.