nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

schfs: the commit barrier, reflink, and attributes with an index over them

Browse source 
The last three phases of the filesystem, which were written together and
touch the same files throughout -- kmount.c, objw.c and vfs.c are each
rewritten by all three -- so they land as one commit rather than three
that could not be built separately.

Durability. A transaction that reaches the platter out of order is worse
than one that never reached it: the uberblock can name a tree whose
blocks are still in the drive's write cache. bdirty/bflush give the
buffer layer an explicit barrier, ideflush issues ATA FLUSH CACHE and
usb_storage_flush the SCSI SYNCHRONIZE CACHE that means the same thing,
and a commit now waits for the data before it writes the root that
points at it. The uberblock groups are read newest-first with fallback,
so a volume whose newest roots are torn mounts at an older one and says
which generations were lost.

Reflink. Extents carry a refcount, so two objects can name the same
blocks and neither frees them while the other holds them. Exposed as
FICLONE -- Linux's ioctl value, so `cp --reflink` works unmodified --
and as cmd/reflink for the case where there is no cp to hand.

Attributes, and the one genuinely new idea here. attr.c stores named
values on an object, reachable through setxattr/getxattr/removexattr at
Linux's syscall numbers. On top of that, schfs_index declares an index
over an attribute name and schfs_query answers "which objects carry this
value" from the B-tree rather than by walking. A query against an
unindexed attribute is refused with an error, never silently answered by
a full scan: a query that quietly becomes a scan is worse than one that
admits it cannot help. cmd/schattr drives both.

Three bugs fixed on the way, each found by the numbers being wrong
rather than by inspection:

  - statfs's f_fsid was declared int64[2]. Linux's fsid_t is two *ints*,
    so every field after it landed one slot late: userland read f_frsize
    out of f_namelen, got 255, and df reported a 32 MB root filesystem
    as 1992 KB. It hid for as long as it did because the fields used to
    be invented constants, so wrong numbers looked like the usual ones.
  - struct inode's nlink was a short against SchFS's uint32, so a file
    past 32767 names came back negative -- and iput frees an object
    whose count reaches zero. It is an int now, signed on purpose, so an
    underflow lands somewhere a check can see rather than wrapping.
  - filewrite split every write to fit v1's log. That limit belongs to
    the journal, not to writing, and SchFS makes one transaction of
    whatever it is handed: a 1 MB write was becoming 170 commits and
    170 flushes for nothing. fs_ops carries max_write now, and SchFS
    sets no limit.

Also here: a real mount table, so `mount -t schfs /dev/hdb /mnt` shows
up in /proc/mounts -- df looks a path up there before it calls statfs at
all, so a mount missing from that list was invisible to it -- several
volumes mounted at once, snapshots mounted by name, and /proc/fsstat
with an rdtsc-based counter, which is the only instrument fine enough to
see inside a single syscall.

Verified: `make schfs-check` passes in full, including the sweep that
crashes the volume at each of 119 points mid-transaction and requires
every one to mount with exactly its committed data, and the case where
both newest uberblocks are rubbish and the volume still comes back. The
headless boot check passes with 27 [ok] and no failures, on a SchFS root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-08-15 16:01:22 +02:00
commit 03cc316001
39 changed files with 3942 additions and 550 deletions
Download patch file Download diff file Expand all files Collapse all files

327
kernel/fs/schfs/objw.c
View file

@ -295,6 +295,135 @@ csum_del(struct schfs_vol *v, uint64_t dblk)
return r == -1 ? 0 : r; /* absent is not a failure */
}
/* ------------------------------------------------------------------ */
/* Extent references */
/* */
/* How many files point at a run of blocks. Only shared runs have a */
/* record; absence means one reference. See SCHFS_EXTREF_OBJECTID. */
/* ------------------------------------------------------------------ */
// Find the reference record *covering* `dblk`, which is not necessarily one
// keyed at it.
//
// This is the whole subtlety of the extent tree. A record is keyed by the block
// a run was allocated at, but a file's extent may point into the middle of that
// run -- splitting a shared extent around a written block leaves a tail whose
// first block is several into the run. Looking up by that block finds nothing,
// and "nothing" means "unshared", so the tail would be freed while another file
// still reached it. Every lookup therefore has to be a range query.
//
// Returns 1 and fills `out` (and `*runstart`) if a record covers dblk, 0 if
// none does -- meaning one reference -- or -2 on corruption.
static int
extref_find(struct schfs_vol *v, uint64_t dblk, uint64_t *runstart,
struct schfs_extref *out)
{
struct schfs_cursor c;
schfs_cursor_init(&c, &v->tree, v->ctx.a);
struct schfs_key probe;
schfs_mkkey(&probe, SCHFS_EXTREF_OBJECTID, SCHFS_KEY_EXTREF, dblk);
int r = schfs_btree_lookup_le(&c, &probe);
if (r == -2)
return -2;
if (r != 0)
return 0;
const struct schfs_key *got = schfs_cursor_key(&c);
unsigned vlen;
const struct schfs_extref *found =
(const struct schfs_extref *)schfs_cursor_val(&c, &vlen);
if (!got || !found)
return 0;
if (got->objectid != SCHFS_EXTREF_OBJECTID || got->type != SCHFS_KEY_EXTREF)
return 0;
if (vlen != sizeof(*out))
return -2;
if (dblk >= got->offset + found->nblocks)
return 0; /* the nearest record ends before dblk */
if (runstart) *runstart = got->offset;
*out = *found;
return 1;
}
static int
extref_put(struct schfs_vol *v, uint64_t dblk, const struct schfs_extref *e)
{
struct schfs_key k;
schfs_mkkey(&k, SCHFS_EXTREF_OBJECTID, SCHFS_KEY_EXTREF, dblk);
return schfs_btree_insert(&v->tree, &v->ctx, &k, e, sizeof(*e));
}
static int
extref_del(struct schfs_vol *v, uint64_t dblk)
{
struct schfs_key k;
schfs_mkkey(&k, SCHFS_EXTREF_OBJECTID, SCHFS_KEY_EXTREF, dblk);
int r = schfs_btree_delete(&v->tree, &v->ctx, &k);
return r == -1 ? 0 : r; /* absent is not a failure */
}
// One more file now points at the run covering `dblk`. `nblocks` is used only
// when no record exists yet, to record the run's length.
static int
extref_inc(struct schfs_vol *v, uint64_t dblk, uint64_t nblocks)
{
struct schfs_extref e;
uint64_t start = dblk;
int have = extref_find(v, dblk, &start, &e);
if (have == -2)
return -2;
if (have == 0) {
// First share: the run went from one holder to two, and this is where a
// record starts existing. `nblocks` is recorded now and never changed,
// because it is the length of the *run as allocated* -- see extref_dec.
e.nblocks = nblocks;
e.refs = 2;
} else {
e.refs++;
}
return extref_put(v, start, &e);
}
// One fewer holder. Returns 1 if the run is now unreferenced, filling *nblocks
// with the length to release; 0 if somebody still holds it; -2 on corruption.
//
// The record survives down to one holder rather than being deleted at that
// point, which is the subtle part. A file that shares a run may later be
// truncated inside it, leaving its extent record describing a *shorter* run
// than was allocated. If the record vanished at one reference, the last holder
// would free only as much as its own shortened record described and the tail
// would leak. Keeping the record keeps the original length, so the final
// release frees exactly what was allocated.
static int
extref_dec(struct schfs_vol *v, uint64_t dblk, uint64_t *runstart,
uint64_t *nblocks)
{
struct schfs_extref e;
uint64_t start = dblk;
int have = extref_find(v, dblk, &start, &e);
if (have == -2)
return -2;
if (have == 0) {
*runstart = dblk;
*nblocks = 0; /* no record: the caller's own view is it */
return 1;
}
*runstart = start;
if (e.refs <= 1) {
*nblocks = e.nblocks;
int r = extref_del(v, start);
return r == 0 ? 1 : r;
}
e.refs--;
*nblocks = e.nblocks;
int r = extref_put(v, start, &e);
return r == 0 ? 0 : r;
}
/* ------------------------------------------------------------------ */
/* Extents */
/* ------------------------------------------------------------------ */
@ -417,23 +546,151 @@ extent_grow(struct schfs_vol *v, uint64_t ino, uint64_t fblk,
}
// Release a run of data blocks and the checksum records that describe them.
// Shared runs are left alone: another file still points at them, and proper
// refcounting needs an extent tree that nothing creates yet.
//
// A run somebody else still points at is not released, only dropped: the
// reference count in the extent tree decides, because this file's own copy of
// `refs` cannot be decremented by the other holder.
//
// Note the asymmetry with a partial release. A file whose extent is being
// truncated in the middle releases only the tail, but the reference count is
// per *run*, not per block -- so a shared run is only ever dropped whole. That
// is why extent_release consults the count using the run's first block, and why
// a partial free of a shared run leaves the blocks alone: they are still
// reachable through the other file, in their entirety.
static int
extent_release(struct schfs_vol *v, const struct schfs_extent *e,
uint64_t first, uint64_t n)
{
if (e->refs != 1)
struct schfs_extref xr;
uint64_t runstart = 0;
int shared = extref_find(v, e->disk_block, &runstart, &xr);
if (shared == -2)
return -2;
// Nothing has ever cloned this run, so this file is the only holder and may
// release exactly what it was asked to -- a whole run or the tail of one.
// This is the path every ordinary file takes.
if (shared == 0) {
for (uint64_t i = 0; i < n; i++) {
int r = csum_del(v, first + i);
if (r != 0)
return r;
v->dev.bfree(v->dev.ctx, first + i);
}
return 0;
for (uint64_t i = 0; i < n; i++) {
int r = csum_del(v, first + i);
}
// Shared. A partial release means this file is keeping the head of the run,
// so its record still points into it and the reference stands: the blocks
// stay, because the other holder reaches all of them anyway.
if (first != e->disk_block || n != e->nblocks)
return 0;
uint64_t runlen = 0;
int last = extref_dec(v, e->disk_block, &runstart, &runlen);
if (last == -2)
return -2;
if (!last)
return 0; /* somebody else still points at it */
// Freed from the run's own start and for its allocated length, not from
// this file's view of it -- which may be a slice several blocks in.
if (runlen == 0) {
runstart = e->disk_block;
runlen = e->nblocks;
}
for (uint64_t i = 0; i < runlen; i++) {
int r = csum_del(v, runstart + i);
if (r != 0)
return r;
v->dev.bfree(v->dev.ctx, first + i);
v->dev.bfree(v->dev.ctx, runstart + i);
}
return 0;
}
static int extents_drop_from(struct schfs_vol *v, uint64_t ino,
struct schfs_inode *in, uint64_t from);
// Share one file's data with another, without copying any of it.
//
// The reflink. `dst` must be an existing object; whatever it held is released
// first. Afterwards both files name the same blocks, both are ordinary writable
// files, and the first write to either copies only the run it touches -- which
// is what copy-on-write already does for the tree and now does for data.
//
// The whole operation is metadata: one extent record per run in the source,
// duplicated, plus a reference record per run. Cloning a gigabyte costs the
// same as cloning a byte.
//
// Both sides' extent records are updated, not just the destination's. The
// per-record `refs` is the hint the write path consults before rewriting a run
// in place; leaving the source saying "mine alone" would let a later write to
// the source scribble over the clone.
int
schfs_file_reflink(struct schfs_vol *v, uint64_t src, uint64_t dst)
{
struct schfs_inode si, di;
struct schfs_cursor c;
if (src == dst)
return -1;
schfs_cache_invalidate();
schfs_cursor_init(&c, &v->tree, v->ctx.a);
if (schfs_inode_get(v, &c, src, &si) != 0)
return -1;
schfs_cursor_init(&c, &v->tree, v->ctx.a);
if (schfs_inode_get(v, &c, dst, &di) != 0)
return -1;
if (si.type != SCHFS_T_FILE || di.type != SCHFS_T_FILE)
return -1;
// The destination starts empty. Doing this first means a failure part way
// through leaves a short file rather than one holding a mixture.
if (extents_drop_from(v, dst, &di, 0) != 0)
return -2;
di.blocks = 0;
// Walk the source by file block. Not with a cursor held across the loop:
// every iteration inserts records, which rewrites the very nodes a cursor
// would be standing on.
uint64_t fblk = 0;
uint64_t nblk = (si.size + SCHFS_BLKSIZE - 1) / SCHFS_BLKSIZE;
while (fblk < nblk) {
uint64_t start;
struct schfs_extent e;
int m = extent_find(v, src, fblk, &start, &e);
if (m == -2)
return -2;
if (m == 0) {
fblk++; /* a hole: nothing to share */
continue;
}
int r = extref_inc(v, e.disk_block, e.nblocks);
if (r != 0)
return -2;
// Both sides now say "shared", so neither will rewrite the run in
// place. 2 rather than a real count: it is only ever compared with 1.
struct schfs_extent se = e;
se.refs = 2;
if (extent_put(v, src, start, &se) != 0)
return -2;
if (extent_put(v, dst, start, &se) != 0)
return -2;
di.blocks += e.nblocks;
fblk = start + e.nblocks;
}
di.size = si.size;
di.gen = v->tree.generation;
return schfs_inode_put(v, dst, &di);
}
// Drop everything from file block `from` onward. Used by truncate and by the
// deletion of an object.
static int
@ -616,11 +873,59 @@ block_for_write(struct schfs_vol *v, uint64_t ino, uint64_t fblk,
return r;
}
// The block just replaced rejoins the free map at the next commit, not now:
// until then the committed tree still points at it and a crash lands there.
int r = extent_release(v, &e, old, 1);
if (r != 0)
return r;
// Account for what the split did to the old run.
//
// The reference count on a shared run is "how many file-extent records
// point into it", so a split that leaves a head *and* a tail has added a
// reference, not kept one. Getting this wrong is invisible until every
// holder is deleted and the run is either freed while still in use or
// never freed at all -- the second is what happened first, leaking four
// blocks per split.
{
int pieces = (fblk > start ? 1 : 0)
+ (fblk + 1 < start + e.nblocks ? 1 : 0);
struct schfs_extref xr;
uint64_t runstart = 0;
int shared = extref_find(v, e.disk_block, &runstart, &xr);
if (shared == -2)
return -2;
if (shared) {
if (pieces == 2) {
if (extref_inc(v, e.disk_block, e.nblocks) != 0)
return -2;
} else if (pieces == 0) {
// A one-block run, entirely replaced: this file's only
// reference to it is gone.
uint64_t runlen = 0;
int last = extref_dec(v, e.disk_block, &runstart, &runlen);
if (last == -2)
return -2;
if (last) {
if (runlen == 0) {
runstart = e.disk_block;
runlen = e.nblocks;
}
for (uint64_t i = 0; i < runlen; i++) {
int rr = csum_del(v, runstart + i);
if (rr != 0)
return rr;
v->dev.bfree(v->dev.ctx, runstart + i);
}
}
}
// The replaced block itself is never freed here: it is part of a
// run somebody else still reaches in full.
} else {
// The block just replaced rejoins the free map at the next commit,
// not now: until then the committed tree still points at it and a
// crash lands there.
int r = extent_release(v, &e, old, 1);
if (r != 0)
return r;
}
}
w->disk = nd;
w->src = old;