nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

schfs: the commit barrier, reflink, and attributes with an index over them

Browse source 
The last three phases of the filesystem, which were written together and
touch the same files throughout -- kmount.c, objw.c and vfs.c are each
rewritten by all three -- so they land as one commit rather than three
that could not be built separately.

Durability. A transaction that reaches the platter out of order is worse
than one that never reached it: the uberblock can name a tree whose
blocks are still in the drive's write cache. bdirty/bflush give the
buffer layer an explicit barrier, ideflush issues ATA FLUSH CACHE and
usb_storage_flush the SCSI SYNCHRONIZE CACHE that means the same thing,
and a commit now waits for the data before it writes the root that
points at it. The uberblock groups are read newest-first with fallback,
so a volume whose newest roots are torn mounts at an older one and says
which generations were lost.

Reflink. Extents carry a refcount, so two objects can name the same
blocks and neither frees them while the other holds them. Exposed as
FICLONE -- Linux's ioctl value, so `cp --reflink` works unmodified --
and as cmd/reflink for the case where there is no cp to hand.

Attributes, and the one genuinely new idea here. attr.c stores named
values on an object, reachable through setxattr/getxattr/removexattr at
Linux's syscall numbers. On top of that, schfs_index declares an index
over an attribute name and schfs_query answers "which objects carry this
value" from the B-tree rather than by walking. A query against an
unindexed attribute is refused with an error, never silently answered by
a full scan: a query that quietly becomes a scan is worse than one that
admits it cannot help. cmd/schattr drives both.

Three bugs fixed on the way, each found by the numbers being wrong
rather than by inspection:

  - statfs's f_fsid was declared int64[2]. Linux's fsid_t is two *ints*,
    so every field after it landed one slot late: userland read f_frsize
    out of f_namelen, got 255, and df reported a 32 MB root filesystem
    as 1992 KB. It hid for as long as it did because the fields used to
    be invented constants, so wrong numbers looked like the usual ones.
  - struct inode's nlink was a short against SchFS's uint32, so a file
    past 32767 names came back negative -- and iput frees an object
    whose count reaches zero. It is an int now, signed on purpose, so an
    underflow lands somewhere a check can see rather than wrapping.
  - filewrite split every write to fit v1's log. That limit belongs to
    the journal, not to writing, and SchFS makes one transaction of
    whatever it is handed: a 1 MB write was becoming 170 commits and
    170 flushes for nothing. fs_ops carries max_write now, and SchFS
    sets no limit.

Also here: a real mount table, so `mount -t schfs /dev/hdb /mnt` shows
up in /proc/mounts -- df looks a path up there before it calls statfs at
all, so a mount missing from that list was invisible to it -- several
volumes mounted at once, snapshots mounted by name, and /proc/fsstat
with an rdtsc-based counter, which is the only instrument fine enough to
see inside a single syscall.

Verified: `make schfs-check` passes in full, including the sweep that
crashes the volume at each of 119 points mid-transaction and requires
every one to mount with exactly its committed data, and the case where
both newest uberblocks are rubbish and the volume still comes back. The
headless boot check passes with 27 [ok] and no failures, on a SchFS root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-08-15 16:01:22 +02:00
commit 03cc316001
39 changed files with 3942 additions and 550 deletions
Download patch file Download diff file Expand all files Collapse all files

507
kernel/fs/schfs/attr.c Normal file
View file

@ -0,0 +1,507 @@
// SPDX-License-Identifier: Apache-2.0
//
// Attributes, and the index over them.
//
// An attribute is a named value on an object. The interesting part is not
// storing them -- that is another key type in a tree that already exists -- but
// the index: an indexed attribute gets a second set of records keyed by *value*
// rather than by object, so "every object whose `tag` is `holiday`" is a range
// scan of a few keys instead of a walk of the whole filesystem.
//
// That is the whole of what "a filesystem you can query" means here, and the
// reason the design was one B-tree with many key types from the start. There is
// no query engine, no parser and no planner; there is an index, which is the
// part that actually makes the question fast. See docs/architecture/schfs.md §6.
//
// ---------------------------------------------------------------------------
// Packed lists, again
//
// Attributes are keyed (ino, ATTR, hash-of-name) and index entries
// (name-hash, INDEX, hash-of-value). Both hash something that can collide, so
// both store a *packed list* of records rather than a single one -- the same
// shape directory entries use, for the same reason. A 64-bit hash makes a
// collision vanishingly unlikely and "vanishingly unlikely" is not "impossible".
#include "schfs_int.h"
#ifdef SCHFS_HOSTED
#include <string.h>
#else
#include "types.h"
#include "defs.h"
#endif
/* ------------------------------------------------------------------ */
/* Reading */
/* ------------------------------------------------------------------ */
// Find one attribute in the packed list under its name's hash.
//
// Returns 1 and fills the caller's buffers if present, 0 if absent, -2 on
// corruption. `*valuelen` is set to the value's true length even when the
// buffer is too small, so a caller can size a second call -- which is what
// getxattr(2) promises.
int
schfs_attr_get(struct schfs_vol *v, uint64_t ino, const char *name,
unsigned namelen, void *out, unsigned cap, unsigned *valuelen,
uint16_t *type)
{
if (namelen == 0 || namelen > SCHFS_ATTR_NAME_MAX)
return 0;
struct schfs_key k;
schfs_mkkey(&k, ino, SCHFS_KEY_ATTR, schfs_namehash(name, namelen));
unsigned len = SCHFS_MAX_INLINE;
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->dirbuf, &len);
if (r == -1)
return 0;
if (r != 0)
return r;
if (len > SCHFS_MAX_INLINE)
return -2;
unsigned off = 0;
while (off + sizeof(struct schfs_attr_rec) <= len) {
const struct schfs_attr_rec *a =
(const struct schfs_attr_rec *)(v->dirbuf + off);
unsigned step = SCHFS_ATTR_LEN(a->namelen, a->valuelen);
if (a->namelen == 0 || a->namelen > SCHFS_ATTR_NAME_MAX ||
a->valuelen > SCHFS_ATTR_VALUE_MAX || off + step > len)
return -2;
const char *nm = (const char *)a + sizeof(*a);
if (a->namelen == namelen && memcmp(nm, name, namelen) == 0) {
if (valuelen) *valuelen = a->valuelen;
if (type) *type = a->type;
if (out && cap >= a->valuelen)
memmove(out, nm + a->namelen, a->valuelen);
else if (out)
return -3; /* buffer too small; length is set */
return 1;
}
off += step;
}
return 0;
}
// Hand every attribute on an object to a callback, in the order they are
// stored. Used by listxattr and by the reindexing walk.
int
schfs_attr_iterate(struct schfs_vol *v, struct schfs_cursor *c, uint64_t ino,
int (*fn)(const char *name, unsigned namelen,
const void *val, unsigned vallen,
uint16_t type, void *arg),
void *arg)
{
struct schfs_key start;
schfs_mkkey(&start, ino, SCHFS_KEY_ATTR, 0);
// Re-point at the volume's current tree. The root moves on every commit
// under copy-on-write, so a cursor is bound per operation rather than once.
schfs_cursor_init(c, &v->tree, c->blk);
if (schfs_cursor_seek(c, &start) != 0)
return 0;
for (;;) {
const struct schfs_key *k = schfs_cursor_key(c);
if (!k || k->objectid != ino || k->type != SCHFS_KEY_ATTR)
return 0;
unsigned len;
const char *val = (const char *)schfs_cursor_val(c, &len);
if (!val || len > SCHFS_MAX_INLINE)
return -2;
// Copied out before the callback runs: the callback may look something
// up, and that reuses the very buffer this points into. The same trap
// that once reparented directories under their own children.
memmove(v->dirbuf, val, len);
unsigned off = 0;
while (off + sizeof(struct schfs_attr_rec) <= len) {
const struct schfs_attr_rec *a =
(const struct schfs_attr_rec *)(v->dirbuf + off);
unsigned step = SCHFS_ATTR_LEN(a->namelen, a->valuelen);
if (a->namelen == 0 || a->namelen > SCHFS_ATTR_NAME_MAX ||
a->valuelen > SCHFS_ATTR_VALUE_MAX || off + step > len)
return -2;
const char *nm = (const char *)a + sizeof(*a);
int stop = fn(nm, a->namelen, nm + a->namelen, a->valuelen,
a->type, arg);
if (stop)
return 0;
off += step;
}
if (schfs_cursor_next(c) != 0)
return 0;
}
}
/* ------------------------------------------------------------------ */
/* The index */
/* */
/* Which attributes are indexed is one record, and it is consulted on */
/* every attribute write -- so it is read into the volume rather than */
/* fetched each time. See schfs_index_reload. */
/* ------------------------------------------------------------------ */
// Is `name` an indexed attribute?
//
// Answered from the cached list rather than the tree. setxattr asks this every
// time, and the answer is almost always no; a tree descent to say so would make
// every attribute write pay for a feature it is not using.
static int
index_wanted(struct schfs_vol *v, const char *name, unsigned namelen)
{
unsigned off = 0;
while (off + 1 <= v->idxnames_len) {
unsigned l = (unsigned char)v->idxnames[off];
if (l == 0 || off + 1 + l > v->idxnames_len)
return 0;
if (l == namelen && memcmp(v->idxnames + off + 1, name, namelen) == 0)
return 1;
off += 1 + l;
}
return 0;
}
// Read the indexed-attribute list off the disk into the volume.
//
// Called at mount and after any change to the list. A volume with no indexes --
// which is every volume until somebody asks for one -- ends up with an empty
// list and index_wanted answers no without touching anything.
int
schfs_index_reload(struct schfs_vol *v)
{
struct schfs_key k;
schfs_mkkey(&k, SCHFS_INDEX_OBJECTID, SCHFS_KEY_INDEX, SCHFS_INDEX_LIST_OFF);
unsigned len = sizeof(v->idxnames);
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->idxnames, &len);
if (r == -1) {
v->idxnames_len = 0;
return 0;
}
if (r != 0)
return r;
v->idxnames_len = len;
return 0;
}
// Add one object to the index for (attribute name, value).
static int
index_add(struct schfs_vol *v, const char *name, unsigned namelen,
const void *val, unsigned vallen, uint64_t ino)
{
struct schfs_key k;
schfs_mkkey(&k, schfs_namehash(name, namelen), SCHFS_KEY_INDEX,
schfs_namehash((const char *)val, vallen));
unsigned len = SCHFS_MAX_INLINE;
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->dirbuf, &len);
if (r == -1)
len = 0;
else if (r != 0)
return r;
// Already listed? An object that sets the same value twice must not appear
// twice, or a query would return it twice.
unsigned off = 0;
while (off + sizeof(struct schfs_index_rec) <= len) {
const struct schfs_index_rec *ir =
(const struct schfs_index_rec *)(v->dirbuf + off);
unsigned step = SCHFS_INDEX_LEN(ir->valuelen);
if (off + step > len)
return -2;
if (ir->objectid == ino)
return 0;
off += step;
}
unsigned add = SCHFS_INDEX_LEN(vallen);
if (len + add > SCHFS_MAX_INLINE)
return -3; /* too many objects share this value */
struct schfs_index_rec ir;
memset(&ir, 0, sizeof(ir));
ir.objectid = ino;
ir.valuelen = vallen;
memmove(v->dirbuf + len, &ir, sizeof(ir));
memmove(v->dirbuf + len + sizeof(ir), val, vallen);
// The padding is zeroed rather than left as whatever was in the buffer, so
// two identical filesystems produce byte-identical images.
memset(v->dirbuf + len + sizeof(ir) + vallen, 0,
add - sizeof(ir) - vallen);
return schfs_btree_insert(&v->tree, &v->ctx, &k, v->dirbuf, len + add);
}
// Remove one object from the index for (attribute name, value).
static int
index_del(struct schfs_vol *v, const char *name, unsigned namelen,
const void *val, unsigned vallen, uint64_t ino)
{
struct schfs_key k;
schfs_mkkey(&k, schfs_namehash(name, namelen), SCHFS_KEY_INDEX,
schfs_namehash((const char *)val, vallen));
unsigned len = SCHFS_MAX_INLINE;
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->dirbuf, &len);
if (r == -1)
return 0;
if (r != 0)
return r;
unsigned off = 0;
while (off + sizeof(struct schfs_index_rec) <= len) {
struct schfs_index_rec *ir = (struct schfs_index_rec *)(v->dirbuf + off);
unsigned step = SCHFS_INDEX_LEN(ir->valuelen);
if (off + step > len)
return -2;
if (ir->objectid == ino) {
memmove(v->dirbuf + off, v->dirbuf + off + step, len - off - step);
len -= step;
if (len == 0)
return schfs_btree_delete(&v->tree, &v->ctx, &k);
return schfs_btree_insert(&v->tree, &v->ctx, &k, v->dirbuf, len);
}
off += step;
}
return 0;
}
/* ------------------------------------------------------------------ */
/* Writing */
/* ------------------------------------------------------------------ */
// Set an attribute, replacing any previous value of the same name.
//
// The index is kept in step here rather than by a separate pass, because an
// index that can be out of date is worse than none: a query would answer
// confidently and wrongly.
int
schfs_attr_set(struct schfs_vol *v, uint64_t ino, const char *name,
unsigned namelen, const void *val, unsigned vallen,
uint16_t type)
{
if (namelen == 0 || namelen > SCHFS_ATTR_NAME_MAX)
return -1;
if (vallen > SCHFS_ATTR_VALUE_MAX)
return -1;
schfs_cache_invalidate();
// The old value has to come out of the index before the new one goes in,
// and it has to be read before the record is overwritten.
int indexed = index_wanted(v, name, namelen);
char oldval[SCHFS_ATTR_VALUE_MAX];
unsigned oldlen = 0;
int had_old = 0;
if (indexed) {
unsigned vl = 0;
int g = schfs_attr_get(v, ino, name, namelen, oldval, sizeof(oldval),
&vl, 0);
if (g == 1) { had_old = 1; oldlen = vl; }
}
struct schfs_key k;
schfs_mkkey(&k, ino, SCHFS_KEY_ATTR, schfs_namehash(name, namelen));
unsigned len = SCHFS_MAX_INLINE;
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->dirbuf, &len);
if (r == -1)
len = 0;
else if (r != 0)
return r;
// Drop any record with this name; what remains is everything that merely
// hashed the same.
unsigned off = 0;
while (off + sizeof(struct schfs_attr_rec) <= len) {
struct schfs_attr_rec *a = (struct schfs_attr_rec *)(v->dirbuf + off);
unsigned step = SCHFS_ATTR_LEN(a->namelen, a->valuelen);
if (a->namelen == 0 || off + step > len)
return -2;
const char *nm = (const char *)a + sizeof(*a);
if (a->namelen == namelen && memcmp(nm, name, namelen) == 0) {
memmove(v->dirbuf + off, v->dirbuf + off + step, len - off - step);
len -= step;
break;
}
off += step;
}
unsigned add = SCHFS_ATTR_LEN(namelen, vallen);
if (len + add > SCHFS_MAX_INLINE)
return -3;
struct schfs_attr_rec a;
memset(&a, 0, sizeof(a));
a.namelen = namelen;
a.valuelen = vallen;
a.type = type;
memmove(v->dirbuf + len, &a, sizeof(a));
memmove(v->dirbuf + len + sizeof(a), name, namelen);
memmove(v->dirbuf + len + sizeof(a) + namelen, val, vallen);
memset(v->dirbuf + len + sizeof(a) + namelen + vallen, 0,
add - sizeof(a) - namelen - vallen);
r = schfs_btree_insert(&v->tree, &v->ctx, &k, v->dirbuf, len + add);
if (r != 0)
return r;
if (indexed) {
if (had_old) {
r = index_del(v, name, namelen, oldval, oldlen, ino);
if (r != 0)
return r;
}
r = index_add(v, name, namelen, val, vallen, ino);
if (r != 0)
return r;
}
return 0;
}
// Remove an attribute. Absent is not a failure.
int
schfs_attr_del(struct schfs_vol *v, uint64_t ino, const char *name,
unsigned namelen)
{
if (namelen == 0 || namelen > SCHFS_ATTR_NAME_MAX)
return -1;
schfs_cache_invalidate();
int indexed = index_wanted(v, name, namelen);
char oldval[SCHFS_ATTR_VALUE_MAX];
unsigned oldlen = 0;
int had_old = 0;
if (indexed) {
unsigned vl = 0;
int g = schfs_attr_get(v, ino, name, namelen, oldval, sizeof(oldval),
&vl, 0);
if (g == 1) { had_old = 1; oldlen = vl; }
}
struct schfs_key k;
schfs_mkkey(&k, ino, SCHFS_KEY_ATTR, schfs_namehash(name, namelen));
unsigned len = SCHFS_MAX_INLINE;
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->dirbuf, &len);
if (r == -1)
return 0;
if (r != 0)
return r;
unsigned off = 0;
int found = 0;
while (off + sizeof(struct schfs_attr_rec) <= len) {
struct schfs_attr_rec *a = (struct schfs_attr_rec *)(v->dirbuf + off);
unsigned step = SCHFS_ATTR_LEN(a->namelen, a->valuelen);
if (a->namelen == 0 || off + step > len)
return -2;
const char *nm = (const char *)a + sizeof(*a);
if (a->namelen == namelen && memcmp(nm, name, namelen) == 0) {
memmove(v->dirbuf + off, v->dirbuf + off + step, len - off - step);
len -= step;
found = 1;
break;
}
off += step;
}
if (!found)
return 0;
r = (len == 0) ? schfs_btree_delete(&v->tree, &v->ctx, &k)
: schfs_btree_insert(&v->tree, &v->ctx, &k, v->dirbuf, len);
if (r != 0)
return r;
if (indexed && had_old)
return index_del(v, name, namelen, oldval, oldlen, ino);
return 0;
}
/* ------------------------------------------------------------------ */
/* Declaring an index, and querying it */
/* ------------------------------------------------------------------ */
// Start indexing an attribute.
//
// Only affects values written *after* this: there is no reindexing pass, and
// pretending otherwise would be the kind of half-truth that makes a query
// answer confidently and wrongly. A caller that wants existing objects in the
// index rewrites their attributes.
int
schfs_index_create(struct schfs_vol *v, const char *name, unsigned namelen)
{
if (namelen == 0 || namelen > SCHFS_ATTR_NAME_MAX)
return -1;
schfs_cache_invalidate();
if (index_wanted(v, name, namelen))
return 0; /* already indexed */
if (v->idxnames_len + 1 + namelen > sizeof(v->idxnames))
return -3;
v->idxnames[v->idxnames_len] = (char)namelen;
memmove(v->idxnames + v->idxnames_len + 1, name, namelen);
v->idxnames_len += 1 + namelen;
struct schfs_key k;
schfs_mkkey(&k, SCHFS_INDEX_OBJECTID, SCHFS_KEY_INDEX, SCHFS_INDEX_LIST_OFF);
return schfs_btree_insert(&v->tree, &v->ctx, &k, v->idxnames,
v->idxnames_len);
}
// Every object whose `name` attribute equals `val`.
//
// One tree descent, whatever the size of the filesystem -- which is the entire
// point. Returns how many object ids were written to `out`, or negative on
// error. A caller with a full buffer gets `cap` and no way to ask for more,
// which is a limit worth having rather than an unbounded kernel allocation.
int
schfs_query_equal(struct schfs_vol *v, const char *name, unsigned namelen,
const void *val, unsigned vallen, uint64_t *out, int cap)
{
if (namelen == 0 || namelen > SCHFS_ATTR_NAME_MAX)
return -1;
if (!index_wanted(v, name, namelen))
return -1; /* not indexed: there is nothing to scan */
struct schfs_key k;
schfs_mkkey(&k, schfs_namehash(name, namelen), SCHFS_KEY_INDEX,
schfs_namehash((const char *)val, vallen));
unsigned len = SCHFS_MAX_INLINE;
int r = schfs_btree_lookup(&v->tree, &v->ctx, &k, v->dirbuf, &len);
if (r == -1)
return 0; /* no object carries that value */
if (r != 0)
return r;
int n = 0;
unsigned off = 0;
while (off + sizeof(struct schfs_index_rec) <= len && n < cap) {
const struct schfs_index_rec *ir =
(const struct schfs_index_rec *)(v->dirbuf + off);
unsigned step = SCHFS_INDEX_LEN(ir->valuelen);
if (off + step > len)
return -2;
// The value is stored, not just its hash, and this is why: two values
// can hash alike, and a query that returned the wrong file would be a
// far worse failure than a slow one.
if (ir->valuelen == vallen &&
memcmp((const char *)ir + sizeof(*ir), val, vallen) == 0)
out[n++] = ir->objectid;
off += step;
}
return n;
}

94
kernel/fs/schfs/btree.c
View file

@ -30,16 +30,44 @@
/* CRC32c (Castagnoli), table-free so this stays usable very early. */
/* ------------------------------------------------------------------ */
// Table-driven, with the table built on first use.
//
// This was a bit-at-a-time loop -- eight shifts per byte, 32768 iterations to
// checksum one 4096-byte block -- justified by staying usable before anything
// is initialised. The table keeps that property, because it builds itself on
// the first call rather than needing an init hook: the cost is one 1 KB static
// array and a first call that does the work the old loop did for 256 bytes.
//
// It matters because every block read off the medium is checksummed in full.
// After the cursor stopped re-reading resident blocks this became the largest
// remaining cost in the read path, which is the usual shape of these things --
// removing the top item promotes whatever was underneath it.
static uint32_t crc32c_tab[256];
static int crc32c_tab_ready;
static void
crc32c_build(void)
{
for (unsigned i = 0; i < 256; i++) {
uint32_t c = i;
for (int k = 0; k < 8; k++)
c = (c >> 1) ^ (0x82F63B78u & (uint32_t)(-(int32_t)(c & 1)));
crc32c_tab[i] = c;
}
crc32c_tab_ready = 1;
}
uint32_t
schfs_crc32c(uint32_t crc, const void *buf, unsigned len)
{
const uint8_t *p = (const uint8_t *)buf;
if (!crc32c_tab_ready)
crc32c_build();
crc = ~crc;
while (len--) {
crc ^= *p++;
for (int i = 0; i < 8; i++)
crc = (crc >> 1) ^ (0x82F63B78u & (uint32_t)(-(int32_t)(crc & 1)));
}
while (len--)
crc = (crc >> 8) ^ crc32c_tab[(crc ^ *p++) & 0xFF];
return ~crc;
}
@ -260,18 +288,64 @@ schfs_btree_create(struct schfs_btree *t, struct schfs_bdev *dev,
void
schfs_cursor_init(struct schfs_cursor *c, struct schfs_btree *t, void *blk)
{
c->t = t;
c->blk = blk;
c->depth = 0;
c->valid = 0;
c->t = t;
c->blk = blk;
c->depth = 0;
c->valid = 0;
// Whatever the previous user of this buffer left in it is not ours to
// trust: a mutation between two cursors could have rewritten that block in
// place, and the copy here would be a generation behind.
c->resident = 0;
c->epoch = 0;
}
// Bumped by schfs_cache_invalidate(), which every write entry point calls
// before it touches anything. A cursor's resident block is only trusted while
// this has not moved: within one transaction group the tree rewrites its own
// nodes *in place*, so a cached copy of block N can be a generation behind
// while still carrying block N's own number in its header. Nothing else
// distinguishes the two, which is why identity checks alone are not enough.
uint64_t schfs_tree_epoch = 1;
// Load path[d]'s block into the cursor buffer, verifying it against the
// checksum recorded alongside it.
//
// The early return is the whole reason this system was slow.
//
// schfs_cursor_val calls this on *every* access to the current record, and
// iter_item calls schfs_cursor_val once per directory entry -- deliberately, so
// that a callback which disturbs the shared buffer cannot corrupt the walk. But
// node_read is not cheap: it copies 4096 bytes out of the buffer cache and then
// runs CRC32c over all 4096. Listing a 243-entry directory did that once per
// entry, and `ls /bin` moved several megabytes through both. Measured across
// one session: 398,999 block reads against 445 actual disk misses, so ~99.9% of
// them were re-reading a block already sitting in the buffer.
//
// Two conditions, and both are needed. `resident` says this cursor put that
// block here; the header says the page still holds it, which is what catches a
// second cursor sharing the same page having loaded something else in between.
// The header alone would be enough for identity, but a block cached from before
// an in-place rewrite has the same blocknr -- so the per-cursor field, cleared
// at init, is what bounds the trust to one entry point's lifetime.
static int
cur_load(struct schfs_cursor *c, int d)
{
return node_read(c->t->dev, c->path[d].blocknr, c->path[d].crc, c->blk);
uint64_t want = c->path[d].blocknr;
if (c->resident == want && c->epoch == schfs_tree_epoch) {
const struct schfs_node_hdr *h = HDR(c->blk);
if (h->magic == SCHFS_NODE_MAGIC && h->blocknr == want &&
(c->path[d].crc == 0 || c->path[d].crc == h->crc))
return 0;
}
if (node_read(c->t->dev, want, c->path[d].crc, c->blk) != 0) {
c->resident = 0;
return -2;
}
c->resident = want;
c->epoch = schfs_tree_epoch;
return 0;
}
// From the node currently in the buffer at depth `d`, walk down to a leaf,

28
kernel/fs/schfs/btree.h
View file

@ -40,6 +40,19 @@ struct schfs_bdev {
// blocks from balloc. NULL means unsupported; callers then fall back to
// allocating singly and accept the extra extent records.
uint64_t (*balloc_run)(void *ctx, uint64_t n);
// Optional: make every write issued so far durable, and do not return until
// it is. NULL means the backing store has no volatile stage to worry about
// -- a memory-backed device, or a host file the tools fsync themselves.
//
// This is the one hook the format's crash-safety argument actually rests
// on. Copy-on-write guarantees that a tree is never half-modified, but only
// if the blocks a root points at reach the medium *before* the block that
// names that root. Nothing in the ordering of bwrite calls guarantees that:
// a drive with a write cache is free to complete them in any order it
// likes. Without a barrier between the two, a power cut lands on a newest
// uberblock whose tree was never written -- exactly the failure copy-on-
// write is supposed to make impossible.
int (*bflush)(void *ctx);
};
struct schfs_btree {
@ -93,9 +106,24 @@ struct schfs_cursor {
void *blk; // caller-owned, SCHFS_BLKSIZE bytes
int depth; // path[depth] is the leaf
int valid; // 0 once the walk has run off either end
// Which block `blk` currently holds, as far as this cursor knows, or 0 for
// "nothing yet". See cur_load: re-reading a block that is already in the
// buffer costs a 4 KB copy and a CRC32c over the whole 4 KB, and the code
// was paying both on every single access to the current record.
//
// Cleared by schfs_cursor_init rather than trusted across cursors, because
// the buffer belongs to the caller and the *next* caller's contents are not
// this cursor's business. Cross-checked against the block header on every
// use, so a second cursor sharing the same page cannot fool it either.
uint64_t resident;
uint64_t epoch; // schfs_tree_epoch when it was loaded
struct schfs_path_ent path[SCHFS_MAX_LEVEL + 1];
};
// Every mutation bumps this, retiring every cursor's cached block. See
// cur_load in btree.c.
extern uint64_t schfs_tree_epoch;
// `blk` must point at SCHFS_BLKSIZE bytes that outlive the cursor.
void schfs_cursor_init(struct schfs_cursor *c, struct schfs_btree *t, void *blk);

1031
kernel/fs/schfs/kmount.c
View file

File diff suppressed because it is too large Load diff

48
kernel/fs/schfs/obj.c
View file

@ -246,7 +246,9 @@ schfs_dir_lookup(struct schfs_vol *v, struct schfs_cursor *c,
}
struct iter_ctx {
int (*fn)(uint64_t, uint16_t, const char *, unsigned, void *);
// The hash-aware form. schfs_dir_iterate's plain callback is wrapped in one
// of these, so there is one walk rather than two.
schfs_dir_fn fn;
void *arg;
struct schfs_cursor *cur;
uint64_t dir;
@ -310,7 +312,7 @@ iter_item(const struct schfs_key *k, const void *val, unsigned len, void *arg)
uint16_t type = d->type;
memmove(name, (const char *)d + sizeof(*d), namelen);
int r = ic->fn(oid, type, name, namelen, ic->arg);
int r = ic->fn(oid, type, name, namelen, k->offset, ic->arg);
if (r) {
ic->stop = r;
return 1;
@ -320,13 +322,17 @@ iter_item(const struct schfs_key *k, const void *val, unsigned len, void *arg)
return 0;
}
// Walk a directory from a given point in its key space.
//
// `start_hash` is where to resume: entries are keyed (dir, DIRENT, hash), so
// seeking straight to a hash is what makes a partial listing continue in one
// descent instead of re-walking everything before it. See schfs_vfs_getdents.
int
schfs_dir_iterate(struct schfs_vol *v, struct schfs_cursor *c, uint64_t dir,
int (*fn)(uint64_t, uint16_t, const char *, unsigned, void *),
void *arg)
schfs_dir_iterate_from(struct schfs_vol *v, struct schfs_cursor *c, uint64_t dir,
uint64_t start_hash, schfs_dir_fn fn, void *arg)
{
struct schfs_key start;
schfs_mkkey(&start, dir, SCHFS_KEY_DIRENT, 0);
schfs_mkkey(&start, dir, SCHFS_KEY_DIRENT, start_hash);
struct iter_ctx ic;
ic.fn = fn; ic.arg = arg; ic.cur = c; ic.dir = dir; ic.stop = 0;
@ -338,6 +344,31 @@ schfs_dir_iterate(struct schfs_vol *v, struct schfs_cursor *c, uint64_t dir,
return ic.stop == 1 ? 0 : ic.stop;
}
// The plain form, for callers that want the whole directory and do not care
// where in the key space an entry sat.
struct plain_ctx {
int (*fn)(uint64_t, uint16_t, const char *, unsigned, void *);
void *arg;
};
static int
plain_entry(uint64_t ino, uint16_t type, const char *name, unsigned namelen,
uint64_t hash, void *arg)
{
struct plain_ctx *p = arg;
(void)hash;
return p->fn(ino, type, name, namelen, p->arg);
}
int
schfs_dir_iterate(struct schfs_vol *v, struct schfs_cursor *c, uint64_t dir,
int (*fn)(uint64_t, uint16_t, const char *, unsigned, void *),
void *arg)
{
struct plain_ctx p = { fn, arg };
return schfs_dir_iterate_from(v, c, dir, 0, plain_entry, &p);
}
/* ------------------------------------------------------------------ */
/* File data */
/* ------------------------------------------------------------------ */
@ -375,6 +406,11 @@ static struct {
void
schfs_cache_invalidate(void)
{
// Retires every cursor's resident block as well as the two caches below:
// an in-place rewrite leaves a stale copy that is indistinguishable from
// the real thing by inspection, so the only safe signal is "something
// changed". See cur_load in btree.c.
schfs_tree_epoch++;
for (unsigned i = 0; i < INO_CACHE_N; i++)
ino_cache[i].valid = 0;
ext_cache.valid = 0;

327
kernel/fs/schfs/objw.c
View file

@ -295,6 +295,135 @@ csum_del(struct schfs_vol *v, uint64_t dblk)
return r == -1 ? 0 : r; /* absent is not a failure */
}
/* ------------------------------------------------------------------ */
/* Extent references */
/* */
/* How many files point at a run of blocks. Only shared runs have a */
/* record; absence means one reference. See SCHFS_EXTREF_OBJECTID. */
/* ------------------------------------------------------------------ */
// Find the reference record *covering* `dblk`, which is not necessarily one
// keyed at it.
//
// This is the whole subtlety of the extent tree. A record is keyed by the block
// a run was allocated at, but a file's extent may point into the middle of that
// run -- splitting a shared extent around a written block leaves a tail whose
// first block is several into the run. Looking up by that block finds nothing,
// and "nothing" means "unshared", so the tail would be freed while another file
// still reached it. Every lookup therefore has to be a range query.
//
// Returns 1 and fills `out` (and `*runstart`) if a record covers dblk, 0 if
// none does -- meaning one reference -- or -2 on corruption.
static int
extref_find(struct schfs_vol *v, uint64_t dblk, uint64_t *runstart,
struct schfs_extref *out)
{
struct schfs_cursor c;
schfs_cursor_init(&c, &v->tree, v->ctx.a);
struct schfs_key probe;
schfs_mkkey(&probe, SCHFS_EXTREF_OBJECTID, SCHFS_KEY_EXTREF, dblk);
int r = schfs_btree_lookup_le(&c, &probe);
if (r == -2)
return -2;
if (r != 0)
return 0;
const struct schfs_key *got = schfs_cursor_key(&c);
unsigned vlen;
const struct schfs_extref *found =
(const struct schfs_extref *)schfs_cursor_val(&c, &vlen);
if (!got || !found)
return 0;
if (got->objectid != SCHFS_EXTREF_OBJECTID || got->type != SCHFS_KEY_EXTREF)
return 0;
if (vlen != sizeof(*out))
return -2;
if (dblk >= got->offset + found->nblocks)
return 0; /* the nearest record ends before dblk */
if (runstart) *runstart = got->offset;
*out = *found;
return 1;
}
static int
extref_put(struct schfs_vol *v, uint64_t dblk, const struct schfs_extref *e)
{
struct schfs_key k;
schfs_mkkey(&k, SCHFS_EXTREF_OBJECTID, SCHFS_KEY_EXTREF, dblk);
return schfs_btree_insert(&v->tree, &v->ctx, &k, e, sizeof(*e));
}
static int
extref_del(struct schfs_vol *v, uint64_t dblk)
{
struct schfs_key k;
schfs_mkkey(&k, SCHFS_EXTREF_OBJECTID, SCHFS_KEY_EXTREF, dblk);
int r = schfs_btree_delete(&v->tree, &v->ctx, &k);
return r == -1 ? 0 : r; /* absent is not a failure */
}
// One more file now points at the run covering `dblk`. `nblocks` is used only
// when no record exists yet, to record the run's length.
static int
extref_inc(struct schfs_vol *v, uint64_t dblk, uint64_t nblocks)
{
struct schfs_extref e;
uint64_t start = dblk;
int have = extref_find(v, dblk, &start, &e);
if (have == -2)
return -2;
if (have == 0) {
// First share: the run went from one holder to two, and this is where a
// record starts existing. `nblocks` is recorded now and never changed,
// because it is the length of the *run as allocated* -- see extref_dec.
e.nblocks = nblocks;
e.refs = 2;
} else {
e.refs++;
}
return extref_put(v, start, &e);
}
// One fewer holder. Returns 1 if the run is now unreferenced, filling *nblocks
// with the length to release; 0 if somebody still holds it; -2 on corruption.
//
// The record survives down to one holder rather than being deleted at that
// point, which is the subtle part. A file that shares a run may later be
// truncated inside it, leaving its extent record describing a *shorter* run
// than was allocated. If the record vanished at one reference, the last holder
// would free only as much as its own shortened record described and the tail
// would leak. Keeping the record keeps the original length, so the final
// release frees exactly what was allocated.
static int
extref_dec(struct schfs_vol *v, uint64_t dblk, uint64_t *runstart,
uint64_t *nblocks)
{
struct schfs_extref e;
uint64_t start = dblk;
int have = extref_find(v, dblk, &start, &e);
if (have == -2)
return -2;
if (have == 0) {
*runstart = dblk;
*nblocks = 0; /* no record: the caller's own view is it */
return 1;
}
*runstart = start;
if (e.refs <= 1) {
*nblocks = e.nblocks;
int r = extref_del(v, start);
return r == 0 ? 1 : r;
}
e.refs--;
*nblocks = e.nblocks;
int r = extref_put(v, start, &e);
return r == 0 ? 0 : r;
}
/* ------------------------------------------------------------------ */
/* Extents */
/* ------------------------------------------------------------------ */
@ -417,23 +546,151 @@ extent_grow(struct schfs_vol *v, uint64_t ino, uint64_t fblk,
}
// Release a run of data blocks and the checksum records that describe them.
// Shared runs are left alone: another file still points at them, and proper
// refcounting needs an extent tree that nothing creates yet.
//
// A run somebody else still points at is not released, only dropped: the
// reference count in the extent tree decides, because this file's own copy of
// `refs` cannot be decremented by the other holder.
//
// Note the asymmetry with a partial release. A file whose extent is being
// truncated in the middle releases only the tail, but the reference count is
// per *run*, not per block -- so a shared run is only ever dropped whole. That
// is why extent_release consults the count using the run's first block, and why
// a partial free of a shared run leaves the blocks alone: they are still
// reachable through the other file, in their entirety.
static int
extent_release(struct schfs_vol *v, const struct schfs_extent *e,
uint64_t first, uint64_t n)
{
if (e->refs != 1)
struct schfs_extref xr;
uint64_t runstart = 0;
int shared = extref_find(v, e->disk_block, &runstart, &xr);
if (shared == -2)
return -2;
// Nothing has ever cloned this run, so this file is the only holder and may
// release exactly what it was asked to -- a whole run or the tail of one.
// This is the path every ordinary file takes.
if (shared == 0) {
for (uint64_t i = 0; i < n; i++) {
int r = csum_del(v, first + i);
if (r != 0)
return r;
v->dev.bfree(v->dev.ctx, first + i);
}
return 0;
for (uint64_t i = 0; i < n; i++) {
int r = csum_del(v, first + i);
}
// Shared. A partial release means this file is keeping the head of the run,
// so its record still points into it and the reference stands: the blocks
// stay, because the other holder reaches all of them anyway.
if (first != e->disk_block || n != e->nblocks)
return 0;
uint64_t runlen = 0;
int last = extref_dec(v, e->disk_block, &runstart, &runlen);
if (last == -2)
return -2;
if (!last)
return 0; /* somebody else still points at it */
// Freed from the run's own start and for its allocated length, not from
// this file's view of it -- which may be a slice several blocks in.
if (runlen == 0) {
runstart = e->disk_block;
runlen = e->nblocks;
}
for (uint64_t i = 0; i < runlen; i++) {
int r = csum_del(v, runstart + i);
if (r != 0)
return r;
v->dev.bfree(v->dev.ctx, first + i);
v->dev.bfree(v->dev.ctx, runstart + i);
}
return 0;
}
static int extents_drop_from(struct schfs_vol *v, uint64_t ino,
struct schfs_inode *in, uint64_t from);
// Share one file's data with another, without copying any of it.
//
// The reflink. `dst` must be an existing object; whatever it held is released
// first. Afterwards both files name the same blocks, both are ordinary writable
// files, and the first write to either copies only the run it touches -- which
// is what copy-on-write already does for the tree and now does for data.
//
// The whole operation is metadata: one extent record per run in the source,
// duplicated, plus a reference record per run. Cloning a gigabyte costs the
// same as cloning a byte.
//
// Both sides' extent records are updated, not just the destination's. The
// per-record `refs` is the hint the write path consults before rewriting a run
// in place; leaving the source saying "mine alone" would let a later write to
// the source scribble over the clone.
int
schfs_file_reflink(struct schfs_vol *v, uint64_t src, uint64_t dst)
{
struct schfs_inode si, di;
struct schfs_cursor c;
if (src == dst)
return -1;
schfs_cache_invalidate();
schfs_cursor_init(&c, &v->tree, v->ctx.a);
if (schfs_inode_get(v, &c, src, &si) != 0)
return -1;
schfs_cursor_init(&c, &v->tree, v->ctx.a);
if (schfs_inode_get(v, &c, dst, &di) != 0)
return -1;
if (si.type != SCHFS_T_FILE || di.type != SCHFS_T_FILE)
return -1;
// The destination starts empty. Doing this first means a failure part way
// through leaves a short file rather than one holding a mixture.
if (extents_drop_from(v, dst, &di, 0) != 0)
return -2;
di.blocks = 0;
// Walk the source by file block. Not with a cursor held across the loop:
// every iteration inserts records, which rewrites the very nodes a cursor
// would be standing on.
uint64_t fblk = 0;
uint64_t nblk = (si.size + SCHFS_BLKSIZE - 1) / SCHFS_BLKSIZE;
while (fblk < nblk) {
uint64_t start;
struct schfs_extent e;
int m = extent_find(v, src, fblk, &start, &e);
if (m == -2)
return -2;
if (m == 0) {
fblk++; /* a hole: nothing to share */
continue;
}
int r = extref_inc(v, e.disk_block, e.nblocks);
if (r != 0)
return -2;
// Both sides now say "shared", so neither will rewrite the run in
// place. 2 rather than a real count: it is only ever compared with 1.
struct schfs_extent se = e;
se.refs = 2;
if (extent_put(v, src, start, &se) != 0)
return -2;
if (extent_put(v, dst, start, &se) != 0)
return -2;
di.blocks += e.nblocks;
fblk = start + e.nblocks;
}
di.size = si.size;
di.gen = v->tree.generation;
return schfs_inode_put(v, dst, &di);
}
// Drop everything from file block `from` onward. Used by truncate and by the
// deletion of an object.
static int
@ -616,11 +873,59 @@ block_for_write(struct schfs_vol *v, uint64_t ino, uint64_t fblk,
return r;
}
// The block just replaced rejoins the free map at the next commit, not now:
// until then the committed tree still points at it and a crash lands there.
int r = extent_release(v, &e, old, 1);
if (r != 0)
return r;
// Account for what the split did to the old run.
//
// The reference count on a shared run is "how many file-extent records
// point into it", so a split that leaves a head *and* a tail has added a
// reference, not kept one. Getting this wrong is invisible until every
// holder is deleted and the run is either freed while still in use or
// never freed at all -- the second is what happened first, leaking four
// blocks per split.
{
int pieces = (fblk > start ? 1 : 0)
+ (fblk + 1 < start + e.nblocks ? 1 : 0);
struct schfs_extref xr;
uint64_t runstart = 0;
int shared = extref_find(v, e.disk_block, &runstart, &xr);
if (shared == -2)
return -2;
if (shared) {
if (pieces == 2) {
if (extref_inc(v, e.disk_block, e.nblocks) != 0)
return -2;
} else if (pieces == 0) {
// A one-block run, entirely replaced: this file's only
// reference to it is gone.
uint64_t runlen = 0;
int last = extref_dec(v, e.disk_block, &runstart, &runlen);
if (last == -2)
return -2;
if (last) {
if (runlen == 0) {
runstart = e.disk_block;
runlen = e.nblocks;
}
for (uint64_t i = 0; i < runlen; i++) {
int rr = csum_del(v, runstart + i);
if (rr != 0)
return rr;
v->dev.bfree(v->dev.ctx, runstart + i);
}
}
}
// The replaced block itself is never freed here: it is part of a
// run somebody else still reaches in full.
} else {
// The block just replaced rejoins the free map at the next commit,
// not now: until then the committed tree still points at it and a
// crash lands there.
int r = extent_release(v, &e, old, 1);
if (r != 0)
return r;
}
}
w->disk = nd;
w->src = old;

71
kernel/fs/schfs/schfs_int.h
View file

@ -75,6 +75,11 @@ struct schfs_vol {
int uber_slot; // ring slot the last commit used
int readonly;
// How many newer uberblocks schfs_vol_open had to reject before it found a
// root it could actually read. Zero on every healthy mount. Anything else
// means transactions were lost, which is a recovery the operator has to be
// told about rather than something to absorb quietly.
int recovered;
char label[SCHFS_LABEL_MAX];
uint8_t uuid[16];
@ -82,6 +87,14 @@ struct schfs_vol {
// ctx buffers: a split reuses ctx.b and ctx.c while the value being
// inserted is still being read from, so the value needs storage of its own.
char dirbuf[SCHFS_MAX_INLINE];
// Which attributes are indexed, as a packed list of (length, name), read
// off the disk at mount. Cached here rather than looked up because every
// attribute write asks the question and the answer is almost always no --
// a tree descent to say so would make every write pay for a feature it is
// not using. See kernel/fs/schfs/attr.c.
char idxnames[512];
unsigned idxnames_len;
};
// ---------------------------------------------------------------------------
@ -141,6 +154,12 @@ void schfs_vol_free(struct schfs_vol *v, uint64_t blk);
int schfs_space_init(struct schfs_space *s, uint64_t nblocks,
void *(*palloc)(void));
// Return an already-sized map to "empty volume", reusing its pages. Mounting
// may try several uberblocks, and each candidate root needs its scan to start
// from a clean map -- but the pages were taken once and must not be taken
// again, which is the one thing schfs_space_init cannot do twice.
void schfs_space_reset(struct schfs_space *s, uint64_t nblocks);
// Record a run as allocated. Idempotent, because an extent range may be shared
// by several files.
void schfs_space_mark(struct schfs_space *s, uint64_t blk, uint64_t n);
@ -187,6 +206,15 @@ int schfs_dir_lookup(struct schfs_vol *v, struct schfs_cursor *c,
// Iterate a directory's real entries in hash order. "." and ".." are not
// stored and are not reported; callers that need them synthesise them.
// A non-zero return from fn stops the walk and is passed back.
// A directory entry, with the tree key's hash it was found under. The hash is
// what a partial listing resumes from -- see schfs_dir_iterate_from.
typedef int (*schfs_dir_fn)(uint64_t ino, uint16_t type, const char *name,
unsigned namelen, uint64_t hash, void *arg);
int schfs_dir_iterate_from(struct schfs_vol *v, struct schfs_cursor *c,
uint64_t dir, uint64_t start_hash,
schfs_dir_fn fn, void *arg);
int schfs_dir_iterate(struct schfs_vol *v, struct schfs_cursor *c, uint64_t dir,
int (*fn)(uint64_t ino, uint16_t type, const char *name,
unsigned namelen, void *arg),
@ -244,6 +272,49 @@ int schfs_dir_empty(struct schfs_vol *v, uint64_t dir);
// schfs_file_write keeps the extent count down.
int schfs_file_reserve(struct schfs_vol *v, uint64_t ino, uint64_t nblocks);
// Share `src`'s data blocks with `dst` without copying them. Both remain
// ordinary writable files; the first write to either copies only the run it
// touches. Costs one metadata record per run, whatever the file's size.
int schfs_file_reflink(struct schfs_vol *v, uint64_t src, uint64_t dst);
// ---------------------------------------------------------------------------
// attr.c -- attributes, the index over them, and the query that uses it
// ---------------------------------------------------------------------------
// One attribute. Returns 1 and fills the buffers if present, 0 if absent, -3 if
// `out` is too small (the length is still reported), -2 on corruption.
int schfs_attr_get(struct schfs_vol *v, uint64_t ino, const char *name,
unsigned namelen, void *out, unsigned cap,
unsigned *valuelen, uint16_t *type);
// Every attribute on an object. The callback returns non-zero to stop.
int schfs_attr_iterate(struct schfs_vol *v, struct schfs_cursor *c, uint64_t ino,
int (*fn)(const char *name, unsigned namelen,
const void *val, unsigned vallen,
uint16_t type, void *arg),
void *arg);
// Set or replace an attribute, keeping any index in step.
int schfs_attr_set(struct schfs_vol *v, uint64_t ino, const char *name,
unsigned namelen, const void *val, unsigned vallen,
uint16_t type);
// Remove one. Absent is not a failure.
int schfs_attr_del(struct schfs_vol *v, uint64_t ino, const char *name,
unsigned namelen);
// Start indexing an attribute. Affects values written after this only.
int schfs_index_create(struct schfs_vol *v, const char *name, unsigned namelen);
// Load the indexed-attribute list from the tree into the volume. Called at
// mount, and after the list changes.
int schfs_index_reload(struct schfs_vol *v);
// Every object whose `name` attribute equals `val`. One tree descent, whatever
// the size of the filesystem. Returns how many ids were written, or negative.
int schfs_query_equal(struct schfs_vol *v, const char *name, unsigned namelen,
const void *val, unsigned vallen, uint64_t *out, int cap);
// Write at an arbitrary offset, allocating blocks, extent records and data
// checksums, copying any block that a committed tree can still reach, and
// splitting the extent record around it. Writing past the end extends the file;

68
kernel/fs/schfs/space.c
View file

@ -93,6 +93,39 @@ popcount64(uint64_t v)
/* Setup */
/* ------------------------------------------------------------------ */
// Bring an already-sized map back to "empty volume": every block free except
// the ones that are never allocatable. Split out of schfs_space_init because
// mounting may have to try more than one uberblock, and each candidate root
// needs a scan starting from a clean map -- while the pages themselves were
// taken once and must not be taken again.
void
schfs_space_reset(struct schfs_space *s, uint64_t nblocks)
{
for (int i = 0; i < s->npages; i++) {
memset(s->inuse[i], 0, SCHFS_BLKSIZE);
memset(s->pend[i], 0, SCHFS_BLKSIZE);
}
s->nblocks = nblocks;
s->free_blocks = nblocks - SCHFS_FIRST_BLOCK;
s->pending = 0;
s->hint = SCHFS_FIRST_BLOCK;
// Cleared, not preserved: schfs_space_scan sets RETAIN_ALL when it finds a
// snapshot, so carrying one candidate root's answer into the next root's
// scan would pin blocks for a snapshot that tree may not even have.
s->flags = 0;
// The boot block and the uberblock ring are never allocatable, and neither
// is anything past the end of the volume: the last bitmap page covers more
// blocks than exist, and a stray allocation out there would be written to a
// sector the device does not have.
for (uint64_t b = 0; b < SCHFS_FIRST_BLOCK; b++)
bit_set(s->inuse, b);
for (uint64_t b = nblocks;
b < (uint64_t)s->npages * SCHFS_SPACE_BITS_PER_PAGE; b++)
bit_set(s->inuse, b);
}
int
schfs_space_init(struct schfs_space *s, uint64_t nblocks,
void *(*palloc)(void))
@ -111,30 +144,16 @@ schfs_space_init(struct schfs_space *s, uint64_t nblocks,
void *p = palloc();
if (p == 0)
return -5; /* caller frees what was taken */
memset(p, 0, SCHFS_BLKSIZE);
s->inuse[i] = (uint64_t *)p;
p = palloc();
if (p == 0)
return -5;
memset(p, 0, SCHFS_BLKSIZE);
s->pend[i] = (uint64_t *)p;
s->npages = (int)(i + 1);
}
s->nblocks = nblocks;
s->free_blocks = nblocks - SCHFS_FIRST_BLOCK;
s->hint = SCHFS_FIRST_BLOCK;
// The boot block and the uberblock ring are never allocatable, and neither
// is anything past the end of the volume: the last bitmap page covers more
// blocks than exist, and a stray allocation out there would be written to a
// sector the device does not have.
for (uint64_t b = 0; b < SCHFS_FIRST_BLOCK; b++)
bit_set(s->inuse, b);
for (uint64_t b = nblocks; b < (uint64_t)npages * SCHFS_SPACE_BITS_PER_PAGE; b++)
bit_set(s->inuse, b);
schfs_space_reset(s, nblocks);
return 0;
}
@ -276,6 +295,25 @@ mark_leaf_extents(struct schfs_space *s, const void *blk)
(const struct schfs_extent *)((const char *)blk + it[i].offset);
schfs_space_mark(s, e->disk_block, e->nblocks);
}
// Shared runs, at the length they were *allocated*.
//
// Not redundant with the loop above, and the difference matters. A file
// that shares a run may since have been truncated inside it, so its extent
// record describes a shorter run than exists -- and the tail would be
// derived as free while the reference record still holds it. The allocator
// would then hand out a block that a later release frees a second time.
for (int i = 0; i < n; i++) {
if (it[i].key.type != SCHFS_KEY_EXTREF)
continue;
if (it[i].size != sizeof(struct schfs_extref))
continue;
if ((int)it[i].offset + (int)it[i].size > SCHFS_BLKSIZE)
continue;
const struct schfs_extref *x =
(const struct schfs_extref *)((const char *)blk + it[i].offset);
schfs_space_mark(s, it[i].key.offset, x->nblocks);
}
}
// Snapshot roots found while walking, so they can be walked in turn.

219
kernel/fs/schfs/vfs.c
View file

@ -39,6 +39,7 @@
#include "buf.h"
#include "file.h"
#include "errno.h"
#include "fsstat.h"
#include "schfs_int.h"
@ -91,7 +92,10 @@ schfs_vfs_iload(struct inode *ip)
ip->type = type_to_vfs(in.type);
ip->major = (short)in.rdev_major;
ip->minor = (short)in.rdev_minor;
ip->nlink = (short)in.nlink;
// Clamped, not cast. The on-disk count is a uint32 and the VFS's is a signed
// int, so the top half of the range has nowhere to go; saturating keeps it
// positive, and a file with two billion names is not going to lose one here.
ip->nlink = in.nlink > 0x7fffffffu ? 0x7fffffff : (int)in.nlink;
ip->size = in.size;
ip->rwmode = RW;
ip->supermode = 0;
@ -123,7 +127,7 @@ schfs_vfs_istore(struct inode *ip)
// Read-modify-write rather than build-from-scratch: the on-disk record holds
// fields the VFS has no slot for (uid, gid, the real mode bits, the
// generation), and rebuilding it here would quietly zero every one of them.
if (schfs_kstat(ip->inum, &in) != 0) {
if (schfs_kstat_dev(ip->dev, ip->inum, &in) != 0) {
memset(&in, 0, sizeof(in));
in.mode = ip->type == T_DIR ? 0755 : 0644;
}
@ -137,7 +141,7 @@ schfs_vfs_istore(struct inode *ip)
in.mtime = ip->mtime;
in.ctime = ip->ctime;
schfs_ksetattr(ip->inum, &in);
schfs_ksetattr(ip->dev, ip->inum, &in);
}
static void
@ -145,7 +149,7 @@ schfs_vfs_ifree(struct inode *ip)
{
if (readonly(ip))
return;
schfs_kfree(ip->inum);
schfs_kfree(ip->dev, ip->inum);
}
/* ------------------------------------------------------------------ */
@ -173,6 +177,9 @@ schfs_vfs_ifree(struct inode *ip)
static int
schfs_vfs_read(struct inode *ip, int user_dst, uint64 dst, uint64 off, uint n)
{
uint64 t0 = rdtsc();
fsstat.read_calls++;
char *tmp = kalloc();
if (tmp == 0)
return -1;
@ -199,6 +206,8 @@ schfs_vfs_read(struct inode *ip, int user_dst, uint64 dst, uint64 off, uint n)
}
kfree(tmp);
fsstat.read_bytes += done;
fsstat.read_cycles += rdtsc() - t0;
return (int)done;
}
@ -212,6 +221,22 @@ schfs_vfs_write(struct inode *ip, int user_src, uint64 src, uint64 off, uint n)
if (tmp == 0)
return -1;
// One transaction for the whole call, not one per 4 KB chunk.
//
// Each schfs_kwrite is its own group otherwise, and a group ends in two cache
// flushes and an uberblock write. A flush is a barrier, so committing per
// chunk forbids the buffer cache to coalesce the tree writes that every chunk
// repeats — the same leaf, holding this file's data checksums, went to the
// disk once per chunk. Measured over 1 MB: 1410 block writes per chunk-commit
// against 267 for one group. It is also better semantics, since a short write
// now leaves the file at a length some prefix of this call actually reached
// rather than at whichever chunk happened to commit last.
//
// copyin under the bracket is safe: it walks the page table and returns -1 on
// an unmapped address rather than faulting, and a page fault here would be a
// SIGSEGV in any case — there is no demand paging to re-enter the filesystem.
schfs_ktx_open(ip->dev);
uint done = 0;
while (done < n) {
uint want = n - done;
@ -225,13 +250,19 @@ schfs_vfs_write(struct inode *ip, int user_src, uint64 src, uint64 off, uint n)
memmove(tmp, (char *)(src + done), want);
}
long put = schfs_kwrite(ip->inum, off + done, tmp, want);
long put = schfs_kwrite(ip->dev, ip->inum, off + done, tmp, want);
if (put <= 0) break;
done += (uint)put;
if ((uint)put < want) break;
}
int cr = schfs_ktx_close(ip->dev);
kfree(tmp);
// A failed commit means nothing durably happened, so report it that way
// rather than claiming bytes that are not on the disk.
if (cr != 0)
return -1;
if (off + done > ip->size)
ip->size = off + done;
return (int)done;
@ -242,7 +273,7 @@ schfs_vfs_truncate(struct inode *ip, uint64 newsize)
{
if (readonly(ip))
return;
if (schfs_ktruncate(ip->inum, newsize) == 0)
if (schfs_ktruncate(ip->dev, ip->inum, newsize) == 0)
ip->size = newsize;
}
@ -297,7 +328,7 @@ schfs_vfs_dirlink(struct inode *dp, char *name, uint64 inum)
if (schfs_kstat_dev(dp->dev, inum, &in) != 0)
return -1;
int r = schfs_kdiradd(dp->inum, name, namelen, inum, in.type);
int r = schfs_kdiradd(dp->dev, dp->inum, name, namelen, inum, in.type);
if (r != 0)
return -1;
@ -306,7 +337,7 @@ schfs_vfs_dirlink(struct inode *dp, char *name, uint64 inum)
// parent -- and every ".." walk out of it lands somewhere else entirely.
if (in.type == SCHFS_T_DIR && in.parent != dp->inum) {
in.parent = dp->inum;
schfs_ksetattr(inum, &in);
schfs_ksetattr(dp->dev, inum, &in);
}
return 0;
}
@ -320,21 +351,21 @@ schfs_vfs_dirunlink(struct inode *dp, char *name, uint off)
(void)off;
unsigned namelen = 0;
while (name[namelen]) namelen++;
return schfs_kunlink(dp->inum, name, namelen) == 0 ? 0 : -1;
return schfs_kunlink(dp->dev, dp->inum, name, namelen) == 0 ? 0 : -1;
}
static int
schfs_vfs_dirempty(struct inode *dp)
{
return schfs_kdirempty(dp->inum);
return schfs_kdirempty(dp->dev, dp->inum);
}
static struct inode *
schfs_vfs_ialloc(uint dev, short type)
{
uint64 ino = 0;
if (schfs_kalloc_object(type_from_vfs(type), type == T_DIR ? 0755 : 0644,
&ino) != 0)
if (schfs_kalloc_object(dev, type_from_vfs(type),
type == T_DIR ? 0755 : 0644, &ino) != 0)
return 0;
return iget(dev, ino);
}
@ -345,44 +376,60 @@ schfs_vfs_ialloc(uint dev, short type)
// One record's worth of state for the callback below.
//
// Entries come back in name-hash order, which has nothing to do with their
// names -- POSIX promises no order, and hash order is what makes a lookup one
// tree descent instead of a scan. The position handed back to userland is
// therefore an *index*: entry N of the walk. Resuming means walking to N again,
// which readdir already does linearly, so a full listing stays linear overall
// only because the common case is resuming exactly where the last call stopped.
// ---------------------------------------------------------------------------
// The resume cookie, and why it is a key rather than a count
//
// getdents is called repeatedly until it returns nothing, and each call has to
// carry on where the last stopped. `*poff` is the only state there is, so
// whatever goes in it must be enough to find that place again.
//
// It used to be an ordinal -- "entry N of the walk" -- and resuming meant
// walking from the first entry and skipping N. That is quadratic in the size of
// the directory, and measurably so: listing /bin, 243 entries, walked 914
// entries across five calls, because each call started over. A larger directory
// gets worse faster than it gets bigger.
//
// It is now the tree key to seek to. Entries are keyed (dir, DIRENT, hash), so
// the cookie is a hash and resuming is one descent to that hash. The three
// values below `SCHFS_DIRENT_FIRST` are reserved for the synthesised "." and
// "..", which are not records and would otherwise have no position at all.
//
// One consequence worth stating: a whole hash bucket is emitted or none of it
// is. Buckets hold colliding names, at most SCHFS_MAX_INLINE packed, which is
// under 3 KB of output records -- so any caller with a 4 KB buffer never
// notices. A caller whose buffer cannot hold one bucket gets EINVAL, which is
// what Linux does when a buffer cannot hold a single record.
#define GD_DOT 0 /* emit "." next */
#define GD_DOTDOT 1 /* emit ".." next */
#define GD_ENTRIES 2 /* emit real entries, from hash (cookie - 2) */
struct gd_ctx {
char *out;
int cap;
int written;
uint64 skip; // entries to step over before emitting
uint64 index; // entries seen so far
int full;
uint64 from; // hash to resume at
uint64 next; // hash the *following* call should resume at
int full; // buffer filled; stop the walk
int toobig; // a single bucket did not fit in an empty buffer
uint64 bucket; // hash of the bucket being emitted
int bucket_started;
};
static int
gd_entry(uint64 ino, unsigned short type, const char *name, unsigned namelen,
void *arg)
gd_emit(struct gd_ctx *g, uint64 ino, unsigned short type,
const char *name, unsigned namelen)
{
struct gd_ctx *g = arg;
if (g->index++ < g->skip)
return 0;
if (g->full)
return 1;
if (namelen > 255)
namelen = 255;
int reclen = (8 + 8 + 2 + 1 + (int)namelen + 1 + 7) & ~7;
if (g->written + reclen > g->cap) {
g->full = 1;
return 1; /* stop; the caller will come back */
}
if (g->written + reclen > g->cap)
return -1;
char *k = g->out + g->written;
uint64 *p64 = (uint64 *)k;
p64[0] = ino;
p64[1] = g->index; /* resume position: entries consumed */
p64[1] = g->next;
*(uint16 *)(k + 16) = (uint16)reclen;
*(uint8 *)(k + 18) = type == SCHFS_T_DIR ? 4 /* DT_DIR */
: type == SCHFS_T_DEV ? 2 /* DT_CHR */
@ -391,44 +438,98 @@ gd_entry(uint64 ino, unsigned short type, const char *name, unsigned namelen,
memmove(k + 19, name, namelen);
memset(k + 19 + namelen, 0, reclen - 19 - namelen);
g->written += reclen;
fsstat.getdents_walked++; // entries emitted; no longer entries re-walked
return 0;
}
static int
gd_entry(uint64 ino, unsigned short type, const char *name, unsigned namelen,
uint64 hash, void *arg)
{
struct gd_ctx *g = arg;
if (g->full)
return 1;
// A new bucket. Everything emitted so far is safe to keep, because the
// cookie already points past the previous bucket.
if (!g->bucket_started || hash != g->bucket) {
g->bucket = hash;
g->bucket_started = 1;
g->next = hash + GD_ENTRIES; // resume *at* this bucket if we stop here
}
// The cookie the emitted records carry has to be "past this bucket", or a
// caller that stops reading mid-buffer would come back and get the bucket
// again. Setting it before the emit is what makes that true.
uint64 saved = g->next;
g->next = hash + 1 + GD_ENTRIES;
if (gd_emit(g, ino, type, name, namelen) < 0) {
// Did not fit. Roll back to the start of this bucket so the whole thing is
// re-read next time, and stop.
g->next = saved;
g->full = 1;
if (g->written == 0)
g->toobig = 1;
return 1;
}
return 0;
}
static int
schfs_vfs_getdents(struct inode *dp, char *out, int cap, uint64 *poff)
{
uint64 t0 = rdtsc();
fsstat.getdents_calls++;
struct gd_ctx g;
g.out = out;
g.cap = cap;
g.written = 0;
g.skip = *poff;
g.index = 0;
g.full = 0;
g.out = out;
g.cap = cap;
g.written = 0;
g.from = *poff;
g.next = *poff;
g.full = 0;
g.toobig = 0;
g.bucket = 0;
g.bucket_started = 0;
// "." and ".." are synthesised: they are not records, so a walk would never
// produce them, and every readdir caller expects both.
static const char *dots[2] = { ".", ".." };
for (int i = 0; i < 2; i++) {
if (g.index++ < g.skip)
continue;
int reclen = (8 + 8 + 2 + 1 + (i + 1) + 1 + 7) & ~7;
if (g.written + reclen > cap)
return g.written;
char *k = out + g.written;
uint64 *p64 = (uint64 *)k;
p64[0] = dp->inum; /* close enough for ".."; nothing reads it */
p64[1] = g.index;
*(uint16 *)(k + 16) = (uint16)reclen;
*(uint8 *)(k + 18) = 4; /* DT_DIR */
memmove(k + 19, dots[i], i + 1);
memset(k + 19 + i + 1, 0, reclen - 19 - i - 1);
g.written += reclen;
if (g.from <= GD_DOTDOT) {
static const char *dots[2] = { ".", ".." };
for (int i = (int)g.from; i < 2; i++) {
g.next = (uint64)i + 1;
if (gd_emit(&g, dp->inum, SCHFS_T_DIR, dots[i], (unsigned)(i + 1)) < 0) {
if (g.written == 0)
return -EINVAL;
*poff = g.next;
return g.written;
}
}
g.next = GD_ENTRIES; // real entries start at hash 0
g.from = GD_ENTRIES;
}
if (schfs_kreaddir_dev(dp->dev, dp->inum, gd_entry, &g) != 0 && g.written == 0)
uint64 start_hash = g.from - GD_ENTRIES;
if (schfs_kreaddir_from_dev(dp->dev, dp->inum, start_hash, gd_entry, &g) != 0
&& g.written == 0)
return -1;
*poff = g.index;
if (g.toobig)
return -EINVAL;
// Nothing emitted and the walk ran to the end: this is the end of the
// directory, and the cookie must not move or the caller loops.
if (!g.full && g.written == 0) {
*poff = g.next;
fsstat.getdents_cycles += rdtsc() - t0;
return 0;
}
*poff = g.next;
fsstat.getdents_cycles += rdtsc() - t0;
return g.written;
}

173
kernel/fs/schfs/vol.c
View file

@ -85,70 +85,118 @@ schfs_vol_open(struct schfs_vol *v, struct schfs_bdev *dev,
struct schfs_ctx *x, void *(*palloc)(void))
{
struct schfs_uberblock *u = (struct schfs_uberblock *)x->a;
int best_slot = -1;
uint64_t best_gen = 0;
int io_errors = 0;
int io_errors = 0;
int seen = 0; // valid uberblocks found, at any generation
uint64_t ceiling = ~0ULL; // consider only generations below this
int spaced = 0; // the allocation map is built and owned
v->dev = *dev;
v->ctx = *x;
v->tree.dev = &v->dev;
v->recovered = 0;
// Remember which slot won, not a copy of it.
// Walk the ring newest-first, and keep walking.
//
// An uberblock is a whole SCHFS_BLKSIZE block, and keeping one as a local
// put 4096 bytes on a kernel stack that is 4096 bytes in total — the canary
// in kalloc'd stacks caught it as "kstack overflow" on the very first boot.
// Re-reading the winner costs one block read, once, at mount.
for (int i = 0; i < SCHFS_UBER_COUNT; i++) {
if (v->dev.bread(v->dev.ctx, SCHFS_UBER_START + i, u) != 0) {
io_errors++;
continue; // an unreadable slot is not fatal: try the rest
// This used to take the highest valid generation, check its tree, and give
// up if that failed. But an uberblock passing its own checksum only proves
// the uberblock landed -- not the tree it names. That is exactly the state
// a power cut used to leave behind before commit had a barrier, and it is
// still what a misdirected write or a bad sector under the root produces.
//
// A ring of SCHFS_UBER_COUNT slots exists precisely so there is somewhere
// to fall back *to*. Refusing the mount while 127 older, entirely intact
// roots sat on the disk was throwing away the only thing the ring is for.
for (;;) {
int best_slot = -1;
uint64_t best_gen = 0;
for (int i = 0; i < SCHFS_UBER_COUNT; i++) {
if (v->dev.bread(v->dev.ctx, SCHFS_UBER_START + i, u) != 0) {
if (ceiling == ~0ULL)
io_errors++; // count each slot once, on the first sweep
continue; // an unreadable slot is not fatal: try the rest
}
if (!uber_valid(u))
continue;
if (u->generation >= ceiling)
continue; // already tried, and rejected
if (best_slot < 0 || u->generation > best_gen) {
best_gen = u->generation;
best_slot = i;
}
}
if (ceiling == ~0ULL)
seen = best_slot >= 0;
if (best_slot < 0)
break; // nothing older left to try
ceiling = best_gen; // next round considers strictly older ones
// Re-read the winner rather than keeping a copy of it.
//
// An uberblock is a whole SCHFS_BLKSIZE block, and keeping one as a
// local put 4096 bytes on a kernel stack that is 4096 bytes in total --
// the canary in kalloc'd stacks caught it as "kstack overflow" on the
// very first boot. Re-reading costs one block read per candidate.
if (v->dev.bread(v->dev.ctx, SCHFS_UBER_START + best_slot, u) != 0)
goto next;
if (!uber_valid(u))
continue;
if (best_slot < 0 || u->generation > best_gen) {
best_gen = u->generation;
best_slot = i;
goto next; /* changed under us, or a flaky device */
vol_from_uber(v, u, best_slot);
v->readonly = 1; // until an allocation map exists, below
if (v->dev.bread(v->dev.ctx, v->tree.root, x->a) != 0)
goto next;
{
struct schfs_node_hdr *h = (struct schfs_node_hdr *)x->a;
if (h->magic != SCHFS_NODE_MAGIC || h->blocknr != v->tree.root)
goto next;
}
// Rebuild the allocation map from the tree just adopted. Without a
// palloc the volume stays read-only, which is exactly right for a tool
// that only inspects an image: it can then never allocate, rather than
// allocating from a map that says everything is free.
//
// The scan is also the deepest check this root gets: it visits every
// metadata block and verifies each against the checksum in its parent.
// A root that survives it is not merely present, it is whole -- so a
// scan failure is a reason to try the next candidate, not to give up.
if (palloc) {
if (!spaced) {
// Sizing failures are the caller's problem, not this root's:
// a volume too large to map or a machine out of pages will
// fail identically on every slot in the ring.
int sr = schfs_space_init(&v->space, v->total_blocks, palloc);
if (sr != 0)
return sr;
spaced = 1;
} else {
schfs_space_reset(&v->space, v->total_blocks);
}
if (schfs_space_scan(v) != 0)
goto next;
v->free_blocks = v->space.free_blocks;
v->readonly = 0;
}
// Which attributes are indexed, so attr.c can answer without a lookup
// on every write. A volume with no indexes -- every volume until
// somebody asks for one -- ends up with an empty list.
schfs_index_reload(v);
return 0;
next:
v->recovered++; // this candidate was newer, and unusable
continue;
}
if (best_slot < 0)
return io_errors == SCHFS_UBER_COUNT ? -2 : -1;
if (v->dev.bread(v->dev.ctx, SCHFS_UBER_START + best_slot, u) != 0)
// Nothing usable. Distinguish "not a SchFS volume" from "a SchFS volume
// that cannot be read", because the first is an ordinary answer to a probe
// and the second is a damaged disk somebody needs to hear about.
if (io_errors == SCHFS_UBER_COUNT)
return -2;
if (!uber_valid(u))
return -2; /* changed under us, or a flaky device */
vol_from_uber(v, u, best_slot);
v->readonly = 1; // until an allocation map exists, below
// The newest uberblock passing its own checksum does not prove the tree it
// names is readable — a misdirected write could have landed on the root.
// Check now, while there is still the option of falling back, rather than
// discovering it on the first lookup.
if (v->dev.bread(v->dev.ctx, v->tree.root, x->a) != 0)
return -2;
{
struct schfs_node_hdr *h = (struct schfs_node_hdr *)x->a;
if (h->magic != SCHFS_NODE_MAGIC || h->blocknr != v->tree.root)
return -2;
}
// Rebuild the allocation map from the tree just adopted. Without a palloc
// the volume stays read-only, which is exactly right for a tool that only
// inspects an image: it can then never allocate, rather than allocating
// from a map that says everything is free.
if (palloc) {
int sr = schfs_space_init(&v->space, v->total_blocks, palloc);
if (sr != 0)
return sr;
if (schfs_space_scan(v) != 0)
return -2;
v->free_blocks = v->space.free_blocks;
v->readonly = 0;
}
return 0;
return seen ? -2 : -1;
}
int
@ -181,9 +229,30 @@ schfs_vol_commit(struct schfs_vol *v)
memmove(u->label, v->label, sizeof(u->label));
u->crc = uber_crc(u);
// The barrier the whole design rests on. Every block the new root reaches
// has been handed to bwrite by now, but "handed to bwrite" is not "on the
// medium": the kernel defers tree writes into the buffer cache, and the
// drive under it has a write cache of its own. Both are free to reorder.
//
// So: drain everything, *then* write the block that makes it live. A crash
// before this point leaves the previous uberblock newest and nothing is
// lost; a crash after it finds a tree that is fully present. Without the
// barrier there is a third outcome -- a newest uberblock naming a tree that
// never landed -- and it is unrecoverable, because the checksum on the
// uberblock is perfectly valid.
if (v->dev.bflush && v->dev.bflush(v->dev.ctx) != 0)
return -2;
if (v->dev.bwrite(v->dev.ctx, SCHFS_UBER_START + v->uber_slot, u) != 0)
return -2;
// And the uberblock itself, before this function claims the transaction is
// durable. Everything below acts on that claim: the blocks this group freed
// are handed back to the allocator, which is only safe once the group that
// still referenced them can no longer be what a crash comes back to.
if (v->dev.bflush && v->dev.bflush(v->dev.ctx) != 0)
return -2;
// Order matters here, and it is the whole crash-safety argument in three
// lines. Only now that the new uberblock is on the disk is the group this
// freed blocks in superseded, so only now may those blocks be handed out