diff --git a/docs/architecture/schfs.md b/docs/architecture/schfs.md index 02eac53..cfed11b 100644 --- a/docs/architecture/schfs.md +++ b/docs/architecture/schfs.md @@ -156,21 +156,72 @@ Directory *depth* is unbounded on disk; path resolution is what limits it. | | limit | where it comes from | |---|---|---| -| Volume, mounted | **64 GiB** | the in-RAM allocation bitmap: `SCHFS_SPACE_PAGES` (512) pages × 32768 blocks | -| Volumes mounted at once | **4** | `SCHFS_VOL_SLOTS`; slot 0 is the root | -| Snapshots mounted at once | **4** | `SCHFS_SNAP_SLOTS` | -| Volume, via the buffer cache | **4 TiB** | `bio.c` addresses blocks with a `uint`, and one SchFS block spans four of them | -| RAM per mount | **8 KiB per GiB** of volume | two bitmaps, one bit per block each | +| Volume, mounted | **64 TiB** | the in-RAM allocation bitmap: `SCHFS_SPACE_DIRS` (1024) pointer pages × 512 bitmap pages × 32768 blocks | +| Volumes mounted at once | **16** | `SCHFS_VOL_SLOTS`; slot 0 is the root | +| Snapshots mounted at once | **8** | `SCHFS_SNAP_SLOTS` | +| RAM per mount | **32 KiB per GiB** of volume | one bit per block, plus a page per 128 MiB that has seen a free since the last commit; 32 MiB for 1 TiB | +| Metadata reserve | **1%** of the volume, 1–64 MiB | data and new names stop there with `ENOSPC`; see below | +| Changes lost in a crash | at most **5 s** of them | `SCHFS_COMMIT_TICKS`; `sync` and `fsync` commit at once | | Inline value in a leaf | **1016 bytes** | `SCHFS_MAX_INLINE` = a quarter of a node | | Colliding names in one directory | ~**60**, sharing a hash | the packed list for one hash must fit in one inline value | | Directory entries at 8-byte names | ~**42** per key, unlimited keys | as above | | Records per leaf / children per node | **127 / 84** | 4 KiB minus a 32-byte header | | Mount time | one pass over the metadata | the allocation map is derived, not stored | -The 64 GiB mount ceiling is the one that bites first, and it is one constant: -raising `SCHFS_SPACE_PAGES` to 4096 makes it 512 GiB at 64 KiB of pointers per -mount. It is set where it is because the page-pointer vector lives inside -`struct schfs_vol`, which the kernel holds as static storage. +The mount ceiling was 64 GiB, which refused most laptop disks: the page +vector was one level of 512 pointers inside `struct schfs_vol`. It is two +levels now, the way a page table is, so the struct holds 1024 pointers to +pointer pages and the ceiling is 64 TiB for the same 8 KiB. Raising +`SCHFS_SPACE_DIRS` raises it further. + +## Commits, full disks and deletes + +**Transaction groups commit every 5 seconds**, not after every operation: +from the `schfsd` kernel thread, on `sync(2)` and `fsync(2)`, on unmount and +shutdown, and early when space is short (below). This is the contract ext4 +(5 s), ZFS (5 s) and btrfs (30 s) have, and the crash argument is unchanged: a +commit is one uberblock write, so a crash lands on the last group whole. What +it changes is how much can be lost: at most the last few seconds, and nothing +an `fsync` returned for. + +It used to commit before every `write(2)` returned, two cache flushes and an +uberblock each time. Inside one group a block the group already wrote is +changed in place, so a tree path rewritten a hundred times between commits +goes to the disk once; committing per call threw that away. 1 MB in 4 KB +writes went from 2.8 to 6.6 MB/s on AHCI under QEMU, and from 66 KB/s to +3.9 MB/s on the emulated IDE disk. + +**A full disk stays mountable.** A copy-on-write tree needs new blocks to +change anything, deleting included, and an update that runs out halfway +leaves a half-built tree the next commit publishes. That is what filling the +disk did: the root volume would not mount again. So 1% of the volume (at least +1 MiB, at most 64 MiB) is kept for metadata, as btrfs's global reserve and +ZFS's slop space are. File data stops at it with `ENOSPC`, as a short write +of what fitted. New names stop at half of it. Tree updates in progress and +deletes may use the rest. + +Blocks a group frees stay unusable until it commits, because the committed +tree still points at them. So a group commits early once those are a quarter +of what is free, rather than report `ENOSPC` on a disk with room on it. + +**Deleting a large file goes in 4 MiB steps** from the end, each one a whole +operation the group may commit after, as ext4 restarts its transaction through +a long truncate. One 80 MB delete was 20000 checksum-record deletions in a +single operation, which on a full disk ran out of room halfway. + +**A file is freed on its last close**, not when its last name goes. The VFS +unlinks the name, drops `nlink`, and frees the object when nothing holds it, +so a file removed while open still reads. One that was still open when the +machine went down has no names and no holder: the mount finds such objects +during the scan it already makes, and frees them, as ext4 does with its orphan +list. + +**Snapshots no longer stop space being reused.** A snapshot holds the tree +committed before the group it was made in, so it can only hold blocks written +before that group. A freed block written in or after the newest snapshot's +group goes back to the free map at once (ZFS's birth-time rule). The older +ones are kept until a mount works out which snapshots still reach them. It used +to keep every freed block while any snapshot existed. ### The rest of SchOS @@ -179,7 +230,7 @@ These are VFS and driver limits rather than SchFS ones: - A path is at most **4096 bytes** and a name **255**, Linux's `PATH_MAX` and `NAME_MAX` (`VFS_PATH_MAX`, `VFS_NAME_MAX`). Longer is `ENAMETOOLONG`. Both used to be cut short without a word, at 128 and 29. -- `NMOUNT` allows **8** mounts. +- `NMOUNT` allows **32** mounts, and `NBLKDEV` **32** disks and partitions. - The IDE driver handles the primary channel only: **2** ATA devices. - The IDE driver uses LBA48 past 128 GiB, so the disk is not the limit there. - A single `read`/`write` call is capped at **4 GiB** (`n` is a `uint`); the diff --git a/kernel/fs/schfs/btree.c b/kernel/fs/schfs/btree.c index ab8007e..5e095b3 100644 --- a/kernel/fs/schfs/btree.c +++ b/kernel/fs/schfs/btree.c @@ -240,7 +240,7 @@ cow_block(struct schfs_btree *t, const void *blk, uint64_t old) uint64_t nb = t->dev->balloc(t->dev->ctx); if (nb == 0) return 0; - t->dev->bfree(t->dev->ctx, old); + t->dev->bfree(t->dev->ctx, old, HDR(blk)->generation); return nb; } @@ -1015,7 +1015,7 @@ schfs_btree_delete(struct schfs_btree *t, struct schfs_ctx *x, int drop = (HDR(x->a)->nritems == 0 && depth > 0); if (drop) { - t->dev->bfree(t->dev->ctx, path[depth].blocknr); + t->dev->bfree(t->dev->ctx, path[depth].blocknr, HDR(x->a)->generation); } else { uint64_t newb = cow_block(t, x->a, path[depth].blocknr); if (newb == 0) @@ -1033,7 +1033,8 @@ schfs_btree_delete(struct schfs_btree *t, struct schfs_ctx *x, if (drop) { internal_drop(x->a, path[d].index); if (HDR(x->a)->nritems == 0 && d > 0) { - t->dev->bfree(t->dev->ctx, path[d].blocknr); + t->dev->bfree(t->dev->ctx, path[d].blocknr, + HDR(x->a)->generation); continue; /* keep unlinking upward */ } // A root that lost its last child becomes an empty leaf; leaving it @@ -1074,7 +1075,7 @@ schfs_btree_delete(struct schfs_btree *t, struct schfs_ctx *x, uint64_t old = t->root; t->root = KPTRS(x->a)[0].blocknr; t->root_crc = KPTRS(x->a)[0].crc; - t->dev->bfree(t->dev->ctx, old); + t->dev->bfree(t->dev->ctx, old, HDR(x->a)->generation); } t->root_gen = HDR(x->a)->generation; return 0; diff --git a/kernel/fs/schfs/btree.h b/kernel/fs/schfs/btree.h index e6f2f29..61bc8bd 100644 --- a/kernel/fs/schfs/btree.h +++ b/kernel/fs/schfs/btree.h @@ -32,7 +32,9 @@ struct schfs_bdev { uint64_t (*balloc)(void *ctx); // Release a block that the new tree no longer references. Freeing may only // take effect at commit, since the old tree must stay readable until then. - void (*bfree)(void *ctx, uint64_t blk); + // `gen` is the group that wrote the block, which says whether a snapshot + // can still be holding it -- see schfs_space_free. + void (*bfree)(void *ctx, uint64_t blk, uint64_t gen); // Optional: allocate n physically contiguous blocks, returning the first, // or 0 if that cannot be satisfied. This is what lets a file become one // extent rather than one per block, and it has to be a separate entry point diff --git a/kernel/fs/schfs/kmount.c b/kernel/fs/schfs/kmount.c index e8423c0..68cb8e3 100644 --- a/kernel/fs/schfs/kmount.c +++ b/kernel/fs/schfs/kmount.c @@ -17,16 +17,26 @@ // --------------------------------------------------------------------------- // Transactions // -// Every entry point below that changes anything is one transaction group: it -// takes the volume lock, does its work, and commits before returning. That -// makes an operation durable the moment it returns, and — because a commit is a -// single uberblock write that either lands or does not — indivisible against a -// crash. There is no journal to replay and nothing to undo, because none of the -// work is reachable until that one block is written. +// Every entry point below that changes anything joins the open transaction +// group, and the group commits later: when it is SCHFS_COMMIT_TICKS old, on +// sync(2) or fsync(2), on unmount and shutdown, or early when the blocks it +// freed start to crowd out the free ones. That is how ext4 (5 s by default), +// btrfs (30 s) and ZFS (5 s) all do it, and it is the same contract: a write +// that returned is durable once it has been synced, and a crash loses at most +// the last few seconds and never leaves a half-done operation, because a +// commit is a single uberblock write that either lands or does not. There is +// still no journal to replay and nothing to undo. // -// The cost is two cache flushes and an uberblock write per operation, and it is -// steep enough that callers doing many related changes should bracket them — -// see schfs_ktx_open below. Measured over 1 MB of file data: 1410 block writes +// It used to commit before every operation returned. Durable at once, and two +// cache flushes plus an uberblock per write(2): 4 KB writes ran at 2.8 MB/s on +// a disk that takes 5.6 in 64 KB ones, and every block of data cost about five +// block writes, the same tree path rewritten once per call. Inside one group a +// block the group already wrote is changed in place, so a path rewritten a +// hundred times between commits goes to the disk once. +// +// Callers doing many related changes can still bracket them -- see +// schfs_ktx_open below -- which keeps them in one group whatever the clock +// does. Measured over 1 MB of file data: 1410 block writes // committing every 4 KB against 267 for the same megabyte as one group. Not a // tuning difference. A commit contains a flush, and a flush is a barrier, so // committing often is exactly the same thing as forbidding the buffer cache to @@ -119,8 +129,22 @@ struct schfs_mount { int tx_owner_pid; int tx_depth; int tx_dirty; + + // Changes not yet committed, and since when. Written under `lock`; the + // flusher peeks at them without it, and a stale look only delays a commit + // by one of its rounds. + int dirty; + uint dirty_since; }; +// How long a change may wait for its commit, in 10 ms ticks: 5 s, ext4's and +// ZFS's default. The flusher looks every second. +// How much of a file one step of deleting it frees: 4 MB, 1024 blocks. +#define SCHFS_FREE_STEP (1024ull * SCHFS_BLKSIZE) + +#define SCHFS_COMMIT_TICKS 500 +#define SCHFS_FLUSH_TICKS 100 + static struct schfs_mount kmnt[SCHFS_VOL_SLOTS]; // A mounted snapshot: the same block device with an older tree root, read-only. @@ -289,6 +313,8 @@ schfs_lockinfo(int slot, int *locked, int *owner_pid, int *depth) *depth = m->tx_depth; } +static int kdirty_locked(struct schfs_mount *m); + void schfs_ktx_open(uint dev) { @@ -317,7 +343,7 @@ schfs_ktx_close(uint dev) return 0; if (m->tx_dirty) - r = schfs_vol_commit(&m->vol); + r = kdirty_locked(m); m->tx_dirty = 0; m->tx_owner = 0; m->tx_owner_pid = 0; @@ -325,8 +351,117 @@ schfs_ktx_close(uint dev) return r; } +// Commit the open group, if it has anything in it. Caller holds m->lock and +// is not inside someone's bracket. A failed commit leaves the group dirty, so +// the next one tries again. +static int +kcommit_locked(struct schfs_mount *m) +{ + if (!m->dirty || m->vol.readonly) + return 0; + int r = schfs_vol_commit(&m->vol); + if (r == 0) + m->dirty = 0; + return r; +} + +static void +ksleep_ticks(uint n) +{ + acquire(&tickslock); + uint start = ticks; + while (ticks - start < n) + sleep(&ticks, &tickslock); + release(&tickslock); +} + +// The thread that commits what nobody synced. Started by the first change. +static void +schfs_flusher(void) +{ + for (;;) { + ksleep_ticks(SCHFS_FLUSH_TICKS); + for (int i = 0; i < SCHFS_VOL_SLOTS; i++) { + struct schfs_mount *m = &kmnt[i]; + if (!m->mounted || !m->dirty || + ticks - m->dirty_since < SCHFS_COMMIT_TICKS) + continue; + acquiresleep(&m->lock); + if (m->mounted) { + int r = kcommit_locked(m); + if (r != 0) + cprintf("schfs: commit of volume %d failed (%d); retrying\n", i, r); + } + releasesleep(&m->lock); + } + } +} + +// Note that the open group has changes, and commit it now if waiting would +// hurt. Caller holds m->lock. +// +// Blocks a group frees stay unusable until it commits -- the committed tree +// still points at them -- so a group rewriting a large file can tie up free +// space that is really free. Once that is a quarter of what is left, the +// group commits early rather than report ENOSPC on a disk with room on it. +static int +kdirty_locked(struct schfs_mount *m) +{ + static int flusher; + + if (!m->dirty) { + m->dirty = 1; + m->dirty_since = ticks; + } + if (!flusher && kthread_create(schfs_flusher, "schfsd") >= 0) + flusher = 1; + + struct schfs_space *sp = &m->vol.space; + if (sp->nblocks && sp->pending > sp->free_blocks / 4) + return kcommit_locked(m); + return 0; +} + +// Is the volume down to the part of its reserve that new names may not use? +// See schfs_space_low(). Caller holds the lock. +static int +klow(uint dev) +{ + return schfs_space_low(&mnt_for(dev)->vol.space, 2, 0); +} + +// Shorten a file towards `size` in steps, from the end, each one a whole +// operation the group may commit after -- as ext4 restarts its transaction +// through a long truncate. The caller does the last step. Lock held. +// +// Freeing 80 MB at once deleted 20000 checksum records in one operation, and +// every block that freed stays unusable until a commit, so `rm` on a full disk +// ran out of room halfway through deleting and left the file half there. +// Returns 0, or -1 if there is no such object. +static int +kshrink_locked(uint dev, uint64_t ino, uint64_t size) +{ + struct schfs_vol *v = vol_for(dev); + struct schfs_mount *m = mnt_for(dev); + struct schfs_cursor c; + struct schfs_inode in; + + schfs_cursor_init(&c, &v->tree, curblk_for(dev)); + int r = schfs_inode_get(v, &c, ino, &in); + if (r != 0 || in.type != SCHFS_T_FILE) + return r; + uint64_t at = in.size; + while (r == 0 && at > size + SCHFS_FREE_STEP) { + at -= SCHFS_FREE_STEP; + r = schfs_file_truncate(v, ino, at, datablk_for(dev)); + if (r == 0 && !tx_mine(m)) + r = kdirty_locked(m); + } + return r; +} + // Start and finish one transaction. `changed` says whether anything was -// written, so a read does not pay for an uberblock write. +// written, so a read does not dirty the group. static void ktx_begin(uint dev) { @@ -345,7 +480,7 @@ ktx_end(uint dev, int changed, int r) return r; } if (changed && r >= 0) { - int c = schfs_vol_commit(&m->vol); + int c = kdirty_locked(m); if (c != 0) r = c; } @@ -371,7 +506,7 @@ kd_read(void *ctx, uint64_t blk, void *out) return -1; for (int i = 0; i < SCHFS_BPB; i++) { - struct buf *b = bread(d->dev, (uint)(blk * SCHFS_BPB + i)); + struct buf *b = bread(d->dev, blk * SCHFS_BPB + i); memmove((char *)out + i * BSIZE, b->data, BSIZE); brelse(b); } @@ -404,7 +539,7 @@ kd_write(void *ctx, uint64_t blk, const void *in) // into the last one, and costs nothing in safety because kd_flush below // gives the format the one ordering guarantee it actually asks for. for (int i = 0; i < SCHFS_BPB; i++) { - struct buf *b = bgetblk(d->dev, (uint)(blk * SCHFS_BPB + i)); + struct buf *b = bgetblk(d->dev, blk * SCHFS_BPB + i); memmove(b->data, (const char *)in + i * BSIZE, BSIZE); bdirty(b); brelse(b); @@ -435,9 +570,9 @@ kd_alloc_run(void *ctx, uint64_t n) } static void -kd_free(void *ctx, uint64_t b) +kd_free(void *ctx, uint64_t b, uint64_t gen) { - schfs_vol_free(&((struct schfs_mount *)ctx)->vol, b); + schfs_vol_free(&((struct schfs_mount *)ctx)->vol, b, gen); } // Pages for the allocation bitmap: one per 32768 blocks of volume, taken once @@ -472,6 +607,7 @@ kmount_slot(int slot, uint bdev) return -EBUSY; initsleeplock(&m->lock, "schfs"); + m->dirty = 0; if (m->curblk == 0) m->curblk = kalloc(); if (m->datablk == 0) m->datablk = kalloc(); @@ -534,6 +670,20 @@ kmount_slot(int slot, uint bdev) } m->mounted = 1; + + // Free what was unlinked while open and never closed: the space scan above + // found them with no names. Nothing can reach them, so this is the last + // chance to give their blocks back. + if (m->vol.norphans && !m->vol.readonly) { + int n = m->vol.norphans; + for (int i = 0; i < n; i++) + schfs_kfree(SCHFS_DEV + slot, m->vol.orphans[i]); + schfs_ksync(SCHFS_DEV + slot); + cprintf("schfs: freed %d object%s unlinked while open%s\n", n, + n == 1 ? "" : "s", m->vol.orphans_more ? "; more next mount" : ""); + m->vol.norphans = 0; + } + kinfo("schfs: mounted '%s' on device %d, %d MB, generation %d\n", m->vol.label, bdev, (int)(m->vol.total_blocks * SCHFS_BLKSIZE / (1024 * 1024)), @@ -603,6 +753,9 @@ schfs_kumount(uint dev) return -EBUSY; acquiresleep(&m->lock); + if (kcommit_locked(m) != 0) + cprintf("schfs: unmount: the last commit failed; changes since the " + "previous one are lost\n"); bflush(m->kdev.dev); // Give the allocation map back. It is rebuilt from the tree on every mount, // so there is nothing in it worth keeping -- and at 8 KiB per GiB of volume, @@ -853,6 +1006,8 @@ schfs_kcreate(uint dev, uint64_t dir, const char *name, unsigned namelen, if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); int r = schfs_create(vol_for(dev), dir, name, namelen, type, mode, major, minor, know(), out); return ktx_end(dev, 1, r); @@ -878,7 +1033,9 @@ schfs_ktruncate(uint dev, uint64_t ino, uint64_t size) if (!writable_for(dev)) return -1; ktx_begin(dev); - int r = schfs_file_truncate(vol_for(dev), ino, size, datablk_for(dev)); + int r = kshrink_locked(dev, ino, size); + if (r == 0) + r = schfs_file_truncate(vol_for(dev), ino, size, datablk_for(dev)); return ktx_end(dev, 1, r); } @@ -899,6 +1056,8 @@ schfs_klink(uint dev, uint64_t dir, const char *name, unsigned namelen, if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); int r = schfs_link(vol_for(dev), dir, name, namelen, ino); return ktx_end(dev, 1, r); } @@ -928,17 +1087,26 @@ schfs_ksetattr(uint dev, uint64_t ino, const struct schfs_inode *in) return ktx_end(dev, 1, r); } -// Flush without changing anything. Every operation above commits already, but -// sync(2) has to mean something -- and with writes deferred into the buffer -// cache it now means rather more than it did: an empty commit drives the same -// barrier, so anything a bracket left dirty reaches the medium. +// Commit the open group now. What sync(2) and fsync(2) come down to: the +// group is the unit that commits, so making one file durable makes everything +// written before it durable too, as on ext4's data=ordered. +// +// Inside a bracket the commit belongs to whoever opened it, so this only +// marks the group, and the bracket's close commits it. int schfs_ksync(uint dev) { if (!writable_for(dev)) return 0; - ktx_begin(dev); - return ktx_end(dev, 1, 0); + struct schfs_mount *m = mnt_for(dev); + if (tx_mine(m)) { + m->tx_dirty = 1; + return 0; + } + acquiresleep(&m->lock); + int r = kcommit_locked(m); + releasesleep(&m->lock); + return r; } // Sync every mounted volume. What sys_sync wants: it names no filesystem. @@ -986,6 +1154,8 @@ schfs_kalloc_object(uint dev, uint16_t type, uint32_t mode, uint64_t *out) if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); struct schfs_vol *v = vol_for(dev); uint64_t ino = schfs_alloc_objectid(v); @@ -1010,6 +1180,8 @@ schfs_kdiradd(uint dev, uint64_t dir, const char *name, unsigned namelen, if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); int r = schfs_dir_add(vol_for(dev), dir, name, namelen, ino, type); return ktx_end(dev, 1, r); } @@ -1034,7 +1206,9 @@ schfs_kfree(uint dev, uint64_t ino) if (!writable_for(dev)) return -1; ktx_begin(dev); - int r = schfs_object_free(vol_for(dev), ino); + int r = kshrink_locked(dev, ino, 0); + if (r == 0 || r == -1) + r = schfs_object_free(vol_for(dev), ino); return ktx_end(dev, 1, r); } @@ -1057,6 +1231,8 @@ schfs_kreflink(uint dev, uint64_t src, uint64_t dst) if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); int r = schfs_file_reflink(vol_for(dev), src, dst); return ktx_end(dev, 1, r); } @@ -1072,6 +1248,8 @@ schfs_kattr_set(uint dev, uint64_t ino, const char *name, unsigned namelen, if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); int r = schfs_attr_set(vol_for(dev), ino, name, namelen, val, vallen, type); return ktx_end(dev, 1, r); } @@ -1106,6 +1284,8 @@ schfs_kindex(uint dev, const char *name, unsigned namelen) if (!writable_for(dev)) return -1; ktx_begin(dev); + if (klow(dev)) + return ktx_end(dev, 0, -3); int r = schfs_index_create(vol_for(dev), name, namelen); return ktx_end(dev, 1, r); } diff --git a/kernel/fs/schfs/objw.c b/kernel/fs/schfs/objw.c index 0e6d2a2..c3fff95 100644 --- a/kernel/fs/schfs/objw.c +++ b/kernel/fs/schfs/objw.c @@ -84,8 +84,11 @@ schfs_snap_create(struct schfs_vol *v, uint64_t id, const char *name, struct schfs_key k; schfs_mkkey(&k, SCHFS_SNAP_OBJECTID, SCHFS_KEY_SNAP, id); int r = schfs_btree_insert(&v->tree, &v->ctx, &k, &sn, sizeof sn); - if (r == 0) + if (r == 0) { v->space.flags |= SCHFS_SPACE_RETAIN_ALL; + if (sn.created > v->space.snap_gen) + v->space.snap_gen = sn.created; + } return r; } @@ -103,12 +106,11 @@ schfs_snap_delete(struct schfs_vol *v, uint64_t id) if (r != 0) return r; - // Stop retaining. If other snapshots remain, the next mount's derivation - // sets the flag again -- it is the walk that knows, not this function, and - // guessing here would mean keeping a second count to disagree with it. - // Until that mount the allocator is merely conservative, which is the safe - // direction. - v->space.flags &= ~(unsigned)SCHFS_SPACE_RETAIN_ALL; + // Keep retaining. Another snapshot may hold the same old blocks, and only + // the walk at mount knows which remain -- so until then the allocator stays + // conservative, which is the safe direction. This used to clear the flag, + // the unsafe one: deleting one of two snapshots let the other's blocks be + // freed, handed out and overwritten. return 0; } @@ -575,7 +577,7 @@ extent_release(struct schfs_vol *v, const struct schfs_extent *e, int r = csum_del(v, first + i); if (r != 0) return r; - v->dev.bfree(v->dev.ctx, first + i); + v->dev.bfree(v->dev.ctx, first + i, e->gen); } return 0; } @@ -603,7 +605,7 @@ extent_release(struct schfs_vol *v, const struct schfs_extent *e, int r = csum_del(v, runstart + i); if (r != 0) return r; - v->dev.bfree(v->dev.ctx, runstart + i); + v->dev.bfree(v->dev.ctx, runstart + i, e->gen); } return 0; } @@ -754,6 +756,9 @@ schfs_file_reserve(struct schfs_vol *v, uint64_t ino, uint64_t nblocks) // next data block interleaves metadata with data, so consecutive file // blocks are never physically adjacent. Grabbing the run up front keeps it // contiguous and a whole file becomes one record. + if (schfs_space_low(&v->space, 1, nblocks)) + return -3; + uint64_t first; if (v->dev.balloc_run) { first = v->dev.balloc_run(v->dev.ctx, nblocks); @@ -806,6 +811,8 @@ block_for_write(struct schfs_vol *v, uint64_t ino, uint64_t fblk, return -2; if (m == 0) { + if (schfs_space_low(&v->space, 1, 1)) + return -3; uint64_t d = v->dev.balloc(v->dev.ctx); if (d == 0) return -3; @@ -842,6 +849,8 @@ block_for_write(struct schfs_vol *v, uint64_t ino, uint64_t fblk, // it: a head covering what came before, the new block, and a tail covering // what came after. No deletion is needed — when the block being replaced is // the run's first, the middle record simply overwrites the original key. + if (schfs_space_low(&v->space, 1, 1)) + return -3; uint64_t nd = v->dev.balloc(v->dev.ctx); if (nd == 0) return -3; @@ -911,7 +920,7 @@ block_for_write(struct schfs_vol *v, uint64_t ino, uint64_t fblk, int rr = csum_del(v, runstart + i); if (rr != 0) return rr; - v->dev.bfree(v->dev.ctx, runstart + i); + v->dev.bfree(v->dev.ctx, runstart + i, e.gen); } } } @@ -956,6 +965,12 @@ schfs_file_write(struct schfs_vol *v, uint64_t ino, uint64_t off, struct wblk w; r = block_for_write(v, ino, fblk, &w); + // Out of room, at a block boundary and before anything for this block + // was touched: keep what was written, as a short write, and record it + // in the inode. Returning the error alone left those blocks in the + // tree past an end of file nothing had moved. + if (r == -3 && done > 0) + break; if (r != 0) return r; if (w.fresh) @@ -986,8 +1001,8 @@ schfs_file_write(struct schfs_vol *v, uint64_t ino, uint64_t off, done += want; } - if (off + n > in.size) - in.size = off + n; + if (off + done > in.size) + in.size = off + done; in.gen = v->tree.generation; r = schfs_inode_put(v, ino, &in); if (r != 0) diff --git a/kernel/fs/schfs/schfs_int.h b/kernel/fs/schfs/schfs_int.h index 6223bd8..0a0d302 100644 --- a/kernel/fs/schfs/schfs_int.h +++ b/kernel/fs/schfs/schfs_int.h @@ -24,31 +24,54 @@ // One bit per block, held in RAM and derived from the committed tree at mount // rather than stored on disk. See space.c for why. // -// The bitmap is a vector of pages because the kernel allocates a page at a time -// and gives no promise that two are adjacent. That vector is fixed-size, and -// its length is therefore the hard ceiling on how large a volume this build can -// mount: SCHFS_SPACE_PAGES pages x SCHFS_SPACE_BITS_PER_PAGE blocks each. +// The bitmap is paged because the kernel allocates a page at a time and gives +// no promise that two are adjacent, and the pages are found through a second +// level of pages holding pointers, the way a page table finds them. A page of +// bits covers 128 MiB of volume and a page of pointers 64 GiB of it, so the +// fixed top level of SCHFS_SPACE_DIRS entries is what sets the ceiling. +// +// It was one level, 512 pointers held in the struct, which made 64 GiB the +// largest volume this kernel could mount: most laptop disks were refused. // --------------------------------------------------------------------------- #define SCHFS_SPACE_BITS_PER_PAGE ((uint64_t)SCHFS_BLKSIZE * 8) /* 32768 */ -#define SCHFS_SPACE_PAGES 512 +#define SCHFS_SPACE_PTRS_PER_PAGE (SCHFS_BLKSIZE / 8) /* 512 */ +#define SCHFS_SPACE_DIRS 1024 +#define SCHFS_SPACE_PAGES \ + ((uint64_t)SCHFS_SPACE_DIRS * SCHFS_SPACE_PTRS_PER_PAGE) #define SCHFS_MAX_BLOCKS \ - (SCHFS_SPACE_BITS_PER_PAGE * (uint64_t)SCHFS_SPACE_PAGES) /* 64 GiB */ + (SCHFS_SPACE_BITS_PER_PAGE * SCHFS_SPACE_PAGES) /* 64 TiB */ + +// A paged bit vector: dir[i][j] is the page of bits for page number +// i * SCHFS_SPACE_PTRS_PER_PAGE + j. A null page is all zeros. +struct schfs_bits { + uint64_t **dir[SCHFS_SPACE_DIRS]; +}; struct schfs_space { - uint64_t *inuse[SCHFS_SPACE_PAGES]; // 1 = allocated - uint64_t *pend [SCHFS_SPACE_PAGES]; // 1 = freed, but not yet reusable - int npages; + struct schfs_bits inuse; // 1 = allocated + // 1 = freed, but not yet reusable. Only the pages that have seen a free + // since the last commit exist, so the map costs one bit per block rather + // than two: 32 MiB for a 1 TiB volume, not 64. + struct schfs_bits pend; + uint64_t npages; uint64_t nblocks; uint64_t free_blocks; uint64_t pending; // bits set in pend[], counted + // Blocks only metadata may take: see schfs_space_low(). + uint64_t reserve; uint64_t hint; // rotating search start // SCHFS_SPACE_RETAIN_ALL while a snapshot exists -- see schfs_space_free. unsigned flags; + // The newest snapshot's `created` group. A block written in that group or + // later is in no snapshot. + uint64_t snap_gen; // How to give the pages back. Carried here rather than passed to the // release call so that a map can only ever be freed by the allocator that // filled it: the kernel's pages come from kalloc and a host tool's from - // calloc, and the two are not interchangeable. + // calloc, and the two are not interchangeable. `palloc` is kept as well, + // for the pend pages taken as frees arrive. + void *(*palloc)(void); void (*pfree)(void *); }; @@ -107,6 +130,15 @@ struct schfs_vol { // not using. See kernel/fs/schfs/attr.c. char idxnames[512]; unsigned idxnames_len; + + // Objects the mount found with no names left: unlinked while open, and + // the machine went down before the last close freed them. Collected by + // the space scan, freed by whoever mounted the volume, as ext4 does with + // its orphan list. More than fit are found again by the next mount. +#define SCHFS_ORPHANS_MAX 32 + uint64_t orphans[SCHFS_ORPHANS_MAX]; + int norphans; + int orphans_more; }; // --------------------------------------------------------------------------- @@ -154,7 +186,7 @@ int schfs_snap_get(struct schfs_vol *v, struct schfs_cursor *c, uint64_t id, // handle, a device number). Callers wrap these in a one-line adapter instead. uint64_t schfs_vol_alloc(struct schfs_vol *v); uint64_t schfs_vol_alloc_run(struct schfs_vol *v, uint64_t n); -void schfs_vol_free(struct schfs_vol *v, uint64_t blk); +void schfs_vol_free(struct schfs_vol *v, uint64_t blk, uint64_t gen); // --------------------------------------------------------------------------- // space.c @@ -193,10 +225,26 @@ void schfs_space_mark(struct schfs_space *s, uint64_t blk, uint64_t n); uint64_t schfs_space_alloc(struct schfs_space *s); uint64_t schfs_space_alloc_run(struct schfs_space *s, uint64_t n); +// The metadata reserve, as btrfs and ZFS keep one. +// +// A copy-on-write tree needs new blocks to change anything at all, deleting +// included, and an update that runs out halfway leaves a half-built tree the +// next commit publishes. That is what a full disk used to do: the root volume +// would not mount again. So file data and new names stop at the reserve and +// report ENOSPC there, while a tree update in progress, and rm, may use it. +// +// `div` 1 is the data threshold; 2 is the one for creating names, which use +// far less, so a full disk can still be tidied by `mv` and `touch`. +static inline int +schfs_space_low(const struct schfs_space *s, unsigned div, uint64_t want) +{ + return s->nblocks && s->free_blocks < s->reserve / div + want; +} + // Release a block — but only for the *next* transaction group. Until the // commit that supersedes it, the last committed tree still points here and a // crash would land on it. -void schfs_space_free(struct schfs_space *s, uint64_t blk); +void schfs_space_free(struct schfs_space *s, uint64_t blk, uint64_t gen); // Fold this group's deferred frees back into the free map. Called by // schfs_vol_commit once the new uberblock is on the disk, never before. diff --git a/kernel/fs/schfs/space.c b/kernel/fs/schfs/space.c index 4851ded..4854fbf 100644 --- a/kernel/fs/schfs/space.c +++ b/kernel/fs/schfs/space.c @@ -46,39 +46,84 @@ /* ------------------------------------------------------------------ */ /* Bit vectors, paged */ /* */ -/* The bitmap has to be one flat array to be indexable, and the kernel */ -/* hands out memory a page at a time with no promise that two pages */ -/* are adjacent. So it is a vector of page pointers instead: one page */ -/* covers SCHFS_SPACE_BITS_PER_PAGE blocks, and the page vector itself */ -/* is a fixed-size array, which is what puts a hard ceiling on volume */ -/* size. See docs/architecture/schfs.md for the resulting limits. */ +/* The bitmap has to be indexable, and the kernel hands out memory a */ +/* page at a time with no promise that two pages are adjacent. So it */ +/* is pages of bits found through pages of pointers -- see struct */ +/* schfs_bits -- and a missing page reads as zeros. See */ +/* docs/architecture/schfs.md for the resulting limits. */ /* ------------------------------------------------------------------ */ -static int -bit_test(uint64_t *const *pages, uint64_t b) +static uint64_t * +bits_page(const struct schfs_bits *v, uint64_t pg) { - uint64_t pg = b / SCHFS_SPACE_BITS_PER_PAGE; - uint64_t off = b % SCHFS_SPACE_BITS_PER_PAGE; - return (int)((pages[pg][off / 64] >> (off % 64)) & 1); + uint64_t **d = v->dir[pg / SCHFS_SPACE_PTRS_PER_PAGE]; + return d ? d[pg % SCHFS_SPACE_PTRS_PER_PAGE] : 0; } -static void -bit_set(uint64_t *const *pages, uint64_t b) +// The page for `pg`, taken from `palloc` (zeroed) if it is not there yet. +// 0 when there is no memory for it. +static uint64_t * +bits_page_get(struct schfs_bits *v, uint64_t pg, void *(*palloc)(void)) { - uint64_t pg = b / SCHFS_SPACE_BITS_PER_PAGE; + uint64_t ***d = &v->dir[pg / SCHFS_SPACE_PTRS_PER_PAGE]; + if (!*d) { + if (!palloc || !(*d = (uint64_t **)palloc())) + return 0; + memset(*d, 0, SCHFS_BLKSIZE); + } + uint64_t **slot = &(*d)[pg % SCHFS_SPACE_PTRS_PER_PAGE]; + if (!*slot) { + if (!(*slot = (uint64_t *)palloc())) + return 0; + memset(*slot, 0, SCHFS_BLKSIZE); + } + return *slot; +} + +// Give every page back, pointer pages included. +static void +bits_free(struct schfs_bits *v, void (*pfree)(void *)) +{ + for (int i = 0; i < SCHFS_SPACE_DIRS; i++) { + if (!v->dir[i]) + continue; + if (pfree) { + for (int j = 0; j < (int)SCHFS_SPACE_PTRS_PER_PAGE; j++) + if (v->dir[i][j]) + pfree(v->dir[i][j]); + pfree(v->dir[i]); + } + v->dir[i] = 0; + } +} + +static int +bit_test(const struct schfs_bits *v, uint64_t b) +{ + uint64_t *pg = bits_page(v, b / SCHFS_SPACE_BITS_PER_PAGE); uint64_t off = b % SCHFS_SPACE_BITS_PER_PAGE; - pages[pg][off / 64] |= 1ULL << (off % 64); + return pg ? (int)((pg[off / 64] >> (off % 64)) & 1) : 0; +} + +// Only on a page that exists: the inuse map has all of its pages from init, +// and the pend map takes one in schfs_space_free before it sets anything. +static void +bit_set(const struct schfs_bits *v, uint64_t b) +{ + uint64_t *pg = bits_page(v, b / SCHFS_SPACE_BITS_PER_PAGE); + uint64_t off = b % SCHFS_SPACE_BITS_PER_PAGE; + pg[off / 64] |= 1ULL << (off % 64); } // The whole 64-block word containing b, so a scan can skip a fully used region // sixty-four blocks at a time instead of one. On a nearly full volume that is // the difference between an allocation costing microseconds and milliseconds. static uint64_t -bit_word(uint64_t *const *pages, uint64_t b) +bit_word(const struct schfs_bits *v, uint64_t b) { - uint64_t pg = b / SCHFS_SPACE_BITS_PER_PAGE; + uint64_t *pg = bits_page(v, b / SCHFS_SPACE_BITS_PER_PAGE); uint64_t off = b % SCHFS_SPACE_BITS_PER_PAGE; - return pages[pg][off / 64]; + return pg ? pg[off / 64] : 0; } static int @@ -101,40 +146,44 @@ popcount64(uint64_t v) void schfs_space_reset(struct schfs_space *s, uint64_t nblocks) { - for (int i = 0; i < s->npages; i++) { - memset(s->inuse[i], 0, SCHFS_BLKSIZE); - memset(s->pend[i], 0, SCHFS_BLKSIZE); - } + for (uint64_t i = 0; i < s->npages; i++) + memset(bits_page(&s->inuse, i), 0, SCHFS_BLKSIZE); + bits_free(&s->pend, s->pfree); s->nblocks = nblocks; s->free_blocks = nblocks - SCHFS_FIRST_BLOCK; s->pending = 0; + // 1% of the volume, at least 1 MB and at most 64 MB. A tree update touches + // a path of at most SCHFS_MAX_LEVEL nodes per record, and a group rewrites + // a node once however often it changes, so this is generous; btrfs keeps + // between 16 MB and 512 MB for the same job. + s->reserve = nblocks / 100; + if (s->reserve < 256) + s->reserve = 256; + if (s->reserve > 16384) + s->reserve = 16384; s->hint = SCHFS_FIRST_BLOCK; // Cleared, not preserved: schfs_space_scan sets RETAIN_ALL when it finds a // snapshot, so carrying one candidate root's answer into the next root's // scan would pin blocks for a snapshot that tree may not even have. s->flags = 0; + s->snap_gen = 0; // The boot block and the uberblock ring are never allocatable, and neither // is anything past the end of the volume: the last bitmap page covers more // blocks than exist, and a stray allocation out there would be written to a // sector the device does not have. for (uint64_t b = 0; b < SCHFS_FIRST_BLOCK; b++) - bit_set(s->inuse, b); - for (uint64_t b = nblocks; - b < (uint64_t)s->npages * SCHFS_SPACE_BITS_PER_PAGE; b++) - bit_set(s->inuse, b); + bit_set(&s->inuse, b); + for (uint64_t b = nblocks; b < s->npages * SCHFS_SPACE_BITS_PER_PAGE; b++) + bit_set(&s->inuse, b); } void schfs_space_release(struct schfs_space *s) { - if (s->pfree) { - for (int i = 0; i < s->npages; i++) { - if (s->inuse[i]) s->pfree(s->inuse[i]); - if (s->pend[i]) s->pfree(s->pend[i]); - } - } + bits_free(&s->inuse, s->pfree); + bits_free(&s->pend, s->pfree); // Zeroed rather than merely emptied, so calling this twice is harmless and // a released map cannot be mistaken for a sized one. memset(s, 0, sizeof(*s)); @@ -145,7 +194,8 @@ schfs_space_init(struct schfs_space *s, uint64_t nblocks, void *(*palloc)(void), void (*pfree)(void *)) { memset(s, 0, sizeof(*s)); - s->pfree = pfree; + s->palloc = palloc; + s->pfree = pfree; if (nblocks <= SCHFS_FIRST_BLOCK) return -3; @@ -155,27 +205,14 @@ schfs_space_init(struct schfs_space *s, uint64_t nblocks, uint64_t npages = (nblocks + SCHFS_SPACE_BITS_PER_PAGE - 1) / SCHFS_SPACE_BITS_PER_PAGE; + // Every inuse page now; pend pages as frees need them. A failure part way + // leaves what was taken reachable from the map, so release finds it. for (uint64_t i = 0; i < npages; i++) { - // npages is raised as soon as inuse[i] is ours, not after both pages - // are. It used to be raised after, which meant a failure on the second - // allocation left the first one owned by nobody: outside the count, so - // unreachable to any cleanup, and never freed. The comment here said - // "caller frees what was taken" and there was nothing the caller could - // call. - void *p = palloc(); - if (p == 0) { + if (!bits_page_get(&s->inuse, i, palloc)) { schfs_space_release(s); return -5; } - s->inuse[i] = (uint64_t *)p; - s->npages = (int)(i + 1); - - p = palloc(); - if (p == 0) { - schfs_space_release(s); - return -5; - } - s->pend[i] = (uint64_t *)p; + s->npages = i + 1; } schfs_space_reset(s, nblocks); @@ -189,9 +226,9 @@ schfs_space_mark(struct schfs_space *s, uint64_t blk, uint64_t n) uint64_t b = blk + i; if (b < SCHFS_FIRST_BLOCK || b >= s->nblocks) continue; - if (bit_test(s->inuse, b)) + if (bit_test(&s->inuse, b)) continue; /* already counted; extents may be shared */ - bit_set(s->inuse, b); + bit_set(&s->inuse, b); s->free_blocks--; } } @@ -216,12 +253,12 @@ schfs_space_alloc_run(struct schfs_space *s, uint64_t n) uint64_t run = 0, first = 0; for (uint64_t b = lo; b < hi; ) { - if (b % 64 == 0 && bit_word(s->inuse, b) == ~0ULL) { + if (b % 64 == 0 && bit_word(&s->inuse, b) == ~0ULL) { run = 0; b += 64; continue; } - if (bit_test(s->inuse, b)) { + if (bit_test(&s->inuse, b)) { run = 0; b++; continue; @@ -232,7 +269,7 @@ schfs_space_alloc_run(struct schfs_space *s, uint64_t n) b++; if (run == n) { for (uint64_t i = 0; i < n; i++) - bit_set(s->inuse, first + i); + bit_set(&s->inuse, first + i); s->free_blocks -= n; s->hint = first + n < s->nblocks ? first + n : SCHFS_FIRST_BLOCK; return first; @@ -249,32 +286,37 @@ schfs_space_alloc(struct schfs_space *s) } void -schfs_space_free(struct schfs_space *s, uint64_t blk) +schfs_space_free(struct schfs_space *s, uint64_t blk, uint64_t gen) { - // A snapshot exists, so this block may still be reachable from it. + // A snapshot may still be holding this block, and if so it must stay. // // Copy-on-write frees the old block whenever one is rewritten, and that is // exactly the block a snapshot is holding. Freeing it would hand it out // again and overwrite the data the snapshot exists to keep -- silently, // with the damage only visible later when the snapshot is read. // - // Knowing *which* freed blocks a snapshot still needs requires reference - // counting per extent, which is a real design and not a flag. Until that - // exists, the safe answer is to free nothing while a snapshot is present: - // the free map is derived at mount anyway, so everything genuinely - // unreachable is reclaimed on the next mount rather than lost. Conservative - // in the only direction that is safe -- it can under-report free space, and - // can never hand out a block that is still in use. - if (s->flags & SCHFS_SPACE_RETAIN_ALL) + // ZFS's rule says which: a snapshot names the tree committed before the + // group it was created in, so it can hold only blocks written before that + // group. A block born in or after the newest snapshot's group is in none + // of them and goes back to the free map as usual; only the older ones are + // kept, until a mount derives the map from what the snapshots still reach. + // This used to keep everything while any snapshot existed, so a volume + // with one never reused a block it freed until it was remounted. + if ((s->flags & SCHFS_SPACE_RETAIN_ALL) && gen < s->snap_gen) return; if (blk < SCHFS_FIRST_BLOCK || blk >= s->nblocks) return; - if (!bit_test(s->inuse, blk)) + if (!bit_test(&s->inuse, blk)) return; /* never allocated, or freed twice */ - if (bit_test(s->pend, blk)) + if (bit_test(&s->pend, blk)) return; /* already queued this group */ - bit_set(s->pend, blk); + // No page to note it in, and no memory for one: the block stays in use + // until the next mount derives the map again. Lost space for a while, + // never a block handed out twice. + if (!bits_page_get(&s->pend, blk / SCHFS_SPACE_BITS_PER_PAGE, s->palloc)) + return; + bit_set(&s->pend, blk); s->pending++; } @@ -282,9 +324,11 @@ void schfs_space_commit(struct schfs_space *s) { uint64_t words = SCHFS_SPACE_BITS_PER_PAGE / 64; - for (int p = 0; p < s->npages; p++) { - uint64_t *pend = s->pend[p]; - uint64_t *inuse = s->inuse[p]; + for (uint64_t p = 0; p < s->npages; p++) { + uint64_t *pend = bits_page(&s->pend, p); + uint64_t *inuse = bits_page(&s->inuse, p); + if (!pend) + continue; for (uint64_t w = 0; w < words; w++) { if (pend[w] == 0) continue; @@ -293,6 +337,8 @@ schfs_space_commit(struct schfs_space *s) pend[w] = 0; } } + // The pages stay for the next group: a volume being written frees in the + // same places again, and taking and dropping them every commit is waste. s->pending = 0; } @@ -341,6 +387,29 @@ mark_leaf_extents(struct schfs_space *s, const void *blk) } } +// Inodes with no links left, for the mount to free. See schfs_vol.orphans. +static void +collect_orphans(struct schfs_vol *v, const void *blk) +{ + const struct schfs_item *it = SCHFS_ITEMS(blk); + int n = SCHFS_HDR(blk)->nritems; + + for (int i = 0; i < n; i++) { + if (it[i].key.type != SCHFS_KEY_INODE) + continue; + if (it[i].size != sizeof(struct schfs_inode)) + continue; + if ((int)it[i].offset + (int)it[i].size > SCHFS_BLKSIZE) + continue; + const struct schfs_inode *in = + (const struct schfs_inode *)((const char *)blk + it[i].offset); + if (in->nlink != 0 || it[i].key.objectid == v->root_objectid) + continue; + if (v->norphans >= SCHFS_ORPHANS_MAX) { v->orphans_more = 1; continue; } + v->orphans[v->norphans++] = it[i].key.objectid; + } +} + // Snapshot roots found while walking, so they can be walked in turn. // // Collected rather than descended into immediately: the walk keeps one block @@ -351,6 +420,7 @@ mark_leaf_extents(struct schfs_space *s, const void *blk) struct snap_roots { int n; int overflow; + uint64_t newest; /* the largest `created` seen */ uint64_t nr[SNAP_ROOTS_MAX]; uint32_t crc[SNAP_ROOTS_MAX]; }; @@ -370,6 +440,8 @@ collect_snap_roots(struct snap_roots *sr, const void *blk) continue; const struct schfs_snap *sn = (const struct schfs_snap *)((const char *)blk + it[i].offset); + if (sn->created > sr->newest) + sr->newest = sn->created; if (sr->n >= SNAP_ROOTS_MAX) { sr->overflow = 1; continue; } sr->nr[sr->n] = sn->root; sr->crc[sr->n] = sn->root_crc; @@ -411,8 +483,10 @@ scan_root(struct schfs_vol *v, uint64_t root, uint32_t root_crc, if (SCHFS_HDR(blk)->level == 0) { mark_leaf_extents(s, blk); - if (sr) + if (sr) { collect_snap_roots(sr, blk); + collect_orphans(v, blk); + } d--; } else if (path[d].idx < SCHFS_HDR(blk)->nritems) { struct schfs_kptr *kp = &SCHFS_KPTRS(blk)[path[d].idx]; @@ -452,13 +526,18 @@ schfs_space_scan(struct schfs_vol *v) struct snap_roots sr; sr.n = 0; sr.overflow = 0; + sr.newest = 0; + v->norphans = 0; + v->orphans_more = 0; int r = scan_root(v, v->tree.root, v->tree.root_crc, &sr); if (r != 0) return r; - if (sr.n > 0) - v->space.flags |= SCHFS_SPACE_RETAIN_ALL; + if (sr.n > 0 || sr.overflow) { + v->space.flags |= SCHFS_SPACE_RETAIN_ALL; + v->space.snap_gen = sr.newest; + } for (int i = 0; i < sr.n; i++) { r = scan_root(v, sr.nr[i], sr.crc[i], 0); diff --git a/kernel/fs/schfs/vfs.c b/kernel/fs/schfs/vfs.c index 074f2f3..2f49498 100644 --- a/kernel/fs/schfs/vfs.c +++ b/kernel/fs/schfs/vfs.c @@ -240,6 +240,7 @@ schfs_vfs_write(struct inode *ip, int user_src, uint64 src, uint64 off, uint n) schfs_ktx_open(ip->dev); uint done = 0; + int err = 0; while (done < n) { uint want = n - done; if (want > BOUNCE) @@ -253,6 +254,8 @@ schfs_vfs_write(struct inode *ip, int user_src, uint64 src, uint64 off, uint n) } long put = schfs_kwrite(ip->dev, ip->inum, off + done, tmp, want); + if (put < 0 && done == 0) + err = (put == -3) ? -ENOSPC : -EIO; if (put <= 0) break; done += (uint)put; if ((uint)put < want) break; @@ -264,10 +267,10 @@ schfs_vfs_write(struct inode *ip, int user_src, uint64 src, uint64 off, uint n) // A failed commit means nothing durably happened, so report it that way // rather than claiming bytes that are not on the disk. if (cr != 0) - return -1; + return -EIO; if (off + done > ip->size) ip->size = off + done; - return (int)done; + return done ? (int)done : err; } static void @@ -353,7 +356,13 @@ schfs_vfs_dirunlink(struct inode *dp, char *name, uint off) (void)off; unsigned namelen = 0; while (name[namelen]) namelen++; - return schfs_kunlink(dp->dev, dp->inum, name, namelen) == 0 ? 0 : -1; + // The name only, as the note at the top of this file says. This called + // schfs_kunlink, the object layer's own unlink, which also freed the object + // when its last name went -- so a file removed while still open lost its + // data under the process reading it, and freeing a large one ran as one + // operation big enough to exhaust a full disk halfway. The VFS drops nlink + // and calls ifree on the last iput, which frees it in steps. + return schfs_kdirdel(dp->dev, dp->inum, name, namelen) == 0 ? 0 : -1; } static int diff --git a/kernel/fs/schfs/vol.c b/kernel/fs/schfs/vol.c index 5afb35d..7475ac6 100644 --- a/kernel/fs/schfs/vol.c +++ b/kernel/fs/schfs/vol.c @@ -453,7 +453,7 @@ schfs_vol_alloc_run(struct schfs_vol *v, uint64_t n) } void -schfs_vol_free(struct schfs_vol *v, uint64_t blk) +schfs_vol_free(struct schfs_vol *v, uint64_t blk, uint64_t gen) { - schfs_space_free(&v->space, blk); + schfs_space_free(&v->space, blk, gen); } diff --git a/tools/mkschfs.c b/tools/mkschfs.c index a869817..18c7b6f 100644 --- a/tools/mkschfs.c +++ b/tools/mkschfs.c @@ -154,9 +154,9 @@ img_alloc_run(void *ctx, uint64_t n) } static void -img_free(void *ctx, uint64_t b) +img_free(void *ctx, uint64_t b, uint64_t gen) { - schfs_vol_free(((struct imgdev *)ctx)->vol, b); + schfs_vol_free(((struct imgdev *)ctx)->vol, b, gen); } // Pages for the allocation bitmap. calloc rather than malloc so a bug that diff --git a/tools/schfs_fsck.c b/tools/schfs_fsck.c index d9a4e30..606a9df 100644 --- a/tools/schfs_fsck.c +++ b/tools/schfs_fsck.c @@ -108,7 +108,7 @@ img_write(void *ctx, uint64_t b, const void *buf) } static uint64_t img_alloc(void *ctx) { (void)ctx; return 0; } -static void img_nofree(void *ctx, uint64_t b) { (void)ctx; (void)b; } +static void img_nofree(void *ctx, uint64_t b, uint64_t g) { (void)ctx; (void)b; (void)g; } // Pages for the allocation map --check derives. Nothing is written back; this // is a check on the map, not a use of it. diff --git a/tools/schfs_objtest.c b/tools/schfs_objtest.c index 79e9411..275ac64 100644 --- a/tools/schfs_objtest.c +++ b/tools/schfs_objtest.c @@ -80,7 +80,7 @@ md_write(void *ctx, uint64_t b, const void *buf) static uint64_t md_alloc(void *ctx) { (void)ctx; return schfs_vol_alloc(&VOL); } static uint64_t md_run(void *ctx, uint64_t n) { (void)ctx; return schfs_vol_alloc_run(&VOL, n); } -static void md_free(void *ctx, uint64_t b) { (void)ctx; schfs_vol_free(&VOL, b); } +static void md_free(void *ctx, uint64_t b, uint64_t g) { (void)ctx; schfs_vol_free(&VOL, b, g); } static long md_pages_taken, md_pages_given; static void *md_palloc(void) { md_pages_taken++; return calloc(1, SCHFS_BLKSIZE); } // The matching half. Without it every volume this test opened leaked its whole @@ -548,6 +548,14 @@ main(void) long w = schfs_file_write(&VOL, f, off, chunk, sizeof(chunk), DATABLK); if (w == -3) { saw_enospc = 1; break; } + // Running out part way is a short write of what fitted, as + // write(2) is, and the next one is ENOSPC. + if (w >= 0 && w < (long)sizeof(chunk)) { + off += (uint64_t)w; + saw_enospc = schfs_file_write(&VOL, f, off, chunk, + sizeof(chunk), DATABLK) == -3; + break; + } if (w != (long)sizeof(chunk)) { printf(" FAIL: write returned %ld\n", w); good = 0; @@ -665,6 +673,40 @@ main(void) ok(got == (long)sizeof before && memcmp(buf, before, sizeof before) == 0, "a file deleted from the live tree is still in the snapshot"); + // Data written after the snapshot is in no snapshot, so deleting it + // gives its blocks back now rather than at the next mount. The old + // tree nodes the churn copies are kept -- the snapshot may share them + // -- so the check is that the 64 KB of data came back, not that every + // block did. + { + static char fresh[64 * 1024]; + memset(fresh, 'f', sizeof fresh); + uint64_t before_free = VOL.space.free_blocks; + uint64_t f = mkfile(VOL.root_objectid, "fresh"); + schfs_file_write(&VOL, f, 0, fresh, sizeof fresh, DATABLK); + if (schfs_vol_commit(&VOL) != 0) { printf(" FAIL: commit\n"); failures++; } + schfs_dir_del(&VOL, VOL.root_objectid, "fresh", 5); + schfs_object_free(&VOL, f); + if (schfs_vol_commit(&VOL) != 0) { printf(" FAIL: commit\n"); failures++; } + uint64_t lost = before_free > VOL.space.free_blocks + ? before_free - VOL.space.free_blocks : 0; + ok(lost < sizeof fresh / SCHFS_BLKSIZE, + "data written after a snapshot is freed while the snapshot exists"); + if (lost >= sizeof fresh / SCHFS_BLKSIZE) + printf(" (%llu blocks not given back)\n", + (unsigned long long)lost); + + memset(buf, 0, sizeof buf); + snapvol = VOL; + snapvol.tree.root = sn.root; + snapvol.tree.root_crc = sn.root_crc; + snapvol.tree.root_gen = sn.root_generation; + schfs_cursor_init(&c, &snapvol.tree, CURBLK); + got = schfs_file_read(&snapvol, &c, ino, 0, buf, sizeof before, DATABLK); + ok(got == (long)sizeof before && memcmp(buf, before, sizeof before) == 0, + "and the snapshot still reads back after that churn"); + } + // Drop it again, so what it was holding becomes reclaimable and the // volume goes back to exact accounting for the checks that follow. ok(schfs_snap_delete(&VOL, 1) == 0, "the snapshot can be deleted"); diff --git a/tools/schfs_test.c b/tools/schfs_test.c index 945f55c..912f574 100644 --- a/tools/schfs_test.c +++ b/tools/schfs_test.c @@ -89,8 +89,9 @@ md_alloc(void *ctx) } static void -md_free(void *ctx, uint64_t b) +md_free(void *ctx, uint64_t b, uint64_t gen) { + (void)gen; struct memdev *d = ctx; if (b > 0 && b < NBLOCKS) d->allocated[b] = 0; // reusable only once committed