Sixty-five headers sat side by side in include/, so `#include "irq.h"`
said nothing about where irq.h was or what it belonged to, and errno.h,
stat.h and fcntl.h at the top shadowed the system's for any host tool
built with -I./include. They are in folders now -- arch/ core/ fs/
dev/ gfx/ net/ uapi/ (and ldls/ for the Linux driver layer) -- and
every include names the folder, as Linux's do: "core/proc.h".
Done mechanically and checked the hard way: the kernel built from this
tree disassembles identically to the one built from the tree before
it. The full image boots, and userland is the same set of programs.
Paths in docs and comments follow; docs/guides/layout.md has a table
of what is where, and tools/fix-includes.py rewrites old-style
includes from the folders in the tree, for code written before this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1giNvrHTrPMPw7V6x1kQ4
Transaction groups commit from a schfsd thread every 5 seconds, on sync
and fsync, on unmount and shutdown, and early when freed-but-pending
blocks crowd the free ones -- the contract ext4 and ZFS have. A crash
still lands on the last group whole, and loses at most those seconds.
Committing before every write(2) returned cost two cache flushes and an
uberblock a call: 4 KB writes go from 2.8 to 6.6 MB/s on AHCI, and from
66 KB/s to 3.9 MB/s on QEMU's IDE disk.
Filling the disk made the root volume unmountable, on the last commit
as well: a tree update that ran out of blocks halfway was published by
the next commit. 1% of the volume (1 to 64 MiB) is now a metadata
reserve, as btrfs and ZFS keep one; data stops at it with ENOSPC as a
short write, new names at half of it, and deletes may use the rest. A
write that runs out part way records what it wrote in the inode rather
than leaving blocks past an end of file nothing moved.
Deleting a large file goes in 4 MiB steps from the end with a commit
allowed between them, as ext4 restarts its transaction through a long
truncate. Deleting 80 MB on a full disk ran out of room halfway and
leaked the rest.
The VFS's dirunlink called the object layer's whole unlink, which freed
the object with its last name, so a file removed while open lost its
data under the reader. It removes the name only now, and the object is
freed on the last iput. An object still open when the machine went down
is found by the mount's space scan and freed, like ext4's orphan list.
The allocation bitmap has a second level of pointer pages, so the mount
ceiling is 64 TiB instead of 64 GiB, and the freed-this-group bitmap
has pages only where frees happened: a 1 TiB volume takes 32 MiB.
A freed block written after the newest snapshot is reused at once
(ZFS's birth-time rule, so bfree carries the block's generation); only
older ones are kept while a snapshot exists. Deleting one snapshot no
longer stops retaining for the others, which let their blocks be
handed out again.
Checked under QEMU: crash after sync, after 8 s and within 5 s; fill to
ENOSPC, rm, remount; unlink while open; an orphan freed at the next
mount; a 1 TiB volume. schfs-check passes, with a new test for the
snapshot rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1giNvrHTrPMPw7V6x1kQ4
Nexoniarz is the sole author and SchOS was never published under
Apache-2.0, so this is a straight relicence. LICENSE is the FSF GPLv2 text,
every first-party SPDX header moves to GPL-2.0-or-later, and NOTICE,
README.md, docs/licensing.md and the site say so. projects/schfs-linux
follows the rest of the tree.
The GPL gives copyleft: a fork has to ship its source. It also lifts the
old "Linux drivers, never" rule, since GPL-2.0-only code combines with
GPL-2.0-or-later. "Or later" leaves room for GPLv3, but one binary still
has to pick one version, so docs/licensing.md now keeps Apache-2.0 and
GPLv3 code out of the kernel to keep Linux code possible there. BSD
4-clause code goes from "avoid" to "no", because the advertising clause
is GPL-incompatible.
LDLS still ships no .ko. That boundary is now project policy rather than
a legal requirement, and ldls.md, devices.md and licensing.md section 6
say so. refmod declares license=GPL, and ldls-check passes (27 checks).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last three phases of the filesystem, which were written together and
touch the same files throughout -- kmount.c, objw.c and vfs.c are each
rewritten by all three -- so they land as one commit rather than three
that could not be built separately.
Durability. A transaction that reaches the platter out of order is worse
than one that never reached it: the uberblock can name a tree whose
blocks are still in the drive's write cache. bdirty/bflush give the
buffer layer an explicit barrier, ideflush issues ATA FLUSH CACHE and
usb_storage_flush the SCSI SYNCHRONIZE CACHE that means the same thing,
and a commit now waits for the data before it writes the root that
points at it. The uberblock groups are read newest-first with fallback,
so a volume whose newest roots are torn mounts at an older one and says
which generations were lost.
Reflink. Extents carry a refcount, so two objects can name the same
blocks and neither frees them while the other holds them. Exposed as
FICLONE -- Linux's ioctl value, so `cp --reflink` works unmodified --
and as cmd/reflink for the case where there is no cp to hand.
Attributes, and the one genuinely new idea here. attr.c stores named
values on an object, reachable through setxattr/getxattr/removexattr at
Linux's syscall numbers. On top of that, schfs_index declares an index
over an attribute name and schfs_query answers "which objects carry this
value" from the B-tree rather than by walking. A query against an
unindexed attribute is refused with an error, never silently answered by
a full scan: a query that quietly becomes a scan is worse than one that
admits it cannot help. cmd/schattr drives both.
Three bugs fixed on the way, each found by the numbers being wrong
rather than by inspection:
- statfs's f_fsid was declared int64[2]. Linux's fsid_t is two *ints*,
so every field after it landed one slot late: userland read f_frsize
out of f_namelen, got 255, and df reported a 32 MB root filesystem
as 1992 KB. It hid for as long as it did because the fields used to
be invented constants, so wrong numbers looked like the usual ones.
- struct inode's nlink was a short against SchFS's uint32, so a file
past 32767 names came back negative -- and iput frees an object
whose count reaches zero. It is an int now, signed on purpose, so an
underflow lands somewhere a check can see rather than wrapping.
- filewrite split every write to fit v1's log. That limit belongs to
the journal, not to writing, and SchFS makes one transaction of
whatever it is handed: a 1 MB write was becoming 170 commits and
170 flushes for nothing. fs_ops carries max_write now, and SchFS
sets no limit.
Also here: a real mount table, so `mount -t schfs /dev/hdb /mnt` shows
up in /proc/mounts -- df looks a path up there before it calls statfs at
all, so a mount missing from that list was invisible to it -- several
volumes mounted at once, snapshots mounted by name, and /proc/fsstat
with an rdtsc-based counter, which is the only instrument fine enough to
see inside a single syscall.
Verified: `make schfs-check` passes in full, including the sweep that
crashes the volume at each of 119 points mid-transaction and requires
every one to mount with exactly its committed data, and the case where
both newest uberblocks are rubbish and the volume still comes back. The
headless boot check passes with 27 [ok] and no failures, on a SchFS root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A snapshot is a tree root that is kept. The B-tree is copy-on-write, so
the blocks a committed root reaches are already immutable -- nothing
overwrites a block a committed tree can reach -- and retaining one costs
a record: (SNAP_OBJECTID, KEY_SNAP, id) -> struct schfs_snap.
The record names the root as it was *before* the record was added.
Inserting it rewrites the path under copy-on-write and produces a new
root, so a snapshot that included its own creation would name a block
nobody had written yet.
What makes it survive is the derivation. There is no free-space map on
disk; it is rebuilt at mount by walking the tree, and that walk now
covers every snapshot root as well as the live one. Without it a block
reachable only from a snapshot looks free, gets handed out, and
overwrites the data the snapshot exists to keep -- silently, surfacing
only when the snapshot is read. A snapshot's own tree may hold older
snapshot records; those are deliberately not followed, or deleting a
snapshot could be undone by an older one's memory of it.
Runtime frees are suspended while a snapshot exists. Knowing which freed
blocks a snapshot still needs requires per-extent reference counting,
which is a design and not a flag; until that exists the safe answer is to
free nothing and let the next mount's derivation reclaim what is
genuinely unreachable. Allocation is unaffected and stays correct -- it
works from the in-use bitmap the derivation filled. The running
free-blocks *counter* does drift from the truth while a snapshot is
present, which is cosmetic and is reported by the test rather than
asserted away.
Tested in schfs_objtest: after a snapshot, the live tree shows new
contents while the snapshot still shows the old ones; a file deleted from
the live tree is still readable through the snapshot; deleting the
snapshot removes the record and the space comes back at the next mount.
schfs-check passes including the round-trip; bootcheck 108 [ok].
Not yet: snapshots are not reachable by name from userspace -- there is
no mount-a-snapshot path, so this is the format and the mechanism, not
the feature a user sees.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every read descended the B-tree twice per block -- once for the inode and
once for the extent -- so an 834 KB file cost 418 descents, each a tree
walk plus a 4 KB copy out of the buffer cache per node visited. A
sequential read asks for the same inode every time and usually stays
inside one extent, so both answers were already known and thrown away.
One entry each, not a table: the pattern being optimised is a single file
read straight through, and one entry catches all of it without needing an
eviction policy. A second reader simply misses.
Correctness rests entirely on invalidation, so it is blunt: all five
write entry points in objw.c call schfs_cache_invalidate() before
touching anything. A cache that is merely usually right would hand back
another file's data. The volume pointer is part of the key because the
host tools can hold two images open at once.
Measured in the guest, bracketed inside one command line, TCG:
read 834 KB 1.73s -> 0.31s (5.6x, together with the BOUNCE fix)
ls /bin cold 2.80s -> 1.40s (2x)
ls /bin warm 0.70s -> 0.70s (unchanged)
Warm ls is untouched on purpose-by-consequence: it is 241 separate
inodes, so a one-entry cache never hits. That cost is the per-entry
lookup path -- dirlookup plus iload, each its own descent -- and it is
where the next order of magnitude is, not here.
schfs-check passes, including the round-trip; bootcheck 108 [ok].
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An x86-64 operating system built from xv6, now largely rewritten: SMP,
a copy-on-write root filesystem (SchFS), PCI/USB/virtio drivers, a TTF
rasteriser, and a Linux-numbered syscall ABI.
No history before this point -- the tree was never under version control.
This commit exists so the rename that follows has something to undo to.
