SCFF: the live root ships compressed, with Xorg and Window Maker

SCFF, the SchOS Compressed File Format (include/fs/scff.h), is a block
image cut into 128 KB chunks, each its own zstd frame, behind an index.
tools/mkscff writes it from fs-ram.img on every core (-19, 11 s) and
checks each chunk decompresses to itself. The kernel carries the .scff
as its ramdisk and unpacks a chunk the first time a block in it is
read; zero blocks share one page. The kernel and the ISO are the
compressed size, and the root costs memory only for what is touched.
See docs/architecture/scff.md.

zstd 1.5.7 comes from ports/zstd like uACPI: the decoder is compiled
into the kernel at -O2, with a static context and a 64 KB stack of its
own, and mkscff builds the whole library from the same release.

world now has xorg-server, the fbdev and evdev drivers, xinit, Window
Maker, xterm and DejaVu, and /root/.xinitrc runs wmaker. That root is
358 MB, 102 MB as SCFF.

The ISO no longer carries the root a third time as an ISO 9660 tree,
which nothing read. schos.iso: 1389 MB -> 318 MB. El Torito keeps its
own copy of the EFI image, the layout real machines have booted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S1giNvrHTrPMPw7V6x1kQ4

docs/architecture/scff.md

@@ -0,0 +1,79 @@
+# SCFF: the SchOS Compressed File Format
+
+The live root is shipped compressed. `make iso` builds the root as a SchFS
+image, `tools/mkscff` writes that image as SCFF, and the kernel carries the
+`.scff` file as its ramdisk. Blocks are decompressed the first time something
+reads them.
+
+| | raw image | SCFF |
+|---|---|---|
+| root with Xorg and Window Maker | 358 MB | 102 MB |
+| kernel and ISO | carry all 358 MB | carry 102 MB |
+| memory the root takes at boot | 358 MB | what has been read |
+
+## Layout
+
+The definition is `include/fs/scff.h`, which the kernel and `mkscff` both
+use.
+
+    offset 0          struct scff_header   (64 bytes, "SCFF", version 1)
+    index_off         uint64 off[nchunks + 1]
+    off[i]..off[i+1]  chunk i
+
+The image is cut into chunks of `chunk_bytes`, 128 KB by default. Each chunk
+is stored in one of three ways, and its stored length says which:
+
+- **Zero bytes:** every byte of the chunk is zero. The slack at the end of the
+  image costs nothing.
+- **The chunk's full length:** stored as it is, because it did not compress.
+- **Anything else:** one zstd frame, with a content checksum.
+
+## Why it is shaped like this
+
+**Chunks, not one stream.** Each chunk can be decompressed without the others,
+so reading one file does not mean decompressing the root. This costs a little
+ratio: the 358 MB root is 104 MB in 128 KB chunks and 97 MB in 1 MB chunks.
+128 KB is squashfs's default too.
+
+**A block image, not a compressed filesystem.** squashfs compresses files and
+is read-only, so a writable root needs an overlay filesystem above it. SCFF
+compresses a SchFS image instead, and the ramdisk is still a block device that
+takes writes. A block that is written gets a page of its own, exactly as it
+did before SCFF. SchFS does not know the image was ever compressed.
+
+**zstd.** It compresses close to xz at the levels a build can afford (`-19`).
+It decompresses at memory speed whatever level made the frame. Its decoder is
+dual BSD and GPL-2.0, and it is a plain C library. `ports/zstd` pins the
+release. The kernel compiles only the decoder half. `mkscff` compiles all of
+it from the same release, so neither side depends on the host's libzstd.
+
+## In the kernel
+
+`kernel/dev/ramdisk.c` reads the header at boot, checks it and the index, and
+panics if either is wrong. A damaged image is found before anything mounts.
+
+The ramdisk's page directory covers every block. A block with no page has not
+been unpacked yet. Reading one decompresses its whole chunk into `rd_chunk`,
+and every block in that chunk without a page gets one. A block of zeros points
+at one shared zero page and gets a real page only when it is written.
+
+`kernel/scff/scff.c` holds the decoder: one static zstd context in a 256 KB
+workspace, so nothing allocates. **Decompression runs on a 64 KB stack of its
+own.** A kernel stack is 8 KB, and a chunk is decompressed at the bottom of a
+read that has already used most of it. zstd's sequence decoder needs several
+KB of locals, and Linux gives it a 16 KB stack. The chunk lock (`rd_zlock`)
+makes that stack its holder's alone. `kernel/scff/compat/` stands in for the
+few libc headers zstd includes.
+
+zstd is built at `-O2` whatever `OPT` says, because it runs on every first
+read of a block.
+
+## Building
+
+    make iso                 builds build/fs-ram.scff and links it in
+    make iso SCFF_LEVEL=3    much faster to build, about a third bigger
+    make iso SCFF_CHUNK=256  bigger chunks: a little smaller, more work per miss
+
+`mkscff` uses every core. At `-19` the root takes about 11 seconds on 16
+threads. It decompresses every chunk again before writing it, so an image
+that would not boot fails the build instead.

docs/plan/alpine.md

@@ -74,6 +74,7 @@ All of it found by running apk:
   Xorg installs into it in about a minute. All of it is gone at reboot. Keeping packages needs an installer that writes the root
   to a SATA or NVMe disk, and a kernel that finds its root there by the
   SchFS GPT type rather than as IDE disk 1.
-- Xorg from apk (`xorg-server xf86-video-fbdev xf86-input-evdev`) instead of
-  the hand port in [xorg.md](xorg.md). The packages install; running them is
-  the next test.
+- Xorg and Window Maker are in `world`, so the image starts with them:
+  `startx` runs `wmaker` from `/root/.xinitrc`. Mesa brings LLVM with it,
+  about 200 MB of the 358 MB root; the root ships compressed
+  ([SCFF](../architecture/scff.md)), so that costs about 60 MB of ISO.