SchOS/docs/architecture/scff.md

SCFF: the SchOS Compressed File Format

The live root is shipped compressed. make iso builds the root as a SchFS image, tools/mkscff writes that image as SCFF, and the kernel carries the .scff file as its ramdisk. Blocks are decompressed the first time something reads them.

	raw image	SCFF
root with Xorg and Window Maker	358 MB	102 MB
kernel and ISO	carry all 358 MB	carry 102 MB
memory the root takes at boot	358 MB	what has been read

Layout

The definition is include/fs/scff.h, which the kernel and mkscff both use.

offset 0          struct scff_header   (64 bytes, "SCFF", version 1)
index_off         uint64 off[nchunks + 1]
off[i]..off[i+1]  chunk i

The image is cut into chunks of chunk_bytes, 128 KB by default. Each chunk is stored in one of three ways, and its stored length says which:

• Zero bytes: every byte of the chunk is zero. The slack at the end of the image costs nothing.
• The chunk's full length: stored as it is, because it did not compress.
• Anything else: one zstd frame, with a content checksum.

Why it is shaped like this

Chunks, not one stream. Each chunk can be decompressed without the others, so reading one file does not mean decompressing the root. This costs a little ratio: the 358 MB root is 104 MB in 128 KB chunks and 97 MB in 1 MB chunks. 128 KB is squashfs's default too.

A block image, not a compressed filesystem. squashfs compresses files and is read-only, so a writable root needs an overlay filesystem above it. SCFF compresses a SchFS image instead, and the ramdisk is still a block device that takes writes. A block that is written gets a page of its own, exactly as it did before SCFF. SchFS does not know the image was ever compressed.

zstd. It compresses close to xz at the levels a build can afford (-19). It decompresses at memory speed whatever level made the frame. Its decoder is dual BSD and GPL-2.0, and it is a plain C library. ports/zstd pins the release. The kernel compiles only the decoder half. mkscff compiles all of it from the same release, so neither side depends on the host's libzstd.

In the kernel

kernel/dev/ramdisk.c reads the header at boot, checks it and the index, and panics if either is wrong. A damaged image is found before anything mounts.

The ramdisk's page directory covers every block. A block with no page has not been unpacked yet. Reading one decompresses its whole chunk into rd_chunk, and every block in that chunk without a page gets one. A block of zeros points at one shared zero page and gets a real page only when it is written.

kernel/scff/scff.c holds the decoder: one static zstd context in a 256 KB workspace, so nothing allocates. Decompression runs on a 64 KB stack of its own. A kernel stack is 8 KB, and a chunk is decompressed at the bottom of a read that has already used most of it. zstd's sequence decoder needs several KB of locals, and Linux gives it a 16 KB stack. The chunk lock (rd_zlock) makes that stack its holder's alone. kernel/scff/compat/ stands in for the few libc headers zstd includes.

zstd is built at -O2 whatever OPT says, because it runs on every first read of a block.

Building

make iso                 builds build/fs-ram.scff and links it in
make iso SCFF_LEVEL=3    much faster to build, about a third bigger
make iso SCFF_CHUNK=256  bigger chunks: a little smaller, more work per miss

mkscff uses every core. At -19 the root takes about 11 seconds on 16 threads. It decompresses every chunk again before writing it, so an image that would not boot fails the build instead.