# SCFF: the SchOS Compressed File Format

The live root is shipped compressed. `make iso` builds the root as a SchFS
image, `tools/mkscff` writes that image as SCFF, and the kernel carries the
`.scff` file as its ramdisk. Blocks are decompressed the first time something
reads them.

| | raw image | SCFF |
|---|---|---|
| root with Xorg and Window Maker | 358 MB | 102 MB |
| kernel and ISO | carry all 358 MB | carry 102 MB |
| memory the root takes at boot | 358 MB | what has been read |

## Layout

The definition is `include/fs/scff.h`, which the kernel and `mkscff` both
use.

    offset 0          struct scff_header   (64 bytes, "SCFF", version 1)
    index_off         uint64 off[nchunks + 1]
    off[i]..off[i+1]  chunk i

The image is cut into chunks of `chunk_bytes`, 128 KB by default. Each chunk
is stored in one of three ways, and its stored length says which:

- **Zero bytes:** every byte of the chunk is zero. The slack at the end of the
  image costs nothing.
- **The chunk's full length:** stored as it is, because it did not compress.
- **Anything else:** one zstd frame, with a content checksum.

## Why it is shaped like this

**Chunks, not one stream.** Each chunk can be decompressed without the others,
so reading one file does not mean decompressing the root. This costs a little
ratio: the 358 MB root is 104 MB in 128 KB chunks and 97 MB in 1 MB chunks.
128 KB is squashfs's default too.

**A block image, not a compressed filesystem.** squashfs compresses files and
is read-only, so a writable root needs an overlay filesystem above it. SCFF
compresses a SchFS image instead, and the ramdisk is still a block device that
takes writes. A block that is written gets a page of its own, exactly as it
did before SCFF. SchFS does not know the image was ever compressed.

**zstd.** It compresses close to xz at the levels a build can afford (`-19`).
It decompresses at memory speed whatever level made the frame. Its decoder is
dual BSD and GPL-2.0, and it is a plain C library. `ports/zstd` pins the
release. The kernel compiles only the decoder half. `mkscff` compiles all of
it from the same release, so neither side depends on the host's libzstd.

## In the kernel

`kernel/dev/ramdisk.c` reads the header at boot, checks it and the index, and
panics if either is wrong. A damaged image is found before anything mounts.

The ramdisk's page directory covers every block. A block with no page has not
been unpacked yet. Reading one decompresses its whole chunk into `rd_chunk`,
and every block in that chunk without a page gets one. A block of zeros points
at one shared zero page and gets a real page only when it is written.

`kernel/scff/scff.c` holds the decoder: one static zstd context in a 256 KB
workspace, so nothing allocates. **Decompression runs on a 64 KB stack of its
own.** A kernel stack is 8 KB, and a chunk is decompressed at the bottom of a
read that has already used most of it. zstd's sequence decoder needs several
KB of locals, and Linux gives it a 16 KB stack. The chunk lock (`rd_zlock`)
makes that stack its holder's alone. `kernel/scff/compat/` stands in for the
few libc headers zstd includes.

zstd is built at `-O2` whatever `OPT` says, because it runs on every first
read of a block.

## Building

    make iso                 builds build/fs-ram.scff and links it in
    make iso SCFF_LEVEL=3    much faster to build, about a third bigger
    make iso SCFF_CHUNK=256  bigger chunks: a little smaller, more work per miss

`mkscff` uses every core. At `-19` the root takes about 11 seconds on 16
threads. It decompresses every chunk again before writing it, so an image
that would not boot fails the build instead.