nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

docs: README, plan, guides and the site describe what is built now

Browse source 
README: rewritten around what runs (Linux software, Xorg, apk, Linux's drivers), the honest gaps, measured numbers.
plan/: 1.0.md is the finish line, where each clause of the acceptance test stands, and the register (one failing
clause: a root that survives a reboot); built plans (foundations, hybrid, alpine, xorg) and the decided-against one
(wayland) are gone, their content lives in architecture/; roadmap.md is only what is still open.
guides: building/layout/testing/adding-a-driver (Linux code may be used). history: four entries (ls -lR, TLB
shootdowns, MDIO 0x02460246, netlink MSG_TRUNC). Website: cards, stack, download page, docs index.
Code comments that pointed at deleted plan docs point at the architecture docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EVRhxi9zpdW2uDUN4FVcao
This commit is contained in:
Nexoniarz 2026-10-26 08:08:05 +01:00
commit cb3d28e3d6
57 changed files with 685 additions and 2121 deletions
Download patch file Download diff file Expand all files Collapse all files

584
docs/plan/1.0.md
View file

@ -7,17 +7,16 @@ A finish line, so that the list of open work stops being infinite.
## Why this document exists
[schapi.md](schapi.md) §3 says the stability freeze starts at 1.0 and covers
whatever is marked `stable` at that moment. **1.0 was never defined**, which
means there has been no finish line — and with no finish line every gap is
equally mandatory, the list only grows, and the project feels larger every week
regardless of how much gets done.
whatever is marked `stable` at that moment. 1.0 was never defined, which means
there was no finish line — and with no finish line every gap is equally mandatory,
the list only grows, and the project feels larger every week regardless of how much
gets done.
That feeling is structural, not a judgement about progress. This document fixes
the structure by drawing a line and putting most of the open work on the far
side of it.
That feeling is structural, not a judgement about progress. This document fixes the
structure by drawing a line and putting most of the open work on the far side of it.
> **Anything not named in this document is not part of 1.0.** It is not
> cancelled, not forgotten, and not up for reconsideration until 1.0 ships.
> **Anything not named in this document is not part of 1.0.** It is not cancelled,
> not forgotten, and not up for reconsideration until 1.0 ships.
---
@ -26,31 +25,55 @@ side of it.
> **A machine one person can sit down at and use for an afternoon without
> falling off an edge.**
Concretely — boots on real hardware, roots on SchFS, runs a shell and the real
toybox userland, is on the network, and puts a windowed terminal on the screen.
Concretely — boots on real hardware, roots on SchOS's own filesystem, runs a shell
and a real userland, is on the network, and puts a windowed desktop on the screen.
It is deliberately a **small complete system** rather than a rich incomplete
one. SchOS 1.0 is not Linux, does not run third-party applications, and has no
package manager. It is a system that works, end to end, with nothing in it that
lies about what it did.
It started as a deliberately small system: *no third-party applications and no package
manager*. That has changed on its own, and is worth recording rather than tidying away:
the root is an **apk root** ([packages](../architecture/packages.md)), Alpine's
programs run on it unmodified, and the desktop is Xorg with Window Maker
([desktop](../architecture/desktop.md)). What did not change is the bar: nothing in the
system lies about what it did.
### The acceptance test
One unattended run, asserted from the host, the way everything else here is
tested. When this passes, it is 1.0:
One unattended run, asserted from the host, the way everything else here is tested.
When this passes, it is 1.0:
```
boot on real hardware from a USB stick
→ SchFS root mounts, generation counter advances
→ schinit starts the compositor
→ a terminal window appears and is asserted by pixel
→ in it: ls, cat, a file written, a network fetch
→ schinit starts shells on the virtual terminals
→ X starts; a window appears and is asserted by pixel
→ in it: ls, cat, a file written, a network fetch over HTTPS
→ reboot
→ the file is still there and the checksum matches
```
Nothing in that sentence is a feature list. Every clause is something that is
currently either broken, missing, or unproven.
Where each clause stands:
| Clause | State |
|---|---|
| boot from a USB stick on real hardware | **done** on an AMD B550 desktop, an ASUS TUF A15 laptop and an Ivy Bridge HP all-in-one |
| SchFS root, generation counter | **done** — the root is a SchFS image, in RAM when live |
| a terminal to type in | **done** — a shell on the first six of the twelve virtual terminals (`etc/schinit/*-shell.job`); the rest are there to be started |
| X starts, a window appears | **done** (`startx` → Window Maker + xterm); *asserted by pixel* has no automated test yet — `bootcheck` drives the serial console, and the screenshots are looked at |
| `ls`, `cat`, a file written | **done** |
| a network fetch over HTTPS | **done** (`wget`, `curl`, `apk`) |
| reboot, **the file is still there** | **not met on the live image.** The root lives in RAM and everything written is gone at reboot; SchFS is crash-safe on a disk, but nothing writes a disk root. This is the one clause that fails, and it is the open decision below |
### The one open decision
The acceptance test's last clause needs a root that survives a reboot. There are two honest ways to
close it, and which one is the owner's to choose:
1. **Write the installer** (`projects/schinst/`, [roadmap.md](roadmap.md) §5): partition a disk,
write the root, install the loader. Large, and until now frozen.
2. **Narrow the clause**: 1.0 is a *live* system, and the last line becomes "a file written to a
SchFS disk or partition that was attached survives a reboot" — which already works
(`make disk`, `make qemu` with `schfs.img`). Smaller, and it says what the system is.
Nothing else is on the path to 1.0.
---
@ -59,11 +82,10 @@ currently either broken, missing, or unproven.
**No new capability is started until this register is empty.**
The failure this exists to stop is not laziness and not scope creep — it is
*abandonment by promotion*. A thing gets built to the point where it works, the
next thing becomes more interesting, and the first one is left in a corner:
present in the tree, not finished, not deleted, and not visible in any list.
Enough of those and the tree stops matching its own documentation, which is the
state this register was written to end.
*abandonment by promotion*. A thing gets built to the point where it works, the next
thing becomes more interesting, and the first one is left in a corner: present in the tree,
not finished, not deleted, and not visible in any list. Enough of those and the tree stops
matching its own documentation.
Two rules, and they are the same rule seen from either end:
@ -71,73 +93,47 @@ Two rules, and they are the same rule seen from either end:
>
> **Adding a row requires closing one.** Two things in flight. Never three.
A row leaves this table in one of exactly two ways. Either it is **finished** —
and finished means a description exists in [../architecture/](../architecture/),
not that the code works — or it is **deleted**, and deleted means gone from the
tree, not marked for deletion. A directory that has said "to be deleted" for
months is the corner, not the cure; `projects/schwm/` and `projects/schde/`
proved that and are now actually gone.
A row leaves this table in one of exactly two ways. Either it is **finished** — and finished
means a description exists in [../architecture/](../architecture/), not that the code works —
or it is **deleted**, and deleted means gone from the tree, not marked for deletion. A
directory that has said "to be deleted" for months is the corner, not the cure.
### Built, and not connected to anything
The galling ones. The work exists, is tested, and nothing uses it.
| | Where it stopped |
|---|---|
| ~~**The text layer**~~ | **Half closed, and the half that closed is the one that mattered.** The console draws real TrueType through `font_ops` now: cells hold a codepoint, the cell size comes from the font's measured advance, and `kernel/dev/ttf.c` has a caller that is not a self-test. What is still unused is `text.c`'s *proportional* path — the console remains a fixed grid. See Gate 0. |
| **A font from disk** | Still true, and now deliberate rather than pending: the console typeface is linked into the kernel image, because a panic during boot has to be readable before there is a filesystem. Loading a *second* font from disk is the open part. |
| **Socket activation** | `schinitd` parses `[Socket name]` and remembers the path. Nothing holds the socket. **This is the entire reason for copying launchd**, and it is the half that was not built. |
| **`display_ops.wait_vblank`** | In the vtable, implemented by the VBE driver, left null by virtio-gpu. A compositor that needs it cannot rely on it. |
| **`projects/schcom/`** | A compositor v0 (about 700 lines) over a wire format that nothing speaks any more: the desktop is Xorg. It is the one thing in the tree that has ever put a client's pixels on screen by itself, and it has no consumer. **Delete it**, or say what it is for. |
| **The in-kernel LDLS window** | `vm.c` maps the kernel image a second time in the top 2 GB at every boot, and `include/ldls/ldls.h` describes a `/dev/ldls` loader — the start of loading Linux modules *into the kernel*, which [ldls.md](../architecture/ldls.md) rule 2 says is not how it is done. Nothing calls either. **Delete it**, or decide. |
| **`display_ops`** | `acquire` and `wait_vblank` are in the vtable, implemented by one driver at most, with no consumer: Xorg draws through `/dev/fb0`. Kept for the GPU work ([gpu.md](gpu.md)); that is the reason it stays. |
| **Socket activation** | `schinitd` parses `[Socket name]` and remembers the path. Nothing holds the socket. This is the entire reason for copying launchd, and it is the half that was not built. |
| **`sys_delete`, `sys_show`, `sys_showstat`** | Orphaned entries from the old v1 filesystem's recycle bin. They die with `fs.c`. |
### Started, and stopped part-way
| | Where it stopped |
|---|---|
| **`foundations.md`** | All but item 4 (`setsockopt`) built: descriptor passing, `memfd`, filesystem-named `AF_UNIX`, dynamic linking, and epoll/eventfd/timerfd/signalfd. |
| ~~**`hybrid.md`**~~ | **Closed.** All four mechanisms built; H1, H2 and H3 done. H4 — the USB eviction — is *withdrawn* rather than outstanding: USB is the bus peripherals arrive on, not a peripheral, and an eviction that has to leave an identical driver behind has moved nothing. See [hybrid.md](hybrid.md), "Decided: USB stays in the kernel". |
| ~~**`ldls.md`**~~ | **Closed.** L0 through L4 built: the loader handles 2,440 stock modules, and both `8139too` and Linux's `e1000` run in a SchOS process through `/dev/pci` with `ping` working over each. The e1000 run is a handover — the kernel was driving that NIC and gave it up. Described in [../architecture/ldls.md](../architecture/ldls.md). Two drivers, one kernel version, one machine. |
| **Schcom v0** | 715 lines that composite, over a wire format that is superseded. Frozen: it runs, nothing more is added, and it is kept only because it is the one thing in the tree that has ever put a client's pixels on screen. |
| ~~**`debugger.md`**~~ | **Closed.** D0 through D5 built, including the network transport: `schdbgd` bridges TCP to `/dev/dbgro`, ships disabled, binds loopback, and cannot reach a view that changes state. Described in [../architecture/debugging.md](../architecture/debugging.md). |
**The row LDLS spent, and gave back.** `hybrid.md` closed — H3 built, H4
withdrawn — and `ldls.md` took the slot it freed. It has since closed too, so
the slot is free again. Worth being honest that this is
the largest thing ever to enter this register, and that it was **explicitly not
part of 1.0**: nothing in the acceptance test needs a foreign driver. It was
taken on because the mechanisms it uses were finished and idle, and because
the alternative to it is writing a driver for every device that exists.
**The row the debugger spent, and gave back.** Adding one requires closing one,
and *A released PCI device* closed when H2 landed. It is worth saying plainly
that the debugger is not a capability of the OS the way every other row here is:
nothing in it ships to a user or appears in the acceptance test. The case for it
is that **Gate 0 is a list of bugs to find in a kernel that could not be asked
anything** — so it is the instrument the rest of this register is surveyed with,
and it earned its slot by making every other row cheaper to close.
It has closed. **Every plan in `docs/plan/` that was open is now closed**, which
means the next thing is Gate 0 itself: the survey, with the instrument built.
**Closed since this register was written.** *A released PCI device* — release is
now symmetric with claim (bus mastering cleared before the DMA is freed, a
refusal when the device is the only display), a supervised job starts the
replacement, and it is described in
[../architecture/devices.md](../architecture/devices.md) under "Taking it back".
| **LDLS on real hardware** | `r8169` registers its interface on the laptop and reads its PHY; whether it carries traffic there is **not confirmed**. The native driver is out of the registry and stays as the fallback. |
| **The laptop's embedded controller** | The EC is found and served (battery and AC read through it); **SCI queries are not handled**, so events from it are not seen. |
| **Epiphany (WebKit)** | `apk add epiphany` installs it; its web-process child dies at start-up ("short read from the peer"). NetSurf runs. Not diagnosed. |
### Named in a "what it does not do" and still true
Not corners — these were written down honestly on the day and stayed written
down. They are here so the register is the whole picture rather than the
flattering part of it.
Not corners — these were written down honestly on the day and stayed written down. They are here
so the register is the whole picture rather than the flattering part of it. Each is described where
it lives.
| | |
|---|---|
| `/proc/<pid>/exe` is a regular file | so `readlink()` fails and `ps -o comm` prints `[init]` |
| `getrusage` | does not distinguish `RUSAGE_CHILDREN` from `RUSAGE_SELF` |
| No slab allocator | every subsystem's fixed-size table is a consequence |
| No `getsockname`/`getpeername` | and `accept()` does not report an AF_INET peer |
| No routing table | first interface wins, which is a rule and not routing |
| No page cache | writable `MAP_SHARED` of a file is refused — [kernel.md](../architecture/kernel.md) |
| `mprotect` is a no-op, nothing has NX | [kernel.md](../architecture/kernel.md) |
| No routing table | first interface wins, which is a rule and not routing — [networking.md](../architecture/networking.md) |
| No descriptor garbage collector | a socket passed over itself and never received leaks the pair |
| No udev / uevent socket | Xorg is told its input devices — [desktop.md](../architecture/desktop.md) |
| No EHCI driver | USB 2.0 ports wired only to EHCI do nothing — [usb.md](../architecture/usb.md) |
| Touchpads are mice | no two-finger scroll — [desktop.md](../architecture/desktop.md) |
| `ru_stime` is zero | [kernel.md](../architecture/kernel.md) |
| No IOMMU | a driver in a process is isolated against bugs, not malice — [devices.md](../architecture/devices.md) |
| One audio chip family | AC'97 only; laptops have HDA — [audio.md](../architecture/audio.md) |
### Frozen — scoped, unstarted, and not to be started
@ -145,433 +141,77 @@ Neither cancelled nor in progress. **Do not open these.**
| | |
|---|---|
| `projects/schinst/` | the installer, the installation environment, `.sip` |
| `projects/schinst/` | the installer, the installation environment, `.sip` (see the open decision above) |
| `projects/schfs-linux/` | SchFS as a filesystem for Linux |
| [roadmap.md](roadmap.md) §4 | the interactive crash screen |
| [roadmap.md](roadmap.md) §6 | SchFS beyond SchOS |
| [gpu.md](gpu.md) | amdgpu and nouveau through LDLS — measured, with the work it needs written down |
| [finch.md](finch.md) | Finch inside SchOS (the language itself is finished and lives in its own repository) |
| [schapi.md](schapi.md) | the API layering, designed and not built |
Each of those is well argued and none of them is on the path to the acceptance
test above. That is the whole test for whether something belongs in this section:
**not "is it worth doing" but "does 1.0 fail without it".**
Each of those is well argued and none of them is on the path to the acceptance test above. That is
the whole test for whether something belongs in this section: **not "is it worth doing" but "does 1.0
fail without it".**
---
## The four gates
## The gates
In order. **Do not start a gate before the one before it closes**, and closing
means a description exists in [../architecture/](../architecture/) — see
[the rule](../README.md#the-rule-that-keeps-this-from-rotting).
In order. **Do not start a gate before the one before it closes**, and closing means a description exists in
[../architecture/](../architecture/).
### Gate 0 — Fix what is already wrong
**This comes first, before any new capability.** The desktop work is the
temptation and it is the wrong thing to do next: a compositor built on a kernel
with an unexplained hang and a socket layer that reports success for things it
did not do will fail in ways indistinguishable from its own bugs. That is the
same argument that put `netdev_selftest()` before the TCP/IP stack, and it was
right then.
**Things that are actively wrong:**
- [x] ~~**The `ls -lR` hang.**~~ **Not reproducible.** Instrumented and hunted;
see below. Left here rather than deleted because the absence of a result
is the result.
- [x] ~~**`setsockopt` returns 0 without setting anything.**~~ Six options now
genuinely reach lwIP's pcb and everything else is `ENOPROTOOPT`. Round-trip
checks in `cmd/socktest`, which a permissive stub cannot pass: it has to
read back a value it was set *and* refuse an unknown option.
- [x] ~~**`accept4` silently drops its flags.**~~ It is its own table entry now
rather than an alias for `accept` — an alias would have had to read a
fourth argument that `accept` never passes, inventing a `SOCK_NONBLOCK`
out of whatever the caller last left in that register.
- [x] ~~**Timestamps on everything created after boot read 1970.**~~ Both
`know()` (SchFS) and `now_sec()` (v1) returned `uptime()` under comments
saying the kernel had no wall clock. It has had one since `time_init()`
started reading the RTC; the comments outlived the condition. Two errors,
not one — `uptime()` counts 10 ms ticks and they were used as seconds, so
the clocks also ran 100× fast.
- [x] ~~**User processes are corrupted on SMP.**~~ **Found and fixed.**
`syscall_entry` parked `%rax`, `%rcx` and `%rdx` in three *globals*
across the `FS_BASE` swap, under a comment arguing that `SYSCALL`
clearing `IF` made it safe. `IF` answers for an interrupt on *this* CPU
and says nothing about a second CPU running the same three stores — and
`%rcx` is the user return address `SYSRET` uses, so the CPU that lost the
race returned its process into another process's instruction stream.
That is the whole of it: `rip 0x0` was a process handed a zero and
following it, and `0xffffffffffffffea` was `-EINVAL` in a swapped `%rax`
used as a pointer. Per-CPU words at `%gs:32..48` now. Measured at four
CPUs against the same pristine image: **10 faults in 10 boots before, 0
in 18 after.** See [../architecture/kernel.md](../architecture/kernel.md),
"Per-CPU state".
- [x] ~~**`$(command substitution)` hangs the shell.**~~ **Not a kernel bug.**
`ports/zsh/Makefile` answered configure's "does `kill(pid, 0)` return
ESRCH correctly?" with `no`, which makes zsh `#define ESRCH EINVAL`.
`waitforpid()` ends its wait on `errno != ESRCH`, so with the kernel
correctly returning ESRCH (3) and zsh comparing against EINVAL (22), the
shell waited for ever on a child it had already reaped. One word in a
cross-compile cache. Autoconf's own default for that check is `yes`;
omitting the line would have been right. See
[../history/kernel.md](../history/kernel.md), section 7.
- [x] ~~**`getrusage` does not distinguish `RUSAGE_CHILDREN`** from
`RUSAGE_SELF`.~~ It did not distinguish anything: it ignored `who` and
reported the machine's *uptime* as the caller's user time, so every
process of every age answered with the same number. Per-process CPU time
is sampled at the timer tick now, children's time is rolled up on reap,
and `wait4()`'s fourth argument — a `struct rusage *` that was read off
the stack and thrown away — is filled. That last one is why zsh's `time`
reported `16777216.01s user` for everything: 2^24 seconds is
uninitialised memory, and it never varied because it was never written.
`time` now says 97% cpu for a busy loop and 0% for `sleep 2`.
`ru_stime` is a known zero; see
[../architecture/kernel.md](../architecture/kernel.md).
- [x] ~~**`/proc/<pid>/exe` is a regular file, not a symlink.**~~
`lrwxrwxrwx ... exe -> /bin/schinit`, and `readlink` returns it. It also
needed a real `st_size`, which every other procfs file deliberately
reports as zero: `readlink(2)` bounds its copy by it, so a symlink of
size 0 resolves to the empty string — which looks like it worked and is
worse than the regular file was.
- [x] ~~**OVMF memory reporting.**~~ The region-list fix did close it. 1 GB
reports 1017 MB usable, 2 GB reports 2041 MB, which is the right shape:
the firmware keeps about 7 MB either way.
- [x] ~~**`ps` printed a header and nothing else.**~~ Found while checking the
`exe` fix. `ps` with no arguments shows only processes on *this*
terminal, and `/proc/<pid>/stat` hardcoded `tty_nr` to 0, so nothing
could ever match — `ps -e` always worked, which is what hid it. Fields
14-17 (utime/stime/cutime/cstime) were hardcoded zero too, so `TIME` was
`00:00:00` for everything however long it had run.
**Work already done and not connected:**
These are the galling ones — the replacement exists, is tested, and is unused.
- [x] ~~**Point the console at a real typeface.**~~ It draws JetBrains Mono
through `font_ops` and `ttf.c`. The blocker was not the font: `vtcell.ch`
was **one byte** holding a CP437 index, so `Ą` (U+0104) could not be
*stored*, whatever was drawing it. The cell is a `uint32` codepoint now
and the cell size comes from the font's measured advance.
- [x] ~~**Delete the cell grid.**~~ Done as far as it should be done, and the
rest of it retired — which needs explaining, because the item as written
asked for something a terminal cannot be.
`include/gfx/text.h` made the case against the grid in four charges: it
cannot do variable-width glyphs, cannot position text off a multiple of
eight, cannot anti-alias, and cannot hold a codepoint above 255. **All
four were about the drawing layer, and all four are now false** — glyphs
carry metrics, PSOGM positions text at arbitrary pixels, the TrueType
path blends 8-bit coverage, and `vtcell.ch` is a `uint32`.
What is left is a grid in `vt.c`, and that one is not in the way: it is
what a terminal *is*. VT100 escapes address columns, `TIOCGWINSZ`
reports rows and columns, and every program that draws a box assumes
both. Deleting it would not modernise the console, it would stop it
being a terminal.
So the real remainder was the duplicate: `gfx.c` walked `font_data`
itself and blitted glyphs its own way, while `text.c` did the same job
into a `struct surface` for nobody but a self-test. Two implementations
that had to agree about a cell size only one of them was told had
changed — and they did not agree, which is why every line of PSOGM once
drew the next character's bitmap underneath itself. There is one blitter
now: `gfx.c` hands `text_draw_glyph` the cell *as a surface of its own*,
so clipping a tall glyph to its line is the same bounds check that clips
to a screen. The bitmap is reached through `font_builtin()` like any
other provider, and `gfx.c` no longer includes `font.h`.
- [x] ~~**Load a font from disk**~~ and point the console at it. The built-in
one is linked in on purpose — an early panic has to be readable — so this
is about a *replaceable* font, not about having one. Drop a TrueType file
at `\schos\console.ttf` on the ESP and the console is drawn with it.
**The loader reads it, not the kernel**, and that is the whole design
decision. The kernel could open the file once the root filesystem is
mounted — but the console has been drawing since long before that, so
switching then means re-measuring the cell and re-laying out every
terminal mid-boot, which wipes the kernel log off the screen at exactly
the moment a machine being brought up for the first time needs it. The
loader allocates the pages as `EfiReservedMemoryType`, so the E820
handoff marks them reserved, the page allocator never offers them, and
the kernel uses the file *in place* — no copy, and no need for a
contiguous allocator the kernel does not have.
- [x] ~~**Interrupt-driven receive.**~~ The e1000's handler calls
`netdev_rx_signal()`, which ends the network thread's wait early; the frame
is still *pulled* by the thread, because an interrupt handler can neither
sleep on the buffer cache nor take the lwIP lock. The wait stays on the
tick so lwIP's timers keep their floor. Proven rather than assumed:
`/proc/netif` gained `wake_rx`/`wake_tick`, and across three pings
`rx_frames` and `wake_rx` both went 2 -> 6.
**Structural, because everything after this compounds on it:**
- [x] ~~**A slab or pool allocator.**~~ `kernel/core/kmem.c`. Slabs are pages
from `kalloc()` with a header at the front; free objects hold the free
list in their own first eight bytes, so an object needs no header and
freeing needs no search — the object's page base *is* its slab, and the
slab names its cache. Two interfaces: named `kmem_cache` for a subsystem
with one kind of thing, and `kmalloc`/`kmfree` over power-of-two classes
for everything else. Visible in `/proc/slab`, asserted by
`make kmem-test` on the host — the same arrangement as `schfs-objtest`,
so the bookkeeping is checked in a second rather than in a boot.
**It does not do what this row said it would, and that is worth
recording.** The claim was that every fixed table — `NPROC`, `NSOCK`,
`NDEVICE`, `NFILE`, `NINODE`, `NBUF` — exists to work around `kalloc`.
Measured: `struct proc` is 10656 bytes, `struct socket` 4928, `struct
buf` 4272. **All three are larger than a page**, so a page-backed slab
cannot hold them whatever it does; `NINODE` is a cache and is supposed to
be bounded; and `NFILE`/`NDEVICE` are 48 and 120 bytes, where the table
costs 24 KB and 3.8 KB and the ceiling has never been reached. The tables
were not all working around `kalloc`. Freeing the big ones needs a
contiguous multi-page allocator, which is a different thing and is not
written.
What it does fix is real and was measured rather than assumed. A pipe is
640 bytes and came from a whole 4096-byte page, so every `|` in a shell
command burned 3.5 KB it never touched — **six pipes per page now**. And
`procread()` built its answer in a 2048-byte array on the kernel stack,
under a comment that had warned about exactly that since it was written;
its frame is 64 bytes now instead of ~2100.
- [x] ~~**SchFS on-disk format version policy.**~~ Done. `SCHFS_VERSION` for
frame changes, plus `compat`/`ro_compat`/`incompat` feature flags in what
was `reserved0` — so existing images keep every field offset and read
unchanged. `uber_check()` now distinguishes *damaged* (step over it; that
is what the ring is for) from *too new* (stop), because falling back to an
older uberblock mounts a stale filesystem and reports success. Proven on a
forged image: the old code mounted the stale root and exited 0.
Still owed: an actual upgrade step, which is only writable once there are
two formats. See [../architecture/schfs.md](../architecture/schfs.md).
**Found while doing the above, and fixed:**
- [x] ~~Every `AF_INET` socket failed with `ENFILE` until DHCP bound.~~
`inet_new()` required an *address* before it would hand out a socket, and
reported the refusal as "file table overflow". Creating a socket is not
the same as having somewhere to send; a bind or connect with no route
fails on its own with an error that says so. `net_is_ready()` (lwIP up) is
now distinct from `net_is_up()` (DHCP bound).
- [x] ~~`socktest` asserted "AF_INET is refused, not faked".~~ True when written,
false since AF_INET landed — and **green the whole time**, because the
suite runs before DHCP binds and the `ENFILE` above made the refusal look
real. A test asserting a failure has to say *which* failure.
**The suite itself:**
Both of these were found by running `make schfs-check` rather than by reading
it, and both predate the current work — confirmed by stashing and re-running.
They are here rather than filed as nice-to-haves because a suite that is red for
known reasons stops being read, and then it is red for unknown reasons too.
- [x] ~~**`make schfs-check` fails at baseline.**~~ Fixed, and it was not only a
test bug. The allocation map was taken through a `palloc` callback and
there was **no way to give it back** — no `schfs_space_release`, and a
comment in `schfs_space_init` reading *"caller frees what was taken"* for
an API the caller did not have.
So the kernel leaked the whole map on **every unmount**, up to 2 MB a
time, and `schfs_space_init`'s own error path stranded a page it had
already taken because `npages` was only raised after *both* allocations
succeeded. The test's 518 leaked pages were the symptom that pointed at
it.
The map now carries its own `pfree` so it can only be freed by the
allocator that filled it, `schfs_vol_open`/`schfs_vol_format` release any
map the volume already holds, and unmount releases. `schfs_objtest`
asserts the balance — 520 pages taken, 520 given back — so the next leak
fails the suite instead of being absorbed by it.
- [x] ~~**31 UndefinedBehaviorSanitizer errors.**~~ Fixed, plus two the test
never reached. One root cause: a leaf's value area holds variable-length
records packed end to end, so a record begins wherever the last one ended
— a multiple of 8 only by luck — and the code read them by casting a
pointer into the block, which tells the compiler they are aligned when
they are not.
The seven record types that live at arbitrary offsets now carry
`SCHFS_PACKED`. **No on-disk layout changed**, and that was measured
before it was applied rather than assumed: every one already had explicit
`pad` fields, so sizes and member offsets are identical with and without
the annotation. Seven `_Static_assert`s now pin those sizes, because a
record that silently grows misreads every existing volume from that point
on and looks like corruption rather than like a struct that changed.
`schfs_node_hdr`, `schfs_item`, `schfs_kptr` and `schfs_uberblock` are
deliberately left alone — the node header is exactly 32 bytes, so
everything after it is already 8-aligned, and annotating them would cost
byte-wise access on the tree's hottest paths to fix a problem they do not
have.
Verified by reading an image written *before* the change: full
`schfs_fsck --check` passes on structure, checksums, key ordering, data
blocks and derived free space.
Also caught on the way: `mkschfs` passed a null pointer to `qsort` for an
empty directory. Every implementation ignores it when the count is zero;
the standard does not say they have to.
**The identity item:**
- [x] ~~**Boot from a USB disk.**~~ The root device is probed at boot instead of
compiled in: ramdisk (only in a `RAMDISK_ROOT` build), then USB, then the
IDE disks, first SchFS volume wins. Verified with **no IDE disk attached
at all** — boots, mounts the stick, writes a file, and the file is still
there on the next boot. `dmesg` says `root: USB disk (device 3)`, and a
machine with no root names every candidate it looked at instead of
panicking with "no root filesystem". See
[../architecture/boot.md](../architecture/boot.md).
#### The `ls -lR` hang, and why it is closed
Hunted with the instrumentation it asked for, under a workload heavier than the
one that originally triggered it — ten iterations of `ls -lR /bin`, interleaved
with writes and `cd`, 33 execs and 13378 tree descents. Measured across it:
```
inode_peak 9 / 200 unchanged
file_peak 9 / 512 unchanged
buf_pinned_peak 1 / 256 unchanged
schfs0 locked 0 tx_owner_pid 0 tx_depth 0
```
Nothing climbed, nothing wedged, and the repaired `holdingsleep` assertion never
fired. **That eliminates this file's own stated hypothesis** — "a lock or
reference-count leak somewhere in the `*at()` paths" — and it is very likely the
bug was fixed in passing by the `*at()` dirfd rewrite, the getdents cookie
rewrite, or SchFS becoming the root, all of which landed after it was reported.
It is closed rather than chased further, and that is a deliberate application of
the two-things-in-flight rule. The instrumentation is permanent: if it ever
happens again, `/proc/fsstat` names the table, or shows `locked 1` with
`tx_owner_pid 0` and the volume lock is orphaned. That is a much better position
than the one it was reported from.
Two real defects were found on the way and fixed, neither proven to be the
cause: `holdingsleep()` never compared the owning pid, which disarmed all seven
`panic()` assertions built on it; and SchFS's `tx_mine()` compared a bare
`struct proc *`, which a recycled ptable slot makes ambiguous.
### Gate 1 — The foundations
[foundations.md](foundations.md), items 1–4. Descriptor passing, a shareable
memory object, `AF_UNIX` in the filesystem namespace.
Nothing above needs these; everything below stops without them.
- [x] ~~**Item 1 — `sendmsg`/`recvmsg` with `SCM_RIGHTS`.**~~ Built and tested;
fourteen checks in `cmd/socktest`, each moving data through the passed
descriptor rather than inspecting it. Described in
[../architecture/networking.md](../architecture/networking.md#descriptor-passing).
This was also mechanism 1 of [hybrid.md](hybrid.md) — one primitive for
both, which is the whole reason those are one plan.
- [x] ~~**Item 2 — a shareable memory object.**~~ `memfd_create`, `ftruncate`,
`mmap(MAP_SHARED)`. Built on the `shmget` segments exactly as
[foundations.md](foundations.md) proposed — the pages, refcounts,
`no_free` VMAs and the fork/exit accounting were all already right; what
memfd adds is a segment reached by *descriptor* rather than by name or
System V key, which is the whole reason it exists on Linux.
`cmd/memfdtest`: seventeen checks, ending with a memfd sent over a
socket to a receiver that closes every inherited copy first.
- [x] ~~**Item 3 — `AF_UNIX` sockets in the filesystem namespace.**~~ `bind()`
creates a `T_SOCK` node; `connect()` resolves the path and matches the
socket bound to *that inode*, not to a string. `ls -l` shows
`srwxr-xr-x`, `stat()` reports `S_IFSOCK`, `unlink()` removes it, and a
second bind to a live path is `EADDRINUSE` *from the directory* — which
is what makes a socket file left by a crashed server refuse a rebind, as
it must. `cmd/socktest` section 15. `flock()` is still missing and is
recorded as such.
### Gate 2 — Init as a role — **done**
Milestones I0–I3, all built. PID 1 forks a supervisor and spawns jobs on its
behalf; the job table lives in `/proc/schinit`; a heartbeat runs both ways; and
`proc_reincarnate()` re-execs PID 1 in place rather than panicking. Both failure
paths are exercised by `sysrq` keys rather than reasoned about.
Before the desktop rather than after, because a compositor being brought up for
the first time crashes constantly, and something that restarts it and says why is
worth far more during that work than at the end of it.
The description is now [../architecture/init.md](../architecture/init.md); the
plan document it replaced has been deleted, per
[the rule](../README.md#the-rule-that-keeps-this-from-rotting).
### Gate 3 — The desktop
[xorg.md](xorg.md): Xorg on the framebuffer, twm and xterm, then Window
Maker. Wayland ([wayland.md](wayland.md)) is not being worked on -- what it
needs, DRM/KMS above all, is far larger than what X needs.
**W4 and W5 are not in 1.0.** Multi-window focus policy and a panel are
post-1.0. One terminal window that works is the bar.
| Gate | What | State |
|---|---|---|
| **0 — Fix what is already wrong** | the SMP register corruption, `setsockopt` that lied, timestamps that read 1970, the `ls -lR` hang (hunted, not reproducible — [history](../history/kernel.md) §24), … | **closed** |
| **1 — The foundations** | descriptor passing, a shareable memory object (`memfd`), `AF_UNIX` in the filesystem, dynamic linking, the event descriptors | **closed** — [kernel.md](../architecture/kernel.md), [networking.md](../architecture/networking.md) |
| **2 — Init as a role** | `schinit`, the supervisor, the heartbeat, PID 1 re-exec'd in place | **closed** — [init.md](../architecture/init.md) |
| **3 — The desktop** | Xorg on the framebuffer, Window Maker, a terminal | **closed** — [desktop.md](../architecture/desktop.md) |
| **4 — A root that survives a reboot** | the open decision above | **open** |
---
## Not in 1.0
The point of this list is that it is *written down*, so it can stop occupying
attention. Nothing here is cancelled.
The point of this list is that it is *written down*, so it can stop occupying attention. Nothing here is cancelled.
| | Why it waits |
|---|---|
| **The rest of the hybrid kernel** ([hybrid.md](hybrid.md)) | Still not a completeness requirement — but the mechanisms and the first eviction happened early, because the alternative to a driver compatibility layer is writing a driver for every device that exists, and that is not a plan. What is deferred is the foreign driver host, not the interface it needs. |
| **Mesa, SDL3** | The kernel and musl do dynamic linking now (`cmd/dynhello`), but every shipped program is still static. Mesa is the single biggest post-1.0 unlock and it is a whole project. |
| **A page cache** | `memfd` covers the desktop's needs. General writable `MAP_SHARED` of a file is a real project with no 1.0 consumer. |
| **`.sip` and the installer** | 1.0 installs by writing a USB stick. That is how most systems start and it is honest. |
| **SchFS snapshots, FUSE, Linux module, sparse files, online fsck** | The format version (Gate 0) is the prerequisite for all of them, and it is the only part on a clock. |
| **The interactive crash screen** | PSOGM renders, prints a full report and emits a QR code. That is enough. |
| **GPU acceleration, Mesa** | [gpu.md](gpu.md). A whole project; software rendering is enough to use the machine. |
| **A page cache** | General writable `MAP_SHARED` of a file is a real project with no 1.0 consumer. |
| **SchFS snapshots, FUSE, a Linux module, sparse files, online fsck** | The format version is the prerequisite and it exists; the rest is [roadmap.md](roadmap.md) §6. |
| **CFF/OTF fonts, kerning, text layout** | TrueType and fixed advances are enough for a terminal. |
| **Routing, `getsockname`/`getpeername`, IPv6** | One interface, one default route, works. |
| **Multiple outputs, damage tracking, subsurfaces, xkbcommon** | One screen, full repaints, one keymap. |
| **Audio** | Does not exist in any form and is not required to use a machine. |
| **Panel, notifications, settings, file manager** | A terminal is a desktop. |
| **SchAPI itself** | Deliberately. It should be designed *from* the compositor's experience, not before it. |
| **Routing, IPv6, Wi-Fi** | One interface, one default route, works. |
| **HDA audio, recording, mixing** | AC'97 plays; that is sound. |
| **Threads' loose ends, `ptrace`, `seccomp`** | Whatever a package needs is found by running it. |
| **SchAPI itself** | Deliberately. It should be designed from the desktop's experience, not before it. |
---
## The rule for this list
**Adding to Gate 0–3 requires deleting something else from it.**
**Adding to the gates requires deleting something else from them.**
Not a productivity trick — a forcing function. The reason this project feels
overwhelming is that ~20 fronts have been open simultaneously, and no amount of
discipline makes that workable. The fix is structural.
Not a productivity trick — a forcing function. The reason this project felt overwhelming is that ~20 fronts
were open simultaneously, and no amount of discipline makes that workable. The fix is structural.
And one working limit, worth holding to literally:
> **Two things in flight. Never three.**
One in Gate 0 and one being written up. Everything else is closed — in the
document, and out of your head.
---
## How you will know it is going well
Not by velocity. By this:
**Every closed item leaves a description behind.** When the `ls -lR` hang is
fixed, `history/kernel.md` gains an entry saying what it actually was. When the
console moves onto the text layer, `architecture/graphics.md` loses a line from
its *What it does not do* section.
Those "does not do" sections shrinking is the real progress bar, and it is the
only one that does not lie.
---
Not by velocity. By this: **every closed item leaves a description behind.** When the `ls -lR` hang was
closed, `history/kernel.md` gained an entry. When a gap closed, a "what it does not do" section shrank.
Those sections shrinking is the real progress bar, and it is the only one that does not lie.
## A note worth keeping
This document was written because the work started to feel overwhelming, and
that is worth recording rather than tidying away — because the feeling was
about the *shape* of the work and not its quality, and the two are easy to
confuse when you are inside it.
This document was written because the work started to feel overwhelming, and that is worth recording rather
than tidying away — because the feeling was about the *shape* of the work and not its quality, and the two are
easy to confuse when you are inside it.
The evidence at the time: SMP working on real hardware after five bugs stacked
behind each other; a copy-on-write filesystem whose crash test cuts the power one
write later, 119 times, and demands a clean mount after each; a TCP/IP stack
asserted from both ends; an xHCI driver written from the specification; and a
documentation tree that records why things are strange so they do not get
un-fixed by accident.
The evidence at the time: SMP working on real hardware after five bugs stacked behind each other; a copy-on-write
filesystem whose crash test cuts the power one write later, 119 times, and demands a clean mount after each; a
TCP/IP stack asserted from both ends; an xHCI driver written from the specification; and a documentation tree that
records why things are strange so they do not get un-fixed by accident. Since then: a desktop, real Linux
programs and Linux's own drivers, running.
None of that is the output of a project in trouble. It was a project with too
many tabs open.
None of that is the output of a project in trouble. It was a project with too many tabs open.