SchOS/docs/roadmap.md

Roadmap

Written 2026-08-04, from the plan: USB, GPU, fonts, networking and sockets, then a compositor (Schcom), a window manager (SchWM) and a desktop (SchDE), plus an installer and a .sip package format.

This file records two things that are easy to lose: the order the work has to happen in, and the decisions that constrain it.

Superseded in direction, not in order. SchOS is now aimed at being a system rather than a Unix-like kernel with a userland attached: a NeXTSTEP/ macOS-shaped layering, with POSIX kept complete but demoted from identity to personality, and a native API (SchAPI) above it. See architecture/schapi.md. That decision changes what the phases below are building towards; it does not reorder them, and it does not start until the tree builds and boots natively again.

That assumption about §1 has since been revised. NetBSD's TCP/IP stack was the intended starting point; the per-file licence check found that NetBSD's tcp_input.c and tcp_output.c carry the WIDE Project's 4-clause advertising clause, which Berkeley's rescission does not reach. FreeBSD's equivalent is clean, and lwIP is cleaner still and a far smaller port. See Phase 2 below and docs/licensing.md §3.

---

1. Where drivers can come from

BSD: yes. Linux: no. This is not a preference, it is a licence incompatibility, and it decides the whole driver strategy.

The reasoning — Apache-2.0's patent terms counting as "further restrictions" under GPLv2 §6, which is why the two cannot be combined and redistributed — is in licensing.md, along with the per-licence table and the NetBSD-specific rules. Read it before copying anything. The short version: BSD-2 and BSD-3 code can be used directly with a NOTICE entry; GPL-2.0-only code can never be, at any distance.

There is a third source that is better than either: the specifications. xHCI, AHCI, virtio, NVMe, the Intel e1000 and i915 datasheets are all public documents with no licence attached to what you write from them. That is where kernel/drivers/xhci.c came from.

One caveat about "port it from BSD" as a plan, learned by looking: FreeBSD's sys/dev/usb is roughly 100k lines welded to newbus, devclass, bus_dma, callouts, taskqueues and FreeBSD's locking. Porting it means porting FreeBSD's device framework first and finding the USB stack somewhere underneath. NetBSD's is more portable and the same argument applies with a smaller multiplier. BSD is far more valuable as the reference you check yourself against than as the code you paste. Read it when the spec is ambiguous — which it is, often — and write your own.

Reading Linux for understanding is fine and normal. Translating a Linux driver function-by-function is not; the result is a derivative work whatever the variable names say.

---

2. Ordering, and why it is not the order it was asked in

The requested order was USB → GPU → fonts → network → GUI → installer. That is a reasonable order by interest. By dependency it wants one change:

Shared driver infrastructure comes before all of it. USB, any GPU beyond the VBE framebuffer, and any NIC all need the same three things, none of which existed:

• a way to register an interrupt handler at runtime (kernel/dev/irq.c)
• physically contiguous memory for rings and descriptors (kernel/core/dma.c)
• PCI capability walking and MSI (kernel/dev/pci.c)

That work is done. It was the first step of USB, not a detour from it.

The other honest note: USB is the item that does least for the GUI. Under QEMU there is already a PS/2 keyboard and mouse, so USB buys real hardware support — which matters, but not for getting Schcom on screen. Sockets and raw input events are what actually block the compositor. USB is being done first because it was asked for first; the ordering below reflects that with the dependencies fixed.

---

3. The plan

Phase 1 — USB (done, bar the boot path)

• Shared infrastructure: IRQ registration, DMA arena, PCI capabilities
• xHCI: reset, DCBAA, command ring, event ring, scratchpad
• Port enumeration, Enable Slot, Address Device, control transfers
• Device descriptors read from real devices (USB2 and USB3 paths)
• Configuration and interface descriptor parsing, SET_CONFIGURATION
• Interrupt endpoints, driven from the IRQ handler
• HID boot protocol: keyboard and mouse into the existing input paths
• Bulk endpoints, bulk-only transport, SCSI, a block device at USBDISK_DEV — a superblock reads back through bread()
• Hubs — route strings, hub slot contexts, per-port reset and power. A mouse behind a hub enumerates a tier deep, which is what proves it
• Making a USB disk mountable and bootable (needs the root device to be selectable rather than compiled in)

Phase 2 — Sockets, raw input, and networking

Do sockets before the network stack. Unix domain sockets are what Schcom needs — clients and compositor have to pass messages and share buffers, and a pipe cannot carry an shm handle. Unix sockets are also far simpler than TCP/IP, so they are the right thing to build the socket layer's shape around.

• socket/bind/listen/accept/connect/send/recv syscalls, at their real Linux numbers — they were NO_LINUX stubs, so a musl program calling socket() never even reached the stub
• AF_UNIX SOCK_STREAM, socketpair, shutdown; names live in a table rather than the filesystem (see the comment at the top of kernel/net/unix.c for what that costs)
• Raw input events — /dev/input/event0 and event1, Linux evdev layout, fed by the PS/2 keyboard and mouse and by USB HID. The console keeps its own path; both consumers see every key
• SCM_RIGHTS for passing descriptors. Possibly unnecessary: an shm id is an integer and travels fine over a stream socket, so this waits until something actually needs to pass an open file
• e1000 driver — kernel/drivers/net/e1000/e1000.c, from Intel's public manual 317453. MAC from EEPROM, transmit and receive rings, link detection. Run with -netdev user,id=n0 -device e1000,netdev=n0. Verified to carry real traffic: echo n > /proc/sysrq sends a hand-built ARP request and receives the gateway's reply, which exercises the descriptor formats, the DD handshake, the tail bookkeeping, the DMA addresses and the MAC filter in one exchange. That check came before any protocol stack on purpose -- a stack on an unproven driver fails in ways indistinguishable from its own bugs
• ARP, IPv4, ICMP, UDP, then TCP -- done 2026-08-15, lwIP 2.2.0. Fetched by ports/lwip and compiled into the kernel; the port layer is kernel/net/lwip/ (about 400 lines: a lock, a clock, and the e1000 netif). NO_SYS=1, so lwIP has no threads of its own and one kernel thread (kernel/net/net.c, started by netinit) polls the NIC and runs its timers. DHCP obtains an address at boot. DHCP obtains an address at boot and /proc/netif reports it, along with the MAC, the gateway and frame counts taken at the lwIP seam. Verified end to end, by bootcheck -Net rather than by hand: the guest opens a TCP connection out, the listener on the host receives HELLO-FROM-SCHOS, and the guest receives the host's reply -- so the handshake, both data directions and the close all work, not just the driver. kthread_create was added for this and is generally useful. The reasoning behind not writing it, kept because it is the reason to resist doing so later as well: it is not the packet format that is hard, it is congestion control, retransmission, window management and twenty years of accumulated corner cases that only appear against a real peer. lwIP was preferred over a BSD stack on structure rather than licence -- FreeBSD's sys/netinet is BSD-3 too, but it is not a library: it reaches into mbufs, epoch(9), netisr, VNET and uma(9), so taking it means porting FreeBSD's kernel infrastructure first. NetBSD was ruled out on licence: its tcp_input.c carries the WIDE Project's 4-clause advertising clause. See docs/licensing.md §3.
• AF_INET on top of the same socket layer -- done, exactly as planned: socknew dispatches on the domain, kernel/net/inet.c implements the AF_INET side on lwIP's raw API, and no syscall changed shape. TCP and UDP both work.
• The two things AF_INET owed on the day it landed, both closed 2026-08-15. recvfrom reports the source address -- which meant giving UDP a datagram queue first, since the payloads shared TCP's byte ring and two datagrams arriving together came back as one -- and sendto takes a destination. Blocking sockets sleep on a counter bumped by the lwIP callbacks instead of waking on every timer tick; tcp_sent had to be registered for it, or a write blocked on a full send buffer would sleep with nobody left to wake it.
• bootcheck -Net, so none of the above is verified by hand any more. A listener and a UDP responder run on the host, the guest is told to talk to them, and both ends are asserted -- one side alone cannot tell a working stack from one that completes a handshake and drops every byte. 45 [ok] with it on.
• accept() does not report the peer's address, and there is no getsockname/getpeername. Nothing has needed them yet; a server that wants to log who connected will.
• Receive is still polled. The network thread looks at the NIC every timer tick when the link is idle, so an arriving frame waits up to 10 ms before anything sees it. Waking that thread from the e1000 interrupt is the real fix and was deliberately not the first version: a receive path woken by an interrupt is much harder to debug than one that is asked.

Phase 3 — GPU

"Basic GPU driver" has a right answer and a wrong one. The wrong one is starting with Intel or AMD hardware: modern GPU drivers are enormous and most of the size is mode setting, power management and a command submission model that only pays off once there is something to submit.

• virtio-gpu, 2D. kernel/drivers/virtio_gpu.c, written from the OASIS specification. Full path: GET_DISPLAY_INFO, RESOURCE_CREATE_2D, RESOURCE_ATTACH_BACKING, SET_SCANOUT, TRANSFER_TO_HOST_2D, RESOURCE_FLUSH. Bound as the display class ahead of the VBE framebuffer. Run with -vga std -device virtio-gpu-pci; keep -vga std, because the bootloader's splash goes through VBE and there is no virtio path that early
• A /dev/dri-style device with mode setting, so resolution stops being whatever the bootloader asked VBE for
• Double buffering and vblank — Schcom needs both to not tear
• Only then: real hardware, and only 2D/blit acceleration

Phase 4 — Fonts

This is a bigger job than it sounds and it should be sized honestly. A TTF/OTF rasteriser needs: cmap/glyf/loca/head/hmtx table parsing, quadratic béziers for TrueType outlines and cubic ones for OpenType/CFF, a scanline rasteriser with anti-aliasing, and a glyph cache.

The bigger structural problem is that the console assumes a fixed 8×16 cell grid throughout — vga_putc, the cursor, the ANSI erase operations. Variable width text is not a font change, it is a text-rendering layer.

• The text layer — include/text.h + kernel/dev/text.c. Glyph metrics, a struct surface target so a compositor can draw off-screen, 8-bit coverage with real blending, UTF-8, and font_ops so a rasteriser plugs in as a second provider. This was the actual blocker: the font was never the problem, the character-cell grid was.
• Port the console onto it and delete the cell grid
• A TrueType rasteriser — kernel/dev/ttf.c, glyf outlines, composite glyphs, cmap format 4, anti-aliased scanline fill. Host harness at tools/ttftest.c
• A glyph cache — direct-mapped over ASCII into a 48 KB pool
• Load a font from disk and point the console at it, deleting the cell grid
• CFF/Type 2 charstrings, so OTF fonts work too (a separate font_ops provider; a bytecode interpreter, not an extension of the glyf path)
• A rasteriser. stb_truetype.h is public domain and therefore usable as-is if wanted; writing one is also reasonable and more interesting
• The typefaces, decided 2026-08-09. Both verified, not assumed: Roboto for the desktop, window manager and compositor -- Apache-2.0, the same licence as SchOS, so it needs nothing beyond a NOTICE entry. JetBrains Mono for the terminal and TTY -- SIL OFL 1.1, and notably with no Reserved Font Name declared, so it may be modified and still called JetBrains Mono. That matters more than it sounds: pre-rasterising a font into bitmaps at build time is a modification, and an RFN would have forbidden keeping the name on the result. OFL's one real constraint is that the font may not be sold on its own and must stay under OFL; bundling it in an Apache-2.0 system is exactly the case it is written for.
• Text layout (advance widths, kerning) as a library in lib/
• The console keeps its fixed grid; proportional text belongs to the GUI

Font licensing: public domain or very permissive only. No SIL OFL if it can be avoided, no "free for personal use", no CC-BY-SA. The constraint is the same one that governs drivers (§1): a font file is redistributed inside every SchOS image, so its licence attaches to the image.

Safe choices, in rough order of usefulness:

• CC0 / public domain — best case, nothing to comply with.
• MIT / BSD / Apache-2.0 fonts — fine, needs a NOTICE entry.
• SIL OFL — usable but has a real string attached: a modified font must be renamed, and OFL fonts cannot be sold on their own. Workable, but read it before committing one.

Whatever is chosen, the licence goes in NOTICE before the file is committed, not after.

Phase 5 — The GUI, as separate projects

See projects/README.md. Schcom, SchWM and SchDE live outside the kernel tree on purpose.

• Schcom — the compositor. Owns the framebuffer, composites client buffers, routes input. Needs: unix sockets, shm (done), raw input events, and a GPU driver with double buffering.
• SchWM — window management on top of Schcom's protocol
• SchDE — panel, launcher, settings

Raw input is the missing kernel piece nobody thinks of until it bites: keyboard events currently only arrive through the tty line discipline, cooked and line-buffered and delivered to one foreground process. A compositor needs key-down/key-up with modifiers and no line discipline at all — an evdev-style /dev/input.

Phase 6 — Installer and .sip

.sip (SchOS Installation Package), the stated model being WIM in Windows PE: a file-based image rather than a block image, so one file can hold a whole filesystem, be applied to a partition of any size, and hold several images that share their common files.

The pieces for this mostly exist already. tools/mkschfs.c does content dedup with an (st_dev, st_ino) fast path; SchFS's B-tree is content addressed. A sensible .sip is: a header, a file index (path, metadata, content hash), and a blob store keyed by hash, with compression per blob.

• .sip format specification, written down before any code
• tools/mksip — capture a directory tree into a .sip
• Kernel or userspace apply path
• An installation environment that boots from the ISO, partitions, applies the .sip, and installs the bootloader

---

Phase 7 — The full-stack acceptance test (served its purpose; removed)

For a while this phase was a ported copy of a well-known 1993 shooter, kept as an acceptance test rather than a feature. The reasoning was sound and is worth keeping even though the test is gone: a large program that somebody else wrote, which exercises /dev/fb0, /dev/input/event0, mmap, the ELF loader, gettimeofday and a fixed-rate tic loop simultaneously and notices when any of them lie, is a much harsher test than anything written in-house — because it was not written against our bugs.

It found real ones, all of which are fixed and covered by narrower tests now:

• fcntl(F_SETFL, O_NONBLOCK) was a documented no-op, so the first read blocked forever and looked exactly like a hung renderer.
• /dev/input/eventN replayed its backlog to each new opener, handing a program the keystrokes that had launched it as input.
• Without EVIOCGRAB, a full-screen program and the shell behind it both read the same keystrokes.

It was removed once it stopped finding anything: the port was ~100k lines of third-party source under licences that can never ship in an Apache-2.0 image, and carrying it had a standing cost — 33 MB of non-redistributable data staged into sysroot/, which is linked into the kernel binary for a ramdisk-root build. That combination is a licensing accident waiting to happen, for a test that was passing every time.

What replaced it: tools/windows/bootcheck.ps1 with its -Sock, -Evdev and -Ps modes, plus tools/windows/splashshot.ps1 for the boot path. These are narrower and do not prove the whole stack composes, which is the honest cost of the removal.

The lesson worth keeping is the shape of the test, not the program: the next full-stack test should be something large that we did not write, driven to a real pixel-level assertion rather than a log line. tools/windows/splashshot.ps1's colour counting exists because a log-only harness once passed on a blank screen.

Phase 8 — The system layer: schinit, firmware, and a crash channel

Three ideas parked here rather than started, because each one is a project and the phases above are the agreed order. Recorded so they are not re-derived from scratch later.

schinit, a launchd-shaped init. The point of copying launchd rather than sysvinit is not the config format, it is demand launching: launchd holds the listening socket or watches the path, and starts the job when something actually arrives. That inverts the dependency graph — jobs stop needing to declare what must start before them, because the thing they depend on answers on first use. SchOS already has the two pieces that makes possible: AF_UNIX sockets (kernel/net/unix.c) and a real /proc. What is missing is a supervisor that can hold a socket on a job's behalf and hand it over on exec, plus a job description format. Prerequisite for this being worth anything: ps and top must be able to enumerate processes, or the supervisor is unobservable.

A crash screen you can work in, and a QR code channel out of it. PSOGM (kernel/core/panic.c) currently renders and stops. The idea is two staged things, and the staging matters:

• An interactive mode — "a small OS inside the crash". This is the right framing and it is worth stating precisely, because it decides the design. At panic time the kernel is the only thing running, which is exactly why it can offer what a live system cannot: walk the stack, read another process's memory, dump page tables, inspect the process table — with no scheduler and no locking to respect, because nothing else can race you.

  What makes it a system rather than a print statement is that it keeps the base machine's drivers. Concretely, the ones that work with interrupts off: the VGA/framebuffer (vga_draw_pixel is a plain memory write), the PS/2 keyboard (pollable via port 0x60/0x64), and the serial port. Those three are a complete console. Everything else — IDE, USB, the timer, anything that sleeps or takes a sleeplock — is off limits, because the lock it needs may be precisely the one held by the CPU that just died.

  So the shape is a polled event loop over preallocated state: no kalloc, no sleep/wakeup, no acquire on anything a live path might hold, one static command buffer, and a driver set restricted to the three above. That restriction is not a limitation to work around later; it is the invariant that makes the mode trustworthy when everything else is not. Anything that wants more should be answered by rebooting into a recovery environment instead.

  A useful precedent to read but not copy: DDB (NetBSD/FreeBSD's in-kernel debugger) is exactly this, and its hard-won rule is the same one — a debugger that can deadlock is worse than no debugger.

• A QR code. Done. kernel/core/qr.c (byte mode, EC level L, versions 1–6, no allocation) and a card on the right of the crash screen. A panic report is worth nothing if it cannot leave a machine whose network stack is dead, and a phone camera is an out-of-band channel needing no driver on our side. Deliberately not a URL: it encodes the report itself and reaches nothing.

  It ended up unconditional rather than behind Enter, which is better — it costs nothing to always draw, and requiring a keypress would mean the one case where input is least trustworthy is also the case where the report is hardest to retrieve. Verified end-to-end by make crashshot && make crashqr, which decodes the symbol out of the captured framebuffer.

UEFI. Not a feature so much as the point at which the current bootloader stops being extendable: it is 16-bit real mode, INT 10h VBE, and PIO ATA, none of which exist on a UEFI-only machine. The realistic shape is a second loader — an EFI application that gets the framebuffer from GOP, the memory map from GetMemoryMap(), loads the kernel off a FAT ESP, and enters at the same place boot/bios/stage2main.c does — rather than a port of the existing one. Worth noting that the splash work just done survives that transition: it is a mask, a palette and a layout, none of which care how the pixels were found. What does not survive is every assumption in stage2main.c about where things live in low memory (see docs/layout.md).

SchFS: all five phases are done (2026-08-15)

Phases 1-3 were finished earlier. Phase 3¾ (the commit barrier, uberblock fallback, deferred writes), phase 4 (extent refcounts and reflink) and phase 5 (attributes, the index and the query) all landed on 2026-08-15. See docs/architecture/schfs.md, which is the detailed record.

SchFS beyond SchOS

Three ideas parked 2026-08-08, in the order they should happen.

An on-disk format version, and an upgrade path. This one is not optional and gets harder every day it is postponed: the moment an image exists that someone cares about, changing the layout means being able to read the old one. The superblock already carries a version field and a CRC32c; what is missing is a policy — a documented rule for what a version bump means, a refusal to mount something newer than the driver understands (rather than misreading it), and an explicit upgrade step. Cheap now, expensive later, and it blocks the other two: a second implementation is only meaningful against a format that is pinned.

A Linux implementation. The realistic first form is FUSE, not a kernel module — it needs no GPL entanglement (libfuse is LGPL and used across a process boundary), it can reuse the read/write logic already in tools/mkschfs and tools/schfs_fsck rather than being written twice, and it turns "does our filesystem work" into something testable with fsx and xfstests, which is a far harsher examination than anything in this tree. A Linux kernel module is a different project and carries the licence problem that licensing.md describes.

The missing features. Worth listing honestly before adding any: no journal replay across an unclean shutdown beyond what CoW gives for free, no snapshots (despite the B-tree being copy-on-write, which is most of the work), no extended attributes, no sparse-file support, and no online fsck. Snapshots are the one where the existing design already pays for most of the cost.

---

3b. Known broken

Two things that are wrong now, both reported 2026-08-08 and both diagnosed rather than merely observed. Neither blocks the GUI work, which is why they are here and not in a phase.

SMP is broken (fixed 2026-08-08)

SMP works. One, two and four processors all boot; every CPU reports in with cpu%d: starting, and /proc/cpuinfo lists them all. smp_start_aps in kernel/core/main.c is 1.

Five distinct bugs, each hidden behind the one before it, and all invisible under QEMU because it publishes only one processor in the legacy MP table. VMware published the real count, which is why the triple fault only ever appeared there.

CPU enumeration	mpinit() read the 1994 MP table. kernel/arch/x86_64/acpi.c now reads the ACPI MADT.
triple fault	entry64high had no AP branch — every AP took the BSP's stack and ran kmain() twice. Now tests EBX == 1.
panic("acquire")	seginit() called kalloc() before %fs existed; kalloc locks, locking reads cpu. BSP now preallocates each AP's page.
panic("sched locks")	kernel_fs_base was one global read by trapasm.S on every entry, so all CPUs shared one per-CPU block. Now %gs:0.
null proc in sleep()	cpu/proc were __thread — per thread, so the compiler cached the thread pointer across sched(), but %fs is per CPU. Now mycpu()/myproc().
#GP in syscall_entry	The new per-CPU slots landed on %fs:(8), the syscall scratch word — the same memory as %gs:8. Moved to +16 and +24.

The __thread one is the lesson worth keeping: __thread is wrong for per-CPU kernel data. It means per thread, and the compiler is entitled to cache the thread pointer across calls on that basis. A kernel thread changes CPU at every sched(), so the assumption is false for exactly the variables that most need to be right. include/proc.h carries the disassembly that proves it.

Per-CPU block layout, shared with assembly and not to be reordered:

+0   self-pointer     x86-64 TLS ABI (%fs:0)
+8   syscall scratch  trapasm.S parks %rsp here
+16  struct cpu *
+24  struct proc *

The kernel demands 224 MB of RAM (fixed 2026-08-08)

kernel/arch/x86_64/e820.c reads the firmware's memory map and sets phystop from it. The floor went from 224 MB to 32 MB, and the kernel now sees all the memory a machine actually has: 512 MB in gives MemTotal: 524160 kB, where before it reported 224 MB whatever was installed.

Three details worth keeping:

• It takes the end of the usable region containing the kernel, not the highest usable address anywhere. A memory map has reserved holes in it, and freerange() walks straight from the end of the kernel image to phystop handing out every page between — so the global maximum would put firmware tables on the free list, failing later and elsewhere and looking like anything but a memory map problem.
• The ceiling is 3 GB, because kvmalloc() maps 0–4 GB as four 1 GB pages and marks the top one uncached for PCI BARs.
• Below PHYSTOP_MIN it prints why and stops, rather than triple faulting in silence. At 24 MB: e820: only 23 MB usable; SchOS needs at least 24 MB.

The original text is kept below because the reasoning still applies to the SMP half of §3b, which is the same shape of bug: a constant nobody revisited.

The kernel demanded 224 MB of RAM

Symptom as reported: the OS needs ~256 MB, which is absurd next to Windows 98 or an early Linux.

Diagnosis: it is one constant. PHYSTOP in include/memlayout.h is 0xE000000 — 224 MB — inherited unchanged from xv6, and main() hands P2V(PHYSTOP) straight to kinit1()/dma_init(). The kernel therefore assumes that much physical memory exists and walks off the end of RAM if it does not. Measured 2026-08-08: boots at 224M and above, triple faults at 192M and below. Nothing is actually being used — the requirement is entirely fictional.

The fix is already half-built. boot/bios/stage2.S collects a full E820 memory map in real mode and leaves it at VBE_INFO_ADDR + 0x200 (count) and + 0x208 (24-byte entries); boot/bios/stage2main.c already parses it to print detected RAM. The kernel never reads it. So: have main() take the top of the highest usable E820 region instead of the constant, clamp it to what the direct map covers, and keep PHYSTOP only as a fallback for the case where the firmware returned nothing. That should drop the floor to something in the tens of megabytes and is a contained change.

---

4. Kept out of scope on purpose

• Anything that needs pushing or publishing. This work stays local.