SchOS/docs/plan/hybrid.md

# The kernel becomes hybrid

"Hybrid" is used to mean everything from "a monolith with a nice device model" to
"a microkernel that gave up". **Drawing the line precisely is most of the work**,
so that is what this document is.

The device model it builds on is
[../architecture/devices.md](../architecture/devices.md). It is last on the
plan, and the reason is in [README.md](README.md#the-order-and-why-it-is-not-the-order-they-are-listed-in).

---

## The line

### Stays in the kernel, permanently

- scheduling, address spaces, the trap and syscall path
- IPC and shared memory — the mechanism everything else is built out of
- the VFS and the **root** filesystem
- IRQ demultiplexing
- the console and PSOGM

**That last one is not a size compromise, it is an invariant.** The crash screen
must work when everything else in the system is dead, which means it cannot
depend on anything that could *be* dead. A crash screen served by a userspace
process is a crash screen that is absent in exactly the case it exists for.

### May leave, and should

- GPU drivers
- USB, and every USB class driver under it
- non-root filesystems
- NICs after the first
- input transformation — layouts, gestures, remapping

### The rule that separates the columns

> **Anything needed to reach a shell stays in. Anything discovered at runtime,
> replaceable, or plausibly written by someone else goes out.**

That is NT's compromise and macOS's, arrived at from the same pressure, and it is
the one that survives contact with a real machine.

---

## Built so far

**Mechanisms 2 and 4 exist.** `/dev/pci` (include/dev/pciuser.h, kernel/dev/pciuser.c)
enumerates the bus, hands a device to a process exclusively, maps a BAR into it
and allocates memory a device can be pointed at. `pcitest` proves it end to end:

```
  #  bdf       vendor:device  class        driver
  6  00:04.0   8086:100e     network      e1000
  7  00:07.0   8086:10d3     network      -

device 7: 8086:10d3 (network)
  claimed
  BAR0 at 0x0000000081060000, 128 KB
  regs[0..3] = 0x00140261 0x00000000 0x00080283 0x00000000
  MMIO reaches the device
  DMA 8 KB at virtual 0x40001000, physical 0x3f8eb000
```

Those register values are the e1000e's CTRL and STATUS, with full-duplex and
link-up set -- which is how you tell a working mapping from one that reads
0xFFFFFFFF off an unmapped address. Claiming a device the kernel drives is
refused, and everything a process held is released when it exits, because a
driver that crashed must not keep its hardware away from its replacement.

**Mechanism 3 exists too.** A process asks for a device's interrupts and then
reads them, uio-style:

```
  interrupts routed here (line 10)
  interrupt delivered, count = 1
  ICR = 0x80000004
```

`0x80000004` is the e1000e's cause register with the bit the test asserted and
INT_ASSERTED set, read back by userspace as its acknowledgement.

The line is **masked by the kernel on arrival and unmasked by userspace's
write()**, and that is the entire design rather than a detail. A PCI interrupt
line is level-triggered: it stays asserted until the cause is cleared *at the
device*, which is a different register on every chip and therefore something
only the driver can do. Signalling EOI on a line that is still asserted
re-enters the handler immediately and for ever -- 72,000 times a second, when
this kernel last did it by accident (history/kernel.md §13).

So a driver that dies mid-interrupt leaves the line **masked, not unmasked**.
The instinct is the other way round -- do not leave the device dead -- and it
is wrong: unmasking an asserted line with no handler attached is a crashed
driver taking the machine down with it. Verified by killing one on purpose:

```
  interrupt asserted; exiting WITHOUT acking (simulated crash)
  timer: 3431 ticks taken, 3436 expected
```

Five ticks lost in seven seconds, which is the ordinary background. The next
driver to claim the device re-arms the line, so masked is safe without being
permanent -- also verified.

## The first eviction happened

`cmd/vgpud` is the virtio-gpu driver, outside the kernel. It finds the device
by vendor and device id, asks for it, and the kernel driver hands it over:

```
vgpud: virtio-gpu at 00:03.0, currently held by virtio-gpu
virtio-gpu: released to userspace; screen is the VBE framebuffer
vgpud: claimed
vgpud: display is 1280x800
vgpud: scanout is ours, framebuffer at physical 0x3eceb000
vgpud: drew a test pattern from userspace
```

Full virtio 1.0 bring-up from a process: reset, feature negotiation, virtqueue
in DMA memory, GET_DISPLAY_INFO, RESOURCE_CREATE_2D, ATTACH_BACKING,
SET_SCANOUT, and TRANSFER/FLUSH to present. The pattern it draws is on the
virtio-gpu head while the console carries on in VBE on the other one -- which
is the fallback rule working rather than being described.

**No syscall knows this is a GPU.** The kernel handed over a PCI device; the
process decided what it was. That is the property that makes the interface
general, and it is why the evictions come before anything foreign.

### What the first real driver asked for that a test program had not

- **PCI configuration space** (`PCIIOC_CFG`). A virtio device says where its
  registers are in *vendor capabilities*, which live in config space -- so a
  driver that cannot read it cannot find its own registers. Anything modern is
  discovered the same way.
- **A way for the incumbent to let go.** `struct pci_driver` gained a
  `release` hook. A driver without one keeps its device, which is always safe
  and, for anything on the path to a shell, correct.
- **The outgoing driver freeing its DMA.** Releasing the device while keeping
  its 4 MB framebuffer is half a handover: the incoming driver asks for one and
  is told there is no room. Found by doing it.
- **A bigger arena.** 4 MB was exactly one framebuffer at this resolution, and
  for the moment either side of a handover there are two. 16 MB now.

### And one bug it reproduced immediately

The first working run raised this:

```
irq 11: 77780 interrupts/second -- a device on this line is not acknowledging it.
irq 11:   handler 'e1000' is attached
```

`vgpud` polls the used ring, so it registered no handler -- and a virtio device
interrupts on every completion. A level-triggered line nobody acknowledges
re-fires for ever, drowning the e1000 that shares it. The fix is one line,
`VRING_AVAIL_F_NO_INTERRUPT` before the queue is enabled, and the value of it
is that the storm detector built for history/kernel.md section 13 named the
line, the rate and the neighbour without anyone having to look.

**All four mechanisms now exist**, and with H3 done and H4 withdrawn, nothing
remains on this plan. What the mechanisms were built for next is the foreign
driver host -- [ldls.md](ldls.md).

### One thing the interface is deliberately not

It is about **PCI devices**, not about any driver model, and its first users
will be SchOS's own drivers. That ordering is the design: an interface whose
first consumers are native drivers is a general mechanism that something
foreign can use later, rather than a hole shaped like something foreign. The
difference matters technically -- a Linux-shaped interface would be wrong for
the evictions this plan actually calls for -- and it happens to be the only
version that survives a licensing conversation.

---

## The driver came back

H2 is the milestone this plan calls the one that cannot be skipped, and the
reason is that H1 proves only half of the idea. A driver that runs in a process
is interesting; a driver that runs in a process, *dies*, and is replaced by
another process that picks the device up is the thing the whole hybrid argument
rests on.

`etc/schinit/vgpud.job` makes it a supervised job -- `KeepAlive = yes` -- so
killing it is the whole test:

```
# cat /proc/schinit
1 4 11 os.schos.vgpud
# kill 4
schinitd: restarting os.schos.vgpud
vgpud: virtio-gpu at 00:03.0
vgpud: claimed
vgpud: display is 1280x800
vgpud: scanout is ours, framebuffer at physical 0x3eceb000
vgpud: pid 8 drew a test pattern from userspace
```

The test pattern ends in a twelve-block tally of the drawing process's pid,
because a second run that redraws an identical picture is indistinguishable
from a first run that never stopped. Read off the virtio-gpu scanout across
four generations of the driver it says 4, 8, 10, 12. The same physical address
comes back every time, which is the DMA arena reclaiming cleanly rather than
drifting.

### What it took that H1 had not needed

- **A halt the kernel can apply without knowing what the device is.** A driver
  that dies leaves its device bus-mastering into memory the kernel is about to
  hand to somebody else. The kernel cannot quiesce a device it does not
  understand, but it can clear `PCI_CMD_BUS_MASTER`, and that is enough to stop
  the writes. `pciuser_stop()` does it before the DMA is freed, not after, and
  claim/release became a symmetric pair: claiming sets the bit, releasing
  clears it. `pcitest` grew a `bm` column so the state is visible rather than
  assumed.
- **A refusal, when there is nothing to fall back to.** `vgpu_release()` now
  asks `vga_fb_restore()` whether there is a framebuffer to put the console on,
  and declines to hand the device over if there is not. The rule that the
  kernel keeps a minimal driver for everything on the path to a shell is only
  worth writing down if something enforces it.
- **The DMA arena not being handed to two allocators at once.** This is the bug
  that stopped H2 for a day, and it is worth writing out: `freevm()` frees
  every mapped page below `PHYSTOP`, and the DMA arena is the top 16 MB of
  exactly that range. So a DMA block mapped into a driver and left for
  `freevm()` went onto the page allocator's free list when the driver died,
  while `pciuser_release_all()` handed the same pages back to the arena. Both
  allocators then believed they owned it. The restarted `vgpud` asked for its
  4 MB framebuffer, got pages `kalloc()` had already given out -- including its
  own page tables -- and `dma_alloc()`'s zeroing wiped them, so the driver died
  inside the ioctl, on the return from a syscall whose stack had stopped
  existing. `do_dma()` now records a VMA with `no_free` set, the same way
  shared memory does, so `proc_release_shared()` unmaps the range before the
  page tables are torn down.

### The clause that is not true yet

The milestone says *"the compositor reattaches"*. It does not, because there is
no compositor running: Schcom is v0 and is not started at boot. What has been
shown is the two parts that do not depend on it -- the screen falls back to
VBE, and the driver restarts and takes the device back. During the gap the VBE
head carries a live console and the virtio-gpu head still shows the dead
driver's last frame, which is exactly what a device nobody is driving does.

(That last sentence is still true of a *crashed driver*, and no longer true of
a crashed *kernel* -- see "The clause nobody wrote down" below.)

Reattachment is a Schcom milestone, not a hybrid one, and it is recorded that
way rather than counted here.

---

## A modeset, from a process

H3 is one sentence in the milestone table -- *"a modeset API, which is the
thing an out-of-kernel GPU driver makes affordable and an in-kernel one never
got"* -- and the claim in it is worth testing rather than repeating.

```
# schmode
1280x800 current
1280x800 maximum, framebuffer 4000 KB
# schmode modes
1280x800  *
1280x720
1024x768
800x600
640x480
# schmode 1024 768
1024x768 ok
```

The scanout really resizes: captured from the virtio-gpu head, the frames come
back 1280x800, then 1024x768, then 640x480, then 1280x800 again, with the
checkerboard band showing no shear at any of them.

**Nothing in the kernel changed for this.** No syscall on the path has heard of
a scanout; `schmode` writes a line to a socket and `vgpud` sends five commands
to the device. That is the whole of the milestone, and the reason it is
interesting is the comparison: the same five commands in the kernel would have
been the same five commands *and a reason to be nervous*, because getting the
sequence wrong there takes the machine instead of one process. **That is why
the in-kernel driver never grew one** -- not difficulty, exposure.

### What it costs, and what it refuses

The framebuffer is allocated **once, at the largest mode**, and every smaller
mode reuses it. `PCIIOC_DMA` has no free -- a driver's blocks come back when it
exits and not before -- so a modeset that allocated would spend sixteen slots
and then the arena, and the failure would look like a GPU bug rather than like
counting. A mode larger than the initial allocation is refused, and says why:

```
# schmode 1920 1080
refused: larger than the framebuffer, which is fixed at startup
```

That bound is 4 MB, `PCIIOC_DMA`'s cap, which at 32 bits per pixel is about
1280x800. Raising it is a one-line change to a constant and a question about
how much of a 16 MB arena one driver should be able to take, which is a better
question than it looks and is not answered here.

### The socket, and the argument it settles

`vgpud` used to end in `for(;;) sleep(100);` under a comment arguing that
building a control socket before there is a compositor to judge it is exactly
the mistake this document spends a section warning about.

That argument was right about a *compositor protocol* and wrong as a reason to
have no socket at all. A modeset API has a consumer that exists -- somebody
changing the mode and looking at the screen -- and `schmode` is small enough
that it can judge an interface this small. So the socket is there, it speaks
lines of text for the same reasons
[../architecture/debugging.md](../architecture/debugging.md) gives, and what is
deliberately **not** on it is everything a compositor would need: no buffer
submission, no damage, no vblank. Those still wait, for the original reason.

---

## The mechanisms

Four, and only the fourth is large.

### 1. A control channel — **built**

`AF_UNIX` plus `SCM_RIGHTS` — item 1 of [foundations.md](foundations.md), which
the Wayland work also needed first. A descriptor is how a driver process is
handed the device it is allowed to touch.

Built as `sendmsg`/`recvmsg` in Linux's shape, because the programs that will
use it are not ours: libwayland and toybox both call `recvmsg` with Linux's
structures and would not notice a friendlier door. Descriptors are pinned to the
stream position of the first byte of their message, and the kernel holds a
reference to each until it is received — or closes it when the socket is dropped
with the message still queued, which is the case that leaks if nobody writes it
down. See
[../architecture/networking.md](../architecture/networking.md#descriptor-passing).

**Do not build Mach ports.** The port model is worth its cost when there is a
naming and capability story to hang on it; there is not one yet, and inventing it
before the first userspace driver exists is designing against an imaginary user.

### 2. MMIO to userspace

Map a PCI BAR into a process.

`sys_mmap` already maps the framebuffer's real physical range at its real
address — the path exists. This is that path generalised, behind a check that the
caller was granted that device.

### 3. Interrupts to userspace

**The handler stays in the kernel and does one thing: write to an `eventfd`.**
The driver process reads.

`eventfd` exists already, and that is why it is the right primitive. A signal
would be delivered to a process that cannot safely be interrupted mid-transaction.

### 4. DMA to userspace

Physically contiguous memory whose physical address the process is told.

`kernel/core/dma.c` already owns an arena reserved for exactly this, before
`kalloc` gets anything. It needs a door, a per-device budget, and a way to hand
the region back.

---

## The clause nobody wrote down: a panic has to reach the screen

Handing a device away costs something that was not on the list. The kernel
panics, paints the report into the console framebuffer, and halts — and the
display it gave to a process keeps showing whatever that process last put
there. On a machine with two heads, one says the kernel is dead and the other
shows a shell prompt.

It is not a race that can be won by being careful. There is no scheduling after
a panic, so the process holding the device is never going to run again, and the
kernel let go of the device deliberately.

**So the userspace driver says in advance where it is scanning out**, with
`PCIIOC_PANICFB` (see `include/dev/pciuser.h`), and the kernel does the rest:

1. paints the report into that memory,
2. asks the *kernel* driver the device was taken from to put it on screen —
   `panic_present` in `struct pci_driver`,
3. which resets the device and rebuilds one queue from **static** memory: no
   allocation, no locks, no sleeping, because a corrupted heap is one of the
   ways a machine gets here in the first place.

The registration is checked like the promise it is: the range must lie inside a
block that process got from `PCIIOC_DMA` on that device. Without that it would
be a way to nominate any physical address in the machine to be overwritten at
panic, which is a worse bug than the one it fixes.

What this buys is a property worth stating plainly: **a device can be evicted
to userspace without the kernel losing its ability to tell you the machine
died.** That is the difference between a driver boundary and a blind spot, and
it is one more thing the in-kernel arrangement got for free and the hybrid one
has to earn.

It costs 20 KB of static memory in the virtio-gpu driver, paid whether or not
the machine ever panics. Allocating it on demand would be cheaper and would not
work.

---

## The honest limitation

> **There is no IOMMU, so a userspace driver is isolated against its own bugs and
> not against malice.**

A process that can program a device's DMA engine can write to any physical
address, whatever its page tables say.

That is not a reason to stop — isolation against bugs is most of the value, and a
crashed driver that restarts instead of panicking the machine is the entire
point. But it must be written down, because *"userspace driver"* reads as
*"sandboxed"* and here it does not mean that yet. VT-d/AMD-Vi is its own project
and comes after.

---

## Order, and one thing not to do

### Do not build loadable kernel modules first

It is the obvious first step and it is a generation behind. It means designing a
**stable in-kernel driver ABI**, which is the most expensive promise in this
entire plan, for a mechanism Apple is actively retiring in favour of the one
above.

`../architecture/devices.md` already separates the two ideas. They are separable,
and the second one is the one to build.

### First eviction: the GPU driver

Four reasons it is the right one and not just the interesting one:

- **The boundary already exists and is already correct** — `struct display_ops`.
- **It has exactly one consumer**, the compositor, so the interface has a real
  user to be judged by.
- **There is a working fallback underneath.** The VBE framebuffer stays
  in-kernel, so a GPU driver that fails to start degrades to a working screen
  rather than to no screen.
- It directly serves the work that will actually be happening.

### Second: USB — and why it turned out not to be

`xhci.c` is 1364 lines, self-contained, already spoken to through a bus
abstraction, and it is where third-party drivers will show up first. This is what
DriverKit was built for and the resemblance is not accidental.

#### It is not the same shape as the GPU, and that was missed

Written before the GPU eviction had been done, H4 says *"USB, same shape"*. It
is not, and the difference is one this document's own rule already decides.

The GPU could leave because **there was a working fallback underneath it**: the
VBE framebuffer stays in the kernel, so a GPU driver that fails to start
degrades to a working screen. That is one of the four reasons the GPU was
chosen to go first, listed a few lines above.

USB has no fallback, and it carries two things that are on the path to a shell:

- **The root disk.** `USBDISK_DEV` dispatches through `bio.c` to
  `usb_storage_rw`, and `usbstick.img` is a real boot medium built by `make`.
- **The keyboard.** `usb_hid` feeds `input_report(INPUT_KBD, ...)`, which is how
  anything without PS/2 is typed on — including the Chromebox this is meant to
  boot.

So evicting `xhci.c` wholesale breaks the rule three sections down:

> **The kernel keeps a minimal driver for everything on the path to a shell, and
> userspace owns the good one.**
>
> Break this and you get the circular boot failure where the compositor cannot
> start because the display driver is a process that needs a compositor.

The USB version of that sentence is *the root filesystem cannot mount because
the storage driver is a process that lives on the root filesystem*, and it is
the same failure wearing different clothes.

There is a second difference, subtler and probably the harder one. `/dev/pci`
hands over a **PCI function** — one device, one owner, nothing shared. A USB
device is not that: it sits behind a controller that other devices also sit
behind, so the unit of ownership and the unit of hardware are no longer the
same thing. Handing a process "the xHCI controller" gives it the keyboard too;
handing it "one USB device" is an interface `/dev/pci` does not have and that
this plan has not designed.

#### Decided: USB stays in the kernel

**H4 is withdrawn.** Not deferred, not blocked on the interface above —
withdrawn, because the premise was wrong.

> **USB is not a peripheral.** It is the bus the peripherals arrive on.

Nearly every machine has a USB port in some form, down to a bad micro-B on a
board that has nothing else, and the ones that do not have one are not made
worse by the stack being present. That makes USB structurally the same kind of
thing as PCI, not the same kind of thing as a GPU — and this document has never
proposed evicting PCI.

Reading the eviction rule again with that in mind, it already said so:

> **Anything needed to reach a shell stays in. Anything discovered at runtime,
> replaceable, or plausibly written by someone else goes out.**

USB is how things are *discovered at runtime*, which is not the same as being
one of them. The keyboard and the root disk that arrive over it are on the path
to a shell, and the stack that finds them is part of that path.

The two obstacles in the section above are real and they are why the premise
got questioned, but they are not the reason for this decision. Even with a
minimal in-kernel fallback written and a per-device ownership interface
designed, the answer would be the same: the good USB driver and the minimal one
would be the same driver, and an eviction that leaves an identical copy behind
has moved nothing.

What the eviction was *for* — third-party drivers, written by someone else,
living outside the kernel — is a real goal and it survives this. It is simply
not a USB goal; it belongs to the foreign driver host, which is
[ldls.md](ldls.md).

### The rule that keeps the boot path honest

> **The kernel keeps a minimal driver for everything on the path to a shell, and
> userspace owns the good one.**

Break this and you get the circular boot failure where the compositor cannot
start because the display driver is a process that needs a compositor.

---

## Milestones

| | | |
|---|---|---|
| **H0** | Mechanisms 2, 3 and 4, each with a test that exercises it from a userspace program against a device nothing else is using. | done |
| **H1** | A userspace virtio-gpu driver, publishing `display_ops` across the channel. Schcom talks to it and does not know the difference. | done, except that Schcom does not talk to it -- `cmd/vgpud` holds the device and draws its own frame |
| **H2** | Kill the driver process on purpose. The screen falls back to VBE, the driver restarts, the compositor reattaches. **This is the milestone that proves the whole idea**, and it is the one that cannot be skipped. | done, less the compositor -- see "The driver came back" |
| **H3** | A modeset API, which is the thing an out-of-kernel GPU driver makes affordable and an in-kernel one never got. | done -- see "A modeset, from a process" |
| ~~**H4**~~ | ~~USB, same shape.~~ **Withdrawn** — USB is the bus peripherals arrive on, not a peripheral. See "Decided: USB stays in the kernel". | |

---

## What it will still not do

- **No isolation against a hostile driver**, per the IOMMU note above.
- **No stable driver ABI**, deliberately. Out-of-tree drivers remain a
  recompile-with-the-tree affair until there is a reason to promise otherwise.
- **The root filesystem stays in the kernel**, so SchFS cannot be restarted and a
  bug in it is still fatal. Moving it out needs an initramfs story that does not
  exist.
- **The first NIC stays in the kernel**, because the network thread and lwIP are
  in the kernel and moving them is a separate project with no current benefit.
- **No driver sandboxing beyond address-space separation** — no seccomp-shaped
  syscall restriction, no resource limits per driver.