SchOS/docs/plan/hybrid.md

# The kernel becomes hybrid

"Hybrid" is used to mean everything from "a monolith with a nice device model" to
"a microkernel that gave up". **Drawing the line precisely is most of the work**,
so that is what this document is.

The device model it builds on is
[../architecture/devices.md](../architecture/devices.md). It is last on the
plan, and the reason is in [README.md](README.md#the-order-and-why-it-is-not-the-order-they-are-listed-in).

---

## The line

### Stays in the kernel, permanently

- scheduling, address spaces, the trap and syscall path
- IPC and shared memory — the mechanism everything else is built out of
- the VFS and the **root** filesystem
- IRQ demultiplexing
- the console and PSOGM

**That last one is not a size compromise, it is an invariant.** The crash screen
must work when everything else in the system is dead, which means it cannot
depend on anything that could *be* dead. A crash screen served by a userspace
process is a crash screen that is absent in exactly the case it exists for.

### May leave, and should

- GPU drivers
- USB, and every USB class driver under it
- non-root filesystems
- NICs after the first
- input transformation — layouts, gestures, remapping

### The rule that separates the columns

> **Anything needed to reach a shell stays in. Anything discovered at runtime,
> replaceable, or plausibly written by someone else goes out.**

That is NT's compromise and macOS's, arrived at from the same pressure, and it is
the one that survives contact with a real machine.

---

## Built so far

**Mechanisms 2 and 4 exist.** `/dev/pci` (include/pciuser.h, kernel/dev/pciuser.c)
enumerates the bus, hands a device to a process exclusively, maps a BAR into it
and allocates memory a device can be pointed at. `pcitest` proves it end to end:

```
  #  bdf       vendor:device  class        driver
  6  00:04.0   8086:100e     network      e1000
  7  00:07.0   8086:10d3     network      -

device 7: 8086:10d3 (network)
  claimed
  BAR0 at 0x0000000081060000, 128 KB
  regs[0..3] = 0x00140261 0x00000000 0x00080283 0x00000000
  MMIO reaches the device
  DMA 8 KB at virtual 0x40001000, physical 0x3f8eb000
```

Those register values are the e1000e's CTRL and STATUS, with full-duplex and
link-up set -- which is how you tell a working mapping from one that reads
0xFFFFFFFF off an unmapped address. Claiming a device the kernel drives is
refused, and everything a process held is released when it exits, because a
driver that crashed must not keep its hardware away from its replacement.

**Mechanism 3 exists too.** A process asks for a device's interrupts and then
reads them, uio-style:

```
  interrupts routed here (line 10)
  interrupt delivered, count = 1
  ICR = 0x80000004
```

`0x80000004` is the e1000e's cause register with the bit the test asserted and
INT_ASSERTED set, read back by userspace as its acknowledgement.

The line is **masked by the kernel on arrival and unmasked by userspace's
write()**, and that is the entire design rather than a detail. A PCI interrupt
line is level-triggered: it stays asserted until the cause is cleared *at the
device*, which is a different register on every chip and therefore something
only the driver can do. Signalling EOI on a line that is still asserted
re-enters the handler immediately and for ever -- 72,000 times a second, when
this kernel last did it by accident (history/kernel.md §13).

So a driver that dies mid-interrupt leaves the line **masked, not unmasked**.
The instinct is the other way round -- do not leave the device dead -- and it
is wrong: unmasking an asserted line with no handler attached is a crashed
driver taking the machine down with it. Verified by killing one on purpose:

```
  interrupt asserted; exiting WITHOUT acking (simulated crash)
  timer: 3431 ticks taken, 3436 expected
```

Five ticks lost in seven seconds, which is the ordinary background. The next
driver to claim the device re-arms the line, so masked is safe without being
permanent -- also verified.

## The first eviction happened

`cmd/vgpud` is the virtio-gpu driver, outside the kernel. It finds the device
by vendor and device id, asks for it, and the kernel driver hands it over:

```
vgpud: virtio-gpu at 00:03.0, currently held by virtio-gpu
virtio-gpu: released to userspace; screen is the VBE framebuffer
vgpud: claimed
vgpud: display is 1280x800
vgpud: scanout is ours, framebuffer at physical 0x3eceb000
vgpud: drew a test pattern from userspace
```

Full virtio 1.0 bring-up from a process: reset, feature negotiation, virtqueue
in DMA memory, GET_DISPLAY_INFO, RESOURCE_CREATE_2D, ATTACH_BACKING,
SET_SCANOUT, and TRANSFER/FLUSH to present. The pattern it draws is on the
virtio-gpu head while the console carries on in VBE on the other one -- which
is the fallback rule working rather than being described.

**No syscall knows this is a GPU.** The kernel handed over a PCI device; the
process decided what it was. That is the property that makes the interface
general, and it is why the evictions come before anything foreign.

### What the first real driver asked for that a test program had not

- **PCI configuration space** (`PCIIOC_CFG`). A virtio device says where its
  registers are in *vendor capabilities*, which live in config space -- so a
  driver that cannot read it cannot find its own registers. Anything modern is
  discovered the same way.
- **A way for the incumbent to let go.** `struct pci_driver` gained a
  `release` hook. A driver without one keeps its device, which is always safe
  and, for anything on the path to a shell, correct.
- **The outgoing driver freeing its DMA.** Releasing the device while keeping
  its 4 MB framebuffer is half a handover: the incoming driver asks for one and
  is told there is no room. Found by doing it.
- **A bigger arena.** 4 MB was exactly one framebuffer at this resolution, and
  for the moment either side of a handover there are two. 16 MB now.

### And one bug it reproduced immediately

The first working run raised this:

```
irq 11: 77780 interrupts/second -- a device on this line is not acknowledging it.
irq 11:   handler 'e1000' is attached
```

`vgpud` polls the used ring, so it registered no handler -- and a virtio device
interrupts on every completion. A level-triggered line nobody acknowledges
re-fires for ever, drowning the e1000 that shares it. The fix is one line,
`VRING_AVAIL_F_NO_INTERRUPT` before the queue is enabled, and the value of it
is that the storm detector built for history/kernel.md section 13 named the
line, the rate and the neighbour without anyone having to look.

**The control channel (mechanism 1) is what remains.**

### One thing the interface is deliberately not

It is about **PCI devices**, not about any driver model, and its first users
will be SchOS's own drivers. That ordering is the design: an interface whose
first consumers are native drivers is a general mechanism that something
foreign can use later, rather than a hole shaped like something foreign. The
difference matters technically -- a Linux-shaped interface would be wrong for
the evictions this plan actually calls for -- and it happens to be the only
version that survives a licensing conversation.

---

## The mechanisms

Four, and only the fourth is large.

### 1. A control channel

`AF_UNIX` plus `SCM_RIGHTS` — item 1 of [foundations.md](foundations.md), which
the Wayland work builds first. A descriptor is how a driver process is handed the
device it is allowed to touch.

**Do not build Mach ports.** The port model is worth its cost when there is a
naming and capability story to hang on it; there is not one yet, and inventing it
before the first userspace driver exists is designing against an imaginary user.

### 2. MMIO to userspace

Map a PCI BAR into a process.

`sys_mmap` already maps the framebuffer's real physical range at its real
address — the path exists. This is that path generalised, behind a check that the
caller was granted that device.

### 3. Interrupts to userspace

**The handler stays in the kernel and does one thing: write to an `eventfd`.**
The driver process reads.

`eventfd` exists already, and that is why it is the right primitive. A signal
would be delivered to a process that cannot safely be interrupted mid-transaction.

### 4. DMA to userspace

Physically contiguous memory whose physical address the process is told.

`kernel/core/dma.c` already owns an arena reserved for exactly this, before
`kalloc` gets anything. It needs a door, a per-device budget, and a way to hand
the region back.

---

## The honest limitation

> **There is no IOMMU, so a userspace driver is isolated against its own bugs and
> not against malice.**

A process that can program a device's DMA engine can write to any physical
address, whatever its page tables say.

That is not a reason to stop — isolation against bugs is most of the value, and a
crashed driver that restarts instead of panicking the machine is the entire
point. But it must be written down, because *"userspace driver"* reads as
*"sandboxed"* and here it does not mean that yet. VT-d/AMD-Vi is its own project
and comes after.

---

## Order, and one thing not to do

### Do not build loadable kernel modules first

It is the obvious first step and it is a generation behind. It means designing a
**stable in-kernel driver ABI**, which is the most expensive promise in this
entire plan, for a mechanism Apple is actively retiring in favour of the one
above.

`../architecture/devices.md` already separates the two ideas. They are separable,
and the second one is the one to build.

### First eviction: the GPU driver

Four reasons it is the right one and not just the interesting one:

- **The boundary already exists and is already correct** — `struct display_ops`.
- **It has exactly one consumer**, the compositor, so the interface has a real
  user to be judged by.
- **There is a working fallback underneath.** The VBE framebuffer stays
  in-kernel, so a GPU driver that fails to start degrades to a working screen
  rather than to no screen.
- It directly serves the work that will actually be happening.

### Second: USB

`xhci.c` is 1364 lines, self-contained, already spoken to through a bus
abstraction, and it is where third-party drivers will show up first. This is what
DriverKit was built for and the resemblance is not accidental.

### The rule that keeps the boot path honest

> **The kernel keeps a minimal driver for everything on the path to a shell, and
> userspace owns the good one.**

Break this and you get the circular boot failure where the compositor cannot
start because the display driver is a process that needs a compositor.

---

## Milestones

| | |
|---|---|
| **H0** | Mechanisms 2, 3 and 4, each with a test that exercises it from a userspace program against a device nothing else is using. |
| **H1** | A userspace virtio-gpu driver, publishing `display_ops` across the channel. Schcom talks to it and does not know the difference. |
| **H2** | Kill the driver process on purpose. The screen falls back to VBE, the driver restarts, the compositor reattaches. **This is the milestone that proves the whole idea**, and it is the one that cannot be skipped. |
| **H3** | A modeset API, which is the thing an out-of-kernel GPU driver makes affordable and an in-kernel one never got. |
| **H4** | USB, same shape. |

---

## What it will still not do

- **No isolation against a hostile driver**, per the IOMMU note above.
- **No stable driver ABI**, deliberately. Out-of-tree drivers remain a
  recompile-with-the-tree affair until there is a reason to promise otherwise.
- **The root filesystem stays in the kernel**, so SchFS cannot be restarted and a
  bug in it is still fatal. Moving it out needs an initramfs story that does not
  exist.
- **The first NIC stays in the kernel**, because the network thread and lwIP are
  in the kernel and moving them is a separate project with no current benefit.
- **No driver sandboxing beyond address-space separation** — no seccomp-shaped
  syscall restriction, no resource limits per driver.