9e6d8dd4e3
sendmsg/recvmsg with SCM_RIGHTS. Item 1 of docs/plan/foundations.md and
mechanism 1 of docs/plan/hybrid.md, which are the same primitive seen from two
directions: Wayland's wl_shm_create_pool passes a file descriptor with no
integer-shaped alternative, and handing a driver process the device it is
allowed to touch is the same act. Both plans stopped here.
Linux's structures byte for byte -- msghdr, cmsghdr, the CMSG_ macros -- because
the programs that need this are not ours. libwayland and toybox both call
recvmsg with Linux's headers and would not notice a friendlier door.
The delivery rule is the whole design. A batch of descriptors is pinned to the
stream position of the first byte of the message that carried it, and that has
two halves: a reader arriving at that position is handed them, and a reader that
has not reached it is stopped SHORT of it. Leave out the second half and a
program asking for 4096 bytes swallows three messages and is handed one
message's descriptors, with nothing in the result saying which. socktest checks
exactly that: two one-byte messages, an eight-byte read, one byte back.
The kernel holds a reference to every descriptor in flight, so a sender may
close its own copy the moment sendmsg returns. Giving that reference back is the
part that leaks if it is not written down, and there are three ways a message
ends:
- received with room installed in the receiver's table
- received without room, or by
plain read(2) closed; MSG_CTRUNC for the first case
- never received, socket closed closed by sockclose
The middle row looks like a choice and is not: holding kernel references for a
receiver that has proved it is not looking is a leak with extra steps.
No garbage collector. A socket descriptor sent over that same socket and never
received holds a reference to the socket that holds the reference to it, and
neither close reaches zero. Linux carries several hundred lines for this.
Nothing on either plan passes a socket over a socket -- Wayland passes memory,
the driver work passes devices -- so it is a bounded leak that is documented
rather than a silent one.
Fourteen checks in cmd/socktest, and every one moves data through the descriptor
rather than inspecting it. The descriptor passed is always a pipe end, because a
pipe announces its own lifetime: a read of a pipe whose writers are all gone
returns 0, which turns "was the kernel's reference released" -- otherwise
invisible from userspace -- into a value a test can compare.
Two bugs fell out, neither in the new code:
- sockpair() never set the socket's domain, so every socketpair() socket
claimed domain 0. Invisible while the data paths asked only whether a socket
was AF_INET or AF_NETLINK, since zero answers "neither" correctly.
SCM_RIGHTS asks "is this AF_UNIX", and zero is wrong for that one.
- pipe_poll() reports POLLHUP but not POLLIN at end of file, so a non-blocking
read of a pipe whose writers had all gone returned EAGAIN forever instead of
0 -- a program that polls and reads spins waiting for an end that already
happened. sockreadable() always had the equivalent condition; the pipe path
did not. Found because the test helper for "is this descriptor closed" is a
non-blocking read of a pipe.
projects/schcom/README.md said SCM_RIGHTS was done for a long time while nothing
in the kernel implemented it, and an entire GUI plan was drawn on that line.
It is true now, which is not the same as it having been right, and the README
says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
303 lines
12 KiB
Markdown
303 lines
12 KiB
Markdown
# The kernel becomes hybrid
|
|
|
|
"Hybrid" is used to mean everything from "a monolith with a nice device model" to
|
|
"a microkernel that gave up". **Drawing the line precisely is most of the work**,
|
|
so that is what this document is.
|
|
|
|
The device model it builds on is
|
|
[../architecture/devices.md](../architecture/devices.md). It is last on the
|
|
plan, and the reason is in [README.md](README.md#the-order-and-why-it-is-not-the-order-they-are-listed-in).
|
|
|
|
---
|
|
|
|
## The line
|
|
|
|
### Stays in the kernel, permanently
|
|
|
|
- scheduling, address spaces, the trap and syscall path
|
|
- IPC and shared memory — the mechanism everything else is built out of
|
|
- the VFS and the **root** filesystem
|
|
- IRQ demultiplexing
|
|
- the console and PSOGM
|
|
|
|
**That last one is not a size compromise, it is an invariant.** The crash screen
|
|
must work when everything else in the system is dead, which means it cannot
|
|
depend on anything that could *be* dead. A crash screen served by a userspace
|
|
process is a crash screen that is absent in exactly the case it exists for.
|
|
|
|
### May leave, and should
|
|
|
|
- GPU drivers
|
|
- USB, and every USB class driver under it
|
|
- non-root filesystems
|
|
- NICs after the first
|
|
- input transformation — layouts, gestures, remapping
|
|
|
|
### The rule that separates the columns
|
|
|
|
> **Anything needed to reach a shell stays in. Anything discovered at runtime,
|
|
> replaceable, or plausibly written by someone else goes out.**
|
|
|
|
That is NT's compromise and macOS's, arrived at from the same pressure, and it is
|
|
the one that survives contact with a real machine.
|
|
|
|
---
|
|
|
|
## Built so far
|
|
|
|
**Mechanisms 2 and 4 exist.** `/dev/pci` (include/pciuser.h, kernel/dev/pciuser.c)
|
|
enumerates the bus, hands a device to a process exclusively, maps a BAR into it
|
|
and allocates memory a device can be pointed at. `pcitest` proves it end to end:
|
|
|
|
```
|
|
# bdf vendor:device class driver
|
|
6 00:04.0 8086:100e network e1000
|
|
7 00:07.0 8086:10d3 network -
|
|
|
|
device 7: 8086:10d3 (network)
|
|
claimed
|
|
BAR0 at 0x0000000081060000, 128 KB
|
|
regs[0..3] = 0x00140261 0x00000000 0x00080283 0x00000000
|
|
MMIO reaches the device
|
|
DMA 8 KB at virtual 0x40001000, physical 0x3f8eb000
|
|
```
|
|
|
|
Those register values are the e1000e's CTRL and STATUS, with full-duplex and
|
|
link-up set -- which is how you tell a working mapping from one that reads
|
|
0xFFFFFFFF off an unmapped address. Claiming a device the kernel drives is
|
|
refused, and everything a process held is released when it exits, because a
|
|
driver that crashed must not keep its hardware away from its replacement.
|
|
|
|
**Mechanism 3 exists too.** A process asks for a device's interrupts and then
|
|
reads them, uio-style:
|
|
|
|
```
|
|
interrupts routed here (line 10)
|
|
interrupt delivered, count = 1
|
|
ICR = 0x80000004
|
|
```
|
|
|
|
`0x80000004` is the e1000e's cause register with the bit the test asserted and
|
|
INT_ASSERTED set, read back by userspace as its acknowledgement.
|
|
|
|
The line is **masked by the kernel on arrival and unmasked by userspace's
|
|
write()**, and that is the entire design rather than a detail. A PCI interrupt
|
|
line is level-triggered: it stays asserted until the cause is cleared *at the
|
|
device*, which is a different register on every chip and therefore something
|
|
only the driver can do. Signalling EOI on a line that is still asserted
|
|
re-enters the handler immediately and for ever -- 72,000 times a second, when
|
|
this kernel last did it by accident (history/kernel.md §13).
|
|
|
|
So a driver that dies mid-interrupt leaves the line **masked, not unmasked**.
|
|
The instinct is the other way round -- do not leave the device dead -- and it
|
|
is wrong: unmasking an asserted line with no handler attached is a crashed
|
|
driver taking the machine down with it. Verified by killing one on purpose:
|
|
|
|
```
|
|
interrupt asserted; exiting WITHOUT acking (simulated crash)
|
|
timer: 3431 ticks taken, 3436 expected
|
|
```
|
|
|
|
Five ticks lost in seven seconds, which is the ordinary background. The next
|
|
driver to claim the device re-arms the line, so masked is safe without being
|
|
permanent -- also verified.
|
|
|
|
## The first eviction happened
|
|
|
|
`cmd/vgpud` is the virtio-gpu driver, outside the kernel. It finds the device
|
|
by vendor and device id, asks for it, and the kernel driver hands it over:
|
|
|
|
```
|
|
vgpud: virtio-gpu at 00:03.0, currently held by virtio-gpu
|
|
virtio-gpu: released to userspace; screen is the VBE framebuffer
|
|
vgpud: claimed
|
|
vgpud: display is 1280x800
|
|
vgpud: scanout is ours, framebuffer at physical 0x3eceb000
|
|
vgpud: drew a test pattern from userspace
|
|
```
|
|
|
|
Full virtio 1.0 bring-up from a process: reset, feature negotiation, virtqueue
|
|
in DMA memory, GET_DISPLAY_INFO, RESOURCE_CREATE_2D, ATTACH_BACKING,
|
|
SET_SCANOUT, and TRANSFER/FLUSH to present. The pattern it draws is on the
|
|
virtio-gpu head while the console carries on in VBE on the other one -- which
|
|
is the fallback rule working rather than being described.
|
|
|
|
**No syscall knows this is a GPU.** The kernel handed over a PCI device; the
|
|
process decided what it was. That is the property that makes the interface
|
|
general, and it is why the evictions come before anything foreign.
|
|
|
|
### What the first real driver asked for that a test program had not
|
|
|
|
- **PCI configuration space** (`PCIIOC_CFG`). A virtio device says where its
|
|
registers are in *vendor capabilities*, which live in config space -- so a
|
|
driver that cannot read it cannot find its own registers. Anything modern is
|
|
discovered the same way.
|
|
- **A way for the incumbent to let go.** `struct pci_driver` gained a
|
|
`release` hook. A driver without one keeps its device, which is always safe
|
|
and, for anything on the path to a shell, correct.
|
|
- **The outgoing driver freeing its DMA.** Releasing the device while keeping
|
|
its 4 MB framebuffer is half a handover: the incoming driver asks for one and
|
|
is told there is no room. Found by doing it.
|
|
- **A bigger arena.** 4 MB was exactly one framebuffer at this resolution, and
|
|
for the moment either side of a handover there are two. 16 MB now.
|
|
|
|
### And one bug it reproduced immediately
|
|
|
|
The first working run raised this:
|
|
|
|
```
|
|
irq 11: 77780 interrupts/second -- a device on this line is not acknowledging it.
|
|
irq 11: handler 'e1000' is attached
|
|
```
|
|
|
|
`vgpud` polls the used ring, so it registered no handler -- and a virtio device
|
|
interrupts on every completion. A level-triggered line nobody acknowledges
|
|
re-fires for ever, drowning the e1000 that shares it. The fix is one line,
|
|
`VRING_AVAIL_F_NO_INTERRUPT` before the queue is enabled, and the value of it
|
|
is that the storm detector built for history/kernel.md section 13 named the
|
|
line, the rate and the neighbour without anyone having to look.
|
|
|
|
**All four mechanisms now exist.** What remains on this plan is the second
|
|
eviction and then the foreign driver host.
|
|
|
|
### One thing the interface is deliberately not
|
|
|
|
It is about **PCI devices**, not about any driver model, and its first users
|
|
will be SchOS's own drivers. That ordering is the design: an interface whose
|
|
first consumers are native drivers is a general mechanism that something
|
|
foreign can use later, rather than a hole shaped like something foreign. The
|
|
difference matters technically -- a Linux-shaped interface would be wrong for
|
|
the evictions this plan actually calls for -- and it happens to be the only
|
|
version that survives a licensing conversation.
|
|
|
|
---
|
|
|
|
## The mechanisms
|
|
|
|
Four, and only the fourth is large.
|
|
|
|
### 1. A control channel — **built**
|
|
|
|
`AF_UNIX` plus `SCM_RIGHTS` — item 1 of [foundations.md](foundations.md), which
|
|
the Wayland work also needed first. A descriptor is how a driver process is
|
|
handed the device it is allowed to touch.
|
|
|
|
Built as `sendmsg`/`recvmsg` in Linux's shape, because the programs that will
|
|
use it are not ours: libwayland and toybox both call `recvmsg` with Linux's
|
|
structures and would not notice a friendlier door. Descriptors are pinned to the
|
|
stream position of the first byte of their message, and the kernel holds a
|
|
reference to each until it is received — or closes it when the socket is dropped
|
|
with the message still queued, which is the case that leaks if nobody writes it
|
|
down. See
|
|
[../architecture/networking.md](../architecture/networking.md#descriptor-passing).
|
|
|
|
**Do not build Mach ports.** The port model is worth its cost when there is a
|
|
naming and capability story to hang on it; there is not one yet, and inventing it
|
|
before the first userspace driver exists is designing against an imaginary user.
|
|
|
|
### 2. MMIO to userspace
|
|
|
|
Map a PCI BAR into a process.
|
|
|
|
`sys_mmap` already maps the framebuffer's real physical range at its real
|
|
address — the path exists. This is that path generalised, behind a check that the
|
|
caller was granted that device.
|
|
|
|
### 3. Interrupts to userspace
|
|
|
|
**The handler stays in the kernel and does one thing: write to an `eventfd`.**
|
|
The driver process reads.
|
|
|
|
`eventfd` exists already, and that is why it is the right primitive. A signal
|
|
would be delivered to a process that cannot safely be interrupted mid-transaction.
|
|
|
|
### 4. DMA to userspace
|
|
|
|
Physically contiguous memory whose physical address the process is told.
|
|
|
|
`kernel/core/dma.c` already owns an arena reserved for exactly this, before
|
|
`kalloc` gets anything. It needs a door, a per-device budget, and a way to hand
|
|
the region back.
|
|
|
|
---
|
|
|
|
## The honest limitation
|
|
|
|
> **There is no IOMMU, so a userspace driver is isolated against its own bugs and
|
|
> not against malice.**
|
|
|
|
A process that can program a device's DMA engine can write to any physical
|
|
address, whatever its page tables say.
|
|
|
|
That is not a reason to stop — isolation against bugs is most of the value, and a
|
|
crashed driver that restarts instead of panicking the machine is the entire
|
|
point. But it must be written down, because *"userspace driver"* reads as
|
|
*"sandboxed"* and here it does not mean that yet. VT-d/AMD-Vi is its own project
|
|
and comes after.
|
|
|
|
---
|
|
|
|
## Order, and one thing not to do
|
|
|
|
### Do not build loadable kernel modules first
|
|
|
|
It is the obvious first step and it is a generation behind. It means designing a
|
|
**stable in-kernel driver ABI**, which is the most expensive promise in this
|
|
entire plan, for a mechanism Apple is actively retiring in favour of the one
|
|
above.
|
|
|
|
`../architecture/devices.md` already separates the two ideas. They are separable,
|
|
and the second one is the one to build.
|
|
|
|
### First eviction: the GPU driver
|
|
|
|
Four reasons it is the right one and not just the interesting one:
|
|
|
|
- **The boundary already exists and is already correct** — `struct display_ops`.
|
|
- **It has exactly one consumer**, the compositor, so the interface has a real
|
|
user to be judged by.
|
|
- **There is a working fallback underneath.** The VBE framebuffer stays
|
|
in-kernel, so a GPU driver that fails to start degrades to a working screen
|
|
rather than to no screen.
|
|
- It directly serves the work that will actually be happening.
|
|
|
|
### Second: USB
|
|
|
|
`xhci.c` is 1364 lines, self-contained, already spoken to through a bus
|
|
abstraction, and it is where third-party drivers will show up first. This is what
|
|
DriverKit was built for and the resemblance is not accidental.
|
|
|
|
### The rule that keeps the boot path honest
|
|
|
|
> **The kernel keeps a minimal driver for everything on the path to a shell, and
|
|
> userspace owns the good one.**
|
|
|
|
Break this and you get the circular boot failure where the compositor cannot
|
|
start because the display driver is a process that needs a compositor.
|
|
|
|
---
|
|
|
|
## Milestones
|
|
|
|
| | |
|
|
|---|---|
|
|
| **H0** | Mechanisms 2, 3 and 4, each with a test that exercises it from a userspace program against a device nothing else is using. |
|
|
| **H1** | A userspace virtio-gpu driver, publishing `display_ops` across the channel. Schcom talks to it and does not know the difference. |
|
|
| **H2** | Kill the driver process on purpose. The screen falls back to VBE, the driver restarts, the compositor reattaches. **This is the milestone that proves the whole idea**, and it is the one that cannot be skipped. |
|
|
| **H3** | A modeset API, which is the thing an out-of-kernel GPU driver makes affordable and an in-kernel one never got. |
|
|
| **H4** | USB, same shape. |
|
|
|
|
---
|
|
|
|
## What it will still not do
|
|
|
|
- **No isolation against a hostile driver**, per the IOMMU note above.
|
|
- **No stable driver ABI**, deliberately. Out-of-tree drivers remain a
|
|
recompile-with-the-tree affair until there is a reason to promise otherwise.
|
|
- **The root filesystem stays in the kernel**, so SchFS cannot be restarted and a
|
|
bug in it is still fatal. Moving it out needs an initramfs story that does not
|
|
exist.
|
|
- **The first NIC stays in the kernel**, because the network thread and lwIP are
|
|
in the kernel and moving them is a separate project with no current benefit.
|
|
- **No driver sandboxing beyond address-space separation** — no seccomp-shaped
|
|
syscall restriction, no resource limits per driver.
|