SchOS/docs/architecture/devices.md

# The device model

How hardware gets found, claimed and used, and what the rest of the kernel sees
of it.

Writing a driver is [../guides/adding-a-driver.md](../guides/adding-a-driver.md).
Drivers that live in a process rather than in the kernel are described below
(*Drivers that are not in the kernel*), and the foreign ones in [ldls.md](ldls.md).

---

## What stays in the kernel, and what may leave

"Hybrid" is used to mean everything from "a monolith with a nice device model" to
"a microkernel that gave up", so the line is worth drawing exactly.

**Stays in, permanently:** scheduling, address spaces, the trap and syscall path;
IPC and shared memory; the VFS and the **root** filesystem; IRQ demultiplexing; and
the console and PSOGM. That last one is not a size compromise but an invariant: the
crash screen must work when everything else is dead, so it cannot depend on anything
that could *be* dead. A crash screen served by a userspace process is absent in
exactly the case it exists for.

**May leave, and should:** GPU drivers, non-root filesystems, NICs after the first,
and anything written by someone else.

> **Anything needed to reach a shell stays in. Anything discovered at runtime,
> replaceable, or plausibly written by someone else goes out.**
>
> **The kernel keeps a minimal driver for everything on the path to a shell, and
> userspace owns the good one.** Break that and the compositor cannot start because
> the display driver is a process that needs a compositor.

### USB stays in the kernel, and why

It was on the list to leave, and the premise was wrong. USB is not a peripheral; it
is the bus peripherals arrive on, structurally the same kind of thing as PCI, and
nobody has proposed evicting PCI. The keyboard and the root disk that arrive over it
are on the path to a shell, so the stack that finds them is part of that path. Even
with a minimal in-kernel fallback written, the good driver and the minimal one would
be the same driver, and an eviction that leaves an identical copy behind has moved
nothing. A second, quieter difference: `/dev/pci` hands over one PCI *function*, one
owner and nothing shared, while a USB device sits behind a controller other devices
also sit behind — "the xHCI controller" includes the keyboard. What the eviction was
for, third-party drivers living outside the kernel, survives and belongs to
[ldls.md](ldls.md).

The root filesystem stays in the kernel for the same reason (SchFS cannot be
restarted, and a bug in it is fatal), and so does the first NIC, because the network
thread and lwIP are in the kernel.

---

## One list of everything attached

There is one list of everything attached, whatever bus found it:
`include/dev/device.h` and `kernel/dev/device.c`, readable at `/proc/devices`.

```
name      bus       class     driver    device
kbd0      platform  input     ps2kbd    PS/2 keyboard
mouse0    platform  input     ps2mouse  PS/2 mouse
uart0     platform  other     uart8250  16550 serial port
ide0      platform  storage   ide       ATA disk controller
vga0      pci       display   vga       display controller
```

Before this there were two registries in `kernel/drivers/registry.c` — an array
of PCI drivers and an array of USB drivers — with nothing that knew about both.
That is sufficient for *binding* and useless for the three things wanted next.

**Enumeration.** "What is in this machine, and what claimed it" had no answer
covering a USB disk and a PCI display at once. A device manager is exactly that
question.

**Classes.** A compositor should ask for *the display* and not care whether that
is the VBE framebuffer or virtio-gpu. Without a class every consumer hardcodes a
driver, and adding the second driver means editing the consumers — which is how
a framebuffer address ended up hardcoded in three places here once already.

**A boundary for drivers that are not compiled in.** They have to attach
somewhere that is not an array fixed at link time.

Two vocabularies are kept deliberately separate: a device's **bus** is
provenance (`platform`, `pci`, `usb`), its **class** is capability
(`display`, `input`, `storage`, `net`, `bus`). A USB disk and a SATA disk are
both `storage` despite sharing no numbering at all. Where a driver claims a
device, the driver's opinion wins over the descriptor's: `struct usb_driver` has
a `devclass` field separate from the USB interface class, because the descriptor
says what the device *claims* to be and the driver knows what it *made* of it.

It is a flat list with a bus tag, **not** a device tree. Nothing yet needs to
walk a hierarchy, and a tree nobody traverses is just a harder list. Hot-unplug
would need more; the `in_use` flag is the hook.

Platform devices are registered in one place
(`platform_register_devices()` in `registry.c`) rather than inside each driver's
init, for the same reason the driver arrays are written out by hand: a list you
can read beats a list assembled by side effects.

---

## Drivers that are not in the kernel

`/dev/pci` (major 9) hands a PCI device to a process. It is deliberately about
*PCI devices* and not about any driver model, and its first users are SchOS's
own drivers, evicted one at a time — which is the ordering that keeps it a
general mechanism rather than a hole shaped like something foreign.

What a driver needs from a kernel, and all it needs:

| | |
|---|---|
| find its device, and read what the bus says | `PCIIOC_INFO`, `PCIIOC_CFG` |
| be given it, exclusively | `PCIIOC_CLAIM` |
| reach its registers | `mmap`, where the offset is a BAR's physical address |
| memory it can hand to the device | `PCIIOC_DMA` |
| be told when the device wants attention | `PCIIOC_IRQ`, then `read()` |
| use its ports, if it has an I/O BAR | nothing — `CLAIM` grants them |

Four of those are worth expanding on, because each encodes a decision:

**`mmap` is checked against the caller's claims.** An offset that is not inside
a BAR of a device this process holds is refused. That is the whole difference
between this and `/dev/mem` with extra steps.

The check is against the *pages* a BAR lies in rather than the BAR itself,
because a page is the only granularity an MMU has. A BAR smaller than a page
therefore hands the process the whole page it sits in — forced by the hardware
rather than chosen. Comparing against the BAR's own size instead meant a
sub-page BAR could not be mapped at all, which nothing noticed until an
rtl8139 with 256 bytes of registers turned up; see
[ldls.md](ldls.md), "The bug a foreign driver found in its
first minute".

**DMA is a call, not a mapping.** The process is told the *physical* address,
which is precisely what it cannot work out for itself, and the memory comes from
an arena reserved at boot before the general allocator gets anything.

Everything in that arena has to be **contiguous**, which is the whole reason it
exists — so the number that matters is not how much is free but how large the
biggest piece is. `dma_alloc()` is therefore best fit rather than first fit: a
64 KB ring taken from the middle of a 16 MB arena leaves two 8 MB extents, and
a framebuffer wanting 9 MB then cannot be had on an arena three quarters empty.
Best fit puts each request in the tightest extent that holds it and leaves the
large ones whole for the requests with no alternative.

**One request is capped at 32 MB**, and the cap is not there to ration memory —
the arena does that, and says so properly. It is there so a wild `size` from a
process cannot ask for the whole arena by accident. It was 4 MB, which is one
framebuffer at 1280x800, chosen when that was the only thing anyone asked for.

That is worth recording because it produced a failure that pointed somewhere
else entirely. virtio-gpu's display size is the size of the *host's window*, so
a user who drags it out to 1920x1080 asks for 8100 KB in one piece — refused by
the cap, on an arena with 61 MB free:

```
vgpud: wanted 7365 KB of DMA memory for a 1920x982 framebuffer
...
65536 KB total, 61847 KB free in 2 extent(s)
```

Every number in the diagnostic was about the arena, and the arena was fine.

A failed allocation reports both numbers for the same reason, because they
point at different fixes — a fragmented arena is cured by whoever holds the
block in the middle letting go, a full one by `DMA_ARENA_BYTES`:

```
dma_alloc: no room for 9216000 bytes -- 12591104 free,
           largest contiguous piece 8495104 (`schdbg dma` for the map)
```

**Interrupts arrive by `read()`**, the way Linux's uio does it and for the same
reason: a descriptor composes with `poll`, and a driver that stops reading falls
behind rather than losing events. The kernel masks the line when the interrupt
arrives and leaves it masked until the driver `write()`s back. **That write is
not optional.** A PCI interrupt is level-triggered — it stays asserted until the
cause is cleared *at the device*, which is a different register on every chip and
therefore something only the driver can do. Signal EOI on a line that is still
asserted and it re-fires immediately, for ever. That is not hypothetical: it
happened here at 72,000 interrupts a second, and what it looked like from
userspace was a clock running seven times slow. See §13 of
[../history/kernel.md](../history/kernel.md).

**Ports come with the claim, and only that device's.** There is no ioctl for
this because there is no decision left to make: a process holding a device with
an I/O BAR gets exactly the ports in that BAR, through the TSS I/O permission
bitmap, and loses them when it releases or exits. Granting it is not optional —
a driver's `outl` is an instruction compiled into it, not a call anything can
intercept — and granting it *coarsely* is not acceptable, because `IOPL=3` is
one flag that hands over the interrupt controller and the disk along with the
NIC. See [../../include/ioperm.h](../../include/ioperm.h).

### Handing a device over

`struct pci_driver` has a `release` hook. A driver that does not define one keeps
its device — always safe, and for anything on the path to a shell also correct.
A driver that does define one has to give back everything, not just the device:
the first eviction found this the hard way, because releasing the GPU while
keeping its 4 MB framebuffer meant the incoming driver was told there was no DMA
memory left.

`cmd/vgpud` is the virtio-gpu driver as an ordinary process. It does the whole
virtio 1.0 bring-up from outside — reset, feature negotiation, a virtqueue in DMA
memory, `SET_SCANOUT`, transfer and flush — and while it owns the GPU the kernel
console falls back to the VBE framebuffer.

**No syscall on that path knows it is talking about a GPU.** The kernel hands
over a PCI device; the process decides what it is.

Two things a release has to do beyond quiescing the hardware, both found by
handing over a NIC the kernel was actively using:

- **Mask the line.** The kernel's handler is gone and the incoming driver is
  about to enable the device, so for a while nothing will acknowledge the
  cause. Left unmasked that is an interrupt storm — 450,000 a second, measured.
- **Give up the class device too.** A NIC's `release` calls `net_detach()`, the
  missing half of `net_attach_late()`. An interface left in the stack keeps its
  address and the route to its subnet, so the stack goes on choosing it and
  every frame reaches a `transmit` that now correctly refuses. Nothing reports
  an error. See [ldls.md](ldls.md).

It also **changes the display mode**, on a control socket, with `schmode` as the
client — and that is worth a sentence here because it is the thing the device
model makes possible rather than the thing it does. A modeset is five commands
to the device. In the kernel those five commands are also a reason to be
nervous, because a wrong one takes the machine; in a process the worst case is
that `schinit` starts another one. That is why the in-kernel driver never grew
one — not difficulty, exposure.

```
# schmode
1280x800 current
1280x800 maximum, framebuffer 4000 KB
# schmode 1024 768
1024x768 ok
```

The scanout really resizes, and nothing in the kernel changed for it: `schmode`
writes a line to a socket and `vgpud` sends five commands to the device. The
framebuffer is allocated **once, at the largest mode**, and every smaller mode
reuses it, because `PCIIOC_DMA` has no free — a driver's blocks come back when it
exits and not before — so a modeset that allocated would spend the arena and the
failure would look like a GPU bug rather than like counting. A larger mode is
refused, and says why. The control socket speaks lines of text and has no buffer
submission, damage or vblank, which a compositor would need and nothing has asked
for.

### A panic has to reach the screen

Handing a device away costs something that was not on the list. The kernel panics,
paints the report into the console framebuffer, and halts — and the display it gave
to a process keeps showing whatever that process last put there. On a machine with
two heads, one says the kernel is dead and the other shows a shell prompt. There is
no scheduling after a panic, so the process holding the device is never going to run
again, and the kernel let go of the device deliberately.

So the driver says in advance where it is scanning out, with `PCIIOC_PANICFB`
(`include/dev/pciuser.h`), and the kernel: (1) paints the report into that memory;
(2) asks the *kernel* driver the device was taken from to put it on screen —
`panic_present` in `struct pci_driver`; (3) which resets the device and rebuilds one
queue from **static** memory — no allocation, no locks, no sleeping, because a
corrupted heap is one of the ways a machine gets here. The registration is checked
like the promise it is: the range must lie inside a block that process got from
`PCIIOC_DMA` on that device, or it would be a way to nominate any physical address
to be overwritten at panic. It costs 20 KB of static memory in the virtio-gpu driver,
paid whether or not the machine ever panics; allocating it on demand would be
cheaper and would not work.

### Taking it back

A driver that dies is the case this whole arrangement exists for, so release is a
symmetric operation rather than a courtesy. Three things happen that are not
obvious from "the process exited":

**Bus mastering is cleared before the DMA is freed.** A device whose driver has
just died is still writing into memory the kernel is about to hand to somebody
else. The kernel cannot quiesce a device it does not understand — that is a
different register on every chip — but it can clear `PCI_CMD_BUS_MASTER`, which
stops the writes whatever the device is, and it is the only halt available to a
kernel that knows nothing but the header. So `PCIIOC_CLAIM` sets the bit and
release clears it, in that order relative to freeing the memory. `pcitest` prints
the bit in a `bm` column, with a separate mark for "could not read it", because a
failed read rendered as "off" is worse than no column at all.

**DMA memory a driver mapped is not `freevm()`'s to release.** `freevm()` frees
every mapped page below `PHYSTOP` and the DMA arena is the top 16 MB of exactly
that range, so a DMA block left for it went onto the page allocator's free list
while `pciuser_release_all()` handed the same pages back to the arena — two
allocators, one set of pages. `PCIIOC_DMA` records a VMA with `no_free` set, the
same way shared memory does, and `proc_release_shared()` unmaps the range before
the page tables are torn down. The same rule covers every future case: **anything
mapped into a process that the process does not own the pages of needs a VMA, or
exit will free it twice.**

**A device that is the only display will not be released.** `vgpu_release()` asks
`vga_fb_restore()` whether there is a framebuffer to put the console back on, and
declines if the answer is no. The rule that the kernel keeps a minimal driver for
everything on the path to a shell is only worth writing down if something
enforces it.

Coming back is a job, not a kernel feature. `etc/schinit/vgpud.job` carries
`KeepAlive = yes`, so killing the driver is the entire test: `schinitd` starts
another one, it claims the device, and the pid of whichever process drew the
frame is legible in the frame itself: read off the scanout across four generations of
the driver it said 4, 8, 10, 12, and the DMA arena handed back the same addresses
every time.

### What this is not

**It is not a security boundary.** There is no IOMMU, so a process holding a DMA
buffer can point the device at any physical address. This isolates a driver
against its own *bugs* and not against its own *intent* — a crashed driver that
restarts instead of taking the machine down is the entire benefit, and it is
worth having on those terms. VT-d/AMD-Vi is its own project.

**One device per process.** A driver drives one thing; if you need two, run two.

---

## Linux drivers

[ldls.md](ldls.md) is how SchOS uses them: a Linux `.ko` runs in a process and
reaches the hardware through `/dev/pci`. The licensing position, stated exactly
because it is easy to get wrong in both directions:

- **Distribution is what triggers the GPL.** SchOS is GPL-2.0-or-later, so shipping a
  GPL-2.0-only driver in an image is *lawful* — see [../licensing.md](../licensing.md) §6 —
  and the owner has decided it is allowed (code and firmware blobs). The repository does
  not carry one all the same: `.gitignore` refuses `*.ko`, and the modules are staged into
  a *local* image from the build machine's own Linux kernel (`make ldls-stage`,
  `make ldls-r8169`), where the matching BTF is.
- **Implementing a loader is not copying.** Writing support for a module format and
  a driver ABI from published documentation produces our own work; the layouts come
  from the kernel's BTF at load time on the user's machine, so the source holds none
  of Linux's.

The loader is ours and GPL-2.0-or-later; what it loads is not in the tree.
`RAMDISK_EXCLUDE` and `NOTPORTS` are the mechanisms that keep that true mechanically
rather than by intention.

---

## Interrupts

A PCI driver connects its interrupt with one call, `pci_irq_attach()`, and
does not choose how. In order of preference:

1. **MSI**: the device writes a vector to the local APIC. No pin, no routing,
   nothing to share.
2. **MSI-X**, but only if the driver passes `PCI_IRQF_MSIX`. Some parts change
   how they report interrupt causes in that mode; the 82574 needs its IVAR
   programmed, which `e1000` does not do, so it does not ask.
3. **INTx**, on the pin ACPI's `_PRT` names for it.

`INTERRUPT_LINE` holds the number the firmware gave the 8259. In APIC mode the
device is wired to an IOAPIC input that only `_PRT` describes, and reading
`_PRT` takes an AML interpreter. That is uACPI (`ports/uacpi`, host side in
`kernel/acpi/`): `pci_init()` asks it for every device's pin between scanning
the bus and binding drivers, following link devices' `_CRS` and swizzling
through PCI-to-PCI bridges that have no `_PRT` of their own. The trigger mode
comes from there too -- q35's links are level/active-high, which
`irq_register()` could not express until it took a mode rather than a flag.

MSI still comes first: it needs no routing at all, and a machine whose AML
does not load still has it. INTx is also backed by the poller in
`kernel/dev/irq.c`:

- A level-triggered line that has had a handler for two seconds and has never
  fired gets its handlers called every tick, from a kernel thread with
  interrupts off.
- A line that **storms** (an idle ISA input configured active-low reads as
  asserted forever) is masked and polled instead of being reported and left
  running.

Every handler already has to cope with being called when its device is idle,
because lines are shared, so a spurious call costs only a register read.
`/proc/interrupts` marks such a line `[polled]`, and an MSI line `msi`.

The vector layout:

| Lines | What they are |
|---|---|
| 0–23 | IOAPIC pins |
| 24–29 | MSI vectors, allocated by `irq_alloc_msi()` |
| 30 | LAPIC error (it was 19, which is a PCI pin on Intel chipsets) |
| 31 | spurious |

Nothing goes as high as vector 64, which is `T_SYSCALL`.

Tested under QEMU q35 with the root filesystem on a USB stick, in three
configurations:

- xHCI and e1000e on MSI.
- MSI disabled and the INTx line deliberately misrouted. xHCI took no
  interrupts at all, the poller carried every transfer, and the root still
  mounted.
- The old INTx path, unchanged on i440fx.

And since `_PRT`: on q35 with MSI disabled, xHCI routed to GSI 22 level/high
and taking real interrupts, with nothing polled.

### Power

uACPI also handles the SCI, the power button and S5. The power button syncs
the filesystems and powers off, from the `acpi` kernel thread rather than the
interrupt; `shutdown()` and `reboot()` go through the FADT's PM1 and reset
registers first, and fall back to the emulators' fixed ports only if the AML
did not load.

---

## `/dev/random` and `/dev/urandom`

Seeded once at boot, from `RDRAND` where the CPU has it and from the cycle
counter mixed with the wall clock where it does not. A verbose boot says which:

```
random: seeded from RDRAND
```

That is worth a section only because of what it replaced. It was a linear
congruential generator seeded with a **constant**, so every boot produced the
same bytes in the same order — not weak, *the same*. A program asking for a
key, a nonce, a temporary filename or a hash seed got one it shared with every
other SchOS machine that had ever run.

**It is still not a CSPRNG and does not pretend to be.** It is an LCG, re-stirred
from the cycle counter on each byte so two readers do not continue one
sequence. What changed is the property that made it dangerous rather than
merely poor.

**For a long time it could not be read at all**, and neither could `/dev/zero`.
Both are minors of major 1, and `devsw[1]` is `memread`/`memwrite`, which hand
the console's minors to the console and the rest to their own functions. But
`consoleinit()` installed `consoleread` over the whole major, so a read of
`/dev/zero` waited for a line from the keyboard. `head -c 10 /dev/zero`
"worked" by taking the next command typed. The console now installs only its
`open` and `close` hooks.

## What it does not do

- **It is a flat list, not a device tree.** There is no parent/child nesting,
  only a bus tag. Nothing yet needs to walk a hierarchy, and a tree nobody
  traverses is just a harder list.
- **No hot-plug.** Devices are registered at boot and never removed; the
  `in_use` flag is the hook a hot-unplug path would need, and nothing sets it
  for that purpose.
- **`NDEVICE` is 128**, compile-time, like every other table in the kernel.
- **In-kernel drivers are fixed at link time.** There is no module loader, and
  nothing can be added to the kernel after the build. A driver *outside* the
  kernel is an ordinary program and can be started whenever — it reaches its
  device through `/dev/pci`, and a NIC driver can publish itself to the network
  class through `/dev/netdev` (LDLS does); a display driver publishes nothing to
  `display_ops`, so the console cannot be handed to one.
- **The kernel does not rebind its own driver.** `PCIIOC_RELEASE`, or the process
  exiting, leaves the interrupt line masked and the in-kernel driver unbound —
  the console stays wherever the release put it. A *supervisor* can start a
  replacement driver, and `schinitd` does; the kernel itself watches for
  nothing.
- **Only two class vtables are defined**: `display_ops` and `netdev_ops`.
  Storage and input devices appear in the list with `ops = 0` and are reached
  through their own subsystems instead.