hybrid: the driver is killed, and the driver comes back

H2 of docs/plan/hybrid.md -- the milestone the plan itself calls the one
that proves the whole idea, and the one that cannot be skipped.

    # cat /proc/schinit
    1 4 11 os.schos.vgpud
    # kill 4
    schinitd: restarting os.schos.vgpud
    vgpud: claimed
    vgpud: pid 8 drew a test pattern from userspace

The test pattern ends in a twelve-block tally of the drawing process's
pid, because a second run that redraws an identical picture is
indistinguishable from a first run that never stopped. Read off the
virtio-gpu scanout across four generations it says 4, 8, 10, 12, and the
DMA arena hands back the same physical address every time.

What it took:

* A halt the kernel can apply to a device it does not understand.
  Clearing PCI_CMD_BUS_MASTER is the only one there is, and it has to
  happen before the DMA goes back to the arena -- a dead driver's device
  keeps reading a framebuffer the kernel is about to give to somebody
  else. Claim sets the bit, release clears it; pcitest shows it in a bm
  column, with a third state so a failed read is never printed as off.

* A refusal. vgpu_release() asks vga_fb_restore() whether there is a
  framebuffer to put the console back on, and declines to hand over the
  device if there is not. The rule that the kernel keeps a minimal
  driver for everything on the path to a shell is worth writing down
  only if something enforces it.

* The DMA arena not being owned by two allocators at once. This is the
  bug that stopped H2: freevm() frees every mapped page below PHYSTOP
  and the arena is the top 16 MB of exactly that range, so a DMA block
  mapped into a driver went onto the page allocator's free list when the
  driver died, while pciuser_release_all() handed the same pages back to
  the arena. The restarted vgpud asked for its framebuffer, got pages
  kalloc() had already given out -- including its own page tables -- and
  dma_alloc()'s zeroing wiped them, so it died inside the ioctl on the
  return from a syscall whose stack had stopped existing. do_dma() now
  records a VMA with no_free set, the way shared memory does.

* One exit status. KeepAlive must restart a driver that crashed and must
  not restart one that has looked at the machine and found its device
  absent: on hardware with no virtio-gpu that was fifteen spawns in
  forty-five seconds, each printing a line to the console, for ever.
  SCHINIT_EXIT_NORESTART is 78, sysexits.h's EX_CONFIG. It is not a new
  manifest key -- the wait status has always travelled from PID 1 to the
  supervisor, and note_exit() simply stopped discarding it.

The milestone's third clause, "the compositor reattaches", is not done
and is not claimed: there is no compositor running. That is recorded in
the plan as a Schcom milestone rather than counted here.

Closes the register row "A released PCI device" in docs/plan/1.0.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2

docs/architecture/devices.md

@@ -113,6 +113,44 @@ console falls back to the VBE framebuffer.
 **No syscall on that path knows it is talking about a GPU.** The kernel hands
 over a PCI device; the process decides what it is.
 
+### Taking it back
+
+A driver that dies is the case this whole arrangement exists for, so release is a
+symmetric operation rather than a courtesy. Three things happen that are not
+obvious from "the process exited":
+
+**Bus mastering is cleared before the DMA is freed.** A device whose driver has
+just died is still writing into memory the kernel is about to hand to somebody
+else. The kernel cannot quiesce a device it does not understand — that is a
+different register on every chip — but it can clear `PCI_CMD_BUS_MASTER`, which
+stops the writes whatever the device is, and it is the only halt available to a
+kernel that knows nothing but the header. So `PCIIOC_CLAIM` sets the bit and
+release clears it, in that order relative to freeing the memory. `pcitest` prints
+the bit in a `bm` column, with a separate mark for "could not read it", because a
+failed read rendered as "off" is worse than no column at all.
+
+**DMA memory a driver mapped is not `freevm()`'s to release.** `freevm()` frees
+every mapped page below `PHYSTOP` and the DMA arena is the top 16 MB of exactly
+that range, so a DMA block left for it went onto the page allocator's free list
+while `pciuser_release_all()` handed the same pages back to the arena — two
+allocators, one set of pages. `PCIIOC_DMA` records a VMA with `no_free` set, the
+same way shared memory does, and `proc_release_shared()` unmaps the range before
+the page tables are torn down. The same rule covers every future case: **anything
+mapped into a process that the process does not own the pages of needs a VMA, or
+exit will free it twice.**
+
+**A device that is the only display will not be released.** `vgpu_release()` asks
+`vga_fb_restore()` whether there is a framebuffer to put the console back on, and
+declines if the answer is no. The rule that the kernel keeps a minimal driver for
+everything on the path to a shell is only worth writing down if something
+enforces it.
+
+Coming back is a job, not a kernel feature. `etc/schinit/vgpud.job` carries
+`KeepAlive = yes`, so killing the driver is the entire test: `schinitd` starts
+another one, it claims the device, and the pid of whichever process drew the
+frame is legible in the frame itself. See
+[../plan/hybrid.md](../plan/hybrid.md), "The driver came back".
+
 ### What this is not
 
 **It is not a security boundary.** There is no IOMMU, so a process holding a DMA
@@ -170,9 +208,11 @@ existing mechanisms for keeping that true mechanically rather than by intention.
   kernel is an ordinary program and can be started whenever — but it reaches its
   device through `/dev/pci` and publishes nothing to the class vtables, so the
   console cannot be handed to one yet.
-- **A released device does not come back by itself.** `PCIIOC_RELEASE`, or the
-  process exiting, leaves the interrupt line masked and the in-kernel driver
-  unbound. Something has to start a replacement; nothing watches for one.
+- **The kernel does not rebind its own driver.** `PCIIOC_RELEASE`, or the process
+  exiting, leaves the interrupt line masked and the in-kernel driver unbound —
+  the console stays wherever the release put it. A *supervisor* can start a
+  replacement driver, and `schinitd` does; the kernel itself watches for
+  nothing.
 - **Only two class vtables are defined**: `display_ops` and `netdev_ops`.
   Storage and input devices appear in the list with `ops = 0` and are reached
   through their own subsystems instead.

docs/architecture/init.md

@@ -229,7 +229,7 @@ and this is not a language).
 | `Program` | The binary to execute |
 | `Args` | argv, space separated. argv[0] is conventionally the basename, so `Args = sh` for `/bin/sh` |
 | `RunAtLoad` | Start it when schinit loads |
-| `KeepAlive` | Restart it when it exits |
+| `KeepAlive` | Restart it when it exits — unless it exits `SCHINIT_EXIT_NORESTART`, below |
 | `Disabled` | Load it but do not start it |
 | `ThrottleInterval` | Seconds between respawns (default 1) |
 | `ControllingTerminal` | Claim the console |
@@ -272,6 +272,31 @@ subject to `ThrottleInterval` — a job that exits immediately and forever would
 otherwise spin the machine. launchd defaults to 10 seconds; 1 is enough here,
 where the common case is a shell someone just exited on purpose and expects back.
 
+### One exit status the supervisor reads
+
+Throttling stops a restart loop from costing CPU. It does not stop it being a
+loop, and a job that can never succeed on this machine will print its reason
+every `ThrottleInterval` for the rest of the uptime — measured at fifteen spawns
+in forty-five seconds for `vgpud` on a machine with no virtio-gpu.
+
+So a `KeepAlive` job that exits **`SCHINIT_EXIT_NORESTART`** (78, sysexits.h's
+`EX_CONFIG`) is not restarted, and the supervisor says which job stopped and
+why:
+
+```
+vgpud: no virtio-gpu on this machine
+schinitd: os.schos.vgpud has nothing to do on this machine; not restarting
+```
+
+The distinction it draws is the right one: the status is wait(2)-encoded, so a
+job that was **killed or faulted** has a signal in the low bits, never matches,
+and comes back exactly as `KeepAlive` promises. Only a job that exited
+deliberately, saying this machine is not one it can run on, is believed.
+
+This is not a new manifest key and there is no policy knob. The wait status has
+always travelled from PID 1 to the supervisor inside `struct schinit_reply`;
+`note_exit()` simply stopped discarding it.
+
 ## The fallback
 
 With no manifests found, the supervisor says so and starts `/bin/sh` on the

docs/plan/1.0.md

@@ -94,9 +94,14 @@ The galling ones. The work exists, is tested, and nothing uses it.
 | | Where it stopped |
 |---|---|
 | **`foundations.md`** | Item 1 built. Items 2 (a shareable memory object), 3 (`AF_UNIX` in the filesystem namespace), 4 and 5 open. **Wayland stops at item 2.** |
-| **`hybrid.md`** | All four mechanisms built, H1 done. **H2 — kill the driver on purpose and watch it come back — is the milestone the plan itself calls the one that proves the whole idea**, and it is not done. |
+| **`hybrid.md`** | All four mechanisms built. H1 and H2 done — the driver can be killed and comes back, which is the milestone the plan calls the one that proves the whole idea. H3 (a modeset API) and H4 (USB) open. |
 | **Schcom v0** | 715 lines that composite, over a wire format that is superseded. Frozen: it runs, nothing more is added, and it is kept only because it is the one thing in the tree that has ever put a client's pixels on screen. |
-| **A released PCI device** | `PCIIOC_RELEASE` leaves the line masked and the in-kernel driver unbound. Nothing starts a replacement. Half a handover. |
+
+**Closed since this register was written.** *A released PCI device* — release is
+now symmetric with claim (bus mastering cleared before the DMA is freed, a
+refusal when the device is the only display), a supervised job starts the
+replacement, and it is described in
+[../architecture/devices.md](../architecture/devices.md) under "Taking it back".
 
 ### Named in a "what it does not do" and still true
 

docs/plan/hybrid.md

@@ -172,6 +172,79 @@ version that survives a licensing conversation.
 
 ---
 
+## The driver came back
+
+H2 is the milestone this plan calls the one that cannot be skipped, and the
+reason is that H1 proves only half of the idea. A driver that runs in a process
+is interesting; a driver that runs in a process, *dies*, and is replaced by
+another process that picks the device up is the thing the whole hybrid argument
+rests on.
+
+`etc/schinit/vgpud.job` makes it a supervised job -- `KeepAlive = yes` -- so
+killing it is the whole test:
+
+```
+# cat /proc/schinit
+1 4 11 os.schos.vgpud
+# kill 4
+schinitd: restarting os.schos.vgpud
+vgpud: virtio-gpu at 00:03.0
+vgpud: claimed
+vgpud: display is 1280x800
+vgpud: scanout is ours, framebuffer at physical 0x3eceb000
+vgpud: pid 8 drew a test pattern from userspace
+```
+
+The test pattern ends in a twelve-block tally of the drawing process's pid,
+because a second run that redraws an identical picture is indistinguishable
+from a first run that never stopped. Read off the virtio-gpu scanout across
+four generations of the driver it says 4, 8, 10, 12. The same physical address
+comes back every time, which is the DMA arena reclaiming cleanly rather than
+drifting.
+
+### What it took that H1 had not needed
+
+- **A halt the kernel can apply without knowing what the device is.** A driver
+  that dies leaves its device bus-mastering into memory the kernel is about to
+  hand to somebody else. The kernel cannot quiesce a device it does not
+  understand, but it can clear `PCI_CMD_BUS_MASTER`, and that is enough to stop
+  the writes. `pciuser_stop()` does it before the DMA is freed, not after, and
+  claim/release became a symmetric pair: claiming sets the bit, releasing
+  clears it. `pcitest` grew a `bm` column so the state is visible rather than
+  assumed.
+- **A refusal, when there is nothing to fall back to.** `vgpu_release()` now
+  asks `vga_fb_restore()` whether there is a framebuffer to put the console on,
+  and declines to hand the device over if there is not. The rule that the
+  kernel keeps a minimal driver for everything on the path to a shell is only
+  worth writing down if something enforces it.
+- **The DMA arena not being handed to two allocators at once.** This is the bug
+  that stopped H2 for a day, and it is worth writing out: `freevm()` frees
+  every mapped page below `PHYSTOP`, and the DMA arena is the top 16 MB of
+  exactly that range. So a DMA block mapped into a driver and left for
+  `freevm()` went onto the page allocator's free list when the driver died,
+  while `pciuser_release_all()` handed the same pages back to the arena. Both
+  allocators then believed they owned it. The restarted `vgpud` asked for its
+  4 MB framebuffer, got pages `kalloc()` had already given out -- including its
+  own page tables -- and `dma_alloc()`'s zeroing wiped them, so the driver died
+  inside the ioctl, on the return from a syscall whose stack had stopped
+  existing. `do_dma()` now records a VMA with `no_free` set, the same way
+  shared memory does, so `proc_release_shared()` unmaps the range before the
+  page tables are torn down.
+
+### The clause that is not true yet
+
+The milestone says *"the compositor reattaches"*. It does not, because there is
+no compositor running: Schcom is v0 and is not started at boot. What has been
+shown is the two parts that do not depend on it -- the screen falls back to
+VBE, and the driver restarts and takes the device back. During the gap the VBE
+head carries a live console and the virtio-gpu head still shows the dead
+driver's last frame, which is exactly what a device nobody is driving does.
+
+Reattachment is a Schcom milestone, not a hybrid one, and it is recorded that
+way rather than counted here.
+
+---
+
 ## The mechanisms
 
 Four, and only the fourth is large.
@@ -279,13 +352,13 @@ start because the display driver is a process that needs a compositor.
 
 ## Milestones
 
-| | |
-|---|---|
-| **H0** | Mechanisms 2, 3 and 4, each with a test that exercises it from a userspace program against a device nothing else is using. |
-| **H1** | A userspace virtio-gpu driver, publishing `display_ops` across the channel. Schcom talks to it and does not know the difference. |
-| **H2** | Kill the driver process on purpose. The screen falls back to VBE, the driver restarts, the compositor reattaches. **This is the milestone that proves the whole idea**, and it is the one that cannot be skipped. |
-| **H3** | A modeset API, which is the thing an out-of-kernel GPU driver makes affordable and an in-kernel one never got. |
-| **H4** | USB, same shape. |
+| | | |
+|---|---|---|
+| **H0** | Mechanisms 2, 3 and 4, each with a test that exercises it from a userspace program against a device nothing else is using. | done |
+| **H1** | A userspace virtio-gpu driver, publishing `display_ops` across the channel. Schcom talks to it and does not know the difference. | done, except that Schcom does not talk to it -- `cmd/vgpud` holds the device and draws its own frame |
+| **H2** | Kill the driver process on purpose. The screen falls back to VBE, the driver restarts, the compositor reattaches. **This is the milestone that proves the whole idea**, and it is the one that cannot be skipped. | done, less the compositor -- see "The driver came back" |
+| **H3** | A modeset API, which is the thing an out-of-kernel GPU driver makes affordable and an in-kernel one never got. | |
+| **H4** | USB, same shape. | |
 
 ---