From f38eb8a7fdb7e2c3af86916772d4266ab7233a7d Mon Sep 17 00:00:00 2001 From: Nexoniarz Date: Mon, 14 Sep 2026 13:37:04 +0200 Subject: [PATCH] ldls: L4 was free after all, and the plan closes L4 was listed as "free -- H2 built all of this" and never actually tested, which is a claim rather than a result. Tested now, and it holds: ldlsd running ping 2/2; holds index 7 and DMA slots 3 and 4 killed interface DOWN, device released, bus mastering cleared, DMA reclaimed -- back to vgpud's 3 blocks ping fails honestly: sendto: -5 a second ldlsd ping 3/3, and the arena hands back the same addresses pciuser_release_all() and netuser_release_all() sit beside each other in exit() for the same reason, and that they apply unchanged to a foreign driver is the whole point of having built them against our own. With that, every milestone on ldls.md is done, so it leaves docs/plan/ for docs/architecture/ -- the rule docs/plan/README.md states and the one that keeps the two directories meaning different things. The new document describes what is, not what was planned: the five rules, the four pieces, what it costs (polling, no stack protector, locks that are no-ops), that it can be killed, and a "what it does not do" that leads with one driver, one kernel version, tried once. Sixteen source files pointed at docs/plan/ldls.md and would have been dangling the moment it moved. All repointed, and the milestone references -- which the architecture doc deliberately does not have -- rewritten to name sections instead, so a reader lands somewhere real. The register row closes with it. hybrid.md freed a slot, ldls.md took it, and it is now free again. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2 --- cmd/ldlsd/main.c | 4 +- cmd/netecho/main.c | 9 +- docs/architecture/README.md | 1 + docs/architecture/devices.md | 2 +- docs/architecture/ldls.md | 232 +++++++++ docs/licensing.md | 6 +- docs/plan/1.0.md | 15 +- docs/plan/README.md | 7 +- docs/plan/ldls.md | 888 ----------------------------------- include/netuser.h | 4 +- ldls/btf.h | 3 +- ldls/host/inspect.c | 4 +- ldls/host/run.c | 6 +- ldls/host/types.c | 8 +- ldls/ko.c | 2 +- ldls/ko.h | 2 +- ldls/kpi/kpi.c | 4 +- ldls/kpi/kpi.h | 2 +- ldls/kpi/net.c | 2 +- ldls/stub.h | 2 +- ldls/test/refmod.c | 2 +- tools/python/ldls_check.py | 2 +- 22 files changed, 278 insertions(+), 929 deletions(-) create mode 100644 docs/architecture/ldls.md delete mode 100644 docs/plan/ldls.md diff --git a/cmd/ldlsd/main.c b/cmd/ldlsd/main.c index 73897c1..91718ec 100644 --- a/cmd/ldlsd/main.c +++ b/cmd/ldlsd/main.c @@ -2,7 +2,7 @@ // // ldlsd -- LDLS, on SchOS. // -// docs/plan/ldls.md milestone L3b. `ldls-run` did this on the build machine so +// docs/architecture/ldls.md. `ldls-run` does this on the build machine so // that loader failures cost a second instead of a boot; this is the same // loader, the same BTF reader and the same trampolines, in a process on the // system they were written for. @@ -96,7 +96,7 @@ streq(const char *a, const char *b) // buffers (a kernel's BTF is several megabytes) and memory it can execute -- // which on SchOS is every mapping, because the kernel never sets EFER.NXE. // That is convenient here and is a gap rather than a feature; see the note in -// docs/plan/ldls.md. +// docs/architecture/ldls.md. // // The module lands in the low 2 GB automatically: a process's mmap area starts // at 0x40000000 and bumps upward, which is exactly the window R_X86_64_32S diff --git a/cmd/netecho/main.c b/cmd/netecho/main.c index da9c9d9..fc8ad13 100644 --- a/cmd/netecho/main.c +++ b/cmd/netecho/main.c @@ -2,8 +2,9 @@ // // netecho -- a network, in a process. // -// docs/plan/ldls.md L3a. `/dev/netdev` lets a process be a network interface; -// this is the program that proves it, and it is deliberately *not* a driver. +// docs/architecture/ldls.md, "The class bridge". `/dev/netdev` lets a process +// be a network interface; this is the program that proves it, and it is +// deliberately *not* a driver. // It has no hardware, claims none, and answers every frame itself: // // DHCP so the interface gets an address at all -- lwIP asks the moment @@ -12,8 +13,8 @@ // ARP so the stack can find us // ICMP so `ping 10.9.0.1` answers // -// Why a fake network and not the Linux driver straight away: the two halves of -// L3 fail differently. If `ping` does not work with this, the bridge is wrong. +// Why a fake network and not the Linux driver straight away: the two halves +// fail differently. If `ping` does not work with this, the bridge is wrong. // If it works with this and not with a driver, the driver's side is wrong. A // test that can only fail in one lump tells you nothing about which half. // diff --git a/docs/architecture/README.md b/docs/architecture/README.md index a98ac3d..866e61d 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -94,6 +94,7 @@ Read in this order if you are new: | [networking.md](networking.md) | NIC drivers, lwIP, and both socket families. | | [init.md](init.md) | `schinit`, PID 1 as a role, and the job manifests. | | [debugging.md](debugging.md) | Symbols, a log that outlives the kernel, and asking the kernel a question. | +| [ldls.md](ldls.md) | Running somebody else's drivers: a Linux `.ko`, in a process, reaching hardware through `/dev/pci`. | --- diff --git a/docs/architecture/devices.md b/docs/architecture/devices.md index cb4d833..a889a43 100644 --- a/docs/architecture/devices.md +++ b/docs/architecture/devices.md @@ -86,7 +86,7 @@ therefore hands the process the whole page it sits in — forced by the hardware rather than chosen. Comparing against the BAR's own size instead meant a sub-page BAR could not be mapped at all, which nothing noticed until an rtl8139 with 256 bytes of registers turned up; see -[../plan/ldls.md](../plan/ldls.md), "The bug a foreign driver found in its +[ldls.md](ldls.md), "The bug a foreign driver found in its first minute". **DMA is a call, not a mapping.** The process is told the *physical* address, diff --git a/docs/architecture/ldls.md b/docs/architecture/ldls.md new file mode 100644 index 0000000..0f3c395 --- /dev/null +++ b/docs/architecture/ldls.md @@ -0,0 +1,232 @@ +# LDLS — running somebody else's drivers + +SchOS cannot write a driver for every device that exists. No project can. The +only route to the hardware support an established system has is to run Linux's +drivers, and LDLS is what does that. + +It works. `8139too` — an unmodified GPL kernel module — loads into a SchOS +process, drives QEMU's rtl8139 through `/dev/pci`, and carries packets: + +``` +# ldlsd /lib/modules/8139too.ko /lib/modules/btf & + kpi: the driver read a MAC off the hardware: 52:54:00:ab:cd:ef + ndo_open returned 0 + carrying frames. the interface is ldls0. + +# ping 10.9.0.2 4 +64 bytes from 10.9.0.2: icmp_seq=1 time=30 ms +... +4 packets transmitted, 4 received, 0% packet loss +``` + +`10.9.0.0/24` is reachable only through `ldls0`. The address on it came from +DHCP, over the same driver. + +--- + +## The rules it is built on + +Five, and everything else is an implementation of them. + +### 1. Nothing GPL ever enters an image we build + +The driver is **the user's**: obtained by them, loaded on their machine. SchOS +ships the layer and never the thing it loads. `.gitignore` refuses `*.ko`, +`make ldls-stage` is the one command that puts a module into a local image and +is deliberately not part of `all`, and `make ldls-unstage` takes it out again. + +[../licensing.md](../licensing.md) §6 is the whole argument. The short form is +that GPL obligations attach to *distribution*, we distribute Apache-2.0 code +containing no GPL, and the combination happens in a user process on the user's +machine — so no combined work is distributed by anyone. + +### 2. It runs in a process, never in the kernel + +A module linked into our kernel is the strongest derivative-work argument that +exists. A module combined with our shim inside a user process is a different +thing at arm's length — the position FreeBSD has shipped from for a decade. + +And practically: this is foreign code. A driver we did not write, cannot fully +audit and did not choose the quality of should not be able to take the machine +down. + +### 3. Facts, not expression + +A symbol's signature and a struct's field offsets are facts a compiled driver +depends on. Comments, macro bodies and implementations are expression and are +never carried over. Every file in `ldls/kpi/` records where its shape came +from. + +Mostly the question does not arise, because of rule 3½: + +### 3½. The layouts are not in our source at all + +A kernel built with `CONFIG_DEBUG_INFO_BTF` publishes +`/sys/kernel/btf/vmlinux`: every struct layout and every function prototype, as +generated type data describing the kernel that is installed. `ldls/btf.c` reads +it at load time, on the user's machine, from the user's own kernel. + +``` +$ ldls-types /sys/kernel/btf/vmlinux --need + pci_driver probe offset 16 size 8 + pci_dev resource offset 1000 size 1088 + resource start offset 0 size 8 + all present +``` + +Nothing there is ours to have transcribed, and it is right for the kernel +actually present rather than one pinned by guessing. + +### 4. One kernel version at a time + +Linux states in writing that it has no stable in-kernel ABI. BTF mostly +dissolves this, because it describes the kernel that is there — but a kernel +without BTF needs hand-written layouts for a named version, and that is when +this rule bites. + +### 5. The unresolved-symbol list is the specification + +Do not guess the API surface. Load a real module and let the loader say what is +missing. Every symbol LDLS does not implement still *resolves*, to a stub that +reports itself and returns zero, so the driver carries on and the next thing it +wants becomes visible. The work queue keeps writing itself. + +--- + +## The four pieces + +### The loader — `ldls/ko.c` + +A `.ko` is an ELF64 relocatable object. Index the sections, place the +allocatable ones, resolve the undefined symbols, apply the relocations. Written +from the ELF-64 gABI and the x86-64 psABI, both published specifications, so +there is no licence question in it at all. + +Measured over a stock 6.18.46 tree: **2,440 modules, every one lays out and +relocates to completion**, five relocation types, nothing unhandled. That sweep +is also what retired the fixed section table — `nvidia-modeset` has 7,069 +sections against a median of 50 — so the caller supplies it and there is no +limit to run into. + +It is **freestanding**: no libc, no allocation, no I/O. The same source builds +as a host tool, where a format failure costs a second, and as the SchOS-side +host. That is the arrangement `mkschfs` and the kernel already share. + +#### Two things it has to fix in the object + +**The `%gs` stack guard.** Modules are built `-mstack-protector-guard-reg=gs`, +so the canary is fetched through the GS segment — an ABI between the kernel and +its own per-CPU area that a host which is not that kernel cannot provide. +`ko_defuse_gs()` rewrites those references to `xor %reg, %reg`. There are two +forms: the prologue *loads* the guard with `mov`, the epilogue *compares* it +with `sub` before a `jne`. `xor` is right for both — it zeroes the canary and +sets ZF, so the branch falls through as a passing check does. It checks every +site before patching any and refuses the whole operation on anything it does +not recognise. + +**Where it is loaded.** `-mcmodel=kernel` means `R_X86_64_32S` relocations, so +the object has to live where a signed 32-bit absolute address reaches. A +process's mmap area starts at `0x40000000`, which is inside that window; on the +host harness `MAP_32BIT` reaches it from the other end. + +### The KPI — `ldls/kpi/` + +The Linux symbols, implemented. Register access to a mapped BAR; the PCI calls +to `/dev/pci`; allocation to `mmap` and `PCIIOC_DMA`; `register_netdev` to +`/dev/netdev`. Organised under Linux's own header names so a reader who knows +where something lives in Linux knows where it is here. + +`kpi_io_ok()` bounds every `ioread`/`iowrite` against the mapping `pci_iomap` +returned. A stray access would otherwise be a page fault with no context, and +with no IOMMU a hosted driver's wild writes have to be caught early or not at +all. + +### The back end — `/dev/pci` + +Already built, for `vgpud`, before any of this existed. +[hybrid.md](../plan/hybrid.md) insisted the interface be about *PCI devices* +rather than about a driver model, on the argument that an interface whose first +consumers are native drivers is a general mechanism rather than a hole shaped +like something foreign. A Linux driver's `pci_iomap`, `dma_alloc_coherent` and +`request_irq` land on it unaltered, which is that argument being collected. + +### The class bridge — `/dev/netdev` + +The one piece the kernel had to grow. `netdev_ops` is four entries — +`transmit`, `receive`, `mac`, `link_up` — so a process claims the device, +supplies a MAC, and `read()`/`write()` carry the frames. `/dev/pci` with the +arrows reversed: that hands a device *to* a process, this lets a process *be* +one. See [devices.md](devices.md). + +lwIP never learns that the thing behind the vtable is not a driver. + +--- + +## What it costs + +**It polls.** A Linux driver expects its interrupt handler to be called from an +interrupt. `/dev/pci` does deliver them — by blocking `read()`, which needs a +thread `ldlsd` does not have — so the driver's NAPI poll is called from a loop +instead. The same work, in a context LDLS can provide, at the cost of latency +and a spinning process. + +**The stack protector is gone**, per `ko_defuse_gs()` above. Foreign code, no +IOMMU, one hardening measure fewer. + +**A lock is a no-op.** In one process with no preemption inside the driver +there is nobody to race with. That is correct right up until a driver has an +interrupt thread, and it is the deepest unfixed assumption in the layer. + +--- + +## It can be killed + +The whole H2 machinery applies to a foreign driver unchanged, which was the +point of having built it against our own: + +| | | +|---|---| +| `ldlsd` running | `ping` works; it holds the device and two DMA blocks | +| killed | interface **down**, device released, bus mastering cleared, **DMA reclaimed** | +| `ping` | fails honestly — `sendto: -5` | +| a second `ldlsd` | `ping` works again, and the arena hands back the same addresses | + +`pciuser_release_all()` and `netuser_release_all()` sit beside each other in +`exit()` for the same reason: a process holding a device is one that may crash. + +--- + +## What it does not do + +- **One driver, one kernel version, tried once.** `8139too` on 6.18.46 against + QEMU's rtl8139. Nothing here has been tried on a second driver, a second + kernel, or real silicon, and until it has, "LDLS runs Linux drivers" is a + claim rather than a description. +- **No interrupts**, per the polling note above. +- **It is not a security boundary, and here that is worse than usual.** There + is no IOMMU, so a hosted driver can point its device at any physical address. + For SchOS's own drivers that isolates bugs and not intent; for *foreign* + drivers the distinction between bug and intent is one we are in no position + to make. This is the strongest argument for an IOMMU the project has + produced. +- **It relies on SchOS having no NX.** Every mapping is executable because + `EFER.NXE` is never set. If that is ever fixed — and it should be — `ldlsd` + needs a way to ask for executable memory, and `mmap` does not offer one. +- **Nothing is ever freed.** A hosted driver's allocations last as long as the + process, which is a leak in the strict sense and the right trade for a + program that will be killed to restart it. +- **BTF has to come from somewhere.** SchOS has no `/sys/kernel/btf/vmlinux`, + so the user brings it with the module, from the kernel the module was built + for. A kernel built without `CONFIG_DEBUG_INFO_BTF` has neither. +- **No config-space writes.** `/dev/pci` has none, deliberately — a write is + how a BAR gets moved out from under the kernel — so a driver that needs one + is refused loudly rather than quietly succeeded. +- **No module-to-module dependencies, no firmware loading, no sysfs, no + unload.** `request_firmware` fails, `module_exit` is never called, and the + way to stop a driver is to kill the process. +- **Some symbols are still logging stubs.** Deliberate, per rule 5 — but a + driver reaching one gets zero and carries on, and `memcpy` proved how quiet + that can be: it was a stub while the receive path was being brought up, so + every frame arrived as the right *length* of zeroes and nothing reported an + error. **A stub that returns zero is not safe for a function whose whole job + is a side effect.** diff --git a/docs/licensing.md b/docs/licensing.md index 9831e66..7016d71 100644 --- a/docs/licensing.md +++ b/docs/licensing.md @@ -65,7 +65,7 @@ specification rather than ported. **"Cannot be adopted" is not the same as "cannot be run."** A driver the *user* obtains and loads on their own machine is their act, not a distribution by us — -see §4 and [plan/ldls.md](plan/ldls.md), which is the layer built on exactly +see §4 and [architecture/ldls.md](architecture/ldls.md), which is the layer built on exactly that distinction and which ships no Linux code of any kind. --- @@ -249,7 +249,7 @@ copied file is a file to be understood, attributed and maintained forever. ## 6. LDLS, and the distribution boundary -[plan/ldls.md](plan/ldls.md) runs Linux drivers. That sounds like it should +[architecture/ldls.md](architecture/ldls.md) runs Linux drivers. That sounds like it should contradict §2, and it does not, because the two questions below are different ones and only the first is about us. @@ -257,7 +257,7 @@ ones and only the first is about us. Every file under `ldls/` is ours. The loader was written from the ELF-64 gABI and the x86-64 psABI; the shim is written under the rule in -[plan/ldls.md](plan/ldls.md) §3. **No Linux code is in it and none ever will +[architecture/ldls.md](architecture/ldls.md). **No Linux code is in it and none ever will be**, which is the same standard the xHCI and USB stacks already meet (§5). ### The combination is not ours to make diff --git a/docs/plan/1.0.md b/docs/plan/1.0.md index a554346..2394560 100644 --- a/docs/plan/1.0.md +++ b/docs/plan/1.0.md @@ -95,16 +95,17 @@ The galling ones. The work exists, is tested, and nothing uses it. |---|---| | **`foundations.md`** | Item 1 built. Items 2 (a shareable memory object), 3 (`AF_UNIX` in the filesystem namespace), 4 and 5 open. **Wayland stops at item 2.** | | ~~**`hybrid.md`**~~ | **Closed.** All four mechanisms built; H1, H2 and H3 done. H4 — the USB eviction — is *withdrawn* rather than outstanding: USB is the bus peripherals arrive on, not a peripheral, and an eviction that has to leave an identical driver behind has moved nothing. See [hybrid.md](hybrid.md), "Decided: USB stays in the kernel". | -| **`ldls.md`** | L0, L1, L2 and L3a done — the loader handles 2,440 stock modules; 449 of 500 run `init_module`; `8139too`'s `probe()` runs to completion; and `/dev/netdev` lets a process be an interface, with `ping` working over one. L3b done too — `ldlsd` runs the same loader on SchOS and gets a real driver's `probe()` to return. **L3 done: `ping` works over a Linux driver.** `8139too`, loaded by our loader, running in a SchOS process, driving QEMU's rtl8139 through `/dev/pci`, carrying DHCP and ICMP both ways on a subnet reachable only through it. One driver, one kernel version, tried once — and the network transport for the debugger is the only thing still open on any plan. | +| ~~**`ldls.md`**~~ | **Closed.** L0 through L4 built: the loader handles 2,440 stock modules, `8139too` runs in a SchOS process driving QEMU's rtl8139 through `/dev/pci`, `ping` works over it, and killing the host releases the device and DMA so a replacement takes over. Described in [../architecture/ldls.md](../architecture/ldls.md). One driver, one kernel version, tried once. | | **Schcom v0** | 715 lines that composite, over a wire format that is superseded. Frozen: it runs, nothing more is added, and it is kept only because it is the one thing in the tree that has ever put a client's pixels on screen. | | **`debugger.md`** | D0 through D4 built and described in [../architecture/debugging.md](../architecture/debugging.md). Only the network transport is open — a userspace daemon bridging TCP to `/dev/dbg`, which is deliberately last. | -**The row LDLS spends.** `hybrid.md` closed above — H3 built, H4 withdrawn — -and [ldls.md](ldls.md) takes the slot it freed. Worth being honest that this is -the largest thing ever to enter this register, and that it is **explicitly not -part of 1.0**: nothing in the acceptance test needs a foreign driver. It is -here because the mechanisms it uses are finished and idle, and because the -alternative to it is writing a driver for every device that exists. +**The row LDLS spent, and gave back.** `hybrid.md` closed — H3 built, H4 +withdrawn — and `ldls.md` took the slot it freed. It has since closed too, so +the slot is free again. Worth being honest that this is +the largest thing ever to enter this register, and that it was **explicitly not +part of 1.0**: nothing in the acceptance test needs a foreign driver. It was +taken on because the mechanisms it uses were finished and idle, and because +the alternative to it is writing a driver for every device that exists. **The row the debugger spent.** Adding one requires closing one, and *A released PCI device* closed when H2 landed. It is worth saying plainly that the debugger diff --git a/docs/plan/README.md b/docs/plan/README.md index d0bef00..e7e8c65 100644 --- a/docs/plan/README.md +++ b/docs/plan/README.md @@ -43,7 +43,7 @@ And the one the hybrid mechanisms were built for: | | | |---|---| -| **[ldls.md](ldls.md)** | The Linux Driver Layer for SchOS. Someone else's drivers, in a process, reached through `/dev/pci` — because no project can write a driver for every device that exists. **Not started.** | +| ~~ldls.md~~ | **Built.** The Linux Driver Layer for SchOS: someone else's drivers, in a process, reached through `/dev/pci`. `ping` works over `8139too`. The description moved to [../architecture/ldls.md](../architecture/ldls.md). | And one that is a tool rather than a capability: @@ -106,8 +106,9 @@ mechanism rather than a hole shaped like something foreign. The compositor is still a demanding consumer — it is no longer the *only* one available. Four mechanisms, all built. One eviction done, the second withdrawn with its -reasons recorded, and the foreign driver host — [ldls.md](ldls.md) — is what -they were for. +reasons recorded, and the foreign driver host they were for is +[../architecture/ldls.md](../architecture/ldls.md) — built, and carrying +packets. ### 4. `wayland.md` — with the GPU driver either side of the seam diff --git a/docs/plan/ldls.md b/docs/plan/ldls.md deleted file mode 100644 index 9692eda..0000000 --- a/docs/plan/ldls.md +++ /dev/null @@ -1,888 +0,0 @@ -# LDLS — the Linux Driver Layer for SchOS - -SchOS cannot write a driver for every device that exists. No project can. The -only route to the hardware support an established system has is to run someone -else's drivers, and in practice "someone else's drivers" means Linux's. - -**LDLS itself is Apache-2.0**, like everything else in this tree, and stays that -way because it never contains a line of Linux. What the *user* combines it with -on their own machine is a separate question, and -[../licensing.md](../licensing.md) §6 is where that one is answered. - -[hybrid.md](hybrid.md) built the mechanisms for this and said so in as many -words — MMIO, DMA, interrupts and config space to a process, with SchOS's own -drivers as the first consumers so the interface would be general rather than a -hole shaped like something foreign. `vgpud` proved it. **LDLS is what those -mechanisms were built for.** - ---- - -## The rules - -These are the spine. Everything below is an implementation of them, and -anything that violates one is wrong however well it works. - -### 1. Nothing GPL ever enters an image we build - -The driver is **the user's**, obtained by them, loaded on their machine. SchOS -ships the layer and never the thing it loads. This is not a new position — it -was decided on 2026-08-08 and is written up in -[../architecture/devices.md](../architecture/devices.md) and -[../licensing.md](../licensing.md) §2: - -> Apache-2.0 code and GPL-2.0-only code cannot be combined into one -> distributable work. It is not a matter of taste or of how carefully the files -> are kept apart in the tree. - -`RAMDISK_EXCLUDE` and `NOTPORTS` are the existing mechanisms that keep this -true mechanically rather than by intention, and LDLS must stay inside them. - -### 2. It runs in a process, never in the kernel - -Two independent reasons, and either alone would decide it. - -**Legally**, a module linked into our kernel is the strongest derivative-work -argument that exists — it is the case the FSF has always made. A module -combined with our shim *inside a user process, on the user's machine, at their -direction* is a different thing at arm's length, and it is the position -FreeBSD has shipped from for about a decade: a BSD-licensed `linuxkpi` in base, -GPL `drm-kmod` installed by the user from ports. - -**Practically**, this is foreign code. A driver we did not write, cannot fully -audit and did not choose the quality of, running with the machine's memory in -reach, is exactly the code that should not be able to take the machine down. -Crash isolation is not a nicety here; it is the reason. - -### 3. Facts, not expression - -The shim has to present Linux's internal API, and that API is expressed in GPL -headers. The line LDLS holds: - -| | | -|---|---| -| A symbol exists, and takes these arguments | **fact** — reproduce it | -| A struct has these fields in this order at these offsets | **fact** — a compiled driver indexes them, so there is no other way to express it. And mostly it need not be in our source at all; see BTF below | -| The comment above it | **expression** — never | -| The macro body, the inline function body, the implementation | **expression** — never | - -Every KPI file records where its shape came from. Not decoration: it is the -difference between a clean-room reimplementation and a transcription, and the -only time anyone will ask is the time it matters. - -This is the same discipline [../licensing.md](../licensing.md) §5 already -describes — *"the xHCI, USB, PCI, fbdev and ELF specifications were all read -and implemented directly, which is why those drivers have no licence -entanglement at all"* — applied to a case where the specification is thinner. - -> **Not legal advice, and it should be checked by someone whose job that is -> before anything ships.** The reasoning rests on interfaces being outside -> copyright (EU Software Directive Art. 1(2)) and on *Google v. Oracle* (2021), -> and neither is a blanket permission. - -#### The layouts do not have to be ours to write - -The uncomfortable row above is the struct layouts, and it turns out to be -largely avoidable. Kernels built with `CONFIG_DEBUG_INFO_BTF` publish -`/sys/kernel/btf/vmlinux`, which carries the layouts *and* the function -prototypes as generated binary type data about the kernel that is installed: - -``` -struct pci_dev { - struct list_head bus_list; /* 0 16 */ - struct pci_bus * bus; /* 16 8 */ - -[97814] FUNC 'pci_iomap' type_id=97813 linkage=static -[97815] FUNC_PROTO '(anon)' ret_type_id=108 vlen=4 -``` - -So LDLS reads them at load time, on the user's machine, from the user's own -kernel. Our source contains neither, and there is nothing to have transcribed. -It also mostly dissolves rule 4 below, because BTF describes the kernel that is -actually there rather than one we pinned by guessing. - -**BTF gives shape, not behaviour.** What `napi_complete_done` *does* is still -ours to implement from documentation, and that is the whole of the real work. -A kernel without BTF needs hand-written layouts for a named version, which is -why this section stays rather than being replaced. - -### 4. One kernel version at a time, pinned and named - -*(Mostly superseded by BTF, above — but it still governs any kernel that does -not carry it.)* - -Linux states in writing that it has no stable in-kernel ABI — -`Documentation/process/stable-api-nonsense.rst` exists to say so. Believing -them is cheaper than being surprised by them. - -So LDLS targets **one version**, says which, and is re-pinned deliberately. -FreeBSD's LinuxKPI does exactly this per drm-kmod release. A layer that claims -to support "Linux drivers" without a version is claiming something nobody has -ever delivered. - -### 5. The unresolved-symbol list is the specification - -Do not guess the API surface. **Load a real module and let the loader say what -is missing.** That list is the work queue, it is exact, and it is available -before a single line of shim is written. - ---- - -## What it is made of - -Four pieces, of which three are ordinary and one is the project. - -### The loader — ours, clean, and small - -A `.ko` is a relocatable ELF64 object. Loading one is: lay out the allocatable -sections, apply the relocations, resolve the undefined symbols against a table, -and find the entry point. SchOS already parses ELF in `kernel/fs/exec.c`, for a -different object type but with the same plumbing. - -There is no licence question here at all — ELF is a published specification and -so is the relocation set. - -### The KPI — the shim, and the actual work - -Our Apache-2.0 reimplementation of the Linux symbols a given driver needs, -organised under Linux's own header names so it is navigable: `kpi/slab.h`, -`kpi/pci.h`, `kpi/netdevice.h`. Rule 3 governs every file. - -### The back end — already built - -`/dev/pci`: `PCIIOC_INFO`, `PCIIOC_CLAIM`, `PCIIOC_CFG`, `mmap` of a BAR, -`PCIIOC_DMA`, `PCIIOC_IRQ`. A Linux driver's `pci_iomap`, `dma_alloc_coherent` -and `request_irq` land on these, which is the whole reason they were specified -in terms of PCI devices rather than in terms of drivers. - -### The class bridge — the one new kernel interface - -This is the piece that does not exist, and it is the interesting one. - -A hosted driver produces packets in a process. SchOS's network stack consumes -them through `struct netdev_ops` in the kernel — `transmit`, `receive`, `mac`, -`link_up`, four entries and nothing else. So what is missing is a **netdev -whose backend is a process**: the kernel publishes the vtable, the four calls -are forwarded over a descriptor, and lwIP never learns that the driver is not -in the kernel. - -It is the exact analogue of what `display_ops` was for the GPU, and -`netdev.h`'s own opening comment already anticipates the shape — *"the stack -should ask for the network interfaces and get whatever is bound … exactly as a -compositor asks for the display and does not care which driver answers."* - -One device, one process, claimed and released, on the `/dev/pci` discipline. - ---- - -## Milestones - -| | | | -|---|---|---| -| **L0** | **The loader.** Parse a real `.ko`, lay it out, relocate it, and **print every unresolved symbol**. Nothing executes. The output is the measured API surface. | **done** — `ldls/ko.c`, `make ldls-check` | -| **L1** | **Stubs.** Every symbol resolves, to a function that logs its name. `init_module` runs, which yields the *call order* — the second thing you need and the other thing you cannot guess. | **done** — `ldls/stub.c`, `ldls-run` | -| **L2** | **The hardware.** `probe()` receives a working `struct pci_dev`, maps its BARs, allocates coherent DMA and registers an interrupt, all through `/dev/pci`. The driver reaches its device. | **done**, except `/dev/pci` — see below | -| **L3a** | **The class bridge.** `/dev/netdev`: a process claims it and becomes an interface. `ping` over a *process*. | **done** — `kernel/dev/netuser.c`, `cmd/netecho` | -| **L3b** | The SchOS-side host: `ldls-run`'s loader and BTF work, as a program on SchOS. | **done** — `cmd/ldlsd`; `/dev/pci` is L3c's half | -| **L3c** | The fifteen symbols, implemented rather than stubbed, against a real rtl8139. | **done** — `ldls/kpi/`; the driver reads the real MAC | -| **L3d** | The packet path: `ndo_open`, `ndo_start_xmit`, receive. | **done** | -| **L3** | **A packet.** The two joined: **`ping` over a Linux driver.** This is the milestone that proves the whole idea and the one that cannot be skipped. | **done** | -| **L4** | **Kill it.** The host process dies, the kernel clears bus mastering and reclaims the DMA, a new host starts and the interface comes back. | free — H2 built all of this | - -L4 is listed for completeness rather than as work. `pciuser_stop()`, -`pciuser_release_all()` and `KeepAlive` already do it, and that they apply -unchanged to a foreign driver is the point of having built them against our -own. - -### The first target: `8139too`, not `r8169` - -The obvious choice is a Realtek `r8169`, because that silicon is everywhere. -It is the wrong first choice for one decisive reason: **QEMU cannot emulate -it**, so L0 through L3 would have no way to be tested, and a driver layer -developed against hardware you cannot run is a layer nobody can debug. - -QEMU emulates `rtl8139`. `8139too` is small, self-contained, needs no firmware -blob, and uses the thin `mii` helpers rather than dragging in phylib — which -`r8169` does, and which is a substantial surface on its own. - -So: **`8139too` to build the layer, `r8169` to prove it was general.** The -second is the interesting one and it is much cheaper once the first works. - -This is the `/dev/pci` lesson applied backwards. There, a native driver was the -first consumer so the interface would not be shaped like something foreign. -Here the interface is fixed by Linux, and what is chosen instead is the -*smallest real consumer that can be tested* — because the risk is no longer -designing the wrong interface, it is building a hundred stubs nobody has ever -run. - ---- - -## L0, and what it measured - -`ldls/ko.c` is the loader: index the sections, place the allocatable ones, -resolve the undefined symbols against a table, apply the relocations. It is -freestanding — no libc, no allocation, no I/O — so the same source builds as a -host tool today and as the SchOS-side host later, which is the arrangement -`mkschfs` and the kernel already share. - -``` -$ ldls-inspect 8139too.ko - 119288 bytes, 55 sections, 363 symbols - module 8139too - license GPL - describes RealTek RTL-8139 Fast Ethernet driver - loads to 33988 bytes - - relocations: 1980 - R_X86_64_64 363 - R_X86_64_PC32 991 - R_X86_64_PLT32 428 - R_X86_64_32S 169 - R_X86_64_PC64 29 - - undefined symbols: 94 - ... - stops at: unresolved: __fentry__ -``` - -**The licence is printed first, before anything is loaded**, because it is the -one property that decides whether the object may be on this machine at all and -it is a plain string in `.modinfo`. Rule 1 is easier to keep when the answer is -on screen. - -### What the measurement says - -**94 symbols** is the API surface of `8139too` on Linux 6.18.46. Not an -estimate — the list, exactly, and it is L1's work queue. It is also smaller -than it looks, because a good fraction of it is not API at all: `__fentry__`, -`__stack_chk_fail`, `__ref_stack_chk_guard`, `__x86_return_thunk` and -`__fortify_panic` are compiler and instrumentation artefacts that stub out in a -line each. - -**Five relocation types**, and a corpus run says that is the whole set: over -**2,440 real modules** from a stock 6.18.46 tree, every one lays out and -relocates to completion with nothing unhandled. That includes `nvidia-modeset` -at 7,069 sections and 74,305 relocations, which is what retired the loader's -fixed section table — the caller supplies it now, so there is no limit to run -into rather than a generous one. - -Relocating against stubs is a separate question from listing what is missing, -and it is the one that says the loader is finished: with nothing resolved, the -first unknown name stops everything and no relocation type past it is ever -reached. - -### Nothing foreign is in this tree - -`make ldls-check` runs the loader against `ldls/test/refmod.c` — ours, compiled -with `-c`, written to contain one of each thing the loader has to get right: -undefined functions *and* undefined data, `.text` with calls, `.data` holding a -pointer into `.rodata`, a `.bss` that must arrive zeroed, and a `.modinfo` -shaped the way a module's is. The modules above were read from the host's own -`/nix/store` and never copied here; `.gitignore` refuses `*.ko` outright so -that cannot happen by accident. - ---- - -## L1, and the driver's own voice - -Every undefined symbol resolves to a trampoline — twenty generated bytes that -load the symbol's index and jump to a shared thunk, which saves the argument -registers and reports the call. `init_module` is then simply called. - -``` -$ ldls-run 8139too.ko - module 8139too - license GPL - wants 94 symbols - loaded 33988 bytes at 0x416f8000, 94 stubs at 0x40377000 - - calling init_module at 0x416fc020 - - 1 _printk(0x416fb6c0, 0, 0) - | 8139too: 8139too Fast Ethernet driver 0.9.28 - 2 __pci_register_driver(0x416fb400, 0x416fff80, 0x416fbdbb) - - init_module returned 0 after 2 stub call(s) -``` - -**That second line is the driver's own banner**, printed through a `_printk` -that is ours, from a string in the module's own `.rodata` that a relocation put -there. There is no stronger evidence that a load worked than the loaded thing -speaking. - -The stubs return zero rather than stopping, which is a deliberate departure -from what this milestone originally said. Stopping at the first call tells you -one name; carrying on tells you the *sequence*, and the sequence is the thing -that cannot be read off a symbol table. A driver that then dereferences the -zero faults, and that is a result too — it says exactly how far it got before -it needed the kernel to have really done something. - -### The constraint nobody would guess - -The first run failed on a relocation, not on anything to do with stubs: - -``` -ldls-run: relocate: a signed 32-bit relocation does not fit -``` - -A module is compiled `-mcmodel=kernel`, so the compiler emits `R_X86_64_32S` — -a **signed 32-bit absolute address** — for anything it believes lives in the -top 2 GB. Load the object at an ordinary `mmap` address around `0x7f…` and -those relocations have nowhere to go. Linux avoids this by giving modules a -region within 2 GB of the kernel text; the harness uses `MAP_32BIT` to reach -the same place from the other end. - -**On SchOS this costs nothing** — a process's mmap area starts at `0x40000000` -and is already inside the window — but it is a real constraint on where the -LDLS host may put a module, and it would have been found much later and much -more confusingly without a loader that checks its own arithmetic. - -### What running 500 of them says - -| | | -|---|---| -| `init_module` ran and returned | **449** | -| faulted on a stub's zero | 22 | -| no `init_module` at all | 29 | -| **loader errors** | **0** | - -The 22 are not failures of the loader; they are drivers that needed an -allocation to come back non-null. The 29 without an entry point are mostly -library modules that exist to be depended on. - -### And what it says about the work queue - -Counting *calls* is the wrong metric — one module walking an rbtree called -`rb_first` sixty-four times and told us nothing. Counting **how many modules -need a symbol** is the useful one, and over 110 modules it comes to this: - -| symbol | modules | -|---|---| -| `__kmem_cache_create_args` | 25 | -| `crypto_register_template` | 11 | -| `_printk` | 10 | -| `crypto_register_alg` | 10 | -| `__register_nls` | 9 | -| `crypto_register_shash` | 4 | -| `af_alg_register_type` | 4 | -| `register_filesystem`, `misc_register`, … | 2 each | - -**Forty-four distinct symbols would carry every one of those 110 modules -through `init_module`.** That is a small enough number to be a plan rather than -an estimate, and it is not a number anybody could have guessed — the slab -allocator at the top, and a long tail of registration calls that mostly want to -record a pointer and return zero. - -It says nothing about `probe()`, which is where the real surface is and which -the kernel, not the module, is supposed to call. Reaching that is L2. - ---- - -## L2, and being the PCI core - -In Linux, `__pci_register_driver` ends at the PCI core, which matches the -driver's `id_table` against the bus and calls `probe()`. There is no PCI core -here, so LDLS keeps the pointer that call was given and plays the part itself. - -To do that it has to know where `pci_driver.probe` is — and **it asks the -kernel rather than being told by us**. `ldls/btf.c` reads -`/sys/kernel/btf/vmlinux`: 122,279 types, every struct layout and every -function prototype, as generated data describing the kernel that is installed. - -``` -$ ldls-types /sys/kernel/btf/vmlinux --need - pci_driver probe offset 16 size 8 - pci_driver id_table offset 8 size 8 - pci_dev vendor offset 60 size 2 - pci_dev resource offset 1000 size 1088 - resource start offset 0 size 8 - ... - all present -``` - -Cross-checked against `pahole` on the same file, field for field. **No offset -in this is ours**, which is rule 3's uncomfortable row disappearing rather than -being argued about. - -### What the driver then did - -``` - pci_driver at 0x41662400, from BTF: probe +16, id_table +8 - driver name "8139too" - - devices it claims: - 10ec:8139 class 000000/000000 - ... and 16 more - - calling probe(pci_dev 10ec:8139, ...) - - 1 alloc_etherdev_mqs(0x368, 0x1, 0x1) - 2 pci_enable_device(...) - 3 pci_request_regions(...) - 4 pci_set_master(...) - 5 pci_iomap(pdev, 1, 0) - 6..544 ioread8 / iowrite8 -- 0x88, 0x8a, 0x8c, 0x8e, over and over - 545 dev_addr_mod(...) - 547 netif_napi_add_weight_locked(...) - 549 timer_init_key(...) - 550 register_netdev(...) - 551 netdev_info(...) - | %s at 0x%p, %pM, IRQ %d - - probe returned 0 after 552 stub call(s) -``` - -`10ec:8139` is the real RTL8139 id. The driver mapped **BAR 1**, which is the -one we marked `IORESOURCE_MEM`. And those 538 alternating byte accesses with -values stepping `0x88, 0x8a, 0x8c, 0x8e` are **serial EEPROM clocking** — the -driver bit-banging the chip to read its MAC address, then handing the result to -`dev_addr_mod`. It registered a NAPI instance, started a watchdog timer, -registered the interface, printed its "found it" line, and **returned 0**. - -That is a complete driver bring-up, on shape alone. - -### Fifteen symbols - -The whole of that probe touches **15 distinct symbols**, and 538 of the 552 -calls are two of them: - -| | | -|---|---| -| `ioread8` / `iowrite8` / `ioread32` | 540 calls, 3 symbols | -| PCI setup | `pci_enable_device`, `pci_request_regions`, `pci_set_master`, `pci_iomap` | -| net | `alloc_etherdev_mqs`, `dev_addr_mod`, `netif_napi_add_weight_locked`, `register_netdev`, `netdev_info` | -| other | `mutex_lock`, `mutex_unlock`, `timer_init_key` | - -L1 said 44 symbols carry 110 modules through init. L2 says 15 carry this one -through probe. **Neither is a number anyone would have guessed**, and both came -out of running the thing rather than reading about it. - -### What is not done - -`probe()` receives a `pci_dev` whose *shape* is right and whose *contents are -invented*. The BARs are numbers this harness made up; `pci_iomap` returns a -zeroed page, not a mapping; nothing has touched hardware and nothing could, -because this runs on the build machine. - -**The half of L2 that says "all through `/dev/pci`" belongs to the SchOS -side**, where those calls have somewhere real to land. What is settled is the -part that had to come first: that probe can be reached at all, with a struct -the driver accepts, built from offsets nobody wrote down. - ---- - -## L3a: a network, in a process - -L3 is one sentence in the milestone table and three pieces of work. This is the -first, and it is the one the rest plugs into: **`/dev/netdev`, an interface -whose backend is a process.** - -The kernel side is `netdev_ops` and nothing else — `transmit`, `receive`, -`mac`, `link_up`, the same four entries `e1000` publishes. A process claims the -device, supplies a MAC, and from then on `read()` gives it frames the stack -wants sent and `write()` hands it frames that arrived. It is `/dev/pci`'s shape -with the arrows reversed: that hands a device *to* a process, this lets a -process *be* one. - -`cmd/netecho` is the proof, and it is deliberately **not** a driver — it has no -hardware and claims none. It answers every frame itself: DHCP so the interface -gets an address at all, ARP so the stack can find it, ICMP so it can be pinged. - -``` -# netecho & -netuser: ldls0 is a process now, 2:4c:44:4c:53:1 -net: ldls0 attached as interface 1, asking for an address -netecho: this process is now an interface. I am 10.9.0.1 and will offer 10.9.0.2 -netecho: answered DHCP, offering an address - -# cat /proc/netif -interface ldls0 -driver netuser -address 10.9.0.2 -gateway 10.9.0.1 - -# ping 10.9.0.1 3 -netecho: answered an ARP -netecho: answered a ping -64 bytes from 10.9.0.1: icmp_seq=1 time=30 ms -64 bytes from 10.9.0.1: icmp_seq=2 time=10 ms -64 bytes from 10.9.0.1: icmp_seq=3 time=10 ms -3 packets transmitted, 3 received, 0% packet loss -``` - -**The interface got its address by DHCP from a process**, and then answered -ping from one. lwIP never learns that the thing on the other side of -`netdev_ops` is not a driver — which is the entire property the bridge exists -to have. - -### It dies the way it should - -The same test, continued: - -``` -# kill 8 -# schdbg netuser - interface ldls0 DOWN, nobody holds it -# ping 10.9.0.1 1 -ping: sendto: -5 -0 packets transmitted, 0 received - -# netecho & -# ping 10.9.0.1 2 -2 packets transmitted, 2 received, 0% packet loss -``` - -`link_up()` answers false the moment nobody holds the descriptor, so the stack -refuses to send rather than swallowing packets into a driver that is not there -— and when a new process claims it, the *same* interface comes back, keeping -its address. That is H2's property applied to a network interface, and it came -free: `netuser_release_all()` sits beside `pciuser_release_all()` in `exit()` -for exactly the same reason. - -### One new thing the kernel needed - -`netinit()` starts DHCP on the devices that exist when it runs. An interface -created later — which is how an LDLS-hosted driver arrives — missed that -entirely and would have come up with no address and no way to get one. So -`net_attach_late()` exists, in `kernel/net/net.c` because that is the only file -allowed to include lwIP's headers. - ---- - -## L3b: the same loader, on SchOS - -`ldls/ko.c`, `ldls/btf.c` and `ldls/stub.c` were written freestanding so they -would build unchanged on both sides. `cmd/ldlsd` is the collection of that -promise: the same three files, linked against `lib/` instead of a libc, in an -ordinary SchOS process. - -``` -# ldlsd /lib/modules/8139too.ko /lib/modules/btf -/lib/modules/8139too.ko - module 8139too - license GPL - wants 94 symbols - defused 6 %gs stack-guard reference(s) - loaded 33988 bytes at 0x000000004001f000 - - calling init_module - 1 _printk(...) - | 8139too: 8139too Fast Ethernet driver 0.9.28 - 2 __pci_register_driver(...) - init_module returned 0 after 2 stub call(s) - - btf 122279 types - layout pci_driver.probe +16 pci_dev 2680 bytes - driver "8139too" - devices it claims: - 10ec:8139 - ... and 18 more - - calling probe - 1 alloc_etherdev_mqs(0x368, 0x1) - probe returned -12 after 1 stub call(s) -``` - -Byte for byte the behaviour of the host harness, which is what the shared -source was for. Two predictions from earlier milestones came good without any -work: a process's mmap area starts at `0x40000000`, so the module lands inside -the window `R_X86_64_32S` needs, and the kernel never sets `EFER.NXE`, so every -mapping is executable. - -That second one is **a gap, not a feature**. A process being able to execute -its own heap is normally a thing an operating system prevents, and LDLS is -quietly relying on its absence. It belongs on Gate 0's list. - -### The one thing that did not just work - -The first run on SchOS faulted before probe's first call: - -``` -pid 8 ldlsd: trap 14 err 5 on cpu 0 rip 0x4002138e addr 0xffff80007de5fc60 -``` - -`err 5` is present, read, user mode, and the address is a *kernel* one. The rip -resolved to `.text+0x206e`, which is: - -``` -mov %gs:0x0(%rip),%rax # rtl8139_init_one+0x26 -``` - -Linux builds modules with `-mstack-protector-guard-reg=gs`, so the stack canary -is fetched through the GS segment. That is an ABI between the kernel and its -own per-CPU area, and **a host that is not that kernel cannot provide it** — on -SchOS the GS base belongs to the kernel, so a user-mode read of it faults on a -kernel address. - -`ko_defuse_gs()` rewrites those references to `xor %reg, %reg`. There are two -forms and the second is easy to miss: the prologue *loads* the guard with `mov` -(`8b`), and the epilogue *compares* it with `sub` (`2b`) followed by a `jne` to -`__stack_chk_fail`. `xor` is correct for both — it makes the saved canary zero, -and it sets ZF so the epilogue's branch falls through exactly as a passing -check does. Six sites in `8139too`. - -It checks every site before patching any, and refuses the whole operation on -anything it does not recognise. A loader that half-understands an instruction -should not be editing it — and that refusal is what found the `sub` form, by -declining to guess. - -### Getting the files there - -`make ldls-stage KO=… BTF=…` copies a module and its kernel's BTF into the -local sysroot. It is deliberately not part of `all`, `.gitignore` refuses -`*.ko`, and `make ldls-unstage` removes them again. An image built between -those two commands has somebody else's GPL code in it and must not be -redistributed — which is exactly what [../licensing.md](../licensing.md) §6 -says about the combination, and why it is a thing you type on purpose. - ---- - -## L3c: a Linux driver, driving real hardware - -`ldls/kpi/` is the shim: 41 of `8139too`'s 94 symbols, implemented rather than -logged. Register access goes to a mapped BAR; `pci_enable_device`, -`pci_iomap`, `pci_request_regions` and `pci_set_master` go to `/dev/pci`; -allocation goes to `mmap` and `PCIIOC_DMA`; `register_netdev` goes to -`/dev/netdev`. Every file records its provenance, per rule 3. - -``` -# ldlsd /lib/modules/8139too.ko /lib/modules/btf - found 10ec:8139 at 00:05.0 - bar 0 0x000000000000c000 256 io - bar 1 0x0000000081060000 256 mem - netdev 2688 bytes - - calling probe, with 41 of 94 symbols implemented - - kpi: net_device 2688 bytes + 872 private - kpi: mapped BAR 1 -- phys 0x0000000081060000, 256 bytes - kpi: the driver read a MAC off the hardware: 52:54:00:ab:cd:ef - net: ldls0 attached as interface 1, asking for an address - netuser: ldls0 is a process now, 52:54:0:ab:cd:ef - kpi: registered as an interface, with the hardware's own MAC - - probe returned 0 after 0 stub call(s) -``` - -**`52:54:00:ab:cd:ef` is what the machine was configured with.** Nothing here -invented it: the driver bit-banged the card's EEPROM, serially, through a BAR -mapped by `/dev/pci`, and handed back the address it found. That it matches -means the loader, the relocations, the BTF offsets, the BAR mapping and the -register accessors are all right at once — which is why it was chosen as the -thing to aim at. - -**Zero stub calls.** Every symbol that probe reaches is implemented. - -### The bug a foreign driver found in its first minute - -The first run against real hardware wrote to address `0x4e` and faulted. -`0x4e` is 13 below where the driver meant to write, and 13 is `EACCES`: -`pci_iomap` had returned `-13` as a pointer, because the check for a failed -`mmap` looked only for `-1` while SchOS returns `-errno`. - -Underneath that was the real one. `sys_mmap` rounds a length up to a page — -the only granularity an MMU has — and `pciuser_mmio_ok()` compared that against -the BAR's own size. rtl8139's registers are **256 bytes**, the request became -4096, and the answer was `-EACCES` for a device the process legitimately held. - -**A BAR smaller than a page could never be mapped at all**, and nothing had -noticed because the only consumer until now was virtio-gpu, whose BARs are -large. The check is now against the pages the BAR lies in. The widening is -real and named in the code: a sub-page BAR hands the process the whole page it -sits in, which is forced by the hardware rather than chosen. - -That is precisely what [hybrid.md](hybrid.md) said a foreign driver would be -good for — *"what it asks for is what a foreign driver would ask for too"* — -arriving as a concrete bug rather than as a prediction. - -### Register access is checked - -`kpi_io_ok()` bounds every `ioread`/`iowrite` against the mapping `pci_iomap` -returned, reports the first few that miss, and returns without touching -memory. Two reasons: a stray access would otherwise be a page fault with no -context, and **there is no IOMMU** — a hosted driver's wild writes are exactly -the class of thing that has to be caught early or not at all. - -It is also what turned the bug above from a crash into two lines naming the -offsets, which is how the `-13` was spotted. - -### What is not done - -`probe()` succeeds, so the driver is up and the interface exists with the -hardware's own MAC. **No frame has moved.** `ndo_open` is never called, there -is no `ndo_start_xmit`, and nothing services the receive side — so `ldlsd` -releases the interface rather than leaving it up: - -``` - the driver is up on real hardware and has an interface. - carrying frames is the half of L3 that is not built, so the - interface is released rather than left up and silent. -``` - -A link that reports itself up with nothing draining its queue is a black hole: -the stack would send DHCP into it for ever and every layer above would believe -the network was merely quiet. Down is the true answer until packets move, and -that is L3d. - ---- - -## L3d: ping, over a Linux driver - -``` -# ldlsd /lib/modules/8139too.ko /lib/modules/btf & - kpi: the driver read a MAC off the hardware: 52:54:00:ab:cd:ef - ndo_open returned 0 - carrying frames. the interface is ldls0. - -# cat /proc/netif -interface ldls0 -mac 52:54:00:ab:cd:ef -state up -address 10.9.0.15 -gateway 10.9.0.2 - -# ping 10.9.0.2 4 -64 bytes from 10.9.0.2: icmp_seq=1 time=30 ms -64 bytes from 10.9.0.2: icmp_seq=2 time=20 ms -64 bytes from 10.9.0.2: icmp_seq=3 time=20 ms -64 bytes from 10.9.0.2: icmp_seq=4 time=20 ms -``` - -`10.9.0.0/24` is reachable **only** through `ldls0`, and `ldls0` is `8139too` -running in a SchOS process. The address on it was obtained by DHCP, over the -same driver, before the ping. - -`lwIP` → `netdev_ops` → `/dev/netdev` → `ldlsd` → `ndo_start_xmit` → a BAR -mapped by `/dev/pci` → the wire, and the whole way back. Nothing in the kernel -knows a Linux driver is involved; nothing in the driver knows it is not in -Linux. - -### It polls - -A Linux driver expects its interrupt handler to be called from an interrupt. -`/dev/pci` does deliver them — by blocking `read()`, which needs a thread this -program does not have. So `ldlsd` calls the driver's NAPI poll from a loop -instead: the same work, in a context LDLS can actually provide. It costs -latency and a spinning process, and it is written down here rather than hidden. - -### Four things `alloc_etherdev` does that allocating does not - -A driver gets its `net_device` from `alloc_etherdev_mqs`, and the function does -considerably more than return memory. Each of these was found by the driver -faulting on it, which is a slow way to learn a list and an extremely reliable -one: - -| | | -|---|---| -| `dev->dev_addr` | a **pointer**, aimed at the device's own `perm_addr`. The driver writes its MAC into the chip with `*(u32 *)dev->dev_addr`. | -| `dev->mc`, `dev->uc` | list heads, which are *empty* when they point at themselves. Zeroed means a null `next`, and the receive-mode setup walks them. | -| `dev->_tx` | an array of `netdev_queue`, because `netif_start_queue` clears a bit in `dev->_tx[0].state`. | -| `dev->state` | `__LINK_STATE_START`, set by Linux's `dev_open` **before** it calls `ndo_open`. | - -### The two that did not announce themselves - -Everything above crashed, which made it easy. Two did not. - -**`netif_running(dev)`.** Without `__LINK_STATE_START`, 8139too's receive loop -exited immediately — so the chip received the frame, the driver acknowledged -the interrupt, and nothing came out. Every counter said packets were arriving. -It was found by reading the chip's own `CBR` register through the BAR and -seeing that the hardware had written 600 bytes the driver never collected. - -**`memcpy`.** It was still a logging stub while the receive path was being -brought up. The driver copied each frame into its `sk_buff`, the copy did not -happen, and the stack was handed exactly the right *length* of zeroes. DHCP -never completed and nothing anywhere reported an error. **A stub that returns -zero is not safe for a function whose whole job is a side effect** — and the -memory primitives are the worst case, because they fail silently and -plausibly. `ldls/kpi/string.c` exists because of it. - ---- - -## What it will not do - -- **It is not a security boundary, and here that is worse than usual.** There - is no IOMMU, so a hosted driver can point its device at any physical address. - For SchOS's own drivers that isolates bugs and not intent, which - [hybrid.md](hybrid.md) says at length. For *foreign* drivers the distinction - between bug and intent is one we are in no position to make at all. This is - the single strongest argument for an IOMMU that this project has produced, - and it should be recorded as such rather than waved past. -- **It implements no licence check, and that is not circumvention.** - `EXPORT_SYMBOL_GPL` is Linux marking *Linux's own* symbols, and the check it - drives protects those. Every symbol LDLS provides is ours, reimplemented from - facts under Apache-2.0, so there is no gate being bypassed because there is - no Linux symbol being exposed. LDLS **reads** the declared licence and prints - it before loading anything, because that is the property that decides whether - an object belongs on a machine at all. See - [../licensing.md](../licensing.md) §6. -- **No firmware loading.** `request_firmware` is a stub that fails. Many - drivers need a blob, most blobs are redistributable only under their own - terms, and that is a second licensing problem which is not this one. -- **No module-to-module dependencies.** One object, no `depmod`, no symbol - exports between hosted modules. -- **No sysfs, procfs or debugfs.** Drivers that only *report* through them will - work; drivers that are *configured* through them will not. -- **No unload.** `module_exit` is never called; the way to stop a driver is to - kill the process, which H2 made a supported operation rather than a failure. -- **No sleeping-context fidelity.** Linux drivers assume a kernel's rules about - what may sleep, what runs in interrupt context and what `spin_lock_irqsave` - costs. A process has different rules, and where they differ the shim will be - wrong in ways that show up as rare races rather than as failures to build. - This is the deepest technical risk in the whole plan. -- **Running init is not working.** L0 proves 2,440 modules lay out and - relocate; L1 proves 449 of 500 run their init to completion. Both say the - *mechanism* is right and neither says a driver would function: every symbol - is a stub that logs and returns zero, so what has been demonstrated is that - drivers ask for things in a readable order, not that anything answers them. - The distance from here to a packet is the rest of this plan. -- **The stubs cannot tell code from data.** Every undefined symbol in a module - is `NOTYPE` in the ELF symbol table, so `jiffies` and `pci_iomap` are - indistinguishable and both get a trampoline. A driver that *reads* `jiffies` - reads trampoline bytes. BTF distinguishes them — `VAR` against `FUNC` — and - that is where the fix belongs rather than in a guess. -- **`ldls-run` is a development harness, not the deliverable.** It runs on the - build machine so that loader failures cost a second instead of a boot. The - thing that ships is the SchOS-side host. -- **The stack protector is gone from a hosted driver.** `ko_defuse_gs()` - removes it because the ABI it depends on cannot be supplied here. That is a - hardening measure lost, on foreign code, in a host with no IOMMU — worth - saying plainly rather than filing as a detail. -- **LDLS relies on SchOS having no NX.** Every mapping is executable because - `EFER.NXE` is never set. If that is ever fixed — and it should be — `ldlsd` - needs a way to ask for executable memory, and nothing in `mmap` currently - offers one. -- **BTF has to come from somewhere.** On Linux it is `/sys/kernel/btf/vmlinux`. - SchOS has no such file and never will — so the user brings the BTF along with - the module, from the kernel the module was built for. Both are theirs, - neither is shipped, and a kernel built without `CONFIG_DEBUG_INFO_BTF` has - neither. -- **L3a proves the bridge, not the driver.** `ping` works over a *process*; - nothing in that test loads a module, and `netecho` is a fake network rather - than a NIC. That is the point of doing it first — if `ping` had not worked - here the bridge would be wrong, and now a failure with a real driver can only - be the driver's side. -- **One interface, one process.** `/dev/netdev` is a single device and a second - claimant gets `-EBUSY`. Two hosted NICs would need it to be indexed like - `/dev/input/eventN`, which is a change of shape rather than of size. -- **The interface is never removed.** Releasing it leaves the device registered - with the link down, because nothing in lwIP supports taking an interface out - of its list. From the stack's point of view that is the same thing; from - `/proc/netif`'s it is a `ldls0` that stays listed after its process is gone. -- **It polls instead of taking interrupts**, per the note above. A busy - interface spins a processor; an idle one wakes 200 times before sleeping. -- **One driver, one kernel version, tried once.** `8139too` on 6.18.46 against - QEMU's rtl8139. That is what the milestone table has always promised and it - is worth repeating next to a working `ping`: nothing here has been tried on a - second driver, a second kernel, or real silicon. -- **Some symbols are still logging stubs.** That is deliberate — an - unimplemented symbol that reports itself keeps the work queue writing itself, - exactly as it did at L1 — but it means a driver reaching one gets zero and - carries on, and `memcpy` proved how quiet that can be. -- **A lock is a no-op.** In one process with no preemption inside the driver - there is nobody to race with, which is correct right up until the driver has - an interrupt thread. That is the sleeping-context assumption named above and - it is the deepest unfixed thing in the layer. -- **It will not support "Linux drivers".** It will support *one driver, on one - kernel version, that has been tried*. Anything beyond that is a claim, and - the list of what has actually been run belongs in the architecture document - when there is one. diff --git a/include/netuser.h b/include/netuser.h index c8008bd..547b879 100644 --- a/include/netuser.h +++ b/include/netuser.h @@ -2,8 +2,8 @@ // /dev/netdev -- being a network interface, from a process. // -// docs/plan/ldls.md calls this "the class bridge", and it is the one new -// kernel interface that plan needs. A driver hosted by LDLS produces packets +// docs/architecture/ldls.md calls this "the class bridge", and it is the one +// kernel interface LDLS needed that did not already exist. A driver hosted by LDLS produces packets // in a process; the network stack consumes them through `struct netdev_ops` in // the kernel. This is what joins the two. // diff --git a/ldls/btf.h b/ldls/btf.h index fd3441c..e843df4 100644 --- a/ldls/btf.h +++ b/ldls/btf.h @@ -3,7 +3,8 @@ // Reading BTF: struct layouts and function prototypes, from the kernel that // the driver was built for. // -// docs/plan/ldls.md milestone L2, and the thing that makes rule 3 comfortable. +// docs/architecture/ldls.md, rule 3½ -- the thing that makes rule 3 comfortable. +// // The shim has to agree with a compiled driver about where `pci_driver.probe` // sits, and a compiled driver is not negotiable about it. Writing those // offsets into our source would mean a field-for-field match with Linux's diff --git a/ldls/host/inspect.c b/ldls/host/inspect.c index 0de2abb..d172d9e 100644 --- a/ldls/host/inspect.c +++ b/ldls/host/inspect.c @@ -2,7 +2,7 @@ // // ldls-inspect -- what is in a relocatable object, and what it wants. // -// docs/plan/ldls.md milestone L0, run on the machine doing the building rather +// docs/architecture/ldls.md, "The loader", run on the machine doing the building rather // than inside SchOS. That is the same arrangement `mkschfs` has with the // kernel, for the same reason: the interesting failures here are format // failures, and a format failure found on the host costs a second instead of a @@ -15,7 +15,7 @@ // ldls-inspect 8139too.ko // // **No module is committed to this tree**, and none ever will be; see -// docs/plan/ldls.md rule 1. `make ldls-check` runs this against an object +// docs/architecture/ldls.md rule 1. `make ldls-check` runs this against an object // built from ldls/test/refmod.c, which is ours. #include diff --git a/ldls/host/run.c b/ldls/host/run.c index 79a6957..ea9b1d1 100644 --- a/ldls/host/run.c +++ b/ldls/host/run.c @@ -2,7 +2,7 @@ // // ldls-run -- load a module and call its init, on the build machine. // -// docs/plan/ldls.md milestone L1. L0 answered "what does this driver ask +// docs/architecture/ldls.md. The loader answered "what does this driver ask // for"; this answers "and in what order does it ask", which is the other thing // that cannot be guessed and falls straight out of running the thing. // @@ -18,7 +18,7 @@ // that logs and returns zero, so a driver gets as far as its first real // dependency on the kernel doing something, and then says so. // -// **No module is committed to this tree.** See docs/plan/ldls.md rule 1. +// **No module is committed to this tree.** See docs/architecture/ldls.md rule 1. #include #include @@ -264,7 +264,7 @@ wr64(unsigned long base, unsigned int off, unsigned long v) // The PCI resource flag bits. These are not layout and BTF does not carry // them -- they are #defines. They are ABI between the kernel and every driver // ever written against it, which makes them facts in the sense of -// docs/plan/ldls.md rule 3, and there are two of them. +// docs/architecture/ldls.md rule 3, and there are two of them. #define LDLS_IORESOURCE_IO 0x00000100 #define LDLS_IORESOURCE_MEM 0x00000200 diff --git a/ldls/host/types.c b/ldls/host/types.c index 5981c7d..299a98f 100644 --- a/ldls/host/types.c +++ b/ldls/host/types.c @@ -2,15 +2,15 @@ // // ldls-types -- ask a kernel's BTF where its fields are. // -// docs/plan/ldls.md milestone L2. This is the tool that proves the layouts LDLS -// needs can be read from the kernel the user already has, rather than written -// into our source from theirs. +// docs/architecture/ldls.md, rule 3½. This is the tool that proves the layouts +// LDLS needs can be read from the kernel the user already has, rather than +// written into our source from theirs. // // ldls-types /sys/kernel/btf/vmlinux pci_driver probe id_table // ldls-types /sys/kernel/btf/vmlinux --need // // `--need` prints every offset the PCI path needs at once, which is both a -// test and the thing L2's driver code consumes. +// test and the thing cmd/ldlsd consumes. #include #include diff --git a/ldls/ko.c b/ldls/ko.c index 89b0ae1..4b720c7 100644 --- a/ldls/ko.c +++ b/ldls/ko.c @@ -647,7 +647,7 @@ ko_name(const struct ko *k) // The module's `struct module` lives in `.gnu.linkonce.this_module` and its // name is at a fixed offset -- but that offset is a property of the kernel - // version, which is exactly the kind of thing docs/plan/ldls.md rule 4 says + // version, which is exactly the kind of thing docs/architecture/ldls.md rule 4 says // not to assume. The symbol table has the same information portably: the // object defines `__this_module`, and `.modinfo` carries the rest. for (i = 0; i < k->nsym; i++) { diff --git a/ldls/ko.h b/ldls/ko.h index 1533f67..62ae3f3 100644 --- a/ldls/ko.h +++ b/ldls/ko.h @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 #pragma once -// Loading a relocatable object. docs/plan/ldls.md milestone L0. +// Loading a relocatable object. docs/architecture/ldls.md, "The loader". // // A Linux `.ko` is an ordinary ELF64 relocatable object -- the same thing // `gcc -c` produces -- and loading one is four steps with no mystery in any of diff --git a/ldls/kpi/kpi.c b/ldls/kpi/kpi.c index f01cbb4..05ba455 100644 --- a/ldls/kpi/kpi.c +++ b/ldls/kpi/kpi.c @@ -7,7 +7,7 @@ // and there is no other way to express that. What each one *does* is written // here from the published behaviour and from what the driver requires of it. // No comment, macro body or implementation has been carried over from -// anywhere. docs/plan/ldls.md §3. +// anywhere. docs/architecture/ldls.md §3. #include "types.h" #include "user.h" @@ -172,7 +172,7 @@ kpi_dma_free_attrs(void *dev, unsigned long size, void *va, // *correct* rather than merely convenient: there is nobody to race with. // // This stops being true the moment a driver has an interrupt thread, and the -// note in docs/plan/ldls.md about sleeping-context fidelity is about exactly +// note in docs/architecture/ldls.md about sleeping-context fidelity is about exactly // this. It is the deepest unfixed assumption in the whole layer. void kpi_nop(void) { } diff --git a/ldls/kpi/kpi.h b/ldls/kpi/kpi.h index 34a70fc..f5e4488 100644 --- a/ldls/kpi/kpi.h +++ b/ldls/kpi/kpi.h @@ -2,7 +2,7 @@ // // The KPI: the Linux symbols a hosted driver actually calls, implemented. // -// docs/plan/ldls.md milestone L3c. L1 and L2 measured what a driver asks for +// docs/architecture/ldls.md, "The KPI". The stub passes measured what a driver asks for // and in what order; this is what answers, for real, against `/dev/pci` and // `/dev/netdev`. // diff --git a/ldls/kpi/net.c b/ldls/kpi/net.c index becac02..13e81c5 100644 --- a/ldls/kpi/net.c +++ b/ldls/kpi/net.c @@ -190,7 +190,7 @@ kpi_dev_addr_mod(void *dev, unsigned int off, const unsigned char *addr, // Become an interface. // -// /dev/netdev is the bridge docs/plan/ldls.md calls the class bridge, built +// /dev/netdev is the bridge docs/architecture/ldls.md calls the class bridge, built // and proved at L3a by `netecho` with no hardware behind it at all. Here there // is: the same four calls, answered by a Linux driver. int diff --git a/ldls/stub.h b/ldls/stub.h index 9dc633c..90f8df0 100644 --- a/ldls/stub.h +++ b/ldls/stub.h @@ -2,7 +2,7 @@ // // Stubs: giving every undefined symbol an address that says who it was. // -// docs/plan/ldls.md milestone L1. L0 measured what a module asks for; this is +// docs/architecture/ldls.md, rule 5. The loader measures what a module asks for; this is // what answers. Every name resolves to a small block of generated code, and // when the driver calls it the block says which name it was and what arguments // it came with. diff --git a/ldls/test/refmod.c b/ldls/test/refmod.c index c19ce93..9a2c423 100644 --- a/ldls/test/refmod.c +++ b/ldls/test/refmod.c @@ -2,7 +2,7 @@ // // The loader's test object. **Ours, on purpose.** // -// docs/plan/ldls.md rule 1 says nothing GPL ever enters an image we build, and +// docs/architecture/ldls.md rule 1 says nothing GPL ever enters an image we build, and // the same rule applies to the tree itself: there is no `.ko` committed here // and there never will be. So the thing `make ldls-check` loads is this file, // compiled with `-c`, which produces exactly what a `.ko` is -- an ELF64 diff --git a/tools/python/ldls_check.py b/tools/python/ldls_check.py index 073b3b6..39c3997 100644 --- a/tools/python/ldls_check.py +++ b/tools/python/ldls_check.py @@ -3,7 +3,7 @@ # # Check the LDLS loader against an object we compiled ourselves. # -# docs/plan/ldls.md milestone L0. The thing being tested is ldls/ko.c; the +# docs/architecture/ldls.md, "The loader". The thing being tested is ldls/ko.c; the # input is ldls/test/refmod.c compiled with -c, which is exactly what a `.ko` # is. **No kernel module is involved and none is in this tree**, which is rule # 1 applied to the repository as well as to the image.