nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

ldls: L4 was free after all, and the plan closes

Browse source 
L4 was listed as "free -- H2 built all of this" and never actually tested,
which is a claim rather than a result. Tested now, and it holds:

    ldlsd running     ping 2/2; holds index 7 and DMA slots 3 and 4
    killed            interface DOWN, device released, bus mastering
                      cleared, DMA reclaimed -- back to vgpud's 3 blocks
    ping              fails honestly: sendto: -5
    a second ldlsd    ping 3/3, and the arena hands back the same addresses

pciuser_release_all() and netuser_release_all() sit beside each other in
exit() for the same reason, and that they apply unchanged to a foreign driver
is the whole point of having built them against our own.

With that, every milestone on ldls.md is done, so it leaves docs/plan/ for
docs/architecture/ -- the rule docs/plan/README.md states and the one that
keeps the two directories meaning different things. The new document describes
what is, not what was planned: the five rules, the four pieces, what it costs
(polling, no stack protector, locks that are no-ops), that it can be killed,
and a "what it does not do" that leads with one driver, one kernel version,
tried once.

Sixteen source files pointed at docs/plan/ldls.md and would have been dangling
the moment it moved. All repointed, and the milestone references -- which the
architecture doc deliberately does not have -- rewritten to name sections
instead, so a reader lands somewhere real.

The register row closes with it. hybrid.md freed a slot, ldls.md took it, and
it is now free again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
Nexoniarz 2026-09-14 13:37:04 +02:00
commit f38eb8a7fd
22 changed files with 278 additions and 929 deletions
Download patch file Download diff file Expand all files Collapse all files

15
docs/plan/1.0.md
View file

@ -95,16 +95,17 @@ The galling ones. The work exists, is tested, and nothing uses it.
|---|---|
| **`foundations.md`** | Item 1 built. Items 2 (a shareable memory object), 3 (`AF_UNIX` in the filesystem namespace), 4 and 5 open. **Wayland stops at item 2.** |
| ~~**`hybrid.md`**~~ | **Closed.** All four mechanisms built; H1, H2 and H3 done. H4 — the USB eviction — is *withdrawn* rather than outstanding: USB is the bus peripherals arrive on, not a peripheral, and an eviction that has to leave an identical driver behind has moved nothing. See [hybrid.md](hybrid.md), "Decided: USB stays in the kernel". |
| **`ldls.md`** | L0, L1, L2 and L3a done — the loader handles 2,440 stock modules; 449 of 500 run `init_module`; `8139too`'s `probe()` runs to completion; and `/dev/netdev` lets a process be an interface, with `ping` working over one. L3b done too — `ldlsd` runs the same loader on SchOS and gets a real driver's `probe()` to return. **L3 done: `ping` works over a Linux driver.** `8139too`, loaded by our loader, running in a SchOS process, driving QEMU's rtl8139 through `/dev/pci`, carrying DHCP and ICMP both ways on a subnet reachable only through it. One driver, one kernel version, tried once — and the network transport for the debugger is the only thing still open on any plan. |
| ~~**`ldls.md`**~~ | **Closed.** L0 through L4 built: the loader handles 2,440 stock modules, `8139too` runs in a SchOS process driving QEMU's rtl8139 through `/dev/pci`, `ping` works over it, and killing the host releases the device and DMA so a replacement takes over. Described in [../architecture/ldls.md](../architecture/ldls.md). One driver, one kernel version, tried once. |
| **Schcom v0** | 715 lines that composite, over a wire format that is superseded. Frozen: it runs, nothing more is added, and it is kept only because it is the one thing in the tree that has ever put a client's pixels on screen. |
| **`debugger.md`** | D0 through D4 built and described in [../architecture/debugging.md](../architecture/debugging.md). Only the network transport is open — a userspace daemon bridging TCP to `/dev/dbg`, which is deliberately last. |
**The row LDLS spends.** `hybrid.md` closed above — H3 built, H4 withdrawn —
and [ldls.md](ldls.md) takes the slot it freed. Worth being honest that this is
the largest thing ever to enter this register, and that it is **explicitly not
part of 1.0**: nothing in the acceptance test needs a foreign driver. It is
here because the mechanisms it uses are finished and idle, and because the
alternative to it is writing a driver for every device that exists.
**The row LDLS spent, and gave back.** `hybrid.md` closed — H3 built, H4
withdrawn — and `ldls.md` took the slot it freed. It has since closed too, so
the slot is free again. Worth being honest that this is
the largest thing ever to enter this register, and that it was **explicitly not
part of 1.0**: nothing in the acceptance test needs a foreign driver. It was
taken on because the mechanisms it uses were finished and idle, and because
the alternative to it is writing a driver for every device that exists.
**The row the debugger spent.** Adding one requires closing one, and *A released
PCI device* closed when H2 landed. It is worth saying plainly that the debugger

7
docs/plan/README.md
View file

@ -43,7 +43,7 @@ And the one the hybrid mechanisms were built for:
| | |
|---|---|
| **[ldls.md](ldls.md)** | The Linux Driver Layer for SchOS. Someone else's drivers, in a process, reached through `/dev/pci` — because no project can write a driver for every device that exists. **Not started.** |
| ~~ldls.md~~ | **Built.** The Linux Driver Layer for SchOS: someone else's drivers, in a process, reached through `/dev/pci`. `ping` works over `8139too`. The description moved to [../architecture/ldls.md](../architecture/ldls.md). |
And one that is a tool rather than a capability:
@ -106,8 +106,9 @@ mechanism rather than a hole shaped like something foreign. The compositor is
still a demanding consumer — it is no longer the *only* one available.
Four mechanisms, all built. One eviction done, the second withdrawn with its
reasons recorded, and the foreign driver host — [ldls.md](ldls.md) — is what
they were for.
reasons recorded, and the foreign driver host they were for is
[../architecture/ldls.md](../architecture/ldls.md) — built, and carrying
packets.
### 4. `wayland.md` — with the GPU driver either side of the seam

888
docs/plan/ldls.md
View file

@ -1,888 +0,0 @@
# LDLS — the Linux Driver Layer for SchOS
SchOS cannot write a driver for every device that exists. No project can. The
only route to the hardware support an established system has is to run someone
else's drivers, and in practice "someone else's drivers" means Linux's.
**LDLS itself is Apache-2.0**, like everything else in this tree, and stays that
way because it never contains a line of Linux. What the *user* combines it with
on their own machine is a separate question, and
[../licensing.md](../licensing.md) §6 is where that one is answered.
[hybrid.md](hybrid.md) built the mechanisms for this and said so in as many
words — MMIO, DMA, interrupts and config space to a process, with SchOS's own
drivers as the first consumers so the interface would be general rather than a
hole shaped like something foreign. `vgpud` proved it. **LDLS is what those
mechanisms were built for.**
---
## The rules
These are the spine. Everything below is an implementation of them, and
anything that violates one is wrong however well it works.
### 1. Nothing GPL ever enters an image we build
The driver is **the user's**, obtained by them, loaded on their machine. SchOS
ships the layer and never the thing it loads. This is not a new position — it
was decided on 2026-08-08 and is written up in
[../architecture/devices.md](../architecture/devices.md) and
[../licensing.md](../licensing.md) §2:
> Apache-2.0 code and GPL-2.0-only code cannot be combined into one
> distributable work. It is not a matter of taste or of how carefully the files
> are kept apart in the tree.
`RAMDISK_EXCLUDE` and `NOTPORTS` are the existing mechanisms that keep this
true mechanically rather than by intention, and LDLS must stay inside them.
### 2. It runs in a process, never in the kernel
Two independent reasons, and either alone would decide it.
**Legally**, a module linked into our kernel is the strongest derivative-work
argument that exists — it is the case the FSF has always made. A module
combined with our shim *inside a user process, on the user's machine, at their
direction* is a different thing at arm's length, and it is the position
FreeBSD has shipped from for about a decade: a BSD-licensed `linuxkpi` in base,
GPL `drm-kmod` installed by the user from ports.
**Practically**, this is foreign code. A driver we did not write, cannot fully
audit and did not choose the quality of, running with the machine's memory in
reach, is exactly the code that should not be able to take the machine down.
Crash isolation is not a nicety here; it is the reason.
### 3. Facts, not expression
The shim has to present Linux's internal API, and that API is expressed in GPL
headers. The line LDLS holds:
| | |
|---|---|
| A symbol exists, and takes these arguments | **fact** — reproduce it |
| A struct has these fields in this order at these offsets | **fact** — a compiled driver indexes them, so there is no other way to express it. And mostly it need not be in our source at all; see BTF below |
| The comment above it | **expression** — never |
| The macro body, the inline function body, the implementation | **expression** — never |
Every KPI file records where its shape came from. Not decoration: it is the
difference between a clean-room reimplementation and a transcription, and the
only time anyone will ask is the time it matters.
This is the same discipline [../licensing.md](../licensing.md) §5 already
describes — *"the xHCI, USB, PCI, fbdev and ELF specifications were all read
and implemented directly, which is why those drivers have no licence
entanglement at all"* — applied to a case where the specification is thinner.
> **Not legal advice, and it should be checked by someone whose job that is
> before anything ships.** The reasoning rests on interfaces being outside
> copyright (EU Software Directive Art. 1(2)) and on *Google v. Oracle* (2021),
> and neither is a blanket permission.
#### The layouts do not have to be ours to write
The uncomfortable row above is the struct layouts, and it turns out to be
largely avoidable. Kernels built with `CONFIG_DEBUG_INFO_BTF` publish
`/sys/kernel/btf/vmlinux`, which carries the layouts *and* the function
prototypes as generated binary type data about the kernel that is installed:
```
struct pci_dev {
struct list_head bus_list; /* 0 16 */
struct pci_bus * bus; /* 16 8 */
[97814] FUNC 'pci_iomap' type_id=97813 linkage=static
[97815] FUNC_PROTO '(anon)' ret_type_id=108 vlen=4
```
So LDLS reads them at load time, on the user's machine, from the user's own
kernel. Our source contains neither, and there is nothing to have transcribed.
It also mostly dissolves rule 4 below, because BTF describes the kernel that is
actually there rather than one we pinned by guessing.
**BTF gives shape, not behaviour.** What `napi_complete_done` *does* is still
ours to implement from documentation, and that is the whole of the real work.
A kernel without BTF needs hand-written layouts for a named version, which is
why this section stays rather than being replaced.
### 4. One kernel version at a time, pinned and named
*(Mostly superseded by BTF, above — but it still governs any kernel that does
not carry it.)*
Linux states in writing that it has no stable in-kernel ABI —
`Documentation/process/stable-api-nonsense.rst` exists to say so. Believing
them is cheaper than being surprised by them.
So LDLS targets **one version**, says which, and is re-pinned deliberately.
FreeBSD's LinuxKPI does exactly this per drm-kmod release. A layer that claims
to support "Linux drivers" without a version is claiming something nobody has
ever delivered.
### 5. The unresolved-symbol list is the specification
Do not guess the API surface. **Load a real module and let the loader say what
is missing.** That list is the work queue, it is exact, and it is available
before a single line of shim is written.
---
## What it is made of
Four pieces, of which three are ordinary and one is the project.
### The loader — ours, clean, and small
A `.ko` is a relocatable ELF64 object. Loading one is: lay out the allocatable
sections, apply the relocations, resolve the undefined symbols against a table,
and find the entry point. SchOS already parses ELF in `kernel/fs/exec.c`, for a
different object type but with the same plumbing.
There is no licence question here at all — ELF is a published specification and
so is the relocation set.
### The KPI — the shim, and the actual work
Our Apache-2.0 reimplementation of the Linux symbols a given driver needs,
organised under Linux's own header names so it is navigable: `kpi/slab.h`,
`kpi/pci.h`, `kpi/netdevice.h`. Rule 3 governs every file.
### The back end — already built
`/dev/pci`: `PCIIOC_INFO`, `PCIIOC_CLAIM`, `PCIIOC_CFG`, `mmap` of a BAR,
`PCIIOC_DMA`, `PCIIOC_IRQ`. A Linux driver's `pci_iomap`, `dma_alloc_coherent`
and `request_irq` land on these, which is the whole reason they were specified
in terms of PCI devices rather than in terms of drivers.
### The class bridge — the one new kernel interface
This is the piece that does not exist, and it is the interesting one.
A hosted driver produces packets in a process. SchOS's network stack consumes
them through `struct netdev_ops` in the kernel — `transmit`, `receive`, `mac`,
`link_up`, four entries and nothing else. So what is missing is a **netdev
whose backend is a process**: the kernel publishes the vtable, the four calls
are forwarded over a descriptor, and lwIP never learns that the driver is not
in the kernel.
It is the exact analogue of what `display_ops` was for the GPU, and
`netdev.h`'s own opening comment already anticipates the shape — *"the stack
should ask for the network interfaces and get whatever is bound … exactly as a
compositor asks for the display and does not care which driver answers."*
One device, one process, claimed and released, on the `/dev/pci` discipline.
---
## Milestones
| | | |
|---|---|---|
| **L0** | **The loader.** Parse a real `.ko`, lay it out, relocate it, and **print every unresolved symbol**. Nothing executes. The output is the measured API surface. | **done** — `ldls/ko.c`, `make ldls-check` |
| **L1** | **Stubs.** Every symbol resolves, to a function that logs its name. `init_module` runs, which yields the *call order* — the second thing you need and the other thing you cannot guess. | **done** — `ldls/stub.c`, `ldls-run` |
| **L2** | **The hardware.** `probe()` receives a working `struct pci_dev`, maps its BARs, allocates coherent DMA and registers an interrupt, all through `/dev/pci`. The driver reaches its device. | **done**, except `/dev/pci` — see below |
| **L3a** | **The class bridge.** `/dev/netdev`: a process claims it and becomes an interface. `ping` over a *process*. | **done** — `kernel/dev/netuser.c`, `cmd/netecho` |
| **L3b** | The SchOS-side host: `ldls-run`'s loader and BTF work, as a program on SchOS. | **done** — `cmd/ldlsd`; `/dev/pci` is L3c's half |
| **L3c** | The fifteen symbols, implemented rather than stubbed, against a real rtl8139. | **done** — `ldls/kpi/`; the driver reads the real MAC |
| **L3d** | The packet path: `ndo_open`, `ndo_start_xmit`, receive. | **done** |
| **L3** | **A packet.** The two joined: **`ping` over a Linux driver.** This is the milestone that proves the whole idea and the one that cannot be skipped. | **done** |
| **L4** | **Kill it.** The host process dies, the kernel clears bus mastering and reclaims the DMA, a new host starts and the interface comes back. | free — H2 built all of this |
L4 is listed for completeness rather than as work. `pciuser_stop()`,
`pciuser_release_all()` and `KeepAlive` already do it, and that they apply
unchanged to a foreign driver is the point of having built them against our
own.
### The first target: `8139too`, not `r8169`
The obvious choice is a Realtek `r8169`, because that silicon is everywhere.
It is the wrong first choice for one decisive reason: **QEMU cannot emulate
it**, so L0 through L3 would have no way to be tested, and a driver layer
developed against hardware you cannot run is a layer nobody can debug.
QEMU emulates `rtl8139`. `8139too` is small, self-contained, needs no firmware
blob, and uses the thin `mii` helpers rather than dragging in phylib — which
`r8169` does, and which is a substantial surface on its own.
So: **`8139too` to build the layer, `r8169` to prove it was general.** The
second is the interesting one and it is much cheaper once the first works.
This is the `/dev/pci` lesson applied backwards. There, a native driver was the
first consumer so the interface would not be shaped like something foreign.
Here the interface is fixed by Linux, and what is chosen instead is the
*smallest real consumer that can be tested* — because the risk is no longer
designing the wrong interface, it is building a hundred stubs nobody has ever
run.
---
## L0, and what it measured
`ldls/ko.c` is the loader: index the sections, place the allocatable ones,
resolve the undefined symbols against a table, apply the relocations. It is
freestanding — no libc, no allocation, no I/O — so the same source builds as a
host tool today and as the SchOS-side host later, which is the arrangement
`mkschfs` and the kernel already share.
```
$ ldls-inspect 8139too.ko
119288 bytes, 55 sections, 363 symbols
module 8139too
license GPL
describes RealTek RTL-8139 Fast Ethernet driver
loads to 33988 bytes
relocations: 1980
R_X86_64_64 363
R_X86_64_PC32 991
R_X86_64_PLT32 428
R_X86_64_32S 169
R_X86_64_PC64 29
undefined symbols: 94
...
stops at: unresolved: __fentry__
```
**The licence is printed first, before anything is loaded**, because it is the
one property that decides whether the object may be on this machine at all and
it is a plain string in `.modinfo`. Rule 1 is easier to keep when the answer is
on screen.
### What the measurement says
**94 symbols** is the API surface of `8139too` on Linux 6.18.46. Not an
estimate — the list, exactly, and it is L1's work queue. It is also smaller
than it looks, because a good fraction of it is not API at all: `__fentry__`,
`__stack_chk_fail`, `__ref_stack_chk_guard`, `__x86_return_thunk` and
`__fortify_panic` are compiler and instrumentation artefacts that stub out in a
line each.
**Five relocation types**, and a corpus run says that is the whole set: over
**2,440 real modules** from a stock 6.18.46 tree, every one lays out and
relocates to completion with nothing unhandled. That includes `nvidia-modeset`
at 7,069 sections and 74,305 relocations, which is what retired the loader's
fixed section table — the caller supplies it now, so there is no limit to run
into rather than a generous one.
Relocating against stubs is a separate question from listing what is missing,
and it is the one that says the loader is finished: with nothing resolved, the
first unknown name stops everything and no relocation type past it is ever
reached.
### Nothing foreign is in this tree
`make ldls-check` runs the loader against `ldls/test/refmod.c` — ours, compiled
with `-c`, written to contain one of each thing the loader has to get right:
undefined functions *and* undefined data, `.text` with calls, `.data` holding a
pointer into `.rodata`, a `.bss` that must arrive zeroed, and a `.modinfo`
shaped the way a module's is. The modules above were read from the host's own
`/nix/store` and never copied here; `.gitignore` refuses `*.ko` outright so
that cannot happen by accident.
---
## L1, and the driver's own voice
Every undefined symbol resolves to a trampoline — twenty generated bytes that
load the symbol's index and jump to a shared thunk, which saves the argument
registers and reports the call. `init_module` is then simply called.
```
$ ldls-run 8139too.ko
module 8139too
license GPL
wants 94 symbols
loaded 33988 bytes at 0x416f8000, 94 stubs at 0x40377000
calling init_module at 0x416fc020
1 _printk(0x416fb6c0, 0, 0)
| 8139too: 8139too Fast Ethernet driver 0.9.28
2 __pci_register_driver(0x416fb400, 0x416fff80, 0x416fbdbb)
init_module returned 0 after 2 stub call(s)
```
**That second line is the driver's own banner**, printed through a `_printk`
that is ours, from a string in the module's own `.rodata` that a relocation put
there. There is no stronger evidence that a load worked than the loaded thing
speaking.
The stubs return zero rather than stopping, which is a deliberate departure
from what this milestone originally said. Stopping at the first call tells you
one name; carrying on tells you the *sequence*, and the sequence is the thing
that cannot be read off a symbol table. A driver that then dereferences the
zero faults, and that is a result too — it says exactly how far it got before
it needed the kernel to have really done something.
### The constraint nobody would guess
The first run failed on a relocation, not on anything to do with stubs:
```
ldls-run: relocate: a signed 32-bit relocation does not fit
```
A module is compiled `-mcmodel=kernel`, so the compiler emits `R_X86_64_32S` —
a **signed 32-bit absolute address** — for anything it believes lives in the
top 2 GB. Load the object at an ordinary `mmap` address around `0x7f…` and
those relocations have nowhere to go. Linux avoids this by giving modules a
region within 2 GB of the kernel text; the harness uses `MAP_32BIT` to reach
the same place from the other end.
**On SchOS this costs nothing** — a process's mmap area starts at `0x40000000`
and is already inside the window — but it is a real constraint on where the
LDLS host may put a module, and it would have been found much later and much
more confusingly without a loader that checks its own arithmetic.
### What running 500 of them says
| | |
|---|---|
| `init_module` ran and returned | **449** |
| faulted on a stub's zero | 22 |
| no `init_module` at all | 29 |
| **loader errors** | **0** |
The 22 are not failures of the loader; they are drivers that needed an
allocation to come back non-null. The 29 without an entry point are mostly
library modules that exist to be depended on.
### And what it says about the work queue
Counting *calls* is the wrong metric — one module walking an rbtree called
`rb_first` sixty-four times and told us nothing. Counting **how many modules
need a symbol** is the useful one, and over 110 modules it comes to this:
| symbol | modules |
|---|---|
| `__kmem_cache_create_args` | 25 |
| `crypto_register_template` | 11 |
| `_printk` | 10 |
| `crypto_register_alg` | 10 |
| `__register_nls` | 9 |
| `crypto_register_shash` | 4 |
| `af_alg_register_type` | 4 |
| `register_filesystem`, `misc_register`, … | 2 each |
**Forty-four distinct symbols would carry every one of those 110 modules
through `init_module`.** That is a small enough number to be a plan rather than
an estimate, and it is not a number anybody could have guessed — the slab
allocator at the top, and a long tail of registration calls that mostly want to
record a pointer and return zero.
It says nothing about `probe()`, which is where the real surface is and which
the kernel, not the module, is supposed to call. Reaching that is L2.
---
## L2, and being the PCI core
In Linux, `__pci_register_driver` ends at the PCI core, which matches the
driver's `id_table` against the bus and calls `probe()`. There is no PCI core
here, so LDLS keeps the pointer that call was given and plays the part itself.
To do that it has to know where `pci_driver.probe` is — and **it asks the
kernel rather than being told by us**. `ldls/btf.c` reads
`/sys/kernel/btf/vmlinux`: 122,279 types, every struct layout and every
function prototype, as generated data describing the kernel that is installed.
```
$ ldls-types /sys/kernel/btf/vmlinux --need
pci_driver probe offset 16 size 8
pci_driver id_table offset 8 size 8
pci_dev vendor offset 60 size 2
pci_dev resource offset 1000 size 1088
resource start offset 0 size 8
...
all present
```
Cross-checked against `pahole` on the same file, field for field. **No offset
in this is ours**, which is rule 3's uncomfortable row disappearing rather than
being argued about.
### What the driver then did
```
pci_driver at 0x41662400, from BTF: probe +16, id_table +8
driver name "8139too"
devices it claims:
10ec:8139 class 000000/000000
... and 16 more
calling probe(pci_dev 10ec:8139, ...)
1 alloc_etherdev_mqs(0x368, 0x1, 0x1)
2 pci_enable_device(...)
3 pci_request_regions(...)
4 pci_set_master(...)
5 pci_iomap(pdev, 1, 0)
6..544 ioread8 / iowrite8 -- 0x88, 0x8a, 0x8c, 0x8e, over and over
545 dev_addr_mod(...)
547 netif_napi_add_weight_locked(...)
549 timer_init_key(...)
550 register_netdev(...)
551 netdev_info(...)
| %s at 0x%p, %pM, IRQ %d
probe returned 0 after 552 stub call(s)
```
`10ec:8139` is the real RTL8139 id. The driver mapped **BAR 1**, which is the
one we marked `IORESOURCE_MEM`. And those 538 alternating byte accesses with
values stepping `0x88, 0x8a, 0x8c, 0x8e` are **serial EEPROM clocking** — the
driver bit-banging the chip to read its MAC address, then handing the result to
`dev_addr_mod`. It registered a NAPI instance, started a watchdog timer,
registered the interface, printed its "found it" line, and **returned 0**.
That is a complete driver bring-up, on shape alone.
### Fifteen symbols
The whole of that probe touches **15 distinct symbols**, and 538 of the 552
calls are two of them:
| | |
|---|---|
| `ioread8` / `iowrite8` / `ioread32` | 540 calls, 3 symbols |
| PCI setup | `pci_enable_device`, `pci_request_regions`, `pci_set_master`, `pci_iomap` |
| net | `alloc_etherdev_mqs`, `dev_addr_mod`, `netif_napi_add_weight_locked`, `register_netdev`, `netdev_info` |
| other | `mutex_lock`, `mutex_unlock`, `timer_init_key` |
L1 said 44 symbols carry 110 modules through init. L2 says 15 carry this one
through probe. **Neither is a number anyone would have guessed**, and both came
out of running the thing rather than reading about it.
### What is not done
`probe()` receives a `pci_dev` whose *shape* is right and whose *contents are
invented*. The BARs are numbers this harness made up; `pci_iomap` returns a
zeroed page, not a mapping; nothing has touched hardware and nothing could,
because this runs on the build machine.
**The half of L2 that says "all through `/dev/pci`" belongs to the SchOS
side**, where those calls have somewhere real to land. What is settled is the
part that had to come first: that probe can be reached at all, with a struct
the driver accepts, built from offsets nobody wrote down.
---
## L3a: a network, in a process
L3 is one sentence in the milestone table and three pieces of work. This is the
first, and it is the one the rest plugs into: **`/dev/netdev`, an interface
whose backend is a process.**
The kernel side is `netdev_ops` and nothing else — `transmit`, `receive`,
`mac`, `link_up`, the same four entries `e1000` publishes. A process claims the
device, supplies a MAC, and from then on `read()` gives it frames the stack
wants sent and `write()` hands it frames that arrived. It is `/dev/pci`'s shape
with the arrows reversed: that hands a device *to* a process, this lets a
process *be* one.
`cmd/netecho` is the proof, and it is deliberately **not** a driver — it has no
hardware and claims none. It answers every frame itself: DHCP so the interface
gets an address at all, ARP so the stack can find it, ICMP so it can be pinged.
```
# netecho &
netuser: ldls0 is a process now, 2:4c:44:4c:53:1
net: ldls0 attached as interface 1, asking for an address
netecho: this process is now an interface. I am 10.9.0.1 and will offer 10.9.0.2
netecho: answered DHCP, offering an address
# cat /proc/netif
interface ldls0
driver netuser
address 10.9.0.2
gateway 10.9.0.1
# ping 10.9.0.1 3
netecho: answered an ARP
netecho: answered a ping
64 bytes from 10.9.0.1: icmp_seq=1 time=30 ms
64 bytes from 10.9.0.1: icmp_seq=2 time=10 ms
64 bytes from 10.9.0.1: icmp_seq=3 time=10 ms
3 packets transmitted, 3 received, 0% packet loss
```
**The interface got its address by DHCP from a process**, and then answered
ping from one. lwIP never learns that the thing on the other side of
`netdev_ops` is not a driver — which is the entire property the bridge exists
to have.
### It dies the way it should
The same test, continued:
```
# kill 8
# schdbg netuser
interface ldls0 DOWN, nobody holds it
# ping 10.9.0.1 1
ping: sendto: -5
0 packets transmitted, 0 received
# netecho &
# ping 10.9.0.1 2
2 packets transmitted, 2 received, 0% packet loss
```
`link_up()` answers false the moment nobody holds the descriptor, so the stack
refuses to send rather than swallowing packets into a driver that is not there
— and when a new process claims it, the *same* interface comes back, keeping
its address. That is H2's property applied to a network interface, and it came
free: `netuser_release_all()` sits beside `pciuser_release_all()` in `exit()`
for exactly the same reason.
### One new thing the kernel needed
`netinit()` starts DHCP on the devices that exist when it runs. An interface
created later — which is how an LDLS-hosted driver arrives — missed that
entirely and would have come up with no address and no way to get one. So
`net_attach_late()` exists, in `kernel/net/net.c` because that is the only file
allowed to include lwIP's headers.
---
## L3b: the same loader, on SchOS
`ldls/ko.c`, `ldls/btf.c` and `ldls/stub.c` were written freestanding so they
would build unchanged on both sides. `cmd/ldlsd` is the collection of that
promise: the same three files, linked against `lib/` instead of a libc, in an
ordinary SchOS process.
```
# ldlsd /lib/modules/8139too.ko /lib/modules/btf
/lib/modules/8139too.ko
module 8139too
license GPL
wants 94 symbols
defused 6 %gs stack-guard reference(s)
loaded 33988 bytes at 0x000000004001f000
calling init_module
1 _printk(...)
| 8139too: 8139too Fast Ethernet driver 0.9.28
2 __pci_register_driver(...)
init_module returned 0 after 2 stub call(s)
btf 122279 types
layout pci_driver.probe +16 pci_dev 2680 bytes
driver "8139too"
devices it claims:
10ec:8139
... and 18 more
calling probe
1 alloc_etherdev_mqs(0x368, 0x1)
probe returned -12 after 1 stub call(s)
```
Byte for byte the behaviour of the host harness, which is what the shared
source was for. Two predictions from earlier milestones came good without any
work: a process's mmap area starts at `0x40000000`, so the module lands inside
the window `R_X86_64_32S` needs, and the kernel never sets `EFER.NXE`, so every
mapping is executable.
That second one is **a gap, not a feature**. A process being able to execute
its own heap is normally a thing an operating system prevents, and LDLS is
quietly relying on its absence. It belongs on Gate 0's list.
### The one thing that did not just work
The first run on SchOS faulted before probe's first call:
```
pid 8 ldlsd: trap 14 err 5 on cpu 0 rip 0x4002138e addr 0xffff80007de5fc60
```
`err 5` is present, read, user mode, and the address is a *kernel* one. The rip
resolved to `.text+0x206e`, which is:
```
mov %gs:0x0(%rip),%rax # rtl8139_init_one+0x26
```
Linux builds modules with `-mstack-protector-guard-reg=gs`, so the stack canary
is fetched through the GS segment. That is an ABI between the kernel and its
own per-CPU area, and **a host that is not that kernel cannot provide it** — on
SchOS the GS base belongs to the kernel, so a user-mode read of it faults on a
kernel address.
`ko_defuse_gs()` rewrites those references to `xor %reg, %reg`. There are two
forms and the second is easy to miss: the prologue *loads* the guard with `mov`
(`8b`), and the epilogue *compares* it with `sub` (`2b`) followed by a `jne` to
`__stack_chk_fail`. `xor` is correct for both — it makes the saved canary zero,
and it sets ZF so the epilogue's branch falls through exactly as a passing
check does. Six sites in `8139too`.
It checks every site before patching any, and refuses the whole operation on
anything it does not recognise. A loader that half-understands an instruction
should not be editing it — and that refusal is what found the `sub` form, by
declining to guess.
### Getting the files there
`make ldls-stage KO=… BTF=…` copies a module and its kernel's BTF into the
local sysroot. It is deliberately not part of `all`, `.gitignore` refuses
`*.ko`, and `make ldls-unstage` removes them again. An image built between
those two commands has somebody else's GPL code in it and must not be
redistributed — which is exactly what [../licensing.md](../licensing.md) §6
says about the combination, and why it is a thing you type on purpose.
---
## L3c: a Linux driver, driving real hardware
`ldls/kpi/` is the shim: 41 of `8139too`'s 94 symbols, implemented rather than
logged. Register access goes to a mapped BAR; `pci_enable_device`,
`pci_iomap`, `pci_request_regions` and `pci_set_master` go to `/dev/pci`;
allocation goes to `mmap` and `PCIIOC_DMA`; `register_netdev` goes to
`/dev/netdev`. Every file records its provenance, per rule 3.
```
# ldlsd /lib/modules/8139too.ko /lib/modules/btf
found 10ec:8139 at 00:05.0
bar 0 0x000000000000c000 256 io
bar 1 0x0000000081060000 256 mem
netdev 2688 bytes
calling probe, with 41 of 94 symbols implemented
kpi: net_device 2688 bytes + 872 private
kpi: mapped BAR 1 -- phys 0x0000000081060000, 256 bytes
kpi: the driver read a MAC off the hardware: 52:54:00:ab:cd:ef
net: ldls0 attached as interface 1, asking for an address
netuser: ldls0 is a process now, 52:54:0:ab:cd:ef
kpi: registered as an interface, with the hardware's own MAC
probe returned 0 after 0 stub call(s)
```
**`52:54:00:ab:cd:ef` is what the machine was configured with.** Nothing here
invented it: the driver bit-banged the card's EEPROM, serially, through a BAR
mapped by `/dev/pci`, and handed back the address it found. That it matches
means the loader, the relocations, the BTF offsets, the BAR mapping and the
register accessors are all right at once — which is why it was chosen as the
thing to aim at.
**Zero stub calls.** Every symbol that probe reaches is implemented.
### The bug a foreign driver found in its first minute
The first run against real hardware wrote to address `0x4e` and faulted.
`0x4e` is 13 below where the driver meant to write, and 13 is `EACCES`:
`pci_iomap` had returned `-13` as a pointer, because the check for a failed
`mmap` looked only for `-1` while SchOS returns `-errno`.
Underneath that was the real one. `sys_mmap` rounds a length up to a page —
the only granularity an MMU has — and `pciuser_mmio_ok()` compared that against
the BAR's own size. rtl8139's registers are **256 bytes**, the request became
4096, and the answer was `-EACCES` for a device the process legitimately held.
**A BAR smaller than a page could never be mapped at all**, and nothing had
noticed because the only consumer until now was virtio-gpu, whose BARs are
large. The check is now against the pages the BAR lies in. The widening is
real and named in the code: a sub-page BAR hands the process the whole page it
sits in, which is forced by the hardware rather than chosen.
That is precisely what [hybrid.md](hybrid.md) said a foreign driver would be
good for — *"what it asks for is what a foreign driver would ask for too"* —
arriving as a concrete bug rather than as a prediction.
### Register access is checked
`kpi_io_ok()` bounds every `ioread`/`iowrite` against the mapping `pci_iomap`
returned, reports the first few that miss, and returns without touching
memory. Two reasons: a stray access would otherwise be a page fault with no
context, and **there is no IOMMU** — a hosted driver's wild writes are exactly
the class of thing that has to be caught early or not at all.
It is also what turned the bug above from a crash into two lines naming the
offsets, which is how the `-13` was spotted.
### What is not done
`probe()` succeeds, so the driver is up and the interface exists with the
hardware's own MAC. **No frame has moved.** `ndo_open` is never called, there
is no `ndo_start_xmit`, and nothing services the receive side — so `ldlsd`
releases the interface rather than leaving it up:
```
the driver is up on real hardware and has an interface.
carrying frames is the half of L3 that is not built, so the
interface is released rather than left up and silent.
```
A link that reports itself up with nothing draining its queue is a black hole:
the stack would send DHCP into it for ever and every layer above would believe
the network was merely quiet. Down is the true answer until packets move, and
that is L3d.
---
## L3d: ping, over a Linux driver
```
# ldlsd /lib/modules/8139too.ko /lib/modules/btf &
kpi: the driver read a MAC off the hardware: 52:54:00:ab:cd:ef
ndo_open returned 0
carrying frames. the interface is ldls0.
# cat /proc/netif
interface ldls0
mac 52:54:00:ab:cd:ef
state up
address 10.9.0.15
gateway 10.9.0.2
# ping 10.9.0.2 4
64 bytes from 10.9.0.2: icmp_seq=1 time=30 ms
64 bytes from 10.9.0.2: icmp_seq=2 time=20 ms
64 bytes from 10.9.0.2: icmp_seq=3 time=20 ms
64 bytes from 10.9.0.2: icmp_seq=4 time=20 ms
```
`10.9.0.0/24` is reachable **only** through `ldls0`, and `ldls0` is `8139too`
running in a SchOS process. The address on it was obtained by DHCP, over the
same driver, before the ping.
`lwIP` → `netdev_ops` → `/dev/netdev` → `ldlsd` → `ndo_start_xmit` → a BAR
mapped by `/dev/pci` → the wire, and the whole way back. Nothing in the kernel
knows a Linux driver is involved; nothing in the driver knows it is not in
Linux.
### It polls
A Linux driver expects its interrupt handler to be called from an interrupt.
`/dev/pci` does deliver them — by blocking `read()`, which needs a thread this
program does not have. So `ldlsd` calls the driver's NAPI poll from a loop
instead: the same work, in a context LDLS can actually provide. It costs
latency and a spinning process, and it is written down here rather than hidden.
### Four things `alloc_etherdev` does that allocating does not
A driver gets its `net_device` from `alloc_etherdev_mqs`, and the function does
considerably more than return memory. Each of these was found by the driver
faulting on it, which is a slow way to learn a list and an extremely reliable
one:
| | |
|---|---|
| `dev->dev_addr` | a **pointer**, aimed at the device's own `perm_addr`. The driver writes its MAC into the chip with `*(u32 *)dev->dev_addr`. |
| `dev->mc`, `dev->uc` | list heads, which are *empty* when they point at themselves. Zeroed means a null `next`, and the receive-mode setup walks them. |
| `dev->_tx` | an array of `netdev_queue`, because `netif_start_queue` clears a bit in `dev->_tx[0].state`. |
| `dev->state` | `__LINK_STATE_START`, set by Linux's `dev_open` **before** it calls `ndo_open`. |
### The two that did not announce themselves
Everything above crashed, which made it easy. Two did not.
**`netif_running(dev)`.** Without `__LINK_STATE_START`, 8139too's receive loop
exited immediately — so the chip received the frame, the driver acknowledged
the interrupt, and nothing came out. Every counter said packets were arriving.
It was found by reading the chip's own `CBR` register through the BAR and
seeing that the hardware had written 600 bytes the driver never collected.
**`memcpy`.** It was still a logging stub while the receive path was being
brought up. The driver copied each frame into its `sk_buff`, the copy did not
happen, and the stack was handed exactly the right *length* of zeroes. DHCP
never completed and nothing anywhere reported an error. **A stub that returns
zero is not safe for a function whose whole job is a side effect** — and the
memory primitives are the worst case, because they fail silently and
plausibly. `ldls/kpi/string.c` exists because of it.
---
## What it will not do
- **It is not a security boundary, and here that is worse than usual.** There
is no IOMMU, so a hosted driver can point its device at any physical address.
For SchOS's own drivers that isolates bugs and not intent, which
[hybrid.md](hybrid.md) says at length. For *foreign* drivers the distinction
between bug and intent is one we are in no position to make at all. This is
the single strongest argument for an IOMMU that this project has produced,
and it should be recorded as such rather than waved past.
- **It implements no licence check, and that is not circumvention.**
`EXPORT_SYMBOL_GPL` is Linux marking *Linux's own* symbols, and the check it
drives protects those. Every symbol LDLS provides is ours, reimplemented from
facts under Apache-2.0, so there is no gate being bypassed because there is
no Linux symbol being exposed. LDLS **reads** the declared licence and prints
it before loading anything, because that is the property that decides whether
an object belongs on a machine at all. See
[../licensing.md](../licensing.md) §6.
- **No firmware loading.** `request_firmware` is a stub that fails. Many
drivers need a blob, most blobs are redistributable only under their own
terms, and that is a second licensing problem which is not this one.
- **No module-to-module dependencies.** One object, no `depmod`, no symbol
exports between hosted modules.
- **No sysfs, procfs or debugfs.** Drivers that only *report* through them will
work; drivers that are *configured* through them will not.
- **No unload.** `module_exit` is never called; the way to stop a driver is to
kill the process, which H2 made a supported operation rather than a failure.
- **No sleeping-context fidelity.** Linux drivers assume a kernel's rules about
what may sleep, what runs in interrupt context and what `spin_lock_irqsave`
costs. A process has different rules, and where they differ the shim will be
wrong in ways that show up as rare races rather than as failures to build.
This is the deepest technical risk in the whole plan.
- **Running init is not working.** L0 proves 2,440 modules lay out and
relocate; L1 proves 449 of 500 run their init to completion. Both say the
*mechanism* is right and neither says a driver would function: every symbol
is a stub that logs and returns zero, so what has been demonstrated is that
drivers ask for things in a readable order, not that anything answers them.
The distance from here to a packet is the rest of this plan.
- **The stubs cannot tell code from data.** Every undefined symbol in a module
is `NOTYPE` in the ELF symbol table, so `jiffies` and `pci_iomap` are
indistinguishable and both get a trampoline. A driver that *reads* `jiffies`
reads trampoline bytes. BTF distinguishes them — `VAR` against `FUNC` — and
that is where the fix belongs rather than in a guess.
- **`ldls-run` is a development harness, not the deliverable.** It runs on the
build machine so that loader failures cost a second instead of a boot. The
thing that ships is the SchOS-side host.
- **The stack protector is gone from a hosted driver.** `ko_defuse_gs()`
removes it because the ABI it depends on cannot be supplied here. That is a
hardening measure lost, on foreign code, in a host with no IOMMU — worth
saying plainly rather than filing as a detail.
- **LDLS relies on SchOS having no NX.** Every mapping is executable because
`EFER.NXE` is never set. If that is ever fixed — and it should be — `ldlsd`
needs a way to ask for executable memory, and nothing in `mmap` currently
offers one.
- **BTF has to come from somewhere.** On Linux it is `/sys/kernel/btf/vmlinux`.
SchOS has no such file and never will — so the user brings the BTF along with
the module, from the kernel the module was built for. Both are theirs,
neither is shipped, and a kernel built without `CONFIG_DEBUG_INFO_BTF` has
neither.
- **L3a proves the bridge, not the driver.** `ping` works over a *process*;
nothing in that test loads a module, and `netecho` is a fake network rather
than a NIC. That is the point of doing it first — if `ping` had not worked
here the bridge would be wrong, and now a failure with a real driver can only
be the driver's side.
- **One interface, one process.** `/dev/netdev` is a single device and a second
claimant gets `-EBUSY`. Two hosted NICs would need it to be indexed like
`/dev/input/eventN`, which is a change of shape rather than of size.
- **The interface is never removed.** Releasing it leaves the device registered
with the link down, because nothing in lwIP supports taking an interface out
of its list. From the stack's point of view that is the same thing; from
`/proc/netif`'s it is a `ldls0` that stays listed after its process is gone.
- **It polls instead of taking interrupts**, per the note above. A busy
interface spins a processor; an idle one wakes 200 times before sleeping.
- **One driver, one kernel version, tried once.** `8139too` on 6.18.46 against
QEMU's rtl8139. That is what the milestone table has always promised and it
is worth repeating next to a working `ping`: nothing here has been tried on a
second driver, a second kernel, or real silicon.
- **Some symbols are still logging stubs.** That is deliberate — an
unimplemented symbol that reports itself keeps the work queue writing itself,
exactly as it did at L1 — but it means a driver reaching one gets zero and
carries on, and `memcpy` proved how quiet that can be.
- **A lock is a no-op.** In one process with no preemption inside the driver
there is nobody to race with, which is correct right up until the driver has
an interrupt thread. That is the sleeping-context assumption named above and
it is the deepest unfixed thing in the layer.
- **It will not support "Linux drivers".** It will support *one driver, on one
kernel version, that has been tried*. Anything beyond that is a claim, and
the list of what has actually been run belongs in the architecture document
when there is one.