# Networking ARP, IPv4, ICMP, UDP, TCP and DHCP, over a driver-agnostic interface class, reached through the same socket syscalls as Unix domain sockets. Verified end to end rather than by inspection: `bootcheck -Net` starts a listener and a UDP responder on the host, tells the guest to talk to them, and asserts on **both** ends — one side alone cannot tell a working stack from one that completes a handshake and drops every byte. --- ## The four layers ``` socket() bind() connect() send() recv() syscalls, one set ┌────────────────┴───────────────────┐ │ AF_UNIX AF_INET │ socknew() dispatches on domain │ kernel/net/unix.c kernel/net/inet.c └────────────────┬───────────────────┘ │ lwIP raw API ┌────────────────┴───────────────────┐ │ lwIP 2.2.0, NO_SYS=1 │ ports/lwip, compiled into the kernel │ port layer: kernel/net/lwip/ │ ~400 lines: a lock, a clock, a netif └────────────────┬───────────────────┘ │ struct netdev_ops ┌────────────────┴───────────────────┐ │ e1000, and any future NIC driver │ └────────────────────────────────────┘ ``` ## The netdev class A driver publishes a [`struct netdev_ops`](../../include/netdev.h) and is then finished with the network stack forever: it never mentions lwIP, and the stack never mentions it. ```c struct netdev_ops { int (*transmit)(struct device *, const void *frame, int len); int (*receive) (struct device *, void *out, int cap); const unsigned char *(*mac)(struct device *); int (*link_up) (struct device *); // optional }; ``` `transmit` returning -1 is **congestion, not failure** — the ring is full and the caller is expected to retry. `receive` never blocks; it returns 0 when nothing is waiting, and the caller decides how to wait. This class was added late, and the reason it had to be is instructive: the stack had reached straight into the driver. `net.c` called `e1000_present()`, the lwIP glue was named `netif_e1000.c`, and the e1000 shipped with `.ops = 0` and a comment saying so. A second NIC driver would have meant editing the protocol stack to add itself. lwIP now gets one netif per bound device, `/proc/netif` lists them all, and the e1000 drives up to four cards rather than refusing the second — verified with two attached, each with its own MAC and its own DHCP lease. ### `netdev_selftest()` `echo n > /proc/sysrq` sends a hand-built ARP request and waits for the reply. It proves the whole driver path in one exchange — descriptor formats, the DD handshake, tail bookkeeping, DMA addresses, the receive ring hand-back, and the MAC filter, because an ARP reply is addressed to this station and does not arrive if the filter is wrong. It was written *before* there was any protocol stack, deliberately: a stack on an unproven driver fails in ways indistinguishable from its own bugs. It belongs to the class rather than to the e1000, because it is the check every new NIC driver will want first. ## lwIP lwIP 2.2.0, fetched by `ports/lwip` and **compiled into the kernel** — it is the one port that installs nothing into the sysroot, because it is kernel code. `NO_SYS=1`, so lwIP has no threads, mailboxes or semaphores of its own. The port layer is `kernel/net/lwip/`: an architecture header, a lock, a clock, an Ethernet netif, and a set of shim headers standing in for the libc lwIP expects. ### Why lwIP and not a BSD stack The licence was not the deciding factor — FreeBSD's `sys/netinet` is BSD-3 too. **It is that FreeBSD's stack is not a library.** `tcp_input.c` alone is 122 KB and reaches into mbufs, `epoch(9)`, netisr, VNET, `uma(9)`, sockbufs and inpcb hash tables. Taking it means porting FreeBSD's kernel infrastructure first and finding the TCP somewhere underneath — the same conclusion already reached about FreeBSD's USB stack. NetBSD was ruled out on licence: its `tcp_input.c` and `tcp_output.c` carry the WIDE Project's 4-clause advertising clause, which Berkeley's rescission does not reach. See [../licensing.md](../licensing.md) §3. lwIP was written for exactly the case SchOS is in: a small operating system with a raw Ethernet driver and no infrastructure to speak of. **The reason not to write one, kept because it is the reason to resist later too:** it is not the packet formats that are hard. It is congestion control, retransmission, window management and twenty years of corner cases that only appear against a real peer. ## The network thread `kernel/net/net.c`. One kernel thread, started by `netinit()`, feeds lwIP received frames and runs its timers. **A thread rather than the interrupt handler, and the reason is the lock:** a call into lwIP can reach the driver, the driver can sleep, and an interrupt handler cannot. So the frame is always *pulled*, by this thread, through `.receive`. The interrupt's only job is to end the thread's wait early. `netdev_rx_signal()` is what a driver calls from its handler; it delivers no frame and reads no ring. Without it a frame that arrived just after a poll sat there until the next clock tick — up to 10 ms, and the largest fixed latency in the system. **The wait stays on the timer tick rather than moving to a channel of its own**, and that is the design rather than an accident. lwIP's timers must still run when the link is silent, so the clock has to remain a floor under the wait, and `sleep()` cannot wait on two things at once. Joining the tick's existing wakeup costs nothing and leaves one wait with one predicate. The flag lives under `tickslock` for the same reason it does not get a lock of its own: `sleep()` releases `tickslock` atomically, so a driver setting the flag and waking under that lock cannot lose the wakeup by arriving between the predicate test and the sleep. A private lock would not be held across the sleep, and that window would be back. `/proc/netif` reports `wake_rx` and `wake_tick` so this is observable rather than assumed — if `wake_rx` stays at zero while frames arrive, the interrupt is not reaching the stack and the thread is running on the clock exactly as it used to. Measured across three pings: `rx_frames` and `wake_rx` both went 2 → 6. `netinit()` runs after `sinit()` because `kthread_create` needs a usable process table, and it returns immediately: everything happens in that thread, which cannot run until the scheduler is entered. DHCP obtains an address at boot. `/proc/netif` reports each interface's address, netmask, gateway, MAC and frame counts, taken at the lwIP seam. The lease announcement goes to the **serial line only**, through `klog()` rather than `cprintf()`. A lease arrives seconds *after* the shell prompt does, so on the display it lands in the middle of whatever the user is doing — which is the same rule that makes `cons_serial_only` start set: the display is the product, the serial line is the diagnostics. Someone who wants to know their address reads `/proc/netif`. > It is `/proc/netif` and not `/proc/net` deliberately: on Linux `/proc/net` is a > directory of many files, and a single file by that name would be a lie about a > well-known path. ## Sockets One set of syscalls, at their real Linux numbers. `socknew()` dispatches on the domain, and **no syscall changes shape** between the two families. ### AF_UNIX `kernel/net/unix.c`. `SOCK_STREAM`, `socketpair`, `shutdown`. A connected pair is **two buffers, not one**: writing appends to the peer's buffer and reading drains your own, which is what makes a socket bidirectional where a pipe is not, without needing two of anything. **One lock covers the whole table.** A pipe gets away with a lock per pipe because every operation touches one object; a socket operation touches two, and two-object locking needs an ordering rule every path obeys or it deadlocks. Worse, the wakeup races are subtle — a reader testing `peer_closed` under its own lock while the closer sets it under the peer's has no interlock at all. **Names live in a table here, not in the filesystem.** Binding creates no directory entry. The table gives the behaviour that matters — a client finds a server by a name agreed in advance — and what is lost is `ls -l` showing an `s`, and `unlink()` of a socket path working. Both want a VFS change rather than a socket change. ### Descriptor passing `sendmsg`/`recvmsg` with `SCM_RIGHTS`, which is the one thing a socket can carry that a pipe cannot: a reference to an object the kernel already knows, rather than a copy of its contents. Wayland's `wl_shm_create_pool` passes a descriptor and has no integer-shaped alternative, and [hybrid.md](../plan/hybrid.md)'s mechanism 1 is how a driver process is handed the device it is allowed to touch. Both wanted the same primitive. **A batch of descriptors is pinned to the stream position of the first byte of the message that carried it.** That single rule is the whole delivery model, and it has two halves that are easy to get half-right: - a reader that arrives at that position is handed the descriptors, and - a reader that has not reached it yet is **stopped short of it**. Without the second half, a program that asked for 4096 bytes would swallow three messages in one call and be handed one message's descriptors, with nothing in the result saying which message they belonged to. `socktest` checks exactly this: two messages, each one byte and one descriptor, read with an eight-byte buffer, and the first read must return one byte. The kernel holds a reference on every descriptor in flight, so a sender may close its own copy the moment `sendmsg` returns. That reference is also the thing that has to be given back, and there are three ways a message ends: | | | |---|---| | received by `recvmsg` with room | installed in the receiver's table | | received by `recvmsg` without room, or by plain `read(2)` | **closed**, and `MSG_CTRUNC` set for the first case | | never received — the socket is closed | **closed** by `sockclose` | The second row is the one that looks like a choice and is not. Holding kernel references for a receiver that has already proved it is not looking is a leak with extra steps, and it is what Linux does for the same reason. Descriptors need at least one byte to ride on; `sendmsg` refuses a message of pure ancillary data with `EINVAL` rather than accepting it and pinning the descriptors to a position that never arrives. **There is no garbage collector.** An AF_UNIX socket descriptor, sent over that same socket and never received, holds a reference to the socket that holds the reference to it, and neither close can reach zero. Linux carries several hundred lines to break these cycles. Nothing on either plan passes a socket over a socket — Wayland passes memory, the driver work passes devices — so this is a known and bounded leak (thirty-two sockets) rather than a silent one. ### AF_INET `kernel/net/inet.c`, on lwIP's **raw** API rather than lwIP's own sockets layer, which would need `NO_SYS=0` and a second socket abstraction alongside this kernel's own. **Everything is protected by the lwIP lock and nothing else**, and that is not a shortcut: lwIP's callbacks run in the network thread with that lock held and are the only writers to a socket's receive buffer. A syscall reading it takes the same lock. There is no second lock because there is no second writer. #### How a blocking socket waits Worth reading before touching, because the obvious implementation does not work. A blocking call cannot `sleep(&socket, &lwip_lock)`: `sleep()` wants a spinlock and the lwIP lock is a *sleeplock* — it has to be, because a call into lwIP can reach the driver. And a waiter cannot hold that lock while sleeping, or the network thread could never deliver the thing being waited for. So the wait is on a **counter** with a spinlock of its own: ``` reader: lwip_lock; check condition; s = net_seq(); lwip_unlock; net_wait(s) waker: (inside a callback, lwIP lock held) net_wake() ``` No wakeup can be lost. `s` is read while the reader still holds the lwIP lock, so anything changing the condition must do so afterwards and therefore bumps the counter after the read; `net_wait` re-checks under its own spinlock and does not sleep. One counter and one channel for **all** inet sockets rather than one each. At this scale waking sixteen sockets costs a handful of instructions in the ones with nothing to do, and a per-socket channel would need careful teardown in `close()` against callbacks that can still arrive. Before this, a blocked socket woke on every timer tick and looked — costing a wakeup per socket per 10 ms and answering up to 10 ms late even when the data was already there. `tcp_sent` had to be registered for it as well, or a write blocked on a full send buffer slept with nobody left to wake it. #### ICMP, and ping sockets `socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP)` — Linux's *ping socket*, which gives an unprivileged process the one thing raw sockets were used for. The caller reads and writes the ICMP message alone; the kernel owns the IP header. Built on lwIP's raw API: `raw_new(IP_PROTO_ICMP)`, with the receive callback stripping the variable-length IP header before queueing the message on the same datagram queue UDP uses. **Only echo replies are consumed**, and that restriction is load-bearing. `raw_input` runs *before* lwIP's own `icmp_input`, so a callback returning "eaten" for every ICMP message would swallow the echo **requests** too and this machine would silently stop answering pings from outside. Returning 0 leaves the packet for lwIP, which is what should happen to every ICMP message nobody asked for. There is no per-socket echo-id filtering. Linux rewrites the id and matches on it; here a reply goes to every open ICMP socket, which is invisible with one ping running and means a second one may see a sequence number it never sent. ### AF_NETLINK, and why it exists `getifaddrs(3)` is how every program asks *what interfaces does this machine have, and what addresses do they carry* — and on Linux there is exactly one answer: a `NETLINK_ROUTE` socket. musl implements the libc call over it, so what looks like a syscall is really a small protocol conversation, and without this side of it every caller fails identically. `kernel/net/netlink.c` is a **responder**, not a general socket family. There is no kernel-to-userspace event stream, no multicast groups, nothing subscribes to link state. A request arrives, the whole answer is generated into the socket's buffer, and reads drain it. That shape is deliberate. `getifaddrs` sends a dump request and reads until `NLMSG_DONE` with `MSG_DONTWAIT`, so generating the answer at send time means a read never blocks and the non-blocking flag is honoured for free. Two message types are served — `RTM_GETLINK` returns one `RTM_NEWLINK` per interface (name, flags, MAC), `RTM_GETADDR` returns one `RTM_NEWADDR` per address (address, prefix length, label). Anything else gets a bare `NLMSG_DONE`. **The data is not netlink's.** Interfaces come from the netdev class and addresses from lwIP's netifs, exactly as `/proc/netif` reads them. Nothing in the file knows what an e1000 is. Interface indices matter more than they look: `getifaddrs` matches address records to link records *by index*, so the two dumps have to agree. Loopback is 1 and the netdevs follow from 2. What it produces, read back through musl: ``` lo flags=0x49 fam=-1 eth0 flags=0x1043 fam=17 (AF_PACKET, the MAC) lo flags=0x49 fam=2 addr=127.0.0.1 mask=255.0.0.0 eth0 flags=0x1043 fam=2 addr=10.0.2.15 mask=255.255.255.0 ``` ### Loopback `127.0.0.1` is a real destination: `LWIP_HAVE_LOOPIF` creates the interface and `lwip_init()` brings it up. `LWIP_NETIF_LOOPBACK` is the half that is easy to miss — without it, a packet addressed to an interface's *own* address goes out of the wire and never comes back. With `NO_SYS=1` there is no thread to drain the loopback queue, so the network thread calls `netif_poll_all()` alongside `sys_check_timeouts()`. ### Datagrams UDP has a **datagram queue**, not a byte ring. It shared TCP's ring at first, and two datagrams arriving together came back as one — which is the bug that had to be fixed before `recvfrom` could report a source address at all. --- ## Names For a long time nothing on this system could turn a name into an address, and the shape of that failure is worth keeping: it did not look like one missing feature. `wget` did not work. `telnet` did not work. Anything taking a hostname did not work, each in its own way, and each looked like a separate broken program — which is a good argument for a userland with fewer applets, and a wrong one. **There is still no DNS client in SchOS, and there does not need to be.** musl contains a complete resolver: `getaddrinfo` reads `/etc/resolv.conf`, sends a query over an ordinary UDP socket and parses the reply. Every one of those pieces already worked. The single missing fact was *which nameserver* — and that arrives in a DHCP reply the kernel handled and never told anyone about. So the mechanism is one file: ``` # cat /proc/resolv.conf nameserver 10.0.2.3 ``` `net_resolv()` renders **resolv.conf's exact syntax**, so that putting it where the resolver looks is a copy and not a translation. It is empty until DHCP answers, which is the honest report: a file saying nothing is right, and one naming a server that does not exist is not. The policy is `cmd/resolvconf`, run once at boot by a `schinit` job. It waits for a nameserver, writes `/etc/resolv.conf` and exits — and **will not overwrite an existing one** without `-f`, because a machine with a hand-written `resolv.conf` has been configured on purpose. That division is the usual one here: the kernel knows a fact a process cannot find out for itself and publishes it; whether that fact should become `/etc/resolv.conf` is not the kernel's business. ``` # wget -O - http://example.com/ | head -1 Example Domain ``` ## Serving `cmd/httpd` is the other direction, and it existed for the same reason this section does: a machine that could fetch a page but not serve one is a strange shape for something with a TCP stack and a filesystem. GET and HEAD, static files out of a directory, `index.html` for a directory and a generated listing when there is none. A child per connection, so one client that stops reading cannot stop the server. No CGI, no keep-alive, no TLS, no configuration file — it is a *file server*, and each of those is a decision somebody should make on purpose. **The path check is the only security-relevant code in it**, and it refuses rather than normalises. The tempting version resolves `..`, collapses slashes and then checks the result — and it has to agree exactly with the kernel's resolver, and where it does not, the difference is a file outside the root. So no component may be `..` at all. Percent-decoding happens first, because `%2e%2e` is `..` and a check that runs before decoding does not see it. ``` $ curl -s -o /dev/null -w '%{http_code}\n' --path-as-is host/%2e%2e/etc/passwd 403 ``` ### A listening socket is a different object Worth its own note, because it cost a kernel panic and the shape of it is not obvious from the API. `tcp_listen()` does not put a pcb into a listening state. It **frees the connection pcb and returns a new, smaller one** — a `struct tcp_pcb_listen`, which has no `recv`, `sent`, `err` or `poll` field at all. The two are different types behind one `struct tcp_pcb *`, and the only callback a listening one has is `accept`. `inet_close()` did not know that, and cleared the callbacks the same way for both. On a listening socket that is a write past the end of the allocation; lwIP asserts rather than allowing it, and an assert here is a panic: ``` lwip assert: invalid socket state for recv callback at tcp.c:2024 #1 tcp_recv #2 inet_close #3 sockclose #4 fileclose #5 exit #6 syscall ``` Which is **a user process taking the machine down by pressing Ctrl-C** — the close path runs from `exit()`, so any program that listened and then died did it. Nothing had, because until `cmd/httpd` nothing in the tree listened on TCP and exited; `schdbgd` would have done it too, a few days later. The lesson is not about lwIP. It is that `listening` was already tracked one field away in `struct inet_sock` and the close path did not consult it — the information was there and the code did not ask. ## What it does not do - **`ping` is a first-party program**, `cmd/ping`, not toybox's. `recvmsg` now exists, so half of what stopped toybox's is gone; what remains is `IP_RECVTTL`, an `IPPROTO_IP` control message that would have to be generated on the receive path rather than merely carried. Ours has no `-I`, no flood mode and no TTL display. - **`SCM_RIGHTS` is the only control message.** `SCM_CREDENTIALS` is refused with `EINVAL` rather than ignored: a caller told its credentials were passed would have the receiver read a uid out of uninitialised memory and act on it. - **A message may carry eight descriptors, and a socket eight such messages.** Past that `sendmsg` answers `ENOBUFS`. The cap exists because the kernel holds a reference to each one until it is received; a sender eight unreceived messages ahead of its receiver is not rescued by a ninth slot. - **`ping` still takes a dotted quad**, even though names now work. It is a first-party program that does not link musl's resolver, and `localhost` is special-cased in it rather than resolved. Every *ported* program resolves names properly; ours is the one that does not. - **Nothing re-runs `resolvconf`.** The nameserver is written once at boot. A lease that changes the DNS server, or a `/dev/pci` handover that brings up a new interface, leaves a stale `/etc/resolv.conf` until somebody runs it again. - **lwIP's own DNS client is still unused.** It is compiled in and only `dns_getserver()` is ever called; resolution happens entirely in the process. Nothing in the kernel can resolve a name, which matters the day something in the kernel wants to. - **`httpd` does not serve HTTP/1.1.** Every response is HTTP/1.0 with `Connection: close`, because the directory listing is generated straight into the socket and so has no `Content-Length` to give. One request per connection is the cost. - **A queued-but-never-accepted connection is closed without clearing its callbacks.** `inet_close()` clears the argument and closes, which is enough because the pcb is freed — but it is a narrower guarantee than the one the main path makes, and it relies on `tcp_close()` succeeding. - **`httpd` has no timeouts.** A client that connects and says nothing ties up a child process until it goes away; `SO_RCVTIMEO` does not exist here (see below), so there is nothing to set. - **No routing table**, and no `/proc/net/route` to report one. With two interfaces up, the first is the default route and the second is reachable only by an address on its own subnet. That is a rule rather than an accident, and it is not routing. The missing *file* is the more immediately visible half: it is how a program asks which interface carries the default route, and without it anything that filters on "the default one" sees nothing. - **Netlink serves two message types.** No `RTM_GETROUTE`, no `RTM_NEWROUTE`, no event stream, no multicast groups. A program that expects to *subscribe* to link changes gets a socket that answers dumps and never speaks again. - **No `getsockname`/`getpeername`**, and `accept()` does not report the peer's address for an AF_INET socket. A server that wants to log who connected cannot. - **`setsockopt` covers six options**, and refuses the rest with `ENOPROTOOPT` rather than pretending: `SO_REUSEADDR`, `SO_KEEPALIVE`, `SO_BROADCAST` and `TCP_NODELAY` reach lwIP's pcb; `SO_SNDBUF`/`SO_RCVBUF` are accepted and report the real size on read. Notably absent: `SO_RCVTIMEO`/`SO_SNDTIMEO`. - **`SOCK_DGRAM` over AF_UNIX does not exist** — stream only. - **AF_UNIX names are not filesystem paths**, so nothing can `stat()` or `unlink()` a socket. - **One NIC driver exists.** The class is proven with two e1000 cards, not with two different drivers. - **No IPv6**, no DNS resolver in the kernel (lwIP's is compiled but unused by anything), and no firewall or packet filter of any kind.