SchOS/docs/architecture/networking.md

# Networking

ARP, IPv4, ICMP, UDP, TCP and DHCP, over a driver-agnostic interface class,
reached through the same socket syscalls as Unix domain sockets.

Verified end to end rather than by inspection: `bootcheck -Net` starts a
listener and a UDP responder on the host, tells the guest to talk to them, and
asserts on **both** ends — one side alone cannot tell a working stack from one
that completes a handshake and drops every byte.

---

## The four layers

```
        socket() bind() connect() send() recv()        syscalls, one set
    ┌────────────────┴───────────────────┐
    │  AF_UNIX              AF_INET      │   socknew() dispatches on domain
    │  kernel/net/unix.c    kernel/net/inet.c
    └────────────────┬───────────────────┘
                     │  lwIP raw API
    ┌────────────────┴───────────────────┐
    │  lwIP 2.2.0, NO_SYS=1              │   ports/lwip, compiled into the kernel
    │  port layer: kernel/net/lwip/      │   ~400 lines: a lock, a clock, a netif
    └────────────────┬───────────────────┘
                     │  struct netdev_ops
    ┌────────────────┴───────────────────┐
    │  e1000, and any future NIC driver  │
    └────────────────────────────────────┘
```

## The netdev class

A driver publishes a [`struct netdev_ops`](../../include/netdev.h) and is then
finished with the network stack forever: it never mentions lwIP, and the stack
never mentions it.

```c
struct netdev_ops {
  int (*transmit)(struct device *, const void *frame, int len);
  int (*receive) (struct device *, void *out, int cap);
  const unsigned char *(*mac)(struct device *);
  int (*link_up) (struct device *);        // optional
};
```

`transmit` returning -1 is **congestion, not failure** — the ring is full and
the caller is expected to retry. `receive` never blocks; it returns 0 when
nothing is waiting, and the caller decides how to wait.

This class was added late, and the reason it had to be is instructive: the stack
had reached straight into the driver. `net.c` called `e1000_present()`, the lwIP
glue was named `netif_e1000.c`, and the e1000 shipped with `.ops = 0` and a
comment saying so. A second NIC driver would have meant editing the protocol
stack to add itself. lwIP now gets one netif per bound device, `/proc/netif`
lists them all, and the e1000 drives up to four cards rather than refusing the
second — verified with two attached, each with its own MAC and its own DHCP
lease.

### `netdev_selftest()`

`echo n > /proc/sysrq` sends a hand-built ARP request and waits for the reply.

It proves the whole driver path in one exchange — descriptor formats, the DD
handshake, tail bookkeeping, DMA addresses, the receive ring hand-back, and the
MAC filter, because an ARP reply is addressed to this station and does not
arrive if the filter is wrong.

It was written *before* there was any protocol stack, deliberately: a stack on
an unproven driver fails in ways indistinguishable from its own bugs. It belongs
to the class rather than to the e1000, because it is the check every new NIC
driver will want first.

## lwIP

lwIP 2.2.0, fetched by `ports/lwip` and **compiled into the kernel** — it is the
one port that installs nothing into the sysroot, because it is kernel code.

`NO_SYS=1`, so lwIP has no threads, mailboxes or semaphores of its own. The port
layer is `kernel/net/lwip/`: an architecture header, a lock, a clock, an
Ethernet netif, and a set of shim headers standing in for the libc lwIP expects.

### Why lwIP and not a BSD stack

The licence was not the deciding factor — FreeBSD's `sys/netinet` is BSD-3 too.
**It is that FreeBSD's stack is not a library.** `tcp_input.c` alone is 122 KB
and reaches into mbufs, `epoch(9)`, netisr, VNET, `uma(9)`, sockbufs and inpcb
hash tables. Taking it means porting FreeBSD's kernel infrastructure first and
finding the TCP somewhere underneath — the same conclusion already reached about
FreeBSD's USB stack.

NetBSD was ruled out on licence: its `tcp_input.c` and `tcp_output.c` carry the
WIDE Project's 4-clause advertising clause, which Berkeley's rescission does not
reach. See [../licensing.md](../licensing.md) §3.

lwIP was written for exactly the case SchOS is in: a small operating system with
a raw Ethernet driver and no infrastructure to speak of.

**The reason not to write one, kept because it is the reason to resist later
too:** it is not the packet formats that are hard. It is congestion control,
retransmission, window management and twenty years of corner cases that only
appear against a real peer.

## The network thread

`kernel/net/net.c`. One kernel thread, started by `netinit()`, feeds lwIP
received frames and runs its timers.

**A thread rather than the interrupt handler, and the reason is the lock:** a
call into lwIP can reach the driver, the driver can sleep, and an interrupt
handler cannot. So the interrupt keeps doing what it always did — clear the
cause register — and this thread polls.

`netinit()` runs after `sinit()` because `kthread_create` needs a usable process
table, and it returns immediately: everything happens in that thread, which
cannot run until the scheduler is entered.

DHCP obtains an address at boot. `/proc/netif` reports each interface's address,
netmask, gateway, MAC and frame counts, taken at the lwIP seam.

> It is `/proc/netif` and not `/proc/net` deliberately: on Linux `/proc/net` is a
> directory of many files, and a single file by that name would be a lie about a
> well-known path.

## Sockets

One set of syscalls, at their real Linux numbers. `socknew()` dispatches on the
domain, and **no syscall changes shape** between the two families.

### AF_UNIX

`kernel/net/unix.c`. `SOCK_STREAM`, `socketpair`, `shutdown`.

A connected pair is **two buffers, not one**: writing appends to the peer's
buffer and reading drains your own, which is what makes a socket bidirectional
where a pipe is not, without needing two of anything.

**One lock covers the whole table.** A pipe gets away with a lock per pipe
because every operation touches one object; a socket operation touches two, and
two-object locking needs an ordering rule every path obeys or it deadlocks.
Worse, the wakeup races are subtle — a reader testing `peer_closed` under its own
lock while the closer sets it under the peer's has no interlock at all.

**Names live in a table here, not in the filesystem.** Binding creates no
directory entry. The table gives the behaviour that matters — a client finds a
server by a name agreed in advance — and what is lost is `ls -l` showing an `s`,
and `unlink()` of a socket path working. Both want a VFS change rather than a
socket change.

### AF_INET

`kernel/net/inet.c`, on lwIP's **raw** API rather than lwIP's own sockets layer,
which would need `NO_SYS=0` and a second socket abstraction alongside this
kernel's own.

**Everything is protected by the lwIP lock and nothing else**, and that is not a
shortcut: lwIP's callbacks run in the network thread with that lock held and are
the only writers to a socket's receive buffer. A syscall reading it takes the
same lock. There is no second lock because there is no second writer.

#### How a blocking socket waits

Worth reading before touching, because the obvious implementation does not work.
A blocking call cannot `sleep(&socket, &lwip_lock)`: `sleep()` wants a spinlock
and the lwIP lock is a *sleeplock* — it has to be, because a call into lwIP can
reach the driver. And a waiter cannot hold that lock while sleeping, or the
network thread could never deliver the thing being waited for.

So the wait is on a **counter** with a spinlock of its own:

```
  reader:  lwip_lock; check condition; s = net_seq(); lwip_unlock; net_wait(s)
  waker:   (inside a callback, lwIP lock held) net_wake()
```

No wakeup can be lost. `s` is read while the reader still holds the lwIP lock,
so anything changing the condition must do so afterwards and therefore bumps the
counter after the read; `net_wait` re-checks under its own spinlock and does not
sleep.

One counter and one channel for **all** inet sockets rather than one each. At
this scale waking sixteen sockets costs a handful of instructions in the ones
with nothing to do, and a per-socket channel would need careful teardown in
`close()` against callbacks that can still arrive.

Before this, a blocked socket woke on every timer tick and looked — costing a
wakeup per socket per 10 ms and answering up to 10 ms late even when the data
was already there. `tcp_sent` had to be registered for it as well, or a write
blocked on a full send buffer slept with nobody left to wake it.

#### ICMP, and ping sockets

`socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP)` — Linux's *ping socket*, which gives
an unprivileged process the one thing raw sockets were used for. The caller
reads and writes the ICMP message alone; the kernel owns the IP header.

Built on lwIP's raw API: `raw_new(IP_PROTO_ICMP)`, with the receive callback
stripping the variable-length IP header before queueing the message on the same
datagram queue UDP uses.

**Only echo replies are consumed**, and that restriction is load-bearing.
`raw_input` runs *before* lwIP's own `icmp_input`, so a callback returning
"eaten" for every ICMP message would swallow the echo **requests** too and this
machine would silently stop answering pings from outside. Returning 0 leaves the
packet for lwIP, which is what should happen to every ICMP message nobody asked
for.

There is no per-socket echo-id filtering. Linux rewrites the id and matches on
it; here a reply goes to every open ICMP socket, which is invisible with one
ping running and means a second one may see a sequence number it never sent.

### Loopback

`127.0.0.1` is a real destination: `LWIP_HAVE_LOOPIF` creates the interface and
`lwip_init()` brings it up. `LWIP_NETIF_LOOPBACK` is the half that is easy to
miss — without it, a packet addressed to an interface's *own* address goes out
of the wire and never comes back.

With `NO_SYS=1` there is no thread to drain the loopback queue, so the network
thread calls `netif_poll_all()` alongside `sys_check_timeouts()`.

### Datagrams

UDP has a **datagram queue**, not a byte ring. It shared TCP's ring at first, and
two datagrams arriving together came back as one — which is the bug that had to
be fixed before `recvfrom` could report a source address at all.

---

## What it does not do

- **`ping` is a first-party program**, `cmd/ping`, not toybox's. toybox's reads
  replies with `recvmsg(2)` and asks for the TTL through an `IP_RECVTTL` control
  message, and neither exists — `recvmsg` is item 1 of
  [../plan/foundations.md](../plan/foundations.md), where it is wanted for
  `SCM_RIGHTS` anyway. Ours has no `-I`, no flood mode and no TTL display.
- **No resolver reachable from userland.** lwIP's DNS client is compiled but
  nothing exposes it, so `ping` takes a dotted quad. `localhost` is special-cased
  in the program rather than resolved.
- **Receive is polled.** The network thread looks at the NIC every timer tick
  when the link is idle, so an arriving frame waits up to 10 ms. Waking the
  thread from the e1000 interrupt is the real fix, and was deliberately not the
  first version: a receive path woken by an interrupt is much harder to debug
  than one that is asked.
- **No routing table.** With two interfaces up, the first is the default route
  and the second is reachable only by an address on its own subnet. That is a
  rule rather than an accident, and it is not routing.
- **No `getsockname`/`getpeername`**, and `accept()` does not report the peer's
  address for an AF_INET socket. A server that wants to log who connected cannot.
- **`setsockopt` covers six options**, and refuses the rest with `ENOPROTOOPT`
  rather than pretending: `SO_REUSEADDR`, `SO_KEEPALIVE`, `SO_BROADCAST` and
  `TCP_NODELAY` reach lwIP's pcb; `SO_SNDBUF`/`SO_RCVBUF` are accepted and
  report the real size on read. Notably absent: `SO_RCVTIMEO`/`SO_SNDTIMEO`.
- **`SOCK_DGRAM` over AF_UNIX does not exist** — stream only.
- **AF_UNIX names are not filesystem paths**, so nothing can `stat()` or
  `unlink()` a socket.
- **One NIC driver exists.** The class is proven with two e1000 cards, not with
  two different drivers.
- **No IPv6**, no DNS resolver in the kernel (lwIP's is compiled but unused by
  anything), and no firewall or packet filter of any kind.