nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

debugger: D5, the network transport -- and the plan closes

Browse source 
schdbgd bridges TCP to the debugger, which is the one way in that does not
need you to be sitting at the machine. It is a bridge and nothing else: a line
in becomes a line written to /dev/dbgro and the answer comes back, so `nc` is a
complete client and there is no clever client to be broken on the day you need
this.

    # schdbgd &
    schdbgd: debugger on port 4444, loopback only
    $ nc 127.0.0.1 4444
    dma
    arena  0xffff80003da41000-0xffff80003ea41000  16384 KB total, ...

A kernel debugger plus a TCP socket is a remote root hole, so three things were
decided rather than discovered:

  - It does not start by itself. schdbgd.job ships Disabled. SchOS has no
    kernel command line, so the boot flag the plan asked for is the job
    manifest, which is the same thing in this system's own idiom.
  - It binds loopback; -a names an address and says so in a paragraph.
  - It cannot reach a view that changes state, enforced in the kernel.

That last one is /dev/dbgro: /dev/dbg on minor 1. The minor is the one piece of
per-open state a character device already sees, so which node you opened is the
permission -- nothing to latch, nothing to remember between the write and the
read, no way across. Which views are refused comes from dbg_register_rw() on
the registration itself, because a central list of dangerous views goes stale
in the direction where a new view is quietly treated as safe. Today that is
`watch <addr>`; `watch` with no argument is still the listing and still
answered.

Also: socket() now accepts IPPROTO_TCP with SOCK_STREAM and IPPROTO_UDP with
SOCK_DGRAM. They name the default rather than departing from it, and toybox's
nc writes them out -- so TCP was "Protocol not supported" on a system whose
whole network is TCP.

docs/plan/debugger.md retires into docs/architecture/debugging.md; 16 files
repointed. Every plan in docs/plan/ that was open is now closed, so the next
thing is Gate 0 itself: the survey, with the instrument built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
Nexoniarz 2026-09-14 15:33:53 +02:00
commit bc39b73969
23 changed files with 491 additions and 293 deletions
Download patch file Download diff file Expand all files Collapse all files

91
docs/architecture/debugging.md
View file

@ -204,6 +204,7 @@ fine.
| `sysrq g` | `/proc` is mounted |
| `/dev/dbg` | `/dev` is mounted |
| `schdbg` | there is a shell |
| `schdbgd` over TCP | there is a network, and somebody enabled it |
**Ctrl-G is the one that matters for boot problems**, because it is the only
one that does not need a filesystem. It sits beside the older Ctrl-P, which
@ -226,6 +227,78 @@ called from `panic()`.**
---
## Over the network
The fourth way in, and the only one that does not need you to be sitting at the
machine: a target that has locked its display, a board with no keyboard, or a
bug that only appears under load you cannot produce by hand.
```
# schdbgd &
schdbgd: debugger on port 4444, loopback only
$ nc 127.0.0.1 4444
SchOS debugger. 'help' for the views, 'quit' to go.
read-only: views that change state are refused here.
dma
arena 0xffff80003da41000-0xffff80003ea41000
16384 KB total, 16319 KB free in 1 extent(s), 64 KB out
```
`schdbgd` is **a bridge and nothing else**: a line in becomes a line written to
`/dev/dbgro`, and the answer comes back. No framing, no state, no parsing of
anything the kernel said — which is what makes `nc` a complete client, and what
keeps this usable on the day the clever client is one of the broken things.
It is in a process rather than in the kernel because there is no reason for it
to be in the kernel. A TCP listener inside a kernel debugger is a listener that
has to work when the kernel does not, and this one does not pretend to: if the
scheduler is wedged, `schdbgd` is not running either, and the answer is Ctrl-G.
### What a listener had to be decided about
A kernel debugger plus a TCP socket is a remote root hole. That is true of every
one ever written; it is not a reason not to build one, and it is a reason to
decide rather than discover. Three decisions, in increasing order of how much
they actually protect:
**It does not start by itself.** `/etc/schinit/schdbgd.job` ships `Disabled`.
SchOS has no kernel command line, so the boot flag is the job manifest — the
same thing in this system's own idiom. A machine whose owner never asked for a
debugger is not running one.
**It binds loopback**, and binding anywhere else takes `-a` naming the address,
and says so in a paragraph rather than a word. Reaching it then means `ssh -L`,
which is somebody's deliberate act.
**It cannot reach a view that changes state, and that is enforced in the
kernel.** `/dev/dbgro` is `/dev/dbg` on minor 1. The minor is the one piece of
per-open state a character device already sees, so *which node you opened is the
permission*: there is nothing to latch, nothing to remember between the write
and the read, and no way to get from one descriptor to the other.
Which views are refused comes from the registration — `dbg_register_rw` instead
of `dbg_register` — for exactly the reason the registry exists at all. A list of
dangerous views in one central file is a list that goes stale, and it goes stale
in the direction where a newly added view is quietly treated as safe.
Today that is one view. `watch <addr>` arms a debug register, which is a
reasonable thing for a debugger to do and not a reasonable thing for a stranger
on a socket to do:
```
$ nc 127.0.0.1 4444
watch 0xffff80003da41000
'watch' changes state, and this is /dev/dbgro.
use /dev/dbg from the console for that.
```
`watch` with no argument is still answered, because that is the listing and it
arms nothing.
---
## Who owns this page
The views each speak for one subsystem, which is what keeps them cheap and
@ -360,10 +433,16 @@ something that reports nonsense.
`dbg_thread_start()` — which is the last thing `main()` does before entering
the scheduler — nothing answers but the log, and the log is enough because it
is read on the *next* boot.
- **No network transport.** The plan puts a TCP bridge in a userspace daemon,
deliberately outside the kernel; it does not exist, so the debugger is local
or serial only.
- **No memory writes**, and no way to poke anything. Reading is what finds bugs.
- **`schdbgd` has no authentication.** Reaching the port *is* the permission,
which is why the port is on loopback and why the daemon is off. Anyone who can
open a TCP connection to it can read this kernel's memory layout, its process
table and its open sockets. Putting a password on it would be worse than
useless — it would make the current arrangement feel safer than it is.
- **One client at a time.** `/dev/dbg` holds one answer at a time, so two
sessions would read each other's; a second caller waits.
- **A disconnected client is noticed only when written to.** A session that
goes away between commands holds the daemon until TCP notices.
- **`all` truncates at 16 KB** and says so. On a busy machine, ask for one view
at a time.
- **One conversation at a time.** The answer buffer is global, so two programs
@ -379,5 +458,7 @@ something that reports nonsense.
happened in the two hundred events before this" is not a question this can
answer.
What remains — a network transport, kept deliberately outside the kernel — is
in [../plan/debugger.md](../plan/debugger.md).
Nothing remains on the plan this was built from. What is *not* on it, and is the
obvious next thing if this is ever picked up again, is tracing: static
tracepoints and an event ring. That is a different tool that happens to share a
transport.

19
docs/plan/1.0.md
View file

@ -95,9 +95,9 @@ The galling ones. The work exists, is tested, and nothing uses it.
|---|---|
| **`foundations.md`** | Item 1 built. Items 2 (a shareable memory object), 3 (`AF_UNIX` in the filesystem namespace), 4 and 5 open. **Wayland stops at item 2.** |
| ~~**`hybrid.md`**~~ | **Closed.** All four mechanisms built; H1, H2 and H3 done. H4 — the USB eviction — is *withdrawn* rather than outstanding: USB is the bus peripherals arrive on, not a peripheral, and an eviction that has to leave an identical driver behind has moved nothing. See [hybrid.md](hybrid.md), "Decided: USB stays in the kernel". |
| ~~**`ldls.md`**~~ | **Closed.** L0 through L4 built: the loader handles 2,440 stock modules, `8139too` runs in a SchOS process driving QEMU's rtl8139 through `/dev/pci`, `ping` works over it, and killing the host releases the device and DMA so a replacement takes over. Described in [../architecture/ldls.md](../architecture/ldls.md). One driver, one kernel version, tried once. |
| ~~**`ldls.md`**~~ | **Closed.** L0 through L4 built: the loader handles 2,440 stock modules, and both `8139too` and Linux's `e1000` run in a SchOS process through `/dev/pci` with `ping` working over each. The e1000 run is a handover — the kernel was driving that NIC and gave it up. Described in [../architecture/ldls.md](../architecture/ldls.md). Two drivers, one kernel version, one machine. |
| **Schcom v0** | 715 lines that composite, over a wire format that is superseded. Frozen: it runs, nothing more is added, and it is kept only because it is the one thing in the tree that has ever put a client's pixels on screen. |
| **`debugger.md`** | D0 through D4 built and described in [../architecture/debugging.md](../architecture/debugging.md). Only the network transport is open — a userspace daemon bridging TCP to `/dev/dbg`, which is deliberately last. |
| ~~**`debugger.md`**~~ | **Closed.** D0 through D5 built, including the network transport: `schdbgd` bridges TCP to `/dev/dbgro`, ships disabled, binds loopback, and cannot reach a view that changes state. Described in [../architecture/debugging.md](../architecture/debugging.md). |
**The row LDLS spent, and gave back.** `hybrid.md` closed — H3 built, H4
withdrawn — and `ldls.md` took the slot it freed. It has since closed too, so
@ -107,13 +107,16 @@ part of 1.0**: nothing in the acceptance test needs a foreign driver. It was
taken on because the mechanisms it uses were finished and idle, and because
the alternative to it is writing a driver for every device that exists.
**The row the debugger spent.** Adding one requires closing one, and *A released
PCI device* closed when H2 landed. It is worth saying plainly that the debugger
is not a capability of the OS the way every other row here is: nothing in it
ships to a user or appears in the acceptance test. The case for it is that
**Gate 0 is a list of bugs to find in a kernel that could not be asked
**The row the debugger spent, and gave back.** Adding one requires closing one,
and *A released PCI device* closed when H2 landed. It is worth saying plainly
that the debugger is not a capability of the OS the way every other row here is:
nothing in it ships to a user or appears in the acceptance test. The case for it
is that **Gate 0 is a list of bugs to find in a kernel that could not be asked
anything** — so it is the instrument the rest of this register is surveyed with,
and it earns its slot by making every other row cheaper to close.
and it earned its slot by making every other row cheaper to close.
It has closed. **Every plan in `docs/plan/` that was open is now closed**, which
means the next thing is Gate 0 itself: the survey, with the instrument built.
**Closed since this register was written.** *A released PCI device* — release is
now symmetric with claim (bus mastering cleared before the DMA is freed, a

9
docs/plan/README.md
View file

@ -43,13 +43,8 @@ And the one the hybrid mechanisms were built for:
| | |
|---|---|
| ~~ldls.md~~ | **Built.** The Linux Driver Layer for SchOS: someone else's drivers, in a process, reached through `/dev/pci`. `ping` works over `8139too`. The description moved to [../architecture/ldls.md](../architecture/ldls.md). |
And one that is a tool rather than a capability:
| | |
|---|---|
| **[debugger.md](debugger.md)** | A debugger for this kernel — one that observes rather than halts, and answers in this kernel's own vocabulary. **Built**, except the network transport: symbols, a log that outlives the kernel, `/dev/dbg`, the `whose` ownership query and hardware watchpoints are all described in [../architecture/debugging.md](../architecture/debugging.md). |
| ~~ldls.md~~ | **Built.** The Linux Driver Layer for SchOS: someone else's drivers, in a process, reached through `/dev/pci`. `ping` works over `8139too` and over `e1000`. The description moved to [../architecture/ldls.md](../architecture/ldls.md). |
| ~~debugger.md~~ | **Built**, all of it, including the network transport. A debugger that observes rather than halts and answers in this kernel's own vocabulary: symbols, a log that outlives the kernel, `/dev/dbg`, `whose`, hardware watchpoints, and `schdbgd`. The description moved to [../architecture/debugging.md](../architecture/debugging.md). |
Two longer-horizon documents sit behind these: [schapi.md](schapi.md), the API
layering and the compatibility promise, and [roadmap.md](roadmap.md), which holds

243
docs/plan/debugger.md
View file

@ -1,243 +0,0 @@
# A debugger for this kernel
> **Every milestone on this plan is built.** Symbols, a log that outlives the
> kernel, the introspection registry behind `/dev/dbg`, the `whose` ownership
> query and hardware watchpoints are all described in
> [../architecture/debugging.md](../architecture/debugging.md).
>
> What is left here is **level 2, the network transport** — a userspace daemon
> bridging TCP to `/dev/dbg`, deliberately outside the kernel. Nothing else.
Every bug in this project so far has been found the same way: add a `cprintf`,
rebuild, boot, read the serial log, guess again. That loop is about ninety
seconds long and it answers exactly one question per turn — the question you
thought to ask before you rebuilt.
The DMA ownership bug in `do_dma()` is what made the case. Finding it took
several rounds of that loop, and the thing that eventually found it was not a
print at all; it was reading `freevm()` and noticing what `PHYSTOP` meant for
the arena. The kernel could not be *asked* whether two allocators owned the same
page, because nothing in it can be asked anything.
> **The kernel has no way to answer a question it was not already compiled to
> answer.** That is the gap.
---
## What there was, and what was missing
The state this plan was written against. D0 to D2 have since closed the first
three rows; they are kept because they are the argument.
| | |
|---|---|
| `cprintf` | goes to UART and console and is then gone. **There is no log.** `panic.c` says so itself: *"there is no core dump and no log to go back to."* |
| PSOGM | collects a great deal at the moment of a panic — CPU, process, call chain, load address — and paints it. It is the only artefact of a crash, and it does not survive being looked away from. |
| `getcallerpcs()` | walks frames and prints **raw addresses**. Turning one into a function name is a manual `objdump` session. |
| `/proc/sysrq` | a hardcoded `switch` with one letter per action. `xhci_dump`, `usb_storage_dump`, `display_selftest` — the right instinct, in the wrong shape: they are self-tests, not state, and adding one means editing a central file. |
| DR0–DR3 | **unused.** `T_DEBUG` maps to `SIGTRAP` and nothing ever arms a hardware watchpoint. |
---
## The two rules
### It observes; it does not control
No stepping. No software breakpoints. No stopping the world.
This is the decision everything else follows from, and it is not a scope
compromise. Half the bugs this project actually hits are *timing*: an interrupt
storm at 77,000 a second, a level-triggered line nobody acknowledges, a driver
that faults only on the second start, a supervisor that restarts a job fifteen
times in forty-five seconds. **Every one of those disappears when you stop the
world to look at it.** A debugger that can only work by halting the machine
cannot see the class of bug this kernel produces.
So it answers questions, and it watches. Both of those work on a machine that is
still running.
### It must not depend on what it is debugging
The same circularity rule the display already has, pointed at a different
subsystem:
> A debugger that needs lwIP cannot debug lwIP. A debugger that needs a process
> cannot debug the scheduler. A debugger that needs to allocate cannot debug the
> allocator.
That rule decides the layering below rather than taste deciding it, which is the
only reason to write a rule down.
---
## Why not GDB
The licence is not the reason — a stub written here from the published protocol
would be our own Apache-2.0 code, the same clean-room argument as
[../architecture/devices.md](../architecture/devices.md) makes for Linux
drivers. Two technical reasons, and they are both about what a *generic*
debugger structurally cannot do.
**It only knows registers and memory.** GDB can tell you that `0x3eceb000`
contains zeroes. It cannot tell you that the page is in the DMA arena's free
list *and* on `kalloc`'s free list *and* mapped into pid 7 — because those are
this kernel's abstractions, and a debugger that knows about them has to be
written against them. That query is the one that would have ended the session
described above in one line.
**It stops the world**, per the rule above.
There is a third, smaller reason: a `gdbstub` is a protocol you implement and
then debug, and its failure mode is an unresponsive stub with no way to ask it
why. The thing below can always be driven by hand over a serial line.
---
## The layers
Three transports, and the rule decides which gets what. **The kernel's half is
the minimum that works when nothing else does; everything convenient lives in a
process** — which is the same sentence as [hybrid.md](hybrid.md)'s, about a
different thing.
| | Works when | |
|---|---|---|
| **0. Console and UART** | always, including panic context, with interrupts off, no allocation, no scheduler, no lock taken | `sysrq`-driven. If the machine is dying this is the one you have. |
| **1. `/dev/dbg`** | there is a scheduler and a process | A character device. The local client is an ordinary program — exactly the `/dev/pci` discipline, for exactly the same reason. |
| **2. The network** | there is a network | A **userspace daemon** bridging TCP to `/dev/dbg`. Not in the kernel. lwIP must stay out of the debugger's path, or the debugger cannot be pointed at lwIP. |
Level 2 is deliberately last, and it is not in the first cut.
---
## The protocol is lines of text
A request is a line. A response is lines, ending in a blank one.
```
> whose 0x3eceb000
phys 0x3eceb000
dma arena extent 2, FREE (4096000 bytes @ 0x3eceb000)
kalloc free list, position 41
pid 7 vgpud va 0x4000d000 rw-u vma[3] no_free
*** three owners -- this page is claimed twice
> view dma
arena 0x3e000000-0x3f000000 16384 KB, 12284 KB free
free[0] 0x3e0a4000 8192 KB
free[1] 0x3ecec000 4092 KB
in use 4096 KB in 3 blocks
```
Text, and not a typed binary protocol, for three reasons that all turn out to be
the same reason:
- **It works at level 0.** Panic context cannot allocate, cannot frame records,
cannot wait for a client to acknowledge anything. It can put characters on a
serial line. A binary protocol needs a text fallback anyway, and then there are
two protocols.
- **It is usable before a client exists.** `cat`, `echo`, a serial terminal, the
guest's own shell. A protocol that needs its client built first is a protocol
nobody can debug.
- **It survives the thing it is debugging.** When the answer arrives garbled,
you can still read most of it.
A richer client can be layered on later by parsing this, and if that ever
happens the kernel does not change.
---
## Milestones
| | | |
|---|---|---|
| **D0** | **Symbols.** A compact sorted table emitted from `nm` at link time and linked into the image; `ksym_lookup(addr, &off)`. | **done** |
| **D1** | **A log that survives.** A ring behind `cprintf`, timestamped, in a region the allocator never sees — so a panic's last words are readable on the next boot. `/proc/kmsg`. | **done** |
| **D2** | **The introspection registry.** `dbg_register("dma", dma_dump)`. Seven views, `/dev/dbg`, `schdbg`, and `sysrq g`. | **done** |
| **D3** | **`whose <addr>`** — the cross-subsystem ownership query above. | **done** |
| **D4** | **Watchpoints**, DR0–DR3, break-on-write, reporting rip and call chain and **continuing** rather than stopping. | **done** |
| **D5** | **The network transport**, level 2 above: a userspace daemon bridging TCP to `/dev/dbg`. Off unless a boot flag turns it on, loopback by default. | |
### Why D0 and D1 come before the interesting parts
Neither is clever. Both multiply everything after them: a view that prints an
address is worth less than one that prints a name, and a query you can only run
*after* you have noticed something is worth less than a log that was already
running when it happened.
D1 in particular changes the job rather than improving it. Today a triple fault
leaves nothing at all — the screen is gone, the serial buffer is gone, and the
only evidence is whatever you happened to be watching. A log that outlives the
boot turns that into a file.
### What D1 turned out to cost
One measurement changed the design. The log was first placed at the top of RAM
beside the DMA arena, which is the obvious spot for a region nothing should
allocate — and on the first reboot its magic came back as ASCII, because **the
top of usable memory is exactly where the firmware makes its own allocations**
before handing the machine over. It sits above the kernel image instead, which
is both a stable address across boots and memory the loader has already
committed to. Recorded because the reasoning that picked the wrong place was
sound, and only the test was not.
### What D2 is really for
Not the views themselves — the *registry*. A subsystem publishes its state with
one line at the bottom of the file that owns the data, and nothing central has to
be edited. That is the only version of this that stays current; the `sysrq`
switch is the version that does not, and it is already three self-tests and no
state.
---
## The security note, for the parts not yet built
A debugger that reads arbitrary physical memory and a TCP listener are, together,
a remote root hole. That is true of every kernel debugger ever written and it is
not a reason not to build one, but it has to be decided rather than discovered:
- Level 2 is **off unless a boot flag turns it on**, and binds loopback by
default.
- No memory *writes* in the first cut. Reading is what finds bugs; writing is
what a later milestone can argue for on its own.
---
## What it will still not do
- **It will not step, break, or halt**, per the rule. If a bug genuinely needs a
stepper, this is the wrong tool and that is a deliberate trade.
- **No stack unwinding through interrupt frames.** `getcallerpcs()` walks `rbp`
chains and stops where the frame pointer stops; a trap frame in the middle of
a chain ends the walk.
- **No userspace symbolization.** The kernel will name kernel symbols. A fault
in a process reports the binary and the offset, and resolving that stays a
host-side job.
- **No profiling.** No sampling, no counters, no flame anything. A timer-driven
sampler is a different tool that happens to share a transport.
- **Watchpoints are limited to four**, by the hardware, and to 1/2/4/8 bytes.
There is no software watchpoint fallback and there should not be one.
- **It is not a tracer.** Static tracepoints and an event ring are the obvious
next thing after D4 and are not on this plan.
---
## On the freeze
[1.0.md](1.0.md) says no new capability starts until the register is empty, and
the register is not empty. Two things about that.
The bookkeeping answer is that the rule is *"adding a row requires closing one"*,
and **A released PCI device** closed when H2 landed. There is a slot.
The better answer is that the register is a list of **capabilities of the OS**,
and this is not one — nothing in it ships to a user or appears in the acceptance
test. It is a tool, and the case for spending time on a tool is that it makes
everything still on the register cheaper to finish. Gate 0 of [1.0.md](1.0.md)
is *"fix what is already wrong"*, assembled from every *What it does not do*
section in [../architecture/](../architecture/). That gate is a list of bugs to
find in a kernel that cannot currently be asked anything.
**Build the instrument before the survey**, and then the freeze holds for the
rest of it.