113595f80b
docs/plan/debugger.md D3 and D4, which finish the plan apart from the
network transport.
D3 -- `whose <addr>`. The views each speak for one subsystem, which is what
keeps them cheap and current, and it is exactly why none of them can see
the bug that matters: a page on the DMA arena's free list looks fine from
dma.c, the same page on kalloc's free list looks fine from kalloc.c, and
that it is on both is a fact no single subsystem holds.
# schdbg whose 0x3e239000
phys 0x000000003e239000 kernel va 0xffff80003e239000
dma arena, handed out
pcidma slot 2, 0 KB into a 4000 KB block given to pid 4 (vgpud)
mapped pid 4 (vgpud) at 0x000000004000d000 rwu vma[5] no_free
consistent: 1 owner, 1 mapping(s)
Claimants register the way views do -- one line in the file that owns the
data -- so the kernel image, the log region, the arena, the page allocator,
the pciuser DMA blocks and every page table each answer for themselves.
`mapped` is the one no allocator can answer: there is no reverse map, so
vm_walk_user() goes forward over every mapped page of every live process.
Two verdicts are shouted and everything else is reported as fact. More than
one owner means the second allocator to hand the page out is handing out
memory the first is still using -- the H2 bug, which would have printed as
three lines. Free-but-still-mapped means a page will be given away from
under the page tables that point at it. Deliberately *not* a finding: no
owner at all, because kalloc keeps a free list and no record of what it has
handed out, so an ordinary allocated page has nobody to speak for it and
saying otherwise would be the query crying wolf.
D4 -- watchpoints, on the four debug registers that had been sitting unused
since this kernel was written. Armed against the same framebuffer page and
then killing the driver:
watch 0: 0xffff80003e239000 written, 8 byte(s), by stosq+0x1f, pid 12 vgpud
#0 memset+0xba
#1 dma_alloc+0x2e2
#2 do_dma+0x22a
#3 pciuser_ioctl+0x13b
#4 sys_ioctl+0x3c7
#5 syscall+0xf6
The chain is the answer, not the rip. It reports and continues -- a data
watchpoint is a trap, delivered after the store completes, so returning
carries on with the program none the wiser, which is what lets this be used
on a machine that is still running.
Debug registers are per-CPU and there is no general IPI here, so the table
carries a generation number and each CPU reloads from its timer interrupt
when it notices the number has moved: one comparison per tick, one tick of
latency at worst. `watch` prints which CPUs have caught up, because "this
watchpoint is watching" and "this watchpoint is watching one CPU" must not
look the same.
Also fixed while testing: watch_init() was called from mpmain(), which runs
on every processor, so the command appeared in the list once per CPU.
dbg_register() now refuses a duplicate name outright, so the next init that
runs per-CPU fails loudly in the list rather than quietly doubling it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
243 lines
11 KiB
Markdown
243 lines
11 KiB
Markdown
# A debugger for this kernel
|
||
|
||
> **Every milestone on this plan is built.** Symbols, a log that outlives the
|
||
> kernel, the introspection registry behind `/dev/dbg`, the `whose` ownership
|
||
> query and hardware watchpoints are all described in
|
||
> [../architecture/debugging.md](../architecture/debugging.md).
|
||
>
|
||
> What is left here is **level 2, the network transport** — a userspace daemon
|
||
> bridging TCP to `/dev/dbg`, deliberately outside the kernel. Nothing else.
|
||
|
||
Every bug in this project so far has been found the same way: add a `cprintf`,
|
||
rebuild, boot, read the serial log, guess again. That loop is about ninety
|
||
seconds long and it answers exactly one question per turn — the question you
|
||
thought to ask before you rebuilt.
|
||
|
||
The DMA ownership bug in `do_dma()` is what made the case. Finding it took
|
||
several rounds of that loop, and the thing that eventually found it was not a
|
||
print at all; it was reading `freevm()` and noticing what `PHYSTOP` meant for
|
||
the arena. The kernel could not be *asked* whether two allocators owned the same
|
||
page, because nothing in it can be asked anything.
|
||
|
||
> **The kernel has no way to answer a question it was not already compiled to
|
||
> answer.** That is the gap.
|
||
|
||
---
|
||
|
||
## What there was, and what was missing
|
||
|
||
The state this plan was written against. D0 to D2 have since closed the first
|
||
three rows; they are kept because they are the argument.
|
||
|
||
| | |
|
||
|---|---|
|
||
| `cprintf` | goes to UART and console and is then gone. **There is no log.** `panic.c` says so itself: *"there is no core dump and no log to go back to."* |
|
||
| PSOGM | collects a great deal at the moment of a panic — CPU, process, call chain, load address — and paints it. It is the only artefact of a crash, and it does not survive being looked away from. |
|
||
| `getcallerpcs()` | walks frames and prints **raw addresses**. Turning one into a function name is a manual `objdump` session. |
|
||
| `/proc/sysrq` | a hardcoded `switch` with one letter per action. `xhci_dump`, `usb_storage_dump`, `display_selftest` — the right instinct, in the wrong shape: they are self-tests, not state, and adding one means editing a central file. |
|
||
| DR0–DR3 | **unused.** `T_DEBUG` maps to `SIGTRAP` and nothing ever arms a hardware watchpoint. |
|
||
|
||
---
|
||
|
||
## The two rules
|
||
|
||
### It observes; it does not control
|
||
|
||
No stepping. No software breakpoints. No stopping the world.
|
||
|
||
This is the decision everything else follows from, and it is not a scope
|
||
compromise. Half the bugs this project actually hits are *timing*: an interrupt
|
||
storm at 77,000 a second, a level-triggered line nobody acknowledges, a driver
|
||
that faults only on the second start, a supervisor that restarts a job fifteen
|
||
times in forty-five seconds. **Every one of those disappears when you stop the
|
||
world to look at it.** A debugger that can only work by halting the machine
|
||
cannot see the class of bug this kernel produces.
|
||
|
||
So it answers questions, and it watches. Both of those work on a machine that is
|
||
still running.
|
||
|
||
### It must not depend on what it is debugging
|
||
|
||
The same circularity rule the display already has, pointed at a different
|
||
subsystem:
|
||
|
||
> A debugger that needs lwIP cannot debug lwIP. A debugger that needs a process
|
||
> cannot debug the scheduler. A debugger that needs to allocate cannot debug the
|
||
> allocator.
|
||
|
||
That rule decides the layering below rather than taste deciding it, which is the
|
||
only reason to write a rule down.
|
||
|
||
---
|
||
|
||
## Why not GDB
|
||
|
||
The licence is not the reason — a stub written here from the published protocol
|
||
would be our own Apache-2.0 code, the same clean-room argument as
|
||
[../architecture/devices.md](../architecture/devices.md) makes for Linux
|
||
drivers. Two technical reasons, and they are both about what a *generic*
|
||
debugger structurally cannot do.
|
||
|
||
**It only knows registers and memory.** GDB can tell you that `0x3eceb000`
|
||
contains zeroes. It cannot tell you that the page is in the DMA arena's free
|
||
list *and* on `kalloc`'s free list *and* mapped into pid 7 — because those are
|
||
this kernel's abstractions, and a debugger that knows about them has to be
|
||
written against them. That query is the one that would have ended the session
|
||
described above in one line.
|
||
|
||
**It stops the world**, per the rule above.
|
||
|
||
There is a third, smaller reason: a `gdbstub` is a protocol you implement and
|
||
then debug, and its failure mode is an unresponsive stub with no way to ask it
|
||
why. The thing below can always be driven by hand over a serial line.
|
||
|
||
---
|
||
|
||
## The layers
|
||
|
||
Three transports, and the rule decides which gets what. **The kernel's half is
|
||
the minimum that works when nothing else does; everything convenient lives in a
|
||
process** — which is the same sentence as [hybrid.md](hybrid.md)'s, about a
|
||
different thing.
|
||
|
||
| | Works when | |
|
||
|---|---|---|
|
||
| **0. Console and UART** | always, including panic context, with interrupts off, no allocation, no scheduler, no lock taken | `sysrq`-driven. If the machine is dying this is the one you have. |
|
||
| **1. `/dev/dbg`** | there is a scheduler and a process | A character device. The local client is an ordinary program — exactly the `/dev/pci` discipline, for exactly the same reason. |
|
||
| **2. The network** | there is a network | A **userspace daemon** bridging TCP to `/dev/dbg`. Not in the kernel. lwIP must stay out of the debugger's path, or the debugger cannot be pointed at lwIP. |
|
||
|
||
Level 2 is deliberately last, and it is not in the first cut.
|
||
|
||
---
|
||
|
||
## The protocol is lines of text
|
||
|
||
A request is a line. A response is lines, ending in a blank one.
|
||
|
||
```
|
||
> whose 0x3eceb000
|
||
phys 0x3eceb000
|
||
dma arena extent 2, FREE (4096000 bytes @ 0x3eceb000)
|
||
kalloc free list, position 41
|
||
pid 7 vgpud va 0x4000d000 rw-u vma[3] no_free
|
||
*** three owners -- this page is claimed twice
|
||
|
||
> view dma
|
||
arena 0x3e000000-0x3f000000 16384 KB, 12284 KB free
|
||
free[0] 0x3e0a4000 8192 KB
|
||
free[1] 0x3ecec000 4092 KB
|
||
in use 4096 KB in 3 blocks
|
||
```
|
||
|
||
Text, and not a typed binary protocol, for three reasons that all turn out to be
|
||
the same reason:
|
||
|
||
- **It works at level 0.** Panic context cannot allocate, cannot frame records,
|
||
cannot wait for a client to acknowledge anything. It can put characters on a
|
||
serial line. A binary protocol needs a text fallback anyway, and then there are
|
||
two protocols.
|
||
- **It is usable before a client exists.** `cat`, `echo`, a serial terminal, the
|
||
guest's own shell. A protocol that needs its client built first is a protocol
|
||
nobody can debug.
|
||
- **It survives the thing it is debugging.** When the answer arrives garbled,
|
||
you can still read most of it.
|
||
|
||
A richer client can be layered on later by parsing this, and if that ever
|
||
happens the kernel does not change.
|
||
|
||
---
|
||
|
||
## Milestones
|
||
|
||
| | | |
|
||
|---|---|---|
|
||
| **D0** | **Symbols.** A compact sorted table emitted from `nm` at link time and linked into the image; `ksym_lookup(addr, &off)`. | **done** |
|
||
| **D1** | **A log that survives.** A ring behind `cprintf`, timestamped, in a region the allocator never sees — so a panic's last words are readable on the next boot. `/proc/kmsg`. | **done** |
|
||
| **D2** | **The introspection registry.** `dbg_register("dma", dma_dump)`. Seven views, `/dev/dbg`, `schdbg`, and `sysrq g`. | **done** |
|
||
| **D3** | **`whose <addr>`** — the cross-subsystem ownership query above. | **done** |
|
||
| **D4** | **Watchpoints**, DR0–DR3, break-on-write, reporting rip and call chain and **continuing** rather than stopping. | **done** |
|
||
| **D5** | **The network transport**, level 2 above: a userspace daemon bridging TCP to `/dev/dbg`. Off unless a boot flag turns it on, loopback by default. | |
|
||
|
||
### Why D0 and D1 come before the interesting parts
|
||
|
||
Neither is clever. Both multiply everything after them: a view that prints an
|
||
address is worth less than one that prints a name, and a query you can only run
|
||
*after* you have noticed something is worth less than a log that was already
|
||
running when it happened.
|
||
|
||
D1 in particular changes the job rather than improving it. Today a triple fault
|
||
leaves nothing at all — the screen is gone, the serial buffer is gone, and the
|
||
only evidence is whatever you happened to be watching. A log that outlives the
|
||
boot turns that into a file.
|
||
|
||
### What D1 turned out to cost
|
||
|
||
One measurement changed the design. The log was first placed at the top of RAM
|
||
beside the DMA arena, which is the obvious spot for a region nothing should
|
||
allocate — and on the first reboot its magic came back as ASCII, because **the
|
||
top of usable memory is exactly where the firmware makes its own allocations**
|
||
before handing the machine over. It sits above the kernel image instead, which
|
||
is both a stable address across boots and memory the loader has already
|
||
committed to. Recorded because the reasoning that picked the wrong place was
|
||
sound, and only the test was not.
|
||
|
||
### What D2 is really for
|
||
|
||
Not the views themselves — the *registry*. A subsystem publishes its state with
|
||
one line at the bottom of the file that owns the data, and nothing central has to
|
||
be edited. That is the only version of this that stays current; the `sysrq`
|
||
switch is the version that does not, and it is already three self-tests and no
|
||
state.
|
||
|
||
---
|
||
|
||
## The security note, for the parts not yet built
|
||
|
||
A debugger that reads arbitrary physical memory and a TCP listener are, together,
|
||
a remote root hole. That is true of every kernel debugger ever written and it is
|
||
not a reason not to build one, but it has to be decided rather than discovered:
|
||
|
||
- Level 2 is **off unless a boot flag turns it on**, and binds loopback by
|
||
default.
|
||
- No memory *writes* in the first cut. Reading is what finds bugs; writing is
|
||
what a later milestone can argue for on its own.
|
||
|
||
---
|
||
|
||
## What it will still not do
|
||
|
||
- **It will not step, break, or halt**, per the rule. If a bug genuinely needs a
|
||
stepper, this is the wrong tool and that is a deliberate trade.
|
||
- **No stack unwinding through interrupt frames.** `getcallerpcs()` walks `rbp`
|
||
chains and stops where the frame pointer stops; a trap frame in the middle of
|
||
a chain ends the walk.
|
||
- **No userspace symbolization.** The kernel will name kernel symbols. A fault
|
||
in a process reports the binary and the offset, and resolving that stays a
|
||
host-side job.
|
||
- **No profiling.** No sampling, no counters, no flame anything. A timer-driven
|
||
sampler is a different tool that happens to share a transport.
|
||
- **Watchpoints are limited to four**, by the hardware, and to 1/2/4/8 bytes.
|
||
There is no software watchpoint fallback and there should not be one.
|
||
- **It is not a tracer.** Static tracepoints and an event ring are the obvious
|
||
next thing after D4 and are not on this plan.
|
||
|
||
---
|
||
|
||
## On the freeze
|
||
|
||
[1.0.md](1.0.md) says no new capability starts until the register is empty, and
|
||
the register is not empty. Two things about that.
|
||
|
||
The bookkeeping answer is that the rule is *"adding a row requires closing one"*,
|
||
and **A released PCI device** closed when H2 landed. There is a slot.
|
||
|
||
The better answer is that the register is a list of **capabilities of the OS**,
|
||
and this is not one — nothing in it ships to a user or appears in the acceptance
|
||
test. It is a tool, and the case for spending time on a tool is that it makes
|
||
everything still on the register cheaper to finish. Gate 0 of [1.0.md](1.0.md)
|
||
is *"fix what is already wrong"*, assembled from every *What it does not do*
|
||
section in [../architecture/](../architecture/). That gate is a list of bugs to
|
||
find in a kernel that cannot currently be asked anything.
|
||
|
||
**Build the instrument before the survey**, and then the freeze holds for the
|
||
rest of it.
|