nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

debugger: the kernel can be asked a question

Browse source 
docs/plan/debugger.md, milestones D0, D1 and D2. What they replace is the
ninety-second loop -- add a cprintf, rebuild, boot, read the serial log --
which answers exactly the one question you thought of before the rebuild.

The rule the shape follows: **it observes, it does not control.** No
stepping, no software breakpoints, no stopping the world. Half the bugs
this kernel produces are timing -- an interrupt storm, a level-triggered
line nobody acknowledges, a driver that faults only on its second start --
and every one of them disappears when you halt the machine to look. And
the layering rule, which is the circularity rule the display already has:
a debugger that needs lwIP cannot debug lwIP.

D0 -- symbols. tools/ksyms.pl turns the first link pass into a table the
second pass carries. Two passes is enough because .ksym is placed after
.text and .rodata, so carrying the table cannot move anything recorded in
it; the build lists both passes' text symbols and refuses to link if they
ever disagree, because a symbol table that is subtly wrong gives confident
false answers. ksym.c reads it through weak symbols, so pass one -- which
has no table -- still links. A crash chain now reads panic+0x5f <-
proc_sysrq_cmd+0x99 <- ... <- syscall+0xf6.

D1 -- a log that outlives the kernel. Every character through consputc()
lands in a byte ring with a timestamp per line, readable as /proc/kmsg.
It sits in a region carved before the page allocator gets anything, with a
magic header, so a kernel finds the previous kernel's log still there --
including the whole PSOGM report. panic.c's own comment said "there is no
core dump and no log to go back to"; that is no longer true.

  The obvious place for it is the top of RAM beside the DMA arena. That
  was the first attempt and the magic came back as ASCII on the reboot,
  because the top of usable memory is exactly where the firmware makes its
  own allocations. It lives above the kernel image instead, which is a
  stable address across boots of the same build.

D2 -- the introspection registry. A subsystem publishes its state with one
line at the bottom of the file that owns the data:

  dbg_register("dma", dma_dump, "the DMA arena's free extents");

Seven views -- dma, mem, proc, vma, claims, pci, sock -- reachable three
ways: sysrq 'g' (always, no process and no filesystem), /dev/dbg (echo a
line, cat the answer), and schdbg. The protocol is lines of text because
the same bytes have to work where nothing can allocate or frame a record,
and because a protocol you cannot drive by hand is one you cannot debug.
struct sbuf moved out of kernel/dev/procfs.c into include/sbuf.h rather
than being written a second time.

Several views exist because of a specific bug. `vma` is the table that
decides whether exit() frees a page, and the DMA ownership bug that made
this worth building was a block that was never in it.

Also fixed, found by building with clang for the first time in a while:
struct iovec was defined in both posix.h and socket.h, and every file that
included both was a redefinition. GCC 15 defaults to C23, which permits an
identical struct redeclaration, so the gcc path never noticed -- while
clang defaults to gnu17 and rejected it, breaking the one toolchain
shell.nix exists to provide and the UEFI loader requires. Both are now
guarded.

D3 (`whose <addr>`, the cross-subsystem ownership query) and D4 (hardware
watchpoints on DR0-DR3) are not built and are described honestly as
missing in the architecture doc's "What it does not do".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
Nexoniarz 2026-09-14 00:58:45 +02:00
commit 1dd4ba0186
35 changed files with 2296 additions and 83 deletions
Download patch file Download diff file Expand all files Collapse all files

237
docs/architecture/debugging.md Normal file
View file

@ -0,0 +1,237 @@
# Finding out what the kernel is doing
Every bug in this project used to be found the same way: add a `cprintf`,
rebuild, boot, read the serial log, guess again. That loop is about ninety
seconds long and it answers exactly one question per turn — the question you
thought to ask *before* you rebuilt.
What ended it was a real bug. A DMA block was on two allocators' free lists at
once, and finding that took several rounds of that loop, because the kernel
could not be asked whether two allocators owned the same page. It could not be
asked anything.
> **The kernel had no way to answer a question it was not already compiled to
> answer.**
Three things close that, and none of them is a debugger in the stepping sense.
---
## The rule the shape follows
**It observes; it does not control.** No stepping, no software breakpoints, no
stopping the world.
That is not a scope compromise. Half the bugs this kernel actually produces are
*timing*: an interrupt storm at 77,000 a second, a level-triggered line nobody
acknowledges, a driver that faults only on its second start, a supervisor that
restarts a job fifteen times in forty-five seconds. **Every one of those
disappears when you halt the machine to look at it.**
And the layering rule, which is the circularity rule the display already has,
pointed somewhere else:
> A debugger that needs lwIP cannot debug lwIP. A debugger that needs a process
> cannot debug the scheduler. A debugger that needs to allocate cannot debug the
> allocator.
So the kernel's half is the minimum that works when nothing else does, and
everything convenient is a process.
---
## Naming an address
`tools/ksyms.pl` turns the kernel's own symbol table into a table the kernel
carries, and `ksym_lookup()` resolves an address through it. A crash chain used
to be nine hex numbers:
```
Call chain
#0 ffff80000010a9d9 panic+0x5f
#1 ffff80000011e07f proc_sysrq_cmd+0x99
#2 ffff80000011e205 proc_write_sysrq+0x6e
#3 ffff800000130159 procfs_write+0x59
#4 ffff80000012e566 writei+0x45
#5 ffff800000128a96 filewrite+0x2ec
#6 ffff80000016c0fa sys_writev+0x12b
#7 ffff80000016986e syscall+0xf6
#8 ffff800000100d2e syscall_trapret
```
### Two link passes, and why two is enough
The kernel is linked twice. Pass one exists to be read; pass two carries the
table built from it.
The obvious objection is that adding forty kilobytes to the image moves
everything the table describes — which is why Linux's `kallsyms` needs three
passes and a convergence check. **`.ksym` is placed after `.text` and
`.rodata`,** so it cannot move a single address the table records. Two passes,
and the second is stable by construction.
Not by assumption, though: the build lists the text symbols of both passes and
**fails** if they differ. A symbol table that is subtly wrong for every address
is worse than none, because it produces confident, false answers.
`ksym.c` reads the table through **weak** symbols, so pass one — which has no
table — still links, and a tree with the generator removed still boots and
simply cannot name anything.
---
## A log that outlives the kernel
`cprintf` used to go to a serial line and a screen and then be gone. `panic.c`
said so in its own header: *"there is no core dump and no log to go back to."*
There is now. Every character that passes through `consputc()` — which is every
kernel message, including the whole PSOGM report — is also appended to a byte
ring with a timestamp per line. It is readable as **`/proc/kmsg`**.
The ring lives in a region carved out before the page allocator is given
anything, and it has a magic header, so **a kernel that reboots finds the
previous kernel's log still there**:
```
--- boot 1 ---
[ 0.00] e820: 1018 MB usable in 7 regions, top 1016 MB (19 entries)
...
[ 7.18] PSOGM -- Purple Screen of Guru Meditation
[ 7.18] Call chain
[ 7.18] #0 ffff80000010a9d9 panic+0x5f
...
--- boot 2 ---
[ 0.00] klog: 128 KB at ffff80000053e000, 2162 bytes carried over from the previous boot
```
### Where it lives, and why not where you would put it
The obvious place is the top of RAM, beside the DMA arena. That was the first
attempt, and on the reboot the magic came back as ASCII — because **the top of
usable memory is exactly where the firmware puts its own allocations** before it
hands the machine over.
It sits immediately above the kernel image instead. `end` is the same address on
every boot of the same build, which is the property a region has to have to be
found again, and the bytes just past the kernel are the ones the loader has
already committed to leaving alone. No hole has to be punched in the allocator:
`kinit1`'s lower bound is a parameter.
The magic and a header checksum mean a region something else has used is
re-initialised rather than printed as if it were a log. **A reboot that does not
preserve DRAM loses the log, and says so** rather than showing garbage.
Two properties of the writer are deliberate. It **takes no lock** — it runs
inside `consputc()`, and a lock there would be a way for the logger to deadlock
the thing it exists to report on. And it **buffers before it exists**: the region
is carved after `e820_init()`, so the handful of lines before that go into a
static array and are flushed in, because a log that is blank for the earliest
part of the boot is blank for the part that is hardest to debug any other way.
---
## Asking a question
`/dev/dbg` is a conversation: write a line, read the answer.
```
# schdbg dma
arena 0xffff80003e8ed000-0xffff80003f8ed000
16384 KB total, 12295 KB free in 3 extent(s), 4088 KB out
# base phys bytes
0 0xffff80003e8f3000 0x000000003e8f3000 4091904
1 0xffff80003ecea200 0x000000003ecea200 3584
2 0xffff80003f0d3000 0x000000003f0d3000 8495104
```
It is text, not a typed binary protocol, and the reason is that the same bytes
have to work at every level — including the one where nothing can allocate,
frame a record, or wait for a client to acknowledge anything. A protocol you
cannot drive by hand is a protocol you cannot debug. Everything `schdbg` does is
also:
```
# echo dma > /dev/dbg && cat /dev/dbg
```
### The registry is the point, not the views
A subsystem publishes its state with **one line at the bottom of the file that
owns the data**:
```c
dbg_register("dma", dma_dump, "the DMA arena's free extents");
```
Nothing central is edited to add one. That is the difference between something
that stays current and something that does not — `/proc/sysrq` is the version
that does not, a `switch` in one file with a letter per action, and after two
years it holds three self-tests and no state at all.
| View | |
|---|---|
| `dma` | the arena's free extents, in virtual and physical |
| `mem` | the page allocator's free list, and whether it has a cycle in it |
| `proc` | every process, with state, size and its mmap bump pointer |
| `vma` | every mapping a process holds but does not own the pages of |
| `claims` | devices, DMA blocks and interrupts held by userspace drivers |
| `pci` | the bus, with each device's command register decoded |
| `sock` | the socket table, with buffered bytes and descriptors in flight |
`all` prints every view; `help` lists them; a view may take an argument, so
`vma 4` narrows to one process.
Several of those exist because of a specific bug. `vma` is the table that decides
whether `exit()` frees a page or leaves it alone, and the bug that made this
worth building was a DMA block that was never in it. `claims` shows the three
things that go wrong with a userspace driver — a device held by a process that
no longer exists, DMA handed out and never returned, an interrupt line left
masked with nobody reading it. `pci` decodes the command register because
"is bus mastering actually off?" is a question with a wrong answer that looks
fine.
### Three ways in, and the one that always works
| | Works when |
|---|---|
| `sysrq g` | always — every view to the console, no process, no filesystem, no client program |
| `/dev/dbg` | there is a scheduler and a filesystem |
| `schdbg` | there is a shell |
Views run in ordinary process context and may take locks. **They must not be
called from `panic()`.**
---
## What it does not do
- **It does not step, break, or halt.** If a bug genuinely needs a stepper, this
is the wrong tool, and that is a deliberate trade rather than an omission.
- **No cross-subsystem queries.** *"Who owns physical page `0x3eceb000`?"* — the
question that would have ended the investigation that prompted all this — still
cannot be asked. Each view answers for its own subsystem and nothing
correlates them. That is `whose`, and it is not built.
- **No watchpoints.** DR0–DR3 are untouched and `T_DEBUG` maps to `SIGTRAP`, so
"something zeroed my page table, tell me who" has no answer.
- **No network transport.** The plan puts a TCP bridge in a userspace daemon,
deliberately outside the kernel; it does not exist, so the debugger is local
or serial only.
- **No memory writes**, and no way to poke anything. Reading is what finds bugs.
- **`all` truncates at 16 KB** and says so. On a busy machine, ask for one view
at a time.
- **One conversation at a time.** The answer buffer is global, so two programs
asking at once get one answer each in some order — never half of the other's,
but not two independent sessions either.
- **No userspace symbolization.** The kernel names kernel symbols. A fault in a
process still reports a bare address, and resolving it is a host-side job.
- **The stack walk stops where the frame pointer stops.** `getcallerpcs()` walks
`rbp` chains, so a trap frame in the middle of a chain ends the walk.
- **`sysrq g` costs you log history.** Its output goes through `cprintf`, so
sixteen kilobytes of views push the same amount out of a 128 KB ring.
- **No tracing.** No static tracepoints, no event ring, no counters. "What
happened in the two hundred events before this" is not a question this can
answer.
What remains is in [../plan/debugger.md](../plan/debugger.md).