57aff40228
docs/ was ten files where a plan, a description, a guide and a bug report all
sat next to each other. It is now architecture/ (what SchOS is), plan/ (what it
will be), guides/ (how to work on it) and history/ (why it is like this), with
an index that states the one rule holding it together: when something is
finished it moves from plan/ to architecture/ and is rewritten as a description
of the thing rather than a record of building it.
A ticked checkbox says the work happened. It does not say what the work
*produced*, and six months later the checkbox is all that is left. The network
stack proved the point: finished, verified end to end, and for a month the only
description of it anywhere was a checkbox four levels of indentation deep.
So roadmap.md went from 534 lines to 181 -- nothing deleted, the done work
graduated. Seven subsystems that were finished and undescribed now have
descriptions: boot, kernel, syscalls, filesystem, networking, graphics, usb,
init. Every one ends with "What it does not do", which is the section that pays;
a document listing only strengths is marketing, and the reader who needs it most
is the one about to trip over a limitation nobody wrote down.
schfs.md still opened with "phases 1 through 3 1/2 are done, phase 4 is next"
long after all five had landed. It is a description now, with the phase record
and the hard-won bugs moved to history/.
Three corrections found on the way, all load-bearing:
* projects/schcom/README.md claimed SCM_RIGHTS was done. There is no
sendmsg/recvmsg in the kernel at all, and an entire GUI plan rested on it.
* The root README said "No TCP/IP" months after the stack landed.
* history/kernel.md sec.7 was marked "in progress" on a problem that process
groups solved.
Also adds plan/1.0.md, which draws a finish line so the open list stops being
infinite, and marks schwm/ and schde/ superseded: in Wayland the compositor is
the window manager, so that split dissolves rather than being abandoned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
98 lines
4.9 KiB
Markdown
98 lines
4.9 KiB
Markdown
---
|
||
name: schunix-smp-and-memory-bugs
|
||
description: "SMP works as of 2026-08-08 — the five bugs it took, the __thread lesson, and the 224 MB RAM floor that is also gone."
|
||
metadata:
|
||
node_type: memory
|
||
type: project
|
||
originSessionId: 9d8a2aef-bcf0-47a0-96c6-f706efab4a95
|
||
modified: 2026-08-08T19:21:20.756Z
|
||
---
|
||
|
||
Both of Nexoniarz's 2026-08-08 reports are **fixed**. `docs/plan/roadmap.md` §3b has
|
||
the full write-up.
|
||
|
||
## SMP works
|
||
|
||
One, two and four processors boot; every CPU prints `cpu%d: starting` and
|
||
`/proc/cpuinfo` lists them all. `smp_start_aps` in `kernel/core/main.c` is 1.
|
||
Verified on QEMU at `-smp 1/2/4`, with all four `bootcheck` modes passing.
|
||
|
||
**Five bugs, each hidden behind the one before it.** All were invisible under
|
||
QEMU, which publishes only one processor in the legacy MP table; VMware
|
||
published the real count, which is why the triple fault only ever appeared
|
||
there.
|
||
|
||
1. **Enumeration** — `mpinit()` read the 1994 Intel MP table.
|
||
`kernel/arch/x86_64/acpi.c` now reads the ACPI MADT (RSDP → RSDT/XSDT →
|
||
MADT type 0). Do this *first*: nothing else is reproducible until QEMU
|
||
reports more than one CPU.
|
||
2. **Triple fault** — `entry64high` in `entry.S` had **no AP branch at all**;
|
||
the check was a comment saying it had never been written. Every AP fell into
|
||
the BSP's path, took its stack, and ran `kmain()` twice. Now tests `EBX == 1`,
|
||
which `entryother.S` had always been setting.
|
||
3. **`panic("acquire")`** — `seginit()` called `kalloc()` for its per-CPU page.
|
||
kalloc locks, `acquire()` calls `holding()`, `holding()` reads `cpu` — which
|
||
`seginit()` was about to set up. `startothers()` now preallocates each AP's
|
||
page on the BSP.
|
||
4. **`panic("sched locks")`** — `kernel_fs_base` was **one global** written by
|
||
every CPU and read by `trapasm.S` on every entry from user mode, so all CPUs
|
||
shared one per-CPU block. `RESTORE_KERNEL_FS` now reads `%gs:0`; `seginit()`
|
||
sets GS base per CPU. No swapgs, because `sys_arch_prctl()` implements only
|
||
ARCH_SET_FS/GET_FS and musl uses FS for TLS — **if ARCH_SET_GS is ever added
|
||
this must become a swapgs pair.**
|
||
5. **Null `proc` in `sleep()`, then `#GP` in `syscall_entry`** — see below.
|
||
|
||
## The lesson: `__thread` is wrong for per-CPU kernel data
|
||
|
||
`cpu` and `proc` were `__thread`. That means per *thread*, and the compiler is
|
||
entitled to cache the thread pointer across calls on that basis:
|
||
|
||
movq %fs:0x0, %rax # read the thread pointer
|
||
movq %rax, -0x18(%rbp) # spill it
|
||
callq sched # <-- the CPU can change here
|
||
movq -0x18(%rbp), %rax # reload the OLD pointer
|
||
movq (%rax,%rcx), %rax # proc, from the previous CPU's block
|
||
movq $0x0, 0x40(%rax) # NULL, because that CPU set proc = 0
|
||
|
||
But `%fs` here is per *CPU*, and a kernel thread changes CPU at every `sched()`.
|
||
Now `mycpu()`/`myproc()` — inline `volatile` asm reading `%gs` at each use, so
|
||
nothing can be hoisted across a call. Named that way (as xv6-riscv does) and
|
||
**not** `cpu`/`proc`, because a macro by those names would also rewrite every
|
||
`struct cpu` and `struct proc`; macros have no namespace, tags do.
|
||
|
||
**Per-CPU block layout — shared with assembly, do not reorder:**
|
||
|
||
+0 self-pointer x86-64 TLS ABI (%fs:0)
|
||
+8 syscall scratch trapasm.S parks %rsp here
|
||
+16 struct cpu * PERCPU_CPU
|
||
+24 struct proc * PERCPU_PROC
|
||
|
||
**Offset 8 is not free.** `syscall_entry` uses `%fs:(8)` as scratch, and since
|
||
FS base and GS base are the same address, `%fs:8` *is* `%gs:8`. Putting `cpu`
|
||
there made every syscall overwrite it with `%rsp` — a `#GP` inside
|
||
`syscall_entry`. That was bug 5.
|
||
|
||
## The 224 MB RAM floor is gone
|
||
|
||
`kernel/arch/x86_64/e820.c` reads the firmware map and sets `phystop`.
|
||
**Floor is 32 MB, was 224 MB**; 512 MB in gives `MemTotal: 524160 kB`. Take the
|
||
end of the usable region *containing the kernel* (`EXTMEM`), not the global
|
||
maximum — `freerange()` walks from the kernel's end to `phystop`, so jumping a
|
||
reserved hole puts firmware tables on the free list. Ceiling `PHYSTOP_MAX` is
|
||
3 GB: `kvmalloc()` maps 0–4 GB as four 1 GB pages with the top one uncached for
|
||
PCI BARs.
|
||
|
||
## Techniques worth reusing
|
||
|
||
- `qemu -d int,cpu_reset -D log` printed `v=0e` with `SP=ffff80000000ffb0` — a
|
||
stack pointer in low memory, which said instantly the AP was not on its own
|
||
stack and pointed at the entry path rather than anything in C.
|
||
- The symbol table is too sparse for nearest-preceding-symbol. Use
|
||
`build/kernel/kernel.asm` and
|
||
`awk '/^[0-9a-f]+ <.*>:/{fn=$0} index($0,"<addr>:")==1{print fn; print; exit}'`
|
||
— gives the enclosing function *and* the exact instruction.
|
||
- **Do not blanket-`sed` an identifier rename.** Doing that for `proc`/`cpu`
|
||
rewrote `#include "proc.h"`, string literals and declarations, and this is not
|
||
a git repo so there was no undo. The recovery that worked: revert the whole
|
||
substitution, reapply only the safe `proc->` → `myproc()->` form, then let the
|
||
compiler name each remaining site.
|