---
name: schunix-smp-and-memory-bugs
description: "SMP works as of 2026-08-08 — the five bugs it took, the __thread lesson, and the 224 MB RAM floor that is also gone."
metadata: 
  node_type: memory
  type: project
  originSessionId: 9d8a2aef-bcf0-47a0-96c6-f706efab4a95
  modified: 2026-08-08T19:21:20.756Z
---

Both of Nexoniarz's 2026-08-08 reports are **fixed**. `docs/plan/roadmap.md` §3b has
the full write-up.

## SMP works

One, two and four processors boot; every CPU prints `cpu%d: starting` and
`/proc/cpuinfo` lists them all. `smp_start_aps` in `kernel/core/main.c` is 1.
Verified on QEMU at `-smp 1/2/4`, with all four `bootcheck` modes passing.

**Five bugs, each hidden behind the one before it.** All were invisible under
QEMU, which publishes only one processor in the legacy MP table; VMware
published the real count, which is why the triple fault only ever appeared
there.

1. **Enumeration** — `mpinit()` read the 1994 Intel MP table.
   `kernel/arch/x86_64/acpi.c` now reads the ACPI MADT (RSDP → RSDT/XSDT →
   MADT type 0). Do this *first*: nothing else is reproducible until QEMU
   reports more than one CPU.
2. **Triple fault** — `entry64high` in `entry.S` had **no AP branch at all**;
   the check was a comment saying it had never been written. Every AP fell into
   the BSP's path, took its stack, and ran `kmain()` twice. Now tests `EBX == 1`,
   which `entryother.S` had always been setting.
3. **`panic("acquire")`** — `seginit()` called `kalloc()` for its per-CPU page.
   kalloc locks, `acquire()` calls `holding()`, `holding()` reads `cpu` — which
   `seginit()` was about to set up. `startothers()` now preallocates each AP's
   page on the BSP.
4. **`panic("sched locks")`** — `kernel_fs_base` was **one global** written by
   every CPU and read by `trapasm.S` on every entry from user mode, so all CPUs
   shared one per-CPU block. `RESTORE_KERNEL_FS` now reads `%gs:0`; `seginit()`
   sets GS base per CPU. No swapgs, because `sys_arch_prctl()` implements only
   ARCH_SET_FS/GET_FS and musl uses FS for TLS — **if ARCH_SET_GS is ever added
   this must become a swapgs pair.**
5. **Null `proc` in `sleep()`, then `#GP` in `syscall_entry`** — see below.

## The lesson: `__thread` is wrong for per-CPU kernel data

`cpu` and `proc` were `__thread`. That means per *thread*, and the compiler is
entitled to cache the thread pointer across calls on that basis:

    movq %fs:0x0, %rax          # read the thread pointer
    movq %rax, -0x18(%rbp)      # spill it
    callq sched                 # <-- the CPU can change here
    movq -0x18(%rbp), %rax      # reload the OLD pointer
    movq (%rax,%rcx), %rax      # proc, from the previous CPU's block
    movq $0x0, 0x40(%rax)       # NULL, because that CPU set proc = 0

But `%fs` here is per *CPU*, and a kernel thread changes CPU at every `sched()`.
Now `mycpu()`/`myproc()` — inline `volatile` asm reading `%gs` at each use, so
nothing can be hoisted across a call. Named that way (as xv6-riscv does) and
**not** `cpu`/`proc`, because a macro by those names would also rewrite every
`struct cpu` and `struct proc`; macros have no namespace, tags do.

**Per-CPU block layout — shared with assembly, do not reorder:**

    +0   self-pointer     x86-64 TLS ABI (%fs:0)
    +8   syscall scratch  trapasm.S parks %rsp here
    +16  struct cpu *     PERCPU_CPU
    +24  struct proc *    PERCPU_PROC

**Offset 8 is not free.** `syscall_entry` uses `%fs:(8)` as scratch, and since
FS base and GS base are the same address, `%fs:8` *is* `%gs:8`. Putting `cpu`
there made every syscall overwrite it with `%rsp` — a `#GP` inside
`syscall_entry`. That was bug 5.

## The 224 MB RAM floor is gone

`kernel/arch/x86_64/e820.c` reads the firmware map and sets `phystop`.
**Floor is 32 MB, was 224 MB**; 512 MB in gives `MemTotal: 524160 kB`. Take the
end of the usable region *containing the kernel* (`EXTMEM`), not the global
maximum — `freerange()` walks from the kernel's end to `phystop`, so jumping a
reserved hole puts firmware tables on the free list. Ceiling `PHYSTOP_MAX` is
3 GB: `kvmalloc()` maps 0–4 GB as four 1 GB pages with the top one uncached for
PCI BARs.

## Techniques worth reusing

- `qemu -d int,cpu_reset -D log` printed `v=0e` with `SP=ffff80000000ffb0` — a
  stack pointer in low memory, which said instantly the AP was not on its own
  stack and pointed at the entry path rather than anything in C.
- The symbol table is too sparse for nearest-preceding-symbol. Use
  `build/kernel/kernel.asm` and
  `awk '/^[0-9a-f]+ <.*>:/{fn=$0} index($0,"<addr>:")==1{print fn; print; exit}'`
  — gives the enclosing function *and* the exact instruction.
- **Do not blanket-`sed` an identifier rename.** Doing that for `proc`/`cpu`
  rewrote `#include "proc.h"`, string literals and declarations, and this is not
  a git repo so there was no undo. The recovery that worked: revert the whole
  substitution, reapply only the safe `proc->` → `myproc()->` form, then let the
  compiler name each remaining site.