SchOS/.claude/memory/schunix-smp-and-memory-bugs.md

name	description	metadata
schunix-smp-and-memory-bugs	SMP works as of 2026-08-08 — the five bugs it took, the __thread lesson, and the 224 MB RAM floor that is also gone.	node_type type originSessionId modified memory project 9d8a2aef-bcf0-47a0-96c6-f706efab4a95 2026-08-08T19:21:20.756Z

Both of Nexoniarz's 2026-08-08 reports are fixed. docs/roadmap.md §3b has the full write-up.

SMP works

One, two and four processors boot; every CPU prints cpu%d: starting and /proc/cpuinfo lists them all. smp_start_aps in kernel/core/main.c is 1. Verified on QEMU at -smp 1/2/4, with all four bootcheck modes passing.

Five bugs, each hidden behind the one before it. All were invisible under QEMU, which publishes only one processor in the legacy MP table; VMware published the real count, which is why the triple fault only ever appeared there.

1. Enumeration — mpinit() read the 1994 Intel MP table. kernel/arch/x86_64/acpi.c now reads the ACPI MADT (RSDP → RSDT/XSDT → MADT type 0). Do this first: nothing else is reproducible until QEMU reports more than one CPU.
2. Triple fault — entry64high in entry.S had no AP branch at all; the check was a comment saying it had never been written. Every AP fell into the BSP's path, took its stack, and ran kmain() twice. Now tests EBX == 1, which entryother.S had always been setting.
3. panic("acquire") — seginit() called kalloc() for its per-CPU page. kalloc locks, acquire() calls holding(), holding() reads cpu — which seginit() was about to set up. startothers() now preallocates each AP's page on the BSP.
4. panic("sched locks") — kernel_fs_base was one global written by every CPU and read by trapasm.S on every entry from user mode, so all CPUs shared one per-CPU block. RESTORE_KERNEL_FS now reads %gs:0; seginit() sets GS base per CPU. No swapgs, because sys_arch_prctl() implements only ARCH_SET_FS/GET_FS and musl uses FS for TLS — if ARCH_SET_GS is ever added this must become a swapgs pair.
5. Null proc in sleep(), then #GP in syscall_entry — see below.

The lesson: __thread is wrong for per-CPU kernel data

cpu and proc were __thread. That means per thread, and the compiler is entitled to cache the thread pointer across calls on that basis:

movq %fs:0x0, %rax          # read the thread pointer
movq %rax, -0x18(%rbp)      # spill it
callq sched                 # <-- the CPU can change here
movq -0x18(%rbp), %rax      # reload the OLD pointer
movq (%rax,%rcx), %rax      # proc, from the previous CPU's block
movq $0x0, 0x40(%rax)       # NULL, because that CPU set proc = 0

But %fs here is per CPU, and a kernel thread changes CPU at every sched(). Now mycpu()/myproc() — inline volatile asm reading %gs at each use, so nothing can be hoisted across a call. Named that way (as xv6-riscv does) and not cpu/proc, because a macro by those names would also rewrite every struct cpu and struct proc; macros have no namespace, tags do.

Per-CPU block layout — shared with assembly, do not reorder:

+0   self-pointer     x86-64 TLS ABI (%fs:0)
+8   syscall scratch  trapasm.S parks %rsp here
+16  struct cpu *     PERCPU_CPU
+24  struct proc *    PERCPU_PROC

Offset 8 is not free. syscall_entry uses %fs:(8) as scratch, and since FS base and GS base are the same address, %fs:8 is %gs:8. Putting cpu there made every syscall overwrite it with %rsp — a #GP inside syscall_entry. That was bug 5.

The 224 MB RAM floor is gone

kernel/arch/x86_64/e820.c reads the firmware map and sets phystop. Floor is 32 MB, was 224 MB; 512 MB in gives MemTotal: 524160 kB. Take the end of the usable region containing the kernel (EXTMEM), not the global maximum — freerange() walks from the kernel's end to phystop, so jumping a reserved hole puts firmware tables on the free list. Ceiling PHYSTOP_MAX is 3 GB: kvmalloc() maps 0–4 GB as four 1 GB pages with the top one uncached for PCI BARs.

Techniques worth reusing

• qemu -d int,cpu_reset -D log printed v=0e with SP=ffff80000000ffb0 — a stack pointer in low memory, which said instantly the AP was not on its own stack and pointed at the entry path rather than anything in C.
• The symbol table is too sparse for nearest-preceding-symbol. Use build/kernel/kernel.asm and awk '/^[0-9a-f]+ <.*>:/{fn=$0} index($0,"<addr>:")==1{print fn; print; exit}' — gives the enclosing function and the exact instruction.
• Do not blanket-sed an identifier rename. Doing that for proc/cpu rewrote #include "proc.h", string literals and declarations, and this is not a git repo so there was no undo. The recovery that worked: revert the whole substitution, reapply only the safe proc-> → myproc()-> form, then let the compiler name each remaining site.