3182da3965
Claude Code keeps per-project memory at <home>/.claude/projects/<key>/,
where <key> is the project's absolute path with every separator and the
drive colon turned into a dash. That makes it both rename-fragile and
machine-specific, and this project has hit each of those once:
- D:\DEV\Schunix -> D:\DEV\SchOS changed the key, and 43 memories were
invisible until moved across by hand.
- Development happens from Windows and from Linux, and each looks in a
different place for the same checkout.
Committing .claude/memory/ fixes both -- git carries it across renames
and between hosts -- and `import` puts it wherever the local machine
expects. .claude/settings.local.json stays ignored: those are per-machine
permissions.
The colon maps to a dash rather than being dropped. Deleting it gives
D-DEV-SchOS, which matches nothing and silently finds zero memories; that
was the first attempt. On Git Bash the key comes from `cygpath -w`,
because Claude Code is a Windows program and sees D:\DEV\SchOS, not
/d/DEV/SchOS.
98 lines
4.9 KiB
Markdown
98 lines
4.9 KiB
Markdown
---
|
||
name: schunix-smp-and-memory-bugs
|
||
description: "SMP works as of 2026-08-08 — the five bugs it took, the __thread lesson, and the 224 MB RAM floor that is also gone."
|
||
metadata:
|
||
node_type: memory
|
||
type: project
|
||
originSessionId: 9d8a2aef-bcf0-47a0-96c6-f706efab4a95
|
||
modified: 2026-08-08T19:21:20.756Z
|
||
---
|
||
|
||
Both of Nexoniarz's 2026-08-08 reports are **fixed**. `docs/roadmap.md` §3b has
|
||
the full write-up.
|
||
|
||
## SMP works
|
||
|
||
One, two and four processors boot; every CPU prints `cpu%d: starting` and
|
||
`/proc/cpuinfo` lists them all. `smp_start_aps` in `kernel/core/main.c` is 1.
|
||
Verified on QEMU at `-smp 1/2/4`, with all four `bootcheck` modes passing.
|
||
|
||
**Five bugs, each hidden behind the one before it.** All were invisible under
|
||
QEMU, which publishes only one processor in the legacy MP table; VMware
|
||
published the real count, which is why the triple fault only ever appeared
|
||
there.
|
||
|
||
1. **Enumeration** — `mpinit()` read the 1994 Intel MP table.
|
||
`kernel/arch/x86_64/acpi.c` now reads the ACPI MADT (RSDP → RSDT/XSDT →
|
||
MADT type 0). Do this *first*: nothing else is reproducible until QEMU
|
||
reports more than one CPU.
|
||
2. **Triple fault** — `entry64high` in `entry.S` had **no AP branch at all**;
|
||
the check was a comment saying it had never been written. Every AP fell into
|
||
the BSP's path, took its stack, and ran `kmain()` twice. Now tests `EBX == 1`,
|
||
which `entryother.S` had always been setting.
|
||
3. **`panic("acquire")`** — `seginit()` called `kalloc()` for its per-CPU page.
|
||
kalloc locks, `acquire()` calls `holding()`, `holding()` reads `cpu` — which
|
||
`seginit()` was about to set up. `startothers()` now preallocates each AP's
|
||
page on the BSP.
|
||
4. **`panic("sched locks")`** — `kernel_fs_base` was **one global** written by
|
||
every CPU and read by `trapasm.S` on every entry from user mode, so all CPUs
|
||
shared one per-CPU block. `RESTORE_KERNEL_FS` now reads `%gs:0`; `seginit()`
|
||
sets GS base per CPU. No swapgs, because `sys_arch_prctl()` implements only
|
||
ARCH_SET_FS/GET_FS and musl uses FS for TLS — **if ARCH_SET_GS is ever added
|
||
this must become a swapgs pair.**
|
||
5. **Null `proc` in `sleep()`, then `#GP` in `syscall_entry`** — see below.
|
||
|
||
## The lesson: `__thread` is wrong for per-CPU kernel data
|
||
|
||
`cpu` and `proc` were `__thread`. That means per *thread*, and the compiler is
|
||
entitled to cache the thread pointer across calls on that basis:
|
||
|
||
movq %fs:0x0, %rax # read the thread pointer
|
||
movq %rax, -0x18(%rbp) # spill it
|
||
callq sched # <-- the CPU can change here
|
||
movq -0x18(%rbp), %rax # reload the OLD pointer
|
||
movq (%rax,%rcx), %rax # proc, from the previous CPU's block
|
||
movq $0x0, 0x40(%rax) # NULL, because that CPU set proc = 0
|
||
|
||
But `%fs` here is per *CPU*, and a kernel thread changes CPU at every `sched()`.
|
||
Now `mycpu()`/`myproc()` — inline `volatile` asm reading `%gs` at each use, so
|
||
nothing can be hoisted across a call. Named that way (as xv6-riscv does) and
|
||
**not** `cpu`/`proc`, because a macro by those names would also rewrite every
|
||
`struct cpu` and `struct proc`; macros have no namespace, tags do.
|
||
|
||
**Per-CPU block layout — shared with assembly, do not reorder:**
|
||
|
||
+0 self-pointer x86-64 TLS ABI (%fs:0)
|
||
+8 syscall scratch trapasm.S parks %rsp here
|
||
+16 struct cpu * PERCPU_CPU
|
||
+24 struct proc * PERCPU_PROC
|
||
|
||
**Offset 8 is not free.** `syscall_entry` uses `%fs:(8)` as scratch, and since
|
||
FS base and GS base are the same address, `%fs:8` *is* `%gs:8`. Putting `cpu`
|
||
there made every syscall overwrite it with `%rsp` — a `#GP` inside
|
||
`syscall_entry`. That was bug 5.
|
||
|
||
## The 224 MB RAM floor is gone
|
||
|
||
`kernel/arch/x86_64/e820.c` reads the firmware map and sets `phystop`.
|
||
**Floor is 32 MB, was 224 MB**; 512 MB in gives `MemTotal: 524160 kB`. Take the
|
||
end of the usable region *containing the kernel* (`EXTMEM`), not the global
|
||
maximum — `freerange()` walks from the kernel's end to `phystop`, so jumping a
|
||
reserved hole puts firmware tables on the free list. Ceiling `PHYSTOP_MAX` is
|
||
3 GB: `kvmalloc()` maps 0–4 GB as four 1 GB pages with the top one uncached for
|
||
PCI BARs.
|
||
|
||
## Techniques worth reusing
|
||
|
||
- `qemu -d int,cpu_reset -D log` printed `v=0e` with `SP=ffff80000000ffb0` — a
|
||
stack pointer in low memory, which said instantly the AP was not on its own
|
||
stack and pointed at the entry path rather than anything in C.
|
||
- The symbol table is too sparse for nearest-preceding-symbol. Use
|
||
`build/kernel/kernel.asm` and
|
||
`awk '/^[0-9a-f]+ <.*>:/{fn=$0} index($0,"<addr>:")==1{print fn; print; exit}'`
|
||
— gives the enclosing function *and* the exact instruction.
|
||
- **Do not blanket-`sed` an identifier rename.** Doing that for `proc`/`cpu`
|
||
rewrote `#include "proc.h"`, string literals and declarations, and this is not
|
||
a git repo so there was no undo. The recovery that worked: revert the whole
|
||
substitution, reapply only the safe `proc->` → `myproc()->` form, then let the
|
||
compiler name each remaining site.
|