nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

kmem: a slab allocator, and what the fixed tables actually cost

Browse source 
Gate 0's last structural row. kernel/core/kmem.c: slabs are pages from
kalloc() with a header at the front, and free objects hold the free list in
their own first eight bytes.

That last detail is the whole design. **An object carries no header and
freeing performs no search**: the page base of any object *is* its slab, and
the slab names its cache. It is also exactly why an object cannot exceed a
page, which matters below.

Two interfaces. kmem_cache_create/alloc/free for a subsystem with one kind of
thing -- it gets its own line in /proc/slab, which is most of why naming it is
worth the argument -- and kmalloc/kmfree over power-of-two classes from 16 to
2048 for code with no single type to build a cache around. Each cache keeps
partial and full lists; a slab that empties goes straight back to kalloc()
rather than being kept, so /proc/slab never overstates what is held.

`make kmem-test` asserts the bookkeeping on the build machine. Same arrangement
as schfs-objtest, and for the same reason: slab lists, free-list threading, the
page-to-cache lookup and whether pages actually come back are all things that
can be wrong without a kernel to be wrong in. 26 checks, including one that a
permissive allocator cannot pass -- write a distinct pattern through 512
objects and read them all back, because two objects sharing storage shows up
there and nowhere else.

One bug the test found immediately: kmem_max_obj() returned the raw page
remainder, which kmem_cache_create() then rounded *up* past its own limit. The
largest legal size was reported as illegal.

**It does not do what the register said it would.** The claim was that every
fixed table exists to work around kalloc. Measured:

    struct proc    10656    NPROC 128     larger than a page
    struct socket   4928    NSOCK 32      larger than a page
    struct buf      4272    NBUF 256      larger than a page
    struct inode     280    NINODE 200    a cache, bounded on purpose
    struct device    120    NDEVICE 32    3.8 KB, never hit
    struct file       48    NFILE 512     24 KB, never hit

The three largest cannot go in a page-backed slab whatever it does. NINODE is
a cache and is meant to be bounded. NFILE and NDEVICE are small enough that
their tables are rounding errors and their ceilings have never been reached.
Freeing the big ones needs a contiguous multi-page allocator, which is a
different thing and is not written. That is now in the register and in
docs/architecture/kernel.md rather than left as a claim nobody checked.

What it does fix is real, and was measured rather than assumed:

  - **A pipe is 640 bytes and came from a whole 4096-byte page.** Every `|` in
    a shell command burned three and a half kilobytes it never touched. Six
    pipes per page now, and /proc/slab shows the page going back when they
    close.
  - **procread() built its answer in a 2048-byte array on the kernel stack**,
    under a comment that had warned about exactly that since it was written --
    it stayed a local because a whole page for 2 KB felt worse, and kmalloc is
    the answer that was missing. Its frame is 64 bytes now instead of ~2100.
  - procfs's other generator took a page from kalloc for the same 2 KB.

Verified on a boot: /proc/slab tracks pipe and kmalloc-2048 through real
pipelines, allocs and frees balance, and $(), ps, time, readlink, sigtest and
rename are all still green with no traps.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-09-22 09:31:55 +02:00
commit c98bdf0019
10 changed files with 651 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

318
kernel/core/kmem.c Normal file
View file

@ -0,0 +1,318 @@
// Small-object allocation. See include/kmem.h for what this is for.
//
// A slab is one page from kalloc(), with a header at the front and objects
// packed after it. Free objects hold the free list in their own first eight
// bytes -- so an object may never be smaller than a pointer, which is what the
// alignment floor below is really enforcing.
//
// Freeing takes no search and no per-object header: an object's page base is
// its slab, and the slab names its cache. That is the entire design, and it is
// the reason an object cannot exceed a page.
//
// Every cache keeps two lists. `partial` holds slabs with at least one free
// object, `full` holds the rest. Allocation takes the head of `partial`;
// freeing into a full slab moves it back. A slab that empties completely is
// handed back to kalloc() straight away rather than kept: hoarding would make
// /proc/slab lie about what the kernel is using, and a subsystem that
// oscillates across a slab boundary is rare enough not to design around.
#ifdef KMEM_HOST
// Host harness: see tools/kmemtest.c. kalloc/kfree become page-aligned
// malloc/free and the locks vanish, because the point of the harness is the
// allocator's own bookkeeping rather than its concurrency.
#include <stdlib.h>
#include <string.h>
typedef unsigned int uint;
typedef unsigned long uint64;
#define PGSIZE 4096
static void *kalloc(void) { void *p = 0;
if (posix_memalign(&p, PGSIZE, PGSIZE)) return 0;
return p; }
static void kfree(char *p) { free(p); }
struct spinlock { int x; };
static void initlock(struct spinlock *l, char *n) { (void)n; l->x = 0; }
static void acquire(struct spinlock *l) { (void)l; }
static void release(struct spinlock *l) { (void)l; }
#define PGROUNDDOWN(a) (((uint64)(a)) & ~((uint64)PGSIZE - 1))
#else
#include "types.h"
#include "defs.h"
#include "param.h"
#include "mmu.h"
#include "spinlock.h"
#include "sbuf.h"
#endif
#include "kmem.h"
// Objects are 16-byte aligned. Eight would be enough for the free list, but
// sixteen is what the ABI wants for anything that might hold a long double or
// be memcpy'd with SSE, and the waste at these sizes is a rounding error.
#define KMEM_ALIGN 16
#define ALIGN_UP(n) (((n) + KMEM_ALIGN - 1) & ~(uint)(KMEM_ALIGN - 1))
struct slab {
struct slab *next;
struct slab *prev;
struct kmem_cache *cache;
void *freelist; // first free object, or 0
uint inuse;
uint pad;
};
struct kmem_cache {
const char *name;
uint objsize; // after rounding
uint per_slab;
struct slab *partial;
struct slab *full;
uint nslab; // pages held
uint inuse; // objects handed out
uint64 nalloc, nfree;
struct spinlock lock;
struct kmem_cache *next; // the global list, for /proc/slab
};
// Caches are themselves fixed, and deliberately: a cache is created once per
// subsystem at init and there are a handful of them, so making them dynamic
// would mean bootstrapping an allocator out of itself for no benefit.
#define NCACHE 32
static struct kmem_cache caches[NCACHE];
static int ncache;
static struct spinlock caches_lock;
static int kmem_ready;
// Rounded *down* to the alignment, so that the value this returns is one
// kmem_cache_create() will actually accept. It used to return the raw
// remainder, which create() then rounded up past its own limit -- so the
// largest legal size was reported as illegal, and the only caller who would
// ever ask is one trying to use every byte.
uint
kmem_max_obj(void)
{
return (PGSIZE - (uint)sizeof(struct slab)) & ~(uint)(KMEM_ALIGN - 1);
}
static void
kmem_boot(void)
{
if (kmem_ready)
return;
initlock(&caches_lock, "kmem");
kmem_ready = 1;
}
struct kmem_cache *
kmem_cache_create(const char *name, uint size)
{
struct kmem_cache *c;
kmem_boot();
size = ALIGN_UP(size < KMEM_ALIGN ? KMEM_ALIGN : size);
if (size > kmem_max_obj())
return 0;
acquire(&caches_lock);
if (ncache >= NCACHE) {
release(&caches_lock);
return 0;
}
c = &caches[ncache++];
release(&caches_lock);
c->name = name;
c->objsize = size;
c->per_slab = (PGSIZE - (uint)sizeof(struct slab)) / size;
c->partial = 0;
c->full = 0;
c->nslab = c->inuse = 0;
c->nalloc = c->nfree = 0;
initlock(&c->lock, "slab");
return c;
}
// --- slab list surgery -----------------------------------------------------
//
// Doubly linked because a slab moves between `partial` and `full` from either
// end: allocation takes the head of partial, but a free arrives at whichever
// slab happens to own the object.
static void
list_remove(struct slab **head, struct slab *s)
{
if (s->prev) s->prev->next = s->next;
else *head = s->next;
if (s->next) s->next->prev = s->prev;
s->next = s->prev = 0;
}
static void
list_push(struct slab **head, struct slab *s)
{
s->prev = 0;
s->next = *head;
if (*head) (*head)->prev = s;
*head = s;
}
static struct slab *
slab_new(struct kmem_cache *c)
{
char *page = kalloc();
struct slab *s;
uint i;
if (!page)
return 0;
s = (struct slab *)page;
s->next = s->prev = 0;
s->cache = c;
s->inuse = 0;
// Thread the free list through the objects themselves, back to front so the
// list comes out in address order. Nothing depends on that order; it just
// makes a dump of the freelist readable when something has gone wrong.
s->freelist = 0;
for (i = c->per_slab; i-- > 0; ) {
char *obj = page + sizeof(struct slab) + i * c->objsize;
*(void **)obj = s->freelist;
s->freelist = obj;
}
return s;
}
void *
kmem_cache_alloc(struct kmem_cache *c)
{
struct slab *s;
void *obj;
if (!c)
return 0;
acquire(&c->lock);
if (!c->partial) {
release(&c->lock);
// Outside the lock: kalloc() takes its own, and holding two locks in one
// order here and the other order anywhere else is how a kernel deadlocks.
s = slab_new(c);
if (!s)
return 0;
acquire(&c->lock);
list_push(&c->partial, s);
c->nslab++;
}
s = c->partial;
obj = s->freelist;
s->freelist = *(void **)obj;
s->inuse++;
c->inuse++;
c->nalloc++;
if (!s->freelist) {
list_remove(&c->partial, s);
list_push(&c->full, s);
}
release(&c->lock);
return obj;
}
void
kmem_cache_free(struct kmem_cache *c, void *obj)
{
struct slab *s;
int empty;
if (!obj)
return;
s = (struct slab *)PGROUNDDOWN((uint64)obj);
if (!c)
c = s->cache;
acquire(&c->lock);
if (!s->freelist) // was full; it has room again
{ list_remove(&c->full, s); list_push(&c->partial, s); }
*(void **)obj = s->freelist;
s->freelist = obj;
s->inuse--;
c->inuse--;
c->nfree++;
empty = (s->inuse == 0);
if (empty) {
list_remove(&c->partial, s);
c->nslab--;
}
release(&c->lock);
// Outside the lock, for the same reason slab_new() is.
if (empty)
kfree((char *)s);
}
// --- the general case ------------------------------------------------------
//
// Power-of-two classes from 16 to 2048. Above that a caller is within a factor
// of two of a page and should take the page.
static const uint gp_size[] = { 16, 32, 64, 128, 256, 512, 1024, 2048 };
#define NGP ((int)(sizeof(gp_size) / sizeof(gp_size[0])))
static struct kmem_cache *gp[NGP];
static const char *gp_name[NGP] = {
"kmalloc-16", "kmalloc-32", "kmalloc-64", "kmalloc-128",
"kmalloc-256", "kmalloc-512", "kmalloc-1024", "kmalloc-2048",
};
void *
kmalloc(uint n)
{
int i;
if (n == 0 || n > gp_size[NGP - 1])
return 0;
for (i = 0; i < NGP; i++)
if (n <= gp_size[i]) {
// Created on first use rather than at boot: a kernel that never calls
// kmalloc for a given size should not hold a page for it.
if (!gp[i] && !(gp[i] = kmem_cache_create(gp_name[i], gp_size[i])))
return 0;
return kmem_cache_alloc(gp[i]);
}
return 0;
}
void
kmfree(void *p)
{
// The cache comes from the object's own page, which is the whole reason
// kmalloc needs no size argument on the way back.
if (p)
kmem_cache_free(0, p);
}
// --- /proc/slab ------------------------------------------------------------
#ifndef KMEM_HOST
void
kmem_stats(struct sbuf *out)
{
int i;
sb_puts(out, "name size inuse slabs per alloc free\n");
for (i = 0; i < ncache; i++) {
struct kmem_cache *c = &caches[i];
sb_pad(out, c->name, 18);
sb_putu(out, c->objsize); sb_puts(out, " ");
sb_putu(out, c->inuse); sb_puts(out, " ");
sb_putu(out, c->nslab); sb_puts(out, " ");
sb_putu(out, c->per_slab); sb_puts(out, " ");
sb_putu(out, c->nalloc); sb_puts(out, " ");
sb_putu(out, c->nfree);
sb_putc(out, '\n');
}
}
#endif

23
kernel/dev/procfs.c
View file

@ -36,6 +36,7 @@
#include "fsstat.h"
#include "schfs.h"
#include "sbuf.h"
#include "kmem.h"
#include "dbg.h"
#define PROC_BUFSZ 2048
@ -344,6 +345,7 @@ gen_filesystems(struct sbuf *sb)
#define PROC_KMSG 17
#define PROC_RESOLV 18
#define PROC_BOOTMODE 19
#define PROC_SLAB 20
// Render global file `id` (a PROC_* constant) into `buf`. Returns its length.
//
@ -364,6 +366,9 @@ proc_gen_global(int id, char *buf, int cap)
case PROC_LOADAVG: gen_loadavg(&sb); break;
case PROC_MOUNTS: gen_mounts(&sb); break;
case PROC_FSSTAT: gen_fsstat(&sb); break;
// The slab allocator renders its own table, for the same reason SchFS and
// PCI do: it owns the data and must not depend on this file's helpers.
case PROC_SLAB: kmem_stats(&sb); break;
case PROC_FILESYSTEMS: gen_filesystems(&sb); break;
// SchFS renders straight into the buffer rather than through sbuf, because
// it lives in kernel/fs/schfs and must not depend on this file's helpers.
@ -404,15 +409,27 @@ procread(struct inode *ip, uint64 off, char *dst, int n)
if (ip->minor == PROC_KMSG)
return klog_read(off, dst, n);
char buf[PROC_BUFSZ];
int len = proc_gen_global(ip->minor, buf, PROC_BUFSZ);
// **Not a local.** This is PROC_BUFSZ bytes -- a quarter of the kernel
// stack -- on a path that is several frames deep already, and the comment
// above has warned about exactly that since it was written. It stayed a
// local because a page from kalloc() for 2 KB felt worse; kmalloc is the
// answer that was missing.
char *buf = kmalloc(PROC_BUFSZ);
int len;
if (len <= 0 || (int)off >= len)
if (!buf)
return -ENOMEM;
len = proc_gen_global(ip->minor, buf, PROC_BUFSZ);
if (len <= 0 || (int)off >= len) {
kmfree(buf);
return 0; // EOF
}
int avail = len - (int)off;
if (n > avail)
n = avail;
memmove(dst, buf + off, n);
kmfree(buf);
return n;
}

16
kernel/fs/pipe.c
View file

@ -7,6 +7,7 @@
#include "spinlock.h"
#include "sleeplock.h"
#include "file.h"
#include "kmem.h"
#include "posix.h"
#include "errno.h"
@ -21,6 +22,13 @@ struct pipe {
int writeopen; // write fd is still open
};
// A pipe is about 560 bytes and used to come from kalloc(), which is a whole
// 4096-byte page -- so every `|` in a shell command burned three and a half
// kilobytes it never touched. This is what kmem_cache exists for, and the
// cache is created on first use rather than at boot so a kernel that never
// opens a pipe holds no page for one.
static struct kmem_cache *pipe_cache;
int
pipealloc(struct file **f0, struct file **f1)
{
@ -30,7 +38,9 @@ pipealloc(struct file **f0, struct file **f1)
*f0 = *f1 = 0;
if((*f0 = filealloc()) == 0 || (*f1 = filealloc()) == 0)
goto bad;
if((p = (struct pipe*)kalloc()) == 0)
if(!pipe_cache && !(pipe_cache = kmem_cache_create("pipe", sizeof(*p))))
goto bad;
if((p = kmem_cache_alloc(pipe_cache)) == 0)
goto bad;
p->readopen = 1;
p->writeopen = 1;
@ -50,7 +60,7 @@ pipealloc(struct file **f0, struct file **f1)
//PAGEBREAK: 20
bad:
if(p)
kfree((char*)p);
kmem_cache_free(pipe_cache, p);
if(*f0)
fileclose(*f0);
if(*f1)
@ -71,7 +81,7 @@ pipeclose(struct pipe *p, int writable)
}
if(p->readopen == 0 && p->writeopen == 0){
release(&p->lock);
kfree((char*)p);
kmem_cache_free(pipe_cache, p);
} else
release(&p->lock);
}

12
kernel/fs/procfs.c
View file

@ -30,6 +30,7 @@
#include "sleeplock.h"
#include "fs.h"
#include "file.h"
#include "kmem.h"
#include "stat.h"
#include "uaccess.h"
#include "errno.h"
@ -99,6 +100,8 @@ static const struct global_file globals[] = {
{ "resolv.conf", 18 },
// "quiet" or "verbose" -- how this machine was booted. See cmd/schinitd.
{ "bootmode", 19 },
// What the slab allocator is holding. See include/kmem.h.
{ "slab", 20 },
{ "kmsg", 17 },
};
#define NGLOBALS ((int)(sizeof(globals) / sizeof(globals[0])))
@ -372,12 +375,15 @@ procfs_read(struct inode *ip, int user_dst, uint64 dst, uint64 off, uint n)
return r;
}
if ((buf = kalloc()) == 0)
// kmalloc, not kalloc: this buffer is PROCFS_GENBUF and kalloc's unit is a
// whole 4 KB page. The comment above about a page not being wasteful was
// written when a page was the only thing there was to ask for.
if ((buf = kmalloc(PROCFS_GENBUF)) == 0)
return -ENOMEM;
len = gen_file(ip->inum, buf, PROCFS_GENBUF);
if (off >= (uint64)len) {
kfree(buf);
kmfree(buf);
return 0;
}
if (off + n > (uint64)len)
@ -391,7 +397,7 @@ procfs_read(struct inode *ip, int user_dst, uint64 dst, uint64 off, uint n)
memmove((char *)dst, buf + off, n);
}
kfree(buf);
kmfree(buf);
return r;
}