nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions
SchOS/docs/plan/foundations.md
Nexoniarz a79dfac32b memfd: a shared memory object you reach by descriptor 
Gate 1, item 2, and the thing a Wayland buffer crosses on.

memfd_create(2) is the same segment machinery shmget() already had -- pages,
refcounts, no_free VMAs, the fork and exit accounting -- reached by
**descriptor** rather than by System V key. That difference is the entire
point and the reason it exists on Linux at all: two processes share memory by
passing the fd over a socket, so neither has to agree on a path or race to
invent a key. It is also why SCM_RIGHTS had to be item 1.

docs/plan/foundations.md proposed exactly this and it was right: what memfd
adds on top of shmget is a segment that starts empty and is sized by
ftruncate(), and a file that owns one reference to it. FD_MEMFD, and
read/write/lseek/fstat all learn about it -- a descriptor that refuses read()
is a surprise nobody needs.

Three things are refused rather than faked, each the honest version of
something this kernel cannot do:

  - MAP_PRIVATE on a memfd. Every caller is sharing on purpose; a private
    mapping of an anonymous object is an expensive MAP_ANONYMOUS.
  - Mapping past the end. Linux faults those pages in; there is no handler
    here that could.
  - Shrinking while mapped. Linux truncates the pages out from under the
    mapper; here a freed page would stay mapped to memory somebody else now
    owns.

Writable MAP_SHARED of a *real file* is still ENODEV. That needs a page cache,
and an anonymous object has no such contract to break -- which is precisely
why memfd is the narrow answer.

**cmd/memfdtest**, seventeen checks, written so a permissive implementation
cannot pass: an mmap that quietly returned private zeroed pages satisfies every
return code and fails the one assertion that matters, because the reader is a
different process. It ends with the combination the plan singled out -- a memfd
sent over a socketpair to a receiver that closes every inherited copy first,
maps only the descriptor that arrived, reads what the sender wrote and writes
back where the sender can see it.

Two things were found by writing that test, both of which had been refusing
everything that was not FD_INODE:

  - sys_fstat() never reached filestat(), so the memfd branch there was
    unreachable from the syscall that needs it. It also refused pipes and
    sockets, and returned -1 (which musl reads as EPERM) rather than an errno.
  - sys_lseek() the same, and SEEK_END needs a length that is not in an inode.

Verified: 40 consecutive runs leave MemFree at exactly the value it started
at -- 936656 kB before and after -- so nothing leaks across creation, dual
mapping, descriptor passing and exit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 09:47:31 +02:00

7.3 KiB
Raw Blame History

Foundations

The kernel work that wayland.md and hybrid.md both stop at. This is the critical path, and it is the reason those two are one plan rather than two.

The insight is small and it decides the schedule: the substrate for passing a buffer to a compositor and the substrate for handing a device to a driver process are the same substrate. Descriptor passing and shared memory, built once.

What is actually in the tree

Measured against the source, not against a README. Each of these was checked before anything below was planned around it:
sendmsg/recvmsg, and therefore SCM_RIGHTS Built. See item 1 below.
Writable MAP_SHARED of a file Refused — kernel/sys/sys_mem.c:131 returns -ENODEV.
memfd_create Does not exist.
AF_UNIX names In a table in kernel/net/unix.c, not in the filesystem.
setsockopt return 0; unconditionally — kernel/sys/sys_net.c:436.
exec ET_EXEC only. No PT_INTERP, no ET_DYN.

projects/schcom/README.md said SCM_RIGHTS was done while nothing in the kernel implemented it — the most load-bearing wrong line in the tree, with an entire GUI plan drawn on top of it. The line is now true, which is not the same as it having been right, and the README says so.

The MAP_SHARED refusal is worth defending rather than fixing carelessly. The code returns -ENODEV honestly: the contract of a writable shared mapping is that stores land in the file and other mappers see them, and there is no page cache that could make that true. Quietly handing back a private copy would be the same lie in a new place.

The five items

In dependency order. The first three are what Wayland cannot start without. Item 1 is built; items 2 and 3 are what a buffer still cannot cross on.

1. sendmsg / recvmsg with SCM_RIGHTS — done

Not optional and not substitutable. wl_shm_create_pool passes a file descriptor; there is no path in the Wayland protocol that passes an integer instead.

The handle_kind field in projects/schcom/proto.h was good foresight and it cannot help here — our protocol is the flexible one, and Wayland's is not.

Built as described: struct msghdr/cmsghdr in Linux's layout, ancillary data queued alongside the byte stream in kernel/net/unix.c, descriptor installation into the receiver's file table, and the three ways a message can end — received, received without room, never received — each releasing the kernel's reference. Fourteen checks in cmd/socktest, every one of them moving data through the descriptor rather than inspecting it. Documented in ../architecture/networking.md.

The delivery rule is the part worth knowing before building on it: descriptors are pinned to the stream position of the first byte of the message that carried them, and a read that has not reached that position is cut short so it stops before it. That is what keeps "which message did these belong to" answerable.

This was also item 1 of the hybrid work, where it is how a driver process is handed the device it is allowed to touch.

Two bugs fell out of building it, neither in the new code:

  • sockpair() never set the socket's domain, so every socketpair() socket claimed domain 0. Invisible for as long as the data paths asked only whether a socket was AF_INET or AF_NETLINK — zero answers "neither" correctly. SCM_RIGHTS asks "is this AF_UNIX", and zero is wrong for that one.
  • pipe_poll() reports POLLHUP but not POLLIN for a pipe at end of file, so a non-blocking read of a pipe whose writers have all gone returned EAGAIN forever instead of 0. A program that polls and reads would spin waiting for an end that had already happened. sockreadable() always had the equivalent condition; the pipe path did not.

2. A shareable memory object — done

memfd_create, plus mmap(MAP_SHARED) on it genuinely working between two processes. Described in ../architecture/kernel.md; asserted by cmd/memfdtest, seventeen checks including the one that matters — a descriptor sent over a socket, mapped by a receiver that has closed every inherited copy, with stores visible in both directions.

Narrower than a general page cache, and sufficient. An anonymous shared object needs no file behind it, which is exactly why memfd exists on Linux at all — and it means this can be built on the shared-memory machinery already in kernel/sys/sys_mem.c (shmget/shmat/shmdt work, and struct vma already carries shmid and a no_free flag for pages that are not ours to release).

The general answer — a page cache, and writable MAP_SHARED of a real file — stays owed. It is a larger project and nothing on this plan needs it.

3. AF_UNIX sockets in the filesystem namespace

Clients find a Wayland compositor by bind()ing $XDG_RUNTIME_DIR/wayland-0 as a real path, and libwayland flock()s the lock file beside it. A name table can express neither.

kernel/net/unix.c's own header comment already predicted this:

"What is lost is ls -l /tmp/schcom showing an s, and unlink() of a socket path failing with ENOENT. Both are worth fixing when the VFS next gets opened up; neither blocks anything."

They block something now. This is that moment, and it is a VFS change rather than a socket change: a socket inode type threaded through iget, dirlookup and stat, plus flock.

4. setsockopt that does not lie

At minimum SO_PASSCRED and SO_SNDBUF. Returning 0 for an option you ignored is worse than ENOPROTOOPT, because the caller adjusts its behaviour on the strength of the lie.

Small, and grouped here because it is in the same file as item 3 and will be found by the same testing.

5. Dynamic linking

PT_INTERP and ET_DYN in kernel/fs/exec.c.

Deferrable, and deferred. Static binaries are enough to reach a working desktop with software rendering. This becomes necessary when Mesa does — EGL means dlopen means a loader — and that is a separate decision made later.

Verification

Each item gets a test before the thing that needs it is built, because a substrate that is subtly wrong fails inside the consumer and looks like the consumer's bug.

  • Item 1: two processes, one passes an open descriptor, the other reads through it. Plus the leak case: a sender that exits with descriptors in flight.
  • Item 2: two processes map the same memfd, one writes, the other sees it.
  • Item 3: bind() a socket, then stat() and ls -l the path and see an s, then unlink() it.
  • Items 1–3 together: a third test that does all of it at once, because that combination is what Wayland actually performs and the three passing separately does not prove it.

These belong in bootcheck as a new switch, following the -Sock model.

What this does not deliver

  • No page cache, so writable MAP_SHARED of a real file stays refused.
  • No dlopen until item 5 is done, which means no Mesa and no third-party shared libraries.
  • SO_* coverage is the two options named, not the full set.
  • No credential passing (SCM_CREDENTIALS) — only descriptors. Nothing on this plan needs it, and a compositor that wants to know who connected will.