sockets: a descriptor can cross between processes

sendmsg/recvmsg with SCM_RIGHTS. Item 1 of docs/plan/foundations.md and
mechanism 1 of docs/plan/hybrid.md, which are the same primitive seen from two
directions: Wayland's wl_shm_create_pool passes a file descriptor with no
integer-shaped alternative, and handing a driver process the device it is
allowed to touch is the same act. Both plans stopped here.

Linux's structures byte for byte -- msghdr, cmsghdr, the CMSG_ macros -- because
the programs that need this are not ours. libwayland and toybox both call
recvmsg with Linux's headers and would not notice a friendlier door.

The delivery rule is the whole design. A batch of descriptors is pinned to the
stream position of the first byte of the message that carried it, and that has
two halves: a reader arriving at that position is handed them, and a reader that
has not reached it is stopped SHORT of it. Leave out the second half and a
program asking for 4096 bytes swallows three messages and is handed one
message's descriptors, with nothing in the result saying which. socktest checks
exactly that: two one-byte messages, an eight-byte read, one byte back.

The kernel holds a reference to every descriptor in flight, so a sender may
close its own copy the moment sendmsg returns. Giving that reference back is the
part that leaks if it is not written down, and there are three ways a message
ends:

  - received with room            installed in the receiver's table
  - received without room, or by
    plain read(2)                 closed; MSG_CTRUNC for the first case
  - never received, socket closed closed by sockclose

The middle row looks like a choice and is not: holding kernel references for a
receiver that has proved it is not looking is a leak with extra steps.

No garbage collector. A socket descriptor sent over that same socket and never
received holds a reference to the socket that holds the reference to it, and
neither close reaches zero. Linux carries several hundred lines for this.
Nothing on either plan passes a socket over a socket -- Wayland passes memory,
the driver work passes devices -- so it is a bounded leak that is documented
rather than a silent one.

Fourteen checks in cmd/socktest, and every one moves data through the descriptor
rather than inspecting it. The descriptor passed is always a pipe end, because a
pipe announces its own lifetime: a read of a pipe whose writers are all gone
returns 0, which turns "was the kernel's reference released" -- otherwise
invisible from userspace -- into a value a test can compare.

Two bugs fell out, neither in the new code:

  - sockpair() never set the socket's domain, so every socketpair() socket
    claimed domain 0. Invisible while the data paths asked only whether a socket
    was AF_INET or AF_NETLINK, since zero answers "neither" correctly.
    SCM_RIGHTS asks "is this AF_UNIX", and zero is wrong for that one.

  - pipe_poll() reports POLLHUP but not POLLIN at end of file, so a non-blocking
    read of a pipe whose writers had all gone returned EAGAIN forever instead of
    0 -- a program that polls and reads spins waiting for an end that already
    happened. sockreadable() always had the equivalent condition; the pipe path
    did not. Found because the test helper for "is this descriptor closed" is a
    non-blocking read of a pipe.

projects/schcom/README.md said SCM_RIGHTS was done for a long time while nothing
in the kernel implemented it, and an entire GUI plan was drawn on that line.
It is true now, which is not the same as it having been right, and the README
says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2

cmd/socktest/main.c

@@ -402,6 +402,347 @@ test_nonblock(void)
     close(sv[1]);
 }
 
+
+
+// The user library has no string.h; these three are all this file needs.
+static void
+memset_(void *p, int c, int n)
+{
+    char *d = p;
+    while (n-- > 0) *d++ = (char)c;
+}
+
+static void
+memmove_(void *dst, const void *src, int n)
+{
+    char *d = dst;
+    const char *s = src;
+    while (n-- > 0) *d++ = *s++;
+}
+
+static int
+strneq_(const char *a, const char *b, int n)
+{
+    while (n-- > 0 && *a == *b && *a) { a++; b++; }
+    return n < 0 || *a == *b;
+}
+
+// ---------------------------------------------------------------------------
+// 8. SCM_RIGHTS: a descriptor crossing between processes.
+//
+// docs/plan/foundations.md item 1, docs/plan/hybrid.md mechanism 1. The thing
+// being proved is not that recvmsg returns a number -- it is that the number
+// names the *same open file* the other process had, which is why every check
+// below moves data through the descriptor rather than inspecting it.
+//
+// The descriptor passed is always a pipe end, for one reason: a pipe announces
+// its own lifetime. A read of a pipe whose every write end is closed returns 0,
+// and that turns "was the kernel's reference released" -- which userspace
+// cannot otherwise see -- into a value a test can compare.
+// ---------------------------------------------------------------------------
+#define T_F_SETFL 4
+
+// Send `fd` with one byte of data on it.
+static int
+sendfd(int sock, int fd, char tag)
+{
+    struct msghdr m;
+    struct iovec iv;
+    char cbuf[CMSG_SPACE(sizeof(int))];
+    struct cmsghdr *c;
+
+    memset_(&m, 0, sizeof(m));
+    memset_(cbuf, 0, sizeof(cbuf));
+
+    iv.iov_base = &tag;
+    iv.iov_len  = 1;
+    m.msg_iov    = &iv;
+    m.msg_iovlen = 1;
+    m.msg_control    = cbuf;
+    m.msg_controllen = sizeof(cbuf);
+
+    c = (struct cmsghdr *)cbuf;
+    c->cmsg_len   = CMSG_LEN(sizeof(int));
+    c->cmsg_level = SOL_SOCKET;
+    c->cmsg_type  = SCM_RIGHTS;
+    memmove_(CMSG_DATA(c), &fd, sizeof(int));
+
+    return sendmsg(sock, &m, 0);
+}
+
+// Receive into `fds`, up to `want` of them. Returns the byte count; *ngot is
+// how many descriptors arrived and *flags the kernel's msg_flags.
+static int
+recvfd(int sock, char *tag, int *fds, int want, int *ngot, int *flags)
+{
+    struct msghdr m;
+    struct iovec iv;
+    char cbuf[CMSG_SPACE(SCM_MAXFD * sizeof(int))];
+    struct cmsghdr *c;
+    int n;
+
+    memset_(&m, 0, sizeof(m));
+    memset_(cbuf, 0, sizeof(cbuf));
+
+    iv.iov_base = tag;
+    iv.iov_len  = 1;
+    m.msg_iov    = &iv;
+    m.msg_iovlen = 1;
+    m.msg_control    = cbuf;
+    // CMSG_LEN, not CMSG_SPACE: the trailing padding only matters if another
+    // header follows, and sizing by it would leave room for a descriptor the
+    // caller did not ask for -- which is exactly what makes a truncation test
+    // pass by accident.
+    m.msg_controllen = CMSG_LEN(want * sizeof(int));
+
+    *ngot = 0;
+    n = recvmsg(sock, &m, 0);
+    if (flags)
+        *flags = m.msg_flags;
+    if (n < 0)
+        return n;
+
+    c = CMSG_FIRSTHDR(&m);
+    if (c && c->cmsg_level == SOL_SOCKET && c->cmsg_type == SCM_RIGHTS) {
+        int k = (int)((c->cmsg_len - CMSG_LEN(0)) / sizeof(int));
+        for (*ngot = 0; *ngot < k && *ngot < want; (*ngot)++)
+            memmove_(&fds[*ngot], CMSG_DATA(c) + *ngot * sizeof(int), sizeof(int));
+    }
+    return n;
+}
+
+// Is every write end of this pipe gone? Non-blocking, so a descriptor that
+// leaked reports itself instead of hanging the test.
+static int
+pipe_at_eof(int rfd)
+{
+    char c;
+    fcntl(rfd, T_F_SETFL, T_O_NONBLOCK);
+    return read(rfd, &c, 1) == 0;
+}
+
+static void
+test_fdpass(void)
+{
+    int sv[2], p[2], pid, got = 0, fds[SCM_MAXFD], n;
+    char tag = 0, buf[32];
+
+    puts_("SCM_RIGHTS:\n");
+
+    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0 || pipe(p) < 0) {
+        check(0, "socketpair() and pipe() for the descriptor tests");
+        return;
+    }
+
+    pid = fork();
+    if (pid == 0) {
+        close(sv[0]);
+        write(p[1], "through a passed descriptor", 27);
+        sendfd(sv[1], p[0], 'F');
+        // Every process-held reference is gone from here on. If the kernel is
+        // not holding one of its own, the parent receives a dead descriptor.
+        close(p[0]);
+        close(p[1]);
+        close(sv[1]);
+        exit();
+    }
+
+    close(sv[1]);
+    close(p[0]);
+    close(p[1]);
+    wait();
+
+    n = recvfd(sv[0], &tag, fds, SCM_MAXFD, &got, 0);
+    check(n == 1 && tag == 'F', "recvmsg returns the byte the descriptor rode on");
+    check(got == 1, "and one descriptor with it");
+
+    if (got == 1) {
+        n = read(fds[0], buf, sizeof(buf));
+        check(n == 27 && strneq_(buf, "through a passed descriptor", 27),
+              "the descriptor is the sender's pipe, and still has its data");
+        close(fds[0]);
+    }
+    close(sv[0]);
+}
+
+// Two messages, each carrying a descriptor, read with a buffer big enough for
+// both. A read that ran past the second message's first byte would hand back
+// one message's descriptors with two messages' bytes, and nothing in the
+// result would say so -- which is why the kernel stops the read at the
+// boundary instead.
+static void
+test_fdpass_boundary(void)
+{
+    int sv[2], a[2], b[2], got = 0, fds[SCM_MAXFD], n, flags = 0;
+    char tag = 0;
+    struct msghdr m;
+    struct iovec iv;
+    char big[8];
+
+    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0 ||
+        pipe(a) < 0 || pipe(b) < 0) {
+        check(0, "sockets and pipes for the boundary test");
+        return;
+    }
+
+    sendfd(sv[1], a[0], '1');
+    sendfd(sv[1], b[0], '2');
+
+    // Ask for eight bytes. Two are available.
+    memset_(&m, 0, sizeof(m));
+    iv.iov_base = big;
+    iv.iov_len  = sizeof(big);
+    m.msg_iov    = &iv;
+    m.msg_iovlen = 1;
+    n = recvmsg(sv[0], &m, 0);
+    check(n == 1, "a read spanning two messages stops at the second's boundary");
+
+    n = recvfd(sv[0], &tag, fds, SCM_MAXFD, &got, &flags);
+    check(n == 1 && tag == '2' && got == 1,
+          "so the next read starts there, and gets that message's descriptor");
+    if (got == 1)
+        close(fds[0]);
+
+    close(sv[0]); close(sv[1]);
+    close(a[0]); close(a[1]); close(b[0]); close(b[1]);
+}
+
+// A plain read(2) of a message carrying descriptors. There is nowhere to put
+// them, so they are closed -- the alternative is the kernel holding references
+// for a receiver that has proved it is not looking.
+static void
+test_fdpass_dropped(void)
+{
+    int sv[2], p[2];
+    char c;
+
+    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0 || pipe(p) < 0) {
+        check(0, "sockets and a pipe for the dropped-descriptor test");
+        return;
+    }
+
+    sendfd(sv[1], p[1], 'D');
+    close(p[1]);                     // the kernel's reference is the last one
+
+    check(read(sv[0], &c, 1) == 1 && c == 'D',
+          "read(2) of a message carrying a descriptor returns its bytes");
+    check(pipe_at_eof(p[0]),
+          "and closes the descriptor rather than holding it for nobody");
+
+    close(p[0]); close(sv[0]); close(sv[1]);
+}
+
+// A message with descriptors that is never received at all. The socket is
+// closed with them still queued, and the kernel holds the only references --
+// so if close does not drain the queue they are unreachable and permanent.
+static void
+test_fdpass_orphaned(void)
+{
+    int sv[2], p[2];
+
+    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0 || pipe(p) < 0) {
+        check(0, "sockets and a pipe for the orphaned-descriptor test");
+        return;
+    }
+
+    sendfd(sv[1], p[1], 'O');
+    close(p[1]);
+    close(sv[0]);                    // never received
+    close(sv[1]);
+
+    check(pipe_at_eof(p[0]),
+          "closing a socket closes the descriptors still queued on it");
+    close(p[0]);
+}
+
+// More descriptors than the receiver left room for. The ones that do not fit
+// are closed, and MSG_CTRUNC is how the caller is told it happened.
+static void
+test_fdpass_truncated(void)
+{
+    int sv[2], a[2], b[2], got = 0, fds[SCM_MAXFD], flags = 0;
+    char tag = 0;
+    struct msghdr m;
+    struct iovec iv;
+    char cbuf[CMSG_SPACE(2 * sizeof(int))];
+    struct cmsghdr *c;
+    int two[2];
+
+    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0 ||
+        pipe(a) < 0 || pipe(b) < 0) {
+        check(0, "sockets and pipes for the truncation test");
+        return;
+    }
+
+    // One message, two descriptors.
+    memset_(&m, 0, sizeof(m));
+    memset_(cbuf, 0, sizeof(cbuf));
+    tag = 'T';
+    iv.iov_base = &tag;
+    iv.iov_len  = 1;
+    m.msg_iov    = &iv;
+    m.msg_iovlen = 1;
+    m.msg_control    = cbuf;
+    m.msg_controllen = sizeof(cbuf);
+    c = (struct cmsghdr *)cbuf;
+    c->cmsg_len   = CMSG_LEN(2 * sizeof(int));
+    c->cmsg_level = SOL_SOCKET;
+    c->cmsg_type  = SCM_RIGHTS;
+    two[0] = a[1];
+    two[1] = b[1];
+    memmove_(CMSG_DATA(c), two, sizeof(two));
+    check(sendmsg(sv[1], &m, 0) == 1, "sendmsg carries two descriptors at once");
+    close(a[1]);
+    close(b[1]);
+
+    // Receive with room for one.
+    tag = 0;
+    recvfd(sv[0], &tag, fds, 1, &got, &flags);
+    check(got == 1, "a receiver with room for one gets one");
+    check((flags & MSG_CTRUNC) != 0, "and MSG_CTRUNC says the rest were dropped");
+    check(pipe_at_eof(b[0]) || pipe_at_eof(a[0]),
+          "the descriptor that did not fit was closed, not leaked");
+
+    if (got == 1)
+        close(fds[0]);
+    close(a[0]); close(b[0]); close(sv[0]); close(sv[1]);
+}
+
+// Descriptors with no bytes to ride on. Delivery pins them to the position of
+// a first byte, so a message with no bytes has no position at which anyone
+// could ever be handed them; saying so beats accepting it and losing them.
+static void
+test_fdpass_nodata(void)
+{
+    int sv[2], p[2];
+    struct msghdr m;
+    char cbuf[CMSG_SPACE(sizeof(int))];
+    struct cmsghdr *c;
+
+    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) < 0 || pipe(p) < 0) {
+        check(0, "sockets and a pipe for the no-data test");
+        return;
+    }
+
+    memset_(&m, 0, sizeof(m));
+    memset_(cbuf, 0, sizeof(cbuf));
+    m.msg_control    = cbuf;
+    m.msg_controllen = sizeof(cbuf);
+    c = (struct cmsghdr *)cbuf;
+    c->cmsg_len   = CMSG_LEN(sizeof(int));
+    c->cmsg_level = SOL_SOCKET;
+    c->cmsg_type  = SCM_RIGHTS;
+    memmove_(CMSG_DATA(c), &p[1], sizeof(int));
+
+    check(sendmsg(sv[1], &m, 0) < 0,
+          "sendmsg refuses descriptors with no data to carry them");
+    close(p[1]);
+    check(pipe_at_eof(p[0]),
+          "and does not keep the reference it took while trying");
+
+    close(p[0]); close(sv[0]); close(sv[1]);
+}
+
 int
 main(void)
 {
@@ -414,6 +755,12 @@ main(void)
     test_big();
     test_sockopt();
     test_nonblock();
+    test_fdpass();
+    test_fdpass_boundary();
+    test_fdpass_dropped();
+    test_fdpass_orphaned();
+    test_fdpass_truncated();
+    test_fdpass_nodata();
 
     if (failures == 0)
         puts_("socktest: ALL CHECKS PASSED\n");