nexoniarz/SchOS
nexoniarz/SchOS
Watch
1
0
Fork
You've already forked SchOS
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

init: a hang is as detectable as a death, and PID 1 comes back

Browse source 
docs/plan/init.md milestones I2 and I3. The rule change.

**I2 -- the heartbeat.** The failure the two-init scheme could not see is that
an init usually does not die, it *hangs*: wait() never returns for a process
that is merely stuck, so a supervisor watching a wedged init reports
everything normal for ever. PID 1 now pings every two seconds and expects an
answer within ten; silence means stuck, and stuck is worth acting on.

Exercised rather than reasoned about, as the plan asks: `sysrq w` wedges
schinitd by making the scheduler pass over it for ever. It stays alive, keeps
its descriptors and appears in every listing -- which is the point.

Two things that only showed up by running it:

  - kill() sets a flag the target checks when it next runs, and the premise
    here is that it is not running. Waiting for a corpse that may never be
    produced turns the recovery path into a second hang. PID 1 replaces the
    supervisor *without waiting*; the old one's socket is closed, so the moment
    it runs again its first write fails and it exits.
  - the replaced supervisor must notice the hangup. poll() reports POLLHUP and
    never POLLIN again, so a loop that only looks for POLLIN sits there
    complaining that PID 1 has stopped pinging it.

**I3 -- proc_reincarnate().** exit() on initproc panicked. Now the kernel
tears down nothing, keeps pid 1, and re-execs /bin/schinit in place with a
generation counter. The pid stays 1, so every orphan the kernel reparented is
still correctly parented; the children survive; the open files survive, which
is why this is the first line of exit() and not the last; and the job table
survives in /proc/schinit, which is what it is there for.

The escalation ladder: generations 2-3 carry on, generation 4 ignores
/etc/schinit and starts a console shell only -- a manifest that kills init
must not kill it for ever -- and generation 5 stops, with PSOGM rendering
INIT_RESURRECTION_FAILED, a call chain and a QR code.

Three bugs found by running it, each a lesson about exec():

  - exec() returns 0 on success; it does not fail to return. The new image
    starts when the kernel returns to userspace through the rewritten
    trapframe. Returning from exit() is the success path.
  - exit() was reached because `killed` was set, and the trap path checks it
    again on the way out. Leaving it set exits the new image before its first
    instruction: five generations in a few microseconds, a resurrection loop
    that looks exactly like the crash it was meant to survive.
  - exec() keeps open files, which is what gives the new image a console --
    and also keeps the previous life's socket, holding the old supervisor's
    channel up so it never sees a hangup.

Verified: `sysrq i`, and the shell being typed into keeps its pid across the
death and rebirth of PID 1, with /proc/schinit byte-identical either side.

schfs-check passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
Nexoniarz 2026-09-13 14:27:22 +02:00
commit 0def474272
8 changed files with 456 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

171
cmd/schinit/main.c
View file

@ -52,6 +52,8 @@ struct child {
static struct child children[MAXJOBS];
static int sock = -1; // to schinitd
static int supervisor_pid;
static int last_pong; // uptime() when schinitd last answered
static int last_ping; // uptime() when we last asked
static unsigned int generation = 1; // bumped by proc_reincarnate() later
static int
@ -251,6 +253,60 @@ publish(void)
close(fd);
}
// Read the table back after a reincarnation.
//
// This is the one parse in PID 1, and it is worth being uneasy about. It is
// bounded: a fixed buffer, four integers and a label per line, no allocation,
// and it reads a format this same program wrote. It happens once, at startup,
// only when the kernel has just re-exec'd us. The alternative is forgetting
// every running job, which is worse than a twenty-line scan of known text.
static void
reclaim(void)
{
char buf[MAXJOBS * (SCHINIT_MAXLABEL + 40) + 64];
int fd = open("/proc/schinit", O_RDONLY);
int n, slot = 0;
if (fd < 0)
return;
n = read(fd, buf, (int)sizeof(buf) - 1);
close(fd);
if (n <= 0)
return;
buf[n] = 0;
char *p = buf;
while (*p && slot < MAXJOBS) {
char *line = p;
while (*p && *p != '\n') p++;
if (*p == '\n') *p++ = 0;
unsigned int cookie = 0;
int pid = 0, started = 0, digits = 0;
char *q = line;
if (*q < '0' || *q > '9') continue; // a header line
while (*q >= '0' && *q <= '9') { cookie = cookie * 10 + (unsigned)(*q++ - '0'); digits++; }
while (*q == ' ') q++;
while (*q >= '0' && *q <= '9') pid = pid * 10 + (*q++ - '0');
while (*q == ' ') q++;
while (*q >= '0' && *q <= '9') started = started * 10 + (*q++ - '0');
while (*q == ' ') q++;
if (!digits || pid <= 0 || !*q) continue;
children[slot].pid = pid;
children[slot].cookie = cookie;
children[slot].started = started;
scopy(children[slot].label, q, SCHINIT_MAXLABEL);
slot++;
}
if (slot) {
out("schinit: reclaimed ");
outn(slot);
out(" running job(s) from /proc/schinit\n");
}
}
// ---------------------------------------------------------------------------
// Spawning, on request
// ---------------------------------------------------------------------------
@ -377,6 +433,7 @@ start_supervisor(void)
close(sv[1]);
sock = sv[0];
supervisor_pid = pid;
last_pong = last_ping = uptime(); // it has not missed anything yet
return pid;
}
@ -416,6 +473,10 @@ serve(void)
break;
}
case SCHINIT_PONG:
last_pong = uptime();
break;
case SCHINIT_SPAWN: {
int pid = spawn(&m.spawn);
if (pid > 0) {
@ -445,11 +506,75 @@ serve(void)
// ---------------------------------------------------------------------------
int
main(void)
main(int argc, char *argv[])
{
// argv[1] is the generation, and it is only there when the kernel put it
// there -- proc_reincarnate() re-execs this program in place after PID 1
// has died. A normal boot has no argument and is generation 1.
if (argc > 1) {
unsigned int g = 0;
for (char *q = argv[1]; *q >= '0' && *q <= '9'; q++)
g = g * 10 + (unsigned int)(*q - '0');
if (g)
generation = g;
}
bootstrap();
out("schinit: SchOS process 1\n");
// A reincarnated PID 1 has the children but not the table.
//
// exec() replaced this program's memory and kept everything else -- the
// pid, the open files, and crucially the *children*, which are still
// parented here because nothing killed them. What did not survive is the
// array describing them, so it is read back from where it was published
// precisely so that it could be.
//
// Without this the next publish() overwrites the record with an empty
// table, the supervisor reads that, believes nothing is running and starts
// everything a second time. Same split brain as before, one level up.
if (generation > 1) {
// exec() keeps open files, which is what lets this program still have
// a console to complain to -- and also means the previous life's
// socket to the old supervisor is still open here. Leaving it open
// holds the old supervisor's channel up, so it never sees a hangup,
// never exits, and sits there reporting that PID 1 has stopped
// pinging it. Which is true, and no longer its business.
for (int fd = 3; fd < 32; fd++)
close(fd);
reclaim();
}
out("schinit: SchOS process 1");
if (generation > 1) {
out(", generation ");
outn((int)generation);
out(" (reincarnated)");
}
out("\n");
// Generation 4 is the safety net: a manifest that kills init must not kill
// it for ever, so this generation does not read /etc/schinit at all. One
// shell, on the console, and nothing else -- enough to log in and fix
// whatever has been taking the machine down.
if (generation >= 4) {
out("schinit: safe mode -- console shell only, /etc/schinit ignored\n");
if (fork() == 0) {
setsid();
ioctl(0, TIOCSCTTY, 0);
char *sh[] = { (char *)"-zsh", 0 };
execve((char *)"/bin/zsh", sh, 0);
char *fb[] = { (char *)"sh", 0 };
execve((char *)"/bin/sh", fb, 0);
exit();
}
publish();
for (;;) {
int st, pid;
while ((pid = waitpid(-1, &st, WNOHANG)) > 0)
;
sleep(20);
}
}
if (start_supervisor() < 0) {
// No supervisor. Rather than idle usefully-looking, fall back to a
@ -485,6 +610,48 @@ main(void)
sleep(10);
}
// The heartbeat. A supervisor that has stopped answering is stuck,
// and a stuck supervisor is indistinguishable from a working one by
// any means except asking it something -- which is the whole reason
// this exists. See docs/plan/init.md part 3.
if (sock >= 0) {
int now = uptime();
if (now - last_ping >= SCHINIT_PING_SECS * 100) {
struct schinit_hello p;
p.op = SCHINIT_PING;
p.proto = SCHINIT_PROTO;
p.generation = generation;
p.reserved = 0;
write(sock, &p, sizeof(p));
last_ping = now;
}
if (now - last_pong >= SCHINIT_DEADLINE_SECS * 100) {
out("schinit: supervisor stopped answering; replacing it\n");
// Ask it to die, then replace it *without waiting*.
//
// kill() sets a flag the target checks the next time it runs,
// and the whole premise here is that it is not running. A
// supervisor stuck in a loop will notice at its next trap; one
// blocked forever never will. Waiting for a corpse that may
// never be produced is how a recovery path becomes a second
// hang -- which is exactly what the first version of this did.
//
// Two supervisors briefly existing is safe, and safe in the
// way that matters: the old one's socket is closed here, so
// the moment it runs again its first write fails and it exits.
// It cannot spawn anything, because it cannot ask.
kill(supervisor_pid);
close(sock);
sock = -1;
int old = supervisor_pid;
if (start_supervisor() > 0)
out("schinit: new supervisor started\n");
publish(); // the table names the live supervisor
(void)old;
}
}
int status;
int pid;
while ((pid = waitpid(-1, &status, WNOHANG)) > 0) {

74
cmd/schinitd/main.c
View file

@ -590,18 +590,72 @@ main(void)
out("schinitd: ready\n");
// Everything from here is PID 1 telling us a child exited. There is no
// wait() in this program and there cannot be: the children are not ours.
// From here on this program does two things: answer PID 1's heartbeat, and
// act on the exits it reports. There is no wait() in this program and
// there cannot be -- the children are not ours.
//
// The read is polled rather than blocking, so that a PID 1 which has
// stopped talking is noticed. A blocking read would sit there forever
// looking exactly like a quiet machine, which is the failure mode this
// whole heartbeat exists to make visible.
int last_ping = uptime();
for (;;) {
struct schinit_reply r;
int n = read(sock, &r, sizeof(r));
if (n != sizeof(r)) {
// PID 1 closed the channel. It is the only process that cannot
// have gone away, so this means we are being replaced.
out("schinitd: channel closed\n");
struct pollfd pf;
pf.fd = sock;
pf.events = POLLIN;
pf.revents = 0;
int pr = poll(&pf, 1, 1000);
// The channel is gone. This happens to the *old* supervisor when PID 1
// has replaced it: PID 1 closes its end, so poll reports a hangup and
// never reports readable again. Without this the replaced supervisor
// sits in the loop for ever, complaining that PID 1 has stopped
// pinging -- which it has, because it is no longer talking to us.
if (pr > 0 && (pf.revents & (POLLHUP | POLLERR | POLLNVAL))) {
out("schinitd: channel closed; this supervisor has been replaced\n");
exit();
}
if (r.op == SCHINIT_EXITED)
note_exit(r.cookie, r.pid, r.status);
if (pr > 0 && (pf.revents & POLLIN)) {
union schinit_msg m;
int n = read(sock, &m, sizeof(m));
if (n <= 0) {
// PID 1 closed the channel. It is the one process that cannot
// have gone away, so this means we are being replaced.
out("schinitd: channel closed\n");
exit();
}
if (m.op == SCHINIT_PING) {
struct schinit_hello p;
p.op = SCHINIT_PONG;
p.proto = SCHINIT_PROTO;
p.generation = 0;
p.reserved = 0;
write(sock, &p, sizeof(p));
last_ping = uptime();
} else if (m.op == SCHINIT_EXITED) {
note_exit(m.reply.cookie, m.reply.pid, m.reply.status);
}
}
// Silence from PID 1.
//
// There is nothing this program can do about it -- PID 1 cannot be
// killed and cannot be replaced from here -- so it says so, loudly,
// once. docs/plan/init.md is explicit that this is the limit: a hang
// in PID 1 is only caught by noticing, and if both ends are wedged at
// once nothing catches it. There is no hardware watchdog.
//
// Saying so is still worth doing. The alternative is a machine that
// looks idle while its init is stuck, which is how this class of bug
// stays invisible for months.
static int complained;
if (!complained && uptime() - last_ping >= SCHINIT_DEADLINE_SECS * 100) {
out("schinitd: pid 1 has not pinged for "
"the deadline -- it may be wedged\n");
complained = 1;
}
}
}