init: a hang is as detectable as a death, and PID 1 comes back
docs/plan/init.md milestones I2 and I3. The rule change.
**I2 -- the heartbeat.** The failure the two-init scheme could not see is that
an init usually does not die, it *hangs*: wait() never returns for a process
that is merely stuck, so a supervisor watching a wedged init reports
everything normal for ever. PID 1 now pings every two seconds and expects an
answer within ten; silence means stuck, and stuck is worth acting on.
Exercised rather than reasoned about, as the plan asks: `sysrq w` wedges
schinitd by making the scheduler pass over it for ever. It stays alive, keeps
its descriptors and appears in every listing -- which is the point.
Two things that only showed up by running it:
- kill() sets a flag the target checks when it next runs, and the premise
here is that it is not running. Waiting for a corpse that may never be
produced turns the recovery path into a second hang. PID 1 replaces the
supervisor *without waiting*; the old one's socket is closed, so the moment
it runs again its first write fails and it exits.
- the replaced supervisor must notice the hangup. poll() reports POLLHUP and
never POLLIN again, so a loop that only looks for POLLIN sits there
complaining that PID 1 has stopped pinging it.
**I3 -- proc_reincarnate().** exit() on initproc panicked. Now the kernel
tears down nothing, keeps pid 1, and re-execs /bin/schinit in place with a
generation counter. The pid stays 1, so every orphan the kernel reparented is
still correctly parented; the children survive; the open files survive, which
is why this is the first line of exit() and not the last; and the job table
survives in /proc/schinit, which is what it is there for.
The escalation ladder: generations 2-3 carry on, generation 4 ignores
/etc/schinit and starts a console shell only -- a manifest that kills init
must not kill it for ever -- and generation 5 stops, with PSOGM rendering
INIT_RESURRECTION_FAILED, a call chain and a QR code.
Three bugs found by running it, each a lesson about exec():
- exec() returns 0 on success; it does not fail to return. The new image
starts when the kernel returns to userspace through the rewritten
trapframe. Returning from exit() is the success path.
- exit() was reached because `killed` was set, and the trap path checks it
again on the way out. Leaving it set exits the new image before its first
instruction: five generations in a few microseconds, a resurrection loop
that looks exactly like the crash it was meant to survive.
- exec() keeps open files, which is what gives the new image a console --
and also keeps the previous life's socket, holding the old supervisor's
channel up so it never sees a hangup.
Verified: `sysrq i`, and the shell being typed into keeps its pid across the
death and rebirth of PID 1, with /proc/schinit byte-identical either side.
schfs-check passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
parent
c80b7bf3b2
commit
0def474272
8 changed files with 456 additions and 21 deletions
|
|
@ -590,18 +590,72 @@ main(void)
|
|||
|
||||
out("schinitd: ready\n");
|
||||
|
||||
// Everything from here is PID 1 telling us a child exited. There is no
|
||||
// wait() in this program and there cannot be: the children are not ours.
|
||||
// From here on this program does two things: answer PID 1's heartbeat, and
|
||||
// act on the exits it reports. There is no wait() in this program and
|
||||
// there cannot be -- the children are not ours.
|
||||
//
|
||||
// The read is polled rather than blocking, so that a PID 1 which has
|
||||
// stopped talking is noticed. A blocking read would sit there forever
|
||||
// looking exactly like a quiet machine, which is the failure mode this
|
||||
// whole heartbeat exists to make visible.
|
||||
int last_ping = uptime();
|
||||
|
||||
for (;;) {
|
||||
struct schinit_reply r;
|
||||
int n = read(sock, &r, sizeof(r));
|
||||
if (n != sizeof(r)) {
|
||||
// PID 1 closed the channel. It is the only process that cannot
|
||||
// have gone away, so this means we are being replaced.
|
||||
out("schinitd: channel closed\n");
|
||||
struct pollfd pf;
|
||||
pf.fd = sock;
|
||||
pf.events = POLLIN;
|
||||
pf.revents = 0;
|
||||
|
||||
int pr = poll(&pf, 1, 1000);
|
||||
|
||||
// The channel is gone. This happens to the *old* supervisor when PID 1
|
||||
// has replaced it: PID 1 closes its end, so poll reports a hangup and
|
||||
// never reports readable again. Without this the replaced supervisor
|
||||
// sits in the loop for ever, complaining that PID 1 has stopped
|
||||
// pinging -- which it has, because it is no longer talking to us.
|
||||
if (pr > 0 && (pf.revents & (POLLHUP | POLLERR | POLLNVAL))) {
|
||||
out("schinitd: channel closed; this supervisor has been replaced\n");
|
||||
exit();
|
||||
}
|
||||
if (r.op == SCHINIT_EXITED)
|
||||
note_exit(r.cookie, r.pid, r.status);
|
||||
|
||||
if (pr > 0 && (pf.revents & POLLIN)) {
|
||||
union schinit_msg m;
|
||||
int n = read(sock, &m, sizeof(m));
|
||||
if (n <= 0) {
|
||||
// PID 1 closed the channel. It is the one process that cannot
|
||||
// have gone away, so this means we are being replaced.
|
||||
out("schinitd: channel closed\n");
|
||||
exit();
|
||||
}
|
||||
if (m.op == SCHINIT_PING) {
|
||||
struct schinit_hello p;
|
||||
p.op = SCHINIT_PONG;
|
||||
p.proto = SCHINIT_PROTO;
|
||||
p.generation = 0;
|
||||
p.reserved = 0;
|
||||
write(sock, &p, sizeof(p));
|
||||
last_ping = uptime();
|
||||
} else if (m.op == SCHINIT_EXITED) {
|
||||
note_exit(m.reply.cookie, m.reply.pid, m.reply.status);
|
||||
}
|
||||
}
|
||||
|
||||
// Silence from PID 1.
|
||||
//
|
||||
// There is nothing this program can do about it -- PID 1 cannot be
|
||||
// killed and cannot be replaced from here -- so it says so, loudly,
|
||||
// once. docs/plan/init.md is explicit that this is the limit: a hang
|
||||
// in PID 1 is only caught by noticing, and if both ends are wedged at
|
||||
// once nothing catches it. There is no hardware watchdog.
|
||||
//
|
||||
// Saying so is still worth doing. The alternative is a machine that
|
||||
// looks idle while its init is stuck, which is how this class of bug
|
||||
// stays invisible for months.
|
||||
static int complained;
|
||||
if (!complained && uptime() - last_ping >= SCHINIT_DEADLINE_SECS * 100) {
|
||||
out("schinitd: pid 1 has not pinged for "
|
||||
"the deadline -- it may be wedged\n");
|
||||
complained = 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue