kernel: the clock ran 7x slow because the GPU never acknowledged its interrupt

fastfetch reported 3 seconds of uptime after 15 real ones, `time` claimed
3.12s for a 20-second command, and ping's one-second interval was closer to
six. One cause, not three: `ticks` was advancing at ~14.5 Hz instead of 100.

It was not the timer. virtio-gpu-pci shares IRQ 11 with the e1000 and
registered no interrupt handler at all. PCI INTx is level-triggered, so the
device held the line asserted, the kernel signalled EOI, and it fired again
immediately -- 72,000 interrupts a second, 83% of CPU0, starving the tick.
The e1000's handler read ICR as zero half a million times, correctly
answering "not mine", which is what made it look like a network-free problem.

Reading virtio's ISR byte clears it and drops the line. That is the fix.

irq_dispatch now measures each line's rate against the TSC -- not ticks,
which is the first casualty of a storm -- and names the attached handlers
once above 20000/s. It reproduces the original fault as a single line.

Found along the way, and fixed because they were real:

  - lapicinit loaded TICR with xv6's uncalibrated 10000000, whose own comment
    admitted it was never measured. calibrate_timers() now measures the LAPIC
    and the TSC against the 8254 PIT. (It came back at 999.9 MHz, so this was
    not the bug -- but the next machine is not obliged to be 1 GHz.)
  - microdelay() was an empty function body. Every microdelay(1000) in the
    xHCI reset path and the USB hub power-on sequence returned instantly.
  - HZ is a constant in param.h now, not the literal 100 written out at a
    dozen ticks-to-seconds conversions. USER_HZ is separate and deliberately
    does not follow it: musl answers sysconf(_SC_CLK_TCK) with 100 always, so
    times() and /proc/stat scale at the syscall boundary the way Linux does.
  - TIOCGWINSZ answered a hardcoded 24x80. gfx_cols()/gfx_rows() have known
    the real geometry all along; at 1024x768 the console is 128x48, so every
    program that lays out a screen was formatting for the wrong one.
  - TCFLSH fell through to the catch-all `return 0`, claiming to discard
    input without discarding any.

uptime now advances 5.01 seconds per 5 real seconds. schfs-check passes.

Full investigation, including the two wrong hypotheses and the measurement
flaw that protected them, in docs/history/kernel.md section 13.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQ7Qx8E1q1M1bbCq8QLHS2
This commit is contained in:
Nexoniarz 2026-09-12 21:34:36 +02:00
commit 9d4cdb44fc
17 changed files with 479 additions and 37 deletions

View file

@ -95,8 +95,8 @@ input_report(int dev, int type, int code, int value)
d->r++;
e = &d->ev[d->w % NEVENT];
e->time.tv_sec = ticks / 100;
e->time.tv_usec = (ticks % 100) * 10000;
e->time.tv_sec = ticks / HZ;
e->time.tv_usec = (ticks % HZ) * USEC_PER_TICK;
e->type = (uint16)type;
e->code = (uint16)code;
e->value = (int32)value;

View file

@ -6,6 +6,7 @@
#include "traps.h"
#include "spinlock.h"
#include "irq.h"
#include "x86.h"
struct irq_action {
irq_handler_t fn;
@ -134,6 +135,62 @@ irq_unregister(int irq, irq_handler_t fn, void *arg)
release(&irqlock);
}
// Notice a line that is firing far faster than any real device would, and say
// so once.
//
// A level-triggered PCI interrupt that nobody acknowledges re-fires the instant
// EOI is signalled, so the line runs at whatever rate the CPU can take traps --
// tens of thousands per second. That starves everything else on the machine,
// and it does so silently: the symptom appears wherever the missing CPU time
// hurts first, which is nowhere near the device at fault.
//
// It cost a long afternoon to find one of these. virtio-gpu shared IRQ 11 with
// the e1000 and registered no handler, so the line storm at ~72 kHz and took
// 83% of CPU0. What that looked like from userspace was a clock running seven
// times slow -- uptime, `time`, and ping's interval all wrong together, which
// reads as a timekeeping bug and sends you into the timer code. The timer was
// fine the whole time. This is the message that would have said so.
//
// The rate is measured against the TSC, deliberately: `ticks` is the first
// casualty of a storm, so a storm detector that counted ticks would go quiet
// exactly when it was needed.
#define STORM_SAMPLE 4096 // dispatches between rate checks
#define STORM_HZ 20000 // per second; no real device sustains this
static void
storm_check(int irq)
{
static uint64 sample_tsc[NIRQ];
static uint sample_n[NIRQ];
static uchar reported[NIRQ];
if(reported[irq] || !tsc_hz)
return;
if(++sample_n[irq] < STORM_SAMPLE)
return;
uint64 now = rdtsc(), prev = sample_tsc[irq];
sample_n[irq] = 0;
sample_tsc[irq] = now;
if(!prev)
return; // first sample: no interval to divide by
uint64 elapsed = now - prev;
if(elapsed == 0)
return;
uint64 rate = (uint64)STORM_SAMPLE * tsc_hz / elapsed;
if(rate < STORM_HZ)
return;
reported[irq] = 1;
cprintf("irq %d: %d interrupts/second -- a device on this line is not "
"acknowledging it.\n", irq, (int)rate);
for(int i = 0; i < line[irq].n; i++)
cprintf("irq %d: handler '%s' is attached\n", irq, line[irq].act[i].name);
if(line[irq].n == 0)
cprintf("irq %d: no handler is attached at all\n", irq);
}
void
irq_dispatch(int irq)
{
@ -145,6 +202,7 @@ irq_dispatch(int irq)
n = line[irq].n;
if(n == 0){
line[irq].unhandled++;
storm_check(irq);
return;
}
@ -154,6 +212,7 @@ irq_dispatch(int irq)
line[irq].act[i].count++;
line[irq].act[i].fn(line[irq].act[i].arg);
}
storm_check(irq);
}
// ---------------------------------------------------------------------------

View file

@ -130,11 +130,12 @@ gen_meminfo(struct sbuf *sb)
static void
gen_uptime(struct sbuf *sb)
{
uint t = uptime(); // timer ticks, 100 Hz
sb_putu(sb, t / 100);
uint t = uptime(); // timer ticks, HZ of them per second
uint frac = (t % HZ) * 100 / HZ; // hundredths, whatever HZ happens to be
sb_putu(sb, t / HZ);
sb_putc(sb, '.');
sb_putu(sb, (t % 100) / 10);
sb_putu(sb, (t % 100) % 10);
sb_putu(sb, frac / 10);
sb_putu(sb, frac % 10);
sb_puts(sb, " 0.00\n"); // idle time: not tracked
}
@ -214,7 +215,8 @@ gen_version(struct sbuf *sb)
static void
gen_stat(struct sbuf *sb)
{
uint t = uptime();
// USER_HZ, like times() -- everything that parses /proc/stat divides by 100.
uint t = (uint)((uint64)uptime() * USER_HZ / HZ);
sb_puts(sb, "cpu ");
sb_putu(sb, t); sb_puts(sb, " 0 0 0 0 0 0 0 0 0\n");
// Online, for the same reason as cpuinfo: top and friends count the cpuN