diff --git a/docs/architecture/README.md b/docs/architecture/README.md index d50bd62..7e46a09 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -85,6 +85,7 @@ Read in this order if you are new: | [boot.md](boot.md) | Firmware to the first userspace process. The shortest path through the whole system. | | [kernel.md](kernel.md) | Memory, processes, scheduling, traps, signals, SMP. | | [syscalls.md](syscalls.md) | The one table everything is generated from, and how a musl binary runs unmodified. | +| [scheduler.md](scheduler.md) | Fair share by virtual runtime, slices, and `nice`. | | [filesystem.md](filesystem.md) | The VFS, the buffer cache, mounts, procfs. | | [schfs.md](schfs.md) | The copy-on-write root filesystem. | | [fat.md](fat.md) | FAT12/16/32 with long names: USB sticks, SD cards, the ESP. | diff --git a/docs/architecture/kernel.md b/docs/architecture/kernel.md index 0266f20..1de517f 100644 --- a/docs/architecture/kernel.md +++ b/docs/architecture/kernel.md @@ -344,5 +344,7 @@ allocation over 8 MB mapped straight into a neighbouring live mapping. reported up as `-ENOMEM` and that is the whole policy. - **Nested signal delivery is not supported.** `in_signal` guards against re-entering a handler while one is running. -- **Scheduling is round-robin with no priorities**, no fairness accounting, and - no CPU affinity despite `sched_setaffinity` existing. +- **Scheduling is fair-share on one global run queue.** Priorities and + fairness accounting exist now ([scheduler.md](scheduler.md)); per-CPU run + queues, work stealing and CPU affinity do not, despite `sched_setaffinity` + existing. diff --git a/docs/architecture/scheduler.md b/docs/architecture/scheduler.md new file mode 100644 index 0000000..5b1aecb --- /dev/null +++ b/docs/architecture/scheduler.md @@ -0,0 +1,152 @@ + +# The scheduler + +Every runnable process carries one number, `vruntime`: the CPU time it has +consumed, divided by its weight. The scheduler always runs whichever runnable +process has the smallest one. That is the whole rule. + +Everything below is either a consequence of that rule or a guard against one +of the two ways it goes wrong. + +## What it replaced, and why + +It was xv6's round-robin: walk the process table from end to end, run +everything `RUNNABLE` in slot order, and take the CPU back on the next timer +tick. Three things were wrong with it on a machine that now runs a shell, a +network stack and a display server at once. + +**Slot order is not fairness.** A process that blocked after a microsecond and +one that burned its whole slice were treated as equals, because nothing +counted what anyone had used. A compile and the shell you were typing into got +alternate turns, and the shell felt it. + +**Every tick was a context switch.** `yield()` fired unconditionally on the +timer, so a process alone on a CPU paid a full switch — `switchuvm`, a `%cr3` +reload, a flushed TLB — a hundred times a second, only to be handed straight +back to itself. + +**Nothing could be said to matter more than anything else.** No nice, no +priority, nothing for a driver thread that has to answer a device promptly. + +## What falls out of the rule + +- **Fairness**, without anything enforcing it. Running advances your clock, so + the process whose clock is furthest behind is by definition the one that has + had least CPU, and it is the one that runs next. Equal weights converge on + equal shares. + +- **Interactivity**, without a heuristic. A process that spends its life + blocked on a keystroke barely advances its vruntime, so when the keystroke + arrives it is already at the front of the queue. Older schedulers guessed at + "is this interactive?", and that guess was the part that was always either + gameable or wrong. + +- **Priority**, as one multiplication. Weight scales how fast your clock runs; + a process of twice the weight is charged half the vruntime for the same real + time, so it is chosen twice as often. This is the only line in the scheduler + where `nice` does anything. + +## The two guards + +### Sleepers do not get to collect a debt + +A process blocked for ten seconds comes back ten seconds of vruntime behind +everything else. Left alone it would monopolise the CPU until it caught up — +the standard way a naive fair scheduler turns one sleeper into a system-wide +stall. + +`min_vruntime` is the floor: it tracks the front of the queue and only ever +moves forward. A waking process is pulled up to no worse than +`min_vruntime - SCHED_LATENCY_NS`: far enough forward to be scheduled +promptly, not so far as to be owed anything. + +### Forking does not reset your bill + +`fork()` copies the parent's vruntime to the child. Were the child to start +level with everyone, a process could clear its own CPU debt by forking, and +something that forks per request would collect a fresh full share every time +while a long-running process kept paying for everything it had ever used. + +`nice` is inherited for the ordinary reason: the point of `nice make -j8` is +that the compilers it forks are niced too. + +## Slices + +A slice is `SCHED_LATENCY_NS` (20 ms) divided between the runnable processes, +floored at `SCHED_MIN_GRAN_NS` (2 ms). With few processes each gets a long +slice and switches are rare; as the count grows the slice shrinks until the +floor, below which a switch would cost more than the work done inside it, and +the period is allowed to stretch instead. + +The floor is deliberately **shorter than a timer tick**. The tick is when +preemption is *considered* — `trap()` asks `sched_slice_expired()` and only +then yields — but time is *measured* with the TSC, against the `tsc_hz` the +LAPIC calibration already establishes at boot. Tick-granularity accounting on +a 100 Hz timer would make every slice a multiple of 10 ms whether it deserved +it or not. + +## Nice, and who may use it + +`nice` runs −20 to 19 with 1.25 between adjacent levels, so one step is about +a 10% change in share when two processes compete — the convention UNIX has +used long enough that programs and people both expect it. The weight table is +`1024 / 1.25^nice`, with nice 0 at 1024 so the common case multiplies by one. + +`setpriority(2)` is the only way to set it, and it is the one place in +`kernel/proc/proc.c` where the uids are actually consulted rather than merely +tracked. Two rules, with the two errnos POSIX gives them: + +- **`EPERM`** — you may only renice something you own. Without it any user can + push every root process to nice 19 and have the run of the machine. +- **`EACCES`** — *lowering* nice, which is asking for more CPU, is root's + alone. Without it nice is not a courtesy but a way for any process to + outrank the rest of the system by asking. + +Both are inert today, because everything runs as root. They are written now +because the alternative is growing a priority knob and a permission model in +separate commits, and the window between the two is exactly the bug. + +## Deliberately not a red-black tree + +Linux keeps the runnable set in a red-black tree and Zircon in a WAVL tree, so +"smallest vruntime" is O(log n) with the leftmost node cached. Here it is a +linear scan of the process table — the same scan the old round-robin did. + +`NPROC` is 128 slots; the scan is a handful of cache lines and runs with +`ptable.lock` already held, whereas a balanced tree is three hundred lines of +rotations that have to stay correct through every exit path in `proc.c`. The +scan is the right trade until something measures it as the problem, and the +measurement comes first. Replacing it changes `pick_next()` and nothing else, +which is the point of it being one function. + +## Still one lock, still one queue + +Every CPU picks out of the same table under `ptable.lock`. Per-CPU run queues +with work stealing are the other half of a real SMP scheduler, and they are a +separate change: they need a load balancer, and a load balancer that is wrong +is worse than a global lock that is merely contended. At the core counts this +actually runs on, the lock is not yet what hurts. + +## Looking at it + +| | | +| --- | --- | +| `/proc//schedstat` | vruntime in nanoseconds, nice, slices run. | +| `/proc//status` | `voluntary_ctxt_switches` and `nonvoluntary_ctxt_switches`. | + +The middle field of `schedstat` differs from Linux, which reports run-queue +wait time there. That time is not measured here, and inventing a number is +worse than putting something true in its place. + +The two switch counts are worth telling apart: a high **voluntary** count +means a process is waiting on something, and a high **involuntary** count +means it is being preempted. That is the difference between "this is slow" and +"this is competing". + +### A note on measuring it under QEMU + +Without KVM, the guest's timer interrupts coalesce under load, so the tick +clock — and therefore `utime`/`stime` in `/proc//stat`, and `sleep` — +runs slow against the TSC. `schedstat` is TSC-based and does not, which is why +the two disagree on a busy TCG guest and why shares are worth measuring as a +**delta between two samples** rather than as absolute totals. diff --git a/include/defs.h b/include/defs.h index 12663df..f7de5be 100644 --- a/include/defs.h +++ b/include/defs.h @@ -545,6 +545,11 @@ int wait(void); int waitpid(int, int*, int, uint64*, uint64*); void wakeup(void*); void yield(void); +int sched_slice_expired(void); +void sched_init_proc(struct proc*, int); +void sched_set_nice(struct proc*, int); +int proc_setnice(int, int); +int proc_getnice(int, int*); void sinit(void); void kstack_check_canary(const char *where); diff --git a/include/proc.h b/include/proc.h index 79edd50..d929672 100644 --- a/include/proc.h +++ b/include/proc.h @@ -169,6 +169,9 @@ struct procinfo { int ctty; // controlling terminal, 1-based; 0 = none uint64 utime, stime; // CPU time in ticks, this process uint64 cutime, cstime; // and its reaped children + int nice; // scheduling priority, -20..19 + uint64 vruntime; // weighted CPU nanoseconds, for `top` and debugging + uint64 nvcsw, nivcsw; // voluntary / involuntary context switches char name[16]; char exe[64]; // full path, for /proc//exe }; @@ -292,9 +295,23 @@ struct proc { // are tracked honestly so getuid()/setuid() stop lying to userland. uint uid, euid, suid; uint gid, egid, sgid; + + // --- Fair-share scheduling -------------------------------------------- + // The whole of the scheduler's per-process state. What each field means, + // and why the scheduler is shaped this way, is in kernel/proc/proc.c above + // scheduler(); this is deliberately only the declaration. + int nice; // -20..19, UNIX convention: lower means more CPU + uint32 weight; // nice_weight[nice + 20], cached + uint64 vruntime; // weighted CPU time consumed, in nanoseconds + uint64 ran_at; // TSC when this process last entered RUNNING + uint64 slice_ns; // how long the current slice is allowed to be + uint64 slice_used; // how much of it has been spent, in nanoseconds + uint64 nivcsw, nvcsw; // preempted / voluntary context switches + }; #define FD_CLOEXEC 1 + // wait(2) status encoding, as understood by musl's WIFEXITED/WEXITSTATUS/ // WIFSIGNALED/WTERMSIG macros: // normal exit -> (code & 0xff) << 8 diff --git a/kernel/arch/x86_64/trap.c b/kernel/arch/x86_64/trap.c index 1ade375..18c2b83 100644 --- a/kernel/arch/x86_64/trap.c +++ b/kernel/arch/x86_64/trap.c @@ -226,7 +226,15 @@ trap(struct trapframe *tf) if(myproc() && myproc()->killed && (tf->cs&3) == DPL_USER) exit(W_EXITCODE(0, myproc()->killsig ? myproc()->killsig : SIGKILL)); - if(myproc() && myproc()->state == RUNNING && tf->trapno == T_IRQ0+IRQ_TIMER) + // The timer tick is when preemption is *considered*, not when it happens. + // + // This used to yield unconditionally, so a process alone on a CPU paid a + // full context switch -- switchuvm, a %cr3 reload, a flushed TLB -- a + // hundred times a second only to be handed straight back to itself. + // sched_slice_expired() asks whether it has actually had its share; see the + // scheduler in kernel/proc/proc.c. + if(myproc() && myproc()->state == RUNNING && tf->trapno == T_IRQ0+IRQ_TIMER + && sched_slice_expired()) yield(); if(myproc() && myproc()->killed && (tf->cs&3) == DPL_USER) diff --git a/kernel/fs/procfs.c b/kernel/fs/procfs.c index 12d0c53..7997553 100644 --- a/kernel/fs/procfs.c +++ b/kernel/fs/procfs.c @@ -62,6 +62,7 @@ #define PF_CMDLINE 3 #define PF_STATM 4 #define PF_EXE 5 +#define PF_SCHEDSTAT 6 #define PF_MAX 6 #define INO_PID(ino) ((int)((ino) >> 8)) @@ -112,6 +113,14 @@ static const struct global_file pidfiles[] = { { "cmdline", PF_CMDLINE }, { "statm", PF_STATM }, { "exe", PF_EXE }, + // Three numbers about how the scheduler has treated this process, in the + // order Linux writes them: weighted CPU nanoseconds consumed, then -- where + // Linux reports run-queue wait -- this kernel's nice level, then the number + // of slices it has been given. The middle field differs on purpose and is + // documented in docs/architecture/scheduler.md; nothing here reads it, and + // inventing a wait time that is not measured would be worse than saying + // something true in its place. + { "schedstat", PF_SCHEDSTAT }, }; #define NPIDFILES ((int)(sizeof(pidfiles) / sizeof(pidfiles[0]))) @@ -219,6 +228,16 @@ gen_pid_status(struct sb *s, const struct procinfo *pi) sbputc(s, '\t'); sbputi(s, pi->uid); sbputs(s, "\nVmSize:\t"); sbputi(s, (int64)(pi->sz / 1024)); sbputs(s, " kB\nThreads:\t1\n"); + + // Where Linux keeps them, under the names Linux uses. The two are worth + // telling apart: a process with a high voluntary count is waiting on + // something, and one with a high involuntary count is being preempted -- + // which is the difference between "this is slow" and "this is competing". + sbputs(s, "voluntary_ctxt_switches:\t"); + sbputi(s, (int64)pi->nvcsw); + sbputs(s, "\nnonvoluntary_ctxt_switches:\t"); + sbputi(s, (int64)pi->nivcsw); + sbputc(s, '\n'); } static void @@ -258,6 +277,12 @@ gen_file(uint64 ino, char *buf, int cap) case PF_CMDLINE: gen_pid_cmdline(&s, &pi); break; case PF_STATM: gen_pid_statm(&s, &pi); break; case PF_EXE: sbputs(&s, pi.exe[0] ? pi.exe : pi.name); break; + case PF_SCHEDSTAT: + sbputi(&s, (int64)pi.vruntime); sbputc(&s, ' '); + sbputi(&s, (int64)pi.nice); sbputc(&s, ' '); + sbputi(&s, (int64)(pi.nvcsw + pi.nivcsw)); + sbputc(&s, '\n'); + break; default: return 0; } return s.n; diff --git a/kernel/proc/proc.c b/kernel/proc/proc.c index 63416b3..16476a6 100644 --- a/kernel/proc/proc.c +++ b/kernel/proc/proc.c @@ -312,6 +312,14 @@ found: memset(p->sig_restorer, 0, sizeof(p->sig_restorer)); p->pending_signals = 0; p->utime = p->stime = p->cutime = p->cstime = 0; + + // A recycled slot must not inherit a dead process's place in the queue -- + // in either direction. sched_init_proc starts it at min_vruntime, which is + // level with everything currently runnable: neither owed CPU nor in debt. + // fork() then copies the parent's vruntime over this, which is what stops a + // process from getting a fresh share of the CPU simply by forking. + sched_init_proc(p, 0); + p->signal_mask = 0; p->in_signal = 0; p->sigsusp_oldmask = 0; @@ -495,6 +503,18 @@ fork(void) safestrcpy(np->name, myproc()->name, sizeof(myproc()->name)); + // The child inherits the parent's place in the queue, priority and all. + // + // Both halves matter. Inheriting *nice* is what POSIX says and what makes + // `nice make -j8` mean anything at all -- the point of renicing a build is + // that the compilers it forks are reniced too. Inheriting *vruntime* is the + // part with teeth: were the child to start level with everyone, a process + // could reset its own CPU debt by forking, and something that forks per + // request would collect a fresh full share every time while a long-running + // process kept paying for everything it had ever used. + sched_set_nice(np, myproc()->nice); + np->vruntime = myproc()->vruntime; + pid = np->pid; __sync_synchronize(); @@ -795,37 +815,378 @@ proc_wedge(const char *name) return found; } +// --- The scheduler ------------------------------------------------------ +// +// It was round-robin: walk ptable from end to end, run everything RUNNABLE in +// slot order, and let the timer tick take the CPU back after 10 ms. That is +// xv6's scheduler, and three things are wrong with it on a machine that now +// runs a shell, a network stack and a display server at the same time. +// +// **Slot order is not fairness.** A process that blocks on a read after one +// microsecond and a process that burns its whole slice are treated as equals, +// because nothing counts what anyone actually used. The compile and the shell +// you are typing into get alternate turns; the shell feels it. +// +// **Every tick was a context switch.** `yield()` unconditionally on the timer +// means a process alone on a CPU still pays a full switch -- `switchuvm`, a +// `%cr3` reload, a flushed TLB -- a hundred times a second, to be handed +// straight back to itself. +// +// **There is no way to say one thing matters more than another.** No nice, no +// priority, nothing for a driver thread that must answer a device promptly. +// +// ## What replaces it +// +// Fair share by *virtual runtime*, which is the idea behind both Linux's CFS +// and Zircon's fair scheduler, and is much simpler than either implementation +// of it. Each process carries `vruntime`: the CPU time it has consumed, +// divided by its weight. The scheduler always runs the RUNNABLE process with +// the smallest vruntime. +// +// That single rule produces everything the old one lacked: +// +// - **Fairness falls out.** Running advances your clock; the process that +// has had least CPU is by definition the one whose clock is furthest +// behind, so it runs next. Equal weights converge on equal shares without +// anything having to enforce it. +// +// - **Interactivity falls out too, with no heuristic.** A process that +// spends its life blocked on a keystroke barely advances its vruntime, so +// when the keystroke arrives it is already at the front. No "is this +// interactive?" guess, which is the part of older schedulers that was +// always either gameable or wrong. +// +// - **Priority is one multiplication.** Weight scales how fast your clock +// runs. A high-weight process's vruntime advances slowly, so it is chosen +// more often, in exact proportion. +// +// ## Deliberately not a red-black tree +// +// CFS keeps the runnable set in a red-black tree and Zircon in a WAVL tree, +// so "smallest vruntime" is O(log n) and the leftmost node is cached. Here it +// is a linear scan of ptable, O(NPROC), the same scan the old scheduler did. +// +// That is not laziness. NPROC is 128 slots of a few words each; the scan is a +// handful of cache lines and runs with the lock already held, whereas a +// balanced tree is three hundred lines of rotations that must stay correct +// through every exit path in this file. The scan is the right trade until +// something measures it as the problem, and the rule this kernel keeps is +// that the measurement comes first. Swapping in a tree later changes +// pick_next() and nothing else, which is the point of it being one function. +// +// ## Still one lock, still one queue +// +// Every CPU picks out of the same table under `ptable.lock`. Per-CPU run +// queues with work stealing are the other half of a real SMP scheduler and +// they are a separate change -- they need a load balancer, and a load +// balancer that is wrong is worse than a global lock that is merely +// contended. With the number of cores this actually runs on, the lock is not +// yet what hurts. + +// Weight per nice level, indexed by nice + 20. +// +// The ratio between adjacent levels is 1.25, so one step of nice is about a +// 10% change in CPU share when two processes compete -- the convention UNIX +// has used for long enough that programs and people both expect it. The table +// is 1024 / 1.25^nice, rounded, with nice 0 at 1024 so the common case +// multiplies by one. +static const uint32 nice_weight[40] = { + 88818, 71054, 56843, 45475, 36380, 29104, 23283, 18626, + 14901, 11921, 9537, 7629, 6104, 4883, 3906, 3125, + 2500, 2000, 1600, 1280, 1024, 819, 655, 524, + 419, 336, 268, 215, 172, 137, 110, 88, + 70, 56, 45, 36, 29, 23, 18, 15, +}; +#define WEIGHT_NICE0 1024u + +// The period in which every runnable process should get a turn, and the floor +// on how short a turn may be. +// +// A slice is SCHED_LATENCY_NS divided between the runnable processes, so with +// few of them each gets a long one and switches are rare; as the count grows +// the slice shrinks until it hits SCHED_MIN_GRAN_NS, below which the switch +// costs more than the work done in it and the period is allowed to stretch +// instead. 20 ms and 2 ms on a kernel whose timer runs at 100 Hz: the minimum +// is deliberately shorter than a tick, because the tick is when the decision +// is *checked*, not the granularity at which time is *measured*. +#define SCHED_LATENCY_NS 20000000ull +#define SCHED_MIN_GRAN_NS 2000000ull + +// The floor of every runnable vruntime, which is what keeps a process that +// slept from being owed the world. +// +// Without it a process blocked for ten seconds comes back ten seconds of +// vruntime behind everything else and monopolises the CPU until it catches +// up -- the classic way a naive fair scheduler turns one sleeper into a +// system-wide stall. A waking process is pulled forward to no worse than +// min_vruntime - SCHED_LATENCY_NS: far enough ahead in the queue to be +// scheduled promptly, not so far as to be owed a debt. +// +// It only ever moves forward, and it is guarded by ptable.lock like +// everything else here. +static uint64 min_vruntime; + +// TSC cycles to nanoseconds. +// +// tsc_hz is measured against the PIT at boot (kernel/arch/x86_64/lapic.c), so +// this is real time rather than a cycle count that means something different +// on every machine. The multiply is done in 64 bits after dividing the clock +// down to MHz: cycles * 1000 / mhz overflows only after some centuries of +// uptime, where cycles * 1000000000 would overflow in seconds. +static uint64 +tsc_to_ns(uint64 cycles) +{ + uint64 mhz = tsc_hz / 1000000; + if (mhz == 0) + return 0; // before lapicinit; nothing to charge yet + return (cycles * 1000) / mhz; +} + +// Give a process its scheduling state. Called from allocproc, before it is +// ever RUNNABLE. +void +sched_init_proc(struct proc *p, int nice) +{ + if (nice < -20) nice = -20; + if (nice > 19) nice = 19; + p->nice = nice; + p->weight = nice_weight[nice + 20]; + p->vruntime = min_vruntime; // start level with everyone, owed nothing + p->ran_at = 0; + p->slice_ns = SCHED_MIN_GRAN_NS; + p->slice_used = 0; + p->nvcsw = p->nivcsw = 0; +} + +// setpriority(2)/getpriority(2), the only way nice is ever set. +// +// PRIO_PROCESS only. PRIO_PGRP and PRIO_USER are real and are not hard, but +// answering them requires deciding what "the priority of a user" returns when +// their processes disagree, and a wrong answer is worse than -EINVAL. +// +// Lowering nice -- asking for *more* CPU -- is root's alone, which is the one +// permission check in this file that is actually enforced. Without it nice is +// not a courtesy, it is a way for any process to outrank the rest of the +// system by asking. +int +proc_setnice(int pid, int nice) +{ + struct proc *p; + int r = -ESRCH; + + if (nice < -20) nice = -20; + if (nice > 19) nice = 19; + + acquire(&ptable.lock); + for (p = ptable.proc; p < &ptable.proc[NPROC]; p++) { + if (p->state == UNUSED || (pid != 0 && p->pid != pid)) + continue; + if (pid == 0 && p != myproc()) + continue; + + // Two separate rules, and POSIX gives them two separate errnos. + // + // EPERM: you may only renice something you own. Without it any user can + // push every root process to nice 19 and have the run of the machine. + // EACCES: *lowering* nice -- asking for more CPU -- is root's alone. + // + // Both are inert today, because everything runs as root and euid is 0 + // everywhere. They are written now because the alternative is a scheduler + // that grows a priority knob and a permission model separately, and the + // window between the two is exactly the bug. + if (myproc()->euid != 0 && myproc()->euid != p->uid) { + r = -EPERM; + break; + } + if (nice < p->nice && myproc()->euid != 0) { + r = -EACCES; + break; + } + sched_set_nice(p, nice); + r = 0; + break; + } + release(&ptable.lock); + return r; +} + +int +proc_getnice(int pid, int *out) +{ + struct proc *p; + int r = -ESRCH; + + acquire(&ptable.lock); + for (p = ptable.proc; p < &ptable.proc[NPROC]; p++) { + if (p->state == UNUSED || (pid != 0 && p->pid != pid)) + continue; + if (pid == 0 && p != myproc()) + continue; + *out = p->nice; + r = 0; + break; + } + release(&ptable.lock); + return r; +} + +// Change a process's priority. ptable.lock must be held. +// +// vruntime is left alone rather than rescaled: it is a record of time already +// spent, and rewriting history so that renicing hands out a retroactive +// refund (or bill) is how a nice() call becomes a way to jump the queue. +void +sched_set_nice(struct proc *p, int nice) +{ + if (nice < -20) nice = -20; + if (nice > 19) nice = 19; + p->nice = nice; + p->weight = nice_weight[nice + 20]; +} + +// Charge a process for the time it has just run, and advance its vruntime. +// ptable.lock must be held, and p must be the process leaving the CPU. +static void +sched_charge(struct proc *p) +{ + uint64 now = rdtsc(), ns; + + if (p->ran_at == 0 || now <= p->ran_at) { + p->ran_at = 0; + return; + } + ns = tsc_to_ns(now - p->ran_at); + p->ran_at = 0; + + p->slice_used += ns; + + // The weighting, and the only line in the scheduler where nice does + // anything: a process of twice the weight is charged half the vruntime for + // the same real time, so its clock falls behind and pick_next() returns to + // it sooner. + // + // The guard is for the one way weight can be zero: a ptable slot is BSS, + // and BSS is zeros until allocproc() runs sched_init_proc() on it. Nothing + // reaches here on such a slot today -- only a process that has run gets + // charged, and a process that has run went through allocproc -- but the + // failure if that ever stops being true is a divide-by-zero #GP inside the + // scheduler, which is a fault with no good place to report itself. + if (p->weight == 0) + p->weight = WEIGHT_NICE0; + p->vruntime += (ns * WEIGHT_NICE0) / p->weight; +} + +// How long a slice should be, given how many processes want one. +static uint64 +sched_slice(int nrunnable) +{ + uint64 slice; + + if (nrunnable <= 1) + return SCHED_LATENCY_NS; + slice = SCHED_LATENCY_NS / (uint64)nrunnable; + return slice < SCHED_MIN_GRAN_NS ? SCHED_MIN_GRAN_NS : slice; +} + +// Place a process that has just become runnable, and keep min_vruntime +// honest. ptable.lock must be held. +// +// Called from wakeup1() and from anywhere else that moves a process into +// RUNNABLE from a state where it was not accruing vruntime. +static void +sched_enqueue(struct proc *p) +{ + uint64 floor = min_vruntime > SCHED_LATENCY_NS + ? min_vruntime - SCHED_LATENCY_NS : 0; + + if (p->vruntime < floor) + p->vruntime = floor; + p->slice_used = 0; +} + +// The runnable process with the smallest vruntime, and a count of how many +// were in the running. ptable.lock must be held. +static struct proc * +pick_next(int *nrunnable) +{ + struct proc *p, *best = 0; + int n = 0; + + for (p = ptable.proc; p < &ptable.proc[NPROC]; p++) { + // A wedged process is passed over for ever. It is still RUNNABLE, still + // owns its descriptors and still appears in every listing -- which is + // the point: it looks exactly like a process that is merely busy. + if (p->state != RUNNABLE || p->wedged) + continue; + n++; + if (best == 0 || p->vruntime < best->vruntime) + best = p; + } + *nrunnable = n; + return best; +} + +// Whether the running process has used up its slice, so trap() knows whether +// the timer tick should become a yield(). No lock: it reads only the current +// process's own fields, and the only writer of them is the current process. +// +// This is the line that removed a hundred context switches a second from an +// idle machine. The old code yielded on *every* tick unconditionally. +int +sched_slice_expired(void) +{ + struct proc *p = myproc(); + uint64 now, ns; + + if (p == 0 || p->ran_at == 0) + return 0; + now = rdtsc(); + if (now <= p->ran_at) + return 0; + ns = tsc_to_ns(now - p->ran_at); + return (p->slice_used + ns) >= p->slice_ns; +} + void scheduler(void) { struct proc *p; - int skipped = 0; + int nrunnable; + for(;;){ sti(); + acquire(&ptable.lock); - for(p = ptable.proc; p < &ptable.proc[NPROC]; p++){ - // A wedged process is passed over for ever. It is still RUNNABLE, still - // owns its descriptors and still appears in every listing -- which is - // the point: it looks exactly like a process that is merely busy. - if(p->state != RUNNABLE || p->wedged) { - skipped++; - continue; - } - skipped = 0; - - proc_set(p); - switchuvm(p); - p->state = RUNNING; - swtch(&mycpu()->scheduler, p->context); - switchkvm(); - - proc_set(0); - } - release(&ptable.lock); - if (skipped > NPROC) { + p = pick_next(&nrunnable); + if (p == 0) { + // Nothing to run. Drop the lock before halting -- holding it through a + // hlt() would stop every other CPU from scheduling until an interrupt + // happened to arrive here. + release(&ptable.lock); hlt(); - skipped = 0; + continue; } + + // min_vruntime tracks the front of the queue and never moves backwards, + // so a process that sleeps through a busy period cannot return with a + // claim on all the CPU it missed. See the comment on the variable. + if (p->vruntime > min_vruntime) + min_vruntime = p->vruntime; + + p->slice_ns = sched_slice(nrunnable); + p->slice_used = 0; + p->ran_at = rdtsc(); + + proc_set(p); + switchuvm(p); + p->state = RUNNING; + swtch(&mycpu()->scheduler, p->context); + switchkvm(); + proc_set(0); + + // Back from swtch: p has given the CPU up, through sched(). It has + // already been charged there, where the reason for the switch was known. + release(&ptable.lock); } } @@ -843,6 +1204,14 @@ sched(void) if(readeflags()&FL_IF) panic("sched interruptible"); intena = mycpu()->intena; + + // Charge for the time just run before handing the CPU back. Here rather + // than in scheduler(), because this is the one point every path off a CPU + // goes through -- yield(), sleep() and exit() alike -- and doing it on the + // way out means the vruntime the scheduler then compares is already + // up to date rather than a tick stale. + sched_charge(myproc()); + swtch(&myproc()->context, mycpu()->scheduler); mycpu()->intena = intena; } @@ -852,6 +1221,7 @@ yield(void) { acquire(&ptable.lock); myproc()->state = RUNNABLE; + myproc()->nivcsw++; // involuntary: the slice ran out sched(); release(&ptable.lock); } @@ -1048,6 +1418,7 @@ sleep(void *chan, struct spinlock *lk) myproc()->chan = chan; myproc()->state = SLEEPING; + myproc()->nvcsw++; // voluntary: it asked to wait sched(); myproc()->chan = 0; @@ -1064,8 +1435,12 @@ wakeup1(void *chan) struct proc *p; for(p = ptable.proc; p < &ptable.proc[NPROC]; p++) - if(p->state == SLEEPING && p->chan == chan) + if(p->state == SLEEPING && p->chan == chan){ p->state = RUNNABLE; + // Pull it forward to the front of the timeline if it has been asleep + // long enough to have fallen off the back of it -- see sched_enqueue. + sched_enqueue(p); + } } void @@ -1130,8 +1505,10 @@ sigsend(int pid, int sig) p->killed = 1; if(p->killed && p->killsig == 0) p->killsig = sig; // so exit() can report the real cause - if(p->state == SLEEPING) + if(p->state == SLEEPING){ p->state = RUNNABLE; + sched_enqueue(p); // a signal is a wakeup too; same floor + } release(&ptable.lock); return 0; } @@ -1181,8 +1558,10 @@ sigsend_pgrp(int pgid, int sig) if(p->killsig == 0) p->killsig = sig; } - if(p->state == SLEEPING) + if(p->state == SLEEPING){ p->state = RUNNABLE; + sched_enqueue(p); // a signal is a wakeup too; same floor + } } release(&ptable.lock); @@ -1274,8 +1653,10 @@ kill(int pid) for(p = ptable.proc; p < &ptable.proc[NPROC]; p++){ if(p->pid == pid){ p->killed = 1; - if(p->state == SLEEPING) + if(p->state == SLEEPING){ p->state = RUNNABLE; + sched_enqueue(p); // a signal is a wakeup too; same floor + } release(&ptable.lock); return 0; } @@ -1354,6 +1735,10 @@ proc_snapshot(int idx, struct procinfo *out) out->stime = p->stime; out->cutime = p->cutime; out->cstime = p->cstime; + out->nice = p->nice; + out->vruntime = p->vruntime; + out->nvcsw = p->nvcsw; + out->nivcsw = p->nivcsw; safestrcpy(out->name, p->name, sizeof(out->name)); safestrcpy(out->exe, p->exe, sizeof(out->exe)); release(&ptable.lock); diff --git a/kernel/sys/sys_mem.c b/kernel/sys/sys_mem.c index 998ec95..00cdb80 100644 --- a/kernel/sys/sys_mem.c +++ b/kernel/sys/sys_mem.c @@ -842,6 +842,38 @@ addr_t sys_mremap(void) { return -ENOMEM; } // Resource limits: always report unlimited. addr_t sys_setrlimit(void) { return 0; } -// Process priority: always report 0. -addr_t sys_getpriority(void) { return 0; } -addr_t sys_setpriority(void) { return 0; } +// Process priority. These returned a flat 0 for as long as the scheduler had +// no notion of priority to report; it has one now (kernel/proc/proc.c), and a +// stub that says "everything is nice 0" is a lie the moment `renice` works. +// +// PRIO_PROCESS is 0, and `who` of 0 means the caller -- both from POSIX. +#define PRIO_PROCESS 0 + +addr_t +sys_getpriority(void) +{ + int which, who, nice = 0, r; + + argint(0, &which); argint(1, &who); + if (which != PRIO_PROCESS) + return -EINVAL; + if ((r = proc_getnice(who, &nice)) < 0) + return r; + + // getpriority(2) returns the nice value *offset by 20*, because -1 is a + // legal nice level and also how a syscall says "error". libc subtracts the + // 20 back off. Returning the raw nice value here makes `renice -n -1` look + // like a failure to every caller that checks. + return (addr_t)(20 - nice); +} + +addr_t +sys_setpriority(void) +{ + int which, who, nice; + + argint(0, &which); argint(1, &who); argint(2, &nice); + if (which != PRIO_PROCESS) + return -EINVAL; + return (addr_t)proc_setnice(who, nice); +} diff --git a/kernel/sys/sys_proc.c b/kernel/sys/sys_proc.c index 2cdcfb7..e245f5e 100644 --- a/kernel/sys/sys_proc.c +++ b/kernel/sys/sys_proc.c @@ -362,7 +362,11 @@ addr_t sys_setgroups(void) { return 0; } // --- Process/scheduling extensions --- -addr_t sys_sched_yield(void) { return 0; } +// A real yield. It returned 0 without giving anything up, which cost nothing +// while every timer tick was a yield anyway; with the fair scheduler a tick +// only preempts once the slice is spent, so a spin-and-yield loop (musl's +// locks, a polling driver in userland) would hold the CPU for all of it. +addr_t sys_sched_yield(void) { yield(); return 0; } // Every processor that started, since nothing here pins a process anywhere. // diff --git a/kernel/sys/table.inc b/kernel/sys/table.inc index 6ecb48b..ec28437 100644 --- a/kernel/sys/table.inc +++ b/kernel/sys/table.inc @@ -251,8 +251,8 @@ SYSCALL(142, 151, mlockall) SYSCALL(143, 152, munlockall) SYSCALL(144, 221, fadvise64) SYSCALL(145, 160, setrlimit) -SYSCALL(146, NO_LINUX, getpriority) -SYSCALL(147, NO_LINUX, setpriority) +SYSCALL(146, 140, getpriority) +SYSCALL(147, 141, setpriority) // --- at-syscalls --- SYSCALL(158, 258, mkdirat)