sched: fair share by virtual runtime, with nice
The scheduler was xv6's round-robin: run everything RUNNABLE in slot order and yield on every timer tick. Slot order is not fairness, a process alone on a CPU paid a full context switch a hundred times a second to be handed straight back to itself, and there was no way to say that one thing mattered more than another. Every process now carries vruntime, the CPU time it has used divided by its weight, and the scheduler runs the runnable process with the smallest. Fairness and interactivity both fall out of that one rule; nice is a weight, 1.25x per level as UNIX has always had it. A slice is 20 ms shared among the runnable processes, never below 2 ms, and the timer tick preempts only once it is spent. A process waking from a long sleep is brought up to min_vruntime less one period, so it is scheduled promptly without being owed everything it slept through; fork inherits vruntime, so forking does not reset a debt. getpriority/setpriority are real, with Linux's offset of 20 on the return; /proc/<pid>/schedstat and the ctxt_switches lines of status report what the scheduler did. docs/architecture/scheduler.md has the reasoning, including why the run queue is still a scan of ptable. This was written in an earlier session and left uncommitted. Finished here: a signal waking a sleeper goes through the same vruntime floor as wakeup() does, and sched_yield() yields -- it returned 0 doing nothing, harmless while every tick was a yield and a spin of a whole slice once ticks stopped being one. Checked under QEMU on 1 and 4 CPUs: a nice 10 loop beside a nice 0 one gets about a tenth of a shared CPU, and a command typed meanwhile answers in 26 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1giNvrHTrPMPw7V6x1kQ4
This commit is contained in:
Nexoniarz
parent
b70e36e0a2
commit
a2d987db5f
11 changed files with 666 additions and 35 deletions
|
|
@ -85,6 +85,7 @@ Read in this order if you are new:
|
|||
| [boot.md](boot.md) | Firmware to the first userspace process. The shortest path through the whole system. |
|
||||
| [kernel.md](kernel.md) | Memory, processes, scheduling, traps, signals, SMP. |
|
||||
| [syscalls.md](syscalls.md) | The one table everything is generated from, and how a musl binary runs unmodified. |
|
||||
| [scheduler.md](scheduler.md) | Fair share by virtual runtime, slices, and `nice`. |
|
||||
| [filesystem.md](filesystem.md) | The VFS, the buffer cache, mounts, procfs. |
|
||||
| [schfs.md](schfs.md) | The copy-on-write root filesystem. |
|
||||
| [fat.md](fat.md) | FAT12/16/32 with long names: USB sticks, SD cards, the ESP. |
|
||||
|
|
|
|||
|
|
@ -344,5 +344,7 @@ allocation over 8 MB mapped straight into a neighbouring live mapping.
|
|||
reported up as `-ENOMEM` and that is the whole policy.
|
||||
- **Nested signal delivery is not supported.** `in_signal` guards against
|
||||
re-entering a handler while one is running.
|
||||
- **Scheduling is round-robin with no priorities**, no fairness accounting, and
|
||||
no CPU affinity despite `sched_setaffinity` existing.
|
||||
- **Scheduling is fair-share on one global run queue.** Priorities and
|
||||
fairness accounting exist now ([scheduler.md](scheduler.md)); per-CPU run
|
||||
queues, work stealing and CPU affinity do not, despite `sched_setaffinity`
|
||||
existing.
|
||||
|
|
|
|||
152
docs/architecture/scheduler.md
Normal file
152
docs/architecture/scheduler.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
<!-- SPDX-License-Identifier: GPL-2.0-or-later -->
|
||||
# The scheduler
|
||||
|
||||
Every runnable process carries one number, `vruntime`: the CPU time it has
|
||||
consumed, divided by its weight. The scheduler always runs whichever runnable
|
||||
process has the smallest one. That is the whole rule.
|
||||
|
||||
Everything below is either a consequence of that rule or a guard against one
|
||||
of the two ways it goes wrong.
|
||||
|
||||
## What it replaced, and why
|
||||
|
||||
It was xv6's round-robin: walk the process table from end to end, run
|
||||
everything `RUNNABLE` in slot order, and take the CPU back on the next timer
|
||||
tick. Three things were wrong with it on a machine that now runs a shell, a
|
||||
network stack and a display server at once.
|
||||
|
||||
**Slot order is not fairness.** A process that blocked after a microsecond and
|
||||
one that burned its whole slice were treated as equals, because nothing
|
||||
counted what anyone had used. A compile and the shell you were typing into got
|
||||
alternate turns, and the shell felt it.
|
||||
|
||||
**Every tick was a context switch.** `yield()` fired unconditionally on the
|
||||
timer, so a process alone on a CPU paid a full switch — `switchuvm`, a `%cr3`
|
||||
reload, a flushed TLB — a hundred times a second, only to be handed straight
|
||||
back to itself.
|
||||
|
||||
**Nothing could be said to matter more than anything else.** No nice, no
|
||||
priority, nothing for a driver thread that has to answer a device promptly.
|
||||
|
||||
## What falls out of the rule
|
||||
|
||||
- **Fairness**, without anything enforcing it. Running advances your clock, so
|
||||
the process whose clock is furthest behind is by definition the one that has
|
||||
had least CPU, and it is the one that runs next. Equal weights converge on
|
||||
equal shares.
|
||||
|
||||
- **Interactivity**, without a heuristic. A process that spends its life
|
||||
blocked on a keystroke barely advances its vruntime, so when the keystroke
|
||||
arrives it is already at the front of the queue. Older schedulers guessed at
|
||||
"is this interactive?", and that guess was the part that was always either
|
||||
gameable or wrong.
|
||||
|
||||
- **Priority**, as one multiplication. Weight scales how fast your clock runs;
|
||||
a process of twice the weight is charged half the vruntime for the same real
|
||||
time, so it is chosen twice as often. This is the only line in the scheduler
|
||||
where `nice` does anything.
|
||||
|
||||
## The two guards
|
||||
|
||||
### Sleepers do not get to collect a debt
|
||||
|
||||
A process blocked for ten seconds comes back ten seconds of vruntime behind
|
||||
everything else. Left alone it would monopolise the CPU until it caught up —
|
||||
the standard way a naive fair scheduler turns one sleeper into a system-wide
|
||||
stall.
|
||||
|
||||
`min_vruntime` is the floor: it tracks the front of the queue and only ever
|
||||
moves forward. A waking process is pulled up to no worse than
|
||||
`min_vruntime - SCHED_LATENCY_NS`: far enough forward to be scheduled
|
||||
promptly, not so far as to be owed anything.
|
||||
|
||||
### Forking does not reset your bill
|
||||
|
||||
`fork()` copies the parent's vruntime to the child. Were the child to start
|
||||
level with everyone, a process could clear its own CPU debt by forking, and
|
||||
something that forks per request would collect a fresh full share every time
|
||||
while a long-running process kept paying for everything it had ever used.
|
||||
|
||||
`nice` is inherited for the ordinary reason: the point of `nice make -j8` is
|
||||
that the compilers it forks are niced too.
|
||||
|
||||
## Slices
|
||||
|
||||
A slice is `SCHED_LATENCY_NS` (20 ms) divided between the runnable processes,
|
||||
floored at `SCHED_MIN_GRAN_NS` (2 ms). With few processes each gets a long
|
||||
slice and switches are rare; as the count grows the slice shrinks until the
|
||||
floor, below which a switch would cost more than the work done inside it, and
|
||||
the period is allowed to stretch instead.
|
||||
|
||||
The floor is deliberately **shorter than a timer tick**. The tick is when
|
||||
preemption is *considered* — `trap()` asks `sched_slice_expired()` and only
|
||||
then yields — but time is *measured* with the TSC, against the `tsc_hz` the
|
||||
LAPIC calibration already establishes at boot. Tick-granularity accounting on
|
||||
a 100 Hz timer would make every slice a multiple of 10 ms whether it deserved
|
||||
it or not.
|
||||
|
||||
## Nice, and who may use it
|
||||
|
||||
`nice` runs −20 to 19 with 1.25 between adjacent levels, so one step is about
|
||||
a 10% change in share when two processes compete — the convention UNIX has
|
||||
used long enough that programs and people both expect it. The weight table is
|
||||
`1024 / 1.25^nice`, with nice 0 at 1024 so the common case multiplies by one.
|
||||
|
||||
`setpriority(2)` is the only way to set it, and it is the one place in
|
||||
`kernel/proc/proc.c` where the uids are actually consulted rather than merely
|
||||
tracked. Two rules, with the two errnos POSIX gives them:
|
||||
|
||||
- **`EPERM`** — you may only renice something you own. Without it any user can
|
||||
push every root process to nice 19 and have the run of the machine.
|
||||
- **`EACCES`** — *lowering* nice, which is asking for more CPU, is root's
|
||||
alone. Without it nice is not a courtesy but a way for any process to
|
||||
outrank the rest of the system by asking.
|
||||
|
||||
Both are inert today, because everything runs as root. They are written now
|
||||
because the alternative is growing a priority knob and a permission model in
|
||||
separate commits, and the window between the two is exactly the bug.
|
||||
|
||||
## Deliberately not a red-black tree
|
||||
|
||||
Linux keeps the runnable set in a red-black tree and Zircon in a WAVL tree, so
|
||||
"smallest vruntime" is O(log n) with the leftmost node cached. Here it is a
|
||||
linear scan of the process table — the same scan the old round-robin did.
|
||||
|
||||
`NPROC` is 128 slots; the scan is a handful of cache lines and runs with
|
||||
`ptable.lock` already held, whereas a balanced tree is three hundred lines of
|
||||
rotations that have to stay correct through every exit path in `proc.c`. The
|
||||
scan is the right trade until something measures it as the problem, and the
|
||||
measurement comes first. Replacing it changes `pick_next()` and nothing else,
|
||||
which is the point of it being one function.
|
||||
|
||||
## Still one lock, still one queue
|
||||
|
||||
Every CPU picks out of the same table under `ptable.lock`. Per-CPU run queues
|
||||
with work stealing are the other half of a real SMP scheduler, and they are a
|
||||
separate change: they need a load balancer, and a load balancer that is wrong
|
||||
is worse than a global lock that is merely contended. At the core counts this
|
||||
actually runs on, the lock is not yet what hurts.
|
||||
|
||||
## Looking at it
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| `/proc/<pid>/schedstat` | vruntime in nanoseconds, nice, slices run. |
|
||||
| `/proc/<pid>/status` | `voluntary_ctxt_switches` and `nonvoluntary_ctxt_switches`. |
|
||||
|
||||
The middle field of `schedstat` differs from Linux, which reports run-queue
|
||||
wait time there. That time is not measured here, and inventing a number is
|
||||
worse than putting something true in its place.
|
||||
|
||||
The two switch counts are worth telling apart: a high **voluntary** count
|
||||
means a process is waiting on something, and a high **involuntary** count
|
||||
means it is being preempted. That is the difference between "this is slow" and
|
||||
"this is competing".
|
||||
|
||||
### A note on measuring it under QEMU
|
||||
|
||||
Without KVM, the guest's timer interrupts coalesce under load, so the tick
|
||||
clock — and therefore `utime`/`stime` in `/proc/<pid>/stat`, and `sleep` —
|
||||
runs slow against the TSC. `schedstat` is TSC-based and does not, which is why
|
||||
the two disagree on a busy TCG guest and why shares are worth measuring as a
|
||||
**delta between two samples** rather than as absolute totals.
|
||||
|
|
@ -545,6 +545,11 @@ int wait(void);
|
|||
int waitpid(int, int*, int, uint64*, uint64*);
|
||||
void wakeup(void*);
|
||||
void yield(void);
|
||||
int sched_slice_expired(void);
|
||||
void sched_init_proc(struct proc*, int);
|
||||
void sched_set_nice(struct proc*, int);
|
||||
int proc_setnice(int, int);
|
||||
int proc_getnice(int, int*);
|
||||
void sinit(void);
|
||||
void kstack_check_canary(const char *where);
|
||||
|
||||
|
|
|
|||
|
|
@ -169,6 +169,9 @@ struct procinfo {
|
|||
int ctty; // controlling terminal, 1-based; 0 = none
|
||||
uint64 utime, stime; // CPU time in ticks, this process
|
||||
uint64 cutime, cstime; // and its reaped children
|
||||
int nice; // scheduling priority, -20..19
|
||||
uint64 vruntime; // weighted CPU nanoseconds, for `top` and debugging
|
||||
uint64 nvcsw, nivcsw; // voluntary / involuntary context switches
|
||||
char name[16];
|
||||
char exe[64]; // full path, for /proc/<pid>/exe
|
||||
};
|
||||
|
|
@ -292,9 +295,23 @@ struct proc {
|
|||
// are tracked honestly so getuid()/setuid() stop lying to userland.
|
||||
uint uid, euid, suid;
|
||||
uint gid, egid, sgid;
|
||||
|
||||
// --- Fair-share scheduling --------------------------------------------
|
||||
// The whole of the scheduler's per-process state. What each field means,
|
||||
// and why the scheduler is shaped this way, is in kernel/proc/proc.c above
|
||||
// scheduler(); this is deliberately only the declaration.
|
||||
int nice; // -20..19, UNIX convention: lower means more CPU
|
||||
uint32 weight; // nice_weight[nice + 20], cached
|
||||
uint64 vruntime; // weighted CPU time consumed, in nanoseconds
|
||||
uint64 ran_at; // TSC when this process last entered RUNNING
|
||||
uint64 slice_ns; // how long the current slice is allowed to be
|
||||
uint64 slice_used; // how much of it has been spent, in nanoseconds
|
||||
uint64 nivcsw, nvcsw; // preempted / voluntary context switches
|
||||
|
||||
};
|
||||
#define FD_CLOEXEC 1
|
||||
|
||||
|
||||
// wait(2) status encoding, as understood by musl's WIFEXITED/WEXITSTATUS/
|
||||
// WIFSIGNALED/WTERMSIG macros:
|
||||
// normal exit -> (code & 0xff) << 8
|
||||
|
|
|
|||
|
|
@ -226,7 +226,15 @@ trap(struct trapframe *tf)
|
|||
if(myproc() && myproc()->killed && (tf->cs&3) == DPL_USER)
|
||||
exit(W_EXITCODE(0, myproc()->killsig ? myproc()->killsig : SIGKILL));
|
||||
|
||||
if(myproc() && myproc()->state == RUNNING && tf->trapno == T_IRQ0+IRQ_TIMER)
|
||||
// The timer tick is when preemption is *considered*, not when it happens.
|
||||
//
|
||||
// This used to yield unconditionally, so a process alone on a CPU paid a
|
||||
// full context switch -- switchuvm, a %cr3 reload, a flushed TLB -- a
|
||||
// hundred times a second only to be handed straight back to itself.
|
||||
// sched_slice_expired() asks whether it has actually had its share; see the
|
||||
// scheduler in kernel/proc/proc.c.
|
||||
if(myproc() && myproc()->state == RUNNING && tf->trapno == T_IRQ0+IRQ_TIMER
|
||||
&& sched_slice_expired())
|
||||
yield();
|
||||
|
||||
if(myproc() && myproc()->killed && (tf->cs&3) == DPL_USER)
|
||||
|
|
|
|||
|
|
@ -62,6 +62,7 @@
|
|||
#define PF_CMDLINE 3
|
||||
#define PF_STATM 4
|
||||
#define PF_EXE 5
|
||||
#define PF_SCHEDSTAT 6
|
||||
#define PF_MAX 6
|
||||
|
||||
#define INO_PID(ino) ((int)((ino) >> 8))
|
||||
|
|
@ -112,6 +113,14 @@ static const struct global_file pidfiles[] = {
|
|||
{ "cmdline", PF_CMDLINE },
|
||||
{ "statm", PF_STATM },
|
||||
{ "exe", PF_EXE },
|
||||
// Three numbers about how the scheduler has treated this process, in the
|
||||
// order Linux writes them: weighted CPU nanoseconds consumed, then -- where
|
||||
// Linux reports run-queue wait -- this kernel's nice level, then the number
|
||||
// of slices it has been given. The middle field differs on purpose and is
|
||||
// documented in docs/architecture/scheduler.md; nothing here reads it, and
|
||||
// inventing a wait time that is not measured would be worse than saying
|
||||
// something true in its place.
|
||||
{ "schedstat", PF_SCHEDSTAT },
|
||||
};
|
||||
#define NPIDFILES ((int)(sizeof(pidfiles) / sizeof(pidfiles[0])))
|
||||
|
||||
|
|
@ -219,6 +228,16 @@ gen_pid_status(struct sb *s, const struct procinfo *pi)
|
|||
sbputc(s, '\t'); sbputi(s, pi->uid);
|
||||
sbputs(s, "\nVmSize:\t"); sbputi(s, (int64)(pi->sz / 1024));
|
||||
sbputs(s, " kB\nThreads:\t1\n");
|
||||
|
||||
// Where Linux keeps them, under the names Linux uses. The two are worth
|
||||
// telling apart: a process with a high voluntary count is waiting on
|
||||
// something, and one with a high involuntary count is being preempted --
|
||||
// which is the difference between "this is slow" and "this is competing".
|
||||
sbputs(s, "voluntary_ctxt_switches:\t");
|
||||
sbputi(s, (int64)pi->nvcsw);
|
||||
sbputs(s, "\nnonvoluntary_ctxt_switches:\t");
|
||||
sbputi(s, (int64)pi->nivcsw);
|
||||
sbputc(s, '\n');
|
||||
}
|
||||
|
||||
static void
|
||||
|
|
@ -258,6 +277,12 @@ gen_file(uint64 ino, char *buf, int cap)
|
|||
case PF_CMDLINE: gen_pid_cmdline(&s, &pi); break;
|
||||
case PF_STATM: gen_pid_statm(&s, &pi); break;
|
||||
case PF_EXE: sbputs(&s, pi.exe[0] ? pi.exe : pi.name); break;
|
||||
case PF_SCHEDSTAT:
|
||||
sbputi(&s, (int64)pi.vruntime); sbputc(&s, ' ');
|
||||
sbputi(&s, (int64)pi.nice); sbputc(&s, ' ');
|
||||
sbputi(&s, (int64)(pi.nvcsw + pi.nivcsw));
|
||||
sbputc(&s, '\n');
|
||||
break;
|
||||
default: return 0;
|
||||
}
|
||||
return s.n;
|
||||
|
|
|
|||
|
|
@ -312,6 +312,14 @@ found:
|
|||
memset(p->sig_restorer, 0, sizeof(p->sig_restorer));
|
||||
p->pending_signals = 0;
|
||||
p->utime = p->stime = p->cutime = p->cstime = 0;
|
||||
|
||||
// A recycled slot must not inherit a dead process's place in the queue --
|
||||
// in either direction. sched_init_proc starts it at min_vruntime, which is
|
||||
// level with everything currently runnable: neither owed CPU nor in debt.
|
||||
// fork() then copies the parent's vruntime over this, which is what stops a
|
||||
// process from getting a fresh share of the CPU simply by forking.
|
||||
sched_init_proc(p, 0);
|
||||
|
||||
p->signal_mask = 0;
|
||||
p->in_signal = 0;
|
||||
p->sigsusp_oldmask = 0;
|
||||
|
|
@ -495,6 +503,18 @@ fork(void)
|
|||
|
||||
safestrcpy(np->name, myproc()->name, sizeof(myproc()->name));
|
||||
|
||||
// The child inherits the parent's place in the queue, priority and all.
|
||||
//
|
||||
// Both halves matter. Inheriting *nice* is what POSIX says and what makes
|
||||
// `nice make -j8` mean anything at all -- the point of renicing a build is
|
||||
// that the compilers it forks are reniced too. Inheriting *vruntime* is the
|
||||
// part with teeth: were the child to start level with everyone, a process
|
||||
// could reset its own CPU debt by forking, and something that forks per
|
||||
// request would collect a fresh full share every time while a long-running
|
||||
// process kept paying for everything it had ever used.
|
||||
sched_set_nice(np, myproc()->nice);
|
||||
np->vruntime = myproc()->vruntime;
|
||||
|
||||
pid = np->pid;
|
||||
|
||||
__sync_synchronize();
|
||||
|
|
@ -795,37 +815,378 @@ proc_wedge(const char *name)
|
|||
return found;
|
||||
}
|
||||
|
||||
// --- The scheduler ------------------------------------------------------
|
||||
//
|
||||
// It was round-robin: walk ptable from end to end, run everything RUNNABLE in
|
||||
// slot order, and let the timer tick take the CPU back after 10 ms. That is
|
||||
// xv6's scheduler, and three things are wrong with it on a machine that now
|
||||
// runs a shell, a network stack and a display server at the same time.
|
||||
//
|
||||
// **Slot order is not fairness.** A process that blocks on a read after one
|
||||
// microsecond and a process that burns its whole slice are treated as equals,
|
||||
// because nothing counts what anyone actually used. The compile and the shell
|
||||
// you are typing into get alternate turns; the shell feels it.
|
||||
//
|
||||
// **Every tick was a context switch.** `yield()` unconditionally on the timer
|
||||
// means a process alone on a CPU still pays a full switch -- `switchuvm`, a
|
||||
// `%cr3` reload, a flushed TLB -- a hundred times a second, to be handed
|
||||
// straight back to itself.
|
||||
//
|
||||
// **There is no way to say one thing matters more than another.** No nice, no
|
||||
// priority, nothing for a driver thread that must answer a device promptly.
|
||||
//
|
||||
// ## What replaces it
|
||||
//
|
||||
// Fair share by *virtual runtime*, which is the idea behind both Linux's CFS
|
||||
// and Zircon's fair scheduler, and is much simpler than either implementation
|
||||
// of it. Each process carries `vruntime`: the CPU time it has consumed,
|
||||
// divided by its weight. The scheduler always runs the RUNNABLE process with
|
||||
// the smallest vruntime.
|
||||
//
|
||||
// That single rule produces everything the old one lacked:
|
||||
//
|
||||
// - **Fairness falls out.** Running advances your clock; the process that
|
||||
// has had least CPU is by definition the one whose clock is furthest
|
||||
// behind, so it runs next. Equal weights converge on equal shares without
|
||||
// anything having to enforce it.
|
||||
//
|
||||
// - **Interactivity falls out too, with no heuristic.** A process that
|
||||
// spends its life blocked on a keystroke barely advances its vruntime, so
|
||||
// when the keystroke arrives it is already at the front. No "is this
|
||||
// interactive?" guess, which is the part of older schedulers that was
|
||||
// always either gameable or wrong.
|
||||
//
|
||||
// - **Priority is one multiplication.** Weight scales how fast your clock
|
||||
// runs. A high-weight process's vruntime advances slowly, so it is chosen
|
||||
// more often, in exact proportion.
|
||||
//
|
||||
// ## Deliberately not a red-black tree
|
||||
//
|
||||
// CFS keeps the runnable set in a red-black tree and Zircon in a WAVL tree,
|
||||
// so "smallest vruntime" is O(log n) and the leftmost node is cached. Here it
|
||||
// is a linear scan of ptable, O(NPROC), the same scan the old scheduler did.
|
||||
//
|
||||
// That is not laziness. NPROC is 128 slots of a few words each; the scan is a
|
||||
// handful of cache lines and runs with the lock already held, whereas a
|
||||
// balanced tree is three hundred lines of rotations that must stay correct
|
||||
// through every exit path in this file. The scan is the right trade until
|
||||
// something measures it as the problem, and the rule this kernel keeps is
|
||||
// that the measurement comes first. Swapping in a tree later changes
|
||||
// pick_next() and nothing else, which is the point of it being one function.
|
||||
//
|
||||
// ## Still one lock, still one queue
|
||||
//
|
||||
// Every CPU picks out of the same table under `ptable.lock`. Per-CPU run
|
||||
// queues with work stealing are the other half of a real SMP scheduler and
|
||||
// they are a separate change -- they need a load balancer, and a load
|
||||
// balancer that is wrong is worse than a global lock that is merely
|
||||
// contended. With the number of cores this actually runs on, the lock is not
|
||||
// yet what hurts.
|
||||
|
||||
// Weight per nice level, indexed by nice + 20.
|
||||
//
|
||||
// The ratio between adjacent levels is 1.25, so one step of nice is about a
|
||||
// 10% change in CPU share when two processes compete -- the convention UNIX
|
||||
// has used for long enough that programs and people both expect it. The table
|
||||
// is 1024 / 1.25^nice, rounded, with nice 0 at 1024 so the common case
|
||||
// multiplies by one.
|
||||
static const uint32 nice_weight[40] = {
|
||||
88818, 71054, 56843, 45475, 36380, 29104, 23283, 18626,
|
||||
14901, 11921, 9537, 7629, 6104, 4883, 3906, 3125,
|
||||
2500, 2000, 1600, 1280, 1024, 819, 655, 524,
|
||||
419, 336, 268, 215, 172, 137, 110, 88,
|
||||
70, 56, 45, 36, 29, 23, 18, 15,
|
||||
};
|
||||
#define WEIGHT_NICE0 1024u
|
||||
|
||||
// The period in which every runnable process should get a turn, and the floor
|
||||
// on how short a turn may be.
|
||||
//
|
||||
// A slice is SCHED_LATENCY_NS divided between the runnable processes, so with
|
||||
// few of them each gets a long one and switches are rare; as the count grows
|
||||
// the slice shrinks until it hits SCHED_MIN_GRAN_NS, below which the switch
|
||||
// costs more than the work done in it and the period is allowed to stretch
|
||||
// instead. 20 ms and 2 ms on a kernel whose timer runs at 100 Hz: the minimum
|
||||
// is deliberately shorter than a tick, because the tick is when the decision
|
||||
// is *checked*, not the granularity at which time is *measured*.
|
||||
#define SCHED_LATENCY_NS 20000000ull
|
||||
#define SCHED_MIN_GRAN_NS 2000000ull
|
||||
|
||||
// The floor of every runnable vruntime, which is what keeps a process that
|
||||
// slept from being owed the world.
|
||||
//
|
||||
// Without it a process blocked for ten seconds comes back ten seconds of
|
||||
// vruntime behind everything else and monopolises the CPU until it catches
|
||||
// up -- the classic way a naive fair scheduler turns one sleeper into a
|
||||
// system-wide stall. A waking process is pulled forward to no worse than
|
||||
// min_vruntime - SCHED_LATENCY_NS: far enough ahead in the queue to be
|
||||
// scheduled promptly, not so far as to be owed a debt.
|
||||
//
|
||||
// It only ever moves forward, and it is guarded by ptable.lock like
|
||||
// everything else here.
|
||||
static uint64 min_vruntime;
|
||||
|
||||
// TSC cycles to nanoseconds.
|
||||
//
|
||||
// tsc_hz is measured against the PIT at boot (kernel/arch/x86_64/lapic.c), so
|
||||
// this is real time rather than a cycle count that means something different
|
||||
// on every machine. The multiply is done in 64 bits after dividing the clock
|
||||
// down to MHz: cycles * 1000 / mhz overflows only after some centuries of
|
||||
// uptime, where cycles * 1000000000 would overflow in seconds.
|
||||
static uint64
|
||||
tsc_to_ns(uint64 cycles)
|
||||
{
|
||||
uint64 mhz = tsc_hz / 1000000;
|
||||
if (mhz == 0)
|
||||
return 0; // before lapicinit; nothing to charge yet
|
||||
return (cycles * 1000) / mhz;
|
||||
}
|
||||
|
||||
// Give a process its scheduling state. Called from allocproc, before it is
|
||||
// ever RUNNABLE.
|
||||
void
|
||||
sched_init_proc(struct proc *p, int nice)
|
||||
{
|
||||
if (nice < -20) nice = -20;
|
||||
if (nice > 19) nice = 19;
|
||||
p->nice = nice;
|
||||
p->weight = nice_weight[nice + 20];
|
||||
p->vruntime = min_vruntime; // start level with everyone, owed nothing
|
||||
p->ran_at = 0;
|
||||
p->slice_ns = SCHED_MIN_GRAN_NS;
|
||||
p->slice_used = 0;
|
||||
p->nvcsw = p->nivcsw = 0;
|
||||
}
|
||||
|
||||
// setpriority(2)/getpriority(2), the only way nice is ever set.
|
||||
//
|
||||
// PRIO_PROCESS only. PRIO_PGRP and PRIO_USER are real and are not hard, but
|
||||
// answering them requires deciding what "the priority of a user" returns when
|
||||
// their processes disagree, and a wrong answer is worse than -EINVAL.
|
||||
//
|
||||
// Lowering nice -- asking for *more* CPU -- is root's alone, which is the one
|
||||
// permission check in this file that is actually enforced. Without it nice is
|
||||
// not a courtesy, it is a way for any process to outrank the rest of the
|
||||
// system by asking.
|
||||
int
|
||||
proc_setnice(int pid, int nice)
|
||||
{
|
||||
struct proc *p;
|
||||
int r = -ESRCH;
|
||||
|
||||
if (nice < -20) nice = -20;
|
||||
if (nice > 19) nice = 19;
|
||||
|
||||
acquire(&ptable.lock);
|
||||
for (p = ptable.proc; p < &ptable.proc[NPROC]; p++) {
|
||||
if (p->state == UNUSED || (pid != 0 && p->pid != pid))
|
||||
continue;
|
||||
if (pid == 0 && p != myproc())
|
||||
continue;
|
||||
|
||||
// Two separate rules, and POSIX gives them two separate errnos.
|
||||
//
|
||||
// EPERM: you may only renice something you own. Without it any user can
|
||||
// push every root process to nice 19 and have the run of the machine.
|
||||
// EACCES: *lowering* nice -- asking for more CPU -- is root's alone.
|
||||
//
|
||||
// Both are inert today, because everything runs as root and euid is 0
|
||||
// everywhere. They are written now because the alternative is a scheduler
|
||||
// that grows a priority knob and a permission model separately, and the
|
||||
// window between the two is exactly the bug.
|
||||
if (myproc()->euid != 0 && myproc()->euid != p->uid) {
|
||||
r = -EPERM;
|
||||
break;
|
||||
}
|
||||
if (nice < p->nice && myproc()->euid != 0) {
|
||||
r = -EACCES;
|
||||
break;
|
||||
}
|
||||
sched_set_nice(p, nice);
|
||||
r = 0;
|
||||
break;
|
||||
}
|
||||
release(&ptable.lock);
|
||||
return r;
|
||||
}
|
||||
|
||||
int
|
||||
proc_getnice(int pid, int *out)
|
||||
{
|
||||
struct proc *p;
|
||||
int r = -ESRCH;
|
||||
|
||||
acquire(&ptable.lock);
|
||||
for (p = ptable.proc; p < &ptable.proc[NPROC]; p++) {
|
||||
if (p->state == UNUSED || (pid != 0 && p->pid != pid))
|
||||
continue;
|
||||
if (pid == 0 && p != myproc())
|
||||
continue;
|
||||
*out = p->nice;
|
||||
r = 0;
|
||||
break;
|
||||
}
|
||||
release(&ptable.lock);
|
||||
return r;
|
||||
}
|
||||
|
||||
// Change a process's priority. ptable.lock must be held.
|
||||
//
|
||||
// vruntime is left alone rather than rescaled: it is a record of time already
|
||||
// spent, and rewriting history so that renicing hands out a retroactive
|
||||
// refund (or bill) is how a nice() call becomes a way to jump the queue.
|
||||
void
|
||||
sched_set_nice(struct proc *p, int nice)
|
||||
{
|
||||
if (nice < -20) nice = -20;
|
||||
if (nice > 19) nice = 19;
|
||||
p->nice = nice;
|
||||
p->weight = nice_weight[nice + 20];
|
||||
}
|
||||
|
||||
// Charge a process for the time it has just run, and advance its vruntime.
|
||||
// ptable.lock must be held, and p must be the process leaving the CPU.
|
||||
static void
|
||||
sched_charge(struct proc *p)
|
||||
{
|
||||
uint64 now = rdtsc(), ns;
|
||||
|
||||
if (p->ran_at == 0 || now <= p->ran_at) {
|
||||
p->ran_at = 0;
|
||||
return;
|
||||
}
|
||||
ns = tsc_to_ns(now - p->ran_at);
|
||||
p->ran_at = 0;
|
||||
|
||||
p->slice_used += ns;
|
||||
|
||||
// The weighting, and the only line in the scheduler where nice does
|
||||
// anything: a process of twice the weight is charged half the vruntime for
|
||||
// the same real time, so its clock falls behind and pick_next() returns to
|
||||
// it sooner.
|
||||
//
|
||||
// The guard is for the one way weight can be zero: a ptable slot is BSS,
|
||||
// and BSS is zeros until allocproc() runs sched_init_proc() on it. Nothing
|
||||
// reaches here on such a slot today -- only a process that has run gets
|
||||
// charged, and a process that has run went through allocproc -- but the
|
||||
// failure if that ever stops being true is a divide-by-zero #GP inside the
|
||||
// scheduler, which is a fault with no good place to report itself.
|
||||
if (p->weight == 0)
|
||||
p->weight = WEIGHT_NICE0;
|
||||
p->vruntime += (ns * WEIGHT_NICE0) / p->weight;
|
||||
}
|
||||
|
||||
// How long a slice should be, given how many processes want one.
|
||||
static uint64
|
||||
sched_slice(int nrunnable)
|
||||
{
|
||||
uint64 slice;
|
||||
|
||||
if (nrunnable <= 1)
|
||||
return SCHED_LATENCY_NS;
|
||||
slice = SCHED_LATENCY_NS / (uint64)nrunnable;
|
||||
return slice < SCHED_MIN_GRAN_NS ? SCHED_MIN_GRAN_NS : slice;
|
||||
}
|
||||
|
||||
// Place a process that has just become runnable, and keep min_vruntime
|
||||
// honest. ptable.lock must be held.
|
||||
//
|
||||
// Called from wakeup1() and from anywhere else that moves a process into
|
||||
// RUNNABLE from a state where it was not accruing vruntime.
|
||||
static void
|
||||
sched_enqueue(struct proc *p)
|
||||
{
|
||||
uint64 floor = min_vruntime > SCHED_LATENCY_NS
|
||||
? min_vruntime - SCHED_LATENCY_NS : 0;
|
||||
|
||||
if (p->vruntime < floor)
|
||||
p->vruntime = floor;
|
||||
p->slice_used = 0;
|
||||
}
|
||||
|
||||
// The runnable process with the smallest vruntime, and a count of how many
|
||||
// were in the running. ptable.lock must be held.
|
||||
static struct proc *
|
||||
pick_next(int *nrunnable)
|
||||
{
|
||||
struct proc *p, *best = 0;
|
||||
int n = 0;
|
||||
|
||||
for (p = ptable.proc; p < &ptable.proc[NPROC]; p++) {
|
||||
// A wedged process is passed over for ever. It is still RUNNABLE, still
|
||||
// owns its descriptors and still appears in every listing -- which is
|
||||
// the point: it looks exactly like a process that is merely busy.
|
||||
if (p->state != RUNNABLE || p->wedged)
|
||||
continue;
|
||||
n++;
|
||||
if (best == 0 || p->vruntime < best->vruntime)
|
||||
best = p;
|
||||
}
|
||||
*nrunnable = n;
|
||||
return best;
|
||||
}
|
||||
|
||||
// Whether the running process has used up its slice, so trap() knows whether
|
||||
// the timer tick should become a yield(). No lock: it reads only the current
|
||||
// process's own fields, and the only writer of them is the current process.
|
||||
//
|
||||
// This is the line that removed a hundred context switches a second from an
|
||||
// idle machine. The old code yielded on *every* tick unconditionally.
|
||||
int
|
||||
sched_slice_expired(void)
|
||||
{
|
||||
struct proc *p = myproc();
|
||||
uint64 now, ns;
|
||||
|
||||
if (p == 0 || p->ran_at == 0)
|
||||
return 0;
|
||||
now = rdtsc();
|
||||
if (now <= p->ran_at)
|
||||
return 0;
|
||||
ns = tsc_to_ns(now - p->ran_at);
|
||||
return (p->slice_used + ns) >= p->slice_ns;
|
||||
}
|
||||
|
||||
void
|
||||
scheduler(void)
|
||||
{
|
||||
struct proc *p;
|
||||
int skipped = 0;
|
||||
int nrunnable;
|
||||
|
||||
for(;;){
|
||||
sti();
|
||||
|
||||
acquire(&ptable.lock);
|
||||
for(p = ptable.proc; p < &ptable.proc[NPROC]; p++){
|
||||
// A wedged process is passed over for ever. It is still RUNNABLE, still
|
||||
// owns its descriptors and still appears in every listing -- which is
|
||||
// the point: it looks exactly like a process that is merely busy.
|
||||
if(p->state != RUNNABLE || p->wedged) {
|
||||
skipped++;
|
||||
p = pick_next(&nrunnable);
|
||||
if (p == 0) {
|
||||
// Nothing to run. Drop the lock before halting -- holding it through a
|
||||
// hlt() would stop every other CPU from scheduling until an interrupt
|
||||
// happened to arrive here.
|
||||
release(&ptable.lock);
|
||||
hlt();
|
||||
continue;
|
||||
}
|
||||
skipped = 0;
|
||||
|
||||
// min_vruntime tracks the front of the queue and never moves backwards,
|
||||
// so a process that sleeps through a busy period cannot return with a
|
||||
// claim on all the CPU it missed. See the comment on the variable.
|
||||
if (p->vruntime > min_vruntime)
|
||||
min_vruntime = p->vruntime;
|
||||
|
||||
p->slice_ns = sched_slice(nrunnable);
|
||||
p->slice_used = 0;
|
||||
p->ran_at = rdtsc();
|
||||
|
||||
proc_set(p);
|
||||
switchuvm(p);
|
||||
p->state = RUNNING;
|
||||
swtch(&mycpu()->scheduler, p->context);
|
||||
switchkvm();
|
||||
|
||||
proc_set(0);
|
||||
}
|
||||
|
||||
// Back from swtch: p has given the CPU up, through sched(). It has
|
||||
// already been charged there, where the reason for the switch was known.
|
||||
release(&ptable.lock);
|
||||
if (skipped > NPROC) {
|
||||
hlt();
|
||||
skipped = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -843,6 +1204,14 @@ sched(void)
|
|||
if(readeflags()&FL_IF)
|
||||
panic("sched interruptible");
|
||||
intena = mycpu()->intena;
|
||||
|
||||
// Charge for the time just run before handing the CPU back. Here rather
|
||||
// than in scheduler(), because this is the one point every path off a CPU
|
||||
// goes through -- yield(), sleep() and exit() alike -- and doing it on the
|
||||
// way out means the vruntime the scheduler then compares is already
|
||||
// up to date rather than a tick stale.
|
||||
sched_charge(myproc());
|
||||
|
||||
swtch(&myproc()->context, mycpu()->scheduler);
|
||||
mycpu()->intena = intena;
|
||||
}
|
||||
|
|
@ -852,6 +1221,7 @@ yield(void)
|
|||
{
|
||||
acquire(&ptable.lock);
|
||||
myproc()->state = RUNNABLE;
|
||||
myproc()->nivcsw++; // involuntary: the slice ran out
|
||||
sched();
|
||||
release(&ptable.lock);
|
||||
}
|
||||
|
|
@ -1048,6 +1418,7 @@ sleep(void *chan, struct spinlock *lk)
|
|||
|
||||
myproc()->chan = chan;
|
||||
myproc()->state = SLEEPING;
|
||||
myproc()->nvcsw++; // voluntary: it asked to wait
|
||||
sched();
|
||||
|
||||
myproc()->chan = 0;
|
||||
|
|
@ -1064,8 +1435,12 @@ wakeup1(void *chan)
|
|||
struct proc *p;
|
||||
|
||||
for(p = ptable.proc; p < &ptable.proc[NPROC]; p++)
|
||||
if(p->state == SLEEPING && p->chan == chan)
|
||||
if(p->state == SLEEPING && p->chan == chan){
|
||||
p->state = RUNNABLE;
|
||||
// Pull it forward to the front of the timeline if it has been asleep
|
||||
// long enough to have fallen off the back of it -- see sched_enqueue.
|
||||
sched_enqueue(p);
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
|
|
@ -1130,8 +1505,10 @@ sigsend(int pid, int sig)
|
|||
p->killed = 1;
|
||||
if(p->killed && p->killsig == 0)
|
||||
p->killsig = sig; // so exit() can report the real cause
|
||||
if(p->state == SLEEPING)
|
||||
if(p->state == SLEEPING){
|
||||
p->state = RUNNABLE;
|
||||
sched_enqueue(p); // a signal is a wakeup too; same floor
|
||||
}
|
||||
release(&ptable.lock);
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -1181,8 +1558,10 @@ sigsend_pgrp(int pgid, int sig)
|
|||
if(p->killsig == 0)
|
||||
p->killsig = sig;
|
||||
}
|
||||
if(p->state == SLEEPING)
|
||||
if(p->state == SLEEPING){
|
||||
p->state = RUNNABLE;
|
||||
sched_enqueue(p); // a signal is a wakeup too; same floor
|
||||
}
|
||||
}
|
||||
release(&ptable.lock);
|
||||
|
||||
|
|
@ -1274,8 +1653,10 @@ kill(int pid)
|
|||
for(p = ptable.proc; p < &ptable.proc[NPROC]; p++){
|
||||
if(p->pid == pid){
|
||||
p->killed = 1;
|
||||
if(p->state == SLEEPING)
|
||||
if(p->state == SLEEPING){
|
||||
p->state = RUNNABLE;
|
||||
sched_enqueue(p); // a signal is a wakeup too; same floor
|
||||
}
|
||||
release(&ptable.lock);
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -1354,6 +1735,10 @@ proc_snapshot(int idx, struct procinfo *out)
|
|||
out->stime = p->stime;
|
||||
out->cutime = p->cutime;
|
||||
out->cstime = p->cstime;
|
||||
out->nice = p->nice;
|
||||
out->vruntime = p->vruntime;
|
||||
out->nvcsw = p->nvcsw;
|
||||
out->nivcsw = p->nivcsw;
|
||||
safestrcpy(out->name, p->name, sizeof(out->name));
|
||||
safestrcpy(out->exe, p->exe, sizeof(out->exe));
|
||||
release(&ptable.lock);
|
||||
|
|
|
|||
|
|
@ -842,6 +842,38 @@ addr_t sys_mremap(void) { return -ENOMEM; }
|
|||
// Resource limits: always report unlimited.
|
||||
addr_t sys_setrlimit(void) { return 0; }
|
||||
|
||||
// Process priority: always report 0.
|
||||
addr_t sys_getpriority(void) { return 0; }
|
||||
addr_t sys_setpriority(void) { return 0; }
|
||||
// Process priority. These returned a flat 0 for as long as the scheduler had
|
||||
// no notion of priority to report; it has one now (kernel/proc/proc.c), and a
|
||||
// stub that says "everything is nice 0" is a lie the moment `renice` works.
|
||||
//
|
||||
// PRIO_PROCESS is 0, and `who` of 0 means the caller -- both from POSIX.
|
||||
#define PRIO_PROCESS 0
|
||||
|
||||
addr_t
|
||||
sys_getpriority(void)
|
||||
{
|
||||
int which, who, nice = 0, r;
|
||||
|
||||
argint(0, &which); argint(1, &who);
|
||||
if (which != PRIO_PROCESS)
|
||||
return -EINVAL;
|
||||
if ((r = proc_getnice(who, &nice)) < 0)
|
||||
return r;
|
||||
|
||||
// getpriority(2) returns the nice value *offset by 20*, because -1 is a
|
||||
// legal nice level and also how a syscall says "error". libc subtracts the
|
||||
// 20 back off. Returning the raw nice value here makes `renice -n -1` look
|
||||
// like a failure to every caller that checks.
|
||||
return (addr_t)(20 - nice);
|
||||
}
|
||||
|
||||
addr_t
|
||||
sys_setpriority(void)
|
||||
{
|
||||
int which, who, nice;
|
||||
|
||||
argint(0, &which); argint(1, &who); argint(2, &nice);
|
||||
if (which != PRIO_PROCESS)
|
||||
return -EINVAL;
|
||||
return (addr_t)proc_setnice(who, nice);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -362,7 +362,11 @@ addr_t sys_setgroups(void) { return 0; }
|
|||
|
||||
// --- Process/scheduling extensions ---
|
||||
|
||||
addr_t sys_sched_yield(void) { return 0; }
|
||||
// A real yield. It returned 0 without giving anything up, which cost nothing
|
||||
// while every timer tick was a yield anyway; with the fair scheduler a tick
|
||||
// only preempts once the slice is spent, so a spin-and-yield loop (musl's
|
||||
// locks, a polling driver in userland) would hold the CPU for all of it.
|
||||
addr_t sys_sched_yield(void) { yield(); return 0; }
|
||||
|
||||
// Every processor that started, since nothing here pins a process anywhere.
|
||||
//
|
||||
|
|
|
|||
|
|
@ -251,8 +251,8 @@ SYSCALL(142, 151, mlockall)
|
|||
SYSCALL(143, 152, munlockall)
|
||||
SYSCALL(144, 221, fadvise64)
|
||||
SYSCALL(145, 160, setrlimit)
|
||||
SYSCALL(146, NO_LINUX, getpriority)
|
||||
SYSCALL(147, NO_LINUX, setpriority)
|
||||
SYSCALL(146, 140, getpriority)
|
||||
SYSCALL(147, 141, setpriority)
|
||||
|
||||
// --- at-syscalls ---
|
||||
SYSCALL(158, 258, mkdirat)
|
||||
|
|
|
|||
Loading…
Reference in a new issue