diff --git a/.gitattributes b/.gitattributes index 1cca36e..9957a94 100644 --- a/.gitattributes +++ b/.gitattributes @@ -27,3 +27,11 @@ *.webp binary *.pdf binary *.wasm binary + +# The language bar on the repository page (Forgejo, GitHub) counts what +# linguist counts. Data tables are generated, not written, and Niveus is +# another project's design system carried for the website; neither is SchOS +# code. tools/python/stats.py leaves out the same files. +include/gfx/font.h linguist-generated +include/gfx/logo.h linguist-generated +docs/Website/niveus/** linguist-vendored diff --git a/README.md b/README.md index eefb45d..87f9dfc 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,12 @@
-SchOS — Shift, lit, on graphite +Shift, the SchOS mark # SchOS **A 64-bit operating system, written from the metal up.** -SMP · copy-on-write filesystem · TCP/IP · PCI, USB and virtio drivers · TrueType text · Linux syscall ABI +SMP kernel · copy-on-write filesystem · Linux syscall ABI · Alpine packages · X11 on real hardware
@@ -15,13 +15,46 @@ SMP · copy-on-write filesystem · TCP/IP · PCI, USB and virtio drivers · True SchOS began as [xv6](https://github.com/mit-pdos/xv6-public) and is now mostly something else. The scheduler, the memory manager and the trap path have been rewritten for x86-64 and for more than one processor; the filesystem has been -replaced outright; the syscall table speaks Linux numbering so that real -software — musl, toybox, mksh — runs unmodified. +replaced outright; the syscall table speaks Linux numbering, so real Linux +software runs on it unmodified — statically or dynamically linked, straight +out of Alpine's package repositories. It is a hobby system built by one person, and it is honest about that: what works below has been *run*, not inferred, and the gaps are listed as plainly as the features. +## In numbers + + +| | | +|---|---| +| **First-party code** | 61,367 lines, plus 21,128 lines of comments (26%) in 272 files | +| **Kernel** (`kernel/`, `include/`) | 45,207 lines of code, 16,111 of comments | +| **Syscalls** | 208, of which 184 on Linux numbering | +| **Drivers** | 15 in the kernel | +| **First-party programs** | 29 in `cmd/` | +| **Alpine packages in the root** | 88, from 11 asked for in `ports/alpine/world` | +| **History** | 253 commits since 2026-08-09 (76 days) | + +
+Lines by language + +| Language | Code | Comments | +|---|--:|--:| +| C | 56,580 | 19,427 | +| Assembly | 1,750 | 336 | +| PowerShell | 1,362 | 638 | +| Python | 759 | 147 | +| Shell | 716 | 469 | +| Perl | 129 | 45 | +| Linker script | 71 | 66 | + +
+ +Measured by tools/python/stats.py; first-party code only, so +ports/ and generated tables are not counted. + + ## The pieces | Layer | Name | What it is | @@ -29,6 +62,7 @@ the features. | System | **SchOS** | The distribution — kernel, userland, tooling | | Kernel | **SchIFT** | Multi-core x86-64 kernel with a modular VFS | | Filesystem | **SchFS** | Transactional, copy-on-write, the root filesystem | +| Image format | **SCFF** | The SchOS Compressed File Format the live root ships in | | API | **SchAPI** | System programming interface — the stable surface | | Compositor | **SchCom** | Display compositor *(planned)* | | Window manager | **SchWM** | *(planned)* | @@ -39,45 +73,61 @@ its distribution. ## What works -**Kernel** — Long mode, 4-level paging, preemptive multitasking. **SMP**: one, -two and four processors all boot, with CPU enumeration from the ACPI MADT rather -than the 1994 MP table. Memory sized from the E820 map, so it runs in tens of -megabytes rather than demanding hundreds. +**Kernel** — Long mode, 4-level paging, preemptive multitasking. **SMP** up to +64 processors, enumerated from the ACPI MADT. ACPI through +[uACPI](https://github.com/uACPI/uACPI): interrupt routing from `_PRT`, the +power button, clean power-off and reset. Memory sized from the E820/UEFI map. **Filesystem** — **SchFS** is the root: a copy-on-write B-tree with a uberblock ring, so a crash leaves either the old tree or the new one and there -is nothing to repair. Free space is derived from the tree rather than stored -beside it, which removes a whole class of disagreement. `/proc/schfs` shows the -generation counter advancing as writes commit. +is nothing to repair. The live image ships as **[SCFF](docs/architecture/scff.md)**: +the 358 MB root compresses to 102 MB with zstd and is unpacked a chunk at a +time, the first time something reads it. `/proc` has `/proc//fd`, +`/proc/self` and the rest that `ps`, `top` and Xorg read. -**Processes** — fork, exec, wait, pipes, signals, sessions and process groups, -`mmap` (anonymous and file-backed), POSIX shared memory, Unix domain sockets, -`poll`. +**Processes** — fork, exec, wait, pipes, signals, sessions and job control, +`mmap` (anonymous and file-backed), POSIX shared memory, `memfd`, Unix domain +sockets with descriptor passing (`SCM_RIGHTS`), `poll`, `epoll`, `eventfd`, +`timerfd`, `signalfd`. **Dynamic linking**: `exec` loads `ld-musl`, so +binaries come from Alpine as they are. -**Devices** — PCI enumeration with a bus-agnostic device model at -`/proc/devices`. Drivers for **e1000** networking, **virtio-gpu** and stdvga -displays, **xHCI** USB with HID, hub and mass-storage class drivers, IDE, PS/2, -serial. `/dev/fb0` speaks the Linux fbdev interface. +**Packages** — the root is managed by Alpine's **apk**. `ports/alpine/world` +says what is installed; the build resolves it once into `world.lock` and every +image after that is reproducible. SchOS itself is one more package in it. -**Networking** — ARP, IPv4, ICMP, UDP, TCP and DHCP, on lwIP behind a -driver-agnostic interface class. Verified end to end from the host, both -directions, rather than by inspection. Names resolve, so `wget http://host/` -works; `httpd` serves a directory back the other way. +**Graphics** — **Xorg with Window Maker**, preinstalled, on the framebuffer +(`/dev/fb0`, Linux fbdev). Six virtual terminals, switched with +Ctrl+Alt+F1–F6 or Alt+F1–F6, with X on a terminal of its own; Ctrl+Alt+Backspace +ends the session. Input as Linux evdev on `/dev/input/event*`. -**Userland** — musl, toybox, mksh and zsh, built from source. 268 applets: -`ls`, `cat`, `ps`, `top`, `wc`, `cp`, `mount`, `awk`, `bc`, `diff` and the rest -behave the way you expect, because they are the real programs. `edit` is -[kilo](https://github.com/antirez/kilo) — arrow keys, Ctrl-S, Ctrl-Q, nothing -to learn. +**Devices** — PCI with a bus-agnostic device model at `/proc/devices`. + +| | | +|---|---| +| Storage | AHCI (SATA), NVMe, IDE, USB mass storage | +| Network | Intel e1000, Realtek r8169 and RTL8139 | +| USB | xHCI, hubs, HID keyboards and mice | +| Input | PS/2, I2C-HID laptop touchpads (DesignWare I2C, found through ACPI) | +| Display | UEFI GOP framebuffer, virtio-gpu, Bochs/stdvga | +| Audio | AC'97 | + +**Networking** — ARP, IPv4, ICMP, UDP, TCP and DHCP on lwIP. DNS, `wget` +over **HTTPS**, `ping`, and an `httpd` serving a directory back. + +**Userland** — musl, toybox, mksh and zsh, plus whatever Alpine has: `ls`, +`ps`, `top`, `awk`, `diff`, `xterm`, `fastfetch` behave the way you expect, +because they are the real programs. **Text** — a TrueType rasteriser (`glyf` outlines, composite glyphs, cmap -format 4) replacing the 8×16 cell grid. JetBrains Mono for the terminal, Roboto -for the desktop layer to come. +format 4). JetBrains Mono for the terminal, Roboto for the desktop to come. **When it dies** — PSOGM, the Purple Screen of Guru Meditation: stop code, fault address, call chain, and a QR code containing a text copy of the whole report. +**On real hardware** — booted on an AMD B550 desktop (Ryzen 5 4600G), an ASUS +TUF A15 laptop and an Ivy Bridge HP all-in-one, from a USB stick. + ## Quick start ```sh @@ -86,6 +136,10 @@ make qemu # build it and boot it make iso # a bootable ISO for VMware or real hardware ``` +The ISO is a hybrid image: write it to a USB stick with +`dd if=schos.iso of=/dev/sdX bs=4M` (or Rufus/Etcher in DD mode) and boot it +on any UEFI machine. + Builds on **Linux, macOS, FreeBSD/OpenBSD/NetBSD and Windows** (Git Bash). The toolchain is clang + lld everywhere — it is a cross-compiler by construction, so one LLVM install targets the bare-metal kernel, the 16-bit boot code and musl @@ -108,6 +162,9 @@ make schfs-check # SchFS B-tree, object and image suites `bootcheck` is PowerShell because `pwsh` runs on every supported host. +The numbers at the top are measured, not typed: `python3 tools/python/stats.py` +rewrites them here and on the website. + ## Layout ``` @@ -136,16 +193,19 @@ Another: **syscalls are declared in exactly one place**, ## Not there yet -- **No desktop.** Schcom composites and routes input; the plan is now to put it - on Wayland and delete the homegrown protocol. See - [docs/plan/wayland.md](docs/plan/wayland.md). -- **No descriptor passing.** No `sendmsg`/`recvmsg`, so no `SCM_RIGHTS` — which - is what the GUI work stops at. [docs/plan/foundations.md](docs/plan/foundations.md). -- **No dynamic linking.** `exec` handles static `ET_EXEC` only. +- **No threads.** `clone` makes processes, not threads (`CLONE_VM` is + refused), so multi-threaded programs run on one thread or not at all. Xorg + runs with `-dumbSched` for this reason. +- **No udev.** Xorg is told its input devices by a config file instead of + finding them; `/sys` and the device-event socket are what is missing. +- **USB 2.0 controllers (EHCI)** have no driver. On older machines the ports + wired only to EHCI do nothing; xHCI ports work. +- **Touchpads run in mouse mode**: pointer, click and tap, but no two-finger + scroll yet. - **No page cache**, so writable `MAP_SHARED` of a file is refused rather than - faked. -- **A USB disk cannot be booted from** — the root device is compiled in rather - than selectable. + faked. SysV shared memory is missing too. +- **No installer.** SchOS runs live from the USB stick; nothing is written to + the machine's disks. - SchFS has no Linux driver yet; the plan is on-disk format version first, then FUSE, then a kernel module. - CFF/OpenType outlines and TrueType hinting are deliberately out of scope. diff --git a/docs/Website/index.html b/docs/Website/index.html index 80340e8..09feebf 100644 --- a/docs/Website/index.html +++ b/docs/Website/index.html @@ -301,8 +301,8 @@
Four-level paging, multiple cores, preemptive scheduling, a calibrated - LAPIC timer, real signal delivery and job control. 207 syscalls on - Linux numbering. + LAPIC timer, real signal delivery and job control. Close to two + hundred syscalls on Linux numbering.
@@ -545,20 +545,46 @@

Measured, not marketed

+
-
lines in kernel/ and include/
-
46,961
+
lines of code in kernel/ and include/
+
45,207
+
+
+
lines of comments alongside them
+
16,111
+
+
+
first-party lines in all
+
61,367
syscalls, Linux numbering
-
207
+
184
+
+
+
drivers in the kernel
+
15
+
+
+
Alpine packages in the live root
+
88
+
+
+
commits
+
253
+
+
+
days since the first one
+
76
patches carried against fastfetch
0
+ and + -- and everything between them is regenerated. A number in +the README that somebody typed is a number that was true once; these are true +on the commit that ran this. + +First-party code only: ports/ is other people's software, build/ is output, +and docs/Website/niveus is a vendored design system. Comment lines are counted +apart from code, because in this tree they are a large part of the work. +""" + +import datetime +import os +import re +import subprocess +import sys + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +FIRST_PARTY = ["kernel", "include", "boot", "lib", "cmd", "tools"] +LANGS = { + ".c": "C", ".h": "C", ".S": "Assembly", ".s": "Assembly", ".asm": "Assembly", + ".py": "Python", ".sh": "Shell", ".ps1": "PowerShell", ".pl": "Perl", + ".ld": "Linker script", ".inc": "C", +} +GENERATED = re.compile(r"include/gfx/(logo|font)[^/]*\.h$|_data\.h$") + + +def git(*args): + return subprocess.run(["git", "-C", ROOT, *args], capture_output=True, + text=True, check=True).stdout + + +def tracked(): + return [p for p in git("ls-files").splitlines() + if p.split("/")[0] in FIRST_PARTY and not GENERATED.search(p)] + + +def count(path, lang): + """(code, comment, blank) lines. Block comments are followed for C-family + files; for the scripting languages a line starting with # is a comment.""" + code = comment = blank = 0 + in_block = False + try: + with open(os.path.join(ROOT, path), encoding="utf-8", errors="replace") as f: + lines = f.read().splitlines() + except OSError: + return 0, 0, 0 + hash_comments = lang in ("Python", "Shell", "PowerShell", "Perl") + for raw in lines: + s = raw.strip() + if not s: + blank += 1 + continue + if hash_comments: + if s.startswith("#") and not s.startswith("#!"): + comment += 1 + else: + code += 1 + continue + if in_block: + comment += 1 + if "*/" in s: + in_block = False + if s.split("*/", 1)[1].strip(): + comment -= 1 + code += 1 + continue + if s.startswith("//") or (lang == "Assembly" and s.startswith("#") and + not re.match(r"#\s*(include|define|if|endif|else|elif|undef)", s)): + comment += 1 + elif s.startswith("/*"): + comment += 1 + if "*/" not in s[2:]: + in_block = True + else: + code += 1 + return code, comment, blank + + +def measure(): + by_lang = {} + kernel = [0, 0] + files = 0 + for p in tracked(): + ext = os.path.splitext(p)[1] + lang = LANGS.get(ext) + if lang is None: + continue + c, m, _ = count(p, lang) + files += 1 + t = by_lang.setdefault(lang, [0, 0]) + t[0] += c + t[1] += m + if p.startswith(("kernel/", "include/")): + kernel[0] += c + kernel[1] += m + + table = open(os.path.join(ROOT, "kernel/sys/table.inc")).read() + calls = re.findall(r"^SYSCALL\(\s*\d+,\s*(\w+),", table, re.M) + linux = [n for n in calls if n != "NO_LINUX"] + + drivers = set() + for p in tracked(): + if p.startswith("kernel/drivers/") and p.endswith(".c"): + parts = p.split("/") + drivers.add("/".join(parts[2:4]) if len(parts) > 4 else parts[-1]) + drivers.add("i2c-hid") # kernel/acpi/i2c_hid.c, found through ACPI + + programs = {p.split("/")[1] for p in tracked() if p.startswith("cmd/") and p.count("/") >= 2} + + world = [l for l in open(os.path.join(ROOT, "ports/alpine/world")).read().splitlines() + if l.strip() and not l.startswith("#")] + lock = [l for l in open(os.path.join(ROOT, "ports/alpine/world.lock")).read().splitlines() + if l.strip() and not l.startswith("#")] + + commits = int(git("rev-list", "--count", "HEAD").strip()) + first = git("log", "--reverse", "--format=%ad", "--date=short").splitlines()[0] + days = (datetime.date.today() - datetime.date.fromisoformat(first)).days + + code = sum(v[0] for v in by_lang.values()) + comments = sum(v[1] for v in by_lang.values()) + return { + "code": code, "comments": comments, "files": files, + "by_lang": sorted(by_lang.items(), key=lambda kv: -kv[1][0]), + "kernel_code": kernel[0], "kernel_comments": kernel[1], + "syscalls": len(calls), "linux_syscalls": len(linux), + "drivers": len(drivers), "programs": len(programs), + "world": len(world), "packages": len(lock), + "commits": commits, "first": first, "days": days, + } + + +def n(v): + return f"{v:,}" + + +def readme_block(s): + ratio = 100 * s["comments"] / max(1, s["code"] + s["comments"]) + rows = "\n".join(f"| {lang} | {n(c)} | {n(m)} |" for lang, (c, m) in s["by_lang"]) + return f""" +| | | +|---|---| +| **First-party code** | {n(s['code'])} lines, plus {n(s['comments'])} lines of comments ({ratio:.0f}%) in {n(s['files'])} files | +| **Kernel** (`kernel/`, `include/`) | {n(s['kernel_code'])} lines of code, {n(s['kernel_comments'])} of comments | +| **Syscalls** | {s['syscalls']}, of which {s['linux_syscalls']} on Linux numbering | +| **Drivers** | {s['drivers']} in the kernel | +| **First-party programs** | {s['programs']} in `cmd/` | +| **Alpine packages in the root** | {s['packages']}, from {s['world']} asked for in `ports/alpine/world` | +| **History** | {n(s['commits'])} commits since {s['first']} ({s['days']} days) | + +
+Lines by language + +| Language | Code | Comments | +|---|--:|--:| +{rows} + +
+ +Measured by tools/python/stats.py; first-party code only, so +ports/ and generated tables are not counted. +""" + + +def site_block(s): + items = [ + (f"lines of code in kernel/ and include/", n(s["kernel_code"])), + ("lines of comments alongside them", n(s["kernel_comments"])), + ("first-party lines in all", n(s["code"])), + ("syscalls, Linux numbering", str(s["linux_syscalls"])), + ("drivers in the kernel", str(s["drivers"])), + ("Alpine packages in the live root", str(s["packages"])), + ("commits", n(s["commits"])), + ("days since the first one", str(s["days"])), + ("patches carried against fastfetch", "0"), + ] + out = ["", "
"] + for label, value in items: + out += ["
", + f"
{label}
", + f"
{value}
", + "
"] + out += ["
", " "] + return "\n".join(out) + + +def rewrite(path, block): + full = os.path.join(ROOT, path) + text = open(full, encoding="utf-8").read() + new, k = re.subn(r".*?", lambda _: block, + text, flags=re.S) + if k != 1: + sys.exit(f"stats: {path} needs exactly one stats:begin/stats:end pair") + if new != text: + open(full, "w", encoding="utf-8").write(new) + print(f"stats: rewrote {path}") + + +def main(): + s = measure() + if "--print" in sys.argv: + for k, v in s.items(): + print(f"{k}: {v}") + return + rewrite("README.md", readme_block(s)) + rewrite("docs/Website/index.html", site_block(s)) + + +if __name__ == "__main__": + main()