nexoniarz/YAPF
Watch
1
0
Fork
You've already forked YAPF
 0
mirror of https://github.com/Nexoniarz/YAPF.git synced 2026-11-03 04:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity

YAPF v1: fast byte-aligned format, SIMD, tools and plugins

Browse source 
Format (version 1, replaces the earlier experimental versions):
- per-tile YCoCg-R or subtract-green colour transform, 8-bit modular
- 12 row filters per 8-row band (Left, Gradient, Up-right, Up-average,
  Up 1-8 rows), none with more than a one-add serial dependency
- residuals in groups of 8 at the group's width (w bits x 8 = w bytes),
  4-bit control codes with repeat-previous-width runs chosen by DP
- constant and stored tiles, independent 64x64 tiles, mips, GPU hints
- all sizes validated before decoding; corrupt input is rejected

Library:
- SIMD on by default: SSSE3 (runtime CPUID check; GCC, Clang, MSVC,
  MinGW) and NEON on ARM64; -DYAPF_NO_SIMD for plain C
- yapf_load_memory, yapf_load_mt, yapf_load_memory_mt
- 6-10x faster decoding than PNG (stb_image) on one core; -O2 and -O3
  builds perform the same
- fixes for macOS (sysconf) and the Windows import library exports

Tools and integrations:
- `yapf` CLI: convert PNG/JPG/TGA/BMP/PSD/GIF <-> YAPF, info, bench,
  thumbnail, mips
- JavaScript and Python bindings (byte-identical to the C encoder)
- GIMP 3 plugin: exports the flattened image, optional mipmaps
- Blender extension: import (also as plane), export, drag and drop
- VS Code preview, Figma plugin, Photoshop scripts, Linux MIME type and
  thumbnailer
- Makefile, updated build.sh / build.bat, tools/build_release.sh
- tests/test_yapf.c (784 checks) and bindings/js/test.js

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nexoniarz 2026-10-03 14:17:59 +02:00
commit 2ca726e3af
48 changed files with 14905 additions and 710 deletions
Download patch file Download diff file Expand all files Collapse all files

380
README.md
View file

@ -1,90 +1,127 @@
# YAPF — Yet Another Picture Format
**Fast. Small. Easy.**
**Lossless images that decode several times faster than PNG — on one core.**
YAPF is a lossless image codec built around a single principle: decoding should
be as close to a memory copy as possible. Instead of running a general-purpose
compressor like DEFLATE over pixel data, YAPF encodes images with five cheap
operations — a pixel cache lookup, two flavours of delta encoding, run-length
encoding, and verbatim pixel storage — producing files that are smaller than
uncompressed formats while decoding nearly as fast as `memcpy`.
YAPF is a lossless image format for photos, UI and game assets, built for
decode speed first. There is no entropy coder, no bit reader and no context
model: every structure is byte aligned, and the decoder runs short loops that
modern CPUs execute at memory speed — with SSE / NEON where available. The
encoder does the hard work, so the decoder only follows instructions.
- **6–10× faster decoding than PNG** (stb_image), single-threaded; a 4K image
decodes in about 6 ms (5.6 GB/s). Multithreaded decoding is optional.
- **Smaller than PNG** on UI, icons and screenshots; 20–45 % larger on photos
and natural textures — the price of having no entropy coder.
- **Made for games and tools too**: mipmaps, GPU format hint, sRGB /
premultiplied flags, and random access to 64×64 tiles.
- **Tiny and portable**: two C99 files, no dependencies. Plus a CLI, Python,
JavaScript and plugins for the tools you already use.
This is **YAPF version 1**, the first stable format.
---
## Why YAPF?
## Use it in…
Standard lossless formats like PNG compress well, but decompressing DEFLATE
streams is CPU-intensive. On large images (4K and above) or in
latency-sensitive applications (game engines, real-time pipelines, embedded
systems) this is a meaningful bottleneck.
| Where | What you get | Guide | Status |
| :---- | :----------- | :---- | :----- |
| **C / C++** | `yapf.h` + `yapf.c`, drop into any project | [below](#c--c) | ✅ tested on Linux x64, Windows x64, Linux ARM64 |
| **Command line** | convert PNG/JPG/TGA/BMP/PSD/GIF ↔ YAPF, `info`, `bench`, `thumbnail` | [below](#command-line) | ✅ tested on Linux, Windows |
| **GIMP 3** | open, export (with mipmaps) | [extensions/gimp](extensions/gimp) | ✅ tested in GIMP 3.0.8 |
| **Blender 4.2+** | import, export, drag & drop, image as plane | [extensions/blender](extensions/blender) | ✅ tested in Blender 5.1 |
| **VS Code** | image preview with zoom and mip picker | [extensions/vscode](extensions/vscode) | ✅ packaged, preview tested in Chromium |
| **Figma** | import `.yapf`, export selection as `.yapf` | [extensions/figma](extensions/figma) | ✅ plugin UI tested in Chromium |
| **Photoshop** | File → Scripts → Open YAPF / Save as YAPF | [extensions/photoshop](extensions/photoshop) | ⚠️ not yet tested inside Photoshop |
| **Linux desktop** | file type + thumbnails in GNOME Files, Nemo, Caja, Thunar | [extensions/linux](extensions/linux) | ⚠️ thumbnailer tested, not in a file manager |
| **Python** | `yapf.load()` / `yapf.save()` | [bindings/python](bindings/python) | ✅ tested |
| **JavaScript** | browsers, Node.js, workers | [bindings/js](bindings/js) | ✅ tested, byte-identical to C |
YAPF trades a modest compression ratio advantage for dramatically lower decode
complexity. There are no Huffman tables to build, no sliding windows to
maintain, and no bit-level unpacking. The decoder is a tight byte-dispatch
loop that any modern CPU handles at memory bandwidth.
### Performance
Benchmark: loading a 5K professional photograph (5120 × 2880 px).
| Format | Load time |
| :----- | ----------: |
| **YAPF** | **0.111 s** |
| PNG | 0.194 s |
YAPF decodes approximately **1.7× faster** than PNG on the same hardware
without SIMD or threading.
### When YAPF excels
- **Pixel art, sprites, UI textures** — large flat regions of identical colour
compress to single RUN bytes; palette-like images benefit heavily from the
cache.
- **Screenshots and diagrams** — solid backgrounds and repeated gradients hit
DIFF/LUMA and the cache constantly.
- **Game asset pipelines** — load textures at startup without paying a
decompression tax every frame.
- **Embedded / resource-constrained targets** — the entire codec fits in ~200
lines of portable C with zero external dependencies.
Pre-built downloads (command-line tool and libraries for Windows, macOS and
Linux, plus the plugin packages) are published on the
[Releases](https://github.com/Nexoniarz/YAPF/releases) page; everything can
also be built from source as described below.
---
## Integration
## Command line
The core library is two files. Drop them into your project and compile:
```sh
yapf photo.png photo.yapf # encode (formats by file extension)
yapf texture.png texture.yapf --mips # with a full mip chain
yapf photo.yapf photo.png # decode to PNG / JPG / BMP / TGA
yapf texture.yapf mip2.png --mip 2 # export one mip level
yapf info photo.yapf # size, channels, mips, flags, ratio
yapf bench photo.yapf # decode speed, 1 thread and all cores
yapf thumbnail photo.yapf thumb.png 256
```
Options when writing `.yapf`: `--mips`, `--linear` (pixels are not sRGB),
`--premult` (alpha is premultiplied).
---
## Benchmark
Single core, AMD Ryzen 7 5700G, `gcc -O2`. PNG is decoded with `stb_image`;
the PNG files are the ones shipped with the respective projects. "Plain C"
is the library built with `-DYAPF_NO_SIMD`; the default build uses SSE / NEON.
| Image | Raw | PNG | **YAPF** | vs PNG | PNG decode | YAPF, plain C | **YAPF (default)** |
| :--------------------- | ------: | -----: | -------: | -----: | ---------: | ------------: | -----------------: |
| Wallpaper 3840×2160 | 32400 K | 1812 K | 2267 K | 125 % | 44.7 ms¹ | 10.3 ms | **5.9 ms** (5.6 GB/s) |
| UI screenshot 261×430 | 438 K | 176 K | 143 K | 81 % | 1.6 ms | 0.31 ms | **0.17 ms** |
| UI screenshot 305×529 | 630 K | 201 K | 185 K | 92 % | 2.1 ms | 0.45 ms | **0.24 ms** |
| App icon 128×128 | 64 K | 24 K | 23 K | 93 % | 0.26 ms | 0.06 ms | **0.03 ms** |
| Rock texture 512×512 | 768 K | 182 K | 221 K | 121 % | 1.4 ms | 0.35 ms | **0.16 ms** |
| Wood texture 512×512 | 768 K | 185 K | 234 K | 126 % | 1.4 ms | 0.46 ms | **0.16 ms** |
| Canvas texture 512×512 | 768 K | 171 K | 245 K | 143 % | 1.0 ms | 0.35 ms | **0.15 ms** |
| Pixel art 512×512 | 1024 K | — | 66 K | — | — | 0.74 ms | **0.32 ms** |
| Random noise 512×512 | 1024 K | — | 1024 K | — | — | 0.1 ms | 0.1 ms |
¹ PNG decode of the wallpaper measured between 44.7 and 66 ms depending on
the harness; the table uses the fastest. `-O2` and `-O3` builds of YAPF
perform the same.
---
## C / C++
The library is two files. Add them to your project:
```
yapf.h — public API and format documentation
yapf.c — encoder / decoder implementation
yapf.h — public API and the complete format specification
yapf.c — encoder and decoder
```
```c
#include "yapf.h"
int main(void) {
/* Load */
yapf_image_t *img = yapf_load("photo.yapf");
if (!img) { /* file not found or not a valid YAPF stream */ }
printf("%u x %u, %u channel(s)\n",
img->width, img->height, img->channels);
/* Save */
int rc = yapf_save("output.yapf", img);
if (rc != YAPF_OK) { /* write error */ }
yapf_image_t *img = yapf_load("texture.yapf"); /* or yapf_load_memory(buf, len) */
if (img) {
/* img->pixels: width × height × channels bytes, rows top to bottom */
upload_texture(img->width, img->height, img->channels, img->pixels);
yapf_free(img);
return 0;
}
yapf_image_t out = { .width = w, .height = h, .channels = 4,
.mip_levels = 1, .pixels = rgba, .flags = YAPF_FLAG_SRGB };
yapf_save("out.yapf", &out);
```
Compile with any C99-capable compiler — no flags required for the baseline:
Compile with any C99 compiler (`-pthread` on Unix for the `_mt` loaders):
```sh
gcc -O2 -std=c99 main.c yapf.c -o myapp
cc -O2 -std=c99 main.c yapf.c -o app -pthread
```
---
SIMD is on by default: SSSE3 on x86 / x64 (chosen at run time with CPUID,
so binaries still run on older CPUs) and NEON on ARM64, with GCC, Clang,
MSVC and MinGW.
| Define | Effect |
| :------------------ | :----- |
| `-DYAPF_NO_SIMD` | plain C only |
| `-DYAPF_NO_THREADS` | no threads; the `_mt` loaders decode on the calling thread, and only the C standard library is needed |
## API Reference
@ -92,15 +129,20 @@ gcc -O2 -std=c99 main.c yapf.c -o myapp
```c
typedef struct {
uint32_t width; /* pixels per row */
uint32_t height; /* rows */
uint8_t channels; /* 1 gray / 2 gray+alpha / 3 RGB / 4 RGBA */
uint8_t *pixels; /* raw pixel buffer, malloc-owned */
uint32_t width; /* pixels per row */
uint32_t height; /* rows */
uint8_t channels; /* 1 gray / 2 gray+alpha / 3 RGB / 4 RGBA */
uint8_t gpu_format; /* YAPF_GPU_* hint */
uint8_t flags; /* YAPF_FLAG_* (premultiplied alpha, sRGB) */
uint8_t mip_levels; /* 1 = base only, up to 16 */
uint8_t *pixels; /* base level, malloc-owned */
uint8_t **mips; /* mips[0] == pixels, mips[1..] smaller levels */
} yapf_image_t;
```
`pixels` is a tightly packed, row-major byte array of `width × height × channels`
bytes. The layout per pixel matches the `channels` value:
bytes. Mip level *m* is `max(1, width >> m) × max(1, height >> m)`. The
layout per pixel matches the `channels` value:
| `channels` | Layout per pixel |
| :--------- | :--------------- |
@ -113,29 +155,40 @@ bytes. The layout per pixel matches the `channels` value:
```c
yapf_image_t *yapf_load(const char *filename);
yapf_image_t *yapf_load_memory(const void *buffer, size_t size);
```
Decodes a YAPF file from disk. Returns a heap-allocated `yapf_image_t` on
success, `NULL` on any failure (file not found, invalid magic, unsupported
version, out of memory). Free with `yapf_free()`.
Decode a YAPF file from disk or from memory (e.g. a game archive). Return a
heap-allocated image on success, `NULL` on any failure (file not found,
invalid or corrupt data, out of memory). Corrupt input is rejected, never
read out of bounds.
```c
yapf_image_t *yapf_load_mt(const char *filename, int threads);
yapf_image_t *yapf_load_memory_mt(const void *buffer, size_t size, int threads);
```
Same, decoding tiles on up to `threads` threads (`0` = one per CPU core).
The output is identical to the single-threaded loaders.
```c
int yapf_save(const char *filename, const yapf_image_t *img);
```
Encodes an image to disk. Returns `YAPF_OK` (0) on success or a negative
`YAPF_ERR_*` code on failure.
Encodes an image to disk. Writes `img->mip_levels`
levels; `img->mips` may be `NULL` when `mip_levels == 1`. Returns `YAPF_OK`
(0) on success or a negative `YAPF_ERR_*` code on failure.
```c
void yapf_free(yapf_image_t *img);
```
Releases memory returned by `yapf_load()`. Passing `NULL` is safe.
Releases memory returned by the loaders. Passing `NULL` is safe.
### Return codes
| Constant | Value | Meaning |
| :---------------- | ----: | :----------------------------------- |
| `YAPF_OK` | 0 | Success |
| `YAPF_ERR_INVALID`| –1 | NULL or logically invalid argument |
| `YAPF_ERR_IO` | –2 | File could not be opened or written |
| Constant | Value | Meaning |
| :----------------- | ----: | :----------------------------------- |
| `YAPF_OK` | 0 | Success |
| `YAPF_ERR_INVALID` | –1 | NULL or logically invalid argument |
| `YAPF_ERR_IO` | –2 | File could not be opened or written |
| `YAPF_ERR_OOM` | –3 | Out of memory |
### Channel constants
@ -148,115 +201,102 @@ YAPF_CHANNELS_RGBA 4
---
## Building the libraries
## Building
The build scripts uses the [LLVM](https://releases.llvm.org/) compiler
| System | Command | Output |
| :----- | :------ | :----- |
| Linux, macOS, BSD | `make` · `make test` · `sudo make install` | `build/`: `libyapf.a`, `libyapf.so` / `.dylib`, `yapf` |
| Linux, macOS (LLVM) | `./build.sh` | `dist/<platform>/` incl. macOS universal binaries |
| Windows (LLVM) | `build.bat` | `dist\windows-x64\`, `dist\windows-arm64\`: `yapf.dll`, `yapf.lib`, `libyapf.a`, `yapf.exe` |
| Windows (Visual Studio) | `cl /O2 /c yapf.c` and add `yapf.obj` to your project | |
| All platforms at once | `tools/build_release.sh` (needs [zig](https://ziglang.org)) | `dist/release/`: every download on the Releases page |
The command-line tool is `tools/yapf_cli.c` + `yapf.c`; it uses the public
domain [stb](https://github.com/nothings/stb) headers in `third_party/`.
---
## How it works
1. **Two colour transforms** — per 64×64 tile, YCoCg-R or subtract-green
(`G, R−G, B−G`), whichever codes smaller. Both run mod 256, so every
plane stays one byte per sample.
2. **Twelve row filters** — per band of 8 rows: Left, Gradient (`L + A − D`),
Up-right, Up-average, or the row 1–8 rows above (catches repeating
textures and dither patterns). None depends on the previous sample by
more than one add, and the vertical ones decode as plain vector code.
3. **Groups of 8** — residuals are packed in groups of 8 at the group's bit
width: 8 values × *w* bits = exactly *w* bytes, so every group is byte
aligned and unpacks with one 64-bit load (or one SIMD shuffle).
4. **4-bit control codes** — one per group, or one per run of up to 64 groups
repeating the previous width; the encoder picks the cheapest sequence with
a small dynamic program.
5. **Validate first, then run** — all sizes in a tile are checked before the
hot loops start, so the loops need no bounds checks. Corrupt files are
rejected, never read out of bounds.
6. **Constant and stored tiles** — a plane that is constant in a tile costs
one byte; a tile that would not shrink is stored raw (noise never grows).
7. **Independent tiles** — any tile can be decoded on its own, in any order,
on any thread.
The complete specification is the comment at the top of [`yapf.h`](yapf.h):
```
HEADER (20 bytes) "YAPF", version 1, channels, GPU hint, mip count,
flags, 3 reserved, width u32, height u32
MIP INDEX per level: offset u32, size u32
LEVEL DATA tile_size u32 × tile_count, then the tiles
TILE (64×64) header byte: stored raw, or constant-plane mask + colour transform
per plane: filter byte per 8 rows, control nibbles,
w bytes per group of 8 values
```
---
## Testing
```sh
# Unix (Linux / macOS)
chmod +x build.sh
./build.sh
# Windows
build.bat
make test # C suite: 784 checks, plus JavaScript if node is installed
node bindings/js/test.js # JavaScript: round-trips, sample file, corrupt input
```
Output goes to `dist/` with one subdirectory per platform:
What has been verified for this release:
```
dist/
├── windows-x64/ yapf.dll yapf.lib
├── windows-arm64/ yapf.dll yapf.lib
├── macos-x64/ libyapf.dylib
├── macos-arm64/ libyapf.dylib
├── linux-x64/ libyapf.so libyapf.a
└── linux-arm64/ libyapf.so libyapf.a
```
- exact round-trips for 1–4 channels, odd sizes from 1×1 up, mip chains,
single- and multi-threaded decoding;
- the same tests pass on **Linux x64**, **Windows x64** (MinGW, run under
Wine), **Linux ARM64** (NEON, run under qemu), with and without SIMD, and
under AddressSanitizer, UndefinedBehaviorSanitizer and ThreadSanitizer;
- thousands of corrupted and truncated files are rejected without crashing;
- the JavaScript and Python versions produce byte-identical files to C;
- the GIMP plugin and the Blender extension in the real applications.
### Optional build flags
| Flag | Effect |
| :---------------- | :-------------------------------------------------- |
| `-DYAPF_USE_SIMD` | Reserved for future SIMD acceleration paths |
The interactive build script will prompt for these options.
The macOS builds are cross-compiled and not yet run on a Mac.
---
## Technical Specification
### File header — 14 bytes
| Offset | Size | Type | Description |
| :----- | ---: | :--------- | :-------------------------------------- |
| 0x00 | 4 | `char[4]` | Magic: `"YAPF"` |
| 0x04 | 1 | `uint8_t` | Format version (currently `1`) |
| 0x05 | 4 | `uint32_t` | Width in pixels, little-endian |
| 0x09 | 4 | `uint32_t` | Height in pixels, little-endian |
| 0x0D | 1 | `uint8_t` | Channel count: 1 / 2 / 3 / 4 |
All multi-byte integers are always stored little-endian, regardless of host
architecture.
### Byte-stream opcodes
The payload begins immediately after the header and is decoded left-to-right.
A 64-slot pixel cache (zero-initialised) and a running "previous pixel" state
(`r=0, g=0, b=0, a=255`) are maintained throughout.
| Byte range | Opcode | Size | Description |
| :---------- | :----- | ------: | :-------------------------------------------- |
| 0x00–0x3F | INDEX | 1 byte | Restore pixel from cache slot `byte & 0x3F` |
| 0x40–0x7F | DIFF | 1 byte | Per-channel deltas: dr = bits[5:4]–2, dg = bits[3:2]–2, db = bits[1:0]–2. Range –2..+1. Alpha unchanged. |
| 0x80–0xBF | LUMA | 2 bytes | Green delta dg = (byte & 0x3F)–32. Second byte: dr = ((b1>>4)&0xF)–8+dg, db = (b1&0xF)–8+dg. Alpha unchanged. |
| 0xC0–0xFD | RUN | 1 byte | Repeat previous pixel (byte&0x3F)+1 times (1–62). Cache not updated. |
| 0xFE | RGB | 4 bytes | Next 3 bytes = R, G, B. Alpha unchanged. |
| 0xFF | RGBA | 5 bytes | Next 4 bytes = R, G, B, A. |
After every non-RUN opcode, the cache slot `HASH(r,g,b,a)` is updated and the
pixel becomes the new previous pixel.
## Project layout
```
HASH(r,g,b,a) = (r×3 + g×5 + b×7 + a×11) mod 64
yapf.h, yapf.c the library (format spec in yapf.h)
tools/yapf_cli.c the `yapf` command-line tool
tools/build_release.sh builds every download for all platforms
tests/test_yapf.c C test suite
bindings/js/ JavaScript encoder / decoder
bindings/python/ Python module
extensions/gimp/ GIMP 3 plugin
extensions/blender/ Blender extension
extensions/vscode/ VS Code preview
extensions/figma/ Figma plugin
extensions/photoshop/ Photoshop scripts
extensions/linux/ MIME type and thumbnailer
third_party/ stb_image / stb_image_write (public domain, CLI only)
other/YAPF.YAPF sample image (the YAPF logo)
```
The stream ends when exactly `width × height` pixels have been decoded; no
end-of-stream marker is required.
### Pixel buffer layout
Within the decoded buffer, each pixel occupies `channels` consecutive bytes in
the order shown in the API table above. Grayscale images store a single
luminance byte per pixel; the encoder internally promotes gray values to
`r = g = b = Y` for uniform opcode selection.
---
## GIMP Plugin
A plugin for GIMP is located in `extensions/gimp/`. See the
[build instructions](extensions/gimp/howtocompile.txt) for details.
Supported colour modes: Gray, Gray+Alpha, RGB, RGBA.
File extension: `.yapf`. MIME type: `image/x-yapf`.
---
## Platforms
| Platform | Architectures | Library format |
| :------------- | :------------ | :------------------ |
| Windows | x64, ARM64 | `.dll` + `.lib` |
| macOS | ARM64 | `.dylib` |
| Linux (glibc) | x64, ARM64 | `.so` |
| Linux (musl) | x64, ARM64 | `.a` (static) |
The core library (`yapf.c` / `yapf.h`) uses only the C99 standard library
(`<stdio.h>`, `<stdlib.h>`, `<string.h>`, `<stdint.h>`) and compiles cleanly
on any conforming C99 or C++ toolchain with no configuration required.
---
## License
Apache License 2.0 — see [LICENSE](LICENSE).
Apache License 2.0 — see [LICENSE](LICENSE). `third_party/` contains
stb_image and stb_image_write by Sean Barrett (public domain / MIT).