nexoniarz/YAPF
Watch
1
0
Fork
You've already forked YAPF
 0
mirror of https://github.com/Nexoniarz/YAPF.git synced 2026-11-03 04:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity

YAPF 2: texture container with GPU blocks, wide samples and streaming

Browse source 
Adds format version 2 alongside version 1 (still written for plain 8-bit
images, so existing readers keep working). Full specification in
docs/FORMAT.md.

- GPU block payloads: BC1-BC7, BC6H, ETC2/EAC, ASTC 4x4-12x12, in tiles
  of 16x16 blocks. Built-in BC1/BC3/BC4/BC5/BC7/BC6H encoders and
  BC1-BC7 decoders (BC6H/BC7 bit-exact with a reference decoder).
- Lossless 16-bit, half and float pixels as byte planes over the 8-bit
  coder; order-preserving float mapping, transforms at sample width.
- Cubemaps, cube arrays, texture arrays, volumes, animation frames.
- Channel meanings, normal convention, linear mask, primaries/transfer,
  mip filter and coverage-preserving mips (yapf_tex_build_mips).
- Small mips stored first, file size in the header, surface/tile tables:
  decode from a file prefix and single tiles (virtual texturing).
- yapf-sum checksums on header, tables and every tile; needs bits so
  decoders refuse cleanly what they cannot read.
- Optional entropy stage: per-surface Huffman code books with a
  neighbourhood context (~10% smaller photos) and per-lane packing of
  GPU blocks (~3x smaller on disk); planes decoded interleaved.
- Notes: author, licence, source, tool, ICC and application tags.
- CLI: --format, --entropy, --coverage, --normal, --map, colour options,
  16-bit/HDR input, pack cube/array/volume/anim, info, verify, export.
- Tests for all of the above, including fuzzing; v1 API reads 8-bit v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVrVthWN8GsmgHGUhhsCYm
This commit is contained in:
Claude 2026-10-10 21:57:14 +00:00
commit a226b06368
No known key found for this signature in database
7 changed files with 5317 additions and 237 deletions
Download patch file Download diff file Expand all files Collapse all files

138
README.md
View file

@ -14,10 +14,17 @@ encoder does the hard work, so the decoder only follows instructions.
and natural textures — the price of having no entropy coder. and natural textures — the price of having no entropy coder.
- **Made for games and tools too**: mipmaps, GPU format hint, sRGB / - **Made for games and tools too**: mipmaps, GPU format hint, sRGB /
premultiplied flags, and random access to 64×64 tiles. premultiplied flags, and random access to 64×64 tiles.
- **A real texture format (YAPF 2)**: GPU-ready BC1–BC7 / BC6H / ETC2 /
ASTC blocks, 16-bit and float pixels, cubemaps, arrays and volumes,
channel meanings (normal X/Y, roughness, AO, …), coverage-preserving
mips, small-mips-first streaming, per-tile checksums, notes and
animation — see [Textures](#textures-yapf-2).
- **Tiny and portable**: two C99 files, no dependencies. Plus a CLI, Python, - **Tiny and portable**: two C99 files, no dependencies. Plus a CLI, Python,
JavaScript and plugins for the tools you already use. JavaScript and plugins for the tools you already use.
This is **YAPF version 1**, the first stable format. Two format versions: **version 1** (an 8-bit image, the first stable
format) and **version 2** (the texture container). Plain 8-bit images are
still written as version 1, so every existing YAPF reader keeps working.
--- ---
@ -67,7 +74,64 @@ yapf compare photo.png ui.png # size and decode speed vs the original fo
``` ```
Options when writing `.yapf`: `--mips`, `--linear` (pixels are not sRGB), Options when writing `.yapf`: `--mips`, `--linear` (pixels are not sRGB),
`--premult` (alpha is premultiplied). `--premult` (alpha is premultiplied), and the texture options below.
---
## Textures (YAPF 2)
Version 2 turns YAPF into a texture container for engines, designed our own
way rather than copied from KTX2 or DDS (the design and the full
specification are in [docs/FORMAT.md](docs/FORMAT.md)):
| Need | YAPF 2 |
| :--- | :----- |
| **GPU-compressed payloads** | BC1, BC2, BC3, BC4, BC5, BC6H, BC7, ETC2 / EAC and ASTC 4×4–12×12 blocks, stored as the GPU takes them. Built-in encoders for BC1, BC3, BC4, BC5, BC7 and BC6H; or bring blocks from your own encoder. |
| **More than 8 bits** | lossless 16-bit, half and float pixels, coded as byte planes with the same fast machinery |
| **Cubes, arrays, volumes** | cubemaps and cube arrays (reflection probes, skies), texture arrays (terrain layers), 3D textures (colour-grading LUTs) |
| **Channel meaning** | per channel: normal X/Y/Z, roughness, metalness, AO, height, opacity, …; OpenGL or DirectX normal convention; linear channels inside sRGB images |
| **Streaming** | small mips are stored first: a file prefix is a complete low-resolution texture (`yapf info` prints the prefix sizes); single 64 × 64 tiles decode on their own for virtual texturing |
| **Alpha-tested mips** | `--coverage REF` keeps fences and foliage from thinning out with distance; the filter used is recorded |
| **Entropy coding (optional)** | `--entropy`: shared per-surface Huffman code books with a neighbourhood context — about 10 % smaller photos, about 3× smaller GPU blocks on disk; tiles stay independent |
| **Metadata** | notes (author, licence, source, tool, ICC profile, orientation, DPI, HDR light levels, your own tags), colour primaries and transfer (sRGB, Display P3, BT.2020, PQ, HLG, …) |
| **Integrity** | yapf-sum checksums on the header, tables and every tile; *needs* bits make an older decoder refuse cleanly what it cannot read |
| **Animation** | frames with per-frame durations and a loop count |
```sh
yapf albedo.png albedo.yapf --format bc7 --mips # GPU-ready colour
yapf normal.png normal.yapf --format bc5 --mips --normal dx # normal map, DirectX Y
yapf orm.png orm.yapf --map ao,rough,metal --linear --mips # packed masks
yapf foliage.png foliage.yapf --format bc7 --coverage 128 # alpha-tested mips
yapf sky.hdr sky.yapf --format bc6h --mips # HDR, no RGBM tricks
yapf sky.hdr sky_lossless.yapf # lossless half float
yapf pack cube px.png nx.png py.png ny.png pz.png nz.png probe.yapf --mips
yapf pack array grass.png rock.png sand.png terrain.yapf --format bc7 --mips
yapf pack volume lut_00.png lut_01.png ... grading.yapf --linear
yapf pack anim f0.png f1.png f2.png fx.yapf --frame-ms 50
yapf albedo.png albedo.yapf --entropy --author "Keel" --license CC-BY-4.0
yapf info albedo.yapf # everything in the header, notes, stream prefix sizes
yapf verify albedo.yapf # check every checksum
yapf probe.yapf face2.png --face 2 --mip 1 # export one surface
```
In C:
```c
int err;
yapf_texture_t *t = yapf_tex_load("albedo.yapf", 0, 0, &err);
for (uint32_t m = 0; m < t->mip_levels; m++) {
uint32_t s = yapf_tex_surface(t, 0, 0, 0, m), w, h;
yapf_tex_mip_size(t, m, &w, &h, NULL);
if (t->payload == YAPF_PAYLOAD_BLOCKS) /* e.g. YAPF_GPU_BC7 */
glCompressedTexImage2D(GL_TEXTURE_2D, m, gl_format(t->format, t->flags),
w, h, 0, (GLsizei)yapf_tex_surface_bytes(t, m), t->data[s]);
}
yapf_tex_free(t);
```
The language bindings and the JavaScript port speak version 1; the C
library's v1 functions also read 8-bit version 2 files (2D, cube face 0,
array layer 0), so plugins built on them open those too.
--- ---
@ -198,6 +262,47 @@ void yapf_free(yapf_image_t *img);
``` ```
Releases memory returned by the loaders. Passing `NULL` is safe. Releases memory returned by the loaders. Passing `NULL` is safe.
### Textures (version 2)
```c
yapf_texture_t *yapf_tex_load(const char *file, int flags, int threads, int *err);
yapf_texture_t *yapf_tex_load_memory(const void *buf, size_t size, int flags, int threads, int *err);
int yapf_tex_save(const char *file, const yapf_texture_t *t, int flags);
int yapf_tex_encode(const yapf_texture_t *t, int flags, void **out, size_t *size);
yapf_texture_t *yapf_tex_create(const yapf_texture_t *desc); /* zeroed storage */
void yapf_tex_free(yapf_texture_t *t);
```
A texture describes its shape (width, height, depth, layers, frames,
cube / volume, mips), storage (pixels with u8 / u16 / half / float samples,
or GPU blocks), meaning (channel map, linear mask, normal convention,
primaries, transfer, mip filter, alpha reference) and holds `data[s]` per
surface `s = yapf_tex_surface(t, frame, layer, face, mip)`. Load flag
`YAPF_LOAD_TRUST` skips checksum checks; encode flag `YAPF_ENC_ENTROPY`
allows the entropy stage.
```c
yapf_texture_t *yapf_tex_open(const void *buf, size_t avail, int flags, int *err);
uint64_t yapf_tex_stream_bytes(const yapf_texture_t *t, uint32_t first_mip);
int yapf_tex_decode_surface(yapf_texture_t *t, const void *buf, size_t avail, uint32_t s, int threads);
int yapf_tex_decode_tile(const yapf_texture_t *t, const void *buf, size_t avail, uint32_t s,
uint32_t tx, uint32_t ty, uint32_t slice, void *out, size_t stride,
uint32_t *x, uint32_t *y, uint32_t *w, uint32_t *h);
```
Streaming and virtual texturing: open a file from its first bytes, ask how
many bytes hold mip *m* and smaller, decode surfaces as their bytes arrive
(`YAPF_ERR_TRUNCATED` until then), or decode single tiles.
```c
int yapf_tex_build_mips(yapf_texture_t *t, uint32_t levels, uint8_t alpha_ref);
int yapf_tex_compress(const yapf_texture_t *src, uint8_t format, yapf_texture_t **out);
int yapf_blocks_decode(uint8_t format, const void *blocks, uint32_t w, uint32_t h, void *out);
int yapf_tex_add_note(yapf_texture_t *t, const char tag[4], const void *data, uint32_t size);
const yapf_note_t *yapf_tex_find_note(const yapf_texture_t *t, const char tag[4]);
```
Mip chains (linear-light sRGB, optional coverage preservation), the
built-in BCn encoders, BC1–BC7 decoding for previews, and notes. The
complete list, with sizes and helpers, is in [`yapf.h`](yapf.h).
### Return codes ### Return codes
| Constant | Value | Meaning | | Constant | Value | Meaning |
@ -206,6 +311,10 @@ Releases memory returned by the loaders. Passing `NULL` is safe.
| `YAPF_ERR_INVALID` | –1 | NULL or logically invalid argument | | `YAPF_ERR_INVALID` | –1 | NULL or logically invalid argument |
| `YAPF_ERR_IO` | –2 | File could not be opened or written | | `YAPF_ERR_IO` | –2 | File could not be opened or written |
| `YAPF_ERR_OOM` | –3 | Out of memory | | `YAPF_ERR_OOM` | –3 | Out of memory |
| `YAPF_ERR_CORRUPT` | –4 | Corrupt or not a YAPF file |
| `YAPF_ERR_UNSUPPORTED` | –5 | Needs a feature this decoder lacks |
| `YAPF_ERR_CHECKSUM` | –6 | A checksum does not match (v2) |
| `YAPF_ERR_TRUNCATED` | –7 | The data is not (yet) all there |
### Channel constants ### Channel constants
@ -256,7 +365,8 @@ domain [stb](https://github.com/nothings/stb) headers in `third_party/`.
7. **Independent tiles** — any tile can be decoded on its own, in any order, 7. **Independent tiles** — any tile can be decoded on its own, in any order,
on any thread. on any thread.
The complete specification is the comment at the top of [`yapf.h`](yapf.h): The version 1 specification is the comment at the top of [`yapf.h`](yapf.h);
version 2 (textures) is specified in [docs/FORMAT.md](docs/FORMAT.md). Version 1:
``` ```
HEADER (20 bytes) "YAPF", version 1, channels, GPU hint, mip count, HEADER (20 bytes) "YAPF", version 1, channels, GPU hint, mip count,
@ -273,7 +383,7 @@ TILE (64×64) header byte: stored raw, or constant-plane mask + colour tr
## Testing ## Testing
```sh ```sh
make test # C suite: 786 checks, plus JavaScript if node is installed make test # C suite: 2631 checks, plus JavaScript if node is installed
node bindings/js/test.js # JavaScript: round-trips, sample file, corrupt input node bindings/js/test.js # JavaScript: round-trips, sample file, corrupt input
``` ```
@ -286,6 +396,16 @@ What has been verified:
C++, and under AddressSanitizer, UndefinedBehaviorSanitizer and C++, and under AddressSanitizer, UndefinedBehaviorSanitizer and
ThreadSanitizer; ThreadSanitizer;
- thousands of corrupted and truncated files are rejected without crashing; - thousands of corrupted and truncated files are rejected without crashing;
- version 2: lossless round-trips of u8 / u16 / half / float pixels with 1–4
channels, with and without entropy coding; cubes, cube arrays, arrays,
volumes and animations with notes and every description field; GPU block
payloads; checksum, needs-bit and truncation handling; streaming from file
prefixes and single-tile decoding; BCn encoder quality; linear-light and
coverage-preserving mips; fuzzing of entropy-coded and packed-block files;
- the BC1–BC7 decoders were checked against an independent reference decoder
(bcdec) on 300,000 random blocks: BC6H and BC7 bit-exact, BC1–BC5 within
the ±1 rounding the formats allow (a development check, not part of
`make test`);
- every language binding runs its example; JavaScript, Python, C++, C#, - every language binding runs its example; JavaScript, Python, C++, C#,
Rust, Go, Java and Lua re-encode the sample to byte-identical output; Rust, Go, Java and Lua re-encode the sample to byte-identical output;
- the GIMP plugin in GIMP 3.0.8 and the Blender extension in Blender 5.1; - the GIMP plugin in GIMP 3.0.8 and the Blender extension in Blender 5.1;
@ -293,14 +413,20 @@ What has been verified:
**Not yet verified:** running on a Mac (macOS builds are cross-compiled **Not yet verified:** running on a Mac (macOS builds are cross-compiled
only), Windows on ARM, and running inside Photoshop, VS Code, Figma or a only), Windows on ARM, and running inside Photoshop, VS Code, Figma or a
live Dolphin / GNOME file manager. live Dolphin / GNOME file manager. The version 2 code has been run on
Linux x64 (with and without SIMD, as C and C++, under ASan, UBSan and
TSan); its NEON path is compiled for ARM64 but has not been run yet, and
the Windows builds have not been rerun. ETC2 / EAC / ASTC blocks are
stored and loaded, but there is no built-in encoder or preview decoder
for them.
--- ---
## Project layout ## Project layout
``` ```
yapf.h, yapf.c the library (format spec in yapf.h) yapf.h, yapf.c the library (v1 spec in yapf.h)
docs/FORMAT.md the full format specification, version 2
tools/yapf_cli.c the `yapf` command-line tool tools/yapf_cli.c the `yapf` command-line tool
tools/build_release.sh builds every download for all platforms tools/build_release.sh builds every download for all platforms
tests/test_yapf.c C test suite tests/test_yapf.c C test suite

450
docs/FORMAT.md Normal file
View file

@ -0,0 +1,450 @@
# YAPF file format
This is the complete specification of YAPF files. `yapf.c` is the
reference implementation; where this text and the code disagree, that is a
bug in one of them.
YAPF has two versions:
| Version | What it holds | Who writes it |
| :------ | :------------ | :------------ |
| **1** | one 8-bit image (1–4 channels) with optional mips | `yapf_save` / `yapf_encode`, and the CLI for plain 8-bit images |
| **2** | a texture: GPU blocks or lossless pixels in 8/16-bit or float, cubes, arrays, volumes, animation, channel meanings, colour description, notes, checksums, optional entropy coding | `yapf_tex_save` / `yapf_tex_encode`, and the CLI whenever a v2 feature is used |
Version 1 is specified at the top of [`yapf.h`](../yapf.h) and is not
repeated here, except that version 2 reuses its tiles, filters and plane
payloads (the "group form") byte for byte.
---
## Design: how YAPF 2 does things its own way
YAPF 2 is not a wrapper around someone else's container. A few ideas
carry the whole format:
1. **Everything is tiles.** Pixels are cut into 64 × 64 tiles, GPU
blocks into tiles of 16 × 16 blocks. Each tile has its own size and
checksum in a table, so any tile can be found, checked and decoded on its
own: for multithreaded loading, for partial updates, and for virtual
texturing / "megatexture" streaming.
2. **Small mips first.** Surface data is stored from the smallest mip
to the largest. The first *N* bytes of a file are a complete, lower
resolution texture, so an engine can show something after reading a
few kilobytes and stream in the large mips later. The header carries
the file size, so a loader knows how much is still to come.
3. **Needs, not versions.** New capabilities are announced by *needs*
bits. A decoder refuses a file with a needs bit it does not know, and
can ignore everything else safely. No more version bumps that break
every reader.
4. **Wide samples are byte planes.** 16-bit and float samples are split
into byte planes (high bytes, low bytes) and coded with the same fast
8-bit machinery as version 1: smooth high bytes compress very well, and
the decoder stays branch-light and SIMD friendly.
5. **Shared code books.** The optional entropy stage keeps its Huffman
tables once per surface, keyed by a small neighbourhood context, so
tiles stay independent and their own headers stay tiny. For GPU
blocks the tables are per byte lane, so endpoint bytes and index bytes
are coded separately.
6. **Meaning travels with the data.** Channel meanings, the normal-map
convention, linear channels inside sRGB images, colour primaries and
transfer, and how the mips were made are header fields, not file-name
suffixes.
---
## Conventions
All integers are little-endian. `u8`, `u16`, `u32` and `u64` are unsigned
integers of that many bits. Offsets are from the start of the file.
### yapf-sum
The checksum of a byte string of length *n*. `K = 0x9E3779B97F4A7C15`;
all arithmetic is modulo 2⁶⁴.
```
lane[i] = (i + 1)·K XOR n for i = 0, 1, 2, 3
for each 8-byte word w_j of the input (little-endian; the last word is
padded with zero bytes):
a = lane[j mod 4]
a = (a XOR w_j) · K
a = a XOR (a >> 32)
lane[j mod 4] = a
h = lane[0] XOR rotl(lane[1], 16) XOR rotl(lane[2], 32) XOR rotl(lane[3], 48)
h = h · K
h = h XOR (h >> 32)
result = low 32 bits of h
```
Test vectors: `""` → `0xd7550ed0`, `"YAPF"` → `0xc8cc41d4`,
`"The quick brown fox jumps over the lazy dog"` → `0xbce4f410`.
---
## Version 2 layout
```
HEADER 64 bytes
SURFACE TABLE 16 bytes per surface
FRAME TABLE 4 bytes per frame, only when frames > 1
NOTES notes size bytes
SURFACE DATA in stream order: smallest mip first
```
The *front matter* is everything before the surface data:
`64 + 16·surfaces + (frames > 1 ? 4·frames : 0) + notes size` bytes.
### Header
| Offset | Size | Field | Notes |
| -----: | ---: | :---- | :---- |
| 0x00 | 4 | magic | `"YAPF"` |
| 0x04 | 1 | version | 2 |
| 0x05 | 1 | channels | 1–4 |
| 0x06 | 1 | format | block format (block payload) or GPU hint (pixel payload); see [GPU formats](#gpu-formats) |
| 0x07 | 1 | mip levels | 1–16 |
| 0x08 | 1 | flags | bit 0 premultiplied alpha, bit 1 sRGB, bit 2 coverage-preserving mips; other bits 0 |
| 0x09 | 1 | payload | 0 pixels (lossless tiles), 1 GPU blocks |
| 0x0A | 1 | sample | pixel payload: 0 u8, 1 u16, 2 half float, 3 float. Block payload: type of the decoded texels (0, or 2 for BC6H) |
| 0x0B | 1 | shape | 0 2D, 1 cube, 2 volume |
| 0x0C | 4 | width | 1–65535 |
| 0x10 | 4 | height | 1–65535; cubes: width = height |
| 0x14 | 4 | depth | volumes: 1–65535; otherwise 1 |
| 0x18 | 2 | layers | ≥ 1 (array layers; with a cube, a cube array); volumes: 1 |
| 0x1A | 2 | frames | ≥ 1 (animation frames) |
| 0x1C | 4 | needs | see [Needs](#needs) |
| 0x20 | 4 | channel map | meaning of channels 0–3, see [Channel meanings](#channel-meanings) |
| 0x24 | 1 | linear mask | bit *c*: channel *c* is linear even when the sRGB flag is set |
| 0x25 | 1 | normal | bits 0–1 convention (0 none, 1 OpenGL +Y, 2 DirectX −Y); bits 2–3 encoding (0 XYZ, 1 XY with Z reconstructed, 2 octahedral) |
| 0x26 | 1 | primaries | 0 unspecified, 1 BT.709/sRGB, 2 Display P3, 3 BT.2020, 4 ACES AP0, 5 ACEScg (AP1), 6 Adobe RGB |
| 0x27 | 1 | transfer | 0 unspecified (the sRGB flag decides), 1 sRGB, 2 linear, 3 gamma 2.2, 4 PQ (ST 2084), 5 HLG, 6 BT.1886 |
| 0x28 | 1 | mip filter | bits 0–6: 0 unknown, 1 box, 2 tent, 3 Kaiser, 4 Lanczos, 5 Mitchell, 6 authored; bit 7: sRGB channels were filtered in linear light |
| 0x29 | 1 | alpha ref | with the coverage flag: alpha-test reference 0–255 |
| 0x2A | 2 | loop count | animation repeats, 0 = forever |
| 0x2C | 4 | notes size | bytes of the notes area |
| 0x30 | 8 | file size | total bytes of the file |
| 0x38 | 4 | tables check | yapf-sum of bytes [0x40, front matter end) |
| 0x3C | 4 | header check | yapf-sum of bytes [0x00, 0x3C) |
The header check covers the tables check, so the whole front matter is
protected.
### Surfaces
A *surface* is one mip level of one face of one layer of one frame. With
`faces = 6` for cubes (order +X, −X, +Y, −Y, +Z, −Z) and 1 otherwise,
surface number
```
s = ((frame · layers + layer) · faces + face) · mip_levels + mip
```
Mip *m* is `max(1, width >> m) × max(1, height >> m)`, and for volumes
`max(1, depth >> m)` slices deep. A volume surface holds all its slices.
**Surface table entry** (16 bytes, one per surface, in surface order):
| Offset | Size | Field |
| -----: | ---: | :---- |
| +0 | 8 | offset of the surface data |
| +8 | 4 | size of the surface data |
| +12 | 4 | tile-table check: yapf-sum of the surface's book size, book and tile table |
**Stream order.** Writers store surface data for mip `mip_levels − 1`
first (all frames, layers and faces in surface order), then the next larger
mip, and so on, ending with mip 0. Readers must use the offsets and must not
rely on the order. The number of leading bytes that hold every surface of
mip *m* and smaller is the largest `offset + size` over those surfaces.
### Frame table
When `frames > 1`: one `u32` per frame, its display time in microseconds.
### Notes
A sequence of entries that exactly fills the notes area:
```
tag 4 bytes
length u32
data length bytes
```
| Tag | Content |
| :-- | :------ |
| `AUTH` | author (UTF-8) |
| `LICN` | licence (UTF-8, e.g. an SPDX id) |
| `SRCE` | source / origin (UTF-8, e.g. a URL or path) |
| `TOOL` | the tool that wrote the file (UTF-8) |
| `TITL` | title (UTF-8) |
| `DESC` | description (UTF-8) |
| `DATE` | date, ISO 8601 (UTF-8) |
| `ICCP` | an ICC colour profile |
| `ORNT` | orientation, 1 byte: bits 0–1 clockwise quarter turns, bit 2 mirror left–right after turning |
| `DPI ` | 2 × u32: horizontal and vertical dots per inch × 1000 |
| `CLL ` | 2 × u16: maximum content light level, maximum frame-average light level (cd/m²) |
Tags that begin with a lowercase letter belong to applications (for
example `keel`). Readers ignore tags they do not know. Tags may repeat;
order is kept.
### Needs
| Bit | Name | Set when |
| --: | :--- | :------- |
| 0 | WIDE | pixel payload with sample ≠ u8 |
| 1 | BLOCKS | block payload |
| 2 | ENTROPY | any surface has a code book |
| 3 | LAYOUT | shape ≠ 2D, or layers > 1 |
| 4 | FRAMES | frames > 1 |
A decoder must refuse a file that sets a needs bit it does not implement
(this library: any bit above 4). A file that uses a feature without its
bit is corrupt. Everything that is not a needs feature — channel map,
colour fields, mip filter, notes, frame times — may be ignored.
A decoder that only implements 8-bit 2D pixels can therefore read every
version 2 file whose needs are 0, and cleanly refuse the rest.
---
## Surface data
```
book size u32 0 when the surface has no code book
book book size bytes
tile table 8 bytes per tile: u32 size (bit 31: packed), u32 yapf-sum of the tile bytes
tiles back to back; their sizes must add up exactly to the rest of the surface
```
Tiles are numbered slice by slice, rows top to bottom, tiles left to right.
For pixels the grid is `ceil(w / 64) × ceil(h / 64)` tiles per slice (edge
tiles are smaller, no padding); for blocks it is
`ceil(bx / 16) × ceil(by / 16)` tiles of 16 × 16 blocks, where
`bx = ceil(w / block width)` and `by = ceil(h / block height)`.
Bit 31 of a tile size (*packed*) is allowed only in block tiles of a
surface with a code book.
---
## Pixel tiles
Let *B* be the bytes per sample (1, 2 or 4), *ch* the channels, and the
tile `tw × th` pixels.
### Tile header
```
byte 0 0x80 stored: the raw samples follow, interleaved in
channel order, each B bytes little-endian;
exactly tw·th·ch·B bytes
otherwise coded:
bits 0–3 B = 1: bit c set = plane c is constant; B > 1: 0
bit 4 colour transform subtract-green
bit 5 an entropy mask follows
bit 6 no colour transform
bit 7 0
bits 4 and 6 are exclusive and need ch >= 3; with neither
set, an RGB(A) tile uses YCoCg-R
B > 1 u16 constant mask, bit p = plane p is constant
bit 5 entropy mask: u8 (B = 1) or u16 (B > 1); bit p = plane p is
entropy coded. It must not include constant planes.
then one value byte per constant plane, in plane order;
one payload per other plane, in plane order;
the tile ends exactly where its last payload ends
```
With B = 1 and neither bit 5 nor bit 6 set, a coded tile is exactly a
version 1 tile.
### Planes
The colour transform turns R, G, B (+ A) into transformed channels; it is
skipped for 1–2 channels and with bit 6.
* **YCoCg-R** at the sample width *n* = 8·B bits, all arithmetic mod 2ⁿ,
`>>` an arithmetic shift of the *n*-bit two's complement value, `h = 2ⁿ⁻¹`:
```
forward: o = R − B; t = B + (o >> 1); q = G − t;
Y = t + (q >> 1); Co = o + h; Cg = q + h
inverse: o = Co − h; q = Cg − h; t = Y − (q >> 1);
G = q + t; B = t − (o >> 1); R = B + o
```
* **subtract-green**: `G, R − G + h, B − G + h` (+ A), mod 2ⁿ.
Before the transform, float samples (half or float) are mapped to
integers that keep their order: if the sign bit is set every bit is
flipped, otherwise the sign bit is set. The decoder undoes this after the
inverse transform.
Transformed channel *c* gives B planes: plane `c·B + k` holds byte
`B − 1 − k` of the value (the most significant byte first). For B = 1
the planes are the transformed channels.
### Plane payloads
A plane payload has one of two forms, chosen per plane by the entropy mask.
**Group form** — exactly the version 1 plane payload: a filter byte per
band of 8 rows, 4-bit control codes, and groups of 8 residuals at the
group's bit width. See `yapf.h`.
**Entropy form**
```
filter[ceil(th / 8)] as in the group form
u16 L bytes of the bit stream
L bytes the folded residuals of the plane in raster order
```
Each residual is coded with table `(slot, context)` of the surface's code
book:
* `slot = t · 16 + p`, where *p* is the plane and *t* is 1 with
subtract-green, 2 with no transform, else 0;
* `context = min(6, floor(log2(s + 1)))`, `s = left + above`, the folded
residuals of the left and upper neighbours in this plane (0 outside the
tile) — so 0, 1–2, 3–6, 7–14, 15–30, 31–62, 63 and up.
The stream must end in its last byte. Folded residuals then go through
the filters exactly as in the group form.
---
## Block tiles
A block tile of `tbw × tbh` blocks of *S* bytes is either
* **raw**: the blocks, rows top to bottom, exactly `tbw·tbh·S` bytes — as
the GPU takes them; or
* **packed** (bit 31 of its size): one bit stream holding byte 0 of every
block (in raster order), then byte 1 of every block, …, byte S − 1.
Byte lane *k* is coded with table `(slot k, context 0)` of the surface's
code book. The stream must end in the last byte of the tile.
Decoded block surfaces are row-major blocks, slice after slice, ready for
`glCompressedTexImage*` / `vkCmdCopyBufferToImage`.
---
## Code book
```
u16 count number of tables, 1–336
count × {
u8 slot 0–47
u8 context 0–6
code table
}
```
Each `(slot, context)` pair appears at most once. The book must fill its
size exactly.
**Code table**: `u8 n − 1`, then *n* 4-bit code lengths (symbols
0 … n − 1, low nibble first, an odd count padded with a zero nibble).
Length 0 = no code; otherwise 1–11. At least one code; Kraft sum ≤ 1.
Codes are canonical — by length, then by symbol, the first code of each
length one more than the last code of the previous length, shifted left —
and stored least significant bit first (the first bit of a code is the
lowest unread bit of the stream), so a decoder can look up 11 bits at a
time.
---
## GPU formats
Pixel payload hints (`format` field):
| Value | Name | Vulkan |
| ----: | :--- | :----- |
| 0x00 | RGBA8 | R8G8B8A8_UNORM |
| 0x01 | RGB8 | R8G8B8_UNORM |
| 0x02 | RG8 | R8G8_UNORM |
| 0x03 | R8 | R8_UNORM |
| 0x04 | SRGB8_A8 | R8G8B8A8_SRGB |
| 0x05 | SRGB8 | R8G8B8_SRGB |
| 0x06–0x09 | RGBA16, RGB16, RG16, R16 | R16G16B16A16_UNORM … R16_UNORM |
| 0x0A–0x0D | RGBA16F, RGB16F, RG16F, R16F | R16G16B16A16_SFLOAT … R16_SFLOAT |
| 0x0E–0x11 | RGBA32F, RGB32F, RG32F, R32F | R32G32B32A32_SFLOAT … R32_SFLOAT |
Block formats (block payload). With the sRGB flag, use the `_SRGB`
variant where one exists.
| Value | Name | Block | Bytes | Vulkan / DXGI |
| ----: | :--- | :---: | ----: | :------------ |
| 0x40 | BC1 | 4×4 | 8 | BC1_RGB(A)_UNORM / BC1_UNORM |
| 0x41 | BC2 | 4×4 | 16 | BC2_UNORM |
| 0x42 | BC3 | 4×4 | 16 | BC3_UNORM |
| 0x43 | BC4 | 4×4 | 8 | BC4_UNORM |
| 0x44 | BC4_SNORM | 4×4 | 8 | BC4_SNORM |
| 0x45 | BC5 | 4×4 | 16 | BC5_UNORM |
| 0x46 | BC5_SNORM | 4×4 | 16 | BC5_SNORM |
| 0x47 | BC6H | 4×4 | 16 | BC6H_UFLOAT / BC6H_UF16 |
| 0x48 | BC6H_SF | 4×4 | 16 | BC6H_SFLOAT / BC6H_SF16 |
| 0x49 | BC7 | 4×4 | 16 | BC7_UNORM |
| 0x50 | ETC2_RGB8 | 4×4 | 8 | ETC2_R8G8B8_UNORM |
| 0x51 | ETC2_RGB8A1 | 4×4 | 8 | ETC2_R8G8B8A1_UNORM |
| 0x52 | ETC2_RGBA8 | 4×4 | 16 | ETC2_R8G8B8A8_UNORM |
| 0x53 | EAC_R11 | 4×4 | 8 | EAC_R11_UNORM |
| 0x54 | EAC_R11_SNORM | 4×4 | 8 | EAC_R11_SNORM |
| 0x55 | EAC_RG11 | 4×4 | 16 | EAC_R11G11_UNORM |
| 0x56 | EAC_RG11_SNORM | 4×4 | 16 | EAC_R11G11_SNORM |
| 0x60–0x6D | ASTC 4×4, 5×4, 5×5, 6×5, 6×6, 8×5, 8×6, 8×8, 10×5, 10×6, 10×8, 10×10, 12×10, 12×12 | as named | 16 | ASTC_*_UNORM (HDR: transfer linear, sample half) |
---
## Channel meanings
| Value | Meaning | Value | Meaning |
| ----: | :------ | ----: | :------ |
| 0 | default for the channel count (Y / YA / RGB / RGBA) | 11 | metalness |
| 1 | red | 12 | ambient occlusion |
| 2 | green | 13 | height |
| 3 | blue | 14 | specular |
| 4 | alpha | 15 | emissive |
| 5 | luma | 16 | opacity (alpha-test mask) |
| 6 | normal X | 17 | mask |
| 7 | normal Y | 18 | cavity |
| 8 | normal Z | 19 | thickness |
| 9 | roughness | 20 | id |
| 10 | gloss (1 − roughness) | 255 | unused |
Alpha (an explicit 4 or 16, or the last channel of a 2- or 4-channel image
with default meanings) is always linear. The linear mask marks further
channels that are linear inside an sRGB image, such as masks packed next to
colour.
---
## Mips
`mip filter` records how the mips were made. This library's
`yapf_tex_build_mips` uses a 2 × 2 box (2 × 2 × 2 for volumes, edges
repeat), averages sRGB 8-bit channels in linear light (bit 7), and with an
alpha reference rescales alpha on every mip so that the share of texels
with alpha *above* the reference stays that of mip 0 (coverage flag,
`alpha ref`). Alpha-tested foliage and fences then keep their density at
a distance instead of thinning out.
---
## Validation summary
A decoder rejects a file when:
* the magic, version, a size, shape or count is out of range, or a cube is
not square;
* the header or tables check does not match (unless the reader was told to
trust the file);
* a needs bit is unknown (*unsupported*), or a feature is used without its
needs bit (*corrupt*);
* a surface lies outside the file, its book is malformed or present without
ENTROPY, or its tiles do not fill it exactly;
* a tile's yapf-sum does not match, or the tile does not decode to exactly
its size (reserved bits, invalid filters, codes, masks or lengths).
Because every structure is validated before use, corrupt or hostile input
is rejected, never read out of bounds.

8
docs/LANGUAGES.md
View file

@ -22,6 +22,14 @@ the same native code, so they are all about equally fast; the JavaScript
port needs no native library at all. Every example also re-encodes the port needs no native library at all. Every example also re-encodes the
image and checks that the result is byte-identical to the original file. image and checks that the result is byte-identical to the original file.
**YAPF 2 textures.** The bindings wrap the version 1 API (`yapf_load`,
`yapf_save`, …). Through it they also read 8-bit version 2 files (2D, or
face 0 / layer 0 of cubes and arrays). GPU blocks, 16-bit / float pixels,
volumes and the other version 2 features are available from C and C++
through the `yapf_tex_*` functions in [`yapf.h`](../yapf.h) (see
[FORMAT.md](FORMAT.md)); any language with a C FFI can call them the same
way. The JavaScript port reads and writes version 1 only.
## Getting the native library ## Getting the native library
Bindings other than C, C++, Rust, Go, Zig and JavaScript load the shared Bindings other than C, C++, Rust, Go, Zig and JavaScript load the shared

501
tests/test_yapf.c
View file

@ -7,6 +7,8 @@
* Round-trips many sizes, channel counts and image kinds, checks mip * Round-trips many sizes, channel counts and image kinds, checks mip
* chains, single- vs multi-threaded decode, the bundled sample file, and * chains, single- vs multi-threaded decode, the bundled sample file, and
* that corrupted or truncated files are rejected without crashing. * that corrupted or truncated files are rejected without crashing.
* Version 2: wide samples, shapes, GPU blocks, notes, checksums, needs,
* streaming, tile decode, block encoders and mip generation.
* Exit code 0 means every test passed. * Exit code 0 means every test passed.
*/ */
#include "../yapf.h" #include "../yapf.h"
@ -14,6 +16,7 @@
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <math.h>
/* FNV-1a of the decoded pixels of other/YAPF.YAPF (the YAPF logo). */ /* FNV-1a of the decoded pixels of other/YAPF.YAPF (the YAPF logo). */
#define SAMPLE_HASH 0x8891c2b4u #define SAMPLE_HASH 0x8891c2b4u
@ -171,6 +174,494 @@ static uint32_t fnv(const uint8_t *p, size_t n) {
return h; return h;
} }
/* ======================================================================
* Version 2
* ====================================================================== */
static yapf_texture_t desc(uint32_t w, uint32_t h, int ch, int sample) {
yapf_texture_t d;
memset(&d, 0, sizeof(d));
d.width = w; d.height = h; d.depth = 1; d.layers = 1; d.frames = 1;
d.mip_levels = 1; d.channels = (uint8_t)ch; d.sample = (uint8_t)sample;
return d;
}
/* Fills every surface: noise, gradient or noisy curves, in the sample
* type; block payloads get bytes with some structure. */
static void tex_fill(yapf_texture_t *t, int kind) {
uint32_t S = yapf_tex_surface_count(t);
for (uint32_t s = 0; s < S; s++) {
uint32_t m = s % t->mip_levels, w, h, d;
yapf_tex_mip_size(t, m, &w, &h, &d);
uint8_t *p = (uint8_t *)t->data[s];
if (t->payload == YAPF_PAYLOAD_BLOCKS) {
size_t n = yapf_tex_surface_bytes(t, m);
for (size_t i = 0; i < n; i++)
p[i] = kind ? (uint8_t)(i % 16 < 8 ? (i / 64) & 0xFF : rnd()) : (uint8_t)rnd();
continue;
}
size_t cnt = (size_t)w * h * d * t->channels;
for (size_t i = 0; i < cnt; i++) {
size_t px = i / t->channels;
uint32_t x = (uint32_t)(px % w), y = (uint32_t)((px / w) % h);
int c = (int)(i % t->channels);
double v = kind == 0 ? (rnd() % 1000) / 1000.0
: kind == 1 ? (x * 3 + y * 5 + c * 40) % 600 / 600.0
: ((x * x + y * y) / 50 % 700 + rnd() % 5) / 705.0;
switch (t->sample) {
case YAPF_SAMPLE_U8: p[i] = (uint8_t)(v * 255); break;
case YAPF_SAMPLE_U16: ((uint16_t *)p)[i] = (uint16_t)(v * 65535); break;
case YAPF_SAMPLE_F16: ((uint16_t *)p)[i] = yapf_float_to_half((float)(v * 20 - 2)); break;
default: ((float *)p)[i] = (float)(v * 100 - 10) + (kind == 0 ? (float)rnd() : 0.0f); break;
}
}
}
}
static int tex_equal(const yapf_texture_t *a, const yapf_texture_t *b) {
uint32_t S = yapf_tex_surface_count(a);
if (S != yapf_tex_surface_count(b)) return 0;
for (uint32_t s = 0; s < S; s++)
if (!a->data[s] || !b->data[s] ||
memcmp(a->data[s], b->data[s], yapf_tex_surface_bytes(a, s % a->mip_levels)) != 0)
return 0;
return 1;
}
/* Encode, decode, compare; returns the encoded file (caller frees). */
static uint8_t *tex_roundtrip(yapf_texture_t *t, int flags, size_t *len, const char *what) {
void *buf = NULL;
*len = 0;
int e = yapf_tex_encode(t, flags, &buf, len);
CHECK(e == YAPF_OK, "v2 encode %s (%d)", what, e);
if (e != YAPF_OK) return NULL;
int err = 0;
yapf_texture_t *a = yapf_tex_load_memory(buf, *len, 0, 1, &err);
yapf_texture_t *b = yapf_tex_load_memory(buf, *len, 0, 0, &err);
CHECK(a && tex_equal(a, t), "v2 roundtrip %s", what);
CHECK(b && tex_equal(b, t), "v2 roundtrip (threads) %s", what);
yapf_tex_free(a);
yapf_tex_free(b);
uint8_t *copy = (uint8_t *)malloc(*len);
memcpy(copy, buf, *len);
yapf_free_buffer(buf);
return copy;
}
static void v2_pixels(void) {
static const int sizes[][2] = { {1, 1}, {7, 3}, {64, 64}, {65, 63}, {130, 70}, {200, 201} };
static const char *sn[] = { "u8", "u16", "f16", "f32" };
for (size_t si = 0; si < sizeof(sizes) / sizeof(sizes[0]); si++)
for (int ch = 1; ch <= 4; ch++)
for (int sample = 0; sample < 4; sample++)
for (int kind = 0; kind < 3; kind++)
for (int fl = 0; fl < 2; fl++) {
yapf_texture_t d = desc((uint32_t)sizes[si][0], (uint32_t)sizes[si][1], ch, sample);
yapf_texture_t *t = yapf_tex_create(&d);
char what[96];
sprintf(what, "%dx%d ch%d %s kind%d%s", sizes[si][0], sizes[si][1], ch,
sn[sample], kind, fl ? " entropy" : "");
tex_fill(t, kind);
size_t len;
free(tex_roundtrip(t, fl ? YAPF_ENC_ENTROPY : 0, &len, what));
yapf_tex_free(t);
}
}
/* Shapes, frames, notes and every description field survive. */
static void v2_layout(void) {
struct { int w, h, ch, sample, shape, layers, frames, mips, depth; } c[] = {
{ 64, 64, 4, 0, YAPF_SHAPE_CUBE, 1, 1, 7, 1 },
{ 32, 32, 3, 2, YAPF_SHAPE_CUBE, 2, 1, 6, 1 },
{ 40, 30, 4, 0, YAPF_SHAPE_2D, 5, 1, 3, 1 },
{ 33, 17, 1, 1, YAPF_SHAPE_VOLUME, 1, 1, 4, 9 },
{ 50, 50, 4, 0, YAPF_SHAPE_2D, 1, 6, 1, 1 },
};
for (size_t i = 0; i < sizeof(c) / sizeof(c[0]); i++) {
yapf_texture_t d = desc((uint32_t)c[i].w, (uint32_t)c[i].h, c[i].ch, c[i].sample);
d.shape = (uint8_t)c[i].shape; d.layers = (uint16_t)c[i].layers; d.frames = (uint16_t)c[i].frames;
d.mip_levels = (uint8_t)c[i].mips; d.depth = (uint32_t)c[i].depth;
d.channel_map[0] = YAPF_CH_NORMAL_X; d.channel_map[1] = YAPF_CH_ROUGHNESS;
d.linear_mask = 0x08; d.normal = YAPF_NORMAL_DX | YAPF_NORMAL_XY;
d.primaries = YAPF_PRIMARIES_DISPLAY_P3; d.transfer = YAPF_TRANSFER_PQ;
d.mip_filter = YAPF_MIP_KAISER; d.alpha_ref = 77; d.loop_count = 3;
d.flags = YAPF_FLAG_SRGB | YAPF_FLAG_COVERAGE;
yapf_texture_t *t = yapf_tex_create(&d);
tex_fill(t, 2);
if (t->frames > 1) for (int f = 0; f < t->frames; f++) t->frame_us[f] = 16667u * (uint32_t)(f + 1);
yapf_tex_add_note(t, "AUTH", "Keel", 4);
yapf_tex_add_note(t, "ICCP", "\0\1\2", 3);
size_t len;
char what[32];
sprintf(what, "layout case %d", (int)i);
uint8_t *file = tex_roundtrip(t, (int)(i & 1), &len, what);
yapf_texture_t *b = file ? yapf_tex_load_memory(file, len, 0, 1, NULL) : NULL;
int ok = b && b->shape == d.shape && b->layers == d.layers && b->frames == d.frames &&
b->depth == d.depth && b->mip_levels == d.mip_levels &&
memcmp(b->channel_map, d.channel_map, 4) == 0 && b->linear_mask == 0x08 &&
b->normal == d.normal && b->primaries == d.primaries && b->transfer == d.transfer &&
b->mip_filter == d.mip_filter && b->alpha_ref == 77 && b->loop_count == 3 &&
b->flags == d.flags && b->note_count == 2;
const yapf_note_t *n = b ? yapf_tex_find_note(b, "ICCP") : NULL;
ok = ok && n && n->size == 3 && memcmp(n->data, "\0\1\2", 3) == 0;
if (ok && b->frames > 1) ok = b->frame_us[b->frames - 1] == 16667u * b->frames;
CHECK(ok, "v2 fields, notes and frame times survive (%s)", what);
yapf_tex_free(b);
free(file);
yapf_tex_free(t);
}
}
static void v2_blocks(void) {
static const uint8_t fmts[] = { YAPF_GPU_BC1, YAPF_GPU_BC7, YAPF_GPU_BC4, YAPF_GPU_ASTC_6x6,
YAPF_GPU_ASTC_12x12, YAPF_GPU_ETC2_RGBA8 };
for (size_t f = 0; f < sizeof(fmts); f++)
for (int fl = 0; fl < 2; fl++) {
yapf_texture_t d = desc(300, 200, 4, 0);
d.payload = YAPF_PAYLOAD_BLOCKS; d.format = fmts[f]; d.mip_levels = 3;
if (f == 3) { d.shape = YAPF_SHAPE_VOLUME; d.depth = 5; }
yapf_texture_t *t = yapf_tex_create(&d);
tex_fill(t, 1);
char what[48];
sprintf(what, "blocks 0x%02x%s", fmts[f], fl ? " packed" : "");
size_t len;
free(tex_roundtrip(t, fl, &len, what));
yapf_tex_free(t);
}
}
/* Recomputes both header checks after editing a file. */
static void reseal(uint8_t *f, size_t front) {
uint32_t a = yapf_checksum(f + 64, front - 64), b;
f[0x38] = (uint8_t)a; f[0x39] = (uint8_t)(a >> 8); f[0x3A] = (uint8_t)(a >> 16); f[0x3B] = (uint8_t)(a >> 24);
b = yapf_checksum(f, 0x3C);
f[0x3C] = (uint8_t)b; f[0x3D] = (uint8_t)(b >> 8); f[0x3E] = (uint8_t)(b >> 16); f[0x3F] = (uint8_t)(b >> 24);
}
/* Flips, truncation, scribbles: never crash; truncation always fails. */
static void v2_fuzz(const uint8_t *file, size_t len, const char *what) {
uint8_t *bad = (uint8_t *)malloc(len);
int truncated_ok = 1, err;
for (int it = 0; it < 2000; it++) {
memcpy(bad, file, len);
size_t n = len;
switch (it % 3) {
case 0: for (int k = 0; k < 1 + it % 7; k++) bad[rnd() % len] ^= (uint8_t)(1u << (rnd() % 8)); break;
case 1: n = rnd() % len; break;
default: bad[rnd() % len] = (uint8_t)rnd(); break;
}
int fl = (it & 4) ? YAPF_LOAD_TRUST : 0;
yapf_texture_t *x = yapf_tex_load_memory(bad, n, fl, (it & 1) ? 2 : 1, &err);
if (it % 3 == 1 && x) truncated_ok = 0;
yapf_tex_free(x);
if (it % 50 == 0) yapf_free(yapf_load_memory(bad, n));
}
CHECK(truncated_ok, "truncated v2 files are rejected (%s)", what);
free(bad);
}
static void v2_integrity(void) {
yapf_texture_t d = desc(150, 90, 4, YAPF_SAMPLE_F16);
yapf_texture_t *t = yapf_tex_create(&d);
tex_fill(t, 2);
size_t len;
uint8_t *file = tex_roundtrip(t, YAPF_ENC_ENTROPY, &len, "integrity setup");
if (!file) { yapf_tex_free(t); return; }
uint8_t *bad = (uint8_t *)malloc(len);
size_t front = 64 + 16; /* header + one surface entry */
int err;
/* a flipped bit in tile data is caught by its yapf-sum ... */
memcpy(bad, file, len);
bad[len - 10] ^= 0x10;
CHECK(!yapf_tex_load_memory(bad, len, 0, 1, &err) && err == YAPF_ERR_CHECKSUM,
"corrupt tile data reports YAPF_ERR_CHECKSUM (%d)", err);
yapf_texture_t *trusted = yapf_tex_load_memory(bad, len, YAPF_LOAD_TRUST, 1, &err);
yapf_tex_free(trusted); /* may or may not decode; must not crash */
/* ... and in the header by the header check */
memcpy(bad, file, len);
bad[0x0C] ^= 1;
CHECK(!yapf_tex_load_memory(bad, len, 0, 1, &err) && err == YAPF_ERR_CHECKSUM,
"corrupt header reports YAPF_ERR_CHECKSUM");
/* an unknown needs bit: refused as unsupported */
memcpy(bad, file, len);
bad[0x1F] |= 0x80;
reseal(bad, front);
CHECK(!yapf_tex_load_memory(bad, len, 0, 1, &err) && err == YAPF_ERR_UNSUPPORTED,
"unknown needs bit is refused (%d)", err);
/* a feature in use but not announced: corrupt */
memcpy(bad, file, len);
bad[0x1C] &= (uint8_t)~YAPF_NEED_WIDE;
reseal(bad, front);
CHECK(!yapf_tex_load_memory(bad, len, 0, 1, &err) && err == YAPF_ERR_CORRUPT,
"unannounced feature is refused (%d)", err);
/* the old API refuses what it cannot represent */
CHECK(yapf_load_memory(file, len) == NULL, "yapf_load refuses half-float textures");
free(bad);
v2_fuzz(file, len, "half float + entropy");
free(file);
yapf_tex_free(t);
/* 8-bit RGBA with entropy planes (decoded several streams at once) */
d = desc(140, 75, 4, 0);
t = yapf_tex_create(&d);
tex_fill(t, 2);
file = tex_roundtrip(t, YAPF_ENC_ENTROPY, &len, "fuzz setup rgba8");
if (file) v2_fuzz(file, len, "rgba8 + entropy");
free(file);
/* BC7 blocks with packed tiles */
yapf_texture_t *c = NULL;
if (yapf_tex_compress(t, YAPF_GPU_BC7, &c) == YAPF_OK) {
file = tex_roundtrip(c, YAPF_ENC_ENTROPY, &len, "fuzz setup bc7");
if (file) v2_fuzz(file, len, "packed BC7");
free(file);
}
yapf_tex_free(c);
yapf_tex_free(t);
}
/* Streaming: a file prefix holds the small mips; tiles decode alone. */
static void v2_streaming(void) {
yapf_texture_t d = desc(300, 260, 4, 0);
d.shape = YAPF_SHAPE_CUBE; d.width = d.height = 260; d.mip_levels = 5;
yapf_texture_t *t = yapf_tex_create(&d);
tex_fill(t, 2);
void *buf = NULL;
size_t len = 0;
yapf_tex_encode(t, 0, &buf, &len);
uint8_t *file = (uint8_t *)buf;
int ok = 1, err = 0;
for (uint32_t m = 0; m < 5 && ok; m++) {
yapf_texture_t *probe = yapf_tex_open(file, len, 0, &err);
uint64_t need = probe ? yapf_tex_stream_bytes(probe, m) : 0;
yapf_tex_free(probe);
yapf_texture_t *s = yapf_tex_open(file, (size_t)need, 0, &err);
ok = s != NULL && need <= len && (m > 0 || need == len);
for (uint32_t f = 0; ok && f < 6; f++)
for (uint32_t mm = 0; ok && mm < 5; mm++) {
uint32_t sf = yapf_tex_surface(s, 0, 0, f, mm);
int rc = yapf_tex_decode_surface(s, file, (size_t)need, sf, 1);
if (mm >= m) ok = rc == YAPF_OK &&
memcmp(s->data[sf], t->data[sf], yapf_tex_surface_bytes(t, mm)) == 0;
else ok = rc == YAPF_ERR_TRUNCATED;
}
yapf_tex_free(s);
}
CHECK(ok, "stream prefixes decode exactly their small mips");
/* tile decode equals the matching part of the full surface */
yapf_texture_t *s = yapf_tex_open(file, len, 0, &err);
uint32_t sf = yapf_tex_surface(s, 0, 0, 3, 0), tx, ty;
yapf_tex_tile_grid(s, 0, &tx, &ty);
uint8_t tile[64 * 64 * 4];
ok = s && tx == 5 && ty == 5;
for (uint32_t y = 0; ok && y < ty; y++)
for (uint32_t x = 0; ok && x < tx; x++) {
uint32_t px, py, pw, ph;
ok = yapf_tex_decode_tile(s, file, len, sf, x, y, 0, tile, 64 * 4, &px, &py, &pw, &ph) == YAPF_OK;
for (uint32_t r = 0; ok && r < ph; r++)
ok = memcmp(tile + r * 64 * 4, (uint8_t *)t->data[sf] + ((size_t)(py + r) * 260 + px) * 4,
(size_t)pw * 4) == 0;
}
CHECK(ok, "single tiles decode to the same pixels");
yapf_tex_free(s);
yapf_free_buffer(buf);
yapf_tex_free(t);
}
/* v1 and v2 work through either API where the data allows it. */
static void v2_compat(const char *sample) {
yapf_texture_t *t = yapf_tex_load(sample, 0, 0, NULL);
if (t) {
uint32_t h = fnv((const uint8_t *)t->data[0], yapf_tex_surface_bytes(t, 0));
CHECK(h == SAMPLE_HASH, "yapf_tex_load reads the v1 sample");
size_t len;
uint8_t *file = tex_roundtrip(t, YAPF_ENC_ENTROPY, &len, "sample as v2");
yapf_image_t *img = file ? yapf_load_memory(file, len) : NULL;
CHECK(img && img->width == 512 && img->channels == 4 &&
fnv(img->pixels, 512u * 512u * 4u) == SAMPLE_HASH, "yapf_load reads an 8-bit v2 file");
yapf_free(img);
free(file);
yapf_tex_free(t);
}
/* cube: the v1 API sees face 0 with its mips */
yapf_texture_t d = desc(40, 40, 3, 0);
d.shape = YAPF_SHAPE_CUBE; d.mip_levels = 3;
yapf_texture_t *c = yapf_tex_create(&d);
tex_fill(c, 1);
size_t len;
uint8_t *file = tex_roundtrip(c, 0, &len, "cube for v1 API");
yapf_image_t *img = file ? yapf_load_memory_mt(file, len, 2) : NULL;
CHECK(img && img->mip_levels == 3 && memcmp(img->mips[2], c->data[2], 10 * 10 * 3) == 0 &&
memcmp(img->pixels, c->data[0], 40 * 40 * 3) == 0, "yapf_load reads face 0 of a cube");
yapf_free(img);
free(file);
yapf_tex_free(c);
}
static double psnr8(const uint8_t *a, int sa, const uint8_t *b, int sb, size_t n, int ch) {
double se = 0;
for (size_t i = 0; i < n; i++)
for (int c = 0; c < ch; c++) {
double v = (double)a[i * sa + c] - b[i * sb + c];
se += v * v;
}
if (se == 0) return 99.0;
return 10.0 * log10(255.0 * 255.0 * (double)(n * ch) / se);
}
static void v2_gpu(void) {
const uint32_t w = 61, h = 47; /* not multiples of 4 */
yapf_texture_t d = desc(w, h, 4, 0);
yapf_texture_t *t = yapf_tex_create(&d);
uint8_t *p = (uint8_t *)t->data[0];
for (uint32_t y = 0; y < h; y++)
for (uint32_t x = 0; x < w; x++) {
uint8_t *q = p + ((size_t)y * w + x) * 4;
q[0] = (uint8_t)(x * 4); q[1] = (uint8_t)(y * 5); q[2] = (uint8_t)(255 - x * 2);
q[3] = (uint8_t)(((x / 8 + y / 8) & 1) ? 255 : 40 + x);
}
static const struct { uint8_t f; int ch; double min; } F[] = {
{ YAPF_GPU_BC1, 3, 30 }, { YAPF_GPU_BC3, 4, 30 }, { YAPF_GPU_BC7, 4, 34 },
{ YAPF_GPU_BC4, 1, 38 }, { YAPF_GPU_BC5, 2, 38 }
};
uint8_t *out = (uint8_t *)malloc((size_t)w * h * 4);
for (size_t i = 0; i < sizeof(F) / sizeof(F[0]); i++) {
yapf_texture_t *c = NULL;
int rc = yapf_tex_compress(t, F[i].f, &c);
int ok = rc == YAPF_OK && c->payload == YAPF_PAYLOAD_BLOCKS &&
yapf_blocks_decode(F[i].f, c->data[0], w, h, out) == YAPF_OK;
double q = 0;
if (ok && F[i].f == YAPF_GPU_BC1) {
/* BC1 keeps alpha only as cut-out: compare opaque texels */
for (size_t k = 0; k < (size_t)w * h; k++) if (p[k * 4 + 3] < 128) memcpy(out + k * 4, p + k * 4, 3);
}
if (ok) q = psnr8(p, 4, out, 4, (size_t)w * h, F[i].ch);
CHECK(ok && q >= F[i].min, "GPU format 0x%02x encodes (PSNR %.1f dB)", F[i].f, q);
if (c) {
size_t len;
char what[32];
sprintf(what, "compressed 0x%02x", F[i].f);
free(tex_roundtrip(c, YAPF_ENC_ENTROPY, &len, what));
}
yapf_tex_free(c);
}
CHECK(yapf_blocks_decode(YAPF_GPU_ASTC_4x4, out, 4, 4, out) == YAPF_ERR_UNSUPPORTED,
"ASTC blocks are carried, not decoded");
free(out);
yapf_tex_free(t);
/* BC6H: relative error of a smooth HDR sky gradient */
yapf_texture_t dh = desc(32, 32, 3, YAPF_SAMPLE_F16);
yapf_texture_t *th = yapf_tex_create(&dh), *ch = NULL;
uint16_t *hp = (uint16_t *)th->data[0];
for (int i = 0; i < 32 * 32; i++)
for (int k = 0; k < 3; k++) hp[i * 3 + k] = yapf_float_to_half((0.3f + 0.4f * (float)k) *
(1.0f + (float)(i % 32) * 0.02f + (float)(i / 32) * 0.15f));
int ok = yapf_tex_compress(th, YAPF_GPU_BC6H, &ch) == YAPF_OK;
uint16_t back[32 * 32 * 3];
double worst = 0;
if (ok) ok = yapf_blocks_decode(YAPF_GPU_BC6H, ch->data[0], 32, 32, back) == YAPF_OK;
for (int i = 0; ok && i < 32 * 32 * 3; i++) {
float a = yapf_half_to_float(hp[i]), b = yapf_half_to_float(back[i]);
double r = (a > b ? a - b : b - a) / a;
if (r > worst) worst = r;
}
CHECK(ok && worst < 0.05, "BC6H encodes HDR within 5%% (worst %.4f)", worst);
yapf_tex_free(ch);
yapf_tex_free(th);
}
static void v2_mips(void) {
/* sRGB: a black / white checker averages to sRGB 188, not 128 */
yapf_texture_t d = desc(64, 64, 4, 0);
d.flags = YAPF_FLAG_SRGB;
yapf_texture_t *t = yapf_tex_create(&d);
uint8_t *p = (uint8_t *)t->data[0];
for (int i = 0; i < 64 * 64; i++) {
uint8_t v = ((i % 64 + i / 64) & 1) ? 255 : 0;
p[i * 4] = p[i * 4 + 1] = p[i * 4 + 2] = v;
p[i * 4 + 3] = v; /* alpha stays linear */
}
int ok = yapf_tex_build_mips(t, 0, 0) == YAPF_OK && t->mip_levels == 7;
const uint8_t *m1 = ok ? (const uint8_t *)t->data[1] : NULL;
CHECK(ok && m1[0] == 188 && m1[3] == 128 && (t->mip_filter & YAPF_MIP_LINEAR_LIGHT),
"sRGB mips average in linear light (got %d, alpha %d)", m1 ? m1[0] : -1, m1 ? m1[3] : -1);
yapf_tex_free(t);
/* alpha test: speckled foliage (30 % of texels opaque). A plain box
* filter thins it out with distance; with alpha_ref = 128 every mip
* keeps about the same share of texels above 128. */
double plain[5] = {0}, kept[5] = {0};
for (int pass = 0; pass < 2; pass++) {
d = desc(128, 128, 4, 0);
t = yapf_tex_create(&d);
p = (uint8_t *)t->data[0];
rng = 99;
for (int i = 0; i < 128 * 128; i++) p[i * 4 + 3] = rnd() % 100 < 30 ? 255 : 0;
ok = yapf_tex_build_mips(t, 5, pass ? 128 : 0) == YAPF_OK && t->mip_levels == 5;
if (pass) ok = ok && (t->flags & YAPF_FLAG_COVERAGE) && t->alpha_ref == 128;
for (uint32_t m = 1; ok && m < 5; m++) {
uint32_t w, h, n = 0;
yapf_tex_mip_size(t, m, &w, &h, NULL);
const uint8_t *q = (const uint8_t *)t->data[m];
for (uint32_t i = 0; i < w * h; i++) n += q[i * 4 + 3] > 128;
(pass ? kept : plain)[m] = (double)n / (w * h);
}
if (pass) {
size_t len;
free(tex_roundtrip(t, 0, &len, "mipped texture"));
}
yapf_tex_free(t);
}
for (uint32_t m = 1; ok && m < 5; m++) ok = kept[m] > 0.2 && kept[m] < 0.4;
CHECK(ok && plain[2] < 0.15, "coverage-preserving mips keep alpha-test coverage "
"(mip 2: %.2f, plain box %.2f)", kept[2], plain[2]);
/* float volume: 2×2×2 box */
d = desc(4, 4, 1, YAPF_SAMPLE_F32);
d.shape = YAPF_SHAPE_VOLUME; d.depth = 4;
t = yapf_tex_create(&d);
float *f = (float *)t->data[0];
for (int i = 0; i < 64; i++) f[i] = (float)i;
ok = yapf_tex_build_mips(t, 0, 0) == YAPF_OK && t->mip_levels == 3;
CHECK(ok && ((float *)t->data[1])[0] == (0 + 1 + 4 + 5 + 16 + 17 + 20 + 21) / 8.0f &&
((float *)t->data[2])[0] == 31.5f, "volume mips filter in 3D");
yapf_tex_free(t);
}
static void v2_misc(void) {
int ok = 1;
for (uint32_t h = 0; h < 0x10000 && ok; h++) {
if ((h & 0x7C00) == 0x7C00 && (h & 0x3FF)) continue; /* NaN payloads */
ok = yapf_float_to_half(yapf_half_to_float((uint16_t)h)) == h;
}
CHECK(ok, "every half survives half -> float -> half");
CHECK(yapf_float_to_half(1.0f) == 0x3C00 && yapf_float_to_half(65520.0f) == 0x7C00 &&
yapf_float_to_half(1e-8f) == 0 && yapf_float_to_half(-2.0f) == 0xC000, "float -> half rounding");
/* fixed vectors pin yapf-sum to the specification */
CHECK(yapf_checksum("", 0) == 0xd7550ed0u && yapf_checksum("YAPF", 4) == 0xc8cc41d4u &&
yapf_checksum("The quick brown fox jumps over the lazy dog", 43) == 0xbce4f410u,
"yapf-sum test vectors");
/* encoding is deterministic */
yapf_texture_t d = desc(90, 70, 3, YAPF_SAMPLE_U16);
yapf_texture_t *t = yapf_tex_create(&d);
tex_fill(t, 2);
void *a = NULL, *b = NULL;
size_t la = 0, lb = 0;
yapf_tex_encode(t, YAPF_ENC_ENTROPY, &a, &la);
yapf_tex_encode(t, YAPF_ENC_ENTROPY, &b, &lb);
CHECK(a && la == lb && memcmp(a, b, la) == 0, "v2 encoding is deterministic");
yapf_free_buffer(a);
yapf_free_buffer(b);
yapf_tex_free(t);
d = desc(10, 12, 4, 0);
d.shape = YAPF_SHAPE_CUBE;
CHECK(yapf_tex_create(&d) == NULL, "a cube must be square");
}
int main(int argc, char **argv) { int main(int argc, char **argv) {
static const int sizes[][2] = { static const int sizes[][2] = {
{1, 1}, {1, 7}, {7, 1}, {63, 63}, {64, 64}, {65, 65}, {127, 1}, {1, 1}, {1, 7}, {7, 1}, {63, 63}, {64, 64}, {65, 65}, {127, 1},
@ -197,6 +688,16 @@ int main(int argc, char **argv) {
printf("note: sample %s not found, skipped\n", sample); printf("note: sample %s not found, skipped\n", sample);
} }
v2_pixels();
v2_layout();
v2_blocks();
v2_integrity();
v2_streaming();
v2_compat(sample);
v2_gpu();
v2_mips();
v2_misc();
printf("%d tests, %d failures\n", tests, fails); printf("%d tests, %d failures\n", tests, fails);
return fails != 0; return fails != 0;
} }

862
tools/yapf_cli.c
View file

@ -18,20 +18,17 @@
* ------------------------------------------------------------------------- * -------------------------------------------------------------------------
* *
* yapf <input> <output> [options] convert; formats chosen by extension * yapf <input> <output> [options] convert; formats chosen by extension
* yapf info <file.yapf> print header information * yapf pack <cube|array|volume|anim> <inputs…> <out.yapf> [options]
* yapf info <file.yapf> print header, tables and notes
* yapf verify <file.yapf> check every checksum and decode
* yapf bench <file.yapf> time decoding * yapf bench <file.yapf> time decoding
* yapf thumbnail <in.yapf> <out.png> [size] small preview (default 256) * yapf thumbnail <in.yapf> <out.png> [size] small preview (default 256)
* yapf compare <image.png> [more…] size and decode speed: original vs YAPF * yapf compare <image.png> [more…] size and decode speed: original vs YAPF
* *
* Reads PNG, JPEG, BMP, TGA, GIF (first frame), PSD (composite), PNM, HDR * Reads PNG (8 / 16-bit), JPEG, BMP, TGA, GIF (first frame), PSD, PNM,
* (tone-clamped) and YAPF. Writes YAPF, PNG, BMP, TGA and JPEG. * HDR (float) and YAPF. Writes YAPF, PNG, BMP, TGA, JPEG and HDR.
* * Plain 8-bit 2D images are written as YAPF version 1; anything that
* Options for writing YAPF: * needs version 2 (see `yapf` without arguments) is written as version 2.
* --mips also store a full mip chain (2×2 box filter)
* --linear mark pixels as linear instead of sRGB
* --premult mark alpha as premultiplied
* Options for reading YAPF:
* --mip N export mip level N instead of the base image
* *
* Build: cc -O2 -std=c99 -Ithird_party tools/yapf_cli.c yapf.c -o yapf -lm -pthread * Build: cc -O2 -std=c99 -Ithird_party tools/yapf_cli.c yapf.c -o yapf -lm -pthread
*/ */
@ -84,124 +81,532 @@ static double now_s(void) {
#endif #endif
} }
static uint8_t gpu_hint(int ch, int srgb) {
switch (ch) { /* ── Options ─────────────────────────────────────────────────────────── */
case 1: return YAPF_GPU_R8;
case 2: return YAPF_GPU_RG8; typedef struct {
case 3: return srgb ? YAPF_GPU_SRGB8 : YAPF_GPU_RGB8; int mips, srgb, premult, v2, entropy, eight_bit, float32;
default: return srgb ? YAPF_GPU_SRGB8_A8 : YAPF_GPU_RGBA8; int coverage; /* alpha_ref, 0 = off */
} int format; /* block format, -1 = none */
int normal; /* YAPF_NORMAL_* */
int linear_alpha;
int primaries, transfer;
int have_map;
uint8_t map[4];
double frame_ms;
int loops;
/* export selection */
int mip, layer, face, frame, slice;
const char *author, *license, *source;
const char *notes[16];
int nnotes;
} opts_t;
static const struct { const char *name; uint8_t v; } ch_names[] = {
{ "r", YAPF_CH_RED }, { "g", YAPF_CH_GREEN }, { "b", YAPF_CH_BLUE }, { "a", YAPF_CH_ALPHA },
{ "luma", YAPF_CH_LUMA }, { "nx", YAPF_CH_NORMAL_X }, { "ny", YAPF_CH_NORMAL_Y },
{ "nz", YAPF_CH_NORMAL_Z }, { "rough", YAPF_CH_ROUGHNESS }, { "gloss", YAPF_CH_GLOSS },
{ "metal", YAPF_CH_METALNESS }, { "ao", YAPF_CH_OCCLUSION }, { "height", YAPF_CH_HEIGHT },
{ "spec", YAPF_CH_SPECULAR }, { "emissive", YAPF_CH_EMISSIVE }, { "opacity", YAPF_CH_OPACITY },
{ "mask", YAPF_CH_MASK }, { "cavity", YAPF_CH_CAVITY }, { "thickness", YAPF_CH_THICKNESS },
{ "id", YAPF_CH_ID }, { "-", YAPF_CH_UNUSED }, { "default", YAPF_CH_DEFAULT }
};
static const char *ch_name(uint8_t v) {
for (size_t i = 0; i < sizeof(ch_names) / sizeof(ch_names[0]); i++)
if (ch_names[i].v == v) return ch_names[i].name;
return "?";
} }
/* Full mip chain with a 2×2 box filter (edges clamp for odd sizes). */ static const struct { const char *name; uint8_t v; } fmt_names[] = {
static int build_mips(yapf_image_t *img) { { "bc1", YAPF_GPU_BC1 }, { "bc3", YAPF_GPU_BC3 }, { "bc4", YAPF_GPU_BC4 },
int levels = 1; { "bc5", YAPF_GPU_BC5 }, { "bc6h", YAPF_GPU_BC6H }, { "bc7", YAPF_GPU_BC7 }
while (levels < (int)YAPF_MAX_MIPS && };
((img->width >> levels) > 0 || (img->height >> levels) > 0))
levels++;
img->mips = (uint8_t **)calloc((size_t)levels, sizeof(uint8_t *));
if (!img->mips) return -1;
img->mips[0] = img->pixels;
img->mip_levels = (uint8_t)levels;
int ch = img->channels;
for (int m = 1; m < levels; m++) {
uint32_t pw = img->width >> (m - 1), ph = img->height >> (m - 1);
uint32_t w = img->width >> m, h = img->height >> m;
if (pw < 1) pw = 1;
if (ph < 1) ph = 1;
if (w < 1) w = 1;
if (h < 1) h = 1;
uint8_t *src = img->mips[m - 1];
uint8_t *dst = (uint8_t *)malloc((size_t)w * h * ch);
if (!dst) return -1;
for (uint32_t y = 0; y < h; y++)
for (uint32_t x = 0; x < w; x++)
for (int c = 0; c < ch; c++) {
uint32_t x0 = 2 * x < pw ? 2 * x : pw - 1, x1 = 2 * x + 1 < pw ? 2 * x + 1 : pw - 1;
uint32_t y0 = 2 * y < ph ? 2 * y : ph - 1, y1 = 2 * y + 1 < ph ? 2 * y + 1 : ph - 1;
unsigned s = src[((size_t)y0 * pw + x0) * ch + c] + src[((size_t)y0 * pw + x1) * ch + c]
+ src[((size_t)y1 * pw + x0) * ch + c] + src[((size_t)y1 * pw + x1) * ch + c];
dst[((size_t)y * w + x) * ch + c] = (uint8_t)((s + 2) / 4);
}
img->mips[m] = dst;
}
return 0;
}
static int write_image(const char *path, const uint8_t *px, int w, int h, int ch) {
if (ends_with(path, ".png")) return stbi_write_png(path, w, h, ch, px, w * ch);
if (ends_with(path, ".bmp")) return stbi_write_bmp(path, w, h, ch, px);
if (ends_with(path, ".tga")) return stbi_write_tga(path, w, h, ch, px);
if (ends_with(path, ".jpg") || ends_with(path, ".jpeg"))
return stbi_write_jpg(path, w, h, ch, px, 95);
fprintf(stderr, "yapf: unsupported output format: %s (use .yapf .png .bmp .tga .jpg)\n", path);
return 0;
}
static const char *gpu_name(int g) { static const char *gpu_name(int g) {
static const char *n[] = { "RGBA8", "RGB8", "RG8", "R8", "SRGB8_A8", "SRGB8" }; static const char *px[] = { "RGBA8", "RGB8", "RG8", "R8", "SRGB8_A8", "SRGB8",
return g >= 0 && g < 6 ? n[g] : "unknown"; "RGBA16", "RGB16", "RG16", "R16", "RGBA16F", "RGB16F", "RG16F", "R16F",
"RGBA32F", "RGB32F", "RG32F", "R32F" };
static const char *bc[] = { "BC1", "BC2", "BC3", "BC4", "BC4_SNORM", "BC5", "BC5_SNORM",
"BC6H", "BC6H_SF", "BC7" };
static const char *etc[] = { "ETC2_RGB8", "ETC2_RGB8A1", "ETC2_RGBA8", "EAC_R11",
"EAC_R11_SNORM", "EAC_RG11", "EAC_RG11_SNORM" };
static const char *astc[] = { "ASTC_4x4", "ASTC_5x4", "ASTC_5x5", "ASTC_6x5", "ASTC_6x6",
"ASTC_8x5", "ASTC_8x6", "ASTC_8x8", "ASTC_10x5", "ASTC_10x6", "ASTC_10x8",
"ASTC_10x10", "ASTC_12x10", "ASTC_12x12" };
if (g >= 0 && g < 18) return px[g];
if (g >= 0x40 && g < 0x4A) return bc[g - 0x40];
if (g >= 0x50 && g < 0x57) return etc[g - 0x50];
if (g >= 0x60 && g < 0x6E) return astc[g - 0x60];
return "unknown";
}
static uint8_t gpu_hint(int ch, int srgb, int sample) {
static const uint8_t u8[2][4] = { { YAPF_GPU_R8, YAPF_GPU_RG8, YAPF_GPU_RGB8, YAPF_GPU_RGBA8 },
{ YAPF_GPU_R8, YAPF_GPU_RG8, YAPF_GPU_SRGB8, YAPF_GPU_SRGB8_A8 } };
static const uint8_t wide[3][4] = {
{ YAPF_GPU_R16, YAPF_GPU_RG16, YAPF_GPU_RGB16, YAPF_GPU_RGBA16 },
{ YAPF_GPU_R16F, YAPF_GPU_RG16F, YAPF_GPU_RGB16F, YAPF_GPU_RGBA16F },
{ YAPF_GPU_R32F, YAPF_GPU_RG32F, YAPF_GPU_RGB32F, YAPF_GPU_RGBA32F } };
if (sample == YAPF_SAMPLE_U8) return u8[srgb ? 1 : 0][ch - 1];
return wide[sample - 1][ch - 1];
}
static int parse_map(const char *s, uint8_t *map) {
memset(map, 0, 4);
for (int c = 0; c < 4 && *s; c++) {
const char *e = strchr(s, ',');
size_t n = e ? (size_t)(e - s) : strlen(s);
size_t i;
for (i = 0; i < sizeof(ch_names) / sizeof(ch_names[0]); i++)
if (strlen(ch_names[i].name) == n && strncmp(ch_names[i].name, s, n) == 0) break;
if (i == sizeof(ch_names) / sizeof(ch_names[0])) return 0;
map[c] = ch_names[i].v;
s += n + (e ? 1 : 0);
}
return 1;
}
/* ── Reading any input into a texture ────────────────────────────────── */
static yapf_texture_t *new_tex(uint32_t w, uint32_t h, int ch, int sample) {
yapf_texture_t d;
memset(&d, 0, sizeof(d));
d.width = w; d.height = h; d.depth = 1; d.layers = 1; d.frames = 1; d.mip_levels = 1;
d.channels = (uint8_t)ch; d.sample = (uint8_t)sample;
return yapf_tex_create(&d);
}
static yapf_texture_t *read_input(const char *path, const opts_t *o, int *from_yapf) {
*from_yapf = 0;
if (is_yapf(path)) {
int err = 0;
yapf_texture_t *t = yapf_tex_load(path, 0, 0, &err);
if (!t) fprintf(stderr, "yapf: cannot read %s (error %d)\n", path, err);
*from_yapf = 1;
return t;
}
int w, h, ch;
yapf_texture_t *t = NULL;
if (stbi_is_hdr(path)) {
float *f = stbi_loadf(path, &w, &h, &ch, 0);
if (!f) { fprintf(stderr, "yapf: cannot read %s: %s\n", path, stbi_failure_reason()); return NULL; }
int sample = o->float32 ? YAPF_SAMPLE_F32 : YAPF_SAMPLE_F16;
t = new_tex((uint32_t)w, (uint32_t)h, ch, sample);
if (t) {
size_t n = (size_t)w * h * ch;
if (sample == YAPF_SAMPLE_F32) memcpy(t->data[0], f, n * 4);
else for (size_t i = 0; i < n; i++) ((uint16_t *)t->data[0])[i] = yapf_float_to_half(f[i]);
t->transfer = YAPF_TRANSFER_LINEAR;
}
stbi_image_free(f);
} else if (stbi_is_16_bit(path) && !o->eight_bit) {
stbi_us *p = stbi_load_16(path, &w, &h, &ch, 0);
if (!p) { fprintf(stderr, "yapf: cannot read %s: %s\n", path, stbi_failure_reason()); return NULL; }
t = new_tex((uint32_t)w, (uint32_t)h, ch, YAPF_SAMPLE_U16);
if (t) memcpy(t->data[0], p, (size_t)w * h * ch * 2);
stbi_image_free(p);
} else {
stbi_uc *p = stbi_load(path, &w, &h, &ch, 0);
if (!p) { fprintf(stderr, "yapf: cannot read %s: %s\n", path, stbi_failure_reason()); return NULL; }
t = new_tex((uint32_t)w, (uint32_t)h, ch, YAPF_SAMPLE_U8);
if (t) memcpy(t->data[0], p, (size_t)w * h * ch);
stbi_image_free(p);
}
if (!t) fprintf(stderr, "yapf: %s: unsupported size or out of memory\n", path);
return t;
}
/* Several same-sized inputs into one cube / array / volume / animation. */
static yapf_texture_t *assemble(const char *kind, int n, char **paths, const opts_t *o) {
yapf_texture_t *first = NULL, *t = NULL;
for (int i = 0; i < n; i++) {
int y;
yapf_texture_t *in = read_input(paths[i], o, &y);
if (!in) { yapf_tex_free(t); yapf_tex_free(first); return NULL; }
if (!t) {
yapf_texture_t d = *in;
d.mip_levels = 1;
if (strcmp(kind, "cube") == 0) {
if (n % 6) { fprintf(stderr, "yapf: a cube needs 6 faces per layer (+X -X +Y -Y +Z -Z)\n"); yapf_tex_free(in); return NULL; }
d.shape = YAPF_SHAPE_CUBE; d.layers = (uint16_t)(n / 6);
} else if (strcmp(kind, "array") == 0) {
d.layers = (uint16_t)n;
} else if (strcmp(kind, "volume") == 0) {
d.shape = YAPF_SHAPE_VOLUME; d.depth = (uint32_t)n;
} else if (strcmp(kind, "anim") == 0) {
d.frames = (uint16_t)n;
} else {
fprintf(stderr, "yapf: pack kind must be cube, array, volume or anim\n");
yapf_tex_free(in);
return NULL;
}
t = yapf_tex_create(&d);
if (!t) { fprintf(stderr, "yapf: invalid %s (cube faces must be square)\n", kind); yapf_tex_free(in); return NULL; }
first = in;
} else if (in->width != t->width || in->height != t->height ||
in->channels != t->channels || in->sample != t->sample) {
fprintf(stderr, "yapf: %s does not match the size and channels of %s\n", paths[i], paths[0]);
yapf_tex_free(in); yapf_tex_free(t); yapf_tex_free(first);
return NULL;
}
size_t bytes = yapf_tex_surface_bytes(in, 0);
if (t->shape == YAPF_SHAPE_VOLUME) memcpy((uint8_t *)t->data[0] + bytes * i, in->data[0], bytes);
else if (t->frames > 1) memcpy(t->data[yapf_tex_surface(t, (uint32_t)i, 0, 0, 0)], in->data[0], bytes);
else if (t->shape == YAPF_SHAPE_CUBE) memcpy(t->data[yapf_tex_surface(t, 0, (uint32_t)i / 6, (uint32_t)i % 6, 0)], in->data[0], bytes);
else memcpy(t->data[yapf_tex_surface(t, 0, (uint32_t)i, 0, 0)], in->data[0], bytes);
if (in != first) yapf_tex_free(in);
}
if (t && t->frames > 1)
for (int f = 0; f < t->frames; f++) t->frame_us[f] = (uint32_t)(o->frame_ms * 1000.0 + 0.5);
yapf_tex_free(first);
return t;
}
/* ── Writing .yapf ───────────────────────────────────────────────────── */
static int needs_v2(const yapf_texture_t *t, const opts_t *o) {
return o->v2 || o->entropy || o->format >= 0 || o->coverage || o->normal || o->have_map ||
o->linear_alpha || o->primaries || o->transfer || o->author || o->license ||
o->source || o->nnotes || t->sample != YAPF_SAMPLE_U8 || t->shape != YAPF_SHAPE_2D ||
t->layers > 1 || t->frames > 1 || t->payload != YAPF_PAYLOAD_PIXELS || t->note_count;
}
static int write_yapf(yapf_texture_t *t, const char *out, const opts_t *o, int from_yapf) {
if (!from_yapf) {
t->flags = (uint8_t)((o->srgb && t->sample == YAPF_SAMPLE_U8 && !o->normal ? YAPF_FLAG_SRGB : 0) |
(o->premult ? YAPF_FLAG_PREMULT_ALPHA : 0));
t->format = gpu_hint(t->channels, t->flags & YAPF_FLAG_SRGB, t->sample);
}
if (o->normal) {
t->normal = (uint8_t)o->normal;
t->channel_map[0] = YAPF_CH_NORMAL_X;
t->channel_map[1] = YAPF_CH_NORMAL_Y;
if (t->channels >= 3) t->channel_map[2] = YAPF_CH_NORMAL_Z;
else t->normal |= YAPF_NORMAL_XY;
t->flags &= (uint8_t)~YAPF_FLAG_SRGB;
}
if (o->have_map) memcpy(t->channel_map, o->map, 4);
if (o->linear_alpha && (t->channels == 2 || t->channels == 4))
t->linear_mask |= (uint8_t)(1u << (t->channels - 1));
if (o->primaries) t->primaries = (uint8_t)o->primaries;
if (o->transfer) t->transfer = (uint8_t)o->transfer;
if (o->frame_ms > 0 && t->frames > 1)
for (int f = 0; f < t->frames; f++) t->frame_us[f] = (uint32_t)(o->frame_ms * 1000.0 + 0.5);
if (o->loops >= 0) t->loop_count = (uint16_t)o->loops;
if (o->mips && t->payload == YAPF_PAYLOAD_PIXELS) {
int rc = yapf_tex_build_mips(t, 0, (uint8_t)o->coverage);
if (rc != YAPF_OK) {
fprintf(stderr, "yapf: cannot build mips%s\n", o->coverage ? " (--coverage needs an alpha channel)" : "");
return 1;
}
}
int v2 = needs_v2(t, o);
if (o->format >= 0) {
yapf_texture_t *c = NULL;
int rc = yapf_tex_compress(t, (uint8_t)o->format, &c);
if (rc != YAPF_OK) {
fprintf(stderr, "yapf: cannot compress to %s: %s\n", gpu_name(o->format),
o->format == YAPF_GPU_BC6H ? "BC6H needs a 16-bit or HDR (float) image"
: "LDR block formats need an 8-bit image");
return 1;
}
opts_t rest = *o; /* the blocks are made: write them */
rest.format = -1;
rest.mips = 0;
rest.v2 = 1;
int r2 = write_yapf(c, out, &rest, 1);
yapf_tex_free(c);
return r2;
}
if (!v2) {
/* 8-bit 2D: version 1, readable by every YAPF decoder */
yapf_image_t img;
memset(&img, 0, sizeof(img));
img.width = t->width; img.height = t->height; img.channels = t->channels;
img.gpu_format = t->format; img.flags = t->flags; img.mip_levels = t->mip_levels;
img.mips = (uint8_t **)t->data;
img.pixels = (uint8_t *)t->data[0];
if (yapf_save(out, &img) != YAPF_OK) { fprintf(stderr, "yapf: cannot write %s\n", out); return 1; }
return 0;
}
if (!yapf_tex_find_note(t, "TOOL")) yapf_tex_add_note(t, "TOOL", "yapf command-line tool", 22);
if (o->author) yapf_tex_add_note(t, "AUTH", o->author, (uint32_t)strlen(o->author));
if (o->license) yapf_tex_add_note(t, "LICN", o->license, (uint32_t)strlen(o->license));
if (o->source) yapf_tex_add_note(t, "SRCE", o->source, (uint32_t)strlen(o->source));
for (int i = 0; i < o->nnotes; i++) {
const char *eq = strchr(o->notes[i], '=');
if (!eq || eq - o->notes[i] != 4) { fprintf(stderr, "yapf: --note wants TAG=text with a 4-letter tag\n"); return 1; }
yapf_tex_add_note(t, o->notes[i], eq + 1, (uint32_t)strlen(eq + 1));
}
if (yapf_tex_save(out, t, o->entropy ? YAPF_ENC_ENTROPY : 0) != YAPF_OK) {
fprintf(stderr, "yapf: cannot write %s\n", out);
return 1;
}
return 0;
}
/* ── Exporting a surface to a common format ──────────────────────────── */
/* One surface slice as 8-bit pixels (*hdr = 0) or float RGB(A) (*hdr = 1). */
static void *surface_view(const yapf_texture_t *t, uint32_t s, uint32_t slice,
uint32_t *w, uint32_t *h, int *ch, int *hdr)
{
uint32_t m = s % t->mip_levels, d;
yapf_tex_mip_size(t, m, w, h, &d);
if (slice >= d) return NULL;
size_t n = (size_t)*w * *h, slice_bytes = yapf_tex_surface_bytes(t, m) / d;
const uint8_t *src = (const uint8_t *)t->data[s] + slice_bytes * slice;
*hdr = 0;
if (t->payload == YAPF_PAYLOAD_BLOCKS) {
int is6 = t->format == YAPF_GPU_BC6H || t->format == YAPF_GPU_BC6H_SF;
void *out = malloc(n * (is6 ? 6 : 4));
if (!out) return NULL;
if (yapf_blocks_decode(t->format, src, *w, *h, out) != YAPF_OK) { free(out); return NULL; }
if (!is6) { *ch = 4; return out; }
float *f = (float *)malloc(n * 3 * sizeof(float));
for (size_t i = 0; f && i < n * 3; i++) f[i] = yapf_half_to_float(((uint16_t *)out)[i]);
free(out);
*ch = 3; *hdr = 1;
return f;
}
*ch = t->channels;
if (t->sample == YAPF_SAMPLE_U8) {
uint8_t *o = (uint8_t *)malloc(n * *ch);
if (o) memcpy(o, src, n * *ch);
return o;
}
if (t->sample == YAPF_SAMPLE_U16) {
uint8_t *o = (uint8_t *)malloc(n * *ch);
for (size_t i = 0; o && i < n * *ch; i++) o[i] = (uint8_t)((((const uint16_t *)src)[i] + 128) / 257);
return o;
}
float *f = (float *)malloc(n * *ch * sizeof(float));
for (size_t i = 0; f && i < n * *ch; i++)
f[i] = t->sample == YAPF_SAMPLE_F16 ? yapf_half_to_float(((const uint16_t *)src)[i])
: ((const float *)src)[i];
*hdr = 1;
return f;
}
static uint8_t *to8(const float *f, size_t n) {
uint8_t *o = (uint8_t *)malloc(n);
for (size_t i = 0; o && i < n; i++) {
float v = f[i] < 0 ? 0 : f[i] > 1 ? 1 : f[i];
o[i] = (uint8_t)(v * 255.0f + 0.5f);
}
return o;
}
static int write_image(const char *path, const void *px, int w, int h, int ch, int hdr) {
if (ends_with(path, ".hdr")) {
if (!hdr) {
float *f = (float *)malloc((size_t)w * h * ch * sizeof(float));
for (size_t i = 0; f && i < (size_t)w * h * ch; i++) f[i] = ((const uint8_t *)px)[i] / 255.0f;
int ok = f && stbi_write_hdr(path, w, h, ch, f);
free(f);
return ok;
}
return stbi_write_hdr(path, w, h, ch, (const float *)px);
}
uint8_t *tmp = NULL;
if (hdr) {
tmp = to8((const float *)px, (size_t)w * h * ch);
if (!tmp) return 0;
fprintf(stderr, "yapf: note: float pixels clamped to 0..1 for %s (use .hdr to keep them)\n", path);
px = tmp;
}
int ok = 0;
if (ends_with(path, ".png")) ok = stbi_write_png(path, w, h, ch, px, w * ch);
else if (ends_with(path, ".bmp")) ok = stbi_write_bmp(path, w, h, ch, px);
else if (ends_with(path, ".tga")) ok = stbi_write_tga(path, w, h, ch, px);
else if (ends_with(path, ".jpg") || ends_with(path, ".jpeg")) ok = stbi_write_jpg(path, w, h, ch, px, 95);
else fprintf(stderr, "yapf: unsupported output format: %s (use .yapf .png .bmp .tga .jpg .hdr)\n", path);
free(tmp);
return ok;
}
static int export_image(const yapf_texture_t *t, const char *in, const char *out, const opts_t *o) {
uint32_t faces = yapf_tex_faces(t);
if ((uint32_t)o->mip >= t->mip_levels || (uint32_t)o->layer >= t->layers ||
(uint32_t)o->face >= faces || (uint32_t)o->frame >= t->frames) {
fprintf(stderr, "yapf: %s has %u mip(s), %u layer(s), %u face(s), %u frame(s)\n",
in, t->mip_levels, t->layers, faces, t->frames);
return 1;
}
uint32_t s = yapf_tex_surface(t, (uint32_t)o->frame, (uint32_t)o->layer, (uint32_t)o->face, (uint32_t)o->mip);
uint32_t w, h;
int ch, hdr;
void *px = surface_view(t, s, (uint32_t)o->slice, &w, &h, &ch, &hdr);
if (!px) {
fprintf(stderr, "yapf: cannot export %s: %s\n", in,
t->payload == YAPF_PAYLOAD_BLOCKS ? "no built-in decoder for this block format (ETC2 / ASTC)"
: "no such slice, or out of memory");
return 1;
}
if (t->sample == YAPF_SAMPLE_U16 && !ends_with(out, ".hdr"))
fprintf(stderr, "yapf: note: 16-bit samples written as 8-bit\n");
int ok = write_image(out, px, (int)w, (int)h, ch, hdr);
free(px);
if (!ok) { fprintf(stderr, "yapf: cannot write %s\n", out); return 1; }
return 0;
}
/* ── info / verify / bench / thumbnail ───────────────────────────────── */
static uint8_t *read_all(const char *path, size_t *n) {
FILE *f = fopen(path, "rb");
if (!f) return NULL;
fseek(f, 0, SEEK_END);
long sz = ftell(f);
rewind(f);
uint8_t *buf = sz >= 0 ? (uint8_t *)malloc((size_t)sz + 1) : NULL;
if (buf && fread(buf, 1, (size_t)sz, f) != (size_t)sz) { free(buf); buf = NULL; }
fclose(f);
*n = buf ? (size_t)sz : 0;
return buf;
}
static const char *err_name(int e) {
switch (e) {
case YAPF_ERR_INVALID: return "invalid argument";
case YAPF_ERR_IO: return "cannot read the file";
case YAPF_ERR_OOM: return "out of memory";
case YAPF_ERR_CORRUPT: return "corrupt or not a YAPF file";
case YAPF_ERR_UNSUPPORTED: return "needs a feature this decoder does not have";
case YAPF_ERR_CHECKSUM: return "checksum mismatch (damaged file)";
case YAPF_ERR_TRUNCATED: return "file is truncated";
default: return "error";
}
} }
static int cmd_info(const char *path) { static int cmd_info(const char *path) {
FILE *f = fopen(path, "rb"); size_t n;
uint8_t hdr[20]; uint8_t *buf = read_all(path, &n);
if (!f || fread(hdr, 1, 20, f) != 20 || memcmp(hdr, "YAPF", 4) != 0) { int err = 0;
fprintf(stderr, "yapf: %s is not a YAPF file\n", path); yapf_texture_t *t = buf ? yapf_tex_open(buf, n, 0, &err) : NULL;
if (f) fclose(f); if (!t) {
fprintf(stderr, "yapf: %s: %s\n", path, buf ? err_name(err) : "cannot open");
free(buf);
return 1; return 1;
} }
fseek(f, 0, SEEK_END); static const char *shape[] = { "2D", "cube", "volume" };
long size = ftell(f); static const char *sample[] = { "8-bit", "16-bit", "half float", "float" };
fclose(f); static const char *prim[] = { "unspecified", "BT.709 / sRGB", "Display P3", "BT.2020",
unsigned w = hdr[12] | hdr[13] << 8 | (unsigned)hdr[14] << 16 | (unsigned)hdr[15] << 24; "ACES AP0", "ACEScg", "Adobe RGB" };
unsigned h = hdr[16] | hdr[17] << 8 | (unsigned)hdr[18] << 16 | (unsigned)hdr[19] << 24; static const char *tf[] = { "unspecified", "sRGB", "linear", "gamma 2.2", "PQ", "HLG", "BT.1886" };
static const char *mf[] = { "unknown", "box", "tent", "Kaiser", "Lanczos", "Mitchell", "authored" };
uint32_t w, h, d;
yapf_tex_mip_size(t, 0, &w, &h, &d);
printf("%s\n", path); printf("%s\n", path);
printf(" format version : %u\n", hdr[4]); printf(" format version : %u\n", buf[4]);
printf(" size : %u x %u, %u channel(s)\n", w, h, hdr[5]); printf(" size : %u x %u%s", w, h, t->shape == YAPF_SHAPE_VOLUME ? "" : "\n");
printf(" mip levels : %u\n", hdr[7]); if (t->shape == YAPF_SHAPE_VOLUME) printf(" x %u\n", d);
printf(" gpu format : %s\n", gpu_name(hdr[6])); printf(" shape : %s, %u layer(s), %u frame(s)\n", shape[t->shape], t->layers, t->frames);
printf(" flags : %s%s\n", hdr[8] & YAPF_FLAG_SRGB ? "sRGB " : "linear ", printf(" mip levels : %u\n", t->mip_levels);
hdr[8] & YAPF_FLAG_PREMULT_ALPHA ? "premultiplied-alpha" : ""); if (t->payload == YAPF_PAYLOAD_BLOCKS)
printf(" file size : %ld bytes (%.1f%% of raw)\n", size, printf(" payload : GPU blocks, %s (%u channel(s))\n", gpu_name(t->format), t->channels);
100.0 * (double)size / ((double)w * h * (hdr[5] ? hdr[5] : 1))); else
yapf_image_t *img = yapf_load(path); printf(" payload : lossless pixels, %u channel(s), %s, GPU hint %s\n",
printf(" decodes : %s\n", img ? "ok" : "FAILED (corrupt or unsupported)"); t->channels, sample[t->sample], gpu_name(t->format));
yapf_free(img); printf(" flags : %s%s%s\n", t->flags & YAPF_FLAG_SRGB ? "sRGB" : "linear",
return img ? 0 : 1; t->flags & YAPF_FLAG_PREMULT_ALPHA ? ", premultiplied alpha" : "",
t->flags & YAPF_FLAG_COVERAGE ? ", coverage-preserving mips" : "");
if (buf[4] == 2) {
uint32_t needs = (uint32_t)buf[0x1C] | (uint32_t)buf[0x1D] << 8;
printf(" needs : %s%s%s%s%s%s\n", needs ? "" : "nothing beyond 8-bit 2D",
needs & YAPF_NEED_WIDE ? "wide " : "", needs & YAPF_NEED_BLOCKS ? "blocks " : "",
needs & YAPF_NEED_ENTROPY ? "entropy " : "", needs & YAPF_NEED_LAYOUT ? "layout " : "",
needs & YAPF_NEED_FRAMES ? "frames" : "");
printf(" channels :");
for (int c = 0; c < t->channels; c++)
printf(" %s%s", ch_name(t->channel_map[c]), t->linear_mask & (1u << c) ? "(linear)" : "");
printf("\n");
if (t->normal)
printf(" normal map : %s, %s\n",
(t->normal & 3) == YAPF_NORMAL_GL ? "OpenGL (+Y)" : (t->normal & 3) == YAPF_NORMAL_DX ? "DirectX (-Y)" : "no convention",
t->normal & YAPF_NORMAL_XY ? "XY (Z reconstructed)" : t->normal & YAPF_NORMAL_OCT ? "octahedral" : "XYZ");
printf(" colour : primaries %s, transfer %s\n",
t->primaries < 7 ? prim[t->primaries] : "?", t->transfer < 7 ? tf[t->transfer] : "?");
if (t->mip_levels > 1)
printf(" mip filter : %s%s%s\n", (t->mip_filter & 0x7F) < 7 ? mf[t->mip_filter & 0x7F] : "?",
t->mip_filter & YAPF_MIP_LINEAR_LIGHT ? ", linear light" : "",
t->flags & YAPF_FLAG_COVERAGE ? "" : "");
if (t->flags & YAPF_FLAG_COVERAGE) printf(" alpha ref : %u\n", t->alpha_ref);
if (t->frames > 1)
printf(" animation : %u frames, first %.1f ms, loops %s\n", t->frames,
t->frame_us ? t->frame_us[0] / 1000.0 : 0.0, t->loop_count ? "" : "forever");
for (uint32_t i = 0; i < t->note_count; i++) {
const yapf_note_t *nt = &t->notes[i];
int text = 1;
for (uint32_t k = 0; k < nt->size; k++) if (nt->data[k] < 9) text = 0;
if (text && nt->size < 200) printf(" note %.4s : %.*s\n", nt->tag, (int)nt->size, nt->data);
else printf(" note %.4s : %u bytes\n", nt->tag, nt->size);
}
printf(" stream prefix :");
for (uint32_t m = t->mip_levels; m-- > 0; ) {
uint32_t mw;
yapf_tex_mip_size(t, m, &mw, NULL, NULL);
printf(" %u px: %llu B%s", mw, (unsigned long long)yapf_tex_stream_bytes(t, m), m ? "," : "\n");
}
}
size_t raw = 0;
for (uint32_t s = 0; s < yapf_tex_surface_count(t); s++) {
uint32_t mw, mh, md;
yapf_tex_mip_size(t, s % t->mip_levels, &mw, &mh, &md);
raw += (size_t)mw * mh * md * t->channels * (t->payload ? 1 : yapf_sample_bytes(t->sample));
}
printf(" file size : %zu bytes (%.1f%% of raw)\n", n, 100.0 * (double)n / (double)raw);
yapf_tex_free(t);
t = yapf_tex_load_memory(buf, n, 0, 0, &err);
printf(" decodes : %s\n", t ? "ok (checksums verified)" : err_name(err));
yapf_tex_free(t);
free(buf);
return t ? 0 : 1;
}
static int cmd_verify(const char *path) {
size_t n;
uint8_t *buf = read_all(path, &n);
int err = YAPF_ERR_IO;
yapf_texture_t *t = buf ? yapf_tex_load_memory(buf, n, 0, 0, &err) : NULL;
if (t) printf("%s: ok%s\n", path, buf[4] == 2 ? " (header, tables and every tile checksum match)"
: " (version 1: decodes; no checksums in v1)");
else printf("%s: FAILED — %s\n", path, err_name(err));
yapf_tex_free(t);
free(buf);
return t ? 0 : 1;
} }
static int cmd_bench(const char *path) { static int cmd_bench(const char *path) {
FILE *f = fopen(path, "rb"); size_t n;
if (!f) { fprintf(stderr, "yapf: cannot open %s\n", path); return 1; } uint8_t *buf = read_all(path, &n);
fseek(f, 0, SEEK_END); if (!buf) { fprintf(stderr, "yapf: cannot open %s\n", path); return 1; }
long n = ftell(f); int err;
rewind(f); yapf_texture_t *t = yapf_tex_load_memory(buf, n, 0, 1, &err);
uint8_t *buf = (uint8_t *)malloc((size_t)n); if (!t) { fprintf(stderr, "yapf: %s: %s\n", path, err_name(err)); free(buf); return 1; }
if (!buf || fread(buf, 1, (size_t)n, f) != (size_t)n) { fclose(f); free(buf); return 1; } double raw = 0;
fclose(f); for (uint32_t s = 0; s < yapf_tex_surface_count(t); s++) raw += (double)yapf_tex_surface_bytes(t, s % t->mip_levels);
yapf_image_t *img = yapf_load_memory(buf, (size_t)n); yapf_tex_free(t);
if (!img) { fprintf(stderr, "yapf: %s does not decode\n", path); free(buf); return 1; } for (int mode = 0; mode < 3; mode++) {
double raw = (double)img->width * img->height * img->channels; int threads = mode == 1 ? 0 : 1, flags = mode == 2 ? YAPF_LOAD_TRUST : 0;
yapf_free(img); if (mode == 2 && buf[4] != 2) break;
for (int mt = 0; mt < 2; mt++) {
/* Best of 15 rounds; each round repeats enough decodes to last /* Best of 15 rounds; each round repeats enough decodes to last
* ~20 ms, so short images are measured warm and not timer-bound. */ * ~20 ms, so short images are measured warm and not timer-bound. */
double one = now_s(); double one = now_s();
yapf_free(mt ? yapf_load_memory_mt(buf, (size_t)n, 0) : yapf_load_memory(buf, (size_t)n)); yapf_tex_free(yapf_tex_load_memory(buf, n, flags, threads, NULL));
one = now_s() - one; one = now_s() - one;
int reps = (int)(0.02 / (one > 1e-6 ? one : 1e-6)) + 1; int reps = (int)(0.02 / (one > 1e-6 ? one : 1e-6)) + 1;
double best = 1e9; double best = 1e9;
for (int r = 0; r < 15; r++) { for (int r = 0; r < 15; r++) {
double t = now_s(); double tt = now_s();
for (int k = 0; k < reps; k++) for (int k = 0; k < reps; k++) yapf_tex_free(yapf_tex_load_memory(buf, n, flags, threads, NULL));
yapf_free(mt ? yapf_load_memory_mt(buf, (size_t)n, 0) tt = (now_s() - tt) / reps;
: yapf_load_memory(buf, (size_t)n)); if (tt < best) best = tt;
t = (now_s() - t) / reps;
if (t < best) best = t;
} }
printf(" %-12s %8.3f ms %6.2f GB/s\n", mt ? "all cores" : "1 thread", best * 1e3, raw / best / 1e9); static const char *nm[] = { "1 thread", "all cores", "no checks" };
printf(" %-12s %8.3f ms %6.2f GB/s\n", nm[mode], best * 1e3, raw / best / 1e9);
} }
free(buf); free(buf);
return 0; return 0;
@ -210,23 +615,31 @@ static int cmd_bench(const char *path) {
/* Preview no larger than size×size: start from the smallest mip that is /* Preview no larger than size×size: start from the smallest mip that is
* still big enough, then average boxes of source pixels. */ * still big enough, then average boxes of source pixels. */
static int cmd_thumbnail(const char *in, const char *out, int size) { static int cmd_thumbnail(const char *in, const char *out, int size) {
yapf_image_t *img = yapf_load_mt(in, 0); int err;
if (!img) { fprintf(stderr, "yapf: cannot read %s\n", in); return 1; } yapf_texture_t *t = yapf_tex_load(in, 0, 0, &err);
if (!t) { fprintf(stderr, "yapf: cannot read %s: %s\n", in, err_name(err)); return 1; }
if (size < 1) size = 256; if (size < 1) size = 256;
int m = 0; int m = 0;
while (m + 1 < img->mip_levels && while (m + 1 < t->mip_levels &&
(int)(img->width >> (m + 1)) >= size && (int)(img->height >> (m + 1)) >= size) (int)(t->width >> (m + 1)) >= size && (int)(t->height >> (m + 1)) >= size)
m++; m++;
uint32_t sw = img->width >> m ? img->width >> m : 1, sh = img->height >> m ? img->height >> m : 1; uint32_t sw, sh;
const uint8_t *src = img->mips[m]; int ch, hdr;
uint8_t *src = (uint8_t *)surface_view(t, (uint32_t)m, 0, &sw, &sh, &ch, &hdr);
yapf_tex_free(t);
if (!src) { fprintf(stderr, "yapf: cannot preview %s\n", in); return 1; }
if (hdr) {
uint8_t *b = to8((const float *)src, (size_t)sw * sh * ch);
free(src);
if (!(src = b)) return 1;
}
double k = (double)size / (double)(sw > sh ? sw : sh); double k = (double)size / (double)(sw > sh ? sw : sh);
if (k > 1.0) k = 1.0; if (k > 1.0) k = 1.0;
uint32_t dw = (uint32_t)(sw * k + 0.5), dh = (uint32_t)(sh * k + 0.5); uint32_t dw = (uint32_t)(sw * k + 0.5), dh = (uint32_t)(sh * k + 0.5);
if (dw < 1) dw = 1; if (dw < 1) dw = 1;
if (dh < 1) dh = 1; if (dh < 1) dh = 1;
int ch = img->channels;
uint8_t *dst = (uint8_t *)malloc((size_t)dw * dh * ch); uint8_t *dst = (uint8_t *)malloc((size_t)dw * dh * ch);
if (!dst) { yapf_free(img); return 1; } if (!dst) { free(src); return 1; }
for (uint32_t y = 0; y < dh; y++) { for (uint32_t y = 0; y < dh; y++) {
uint32_t y0 = (uint32_t)((uint64_t)y * sh / dh), y1 = (uint32_t)((uint64_t)(y + 1) * sh / dh); uint32_t y0 = (uint32_t)((uint64_t)y * sh / dh), y1 = (uint32_t)((uint64_t)(y + 1) * sh / dh);
if (y1 <= y0) y1 = y0 + 1; if (y1 <= y0) y1 = y0 + 1;
@ -238,14 +651,14 @@ static int cmd_thumbnail(const char *in, const char *out, int size) {
for (uint32_t yy = y0; yy < y1; yy++) for (uint32_t yy = y0; yy < y1; yy++)
for (uint32_t xx = x0; xx < x1; xx++) for (uint32_t xx = x0; xx < x1; xx++)
sum += src[((size_t)yy * sw + xx) * ch + c]; sum += src[((size_t)yy * sw + xx) * ch + c];
uint64_t n = (uint64_t)(y1 - y0) * (x1 - x0); uint64_t nn = (uint64_t)(y1 - y0) * (x1 - x0);
dst[((size_t)y * dw + x) * ch + c] = (uint8_t)((sum + n / 2) / n); dst[((size_t)y * dw + x) * ch + c] = (uint8_t)((sum + nn / 2) / nn);
} }
} }
} }
int ok = stbi_write_png(out, (int)dw, (int)dh, ch, dst, (int)dw * ch); int ok = stbi_write_png(out, (int)dw, (int)dh, ch, dst, (int)dw * ch);
free(dst); free(dst);
yapf_free(img); free(src);
if (!ok) fprintf(stderr, "yapf: cannot write %s\n", out); if (!ok) fprintf(stderr, "yapf: cannot write %s\n", out);
return ok ? 0 : 1; return ok ? 0 : 1;
} }
@ -285,7 +698,7 @@ static const char *ext_name(const char *path) {
* and a lossless check. */ * and a lossless check. */
static int cmd_compare(int n, char **paths) { static int cmd_compare(int n, char **paths) {
int rc = 0; int rc = 0;
double tot_src = 0, tot_yapf = 0, tot_src_t = 0, tot_yapf_t = 0; double tot_src = 0, tot_yapf = 0, tot_src_t = 0, tot_yapf_t = 0, tot_ent = 0;
for (int i = 0; i < n; i++) { for (int i = 0; i < n; i++) {
const char *path = paths[i]; const char *path = paths[i];
if (is_yapf(path)) { if (is_yapf(path)) {
@ -310,7 +723,7 @@ static int cmd_compare(int n, char **paths) {
memset(&img, 0, sizeof(img)); memset(&img, 0, sizeof(img));
img.width = (uint32_t)w; img.height = (uint32_t)h; img.channels = (uint8_t)ch; img.width = (uint32_t)w; img.height = (uint32_t)h; img.channels = (uint8_t)ch;
img.mip_levels = 1; img.pixels = px; img.flags = YAPF_FLAG_SRGB; img.mip_levels = 1; img.pixels = px; img.flags = YAPF_FLAG_SRGB;
img.gpu_format = gpu_hint(ch, 1); img.gpu_format = gpu_hint(ch, 1, YAPF_SAMPLE_U8);
void *enc = NULL; void *enc = NULL;
size_t esz = 0; size_t esz = 0;
@ -339,6 +752,28 @@ static int cmd_compare(int n, char **paths) {
printf(" %-5s %10s %5.1f%% of raw decode %8.3f ms %.1fx faster, %.0f%% of the %s size\n", printf(" %-5s %10s %5.1f%% of raw decode %8.3f ms %.1fx faster, %.0f%% of the %s size\n",
"YAPF", fmt_kb((double)esz, b), 100.0 * esz / raw, t_one * 1e3, "YAPF", fmt_kb((double)esz, b), 100.0 * esz / raw, t_one * 1e3,
t_src / t_one, 100.0 * esz / fsz, ext_name(path)); t_src / t_one, 100.0 * esz / fsz, ext_name(path));
/* version 2 with the optional Huffman planes */
{
yapf_texture_t v;
void *data[1] = { px };
size_t esz2 = 0;
void *enc2 = NULL;
double t_e;
memset(&v, 0, sizeof(v));
v.width = (uint32_t)w; v.height = (uint32_t)h; v.depth = 1; v.layers = 1; v.frames = 1;
v.mip_levels = 1; v.channels = (uint8_t)ch; v.flags = YAPF_FLAG_SRGB; v.data = data;
if (yapf_tex_encode(&v, YAPF_ENC_ENTROPY, &enc2, &esz2) == YAPF_OK) {
yapf_texture_t *c2 = yapf_tex_load_memory(enc2, esz2, 0, 1, NULL);
if (!c2 || memcmp(c2->data[0], px, (size_t)w * h * ch) != 0) lossless = 0;
yapf_tex_free(c2);
YAPF_TIME_BEST(t_e, yapf_tex_free(yapf_tex_load_memory(enc2, esz2, 0, 1, NULL)));
printf(" %-5s %10s %5.1f%% of raw decode %8.3f ms %.1fx faster, %.0f%% of the %s size\n",
"YAPF+E", fmt_kb((double)esz2, c), 100.0 * esz2 / raw, t_e * 1e3,
t_src / t_e, 100.0 * esz2 / fsz, ext_name(path));
tot_ent += (double)esz2;
yapf_free_buffer(enc2);
}
}
printf(" YAPF on all cores: decode %.3f ms (%.1f GB/s), encode %.0f ms, %s\n\n", printf(" YAPF on all cores: decode %.3f ms (%.1f GB/s), encode %.0f ms, %s\n\n",
t_all * 1e3, raw / t_all / 1e9, t_enc * 1e3, t_all * 1e3, raw / t_all / 1e9, t_enc * 1e3,
lossless ? "lossless: yes" : "LOSSLESS CHECK FAILED"); lossless ? "lossless: yes" : "LOSSLESS CHECK FAILED");
@ -350,8 +785,10 @@ static int cmd_compare(int n, char **paths) {
} }
if (n > 1 && tot_yapf_t > 0) { if (n > 1 && tot_yapf_t > 0) {
char a[32], b[32]; char a[32], b[32];
printf("Total: originals %s, YAPF %s (%.0f%%); decoding %.1fx faster on one core\n", char e[32];
fmt_kb(tot_src, a), fmt_kb(tot_yapf, b), 100.0 * tot_yapf / tot_src, tot_src_t / tot_yapf_t); printf("Total: originals %s, YAPF %s (%.0f%%), YAPF+E %s (%.0f%%); decoding %.1fx faster on one core\n",
fmt_kb(tot_src, a), fmt_kb(tot_yapf, b), 100.0 * tot_yapf / tot_src,
fmt_kb(tot_ent, e), 100.0 * tot_ent / tot_src, tot_src_t / tot_yapf_t);
} }
return rc; return rc;
} }
@ -359,96 +796,131 @@ static int cmd_compare(int n, char **paths) {
static void usage(void) { static void usage(void) {
fprintf(stderr, fprintf(stderr,
"usage: yapf <input> <output> [options] convert (formats by extension)\n" "usage: yapf <input> <output> [options] convert (formats by extension)\n"
" yapf info <file.yapf> show header information\n" " yapf pack <cube|array|volume|anim> <inputs...> <out.yapf> [options]\n"
" yapf info <file.yapf> header, tables and notes\n"
" yapf verify <file.yapf> check every checksum\n"
" yapf bench <file.yapf> time decoding\n" " yapf bench <file.yapf> time decoding\n"
" yapf thumbnail <in.yapf> <out.png> [size] preview, default 256 px\n" " yapf thumbnail <in.yapf> <out.png> [size] preview, default 256 px\n"
" yapf compare <image.png> [more...] size and speed vs the original format\n" " yapf compare <image.png> [more...] size and speed vs the original format\n"
"\n" "\n"
"reads: .yapf .png .jpg .bmp .tga .gif .psd .pnm .hdr\n" "reads: .yapf .png (8/16-bit) .jpg .bmp .tga .gif .psd .pnm .hdr\n"
"writes: .yapf .png .jpg .bmp .tga\n" "writes: .yapf .png .jpg .bmp .tga .hdr\n"
"\n" "\n"
"options when writing .yapf:\n" "options when writing .yapf:\n"
" --mips store a full mip chain\n" " --mips store a full mip chain (sRGB averaged in linear light)\n"
" --coverage REF mips keep alpha-test coverage at alpha REF (0-255)\n"
" --format F GPU blocks: bc1 bc3 bc4 bc5 bc6h bc7\n"
" --entropy Huffman-coded planes: smaller, slower to decode\n"
" --linear pixels are linear, not sRGB\n" " --linear pixels are linear, not sRGB\n"
" --premult alpha is premultiplied\n" " --premult alpha is premultiplied\n"
" --linear-alpha alpha is linear even in an sRGB image\n"
" --normal gl|dx normal map (OpenGL +Y / DirectX -Y); implies linear\n"
" --map a,b,c,d channel meanings: r g b a luma nx ny nz rough gloss metal\n"
" ao height spec emissive opacity mask cavity thickness id -\n"
" --primaries P bt709 p3 bt2020 aces acescg adobe\n"
" --transfer T srgb linear gamma22 pq hlg bt1886\n"
" --8bit read 16-bit images as 8-bit\n"
" --float32 keep .hdr input as 32-bit float (default half)\n"
" --frame-ms N animation frame time (pack anim), --loops N\n"
" --author S --license S --source S --note TAG=text\n"
" --v2 write version 2 even for a plain 8-bit image\n"
"options when reading .yapf:\n" "options when reading .yapf:\n"
" --mip N export mip level N\n"); " --mip N --layer N --face N --frame N --slice N which image to export\n");
}
static int parse_opts(int argc, char **argv, int start, opts_t *o) {
static const char *prim[] = { "", "bt709", "p3", "bt2020", "aces", "acescg", "adobe" };
static const char *tf[] = { "", "srgb", "linear", "gamma22", "pq", "hlg", "bt1886" };
memset(o, 0, sizeof(*o));
o->srgb = 1;
o->format = -1;
o->loops = -1;
for (int i = start; i < argc; i++) {
const char *a = argv[i], *v = i + 1 < argc ? argv[i + 1] : NULL;
if (strcmp(a, "--mips") == 0) o->mips = 1;
else if (strcmp(a, "--linear") == 0) o->srgb = 0;
else if (strcmp(a, "--premult") == 0) o->premult = 1;
else if (strcmp(a, "--entropy") == 0) o->entropy = 1;
else if (strcmp(a, "--v2") == 0) o->v2 = 1;
else if (strcmp(a, "--8bit") == 0) o->eight_bit = 1;
else if (strcmp(a, "--float32") == 0) o->float32 = 1;
else if (strcmp(a, "--linear-alpha") == 0) o->linear_alpha = 1;
else if (!v) return 0;
else if (strcmp(a, "--coverage") == 0) { o->coverage = atoi(v); o->mips = 1; i++;
if (o->coverage < 1 || o->coverage > 255) return 0; }
else if (strcmp(a, "--format") == 0) {
size_t k;
for (k = 0; k < sizeof(fmt_names) / sizeof(fmt_names[0]); k++)
if (strcmp(v, fmt_names[k].name) == 0) o->format = fmt_names[k].v;
if (o->format < 0) return 0;
i++;
}
else if (strcmp(a, "--normal") == 0) {
o->normal = strcmp(v, "gl") == 0 ? YAPF_NORMAL_GL : strcmp(v, "dx") == 0 ? YAPF_NORMAL_DX : 0;
if (!o->normal) return 0;
i++;
}
else if (strcmp(a, "--map") == 0) { if (!parse_map(v, o->map)) return 0; o->have_map = 1; i++; }
else if (strcmp(a, "--primaries") == 0 || strcmp(a, "--transfer") == 0) {
const char **names = a[2] == 'p' ? prim : tf;
int found = 0;
for (int k = 1; k < 7; k++) if (strcmp(v, names[k]) == 0) found = k;
if (!found) return 0;
if (a[2] == 'p') o->primaries = found; else o->transfer = found;
i++;
}
else if (strcmp(a, "--frame-ms") == 0) { o->frame_ms = atof(v); i++; }
else if (strcmp(a, "--loops") == 0) { o->loops = atoi(v); i++; }
else if (strcmp(a, "--author") == 0) { o->author = v; i++; }
else if (strcmp(a, "--license") == 0) { o->license = v; i++; }
else if (strcmp(a, "--source") == 0) { o->source = v; i++; }
else if (strcmp(a, "--note") == 0) { if (o->nnotes == 16) return 0; o->notes[o->nnotes++] = v; i++; }
else if (strcmp(a, "--mip") == 0) { o->mip = atoi(v); i++; }
else if (strcmp(a, "--layer") == 0) { o->layer = atoi(v); i++; }
else if (strcmp(a, "--face") == 0) { o->face = atoi(v); i++; }
else if (strcmp(a, "--frame") == 0) { o->frame = atoi(v); i++; }
else if (strcmp(a, "--slice") == 0) { o->slice = atoi(v); i++; }
else return 0;
}
if (o->mip < 0 || o->layer < 0 || o->face < 0 || o->frame < 0 || o->slice < 0) return 0;
return 1;
} }
int main(int argc, char **argv) { int main(int argc, char **argv) {
if (argc == 3 && strcmp(argv[1], "info") == 0) return cmd_info(argv[2]); if (argc == 3 && strcmp(argv[1], "info") == 0) return cmd_info(argv[2]);
if (argc == 3 && strcmp(argv[1], "verify") == 0) return cmd_verify(argv[2]);
if (argc == 3 && strcmp(argv[1], "bench") == 0) return cmd_bench(argv[2]); if (argc == 3 && strcmp(argv[1], "bench") == 0) return cmd_bench(argv[2]);
if (argc >= 3 && strcmp(argv[1], "compare") == 0) return cmd_compare(argc - 2, argv + 2); if (argc >= 3 && strcmp(argv[1], "compare") == 0) return cmd_compare(argc - 2, argv + 2);
if ((argc == 4 || argc == 5) && strcmp(argv[1], "thumbnail") == 0) if ((argc == 4 || argc == 5) && strcmp(argv[1], "thumbnail") == 0)
return cmd_thumbnail(argv[2], argv[3], argc == 5 ? atoi(argv[4]) : 256); return cmd_thumbnail(argv[2], argv[3], argc == 5 ? atoi(argv[4]) : 256);
if (argc < 3) { usage(); return 2; }
const char *in = argv[1], *out = argv[2]; opts_t o;
int mips = 0, srgb = 1, premult = 0, mip_out = 0; if (argc >= 5 && strcmp(argv[1], "pack") == 0) {
for (int i = 3; i < argc; i++) { int first_opt = 3;
if (strcmp(argv[i], "--mips") == 0) mips = 1; while (first_opt < argc && strncmp(argv[first_opt], "--", 2) != 0) first_opt++;
else if (strcmp(argv[i], "--linear") == 0) srgb = 0; int n = first_opt - 3 - 1; /* inputs, then the output */
else if (strcmp(argv[i], "--premult") == 0) premult = 1; if (n < 1 || !parse_opts(argc, argv, first_opt, &o)) { usage(); return 2; }
else if (strcmp(argv[i], "--mip") == 0 && i + 1 < argc) mip_out = atoi(argv[++i]); const char *out = argv[first_opt - 1];
else { usage(); return 2; } if (!is_yapf(out)) { fprintf(stderr, "yapf: pack writes a .yapf file\n"); return 2; }
} if (o.frame_ms <= 0) o.frame_ms = 100;
yapf_texture_t *t = assemble(argv[2], n, argv + 3, &o);
/* Read */ if (!t) return 1;
yapf_image_t src; int rc = write_yapf(t, out, &o, 0);
yapf_image_t *loaded = NULL; yapf_tex_free(t);
uint8_t *stb_px = NULL; return rc;
memset(&src, 0, sizeof(src)); }
if (is_yapf(in)) { if (argc < 3 || !parse_opts(argc, argv, 3, &o)) { usage(); return 2; }
loaded = yapf_load_mt(in, 0);
if (!loaded) { fprintf(stderr, "yapf: cannot read %s (missing, corrupt or not YAPF v1)\n", in); return 1; } const char *in = argv[1], *out = argv[2];
src = *loaded; int from_yapf;
} else { yapf_texture_t *t = read_input(in, &o, &from_yapf);
int w, h, ch; if (!t) return 1;
if (stbi_is_16_bit(in)) int rc;
fprintf(stderr, "yapf: note: %s has 16-bit channels; YAPF stores 8 bits per channel\n", in); if (is_yapf(out)) {
stb_px = stbi_load(in, &w, &h, &ch, 0); rc = write_yapf(t, out, &o, from_yapf);
if (!stb_px) { fprintf(stderr, "yapf: cannot read %s: %s\n", in, stbi_failure_reason()); return 1; } } else {
src.width = (uint32_t)w; src.height = (uint32_t)h; src.channels = (uint8_t)ch; rc = export_image(t, in, out, &o);
src.pixels = stb_px; src.mip_levels = 1; }
src.flags = srgb ? YAPF_FLAG_SRGB : 0; yapf_tex_free(t);
src.gpu_format = gpu_hint(ch, srgb);
}
/* Write */
int rc = 0;
if (is_yapf(out)) {
yapf_image_t dst = src;
if (!loaded) {
if (premult) dst.flags |= YAPF_FLAG_PREMULT_ALPHA;
}
dst.mips = NULL;
dst.mip_levels = 1;
if (mips && build_mips(&dst) != 0) { fprintf(stderr, "yapf: out of memory\n"); rc = 1; }
if (!rc && yapf_save(out, &dst) != YAPF_OK) { fprintf(stderr, "yapf: cannot write %s\n", out); rc = 1; }
if (dst.mips) {
for (int m = 1; m < dst.mip_levels; m++) free(dst.mips[m]);
free(dst.mips);
}
} else {
const uint8_t *px = src.pixels;
uint32_t w = src.width, h = src.height;
if (mip_out > 0) {
if (!loaded || mip_out >= src.mip_levels) {
fprintf(stderr, "yapf: %s has no mip level %d\n", in, mip_out);
rc = 1;
} else {
px = src.mips[mip_out];
w = src.width >> mip_out ? src.width >> mip_out : 1;
h = src.height >> mip_out ? src.height >> mip_out : 1;
}
}
if (!rc && !write_image(out, px, (int)w, (int)h, src.channels)) {
fprintf(stderr, "yapf: cannot write %s\n", out);
rc = 1;
}
}
yapf_free(loaded);
stbi_image_free(stb_px);
return rc; return rc;
} }

3248
yapf.c
View file

File diff suppressed because it is too large Load diff

359
yapf.h
View file

@ -38,6 +38,13 @@
* loader can upload the decoded buffer straight to a texture without any * loader can upload the decoded buffer straight to a texture without any
* additional transcoding step. * additional transcoding step.
* *
* Two versions exist. Version 1 (below) is a single 8-bit 2D image with
* optional mips. Version 2 is the texture container: GPU block payloads
* (BCn / ETC2 / ASTC), 16-bit and float samples, cubemaps, arrays and
* volumes, channel meanings, per-tile checksums, optional entropy-coded
* planes, notes (metadata) and animation frames. Its complete
* specification is docs/FORMAT.md; a summary follows the v1 spec.
*
* ========================================================================= * =========================================================================
* FILE FORMAT (version 1) * FILE FORMAT (version 1)
* ========================================================================= * =========================================================================
@ -140,6 +147,37 @@
* ───────────────────────────────────────────────── * ─────────────────────────────────────────────────
* bit 0 PREMULT_ALPHA alpha is premultiplied into RGB * bit 0 PREMULT_ALPHA alpha is premultiplied into RGB
* bit 1 SRGB pixels are in sRGB gamma (informational only) * bit 1 SRGB pixels are in sRGB gamma (informational only)
* bit 2 COVERAGE (v2) mips keep alpha-test coverage at alpha_ref
*
* =========================================================================
* FILE FORMAT (version 2) — summary; the full text is docs/FORMAT.md
* =========================================================================
*
* HEADER (64 bytes)
* 0x00 "YAPF" 0x04 version 2 0x05 channels 0x06 format
* 0x07 mips 0x08 flags 0x09 payload 0x0A sample 0x0B shape
* 0x0C width u32 0x10 height u32 0x14 depth u32
* 0x18 layers u16 0x1A frames u16 0x1C needs u32
* 0x20 channel map[4] 0x24 linear mask 0x25 normal 0x26 primaries
* 0x27 transfer 0x28 mip filter 0x29 alpha ref 0x2A loop count u16
* 0x2C notes size u32 0x30 file size u64
* 0x38 tables check u32 (yapf-sum of bytes [0x40, data start))
* 0x3C header check u32 (yapf-sum of bytes [0, 0x3C))
* SURFACE TABLE 16 bytes per surface: offset u64, size u32,
* tile-table check u32. Surface s of (frame, layer,
* face, mip) is ((frame·layers + layer)·faces + face)·mips + mip
* FRAME TABLE u32 duration in µs per frame (only when frames > 1)
* NOTES tag[4], length u32, bytes — repeated
* SURFACE DATA smallest mips first, so a file prefix is a complete
* low-resolution texture. Per surface: u32 book size,
* code book (Huffman tables by slot and context, only
* with entropy coding), tile table (size u32 with bit 31 =
* packed, yapf-sum u32 per tile), then the tiles. Pixel
* tiles are the v1 tiles plus byte planes for wide samples
* and entropy-coded planes; block tiles hold 16 × 16 GPU
* blocks, raw or packed lane by lane.
* NEEDS a decoder must refuse a file with a needs bit it does
* not implement; everything else may be ignored safely.
*/ */
#ifndef YAPF_H #ifndef YAPF_H
@ -156,11 +194,58 @@
#define YAPF_GPU_R8 0x03 #define YAPF_GPU_R8 0x03
#define YAPF_GPU_SRGB8_A8 0x04 #define YAPF_GPU_SRGB8_A8 0x04
#define YAPF_GPU_SRGB8 0x05 #define YAPF_GPU_SRGB8 0x05
/* v2 pixel hints for wide samples */
#define YAPF_GPU_RGBA16 0x06
#define YAPF_GPU_RGB16 0x07
#define YAPF_GPU_RG16 0x08
#define YAPF_GPU_R16 0x09
#define YAPF_GPU_RGBA16F 0x0A
#define YAPF_GPU_RGB16F 0x0B
#define YAPF_GPU_RG16F 0x0C
#define YAPF_GPU_R16F 0x0D
#define YAPF_GPU_RGBA32F 0x0E
#define YAPF_GPU_RGB32F 0x0F
#define YAPF_GPU_RG32F 0x10
#define YAPF_GPU_R32F 0x11
/* v2 GPU block formats (payload = YAPF_PAYLOAD_BLOCKS). The sRGB flag
* selects the _SRGB variant where the API has one. */
#define YAPF_GPU_BC1 0x40 /* RGB(A1), 8 bytes per 4×4 */
#define YAPF_GPU_BC2 0x41 /* RGBA, explicit 4-bit alpha */
#define YAPF_GPU_BC3 0x42 /* RGBA */
#define YAPF_GPU_BC4 0x43 /* R */
#define YAPF_GPU_BC4_SNORM 0x44
#define YAPF_GPU_BC5 0x45 /* RG (normal maps) */
#define YAPF_GPU_BC5_SNORM 0x46
#define YAPF_GPU_BC6H 0x47 /* RGB half float, unsigned */
#define YAPF_GPU_BC6H_SF 0x48 /* RGB half float, signed */
#define YAPF_GPU_BC7 0x49 /* RGB(A) */
#define YAPF_GPU_ETC2_RGB8 0x50
#define YAPF_GPU_ETC2_RGB8A1 0x51
#define YAPF_GPU_ETC2_RGBA8 0x52
#define YAPF_GPU_EAC_R11 0x53
#define YAPF_GPU_EAC_R11_SNORM 0x54
#define YAPF_GPU_EAC_RG11 0x55
#define YAPF_GPU_EAC_RG11_SNORM 0x56
#define YAPF_GPU_ASTC_4x4 0x60 /* ASTC: 0x60 + footprint index */
#define YAPF_GPU_ASTC_5x4 0x61
#define YAPF_GPU_ASTC_5x5 0x62
#define YAPF_GPU_ASTC_6x5 0x63
#define YAPF_GPU_ASTC_6x6 0x64
#define YAPF_GPU_ASTC_8x5 0x65
#define YAPF_GPU_ASTC_8x6 0x66
#define YAPF_GPU_ASTC_8x8 0x67
#define YAPF_GPU_ASTC_10x5 0x68
#define YAPF_GPU_ASTC_10x6 0x69
#define YAPF_GPU_ASTC_10x8 0x6A
#define YAPF_GPU_ASTC_10x10 0x6B
#define YAPF_GPU_ASTC_12x10 0x6C
#define YAPF_GPU_ASTC_12x12 0x6D
/* ── Flags ───────────────────────────────────────────────────────────── */ /* ── Flags ───────────────────────────────────────────────────────────── */
#define YAPF_FLAG_PREMULT_ALPHA 0x01 #define YAPF_FLAG_PREMULT_ALPHA 0x01
#define YAPF_FLAG_SRGB 0x02 #define YAPF_FLAG_SRGB 0x02
#define YAPF_FLAG_COVERAGE 0x04 /* v2: mips preserve alpha-test coverage */
/* ── Channel constants ───────────────────────────────────────────────── */ /* ── Channel constants ───────────────────────────────────────────────── */
@ -176,12 +261,16 @@
#define YAPF_ERR_IO -2 #define YAPF_ERR_IO -2
#define YAPF_ERR_OOM -3 #define YAPF_ERR_OOM -3
#define YAPF_ERR_CORRUPT -4 #define YAPF_ERR_CORRUPT -4
#define YAPF_ERR_UNSUPPORTED -5 /* valid file, feature this decoder lacks */
#define YAPF_ERR_CHECKSUM -6 /* a yapf-sum does not match */
#define YAPF_ERR_TRUNCATED -7 /* data lies beyond the bytes available yet */
/* ── Limits ──────────────────────────────────────────────────────────── */ /* ── Limits ──────────────────────────────────────────────────────────── */
#define YAPF_MAX_DIM 65535u #define YAPF_MAX_DIM 65535u
#define YAPF_MAX_MIPS 16u #define YAPF_MAX_MIPS 16u
#define YAPF_TILE_SIZE 64u #define YAPF_TILE_SIZE 64u
#define YAPF_BLOCK_TILE 16u /* v2 block tiles: 16 × 16 GPU blocks */
/* ── Image descriptor ────────────────────────────────────────────────── */ /* ── Image descriptor ────────────────────────────────────────────────── */
@ -196,6 +285,150 @@ typedef struct {
uint8_t **mips; /* mips[0]=base alias, mips[1..mip_levels-1] owned */ uint8_t **mips; /* mips[0]=base alias, mips[1..mip_levels-1] owned */
} yapf_image_t; } yapf_image_t;
/* ── Version 2: texture description ──────────────────────────────────── */
/* payload */
#define YAPF_PAYLOAD_PIXELS 0 /* lossless YAPF tiles */
#define YAPF_PAYLOAD_BLOCKS 1 /* GPU blocks, uploaded as they are */
/* sample: how one channel value is stored (pixel payload); for block
* payloads, the type of the decoded texels (U8, or F16 for BC6H) */
#define YAPF_SAMPLE_U8 0
#define YAPF_SAMPLE_U16 1
#define YAPF_SAMPLE_F16 2 /* IEEE half, as uint16_t bits */
#define YAPF_SAMPLE_F32 3
/* shape */
#define YAPF_SHAPE_2D 0
#define YAPF_SHAPE_CUBE 1 /* 6 faces: +X −X +Y −Y +Z −Z; with layers
> 1 a cube array */
#define YAPF_SHAPE_VOLUME 2 /* depth slices; mips halve depth too */
/* needs: features a decoder must implement to read the file */
#define YAPF_NEED_WIDE 0x01 /* sample other than U8 (pixel payload) */
#define YAPF_NEED_BLOCKS 0x02 /* GPU block payload */
#define YAPF_NEED_ENTROPY 0x04 /* code books: entropy planes, packed blocks */
#define YAPF_NEED_LAYOUT 0x08 /* cube, volume or layers > 1 */
#define YAPF_NEED_FRAMES 0x10 /* frames > 1 */
#define YAPF_NEED_ALL 0x1F /* everything this library implements */
/* channel meanings (channel_map[c]); 0 = default for the channel count */
#define YAPF_CH_DEFAULT 0
#define YAPF_CH_RED 1
#define YAPF_CH_GREEN 2
#define YAPF_CH_BLUE 3
#define YAPF_CH_ALPHA 4
#define YAPF_CH_LUMA 5
#define YAPF_CH_NORMAL_X 6
#define YAPF_CH_NORMAL_Y 7
#define YAPF_CH_NORMAL_Z 8
#define YAPF_CH_ROUGHNESS 9
#define YAPF_CH_GLOSS 10 /* smoothness = 1 − roughness */
#define YAPF_CH_METALNESS 11
#define YAPF_CH_OCCLUSION 12 /* ambient occlusion */
#define YAPF_CH_HEIGHT 13
#define YAPF_CH_SPECULAR 14
#define YAPF_CH_EMISSIVE 15
#define YAPF_CH_OPACITY 16 /* alpha-test / cut-out mask */
#define YAPF_CH_MASK 17 /* generic mask */
#define YAPF_CH_CAVITY 18
#define YAPF_CH_THICKNESS 19
#define YAPF_CH_ID 20 /* material or object id */
#define YAPF_CH_UNUSED 255
/* normal: bits 0–1 convention, bits 2–3 encoding */
#define YAPF_NORMAL_NONE 0x00
#define YAPF_NORMAL_GL 0x01 /* +Y up (OpenGL, Blender, Unity) */
#define YAPF_NORMAL_DX 0x02 /* −Y (DirectX, Unreal) */
#define YAPF_NORMAL_XYZ 0x00
#define YAPF_NORMAL_XY 0x04 /* Z = sqrt(1 − x² − y²) is reconstructed */
#define YAPF_NORMAL_OCT 0x08 /* octahedral */
/* primaries / transfer (0 = unspecified: the sRGB flag decides) */
#define YAPF_PRIMARIES_UNSPECIFIED 0
#define YAPF_PRIMARIES_BT709 1 /* = sRGB */
#define YAPF_PRIMARIES_DISPLAY_P3 2
#define YAPF_PRIMARIES_BT2020 3
#define YAPF_PRIMARIES_ACES_AP0 4
#define YAPF_PRIMARIES_ACES_AP1 5 /* ACEScg */
#define YAPF_PRIMARIES_ADOBE_RGB 6
#define YAPF_TRANSFER_UNSPECIFIED 0
#define YAPF_TRANSFER_SRGB 1
#define YAPF_TRANSFER_LINEAR 2
#define YAPF_TRANSFER_GAMMA22 3
#define YAPF_TRANSFER_PQ 4 /* SMPTE ST 2084 */
#define YAPF_TRANSFER_HLG 5
#define YAPF_TRANSFER_BT1886 6 /* gamma 2.4 */
/* mip filter (bits 0–6) | YAPF_MIP_LINEAR_LIGHT */
#define YAPF_MIP_UNKNOWN 0
#define YAPF_MIP_BOX 1
#define YAPF_MIP_TENT 2
#define YAPF_MIP_KAISER 3
#define YAPF_MIP_LANCZOS 4
#define YAPF_MIP_MITCHELL 5
#define YAPF_MIP_AUTHORED 6 /* made by hand / by an external tool */
#define YAPF_MIP_LINEAR_LIGHT 0x80 /* sRGB channels were filtered in linear light */
/* load flags */
#define YAPF_LOAD_TRUST 0x01 /* skip checksum verification */
/* encode flags */
#define YAPF_ENC_ENTROPY 0x01 /* allow entropy-coded planes and packed
blocks where they are smaller: ~10 %
smaller photos, ~3× smaller GPU blocks
on disk; slower to decode */
/* A note (metadata entry). Well-known tags: "AUTH" author, "LICN"
* licence, "SRCE" source, "TOOL" creating tool, "TITL" title, "DESC"
* description, "DATE" ISO 8601 date (all UTF-8 text); "ICCP" ICC profile;
* "ORNT" orientation (1 byte); "DPI " 2 × u32 milli-dots per inch;
* "CLL " 2 × u16 max content / max frame-average light level (cd/m²).
* Tags that start with a lowercase letter belong to applications. */
typedef struct {
char tag[4];
uint32_t size;
uint8_t *data;
} yapf_note_t;
/*
* A version 2 texture. Surfaces are numbered
* s = ((frame × layers + layer) × faces + face) × mip_levels + mip
* (faces = 6 for cubes, else 1) and data[s] holds the whole surface:
* pixels: width_m × height_m × depth_m × channels samples, rows top to
* bottom, slices in order; samples in the native byte order
* (uint8_t, uint16_t, half bits as uint16_t, float);
* blocks: ceil(width_m / bw) × ceil(height_m / bh) blocks per slice,
* row-major — exactly what glCompressedTexImage / Vulkan take.
* width_m = max(1, width >> m), likewise height and (volumes) depth.
*/
typedef struct {
uint32_t width, height, depth; /* depth: volumes only, else 1 */
uint16_t layers; /* array layers, >= 1 */
uint16_t frames; /* animation frames, >= 1 */
uint8_t shape; /* YAPF_SHAPE_* */
uint8_t mip_levels; /* 1–16 */
uint8_t payload; /* YAPF_PAYLOAD_* */
uint8_t format; /* YAPF_GPU_*: block format, or hint */
uint8_t sample; /* YAPF_SAMPLE_* */
uint8_t channels; /* 1–4 */
uint8_t flags; /* YAPF_FLAG_* */
uint8_t channel_map[4]; /* YAPF_CH_* per channel */
uint8_t linear_mask; /* bit c: channel c is linear even in
an sRGB image (e.g. alpha, masks) */
uint8_t normal; /* YAPF_NORMAL_* */
uint8_t primaries, transfer; /* YAPF_PRIMARIES_*, YAPF_TRANSFER_* */
uint8_t mip_filter; /* YAPF_MIP_* */
uint8_t alpha_ref; /* coverage reference (with COVERAGE) */
uint16_t loop_count; /* animation loops, 0 = forever */
uint32_t *frame_us; /* frames durations in µs, or NULL */
uint32_t note_count;
yapf_note_t *notes;
void **data; /* one pointer per surface */
void *internal_; /* library use: file tables, ownership */
} yapf_texture_t;
/* ── Public API ──────────────────────────────────────────────────────── */ /* ── Public API ──────────────────────────────────────────────────────── */
#ifdef __cplusplus #ifdef __cplusplus
@ -249,6 +482,132 @@ void yapf_free_buffer(void *data);
*/ */
void yapf_free(yapf_image_t *img); void yapf_free(yapf_image_t *img);
/*
* The v1 functions above also read version 2 files that hold 8-bit pixels
* of a 2D, cube or array texture: they return frame 0, layer 0, face 0
* with all its mips. yapf_save / yapf_encode always write version 1.
*/
/* ── Version 2: textures ─────────────────────────────────────────────── */
/*
* yapf_tex_create — a library-owned texture with the description of
* `desc` (every field up to loop_count; notes, frame times and data are
* not copied) and zeroed storage for every surface. frame_us is
* allocated (zeroed) when frames > 1. NULL on invalid description / OOM.
*/
yapf_texture_t *yapf_tex_create(const yapf_texture_t *desc);
/* yapf_tex_free — release a texture made by the library. NULL-safe. */
void yapf_tex_free(yapf_texture_t *tex);
/* Surface helpers. */
uint32_t yapf_tex_faces(const yapf_texture_t *tex); /* 6 or 1 */
uint32_t yapf_tex_surface_count(const yapf_texture_t *tex);
uint32_t yapf_tex_surface(const yapf_texture_t *tex, uint32_t frame,
uint32_t layer, uint32_t face, uint32_t mip);
void yapf_tex_mip_size(const yapf_texture_t *tex, uint32_t mip,
uint32_t *w, uint32_t *h, uint32_t *d);
size_t yapf_tex_surface_bytes(const yapf_texture_t *tex, uint32_t mip);
size_t yapf_sample_bytes(uint8_t sample); /* 1, 2 or 4 */
/* Notes: add (library-owned textures only; data is copied) and find. */
int yapf_tex_add_note(yapf_texture_t *tex, const char tag[4],
const void *data, uint32_t size);
const yapf_note_t *yapf_tex_find_note(const yapf_texture_t *tex, const char tag[4]);
/*
* yapf_tex_encode / yapf_tex_save — write a version 2 file. `flags` is
* YAPF_ENC_*. Works on library-owned and caller-built textures alike
* (nothing is taken over). Returns YAPF_OK or a YAPF_ERR_* code.
*/
int yapf_tex_encode(const yapf_texture_t *tex, int flags,
void **out_data, size_t *out_size);
int yapf_tex_save(const char *filename, const yapf_texture_t *tex, int flags);
/*
* yapf_tex_load / yapf_tex_load_memory — read a version 1 or 2 file and
* decode every surface on up to `threads` threads (0 = one per core).
* `flags` is YAPF_LOAD_*. On failure NULL is returned and *err (if not
* NULL) receives the YAPF_ERR_* reason.
*/
yapf_texture_t *yapf_tex_load(const char *filename, int flags, int threads, int *err);
yapf_texture_t *yapf_tex_load_memory(const void *buffer, size_t size,
int flags, int threads, int *err);
/*
* Streaming and partial decoding.
*
* yapf_tex_open — read only the header, tables and notes. `avail` may be
* less than the whole file (a download in progress): it must cover the
* front matter, else YAPF_ERR_TRUNCATED. All data[] pointers are NULL.
*
* yapf_tex_stream_bytes — how many leading bytes of the file hold every
* surface of mip `first_mip` and smaller (the file stores small mips
* first, so these are a usable texture on their own).
*
* yapf_tex_decode_surface — decode surface s into tex->data[s]
* (allocated). `buffer`/`avail` is the file or a prefix of it; returns
* YAPF_ERR_TRUNCATED when the surface is not in it yet.
*
* yapf_tex_tile_grid / yapf_tex_decode_tile — virtual texturing: the
* surfaces of mip m are cut into tiles (64 × 64 pixels, or 16 × 16
* blocks), tx across, ty down, per slice. decode_tile writes one tile
* into `out`, rows `stride` bytes apart, and reports its pixel rectangle.
*/
yapf_texture_t *yapf_tex_open(const void *buffer, size_t avail, int flags, int *err);
uint64_t yapf_tex_stream_bytes(const yapf_texture_t *tex, uint32_t first_mip);
int yapf_tex_decode_surface(yapf_texture_t *tex, const void *buffer, size_t avail,
uint32_t surface, int threads);
void yapf_tex_tile_grid(const yapf_texture_t *tex, uint32_t mip,
uint32_t *tiles_x, uint32_t *tiles_y);
int yapf_tex_decode_tile(const yapf_texture_t *tex, const void *buffer, size_t avail,
uint32_t surface, uint32_t tx, uint32_t ty, uint32_t slice,
void *out, size_t stride,
uint32_t *x, uint32_t *y, uint32_t *w, uint32_t *h);
/*
* yapf_tex_build_mips — replace the mips of a library-owned pixel texture
* with a chain made from mip 0: `levels` levels (0 = down to 1 × 1).
* 2 × 2 (× 2 for volumes) box filter; sRGB channels are averaged in
* linear light. alpha_ref > 0 rescales alpha on every mip so the share
* of texels above alpha_ref stays that of mip 0 (fences and foliage do
* not thin out); it sets YAPF_FLAG_COVERAGE. Records the filter used.
*/
int yapf_tex_build_mips(yapf_texture_t *tex, uint32_t levels, uint8_t alpha_ref);
/* ── GPU blocks ──────────────────────────────────────────────────────── */
/* Block footprint and size of a YAPF_GPU_* block format; 0 if not one. */
int yapf_block_info(uint8_t format, uint32_t *block_w, uint32_t *block_h,
uint32_t *block_bytes);
/*
* yapf_tex_compress — a new library-owned texture holding `src` (pixel
* payload) as GPU blocks. Built-in encoders: BC1, BC3, BC4, BC5, BC7 and
* BC6H (unsigned). LDR formats take 8-bit pixels; BC6H takes 16-bit,
* half or float pixels. For other formats bring blocks from your own
* encoder: a payload = BLOCKS texture with data[] filled.
*/
int yapf_tex_compress(const yapf_texture_t *src, uint8_t format,
yapf_texture_t **out);
/*
* yapf_blocks_decode — decode a w × h image of BC1–BC7 blocks (previews,
* tools). LDR formats write RGBA8 (BC4 / BC5: R / RG, then 0, 255;
* snorm values shifted by +128); BC6H writes RGB half floats. Returns
* YAPF_ERR_UNSUPPORTED for ETC2 / EAC / ASTC.
*/
int yapf_blocks_decode(uint8_t format, const void *blocks, uint32_t w, uint32_t h,
void *out);
/* yapf-sum, the checksum used by version 2 files. */
uint32_t yapf_checksum(const void *data, size_t size);
/* Half float conversion helpers. */
uint16_t yapf_float_to_half(float f);
float yapf_half_to_float(uint16_t h);
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif