Skip to content

Commit 775ab43

Browse files
committed
docs: cover Rust, WASM, and APE benchmarks
1 parent 2a8c3bf commit 775ab43

8 files changed

Lines changed: 942 additions & 588 deletions

.dirtree-state

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
# Dirtree: Stateful directory trees in the CLI for humans and LLMs. https://github.com/pmarreck/dirtree
22
ver=1.2
33
annotate=[
4+
MEMORIES/WASM benchmarks must stream inputs and APE needs a clean environment = WASI and Cosmopolitan runtime requirements for benchmark adapters.
45
PLAN.md = Issue #1 plan: decode workflow + printable-binary-file.json container
6+
bm/benchmark-zig-opt = Benchmarks all available CLI variants, including Rust stdin, WASM, and APE adapters.
57
docs/plans/2026-06-26-printable-binary-file-container-design.md = Design spec: printable-binary-file.json container + web decode workflow (issue #1)
68
elixir/lib/printable_binary.ex = Elixir ~PB compile-time sigil: decodes printable-binary glyphs to raw bytes at compile time (reads character_map.txt via @external_resource); decode/1 runtime helper (whitespace-tolerant)
79
elixir/mix.exs = Mix project for the ~PB sigil demo (no deps)
@@ -15,5 +17,6 @@ annotate=[
1517
test/module_consumer.zig = Test fixture: minimal downstream importer of the printable_binary Zig module (mirrors how difz/blip consume it) for the FFI-symbol-leak test
1618
test/test_container = Container (.pbf.json) CLI round-trip + self-verify test, parameterized by IMPLEMENTATION_TO_TEST (issue #1)
1719
test/test_container_cross = Cross-impl container differential: impl-A container decodes via impl-B (MFIC, issue #1)
20+
test/test_docs_benchmark_contract = Checks docs coverage, mapping byte names, and benchmark implementation inventory.
1821
test/test_module_no_ffi_symbols = Regression test (nm oracle): importing the module must emit 0 pb_* symbols while the FFI static lib keeps all 12 — guards the difz-blocking duplicate-symbol bug
1922
]
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
The Emscripten/WASI CLI cannot open an arbitrary host file path under wazero's
2+
default sandbox. Benchmark adapters must redirect each generated input file to
3+
stdin and pass `-d` after wazero's `--` separator for decode.
4+
5+
On the Thelio Nix host, the Cosmopolitan APE reports `Exec format error` when
6+
launched directly from the development environment, but works when invoked with
7+
a minimal `env -i` PATH/HOME/TMPDIR—matching `test/printable-binary-ape-wrapper`.
8+
Use that clean-environment adapter for APE benchmarks so the executable itself,
9+
rather than Nix development-shell state, is measured.

PLAN.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,22 @@ maintained_by: agent
66

77
# PLAN — issue #1: decode workflow + printable-binary-file.json
88

9+
## Docs and benchmark parity — active (2026-07-17 EDT)
10+
11+
- [x] Document every supported implementation—especially Rust, WebAssembly, and
12+
the Cosmopolitan APE—in the primary README and implementation guide.
13+
Curiosity poke: distinguish Rust's deliberately minimal stdin codec CLI
14+
from the feature-complete C/Zig/Lua/Node CLI surface. (2026-07-17 01:56 PM EDT)
15+
- [x] Extend the comparative benchmark runner with Rust, WASM (wazero), and APE
16+
adapters, explicit selection, and availability reporting. Curiosity poke:
17+
preserve identical input/round-trip verification despite their different
18+
process invocation conventions. (2026-07-17 01:56 PM EDT)
19+
- [x] Add conventional ASCII byte names to all 256 README mapping rows, including
20+
control-byte mnemonics. Curiosity poke: retain a mechanically checked
21+
distinction between source-byte names and Unicode glyph names. (2026-07-17 01:56 PM EDT)
22+
- [x] Run the documentation/benchmark contracts and full relevant suite; update
23+
file notes. Commit once reattached to the `yolo` branch. (2026-07-17 01:56 PM EDT)
24+
925
GitHub: https://github.com/pmarreck/printable_binary/issues/1
1026
Dispatch: inbox/2026-06-26-handle-issue-1.md (from Einstein)
1127
Spec: docs/plans/2026-06-26-printable-binary-file-container-design.md

README.md

Lines changed: 337 additions & 280 deletions
Large diffs are not rendered by default.

README_IMPLEMENTATIONS.md

Lines changed: 101 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# PrintableBinary Implementations Guide
22

3-
A comprehensive guide to the LuaJIT, C, and Zig implementations of PrintableBinary - a tool for encoding binary data into human-readable UTF-8 strings and decoding them back.
3+
A guide to the LuaJIT, C, Cosmopolitan APE, Zig, Rust, JavaScript/Node, and WebAssembly implementations of PrintableBinarya tool for encoding binary data into human-readable UTF-8 and decoding it back.
44

55
## Overview
66

@@ -18,36 +18,53 @@ PrintableBinary is available in multiple high-performance implementations:
1818
- **CLI parity**: Same flags, environment variables, and character map behavior as the ELF build
1919
- **Great for distribution**: Ship one file (`printable-binary-ape.com`) and it just works
2020

21+
### 🕸️ **WebAssembly Implementation** (Portable Sandbox)
22+
- **Same C CLI core**: Built from `src/printable_binary.c` with Emscripten
23+
- **WASI runner**: Execute it through wazero on a host with no native installation
24+
- **Full CLI parity**: Same arguments and mapping behavior as the native C binary
25+
- **Measured honestly**: The benchmark includes wazero startup and runtime overhead
26+
2127
### 🦎 **Zig Implementation** (Modern, Memory-Safe)
2228
- **Memory-safe**: Zig's safety features catch bugs at compile time and runtime
2329
- **Cross-compilation**: Easy cross-compilation to many platforms from a single host
2430
- **Fast compilation**: Incremental builds and fast compile times
2531
- **CLI parity**: Same flags and behavior as C implementation
2632

33+
### 🦀 **Rust Implementation** (Embedded Raw Codec)
34+
- **Byte-identical core**: Compile-time tables generated from `character_map.txt`
35+
- **Low-allocation API**: `encode_into` and `decode_into` reuse caller-owned buffers
36+
- **Transport-focused CLI**: Minimal stdin→stdout encode/decode path for Rust↔Zig integration
37+
- **Deliberately narrow surface**: Formatting, containers, runtime map overrides, and mapping reports remain the job of the full CLIs
38+
39+
### 🌐 **JavaScript / Node.js Implementation** (Web and Automation)
40+
- **Shared core**: One module powers the browser UI and Node CLI
41+
- **Portable runtime**: Runs in browsers, Node.js, and compatible bundlers
42+
- **CLI parity**: Node exposes the full user-facing command surface
43+
2744
### **LuaJIT Implementation** (Original)
2845
- **Reference implementation**: Easy to modify and extend
2946
- **Well-tested**: Extensive test suite and battle-tested
3047
- **Development-friendly**: Rapid prototyping and debugging
3148

3249
## Performance Comparison
3350

34-
### Benchmark Results (Apple M4 Max)
51+
Performance depends on input mix, CPU, filesystem, runtime version, and whether a runtime must start. Rather than preserve a partial historical C-vs-Lua table as a project-wide claim, run the reproducible comparison locally:
3552

36-
| File Size | Operation | LuaJIT Time | C Time | C Speedup | C Improvement |
37-
|-----------|-----------|-------------|----------|-----------|---------------|
38-
| 1KB | Encode | 19.3 ms | 17.4 ms | 1.11x | 10.9% |
39-
| 1KB | Decode | 19.0 ms | 17.4 ms | 1.09x | 8.6% |
40-
| 100KB | Encode | 19.6 ms | 17.4 ms | 1.12x | 10.9% |
41-
| 100KB | Decode | 24.3 ms | 17.6 ms | 1.38x | **27.7%** |
42-
| 1MB | Encode | 23.9 ms | 20.3 ms | 1.18x | **15.4%** |
43-
| 1MB | Decode | 90.3 ms | 20.9 ms | 4.31x | **76.8%** |
53+
```bash
54+
./build
55+
nix develop -c cargo build --release --manifest-path rust/Cargo.toml
56+
nix develop -c ./bm/benchmark-zig-opt --list-impls
57+
nix develop -c ./bm/benchmark-zig-opt --quick --sizes "1M"
58+
```
4459

45-
### Performance Highlights
60+
| Implementation | Included in comparison runner | Measurement boundary |
61+
| --- | --- | --- |
62+
| C (native), Zig, LuaJIT, Node.js | Yes, direct when built | CLI process and file I/O |
63+
| APE | Yes, through a clean-environment adapter | Cosmopolitan process and file I/O |
64+
| Rust | Yes, through an stdin adapter | Rust process and stdin/stdout I/O |
65+
| WebAssembly | Yes, through wazero | WASM runtime startup plus CLI I/O |
4666

47-
- **Overall Encoding**: C is 1.14x faster (12.2% improvement)
48-
- **Overall Decoding**: C is 2.08x faster (52.0% improvement)
49-
- **Large Files**: Up to **6x speedup** for 1MB+ binary/random data decoding
50-
- **Memory Usage**: C implementation uses significantly less memory
67+
`--impls rust,wasm,ape` makes those three targets mandatory, so a missing artifact fails instead of producing a misleading partial result. The runner round-trips a deterministic random input before it invokes `hyperfine`; it also includes an ASCII input set. Rust's `cargo run --release --example bench` is a separate in-process codec microbenchmark and should not be compared directly with CLI timings.
5168

5269
## Quick Start
5370

@@ -80,6 +97,19 @@ make ape
8097
make test-ape
8198
```
8299

100+
### WebAssembly Implementation
101+
102+
```bash
103+
# Build the standalone WASM CLI from the C core.
104+
./build wasm
105+
106+
# Run it through the WASI runtime supplied by the development shell.
107+
nix develop -c wazero run bin/printable-binary.wasm < input.bin > encoded.pbt
108+
nix develop -c wazero run bin/printable-binary.wasm -- -d < encoded.pbt > restored.bin
109+
```
110+
111+
WASM has the C CLI's feature set. The runtime does not automatically inherit host environment variables; pass `--env=NAME=value` to wazero when a variable such as `PRINTABLE_BINARY_MUTE_STATS` matters.
112+
83113
### Zig Implementation
84114

85115
```bash
@@ -92,6 +122,21 @@ zig build -Doptimize=ReleaseFast
92122
./zig-out/bin/printable-binary-zig file.bin
93123
```
94124

125+
### Rust Implementation
126+
127+
```bash
128+
# Build the raw codec and its deliberately minimal stdin-only CLI.
129+
nix develop -c cargo build --release --manifest-path rust/Cargo.toml
130+
131+
rust/target/release/printable-binary-rs < file.bin > encoded.pbt
132+
rust/target/release/printable-binary-rs --decode < encoded.pbt > restored.bin
133+
134+
# Measure the in-process codec alone (not directly comparable to CLI timings).
135+
nix develop -c cargo run --release --manifest-path rust/Cargo.toml --example bench
136+
```
137+
138+
Use Rust as an embedded codec or a Rust↔Zig transport boundary. Its CLI currently supports only raw stdin encode and `-d`/`--decode`; use a full CLI for file arguments, format options, mapping reports, containers, or runtime map overrides.
139+
95140
### LuaJIT Implementation
96141

97142
```bash
@@ -142,11 +187,14 @@ nix build .#printableBinaryNative # ELF/Mach-O binary
142187
nix build .#printableBinaryApe # Cosmopolitan APE fat binary (x86_64 + arm64, pinned cosmocc 4.0.2)
143188
nix build .#printableBinaryWasm # WebAssembly module
144189
nix build .#default # Suite: native + APE + WASM
190+
191+
# Rust is a Cargo crate; the Nix development shell provides its offline toolchain.
192+
nix develop -c cargo build --release --manifest-path rust/Cargo.toml
145193
```
146194

147195
## Command-Line Usage
148196

149-
All compiled variants (ELF, APE, WASM via wazero) share **identical** command-line interfaces. Use whichever binary suits your platform (`./bin/printable-binary-c`, `./bin/printable-binary-ape.com`, etc.).
197+
The full compiled variants (native C, APE, Zig, and WASM via wazero) share the user-facing command-line interface. The Rust transport CLI is intentionally narrower: raw stdin encode plus `-d`/`--decode`.
150198

151199
### Basic Operations
152200

@@ -205,13 +253,15 @@ Input/Output:
205253
All binaries embed the canonical 256-entry map, so the `--mappings*` flags work even when `character_map.txt` is missing. If you place a custom map alongside the executable (or set `PRINTABLE_BINARY_MAP`), these options will reflect the override automatically. The override file should contain **exactly 256 lines**, each a single UTF-8 glyph (line 0 = byte 0x00, line 255 = byte 0xFF).
206254
```
207255

208-
### Environment Variables (All Implementations)
256+
### Environment Variables (Full CLIs)
209257

210258
```
211259
PRINTABLE_BINARY_MAP – points to an alternate character_map.txt
212260
PRINTABLE_BINARY_MUTE_STATS – set to 1/true/yes to suppress stderr statistics
213261
```
214262

263+
The Rust crate consumes its map at compile time and has no runtime map/environment configuration. When invoking WASM, pass variables explicitly with wazero's `--env` option because WASI does not inherit the host environment by default.
264+
215265
## When to Use Which Implementation
216266

217267
### Use C Implementation For:
@@ -228,10 +278,23 @@ PRINTABLE_BINARY_MUTE_STATS – set to 1/true/yes to suppress stderr statistics
228278

229279
**Cross-compilation** to other platforms
230280
**Memory-safe production** where safety is paramount
231-
**WebAssembly targets** (future capability)
281+
**A safe native CLI** with the complete option surface
232282
**Modern tooling** with built-in package manager
233283
**Environments** where C toolchains are unavailable
234284

285+
### Use APE or WebAssembly For:
286+
287+
**APE** — a single distributable executable across Linux, macOS, and Windows
288+
**WebAssembly** — a sandboxed/WASI deployment where a native binary is unsuitable
289+
**Both** — the familiar full C CLI behavior; factor the runtime/loader into performance expectations
290+
291+
### Use Rust Implementation For:
292+
293+
**Embedding** the byte↔glyph codec in Rust code
294+
**High-frequency transport** with reusable encode/decode buffers
295+
**Rust↔Zig boundaries** that need byte-identical raw codec output
296+
**A deliberately small raw stdin CLI**, not file/container/format workflows
297+
235298
### Use LuaJIT Implementation For:
236299

237300
**Quick scripts** and one-off operations
@@ -242,44 +305,36 @@ PRINTABLE_BINARY_MUTE_STATS – set to 1/true/yes to suppress stderr statistics
242305

243306
## Feature Comparison
244307

245-
| Feature | LuaJIT | C | Zig | Notes |
246-
|---------|--------|---|-----|-------|
247-
| **Performance** | Fast | **Faster** | **Faster** | C/Zig are 1.1-6x faster |
248-
| **Memory Usage** | Good | **Better** | **Better** | Native uses less memory |
249-
| **Memory Safety** || ⚠️ || Zig has built-in safety checks |
250-
| **Basic Encoding/Decoding** |||| Identical output |
251-
| **Passthrough Mode** |||| Same functionality |
252-
| **Formatted Output** |||| Same formatting |
253-
| **Preserve Options** |||| -s/-t/-n/-w/-p flags |
254-
| **Cross-Platform** |||| All work everywhere |
255-
| **Binary Size** | Small | **Smaller** | Small | C compiles to ~50KB |
256-
| **Startup Time** | Fast | **Faster** | **Faster** | No interpreter overhead |
257-
| **Development** | **Easier** | Harder | Medium | Lua is most flexible |
258-
| **Cross-Compilation** | N/A | Manual | **Easy** | Zig has built-in cross-compile |
308+
| Variant | Raw codec | Full CLI | Runtime / distribution focus |
309+
| --- | --- | --- | --- |
310+
| LuaJIT | Yes | Yes | Reference implementation and scripting |
311+
| Native C | Yes | Yes | Fast direct native executable |
312+
| Cosmopolitan APE | Yes | Yes | One portable executable across major desktop OSes |
313+
| Zig | Yes | Yes | Safe native CLI and broad cross-compilation |
314+
| JavaScript / Node | Yes | Yes | Browser UI and Node automation share a core |
315+
| WebAssembly | Yes | Yes | WASI sandbox via wazero; includes runtime startup cost |
316+
| Rust | Yes | No—raw stdin codec only | Embedded transport API with reusable buffers |
259317

260318
## Compatibility
261319

262320
### 100% Output Compatibility ✅
263321

264-
Both implementations produce **byte-for-byte identical** outputs:
322+
All implementations produce **byte-for-byte identical raw codec** outputs:
265323

266324
- ✅ All 256 possible byte values
267325
- ✅ Unicode and special characters
268326
- ✅ Edge cases and corner conditions
269-
- ✅ Formatted output modes
270-
- ✅ Passthrough functionality
327+
- ✅ Full CLI formatting/passthrough/container modes where that surface is implemented
271328

272329
### Tested Compatibility
273330

274331
```bash
275332
# Run comprehensive compatibility tests
276-
./test # LuaJIT test suite
277-
./test_optimized # C test suite (uses same test cases)
278-
./bm/benchmark_c_vs_lua.sh # Performance + compatibility verification
333+
./test/test # LuaJIT CLI suite
334+
nix build .#checks.x86_64-linux.test-rust # Rust + Zig differential guard (Linux)
335+
nix develop -c ./bm/benchmark-zig-opt --quick # Per-implementation round trips before timing
279336
```
280337

281-
**Test Results**: 24/24 compatibility tests passed ✅
282-
283338
## Build Requirements
284339

285340
### C Implementation
@@ -545,12 +600,11 @@ When reporting issues, please include:
545600

546601
## Summary
547602

548-
PrintableBinary offers three excellent native implementations:
549-
550-
- **C Implementation**: Maximum performance for production use
551-
- **Zig Implementation**: Memory-safe with easy cross-compilation
552-
- **LuaJIT Implementation**: Maximum flexibility for development
603+
PrintableBinary offers a byte-identical raw codec through several deliberate deployment choices:
553604

554-
All maintain perfect compatibility while offering different trade-offs. Choose based on your specific needs: performance-critical applications benefit from the C or Zig versions, while development and scripting scenarios may prefer the LuaJIT version.
605+
- **Native C / Zig / LuaJIT / Node**: full CLIs for direct use
606+
- **Cosmopolitan APE**: the full C CLI in one portable executable
607+
- **WebAssembly**: the full C CLI under a WASI runtime
608+
- **Rust**: an embedded, reusable-buffer raw codec and narrow stdin transport CLI
555609

556-
**🚀 For most users, we recommend the C or Zig implementation for superior performance and efficiency.**
610+
Choose a full CLI for files, formatting, mappings, containers, and interactive workflows. Choose Rust when the raw codec belongs inside a Rust process or transport boundary. Use `bm/benchmark-zig-opt` on the deployment hardware before making a performance claim—its adapters make Rust, WASM, and APE first-class comparison targets.

0 commit comments

Comments
 (0)