Merge remote-tracking branch 'origin/yolo' into yolo

# Conflicts:
#	PLAN.md

Author: pmarreck

PLAN.md

@@ -19,6 +19,51 @@ maintained_by: agent
       a webhook HTTP response proves delivery, not queue admission or a passing
       build.
 
+## Container honors `--spaces` (legible-markdown containers) — DONE (2026-07-21 EDT)
+
+Goal (Peter): put a markdown file into a `.pbf.json` container but keep it legible;
+`--spaces` was silently ignored in container mode. Fixed across all 5 container
+surfaces (Lua, Node, Zig, standalone-C, FFI-C) + shared `test/test_container`.
+
+- [x] `--spaces` honored in container encode: literal spaces in the JSON `data`
+      value (letters/digits/`. @ ^ _` already pass through, so markdown reads
+      naturally; newlines stay `¶` glyphs → still valid single-line JSON). (2026-07-21 03:00 PM EDT)
+- [x] **Flagless** design (Peter's call): NO `spaces` schema field, `version` stays 1.
+      Decode uses a **crc-probe** — try keeping literal spaces; if `crc32_encoded`
+      mismatches, strip them as transport noise. The crc is the disambiguation oracle. (2026-07-21 03:00 PM EDT)
+- [x] Ambiguity warning: a non-spaces container (space-glyph present) with a
+      transport-injected literal space → recover by stripping + WARN that literal
+      spaces were assumed formatting because the space glyph was also present.
+      Space glyph read from the map (DRY), never hardcoded. (2026-07-21 03:00 PM EDT)
+- [x] `--tabs`/`--crlf`/`-w`/`--preserve` + `--container` = HARD ERROR (would break
+      JSON-validity + transport-resistance). (2026-07-21 03:00 PM EDT)
+- [x] Corrected `character_map.txt` header comment: documents the 66/94 printable-ASCII
+      passthrough (the legibility property), verified behavior-neutral. (2026-07-21 03:00 PM EDT)
+- [x] Guard #4 (map redefines space→space) proven UNNECESSARY: the "first
+      whitespace-delimited token" parser + 256-glyph-count check make a literal-space
+      glyph structurally unloadable (physics over policy). (2026-07-21 03:00 PM EDT)
+- [x] Fixed FFI NULL-deref: `preserve_chars` is `char *` (NULL) in FFI opts, not an
+      array — guard now NULL-checks. Caught by hermetic nix check, not exit code. (2026-07-21 03:00 PM EDT)
+
+## Integrated compression (z7z/LZMA in `-C`) — DECLINED (2026-07-21 EDT)
+
+Considered then dropped (Peter): a user can compress in a pipe chain *before*
+printable-binary and stay truer to the Unix philosophy — no need to bake it in.
+Also, z7z is an archive tool (no raw stdin→stdout stream; `.7z` header +
+nondeterministic mtime would sink a stable "compressed hash"; max level 9), so
+integration would be awkward anyway. If ever revived, the open question is
+LZMA2-library-vs-CLI cross-impl parity (Zig/C could link z7z's core in-process;
+Lua/Node/Rust/Elixir could not without shelling out or their own LZMA).
+
+## Container metadata injection (`-k`/`-v`/`--kv`/`--key-namespace`) — SPEC'd, FUTURE (2026-07-21 EDT)
+
+Design approved by Peter (all forks ruled), NOT yet implemented — a future
+nice-to-have. Attach arbitrary key/value metadata to a container for other
+tooling to read first (xattr / HTTP-`X-`-header analogue). Full spec + TDD plan:
+`docs/plans/2026-07-21-container-metadata-injection-design.md`.
+- [ ] Implement TDD-first (Lua reference → Node → Zig → standalone-C → FFI-C) +
+      extend the shared `test/test_container` guard. Reserved-key rejection is an
+      MFIC cross-impl contract.
 ## Docs and benchmark parity — active (2026-07-17 EDT)
 
 - [x] Document every supported implementation—especially Rust, WebAssembly, and

README.md

@@ -214,6 +214,25 @@ paste) and restores the file under its original name. Optional metadata
 (timestamps, POSIX permissions/owner) is included when the tool can read it and
 omitted otherwise; decode never fails on a missing optional field.
 
+**Legible-text containers with `-s`/`--spaces`.** Add `--spaces` to keep literal
+spaces in the `data` value instead of encoding them to the space glyph. Since
+letters, digits and `. @ ^ _` already pass through as themselves, a text file —
+e.g. Markdown — reads naturally inside the container while staying valid,
+single-line JSON (newlines remain `¶` glyphs):
+
+```bash
+./bin/printable-binary -C --spaces notes.md > notes.md.pbf.json
+# data value reads like:  "♯ Title¶¶Some ⁎⁎bold⁎⁎ text.¶˗ a list item¶"
+```
+
+No schema flag records this — the format stays `version: 1`. Decode disambiguates
+with the `crc32_encoded` oracle: it keeps literal spaces if they check out as
+data, otherwise strips them as transport formatting (and warns if the space glyph
+was also present, i.e. the spaces were injected into a non-`--spaces` container).
+`--spaces` is the only preserve flag allowed with `-C`; `--tabs`/`--crlf`/`-w`/
+`--preserve` are rejected, since raw tab/CR/LF would break both JSON validity and
+the whitespace-stripping transport-resistance.
+
 ### Hexlike Mode (`-X`)
 
 `-X`/`--hexlike` is a hybrid view: printable ASCII passes through untouched while

bin/printable-binary

@@ -21,8 +21,13 @@ local function crc32_hex(data)
   return bit.tohex(bit.bxor(crc, 0xFFFFFFFF))
 end
 
--- Strip transport whitespace (space/tab/CR/LF) from an encoded payload.
-local function canonical_payload(s)
+-- Strip transport whitespace from an encoded payload. Normally all of space/tab/
+-- CR/LF are transport noise. When keep_spaces is set (container --spaces), literal
+-- spaces are DATA, so only tab/CR/LF are stripped as noise.
+local function canonical_payload(s, keep_spaces)
+  if keep_spaces then
+    return (s:gsub("[\t\r\n]", ""))
+  end
   return (s:gsub("[ \t\r\n]", ""))
 end
 
@@ -32,8 +37,6 @@ end
 local function json_get_string(json, key)
   return json:match('"' .. key .. '"%s*:%s*"(.-)"')
 end
-
-
 -- PrintableBinary: A utility to encode/decode binary data into human-readable UTF-8
 -- Module definition
 local PrintableBinary = {}
@@ -1028,19 +1031,45 @@ if arg then
   -- Container (.pbf.json) mode (issue #1): assemble/parse the flat-JSON envelope
   -- over the shared encode/decode + crc32 (A2). Transport-resistant.
   if container_mode then
+    -- Only --spaces is honored in container mode. --tabs/--crlf/-w/--preserve would
+    -- put raw tab/CR/LF (or arbitrary chars) into the JSON value, breaking both
+    -- single-line-JSON validity and the tab/CR/LF-stripping transport-resistance.
+    if tabs_mode or crlf_mode or preserve_chars ~= "" then
+      io.stderr:write("Error: --tabs/--crlf/-w/--preserve are not supported with "
+        .. "--container (only --spaces is honored; other whitespace stays encoded)\n")
+      os.exit(1)
+    end
     if decode_mode then
       local data_raw = json_get_string(input_data, "data")
       if not data_raw then
         io.stderr:write("Error: not a printable-binary-file container (missing 'data')\n")
         os.exit(1)
       end
-      local clean = canonical_payload(data_raw)
+      -- Flagless crc-probe: no schema flag records whether --spaces was used; the
+      -- crc32_encoded oracle disambiguates. Try keeping literal spaces (they're DATA
+      -- in a --spaces container); if the crc mismatches, strip them as transport noise.
+      local clean = canonical_payload(data_raw, true)
+      local spaces = true
       local ce = json_get_string(input_data, "crc32_encoded")
       if ce and crc32_hex(clean) ~= ce then
-        io.stderr:write("Error: container crc32_encoded mismatch (data corrupted)\n")
-        os.exit(1)
+        local stripped = canonical_payload(data_raw, false)
+        if crc32_hex(stripped) == ce then
+          -- Literal spaces were noise. If the space glyph is ALSO present, the payload
+          -- mixed real (glyph) spaces with formatting spaces -> warn we dropped them.
+          local space_glyph = encode_map[32]
+          if space_glyph and clean:find(space_glyph, 1, true) then
+            io.stderr:write("Warning: literal spaces in container data were assumed to be "
+              .. "ignorable formatting because the space glyph " .. space_glyph
+              .. " was also present; stripping them\n")
+          end
+          clean = stripped
+          spaces = false
+        else
+          io.stderr:write("Error: container crc32_encoded mismatch (data corrupted)\n")
+          os.exit(1)
+        end
       end
-      local decoded = PrintableBinary.decode(clean, {})
+      local decoded = PrintableBinary.decode(clean, { spaces = spaces })
       local co = json_get_string(input_data, "crc32")
       if co and crc32_hex(decoded) ~= co then
         io.stderr:write("Error: container crc32 mismatch (decoded data corrupted)\n")
@@ -1049,8 +1078,8 @@ if arg then
       io.write(decoded)
       os.exit(0)
     else
-      local data = PrintableBinary.encode(input_data, {})
-      local clean = canonical_payload(data)
+      local data = PrintableBinary.encode(input_data, { spaces = spaces_mode })
+      local clean = canonical_payload(data, spaces_mode)
       local crc_orig = crc32_hex(input_data)
       local crc_enc = crc32_hex(clean)
       local fname = ""

bin/printable-binary-node.js

@@ -361,6 +361,13 @@ async function main() {
     }
 
     if (containerMode) {
+      // Only --spaces is honored in container mode; --tabs/--crlf/-w/--preserve
+      // would put raw tab/CR/LF (or arbitrary chars) into the JSON value, breaking
+      // single-line-JSON validity and the tab/CR/LF-stripping transport-resistance.
+      if (tabsMode || crlfMode || preserveChars) {
+        process.stderr.write('Error: --tabs/--crlf/-w/--preserve are not supported with --container (only --spaces is honored; other whitespace stays encoded)\n');
+        process.exit(1);
+      }
       if (decodeMode) {
         const res = pb.decodeText(input.toString('utf8'));
         stats(`Decoded ${res.kind} container${res.filename && res.filename !== 'decoded.bin' ? ' (' + res.filename + ')' : ''}: ${res.bytes.length} bytes`);
@@ -376,7 +383,7 @@ async function main() {
             meta.mode = '0' + (st.mode & 0o777).toString(8);
           } catch (_e) { /* metadata is best-effort */ }
         }
-        const container = pb.encodeToContainer(new Uint8Array(input), meta);
+        const container = pb.encodeToContainer(new Uint8Array(input), meta, { spaces: spacesMode });
         stats(`Encoded ${input.length} bytes -> .pbf.json container (crc32 ${container.crc32})`);
         process.stdout.write(JSON.stringify(container, null, 2) + '\n');
       }

character_map.txt

@@ -5,6 +5,13 @@
 ## is the glyph for byte N. Exactly 256 glyph lines. The glyph is the FIRST
 ## whitespace-delimited token on the line.
 ##
+## ASCII passthrough (the core point): letters (A–Z, a–z), digits (0–9), and
+## . @ ^ _ map to THEMSELVES — 66 of the 94 printable non-space bytes — so ASCII
+## text stays legible in the encoded form even when mixed with control/high bytes.
+## The other printable bytes take look-alike glyphs to stay unambiguous: 0x20
+## space -> ␣, plus the 28 punctuation ! " # $ % & ' ( ) * + , - / : ; < = > ? [ \ ] ` { | } ~
+## (see per-line trailing notes). Byte 0x23 '#' -> ♯ is discussed below.
+##
 ## Comments:
 ##   - A line beginning with `##` (two or more '#') is a full-line comment, skipped.
 ##   - A trailing `<glyph> ## comment` is ignored (only the first token is taken).