Skip to content

Commit 92feb45

Browse files
feat(shadow-diff): shadow-diff harness proving Rust/Zig parity (#49)
* feat(shadow-diff): compare Rust library_get_all against the Zig sidecar Porting library SQL to Zig has no Zig-side ground truth, so the port is only defensible if both implementations can be run against the same input and their output compared. MT_SHADOW_DIFF makes library_get_all serve the Rust result as before while also asking the sidecar the same query and comparing the two compact JSON documents byte for byte -- the strictest comparison available, since zig-core emits keys in Track's declaration order on purpose and a byte compare also catches a value written into the wrong key slot. A divergence is logged, never thrown, with the first differing byte offset, context from both sides, and a structural summary; the Rust response stays what the frontend sees. With the flag off the command costs one env::var read -- the clone, the round-trip and the comparison are all behind it. The fixture-driven test drives the real sidecar binary over a socket across 26 query shapes (every sort column, both orders, ignore_words through the strip_sort_prefix UDF each side registers independently, search with every character the encoder has to agree about, and four pagination edges) over tests/fixtures/mt_fixture.db, built from mt_20260127.sql through the real schema path and generated on demand so CI needn't order the tasks. --sabotage on the sidecar forces one field to a constant so the harness is shown to catch a divergence, not merely to compare. It is a flag rather than an edit-and-revert so the demonstration stays repeatable; respond delegates with false, leaving the production path byte-identical. ci:shadow-diff runs the harness in a new non-blocking test.yml job with continue-on-error, off every other job's needs, so the main rust job keeps neither the Zig toolchain nor the ability to fail on Zig-side work. Verified: Rust 892 passed / 0 failed, zig 36/36, and the Playwright non-@tauri suite at 504 passed with MT_SHADOW_DIFF=1 and zero divergences (the 5 failures reproduce identically with these changes stashed). The repo-root mt.db is a pre-migration schema and cannot be an automated fixture -- the sidecar fails on it with "no such column: disc_number" -- so mt_20260127.sql through Database::new is the fixture, recorded as a divergence rather than worked around. * docs(task-355.5): record webkit re-verification and AC#3 mocked-invoke gap Re-ran the Playwright suite under the repo's actual default engine (webkit, via the official Playwright container image rather than symlinking mismatched host libjpeg/libjxl sonames) and recorded the result: same pre-existing failures as chromium, one parallel-load-flaky test cleared on isolated rerun, no new diff-attributable failures. Also documents a gap found during that follow-up: the non-@tauri Playwright suite never invokes the real Tauri/Rust command backing MT_SHADOW_DIFF (window.__TAURI__ is absent or hand-mocked in every spec), so AC#3's "zero divergences" claim is vacuously true on both engines. Left as-is per reviewer decision; flagged for a follow-on task. * chore(backlog): mark TASK-355.5 Done after human review
1 parent fdf60ab commit 92feb45

13 files changed

Lines changed: 1124 additions & 114 deletions

File tree

β€Ž.github/workflows/test.ymlβ€Ž

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,45 @@ jobs:
113113
- name: Zig build
114114
run: task zig:build
115115

116+
# Shadow-diff parity between the Rust library_get_all and its Zig port
117+
# (TASK-355.5 AC#6). Deliberately not in any other job's `needs`, and
118+
# continue-on-error, so a divergence is reported on the PR without blocking
119+
# an unrelated change: the harness compares two implementations of the same
120+
# query rather than verifying either against a spec, and it needs both the
121+
# Rust and Zig toolchains plus a staged sidecar, which the rust job
122+
# intentionally does not carry.
123+
shadow-diff:
124+
name: Rust/Zig Shadow-Diff Parity
125+
runs-on: blacksmith-4vcpu-ubuntu-2404
126+
timeout-minutes: 25
127+
128+
steps:
129+
- uses: actions/checkout@v6
130+
131+
- name: Setup Tauri build environment
132+
uses: ./.github/actions/setup-tauri-build
133+
with:
134+
mode: test
135+
136+
- name: Read pinned Zig toolchain from .tool-versions
137+
id: zig-version
138+
shell: bash
139+
run: |
140+
version=$(grep '^zig ' .tool-versions | awk '{print $2}')
141+
echo "toolchain=${version}" >> "$GITHUB_OUTPUT"
142+
143+
- name: Install Zig
144+
uses: mlugg/setup-zig@v2
145+
with:
146+
version: ${{ steps.zig-version.outputs.toolchain }}
147+
148+
- name: Install Task
149+
uses: go-task/setup-task@v1
150+
151+
- name: Run shadow-diff parity harness
152+
continue-on-error: true
153+
run: task ci:shadow-diff TARGET=x86_64-unknown-linux-gnu
154+
116155
build:
117156
name: Build (${{ matrix.platform }})
118157
# Gate on lint/format/test jobs so builds don't burn CI minutes

β€Žbacklog/tasks/task-355.5 - Shadow-diff-harness-proving-Rust-Zig-parity.mdβ€Ž

Lines changed: 60 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
id: TASK-355.5
33
title: Shadow-diff harness proving Rust/Zig parity
4-
status: To Do
4+
status: Done
55
assignee: []
66
created_date: '2026-09-11 00:39'
77
labels: []
@@ -20,10 +20,63 @@ Porting roughly 7,500 lines of hand-written SQL from Rust to Zig by hand carries
2020

2121
## Acceptance Criteria
2222
<!-- AC:BEGIN -->
23-
- [ ] #1 A runtime flag causes the relevant Rust command to execute both the existing Rust implementation and a call to the new Zig sidecar endpoint, and compare their canonical JSON output
24-
- [ ] #2 Any divergence between the two is logged with enough detail to diagnose it
25-
- [ ] #3 The full existing Playwright suite passes with the shadow-diff flag enabled, with zero divergences logged
26-
- [ ] #4 The repo-root mt.db and mt_20260127.sql are used as realistic test fixtures
27-
- [ ] #5 A deliberately introduced divergence between the two implementations is demonstrated to be caught by the harness
28-
- [ ] #6 The shadow-diff harness is excluded from the main rust CI job's critical path (e.g. it runs in a separate, non-blocking job or is opt-in)
23+
- [x] #1 A runtime flag causes the relevant Rust command to execute both the existing Rust implementation and a call to the new Zig sidecar endpoint, and compare their canonical JSON output
24+
- [x] #2 Any divergence between the two is logged with enough detail to diagnose it
25+
- [x] #3 The full existing Playwright suite passes with the shadow-diff flag enabled, with zero divergences logged
26+
- [x] #4 The repo-root mt.db and mt_20260127.sql are used as realistic test fixtures
27+
- [x] #5 A deliberately introduced divergence between the two implementations is demonstrated to be caught by the harness
28+
- [x] #6 The shadow-diff harness is excluded from the main rust CI job's critical path (e.g. it runs in a separate, non-blocking job or is opt-in)
2929
<!-- AC:END -->
30+
31+
## Implementation Notes
32+
33+
<!-- SECTION:NOTES:BEGIN -->
34+
### What was built
35+
36+
**`crates/mt-tauri/src/shadow_diff.rs`** (new) β€” flag, Rust-query β†’ Zig-query-string translation, the comparison, and the divergence log.
37+
38+
- **Flag (AC#1): `MT_SHADOW_DIFF`** (`1`/`true`/`TRUE`/`yes`), following `MT_LOG`'s env-var precedent. Read per call rather than cached, matching `db::indexed_prefix_lookup_enabled`'s env branch β€” caching is what made that flag's settings-driven half go stale β€” and this is one cheap `env::var` per library request, not a hot loop. **With the flag off the command costs one `env::var` read and nothing else**: the query clone, the response clone and the comparison all live inside `if enabled()`, so there is no Zig round-trip and no comparison overhead.
39+
- **Command wiring.** `library_get_all` takes `State<'_, SidecarState>` and, when enabled, spawns a detached `tauri::async_runtime::spawn` task β€” not `block_on`, because `library_get_all` is a *sync* command on Tauri's command thread pool and `block_on` inside a tokio worker panics. Detaching also keeps the sidecar's single-threaded accept loop off the frontend's request latency. The Rust response is returned unchanged whatever the comparison finds: a divergence is logged, never thrown.
40+
- **`SidecarState`** gains `Arc` + `Clone` (so a command can hand the same state to a detached task without borrowing `State<'_, _>` past its own invocation), a `pub(crate) endpoint()` accessor that reuses the endpoint the startup health probe already resolved (no second parse of `sidecar.json`), and a `#[cfg(test)] for_test` constructor.
41+
- **Comparison = byte compare of two compact JSON documents** (AC#1/#2). serde_json and `std.json.Stringify` both emit compact JSON, and `zig-core`'s `writeTrack` emits keys in `Track`'s declaration order deliberately, so byte equality is the *strictest* available check β€” it also catches a value written into a different key slot, which a field-wise deep compare would wave through. Confirmed empirically before writing the harness: on the 301-row fixture the two sides agree byte for byte across a 235,990-byte response (`EQUAL true`), covering `total`/`limit`/`offset` and every `Track` field including float rendering, `i64` nanosecond timestamps and NULL handling.
42+
- **Query translation.** `zig_query_string` maps the Rust `LibraryQuery` onto the sidecar's contract with hand-rolled `application/x-www-form-urlencoded` encoding β€” exactly the character set `parseQuery`'s decoder has to agree about (`+`, `&`, `%`, `'`, multi-byte UTF-8). `sort_by` needs the pre-parse wire name (`LibrarySortColumn` renders as the SQL expression, and `Year` is ambiguous once parsed β€” `date` and `year` both collapse into it), so `sort_column_name` supplies it.
43+
44+
**Divergence log (AC#2)** β€” diagnosable rather than "mismatch": `first_diff_offset` plus ~120 chars of context from *both* sides at that offset, `rust_len`/`zig_len`, and a structural summary (`keys=[...] tracks=N`, or `unparseable: <err>`) so a shape divergence is distinguishable from a value divergence. `event = "shadow_diff divergence"` is the CI grep target. Capped at `MAX_DIVERGENCE_LOGS = 20` so one systematically broken port cannot bury the rest of the log under thousands of identical diffs.
45+
46+
**Zig side.** A `--sabotage` flag (`main.zig` β†’ `server.Options.sabotage` β†’ `library.respondWithSabotage`) forces the one field named by `library.sabotage_field` (`genre`) to a constant. `respond` still delegates with `false`, so the production endpoint path is byte-for-byte unchanged. No CI job passes the flag.
47+
48+
**`crates/mt-tauri/src/shadow_diff_parity_test.rs`** (new) β€” the fixture-driven check that drives the *real sidecar binary over a socket*, not two Rust functions in memory. A 26-case query matrix (every sort column in both orders, `ignore_words` β†’ the `strip_sort_prefix` UDF each side registers independently, search including every special character, artist/album filters, and mid-library / past-the-end / wider-than-library / zero pagination) asserting zero divergences, plus the sabotage test asserting exactly one. The sidecar path resolves through Tauri's own `externalBin` staging convention, newest-binary-wins, with a capability probe for `--sabotage`.
49+
50+
### Verification
51+
52+
- **AC#1/#4 β€” `fixture_library_get_all_matches_between_rust_and_zig`: PASS.** 26 query shapes against the real fixture, >1,000 rows compared, zero divergences asserted per shape. The row count is itself asserted so the matrix cannot silently degrade into a trivial comparison.
53+
- **AC#5 β€” `sabotaged_sidecar_divergence_is_caught`: PASS**, with the log captured via `--nocapture` as real evidence:
54+
55+
```text
56+
ERROR mt_lib::shadow_diff: shadow-diff: Rust and Zig returned different JSON
57+
event="shadow_diff divergence" target_impl="zig" what="response_body"
58+
rust_len=81872 zig_len=82572 first_diff_offset=308
59+
rust_context=…"date":"2025","genre":null,"duration":327.541,…
60+
zig_context =…"date":"2025","genre":"SABOTAGED","duration":327.541,…
61+
rust_structure=keys=[limit,offset,total,tracks] tracks=100
62+
zig_structure=keys=[limit,offset,total,tracks] tracks=100 divergence_count=1
63+
```
64+
65+
Field, both values and byte offset β€” a reviewer can localize the bug from the log alone. Verified independently of the test: the sabotaged sidecar returns `genre: "SABOTAGED"`, the clean sidecar returns `genre: null`, Rust returns `null`. Unlike "temporarily edit code, observe, revert", the sabotage is a *flag*, so this demonstration stays repeatable in CI instead of being a one-time manual act whose evidence evaporates on revert.
66+
- **AC#3 β€” Playwright with `MT_SHADOW_DIFF=1`.** Non-`@tauri` suite (per AGENTS.md, `@tauri` tests need a real Tauri runtime and hardware audio and are excluded from default CI): **504 passed, 5 failed, 2 skipped** on chromium (see below for webkit). Those 5 (lastfm auth error handling, type-to-jump debounce, 3Γ— Plex cloud badge) reproduce **identically on the clean baseline with my changes stashed** β€” pre-existing, and my diff touches zero frontend files (`git status app/frontend/` is empty). Two caveats originally stated: this first ran on **chromium, not webkit** (default `fast` mode is webkit-only), because AlmaLinux 10 ships `libjpeg.so.62`/`libjxl.so.0.10` where Playwright's webkit build wants `libjpeg.so.8`/`libjxl.so.0.8` β€” no compatible package exists in the base OS or EPEL repos (confirmed via `dnf repoquery`, both differ by ABI-generation, not just naming); the 8 `visual-regression` snapshot tests failed on the first run only because `*-snapshots/` is gitignored and no baselines existed.
67+
- **Reviewer follow-up (webkit re-verification, post-review).** Rather than symlinking mismatched sonames, ran the actual `webkit` project (this repo's real default engine) inside the official `mcr.microsoft.com/playwright:v1.58.0-noble` container (matches the pinned `@playwright/test` version exactly), bind-mounting this worktree so the project's own pinned Playwright/browser build ran unmodified β€” genuinely compatible libs via the browser vendor's own supported base image, not a manufactured fix. Result: **495 passed, 14 failed, 2 skipped**. Of the 14: the same 5 baseline failures reproduce; 8 are `visual-regression` snapshot tests with no `webkit-linux` baseline committed (same gitignored-snapshot cause, engine-specific baseline images); the 14th, `library-column-resize.spec.js:219` ("no horizontal scroll after window resize"), reproduced only under full-suite 12-worker parallel load and passed cleanly both in isolation and when the whole `library-column-resize.spec.js` file was re-run alone (22/22 passed) β€” parallel-load flakiness, not a regression (this diff touches zero frontend files, confirmed via `git status --short` on the exact reviewed commit). No new, diff-attributable webkit failures.
68+
- **Deeper finding surfaced during this follow-up, not caught in the original run: AC#3's "zero divergences logged" claim is vacuously true on both engines.** The non-`@tauri` Playwright suite runs against `npm run build && npm run preview` β€” a static Vite server, no Tauri runtime, no Rust process, no Zig sidecar. `app/frontend/js/stores/player.js:6-8` falls back `invoke` to a no-op stub (`Promise.resolve(console.warn('Tauri not available'))`) whenever `window.__TAURI__` is absent, which it is in every plain-browser test; the 7 spec files that do set `window.__TAURI__` (`plex`, `settings-audio`, `library-settings`, `settings`, `statistics`, `network-cache-settings`, `watched-folders`) all hand-roll their own mock `invoke`, never a real backend call. So `library_get_all` β€” the one command `MT_SHADOW_DIFF` gates β€” never executes during this suite, on any browser or engine; the env var has no effect here. AC#1/#2/#5's Rust-level parity tests (`shadow_diff_parity_test.rs`) remain the actual evidence the harness works; AC#3 as currently scoped only proves the flag doesn't break the frontend build/UI, not that the harness runs clean under real usage. Left as-is per explicit reviewer instruction (accepted, not re-scoped in this task) β€” flagged here for whoever picks up a follow-on task that wants AC#3 to mean what it currently implies.
69+
- **Rust 892 passed / 0 failed**; `zig build test` 36/36; `deno fmt --check`, `deno lint`, `cargo fmt --all -- --check`, `zig fmt --check` and `actionlint` all clean. `cargo clippy -D warnings` reports 7 findings β€” verified by stash-and-compare to be the **identical 7 on baseline**, all in `plex.rs`/`removed.rs`/`downloader.rs`/`lib.rs:689`, none of which this task touches. My new code contributes zero clippy findings. Vitest shows 17 failures in 4 files (`isRemote is not a function`), matching the pre-existing set already recorded on TASK-355.4, with zero frontend files in my diff.
70+
- **AC#4 β€” fixtures.** `mt_20260127.sql` is the authoritative fixture: `tests/fixtures/mt_fixture.db` is built from it through the real `Database::new`/migrations path (301 rows, current schema) and generated on demand by the test when absent, so the CI job needn't know about `task zig:fixture`. The repo-root `mt.db` was tried and **cannot** be an automated fixture: it is a pre-migration schema (no `disc_number`/`disc_total`/`genre`/`source`/`remote_id`, still carrying the retired `lastfm_loved`), and the sidecar fails on it with `no such column: disc_number`. That is the same reason `fixture_gen.rs` takes its schema from `Database::new` rather than from the dump β€” recorded as a divergence, not "fixed". `mt.db` stays `.gitignore`d and no automated job depends on it.
71+
- **AC#6 β€” CI.** `taskfiles/ci.yml` gains `ci:shadow-diff` (`requires: TARGET`; stages the sidecar via `:zig:stage`, generates the fixture, runs the harness; nextest where installed, plain `cargo test` otherwise, the same fallback `task test` relies on). `test.yml` gains a `shadow-diff` job on the Linux runner with `continue-on-error: true`, deliberately absent from every other job's `needs` β€” mirroring how the existing `zig` job stays off the critical path, and reusing the `continue-on-error` precedent already used three times in that file. The main `rust` job is untouched: no new toolchain, no new dependency, and it still cannot fail on Zig-side work.
72+
73+
### Bugs found while building this
74+
75+
- **The `--sabotage` flag was inert.** The first implementation parsed the flag and logged about it in `main.zig` but never threaded it into `server.Options`, so `library.respond` sabotaged nothing and the AC#5 test failed with 0 divergences. Caught only because the test asserts `== 1` β€” a sabotage switch nobody checks is worse than none, since it invites a false "the harness works" claim.
76+
- **Stale staged binary.** The test first resolved `crates/mt-tauri/binaries/mt-zig-core-<triple>`, built before the flag existed, which rejects `--sabotage` with `unrecognized argument`. Newest-wins plus a capability probe now reports that as *"predates the --sabotage flag; rebuild"* instead of a confusing spawn failure.
77+
- **Fixture generation race.** Both parity tests generate `mt_fixture.db` when absent; concurrently they delete-then-insert into the same file, failing with `UNIQUE constraint failed: library.id` on a half-populated table. Masked locally by a pre-existing fixture and visible only on a clean run. Fixed with a generation lock and a double-check.
78+
79+
### Note for the reviewer
80+
81+
`lib.rs:805` (`tauri::generate_context!`) was reported during this run as an `OUT_DIR` blocker. It is not one: `crates/mt-tauri/build.rs:11` runs `tauri_build::build()`, which is what supplies `OUT_DIR`; `cargo check` and `cargo test --lib --no-run` both exit 0; and `git blame` puts that line in `af04d617` (Jan 2026), untouched here β€” this task's entire `lib.rs` diff is two `mod` declarations. rust-analyzer expands the macro without cargo's env. Deleting the macro or adding a stub build script to satisfy that heuristic would break the real build, so it was left alone.
82+
<!-- SECTION:NOTES:END -->

0 commit comments

Comments
Β (0)