|
| 1 | +name: Bench Regression Guard |
| 2 | + |
| 3 | +# Sub-deliverable 8.3 of the benchmark/optimization milestone. |
| 4 | +# |
| 5 | +# HONEST SCOPE (read this before assuming this gates on timing): |
| 6 | +# * The `bench-compile` job is a REAL, HARD-FAILING regression gate. It runs |
| 7 | +# `cargo bench --no-default-features --no-run`, which type-checks and links |
| 8 | +# EVERY criterion bench in the v2/ workspace without running a single |
| 9 | +# measurement. Benches are not part of `cargo test`, so they silently |
| 10 | +# bit-rot when a public API they call changes — this job catches that the |
| 11 | +# moment it happens. This is the part of this workflow that can fail a PR. |
| 12 | +# |
| 13 | +# * The `bench-fast-run` job runs a small, curated subset of pure-CPU benches |
| 14 | +# in criterion "quick mode" (short warm-up / measurement / 10 samples) and |
| 15 | +# is INFORMATIONAL ONLY (`continue-on-error: true`). It does NOT gate on |
| 16 | +# timing. Wall-clock timings on shared GitHub-hosted runners vary by |
| 17 | +# 2-3x run-to-run (noisy neighbours, CPU throttling, no pinned frequency), |
| 18 | +# so a hard ">X ms" threshold here would flake constantly and teach |
| 19 | +# everyone to ignore it. We deliberately do not pretend to do timing |
| 20 | +# regression-gating we cannot deliver reliably. The numbers are surfaced in |
| 21 | +# the job log + uploaded as an artifact for humans to eyeball trends. |
| 22 | +# |
| 23 | +# WHY NO criterion --baseline COMPARE GATE: |
| 24 | +# criterion's `--save-baseline` / `--baseline` compare is the textbook |
| 25 | +# regression mechanism, but it only produces a trustworthy verdict when the |
| 26 | +# baseline and the candidate were measured on the SAME hardware under the SAME |
| 27 | +# conditions. GitHub-hosted runners give neither (the baseline commit and the |
| 28 | +# PR commit land on different physical machines). Committing a baseline JSON |
| 29 | +# measured on one runner and comparing a different runner against it would |
| 30 | +# manufacture false regressions. If/when these benches run on a dedicated, |
| 31 | +# frequency-pinned self-hosted runner, a `--baseline` compare with a generous |
| 32 | +# (>2x) noise floor becomes honest and can be added then. Until then, |
| 33 | +# compile-verify + informational-run is the honest gate. |
| 34 | + |
| 35 | +on: |
| 36 | + push: |
| 37 | + branches: [ main, develop, 'feat/*' ] |
| 38 | + paths: |
| 39 | + - 'v2/crates/**/benches/**' |
| 40 | + - 'v2/crates/**/Cargo.toml' |
| 41 | + - 'v2/crates/**/src/**' |
| 42 | + - 'v2/Cargo.toml' |
| 43 | + - 'v2/Cargo.lock' |
| 44 | + - '.github/workflows/bench-regression.yml' |
| 45 | + pull_request: |
| 46 | + paths: |
| 47 | + - 'v2/crates/**/benches/**' |
| 48 | + - 'v2/crates/**/Cargo.toml' |
| 49 | + - 'v2/crates/**/src/**' |
| 50 | + - 'v2/Cargo.toml' |
| 51 | + - 'v2/Cargo.lock' |
| 52 | + - '.github/workflows/bench-regression.yml' |
| 53 | + workflow_dispatch: |
| 54 | + |
| 55 | +permissions: |
| 56 | + contents: read |
| 57 | + |
| 58 | +env: |
| 59 | + CARGO_TERM_COLOR: always |
| 60 | + # Debuginfo is useless in CI and the 38-crate workspace target dir otherwise |
| 61 | + # exhausts the runner disk (mirrors ci.yml's rust-tests job). The bench |
| 62 | + # profile inherits release + debug = true (v2/Cargo.toml [profile.bench]); |
| 63 | + # force it off so the link step does not run out of space. |
| 64 | + CARGO_PROFILE_BENCH_DEBUG: "0" |
| 65 | + CARGO_PROFILE_RELEASE_DEBUG: "0" |
| 66 | + |
| 67 | +jobs: |
| 68 | + # ── HARD GATE: every bench must still compile + link ───────────────────── |
| 69 | + bench-compile: |
| 70 | + name: bench compile-verify (--no-run) |
| 71 | + runs-on: ubuntu-latest |
| 72 | + steps: |
| 73 | + - name: Checkout (recursive — wifi-densepose-rufield path-deps vendor/rufield) |
| 74 | + uses: actions/checkout@v4 |
| 75 | + with: |
| 76 | + # The workspace includes `wifi-densepose-rufield`, which path-deps the |
| 77 | + # `vendor/rufield` submodule crates. Without a recursive checkout the |
| 78 | + # whole workspace fails to resolve before any bench is built. |
| 79 | + submodules: recursive |
| 80 | + |
| 81 | + # The workspace pulls in `wifi-densepose-desktop` (Tauri v2) whose -sys |
| 82 | + # crates need the GTK/WebKit/serial dev libraries via pkg-config, exactly |
| 83 | + # as ci.yml's rust-tests job documents. A `--workspace` bench build links |
| 84 | + # the whole graph, so these are required here too. |
| 85 | + - name: Install Tauri / GTK / serial system dev libraries |
| 86 | + run: | |
| 87 | + sudo apt-get update |
| 88 | + sudo apt-get install -y --no-install-recommends \ |
| 89 | + libglib2.0-dev \ |
| 90 | + libgtk-3-dev \ |
| 91 | + libsoup-3.0-dev \ |
| 92 | + libjavascriptcoregtk-4.1-dev \ |
| 93 | + libwebkit2gtk-4.1-dev \ |
| 94 | + libayatana-appindicator3-dev \ |
| 95 | + librsvg2-dev \ |
| 96 | + libxdo-dev \ |
| 97 | + libudev-dev \ |
| 98 | + libdbus-1-dev \ |
| 99 | + libssl-dev \ |
| 100 | + pkg-config |
| 101 | +
|
| 102 | + - name: Install Rust toolchain |
| 103 | + uses: dtolnay/rust-toolchain@stable |
| 104 | + |
| 105 | + - name: Cache cargo (Swatinem/rust-cache) |
| 106 | + uses: Swatinem/rust-cache@v2 |
| 107 | + with: |
| 108 | + workspaces: v2 |
| 109 | + # Distinct cache scope from ci.yml's rust-tests so the bench profile |
| 110 | + # artifacts (release+opt) do not evict the test profile cache. |
| 111 | + key: bench-regression |
| 112 | + |
| 113 | + # The core regression guard. `--no-run` compiles + links every bench |
| 114 | + # target in the workspace's DEFAULT feature set but runs no measurement, |
| 115 | + # so it is deterministic and fast-ish (build only). A bench that no longer |
| 116 | + # compiles — because a type/signature it calls changed and nobody updated |
| 117 | + # the bench — fails the build here. `--no-default-features` is the |
| 118 | + # workspace's standard gate flag (openblas/tch/ort/onnx stay opt-out). |
| 119 | + - name: Compile all workspace benches (default features) |
| 120 | + working-directory: v2 |
| 121 | + run: cargo bench --workspace --no-default-features --no-run |
| 122 | + |
| 123 | + # Feature-gated benches are skipped by the default build above because |
| 124 | + # their `[[bench]]` entries carry `required-features`. Compile the ones we |
| 125 | + # can guard so they are also covered against bit-rot. |
| 126 | + # * cir → wifi-densepose-signal/benches/cir_bench.rs (ADR-134). The |
| 127 | + # `cir` feature is pure-Rust (`cir = []`), so it builds on the stock |
| 128 | + # runner and is a real, hard-failing guard like the step above. |
| 129 | + # |
| 130 | + # NOT guarded here (honest scope): |
| 131 | + # * crv → wifi-densepose-ruvector/benches/crv_bench.rs. The `crv` feature |
| 132 | + # pulls the crates.io dependency `ruvector-crv 0.1.1`, which currently |
| 133 | + # FAILS to compile on stable (E0308 type mismatch in its own |
| 134 | + # `stage_iii.rs` — an UPSTREAM bug, unrelated to bench bit-rot). |
| 135 | + # Adding a hard `--features crv` compile step would make this workflow |
| 136 | + # red for a reason this gate is not meant to police. Re-add this step |
| 137 | + # once `ruvector-crv` ships a fixed release. (mqtt/onnx benches are |
| 138 | + # likewise left to their own crate workflows.) |
| 139 | + - name: Compile feature-gated benches (cir) |
| 140 | + working-directory: v2 |
| 141 | + run: cargo bench -p wifi-densepose-signal --no-default-features --features cir --bench cir_bench --no-run |
| 142 | + |
| 143 | + # ── INFORMATIONAL: run a curated fast subset (never gates) ─────────────── |
| 144 | + bench-fast-run: |
| 145 | + name: bench fast-run (informational, non-gating) |
| 146 | + runs-on: ubuntu-latest |
| 147 | + # NEVER fail the workflow on this job — timings are noise-prone on shared |
| 148 | + # runners (see header). It exists to surface trends for humans, not to gate. |
| 149 | + continue-on-error: true |
| 150 | + needs: [bench-compile] |
| 151 | + steps: |
| 152 | + - name: Checkout (recursive) |
| 153 | + uses: actions/checkout@v4 |
| 154 | + with: |
| 155 | + submodules: recursive |
| 156 | + |
| 157 | + - name: Install Rust toolchain |
| 158 | + uses: dtolnay/rust-toolchain@stable |
| 159 | + |
| 160 | + - name: Cache cargo (Swatinem/rust-cache) |
| 161 | + uses: Swatinem/rust-cache@v2 |
| 162 | + with: |
| 163 | + workspaces: v2 |
| 164 | + key: bench-regression |
| 165 | + |
| 166 | + # Curated subset = pure-CPU, fast, dependency-light criterion benches that |
| 167 | + # finish in seconds under quick-mode flags. Each is targeted by `--bench` |
| 168 | + # (NOT a bare `cargo bench -p`) because the crates' lib targets use the |
| 169 | + # libtest harness, which rejects criterion's CLI flags (--warm-up-time |
| 170 | + # etc.) and aborts the run. Quick-mode: 1s warm-up, 2s measure, 10 samples. |
| 171 | + - name: nvsim pipeline_throughput (quick) |
| 172 | + working-directory: v2 |
| 173 | + run: | |
| 174 | + mkdir -p ../bench-out |
| 175 | + cargo bench -p nvsim --no-default-features --bench pipeline_throughput -- \ |
| 176 | + --warm-up-time 1 --measurement-time 2 --sample-size 10 \ |
| 177 | + | tee ../bench-out/nvsim_pipeline_throughput.txt |
| 178 | +
|
| 179 | + - name: ruvector sketch_bench (quick) |
| 180 | + working-directory: v2 |
| 181 | + run: | |
| 182 | + cargo bench -p wifi-densepose-ruvector --no-default-features --bench sketch_bench -- \ |
| 183 | + --warm-up-time 1 --measurement-time 2 --sample-size 10 \ |
| 184 | + | tee ../bench-out/ruvector_sketch_bench.txt |
| 185 | +
|
| 186 | + - name: ruvector fusion_bench (quick) |
| 187 | + working-directory: v2 |
| 188 | + run: | |
| 189 | + cargo bench -p wifi-densepose-ruvector --no-default-features --bench fusion_bench -- \ |
| 190 | + --warm-up-time 1 --measurement-time 2 --sample-size 10 \ |
| 191 | + | tee ../bench-out/ruvector_fusion_bench.txt |
| 192 | +
|
| 193 | + - name: Upload informational bench logs |
| 194 | + if: always() |
| 195 | + uses: actions/upload-artifact@v4 |
| 196 | + with: |
| 197 | + name: bench-fast-run-logs |
| 198 | + path: bench-out/ |
| 199 | + if-no-files-found: warn |
0 commit comments