Clean up and add tests to perf-smoke POC

Finish env-fingerprint bucketing and per-sample provenance for the
rolling-window history, add unit tests for the orchestrator, rebaseline
tool, and fingerprint/provenance helpers, and document the architecture
in DESIGN.md.

Author: Neil4561

.github/workflows/perf-gate.yml

@@ -76,18 +76,26 @@ jobs:
           echo "Matrix tasks: $tasks_json"
 
   # ---------------------------------------------------------------------------
-  # Comparator logic tests. Pure stdlib unittest -- no GPU, no Isaac Sim, and
-  # arch-independent -- so this validates the PASS/WARN/BLOCK verdict logic on
-  # every trigger and gives fast signal before the GPU job is scheduled.
+  # Pure-logic tests for the comparator, orchestrator, rebaseline tool, and the
+  # history-bucketing helpers. Pure stdlib unittest -- no GPU, no Isaac Sim, and
+  # arch-independent -- so this validates the PASS/WARN/BLOCK verdict logic, the
+  # launch-config plumbing, and the rolling-window writer on every trigger and
+  # gives fast signal before the GPU job is scheduled.
   # ---------------------------------------------------------------------------
   comparator-unit-tests:
     name: Comparator Unit Tests
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
-      - name: Run comparator unittest suite
-        run: python3 tools/perf_smoke/test_check_perf_regression.py
+      - name: Run unittest suite
+        run: |
+          set -e
+          for t in test_check_perf_regression test_history_fingerprint test_run_perf_gate test_rebaseline; do
+            echo "::group::$t"
+            python3 "tools/perf_smoke/$t.py"
+            echo "::endgroup::"
+          done
 
   # ---------------------------------------------------------------------------
   # The GPU gate. One job PER TASK (matrix from baseline.json): each task is its

tools/perf_smoke/DESIGN.md

@@ -0,0 +1,143 @@
+# Perf Smoke Gate — Architecture
+
+This document explains *how* the gate is put together: the data flow, the
+baseline/history model, and the verdict logic. For a plain-English file-by-file
+tour, see [`README.md`](README.md).
+
+## Design goals
+
+1. **Cheap and per-PR.** A handful of short, stable tasks on a fixed GPU — fast
+   enough to run on every pull request.
+2. **Robust to noise.** Small run-to-run wobble must not flake the gate; only a
+   real, sustained drop blocks.
+3. **In-tree and reviewed.** Baselines and history live in the repo and only
+   change through a normal, reviewed PR — never a silent side-write from CI. This
+   keeps the gate auditable and lets `git blame` answer "why did the bar move?".
+4. **No new dependencies.** Everything is standard-library Python so the
+   comparator and its tests run on any runner without Isaac Sim or a GPU.
+
+## Components
+
+| Layer | Module | Responsibility |
+|---|---|---|
+| Orchestration | `run_perf_gate.py` | Per task: resolve launch config from `baseline.json`, launch the benchmark as its own Isaac Sim subprocess (retry once), hand the result to the comparator, aggregate verdicts (worst wins). |
+| Decision (pure logic) | `check_perf_regression.py` | Read one benchmark result + baseline + rolling window, compute the KPI, and return PASS / WARN / BLOCK. No GPU, no Isaac Sim. |
+| Stored state | `baseline.json`, `perf_history/`, `baseline_overrides.json` | The launch config + static fallback, the rolling window of recent samples, and manual threshold overrides. |
+| Maintenance | `rebaseline.py`, `seed_history.py` | Produce/refresh the stored state from fresh or existing runs (always via a reviewed PR). |
+
+## Data flow (one PR)
+
+```
+baseline.json ─┐
+               ├─► run_perf_gate ─► benchmark_non_rl.py (subprocess) ─► result.json
+perf_history/ ─┤                                                            │
+overrides ─────┘                                                            ▼
+                                   check_perf_regression  ◄─────────────────┘
+                                            │
+                                   RESULT=PASS|WARN|BLOCK  +  $GITHUB_STEP_SUMMARY table
+```
+
+The orchestrator only builds commands and aggregates; **all** of the regression
+judgement lives in the comparator, which is why the comparator is independently
+unit-testable without hardware.
+
+## The KPI
+
+The gating metric is the **post-warm-up steady FPS** (`steady_fps`): the
+benchmark's per-frame effective-FPS array with the first `warmup_frames` dropped.
+Using the same statistic the backend already reports — just windowed — keeps the
+measured value directly comparable to the stored history. Wall-clock seconds are
+carried as a secondary, advisory signal only.
+
+## Baseline & history model
+
+There are two stores, deliberately layered:
+
+- **Rolling window (`perf_history/`, primary).** Per `(task, GPU)`, the last
+  *N* known-good samples. The comparator computes its threshold *at test time*
+  from this window with a robust **median + MAD** estimator:
+
+  ```
+  center = median(window)
+  spread = max(1.4826 * MAD(window), min_spread_pct/100 * center)
+  WARN   when measured < center - k_warn  * spread
+  BLOCK  when measured < center - k_block * spread
+  ```
+
+  A `min_spread_pct` floor stops a very low-variance task from blocking on
+  trivial dips.
+
+- **Static fallback (`baseline.json`, secondary).** When the window is too small
+  to trust (`< MIN_WINDOW` samples), the comparator falls back to a static
+  `baseline_fps` + percentage bands calibrated for that task/GPU. This keeps a
+  fresh store from silently passing everything before it has accumulated history.
+
+**Overrides** (`baseline_overrides.json`) are a manual escape hatch keyed by
+*stable* test identity (`task` + GPU), applied on top of either source — used for
+one-off threshold relaxations or `skip` that ride along in the PR.
+
+### Environment fingerprint buckets
+
+Performance is only comparable within the same software stack: a Warp bump or an
+Isaac Sim upgrade can legitimately shift FPS, and mixing those samples into one
+window would corrupt the baseline. So history is **bucketed by an environment
+fingerprint**:
+
+```
+perf_history/
+  <task>__<gpu>.json                     # flat "default" bucket (legacy / no provenance)
+  env-<hash>/<task>__<gpu>.json          # one bucket per (warp, isaaclab, cuda) stack
+```
+
+- `env_fingerprint(result)` hashes the environment-defining provenance
+  (`warp`, `isaaclab`, `cuda`) into a short, stable `env-<hash>` key. GPU is
+  *not* in the hash because it is already in the file name.
+- The comparator derives the fingerprint from the run under test and reads the
+  matching bucket, **falling back to the flat file** when no bucket exists yet —
+  so the change is backward-compatible with already-seeded flat history.
+- A consistent filename (`history_basename`) is shared by the reader and every
+  writer so a written bucket is always found again.
+
+### Per-sample provenance
+
+Every stored sample carries the context needed to audit or re-bucket it without
+re-running the benchmark: `commit`, `warp`, `isaaclab`, `cuda`, plus the
+`fingerprint` recorded at the window level. This makes the in-tree history
+self-describing — a reviewer reading a `perf_history/` diff can see exactly which
+commit and stack produced each number.
+
+## Re-baselining lifecycle
+
+`rebaseline.py` is the only writer of the stored state and serves two jobs from
+one measurement path:
+
+1. **Variance study (default).** Run each task `--repeat` times and report robust
+   stats (median / CV / MAD / min / max) so thresholds can be justified to
+   reviewers.
+2. **Rolling re-baseline (`--apply`).** Append the new samples to the window
+   (pruned to a cap, stamped with provenance, written into the env bucket) and
+   refresh the static fallback in `baseline.json`.
+
+A **boiling-frog guard** keeps a rolling baseline from quietly absorbing a real
+regression: a task whose new median drops the baseline by more than
+`--soft-drop-pct` is *flagged for review*; a drop beyond `--hard-drop-pct` is
+*refused* (old value kept) unless `--force`. Both stores then change only through
+the PR that `perf-rebaseline.yml` opens.
+
+## CI wiring
+
+- `perf-gate.yml` — on a PR it runs a fast, GPU-free **unit-test job** (comparator,
+  orchestrator, rebaseline, fingerprint helpers) for early signal, then a
+  per-task GPU matrix on the L40S fleet that posts the verdict back. Advisory
+  (`continue-on-error`) until cross-runner variance is confirmed.
+- `perf-rebaseline.yml` — manual workflow that runs `rebaseline.py --apply` on the
+  fleet and opens a PR with the `baseline.json` + `perf_history/` diff.
+
+## Why these boundaries
+
+- **Pure-logic comparator** ⇒ the regression rules are fully testable on any
+  runner; the GPU job only *produces* numbers, it never *decides*.
+- **In-tree, reviewed state** ⇒ no opaque external baseline service; every change
+  to the bar is a diff someone approved.
+- **Layered window → static → override** ⇒ robust thresholds when history exists,
+  a safe floor when it doesn't, and a human escape hatch when neither fits.

tools/perf_smoke/README.md

@@ -42,9 +42,15 @@ mild dip is an advisory **WARN** and only a real, sustained drop is a **BLOCK**.
 |---|---|
 | `test_check_perf_regression.py` | Unit tests for the comparator logic (no GPU). |
 | `test_stress_check_perf_regression.py` | Heavier stress/edge-case tests for the comparator. |
+| `test_history_fingerprint.py` | Unit tests for env-fingerprint bucketing + per-sample provenance. |
+| `test_run_perf_gate.py` | Unit tests for the orchestrator (config, command building, aggregation). |
+| `test_rebaseline.py` | Unit tests for the rebaseline tool (window stats, store writer, boiling-frog guard). |
 | `test_perf_gate.py` | The pytest entry point CI uses to drive a single task end-to-end. |
 | `pytest.ini` | Local pytest config for this directory. |
 
+For the architecture (data flow, the baseline/history model, and the verdict
+logic), see [`DESIGN.md`](DESIGN.md).
+
 ### CI wiring (in `.github/`)
 
 | File | Plain-English purpose |

tools/perf_smoke/check_perf_regression.py

@@ -55,6 +55,7 @@
 
 import argparse
 import glob
+import hashlib
 import json
 import os
 import sys
@@ -366,6 +367,66 @@ def _extract_provenance(result: dict) -> dict[str, object]:
     return out
 
 
+def _extract_commit(result: dict) -> str | None:
+    """Pull the source commit the run was built from (best-effort).
+
+    Reads ``version_info.dev.commit_hash`` (the benchmark backend's dev block)
+    with a couple of common fallbacks. Returns ``None`` when unavailable.
+    """
+    version = result.get("version_info")
+    if not isinstance(version, dict):
+        return None
+    dev = version.get("dev")
+    if isinstance(dev, dict):
+        for key in ("commit_hash", "commit_hash_short", "commit"):
+            val = dev.get(key)
+            if isinstance(val, str) and val:
+                return val
+    for key in ("commit_hash", "commit"):
+        val = version.get(key)
+        if isinstance(val, str) and val:
+            return val
+    return None
+
+
+# Provenance keys that define the *environment* (perf regime), and thus the
+# history bucket. GPU is already encoded in the file name, so it is excluded.
+_FINGERPRINT_KEYS = ("warp", "isaaclab", "cuda")
+
+
+def env_fingerprint(result: dict) -> str | None:
+    """Compute a short, stable bucket key from a run's environment provenance.
+
+    The fingerprint partitions the rolling-window history so that samples from
+    incomparable software stacks (e.g. a Warp bump that shifts the perf regime)
+    never pollute one another's baseline. Returns ``None`` when no provenance is
+    available, which makes callers fall back to the flat ("default") bucket.
+    """
+    prov = _extract_provenance(result)
+    parts = {key: prov[key] for key in _FINGERPRINT_KEYS if prov.get(key)}
+    if not parts:
+        return None
+    canonical = json.dumps(parts, sort_keys=True, separators=(",", ":"))
+    return "env-" + hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:12]
+
+
+def sample_provenance(result: dict) -> dict[str, object]:
+    """Return the per-sample provenance stamped into each rolling-window record.
+
+    Bundles the environment versions (warp / isaaclab / cuda), the source commit,
+    and the derived :func:`env_fingerprint` so every stored sample is auditable
+    and re-bucketable without re-running the benchmark.
+    """
+    prov = _extract_provenance(result)
+    commit = _extract_commit(result)
+    if commit:
+        prov["commit"] = commit
+    fingerprint = env_fingerprint(result)
+    if fingerprint:
+        prov["fingerprint"] = fingerprint
+    return prov
+
+
 def _extract_gpu_name(result: dict) -> str | None:
     """Read the runner's GPU model name from the result's hardware metadata."""
     hw = result.get("hardware_info")
@@ -440,6 +501,15 @@ def _resolve_baseline(
     return matched_key, task_entry, entry
 
 
+def history_basename(task: str, gpu_key: str) -> str:
+    """Filesystem-safe ``<task>__<gpu>`` stem shared by the reader and writers.
+
+    Centralising this keeps the comparator (reader) and rebaseline/seed scripts
+    (writers) byte-for-byte consistent so bucketed history is always found.
+    """
+    return f"{task}__{gpu_key}".replace("/", "_").replace(" ", "_")
+
+
 def _history_window(history_dir: str | None, fingerprint: str | None, task: str, gpu_key: str) -> dict:
     """Load the rolling-window samples for ``(task, gpu)`` from the history store.
 
@@ -449,7 +519,7 @@ def _history_window(history_dir: str | None, fingerprint: str | None, task: str,
     """
     if not history_dir:
         return {}
-    safe = f"{task}__{gpu_key}".replace("/", "_").replace(" ", "_")
+    safe = history_basename(task, gpu_key)
     candidates = []
     if fingerprint:
         candidates.append(Path(history_dir) / fingerprint / f"{safe}.json")
@@ -544,7 +614,11 @@ def main(argv: list[str] | None = None) -> int:
     parser.add_argument("--baseline", required=True, help="Path to baseline.json (run config + static fallback).")
     parser.add_argument("--history-dir", default=None, help="Rolling-window store (orphan-branch checkout).")
     parser.add_argument("--overrides", default=None, help="Path to baseline_overrides.json (committed with the PR).")
-    parser.add_argument("--fingerprint", default=None, help="History bucket key (git-subtree+deps hash).")
+    parser.add_argument(
+        "--fingerprint",
+        default=None,
+        help="History bucket key; overrides the env fingerprint auto-derived from the result.",
+    )
     parser.add_argument("--measured-wall-s", type=float, default=None, help="Wall-clock seconds of the run.")
     parser.add_argument("--results-glob", default=None, help=f"Result glob (defaults to {DEFAULT_GLOB_TEMPLATE!r}).")
     parser.add_argument("--gpu-override", default=None, help="Override the GPU name read from the result JSON.")
@@ -581,7 +655,8 @@ def main(argv: list[str] | None = None) -> int:
         _emit("PASS", task=args.task, gpu=gpu_key, note="skipped_by_override")
         return EXIT_PASS
 
-    window = _history_window(args.history_dir, args.fingerprint, args.task, gpu_key)
+    fingerprint = args.fingerprint or env_fingerprint(result)
+    window = _history_window(args.history_dir, fingerprint, args.task, gpu_key)
     center, spread, k_warn, k_block, source = _thresholds(window, entry, ov)
     delta_pct = (measured_fps - center) / center * 100.0
     warn_floor = center - k_warn * spread
@@ -590,6 +665,7 @@ def main(argv: list[str] | None = None) -> int:
     common: dict[str, object] = {
         "task": args.task,
         "gpu": gpu_key,
+        "bucket": fingerprint or "flat",
         "thresholds": source,
         "center_fps": f"{center:.0f}",
         "measured_fps": f"{measured_fps:.0f}",