[ci] Add performance dashboard metadata and visualizations

[Satyam-53]

Purpose

Add a local FastAPI + React performance dashboard for visualizing benchmark records from the Hugging
Face performance-tracking repo, and extend performance records with run-source metadata so PR, local/
manual, and scheduled-main runs can be shown and filtered correctly.

Fixes #

Changes

• Added dashboard API support for run-source filtering and metadata fields, including run_source,
  baseline_eligible, branch, PR number, Buildkite URL, build ID, and job ID.
• Updated dashboard frontend to display metric trends with axes, units, exact hover values, commit/
  source metadata, latest status, baseline eligibility, and source filters.
• Updated performance record normalization to include run-source metadata.
• Replaced hardcoded “persist only on full main” behavior with PERF_UPLOAD_POLICY:
  • never: do not upload
  • pass: upload only passing records
  • always: upload passing and failing records
• Added baseline eligibility logic so only successful scheduled-main records affect rolling regression
  baselines.
• Updated HF record loading to support baseline_eligible_only.
• Propagated Buildkite metadata into performance jobs for dashboard traceability.
• Updated performance benchmark/dashboard documentation.
• Added backend, API, and policy tests for the new behavior.

Test Plan

conda run -n fastvideo python -m pytest \
  fastvideo/tests/performance/test_compare_baseline_policy.py \
  fastvideo/tests/performance/test_dashboard_service.py \
  fastvideo/tests/performance/test_dashboard_api.py \
  -q

 conda run -n fastvideo python -m compileall -q \
   fastvideo/performance_dashboard \
   fastvideo/tests/performance/compare_baseline.py \
   fastvideo/tests/performance/hf_store.py

 cd ../..
 conda run -n fastvideo pre-commit run --files \
   .buildkite/scripts/pr_test.sh \
   docs/contributing/performance_benchmarks.md \
   fastvideo/performance_dashboard/api.py \
   fastvideo/performance_dashboard/service.py \
   fastvideo/tests/performance/compare_baseline.py \
   fastvideo/tests/performance/hf_store.py \
   fastvideo/tests/performance/test_compare_baseline_policy.py \
   fastvideo/tests/performance/test_dashboard_api.py \
   fastvideo/tests/performance/test_dashboard_service.py \
   performance_dashboard/README.md \
   performance_dashboard/frontend/src/App.tsx \
   performance_dashboard/frontend/src/api.ts \
   performance_dashboard/frontend/src/styles.css

Test Results

Test output

.................. [100%]
18 passed in 1.20s

fastvideo-performance-dashboard@0.1.0 build
ruff (legacy alias)......................................................Passed
codespell................................................................Passed
PyMarkdown...............................................................Passed
Lint GitHub Actions workflow files...................(no files to check)Skipped
mypy.....................................................................Passed
Check for spaces in all filenames........................................Passed
Suggestion................................................................Passed

Checklist

• I ran pre-commit run --all-files and fixed all issues
• I added or updated tests for my changes
• I updated documentation if needed
• I considered GPU memory impact of my changes

[mergify]

⚠️ PR title format required

Your PR title must start with a type tag in brackets. Examples:

• [feat] Add new model support
• [bugfix] Fix VAE tiling corruption
• [refactor] Restructure training pipeline
• [perf] Optimize attention kernel
• [ci] Update test infrastructure
• [infra] Add activation trace hooks
• [docs] Add inference guide
• [misc] Clean up configs
• [new-model] Port Flux2 to FastVideo
• [skill] Add add-model agent skill

Valid tags: feat, feature, bugfix, fix, refactor, perf, ci, infra, doc, docs, misc, chore, kernel, new-model, skill, skills

Please update your PR title and the merge protection check will pass automatically.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟠 PR merge requirements

Waiting for

• check-success=full-suite-passed

Waiting checks: full-suite-passed.

• check-success=full-suite-passed
• #approved-reviews-by>=1
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[gemini-code-assist]

Code Review

This pull request introduces source metadata tracking (such as PR, local, and scheduled main runs) and configurable upload policies for performance benchmarks, ensuring only successful scheduled-main runs are eligible for the CI gating baseline. It also updates the performance dashboard API and frontend to support filtering and visualizing these run sources. Feedback on the changes highlights two key issues: first, filtering by run_source before computing the latest summary restricts the baseline pool to that specific source, which can be resolved by applying the filter only when selecting the latest record; second, strictly requiring the baseline_eligible field excludes legacy records, which can be mitigated by falling back to treating them as eligible if both baseline_eligible and run_source are missing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[SolitaryThinker]

Hi @Satyam-53 — automated review from Gob, one of @SolitaryThinker's AI reviewers. Findings aren't all human-verified; ping @SolitaryThinker if anything looks off.

Verdict: COMMENT

Open PR. The Python logic, metadata schema, and React frontend are internally consistent and well-tested. But two CI env-plumbing gaps mean the feature is largely inert in CI as wired — worth fixing before merge. No blockers (nothing crashes/corrupts; both gaps are silent no-ops with trivial fixes).

Major concerns

MAJOR — PERF_UPLOAD_POLICY defaults to never and is set nowhere in the repo, so uploads stop entirely. compare_baseline.py:50,70-72. The old gate persisted on TEST_SCOPE=full && BUILDKITE_BRANCH=main (the scheduled-main suite); the new _should_persist_tracking() returns _normalized_upload_policy() != "never", and a repo-wide grep finds PERF_UPLOAD_POLICY only in this file's own definition + the docs/tests — never in .buildkite/pipeline.yml, pr_test.sh, or pr_test.py. So _upload_allowed() is always False: no record is ever uploaded, none ever becomes baseline_eligible, and (combined with the strict baseline_eligible_only=True filter) the rolling gating baseline goes permanently cold. Fix: set PERF_UPLOAD_POLICY=pass for PR/direct and =always for scheduled-main in the perf pipeline. If you configure it in the Buildkite UI rather than in-repo, please confirm — otherwise persistence is silently dead.

MAJOR — the new Buildkite metadata never reaches the container, so the dashboard "Buildkite" link is dead-on-arrival. .buildkite/scripts/pr_test.sh:79 + fastvideo/tests/modal/pr_test.py:18-35. BUILDKITE_BUILD_URL/BUILDKITE_BUILD_ID/BUILDKITE_JOB_ID are added to MODAL_ENV, but the Modal container only receives env through the fixed allow-list in image.env({...}) (which lists BUILDKITE_REPO/COMMIT/PULL_REQUEST/BRANCH, TEST_SCOPE, IMAGE_VERSION, HF_REPO_ID). That dict isn't updated here, so os.environ.get("BUILDKITE_BUILD_URL", "") in _record_metadata (compare_baseline.py:385-387) always reads "", so build_url/build_id/job_id are empty in every record and the frontend's build_url ? <a>Buildkite</a> : null never renders. (run_source/branch/pr_number work — they derive from already-allow-listed vars.) Fix: add the three keys to image.env({...}) in pr_test.py.

Minor

• PERF_PYTEST_RC is read but never exported by the runner. compare_baseline.py:114-116. run_performance_tests computes a local shell PYTEST_RC (pr_test.py:348) but never exports it as PERF_PYTEST_RC, so the "upload scheduled-main failures as success=false" path the docs advertise (performance_benchmarks.md:99) can't actually fire. Export it or drop the claim.
• Strict baseline_eligible_only drops all legacy HF records on rollout. hf_store.py:233 (also gemini). Existing records predate the field, so every baseline empties to "Initializing" on first deploy. Consider treating a record as eligible when both baseline_eligible and run_source are absent (legacy fallback).
• summary filters by run_source before computing the latest baseline (api.py:150, gemini). Fine for a dashboard view, but confirm that's intended vs. "always show true latest."
• METADATA_KEYS (service.py:21-30) is defined but unused — the code uses the explicit record_metadata() dict. Remove or wire it.

Notes (not findings)

• The build / Deploy Documentation CI failure is a pre-existing unrelated broken link (docs/training/examples/wan_t2v_1.3B_mixkit.md:59), a file this PR doesn't touch — not a regression here.
• Metadata schema is consistent across the Python record, dashboard service, and api.ts types; frontend SVG/axis math and null-handling look sound; secret scan is clean.

— Gob (@SolitaryThinker's AI reviewer).

[SolitaryThinker]

Hi @Satyam-53 — automated review from Gob, one of @SolitaryThinker's AI reviewers. Findings aren't all human-verified; ping @SolitaryThinker if anything looks off.

Verdict: COMMENT (re-review @ e9494e4f)

Thanks for the quick turnaround. This is a differential re-review of the new commit on top of my prior comment (@ bd49f61c). All previously-raised findings are resolved — verified end-to-end (I re-ran the new policy logic standalone; 24/24 assertions pass).

Prior findings — all fixed ✅

• MAJOR PERF_UPLOAD_POLICY was set nowhere → uploads froze. Now set per-context in pr_test.py:352-364: always for scheduled-main, pass for PR/direct, never fallback. End-to-end path closed.
• MAJOR Buildkite vars never reached the container → dead dashboard link. BUILDKITE_BUILD_URL/BUILD_ID/JOB_ID added to the image.env({...}) allow-list (pr_test.py:29-34) and to MODAL_ENV (pr_test.sh:79). build_url now populates and the frontend <a>Buildkite</a> renders.
• MINOR PERF_PYTEST_RC never exported. Now exported (pr_test.py:369) with a run-gate that lets scheduled-main upload failure records.
• MINOR METADATA_KEYS dead constant. Removed.
• MINOR legacy HF records dropped on rollout (also gemini). is_baseline_eligible_record() (hf_store.py:51-61) adds the legacy fallback; covered by a new test.
• MINOR summary shrank the baseline pool by run_source (gemini). build_latest_summary now keeps the full eligible pool and applies the source filter only to the "latest" pick; covered by test_build_latest_summary_run_source_filter_keeps_canonical_baseline.

Two small new-behavior notes (non-blocking)

• compare_baseline.py:70-72 + pr_test.py:356-360 — with policy='pass', PR/direct full-suite perf runs now do a strict HF sync + strict upload, which they didn't before. Net: (1) a transient HF outage now fails a PR perf job loud instead of degrading to "Initializing"; (2) PR records (correctly baseline_eligible=False, so no gating pollution) now accumulate in the shared dataset with no visible TTL. Both look intentional per the docs — just confirm the PR-build token has write scope and that unbounded PR-record growth is acceptable (or add a retention note).
• pr_test.py:355 — a /test full run on a PR classifies as pr/pass (not direct) because BUILDKITE_PULL_REQUEST is set; the TEST_SCOPE==direct branch only catches direct runs off a PR. Reasonable, but the elif ordering is load-bearing — a one-line comment would help.

Nice, thorough fix — schema is consistent across the Python record / dashboard service / api.ts types, the frontend axis math + null-handling look sound, and the secret scan is clean. CI is green (pre-commit, fastcheck-passed). The still-red build / Deploy-Documentation check is the pre-existing unrelated broken link (docs/training/examples/wan_t2v_1.3B_mixkit.md:59), not from this PR.

— Gob (@SolitaryThinker's AI reviewer).

[Satyam-53]

Only Check docs link is failing in the above tests which got introduced in the pr- #1462 by @alexzms and is unrelated to this pr.

@alexzms can you please take a look at that issue. Else we are good to merge this PR @SolitaryThinker .