[ci] [Feature] Add DGX Spark (GB10) performance baseline and CI stress coverage

[Satyam-53]

Motivation

Add a DGX Spark / GB10 cohort to the FastVideo performance dashboard and establish an initial baseline for the machine. This would also exercise the performance CI on CUDA 13, ARM64, and compute capability 12.1 hardware.

Proposed work

• Add a GB10-specific performance benchmark cohort and dashboard baseline.
• Adopt the single-GPU Wan2.1-T2V-1.3B configuration.
• Gate GB10-specific configurations so they do not run automatically on the existing H100/L40S lanes.
• Ensure the benchmark identity distinguishes GPU type, architecture, software environment, and single-GPU topology.
• Exercise the full collection, comparison, upload, and dashboard-ingestion path on the DGX Spark runner.
• Use repeated CI runs to identify flakiness, thermal/runtime variance, memory instability, and ARM64/CUDA 13 compatibility issues.

Relevant Tickets

#1575 #1374

[Mister-Raggs]

I ran FastVideo's perf benchmark on a DGX Spark (GB10) — initial baseline + the single-GPU config below, matching the proposed work.

Environment: GB10 (cc 12.1), ARM64, CUDA 13, torch 2.12.0+cu130, transformers 5.14.0.

Single-GPU config (→ "Adopt the single-GPU Wan2.1-T2V-1.3B configuration"): single-GPU variant of wan-t2v-1.3b.json — same workload_id, num_gpus=1/sp_size=1, 480×832×45, 4 steps, guidance 3, config_schema_version: 2, GB10/default thresholds. Can share the config + raw result JSON, or open it as a gated PR.

Gating (→ "Gate GB10-specific configurations…"): the suite globs every tests/*.json, so this single-GPU config runs on the H100/L40S lane unless guarded by device/topology.

Baseline (median of 5, 2 warmup):

• gen time: 40.6 s
• peak memory: 8.4 GB (pipeline peak_memory_mb, worker-measured; nvidia-smi reads [N/A] on unified memory)

Collection path: pytest fastvideo/tests/performance/test_inference_performance.py on the GB10 ran the single-GPU config and wrote a normalized record; the 2-GPU config auto-skips on 1 GPU. Collection + write work on ARM64 / CUDA 13 / sm_121. No dashboard upload from me — leaving ingest to this effort.

Reusable patterns: #1591 (pin VSA training to H100) is the closest precedent for keeping GB10 off the H100/L40S lanes. Config stays on the v2 identity schema (#1544 / #1551) so GPU/arch/topology are distinguished; can add an explicit single-GPU-topology field if preferred.

Context: #1631 (DGX Spark perf & tuning guide) has the broader findings — few-step is decode-bound, bf16 VAE decode is lossless and the Wan default, FP4 attention works on sm_121 (#1598).

Workloads: few-step is the practical regime (~40 s/gen); full-length is slow (Wan 50-step ~12 min, Cosmos-2.5 ~47 min), so one full-length run is the relevant thermal/runtime long-runner. Happy to also run baseline vs stress models for specific workloads / models.

[Satyam-53]

Thank you for running this @Mister-Raggs. Can. you please open the PR with a separate config file for the GB10.
Also, can you try producing more baselines with the same config and setup to see if we are getting the same latency, memory and throughput metrics, by running the perf CI tests locally on the GB10 machine. If yes, then let's upload them to the HF hub to serve as the baseline for future runs.

[Mister-Raggs]

@Satyam-53
Ran the gated single-GPU GB10 config 5× — metrics are stable: generation 39.18–39.34 s (0.4% spread), peak memory 8429.7 MB (identical across all 5), throughput ~1.145 fps. Component split: DiT ~26 s, VAE decode ~11.3 s, text-encode ~1.6 s.

Config PR: #1679 — adds the config + an opt-in run_config.gpu_types gate so it skips on the H100/L40S lanes, with thresholds grounded in the above.

One thing on seeding the baseline: seed_baseline.py requires a scheduled_main full-suite source artifact (run_source=scheduled_main, branch=main, test_scope=full), so a local box run can't produce a baseline-eligible record by design. A baseline-eligible seed needs a scheduled-main perf run on a GB10 CI runner once one's wired in — the 5× run above is the stability evidence for it. If you want the numbers visible on the dashboard in the interim I can push a local-source record (not baseline-eligible).

[Satyam-53]

@Mister-Raggs let's push the numbers so that they are visible on the dashboard atleast, it's fine if they remain baseline ineligible for now.

[Mister-Raggs]

@Satyam-53
Pushed the five GB10 runs to FastVideo/performance-tracking, they're on the dashboard now as baseline-ineligible records (baseline_eligible=false), so they're visible but don't move the rolling baseline, as agreed.

Numbers held consistent on the re-run: generation 39.16–39.43 s, peak 8429.7 MB (identical across all 5), throughput ~1.14–1.15 fps; DiT ~26 s, VAE decode ~11.2 s, text-encode ~1.6 s. Status is CALIBRATION_NEEDED for the GB10 identity (hardware_profile_id=hw-e40b9140c67b134f), as expected until a baseline exists.

Baseline-eligible seeding remains the follow-up captured in #1679 — a scheduled-main GB10 run or a scoped manual calibration path.