Add lightweight performance smoke comparisons

[AntoineRichard]

Description

This is a focused alternative to #6864. It uses IsaacLab's existing
benchmark runtime CLI to run the performance workloads and adds only the
missing comparison and CI wiring.

The PR:

• runs eight supported benchmark configurations against the exact PR CI image;
• compares total FPS, total startup time, peak GPU memory, and peak
  benchmark-process RSS;
• keeps the established arithmetic mean of per-step FPS by default, while
  exposing a single opt-in --aggregate_throughput flag;
• requires an FPS regression to cross both its percentage threshold and a
  two-standard-error significance threshold computed from coherent per-step
  FPS statistics;
• downloads a read-only authoritative baseline table from the repository
  variable PERF_BASELINE_S3_URI;
• publishes a Markdown workflow summary, a machine-readable JSON comparison,
  and the raw benchmark bundle;
• matches exact provider-qualified Isaac Sim or kit-less Newton runtime
  identities and fails closed on malformed, missing, ambiguous, or non-finite
  input; and
• adds no dependency, baseline-writing path, reseeding workflow, custom GitHub
  status, or parallel benchmark implementation.

Until the S3 bucket is provisioned, leaving PERF_BASELINE_S3_URI unset makes
the comparison explicitly report SKIP; benchmarks and artifacts are still
produced. Once configured, the workflow only requires s3:GetObject access.
Every matching baseline row must provide all four metrics. The FPS row also
stores the reference per-step FPS standard deviation and sample count so the
comparison can reject noisy percentage regressions that are not statistically
significant.

The implementation is 799 additions across eight files, including exactly six
focused tests. This is approximately 92% smaller than the current proposal
while retaining the gating and reporting behavior needed for CI.

Local validation:

• focused CPU-only checks covered both CLI modes, typed-request forwarding,
  and all five comparator scenarios;
• ./isaaclab.sh -f passed immediately before commit and again immediately
  before push;
• real downloaded Newton CI bundles for Cartpole and G1 produce complete
  four-metric SKIP reports while S3 is unconfigured, including startup,
  memory, and measured per-step FPS deviation; and
• the earlier GPU smoke exercised the real runtime bundle and comparison
  outputs. No GPU benchmark was rerun for these reporting-only trims because
  the shared machine is currently busy.

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

Not applicable; this change affects CI reports and artifacts.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh -f
• Public API and CLI documentation are updated in the runtime request docstring and CLI help
• My changes generate no new warnings
• I have added tests that prove my feature works
• I added the required source/isaaclab/changelog.d/ fragment
• My name already exists in CONTRIBUTORS.md

[greptile-apps]

Greptile Summary

The PR adds an eight-configuration GPU performance-smoke matrix that runs the existing runtime benchmark against the PR image and compares throughput and memory metrics with an optional S3 baseline.

• Adds CI benchmark execution, baseline download, report publication, and artifact upload.
• Adds a comparator that emits Markdown, JSON, and JUnit results with PASS, WARN, FAIL, SKIP, or ERROR outcomes.
• Adds focused tests for regression direction, thresholds, malformed baselines, output generation, and unconfigured-baseline behavior.

Confidence Score: 4/5

The PR appears safe to merge, with the non-blocking exception that the new self-hosted CI job should pin its third-party actions to immutable commits.

The comparison implementation aligns with the existing runtime benchmark schema, while the only accepted concern is the avoidable supply-chain exposure from mutable action tags in the new job.

Files Needing Attention: .github/workflows/build.yaml

Security Review

The new self-hosted performance job uses mutable major-version tags for checkout and artifact upload. Pinning both actions to reviewed commit SHAs would prevent upstream tag movement from changing code executed on the runner.

Important Files Changed

Filename	Overview
.github/workflows/build.yaml	Adds the GPU benchmark matrix and comparison pipeline; its new third-party action references remain mutable.
tools/performance_compare.py	Implements strict baseline identity matching, metric regression evaluation, and three report formats with explicit error handling.
tools/test/test_performance_compare.py	Covers metric direction and boundaries, invalid baseline handling, generated outputs, and baseline-disabled behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Build PR CI image] --> B[Run benchmark matrix]
  B --> C[Raw runtime JSON]
  D[Optional S3 baseline] --> E[Performance comparator]
  C --> E
  E --> F[Markdown report]
  E --> G[JSON report]
  E --> H[JUnit report]
  F --> I[Workflow summary and artifact]
  G --> I
  H --> I

Loading

_{Reviews (1): Last reviewed commit: "Harden performance smoke validation" | Re-trigger Greptile}

[greptile-apps]

Mutable actions on self-hosted runner

The new performance job resolves actions/checkout@v6 and actions/upload-artifact@v7 through mutable tags on a self-hosted GPU runner. Pin both references to full commit SHAs so upstream tag movement cannot change the code executed by this workflow or expose runner resources and benchmark artifacts.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[isaaclab-review-bot]

Isaac Lab Review Bot

Adds a matrix-driven performance smoke workflow using the existing runtime benchmark producer and a dependency-free comparison tool that matches benchmark identity against an optional S3 baseline and emits Markdown, JSON, and JUnit results. The workflow should use the repository-mandated uv entry points before merge.

• Design and architecture: The separation between benchmark production and baseline comparison is sound. Exact matching across task, backends, presets, GPU model, Isaac Sim version, and environment count provides deterministic baseline selection, while an unconfigured baseline produces an explicit SKIP.
• API: The new tooling does not alter public package APIs. Its CLI arguments follow the repository's snake_case convention, and no source-package documentation or changelog integration is required.
• Implementation: The metric extraction, throughput-versus-memory regression directions, baseline validation, report generation, and FAIL/ERROR exit behavior are internally consistent. However, the workflow runs the benchmark through ./isaaclab.sh and the comparison through bare python3, contrary to the trusted tooling requirement to use uv run isaaclab and uv run python for routine CLI and script execution.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🔵 Suggestion · Implementation — CI bypasses uv-managed entry points

The benchmark is launched with ./isaaclab.sh benchmark runtime, and line 949 runs the comparison with a bare host python3. Repository tooling rules reserve ./isaaclab.sh for installer workflows (-i <args>) and direct Isaac Lab CLI and script execution to uv run isaaclab benchmark ... and uv run python tools/performance_compare.py. Using the documented entry points keeps the job aligned with the supported CLI surface and avoids depending on an unmanaged interpreter.

[AntoineRichard]

Closing in favor of #7137