Skip to content

Commit ca7907d

Browse files
Neil4561Neil4561
authored
Add perf-gate baseline seeding workflow (#4)
Register the manual performance regression gate baseline seeding workflow on develop so it can be dispatched. GitHub only exposes a workflow_dispatch "Run workflow" control when the workflow file exists on the default branch, so the workflow must live here even though the rest of the perf gate does not. The workflow is manual only (no automatic push or pull_request triggers) and is dispatched against the perf gate POC branch, where the seeding orchestrator and the rest of the gate tooling live. It re-runs the benchmark at historical commits of one or more branches, tags each sample with its real commit and target branch, and appends the results to the perf-baselines branch. This is added now to unblock collecting baseline seed data for the gate.
1 parent 305b23a commit ca7907d

1 file changed

Lines changed: 189 additions & 0 deletions

File tree

‎.github/workflows/perf-gate-seed-baselines.yaml‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2+
# All rights reserved.
3+
#
4+
# SPDX-License-Identifier: BSD-3-Clause
5+
6+
# Seed the perf-baselines branch from real commit history.
7+
#
8+
# This is the deployment-simulation counterpart to the live gate: instead of
9+
# benchmarking one PR head, it walks a slice of a protected branch's history,
10+
# re-runs each commit's own benchmark inside the CI image (source-mounted so the
11+
# container git-tags the real commit), and appends the results to perf-baselines.
12+
# The samples are keyed by their real commit SHA, so the gate's merge-base /
13+
# ancestry isolation has a populated, branch-correct baseline to compare against.
14+
#
15+
# Requires the prebuilt-image fast path: set repo variable PERF_GATE_CI_IMAGE to
16+
# the published CI image (GHCR first, NGC/NVCR fallback). Without it there is no
17+
# image to source-mount into, and the job fails fast with a clear message.
18+
#
19+
# Manual dispatch only — seeding writes to perf-baselines and burns GPU time, so
20+
# it should never run automatically.
21+
22+
name: Performance Gate - Seed Baselines
23+
24+
on:
25+
workflow_dispatch:
26+
inputs:
27+
branches:
28+
description: "Branches to seed in one run (comma/space separated). Use 'branch:target' to override the stamp. Empty = use commit_branch/commits."
29+
required: false
30+
default: "develop, unified-POC"
31+
commits:
32+
description: "Explicit commit SHAs/refs (space/comma separated). Used only when branches is empty."
33+
required: false
34+
default: ""
35+
commit_branch:
36+
description: "Single branch to seed when branches and commits are empty."
37+
required: false
38+
default: "develop"
39+
commit_count:
40+
description: "Number of recent commits to seed from commit_branch."
41+
required: false
42+
default: "5"
43+
samples_per_commit:
44+
description: "Benchmark repetitions per commit/backend."
45+
required: false
46+
default: "5"
47+
tasks:
48+
description: "Comma-separated task_id allowlist (empty = all tasks.json tasks)."
49+
required: false
50+
default: "Isaac-Cartpole-Direct"
51+
backends:
52+
description: "Comma-separated backend_key allowlist (empty = all backends)."
53+
required: false
54+
default: ""
55+
target_branch:
56+
description: "Protected branch stamped onto each seeded sample."
57+
required: false
58+
default: "develop"
59+
dry_run:
60+
description: "Run benchmarks + build samples but DO NOT push to perf-baselines."
61+
type: boolean
62+
required: false
63+
default: false
64+
65+
concurrency:
66+
# Serialize seeding so concurrent pushes can't race the append-only baseline branch.
67+
group: perf-gate-seed
68+
cancel-in-progress: false
69+
70+
permissions:
71+
contents: write # push appended samples to the perf-baselines branch
72+
packages: read # pull the prebuilt CI image from GHCR (PERF_GATE_CI_IMAGE)
73+
74+
env:
75+
NGC_API_KEY: ${{ secrets.NGC_API_KEY }}
76+
CI_IMAGE_TAG: isaac-lab-ci:seed-${{ github.run_id }}
77+
78+
jobs:
79+
seed:
80+
name: Seed baselines (${{ inputs.commit_branch }} x${{ inputs.commit_count }})
81+
runs-on: ${{ fromJSON(vars.PERF_GATE_RUNS_ON || '["linux-amd64-gpu-rtxpro6000-latest-1"]') }}
82+
timeout-minutes: 600
83+
84+
steps:
85+
- name: Checkout Code
86+
uses: actions/checkout@v6
87+
with:
88+
# Full history so historical commits are resolvable and ancestry checks work.
89+
fetch-depth: 0
90+
lfs: true
91+
92+
# Make the seed/target branch history and the baseline branch available to the
93+
# orchestrator. The repo only auto-fetches the checked-out ref, so pull these
94+
# explicitly; the clone the orchestrator makes draws its objects from here.
95+
- name: Fetch seed + baseline refs
96+
env:
97+
SEED_BRANCHES: ${{ inputs.branches }}
98+
SEED_COMMIT_BRANCH: ${{ inputs.commit_branch }}
99+
run: |
100+
set -euo pipefail
101+
# Collect every branch we might seed from: the multi-branch list (stripping
102+
# any ':target' suffix) plus the single-branch fallback. Fetch each so the
103+
# orchestrator's clone can resolve their commits offline.
104+
BRANCHES="${SEED_BRANCHES//,/ } ${SEED_COMMIT_BRANCH}"
105+
for entry in ${BRANCHES}; do
106+
BRANCH="${entry%%:*}"
107+
[ -z "${BRANCH}" ] && continue
108+
git fetch --no-tags origin "+refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}" || \
109+
echo "::warning::Could not fetch ${BRANCH}; skipping (explicit commits may still work)"
110+
done
111+
git fetch --no-tags origin "+refs/heads/perf-baselines:refs/remotes/origin/perf-baselines" || \
112+
echo "::notice::perf-baselines not found yet; first seed run will create it"
113+
114+
- name: Require prebuilt CI image
115+
if: ${{ vars.PERF_GATE_CI_IMAGE == '' }}
116+
run: |
117+
echo "::error::PERF_GATE_CI_IMAGE is not set. Seeding source-mounts the published"
118+
echo "::error::CI image; publish one (perf-gate-publish-image) and set the variable."
119+
exit 1
120+
121+
# Pull the published CI image and retag it locally (mirrors the gate's override path).
122+
- name: Pull prebuilt CI image
123+
env:
124+
OVERRIDE_IMAGE: ${{ vars.PERF_GATE_CI_IMAGE }}
125+
run: |
126+
set -euo pipefail
127+
128+
# The runner's docker credential store backend is broken ("not implemented"),
129+
# so disable credsStore before any login (same trick as ecr-build-push-pull).
130+
DOCKER_CONFIG_DIR="$(mktemp -d)"
131+
echo '{"credsStore":""}' > "${DOCKER_CONFIG_DIR}/config.json"
132+
export DOCKER_CONFIG="${DOCKER_CONFIG_DIR}"
133+
134+
REGISTRY="${OVERRIDE_IMAGE%%/*}"
135+
case "${REGISTRY}" in
136+
ghcr.io)
137+
echo "🔵 Logging into ghcr.io..."
138+
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
139+
;;
140+
nvcr.io)
141+
if [ -n "${NGC_API_KEY:-}" ]; then
142+
echo "🔵 Logging into nvcr.io..."
143+
echo "${NGC_API_KEY}" | docker login nvcr.io -u '$oauthtoken' --password-stdin
144+
else
145+
echo "::warning::NGC_API_KEY not set; attempting anonymous pull from nvcr.io"
146+
fi
147+
;;
148+
*)
149+
echo "::notice::No known login for registry '${REGISTRY}'; attempting anonymous pull"
150+
;;
151+
esac
152+
153+
echo "🔵 Pulling prebuilt image ${OVERRIDE_IMAGE}..."
154+
docker pull "${OVERRIDE_IMAGE}"
155+
docker tag "${OVERRIDE_IMAGE}" "${{ env.CI_IMAGE_TAG }}"
156+
echo "🟢 Tagged ${OVERRIDE_IMAGE} as ${{ env.CI_IMAGE_TAG }}"
157+
rm -rf "${DOCKER_CONFIG_DIR}"
158+
159+
- name: Seed baselines from commit history
160+
run: |
161+
set -euo pipefail
162+
GPU_MODEL="$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1 | xargs)"
163+
python3 tools/perf_regression_gate/seed_baselines.py \
164+
--branches "${{ inputs.branches }}" \
165+
--commits "${{ inputs.commits }}" \
166+
--commit_branch "${{ inputs.commit_branch }}" \
167+
--commit_count "${{ inputs.commit_count }}" \
168+
--samples_per_commit "${{ inputs.samples_per_commit }}" \
169+
--tasks "${{ inputs.tasks }}" \
170+
--backends "${{ inputs.backends }}" \
171+
--image "${{ env.CI_IMAGE_TAG }}" \
172+
--gpu_model "${GPU_MODEL}" \
173+
--target_branch "${{ inputs.target_branch }}" \
174+
--baseline_branch perf-baselines \
175+
--baseline_remote origin \
176+
--baseline_push_retries 3 \
177+
--workdir "${{ github.workspace }}" \
178+
--artifacts_root seed-artifacts \
179+
--source_mount true \
180+
--dry_run "${{ inputs.dry_run }}"
181+
182+
- name: Upload seed artifacts
183+
if: always()
184+
uses: actions/upload-artifact@v7
185+
with:
186+
name: seed-baselines-${{ github.run_id }}
187+
path: seed-artifacts/
188+
retention-days: 7
189+
if-no-files-found: warn

0 commit comments

Comments
 (0)