Skip to content

Commit 98d88f5

Browse files
Neil4561Neil4561
committed
Add perf-smoke performance regression gate
Introduce the perf-smoke gate that benchmarks a matrix of tasks/backends on the self-hosted L40S pool, compares FPS against per-GPU rolling baselines, and reports verdicts as an omni-github artifact. Baselines live on an orphan perf-baselines branch, keyed by a runtime_contract_hash so samples are only pooled across identical environments. The pinned CI image is resolved via an image-era manifest (sha-digest) instead of the floating latest-develop tag, so a baseline stays valid until the container environment actually changes. Baseline upkeep is automated: - era-roll republishes an immutable image and seeds it when the container era changes; - a post-gate under-filled-bucket detector reseeds exactly the tasks whose runtime_contract_hash window is short of MIN_BASELINE_SAMPLES after a protected develop push; - the seeder runs an ancestry preflight and verify_baselines confirms seeded samples are gate-usable. Targets perf-smoke/infra-probe for same-repo L40S validation before the gate is integrated into develop.
1 parent a1b5cd3 commit 98d88f5

41 files changed

Lines changed: 8999 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/actions/ecr-build-push-pull/action.yml‎

Lines changed: 347 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,347 @@
1+
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2+
# All rights reserved.
3+
#
4+
# SPDX-License-Identifier: BSD-3-Clause
5+
6+
name: 'ECR Build-Push-Pull'
7+
description: >
8+
Builds a Docker image and pushes it to ECR, using ECR as the layer cache.
9+
If the image already exists in ECR (same tag), pulls it instead of building.
10+
Drop-in replacement for docker-build/action.yml with ECR-backed caching.
11+
12+
inputs:
13+
image-tag:
14+
description: 'Tag for the Docker image (e.g. my-image:latest).'
15+
required: true
16+
isaacsim-base-image:
17+
description: 'IsaacSim base image (passed as ISAACSIM_BASE_IMAGE_ARG build-arg).'
18+
required: true
19+
isaacsim-version:
20+
description: 'IsaacSim version (passed as ISAACSIM_VERSION_ARG build-arg).'
21+
required: true
22+
dockerfile-path:
23+
description: 'Path to Dockerfile, relative to the repository root'
24+
default: 'docker/Dockerfile.base'
25+
required: false
26+
ecr-url:
27+
description: >
28+
ECR repository URL (e.g. "123456789.dkr.ecr.us-west-2.amazonaws.com/my-repo").
29+
Resolved in the following order:
30+
1. ecr-url input, if provided.
31+
2. ECR_CACHE_URL environment variable on the runner.
32+
3. SSM parameter /github-runner/<instance-id>/ecr-cache-url.
33+
4. If still empty, ECR cache is skipped and the image is built locally.
34+
required: false
35+
default: ''
36+
cache-tag:
37+
description: Tag used for the ECR layer cache image (e.g. "cache-base", "cache-curobo").
38+
required: false
39+
default: 'cache'
40+
gha-cache-scope:
41+
description: >
42+
When non-empty, enables a GitHub Actions (type=gha) buildx layer cache with
43+
this scope on the full-build path. Acts as a registry-free fallback when ECR
44+
is unavailable (e.g. on non-AWS fleets), and as an extra cache source when
45+
ECR is present. Leave empty (default) to preserve the original ECR-only
46+
behavior for callers that do not opt in. The scope isolates this cache from
47+
other workflows sharing the repository's Actions cache.
48+
required: false
49+
default: ''
50+
runs:
51+
using: composite
52+
steps:
53+
54+
##### 1: Setup docker config + Login to nvcr.io #####
55+
56+
# Create a temp docker config with credsStore disabled before any login.
57+
# The runner's credential store backend is broken ("not implemented") and
58+
# causes all docker login calls to fail unless we bypass it upfront.
59+
# The temp config is exported as DOCKER_CONFIG so all subsequent steps
60+
# (including ECR login in step 3) inherit it automatically.
61+
62+
- name: Setup docker config and login to nvcr.io
63+
shell: bash
64+
run: |
65+
DOCKER_CONFIG_DIR=$(mktemp -d)
66+
if [ -f "${HOME}/.docker/config.json" ]; then
67+
python3 -c "import json; cfg=json.load(open('${HOME}/.docker/config.json')); cfg['credsStore']=''; cfg.pop('credHelpers',None); json.dump(cfg,open('${DOCKER_CONFIG_DIR}/config.json','w'))"
68+
else
69+
echo '{"credsStore":""}' > "${DOCKER_CONFIG_DIR}/config.json"
70+
fi
71+
echo "DOCKER_CONFIG=${DOCKER_CONFIG_DIR}" >> "$GITHUB_ENV"
72+
export DOCKER_CONFIG="${DOCKER_CONFIG_DIR}"
73+
74+
if [ -n "${{ env.NGC_API_KEY }}" ]; then
75+
echo "🔵 Logging into nvcr.io..."
76+
docker login -u \$oauthtoken -p ${{ env.NGC_API_KEY }} nvcr.io
77+
else
78+
echo "🟠 NGC_API_KEY not set - skipping nvcr.io login (normal for fork PRs)"
79+
fi
80+
81+
##### 2: Resolve ECR URL #####
82+
83+
# Tries: explicit input >> ECR_CACHE_URL env var >> SSM parameter on EC2.
84+
# Exports ECR_URL to GITHUB_ENV and sets output `available`.
85+
86+
- name: Resolve ECR URL
87+
id: resolve-ecr
88+
shell: bash
89+
env:
90+
INPUT_ECR_URL: ${{ inputs.ecr-url }}
91+
run: |
92+
ECR_URL="${INPUT_ECR_URL:-}"
93+
94+
if [ -z "${ECR_URL}" ]; then
95+
echo "🔵 ecr-url input not set, trying ECR_CACHE_URL env var..."
96+
ECR_URL="${ECR_CACHE_URL:-}"
97+
[ -n "${ECR_URL}" ] && echo "🟢 Using ECR_CACHE_URL env var: ${ECR_URL}"
98+
fi
99+
100+
if [ -z "${ECR_URL}" ]; then
101+
echo "🔵 ECR_CACHE_URL env var not set, trying SSM..."
102+
IMDS_TOKEN=$(curl -sf -X PUT "http://169.254.169.254/latest/api/token" \
103+
-H "X-aws-ec2-metadata-token-ttl-seconds: 21600") || true
104+
INSTANCE_ID=$(curl -sf -H "X-aws-ec2-metadata-token: ${IMDS_TOKEN}" \
105+
"http://169.254.169.254/latest/meta-data/instance-id") || true
106+
INSTANCE_REGION=$(curl -sf -H "X-aws-ec2-metadata-token: ${IMDS_TOKEN}" \
107+
"http://169.254.169.254/latest/meta-data/placement/region") || true
108+
109+
if [ -n "${INSTANCE_ID}" ]; then
110+
ECR_URL=$(aws ssm get-parameter \
111+
--name "/github-runner/${INSTANCE_ID}/ecr-cache-url" \
112+
--region "${INSTANCE_REGION}" \
113+
--query 'Parameter.Value' --output text 2>/dev/null) || ECR_URL=""
114+
if [ -n "${ECR_URL}" ]; then
115+
echo "🟢 Resolved ECR URL from SSM (/github-runner/${INSTANCE_ID}/ecr-cache-url): ${ECR_URL}"
116+
else
117+
echo "🔵 SSM parameter not found for instance ${INSTANCE_ID}"
118+
fi
119+
else
120+
echo "🔵 Not running on EC2 or IMDS unavailable, skipping SSM lookup"
121+
fi
122+
fi
123+
124+
if [ -n "${ECR_URL}" ]; then
125+
echo "ECR_URL=${ECR_URL}" >> "$GITHUB_ENV"
126+
echo "available=true" >> "$GITHUB_OUTPUT"
127+
else
128+
echo "🟠 ECR URL cannot be resolved. Building locally without ECR cache."
129+
fi
130+
131+
##### 3: Setup ECR authentication #####
132+
133+
# Validates the ECR URL, derives ECR image tags, and logs into ECR.
134+
# DOCKER_CONFIG (with credsStore disabled) is already set by step 1.
135+
136+
- name: Setup ECR authentication
137+
if: steps.resolve-ecr.outputs.available == 'true'
138+
shell: bash
139+
run: |
140+
REGISTRY=$(echo "${ECR_URL}" | cut -d'/' -f1)
141+
AWS_REGION=$(echo "${REGISTRY}" | sed 's/.*\.dkr\.ecr\.\(.*\)\.amazonaws\.com/\1/')
142+
143+
if [ "${AWS_REGION}" = "${REGISTRY}" ]; then
144+
echo "🔴 Invalid ECR URL - cannot extract AWS region: ${ECR_URL}"
145+
echo "🔴 Expected format: <account-id>.dkr.ecr.<region>.amazonaws.com/<repo>"
146+
exit 1
147+
fi
148+
149+
ECR_TAG=$(echo "${{ inputs.image-tag }}" | tr ':/' '--')
150+
ECR_IMAGE="${ECR_URL}:${ECR_TAG}"
151+
CACHE_IMAGE="${ECR_URL}:${{ inputs.cache-tag }}"
152+
153+
echo "ECR_IMAGE=${ECR_IMAGE}" >> "$GITHUB_ENV"
154+
echo "CACHE_IMAGE=${CACHE_IMAGE}" >> "$GITHUB_ENV"
155+
156+
echo "🔵 Logging into ECR registry..."
157+
aws ecr get-login-password --region "${AWS_REGION}" | \
158+
docker login --username AWS --password-stdin "${REGISTRY}"
159+
160+
##### 4: Check if exact image exists in ECR #####
161+
162+
# Lightweight manifest check - fetches only the image manifest (~KB),
163+
# not the actual layers. If the exact per-commit image already exists
164+
# in ECR, sets output `hit: true` to skip all subsequent build/push steps.
165+
166+
- name: Check exact image in ECR
167+
id: pull-exact
168+
if: steps.resolve-ecr.outputs.available == 'true'
169+
shell: bash
170+
run: |
171+
echo "🔵 Checking if commit-tagged image exists in ECR >> ${ECR_IMAGE}"
172+
if docker manifest inspect "${ECR_IMAGE}" >/dev/null 2>&1; then
173+
echo "🟢 Commit-tagged image found in ECR, skipping build!"
174+
echo "hit=true" >> "$GITHUB_OUTPUT"
175+
else
176+
echo "🟠 Image ${ECR_IMAGE} not found in ECR, will try deps-cache strategy..."
177+
fi
178+
179+
# Pull the image when the manifest check succeeded but the image is not
180+
# available locally (test jobs need it for `docker run`). Build jobs
181+
# that only push to ECR will already have the image or don't need it.
182+
- name: Pull exact image from ECR
183+
if: steps.pull-exact.outputs.hit == 'true'
184+
shell: bash
185+
run: |
186+
if docker image inspect "${{ inputs.image-tag }}" >/dev/null 2>&1; then
187+
echo "🟢 Image already available locally, skipping pull"
188+
else
189+
echo "🔵 Pulling ${ECR_IMAGE} from ECR..."
190+
docker pull "${ECR_IMAGE}"
191+
docker tag "${ECR_IMAGE}" "${{ inputs.image-tag }}"
192+
echo "🟢 Image pulled and tagged as ${{ inputs.image-tag }}"
193+
fi
194+
195+
##### 5: Check deps cache #####
196+
197+
# Hashes installation-relevant files + the base image digest to produce a stable
198+
# deps-<hash> ECR tag. If the image exists in ECR, the build job succeeds
199+
# immediately and test jobs pull the deps image with a source volume mount.
200+
201+
# Edit DEPS_FILES or DEPS_MANIFEST_PATTERN when install
202+
# inputs change (new packages, new manifests, etc.).
203+
204+
- name: Check deps cache
205+
id: deps-cache
206+
if: steps.resolve-ecr.outputs.available == 'true' && steps.pull-exact.outputs.hit != 'true'
207+
shell: bash
208+
run: |
209+
##### Deps-hash configuration #####
210+
# Exact files/dirs whose full content is hashed. The Dockerfile is first.
211+
DEPS_FILES=(
212+
"${{ inputs.dockerfile-path }}"
213+
isaaclab.sh
214+
environment.yml
215+
source/isaaclab/isaaclab/cli
216+
)
217+
# Manifest files matched repo-wide via git ls-files.
218+
DEPS_MANIFEST_PATTERN='(setup\.py|pyproject\.toml|setup\.cfg|extension\.toml|requirements[^/]*\.txt|uv\.lock)$'
219+
220+
# Resolve the actual base image digest so a new push of a mutable tag
221+
# (e.g. latest-develop) invalidates the deps cache automatically.
222+
BASE_IMAGE_DIGEST=$(docker buildx imagetools inspect \
223+
"${{ inputs.isaacsim-base-image }}:${{ inputs.isaacsim-version }}" \
224+
--format '{{json .Manifest.Digest}}' 2>/dev/null | tr -d '"' || true)
225+
if [ -n "${BASE_IMAGE_DIGEST}" ]; then
226+
BASE_IMAGE_UNIQ_ID="${{ inputs.isaacsim-base-image }}:${{ inputs.isaacsim-version }}:${BASE_IMAGE_DIGEST}"
227+
else
228+
echo "🟠 Could not resolve base image digest, falling back to tag string"
229+
BASE_IMAGE_UNIQ_ID="${{ inputs.isaacsim-base-image }}:${{ inputs.isaacsim-version }}"
230+
fi
231+
232+
echo "🔵 Base image ID: ${BASE_IMAGE_UNIQ_ID}"
233+
234+
MANIFEST_FILES=$(git ls-files | grep -E "${DEPS_MANIFEST_PATTERN}" || true)
235+
FILE_HASH=$(git ls-files -s "${DEPS_FILES[@]}" ${MANIFEST_FILES} 2>/dev/null \
236+
| sha256sum | cut -c1-16)
237+
DEPS_HASH=$(printf '%s %s' "${FILE_HASH}" "${BASE_IMAGE_UNIQ_ID}" | sha256sum | cut -c1-16)
238+
DEPS_ECR_IMAGE="${ECR_URL}:deps-${DEPS_HASH}"
239+
echo "🔵 Deps hash: ${DEPS_HASH}"
240+
echo "🔵 Checking if deps image ${DEPS_ECR_IMAGE} exists in ECR..."
241+
242+
# Lightweight manifest check - fetches only the image manifest (~KB),
243+
# not the actual layers, so this completes in seconds.
244+
if docker manifest inspect "${DEPS_ECR_IMAGE}" >/dev/null 2>&1; then
245+
echo "🟢 Deps cache HIT!!! Image exists in ECR: ${DEPS_ECR_IMAGE}"
246+
# Create a commit-tagged alias pointing to the same manifest (registry-side,
247+
# no layer download). Test jobs will pull this tag normally.
248+
echo "🔵 Tagging as commit image ${ECR_IMAGE}..."
249+
docker buildx imagetools create -t "${ECR_IMAGE}" "${DEPS_ECR_IMAGE}"
250+
echo "🟢 Tagged ${ECR_IMAGE} >> ${DEPS_ECR_IMAGE}"
251+
echo "deps-cache-hit=true" >> "$GITHUB_OUTPUT"
252+
else
253+
echo "🟠 Deps cache MISS 😿😿😿 (${DEPS_HASH}). Will build now. 🐢🐢🐢"
254+
echo "DEPS_ECR_IMAGE=${DEPS_ECR_IMAGE}" >> "$GITHUB_ENV"
255+
echo "PUSH_DEPS_IMAGE=true" >> "$GITHUB_ENV"
256+
fi
257+
258+
##### 6: Full build #####
259+
260+
# Runs when neither the exact image nor the deps cache was available.
261+
# Uses ECR layer cache (--cache-from/--cache-to) when ECR is available.
262+
263+
- name: Full build
264+
if: steps.pull-exact.outputs.hit != 'true' && steps.deps-cache.outputs.deps-cache-hit != 'true'
265+
shell: bash
266+
run: |
267+
BUILD_ARGS=(
268+
--progress=plain
269+
--platform linux/amd64
270+
-f "${{ inputs.dockerfile-path }}"
271+
--build-arg "ISAACSIM_BASE_IMAGE_ARG=${{ inputs.isaacsim-base-image }}"
272+
--build-arg "ISAACSIM_VERSION_ARG=${{ inputs.isaacsim-version }}"
273+
--build-arg "ISAACSIM_ROOT_PATH_ARG=/isaac-sim"
274+
--build-arg "ISAACLAB_PATH_ARG=/workspace/isaaclab"
275+
--build-arg "DOCKER_USER_HOME_ARG=/root"
276+
-t "${{ inputs.image-tag }}"
277+
)
278+
if [ -n "${ECR_URL:-}" ]; then
279+
BUILD_ARGS+=(
280+
--cache-from "type=registry,ref=${CACHE_IMAGE}"
281+
--cache-to "type=registry,ref=${CACHE_IMAGE},mode=max"
282+
-t "${ECR_IMAGE}"
283+
)
284+
fi
285+
286+
# Opt-in GitHub Actions layer cache. Registry-free fallback for fleets
287+
# where ECR cannot be resolved (the block above is skipped), keeping the
288+
# build in line with how publish-images.yaml / docker-build cache this
289+
# same Dockerfile. The scope isolates it from other workflows' gha caches.
290+
if [ -n "${{ inputs.gha-cache-scope }}" ]; then
291+
# mode=min (not max): Dockerfile.base is single-stage, so there are no
292+
# intermediate-stage layers for mode=max to additionally capture. min keeps
293+
# the same cache-hit rate while writing far less into the repo-shared 10 GB
294+
# GitHub Actions cache budget. The large Isaac Sim base image is referenced
295+
# by digest and re-pulled from the registry, not stored in this cache.
296+
BUILD_ARGS+=(
297+
--cache-from "type=gha,scope=${{ inputs.gha-cache-scope }}"
298+
--cache-to "type=gha,scope=${{ inputs.gha-cache-scope }},mode=min"
299+
)
300+
fi
301+
302+
BUILDER_NAME="ci-builder-${{ github.run_id }}-${{ github.job }}"
303+
docker buildx create --use --driver docker-container --name "${BUILDER_NAME}" \
304+
|| docker buildx use "${BUILDER_NAME}"
305+
trap 'docker buildx rm "${BUILDER_NAME}" || true' EXIT
306+
307+
echo "🔵 Building ${{ inputs.image-tag }}..."
308+
docker buildx build --load "${BUILD_ARGS[@]}" .
309+
310+
##### 7: Push to ECR #####
311+
312+
# Pushes the per-commit ECR image after a successful full build.
313+
# Skipped if the image was pulled in (4).
314+
315+
- name: Push to ECR
316+
if: >
317+
steps.resolve-ecr.outputs.available == 'true' &&
318+
steps.pull-exact.outputs.hit != 'true' &&
319+
steps.deps-cache.outputs.deps-cache-hit != 'true'
320+
shell: bash
321+
run: |
322+
echo "🔵 Pushing ${ECR_IMAGE} to ECR..."
323+
docker push "${ECR_IMAGE}"
324+
echo "🟢 Pushed ${ECR_IMAGE}"
325+
326+
##### 8: Push deps tag #####
327+
328+
# Tags the freshly built image as deps-<hash> so future runs with identical
329+
# install inputs hit the fast path (step 5) instead of doing a full build.
330+
331+
- name: Push deps tag
332+
if: env.PUSH_DEPS_IMAGE == 'true'
333+
shell: bash
334+
run: |
335+
echo "🔵 Pushing deps image for future cache hits: ${DEPS_ECR_IMAGE}"
336+
docker tag "${{ inputs.image-tag }}" "${DEPS_ECR_IMAGE}"
337+
docker push "${DEPS_ECR_IMAGE}"
338+
339+
##### 9: Cleanup docker config #####
340+
341+
- name: Cleanup docker config
342+
if: always()
343+
shell: bash
344+
run: |
345+
if [ -n "${DOCKER_CONFIG}" ] && [ -d "${DOCKER_CONFIG}" ]; then
346+
rm -rf "${DOCKER_CONFIG}"
347+
fi

0 commit comments

Comments
 (0)