[CI] Skip the image invariants on a deps-cache hit and link _isaac_sim only in Kit images

[hujc7]

Summary

Every first CI run of a new commit on a dependency-stable branch has failed since #7405 landed
(develop included): Build Base Docker Image dies in 20 s and skips the 17 test jobs behind it,
and Multi-GPU training smoke (kit-less) exits before pytest. Both are CI plumbing, fixed here.

Description

1. Build Base Docker Image

#7405 added a "Verify image invariants" step that runs docker image inspect on the commit tag
after ecr-build-push-pull. The action leaves an image on the runner only after a full build or an
exact-tag pull. On a deps-cache hit it just aliases the commit tag onto the deps-<hash> manifest in
ECR and pulls nothing, so the inspect fails:

Error response from daemon: No such image: isaac-lab-ci:develop-8ca264bd8e5528d7d9b7b6f58a9d8cd051c5933a

develop runs 33733992373, 33740164999 and 33781460422 fail this way; PRs pass only after a deps
change (full build) or on a re-run of an already built head (exact-tag pull).

The step also ran too late to be a gate. The action pushes the commit tag and the deps tag inside
itself, so an image that fails the check is already published. That is not hypothetical: on
newton-world-prefixes (run 33735349483) deps-6aebe721c56a91f7 was pushed at 09:05:07 and the
invariant failed at 09:05:11, leaving an image whose Isaac Sim extensions cannot load in the shared
cache. Any later run whose install inputs hash the same inherits it.

Fix: move the assertion into the action, before every push, behind a verify-command input that the
base image job passes. A failure now publishes nothing, so the next run rebuilds and fails again
rather than serving the bad image from the cache. Skipping it on a cache hit is then correct by
construction — every published image passed when it was built, and the deps hash covers exactly the
install inputs the invariant asserts against.

2. Multi-GPU training smoke (kit-less)

run_tests.sh bind-mounts nine writable runtime directories under /isaac-sim/… into every
container, so /isaac-sim exists as a directory in the kit-less image too, and the container script
then runs ln -s /isaac-sim _isaac_sim unconditionally. #7466 taught isaaclab.sh to treat a
_isaac_sim directory without a source-build marker as a downloaded Isaac Sim and to refuse it next
to an active venv; the kit-less image sets VIRTUAL_ENV=/opt/isaaclab-venv, so every run stops with:

[ERROR] Downloaded Isaac Sim packages cannot be combined with a Python virtual environment.

#7405 reworked the same check (venv-on-bundled-Python exemption) and the run-tests script without
exempting the kit-less lane. Every PR's smoke run since 11:43Z on 2026-09-03 fails identically.

Fix: plant the link only when /isaac-sim/python.sh exists. Kit images are unchanged; the kit-less
image resolves its interpreter through VIRTUAL_ENV, as it did before #7466.

3. Audit of every _isaac_sim site

Every place that creates the link, and how it interacts with the guard:

Site	Creates	Verdict
run-tests/run_tests.sh	/isaac-sim in every image	Fixed here: guarded on python.sh
multi-gpu/multi_gpu_host_launcher.sh	/isaac-sim, unguarded	Correct today: its only caller passes CI_IMAGE_TAG, the Kit image. The kit-less lane goes through run-tests
Dockerfile.base, Dockerfile.curobo	ISAACSIM_ROOT_PATH, alongside VIRTUAL_ENV	Correct: UV_PYTHON=${ISAACSIM_ROOT_PATH}/kit/python/bin/python3, so the venv is on the bundled Python and #7405's exemption applies
Dockerfile.kitless	never; asserts test ! -e "${ISAACLAB_PATH}/_isaac_sim"	Correct
cli/commands/misc.py (--isaacsim_source)	source-build link	Correct: writes .isaaclab_source_build
docs/.../installation/index.rst ×3	user-run ln -s	Out of scope, see below

4. Verification

• docker/test/test_container_profiles.py gains two regression tests — one pinning the kit-less
  guard, one pinning that verification precedes both push steps. Each fails without its fix. 15 pass
  in that file, 20 with test_carb_env_shim.py.
• yaml.safe_load on both YAML files, bash -n on the script, pre-commit on the changed files.
• This PR touches .github/actions/{ecr-build-push-pull,run-tests}/ and build.yaml, which the
  detect-changes patterns of both workflows match, so its own CI exercises both paths: the base image
  job takes the deps-cache-hit path (no dependency change here) and the kit-less smoke job runs.

5. Out of scope

Three checkers implement the same "downloaded Isaac Sim vs. virtual environment" rule and #7405
updated only two, leaving user-facing divergences that this PR does not touch:

• isaaclab.bat has neither the venv-on-bundled-Python exemption nor the --isaacsim_source
  escape, so on Windows the remedy the Linux error message recommends is still rejected.
• cli/commands/envs.py::_reject_downloaded_isaac_sim has no exemption either, so
  isaaclab -c/-u rejects a venv created on the bundled Python that isaaclab.sh accepts.
• The docs link a source build with a bare ln -s, which leaves no .isaaclab_source_build
  marker, so the same tree is treated as "downloaded" unless linked via --isaacsim_source.

isaaclab.sh also lacks the python.sh precondition its Python twin (cli/utils.py) has, but
that is deliberate: test_launcher_rejects_downloaded_isaac_sim_with_active_environment pins the
fail-closed behaviour on a bare directory. That is why the fix here is in the CI script.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Release backport

• Backport this pull request to the active release branch after it merges into develop

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation (the action README documents the new output)
• My changes generate no new warnings
• Tests: CI-only change; the workflows' own runs on this PR are the test
• Changelog: no source package changed
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[hujc7]

run-ci

[hujc7]

run-ci

[greptile-apps]

Greptile Summary

The PR moves base-image invariant checks into the ECR action before its runnable-image push steps and avoids creating _isaac_sim in kit-less test containers.

• Adds an optional fresh-build verification test path to the ECR action.
• Makes the run-tests symlink conditional on an executable Isaac Sim python.sh.
• Adds regression coverage for publication ordering and kit-less behavior.

Confidence Score: 4/5

The PR is not yet safe to merge because legacy or previously verified dependency-cache manifests can still bypass the new image invariant.

Dependency-cache hits alias the existing manifest to the commit tag and skip verification, while the cache key does not incorporate the verification gate or invariant test; therefore images created before the gate, or under an older invariant, remain eligible for downstream use.

Files Needing Attention: .github/actions/ecr-build-push-pull/action.yml and .github/actions/_lib/compute-deps-hash/action.yml

Important Files Changed

Filename	Overview
.github/actions/ecr-build-push-pull/action.yml	Adds pre-push verification for fresh builds, but existing dependency-cache manifests remain exempt and can still be promoted without verification.
.github/actions/run-tests/run_tests.sh	Restricts _isaac_sim creation to images that actually contain the Kit Python launcher.
.github/workflows/build.yaml	Delegates the base-image invariant test to the reusable ECR action.
docker/test/test_container_profiles.py	Adds structural regression tests covering verification ordering and conditional symlink creation.
docker/test/test_image_invariants.py	Removes the obsolete digest-binding assertion now that verification runs directly after the local build.

_{Reviews (3): Last reviewed commit: "Drop the digest binding from image verif..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The _isaac_sim guard correctly preserves Kit-image behavior while unblocking the kit-less lane. However, the new built output conflates build freshness with local image availability, causing exact-tag reruns to skip image-invariant validation even though the image was pulled locally, and its documentation gives callers misleading guidance.

• Design and architecture: Avoiding a multi-GB pull on dependency-cache hits is reasonable, but build freshness is too coarse a gate for checks that require a locally available image. The action should distinguish the non-local dependency-cache path from fresh builds and exact-tag pulls.
• API: The additive built output is safe for existing callers, but its documented recommendation is inconsistent with its semantics: exact-tag hits and dependency hits with pull-on-deps-hit also provide a local image while returning built == 'false'. Restrict the documentation to build-only checks or expose a separate local-availability output.
• Implementation: The /isaac-sim/python.sh precondition and rm -f correctly avoid creating _isaac_sim in kit-less images. In build.yaml, however, gating invariants solely on built == 'true' lets an exact-tag rerun pull an image that previously failed invariants and then skip the validation. The condition should include exact-tag local availability while exempting only dependency-cache hits that do not pull.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.

[hujc7]

run-ci

[hujc7]

@greptileai review

[hujc7]

run-ci

[hujc7]

@greptileai review

[hujc7]

run-ci