Commit df44b19
authored
[CI] Skip the image invariants on a deps-cache hit and link _isaac_sim only in Kit images (#7540)
## Summary
Every first CI run of a new commit on a dependency-stable branch has
failed since #7405 landed
(develop included): `Build Base Docker Image` dies in 20 s and skips the
17 test jobs behind it,
and `Multi-GPU training smoke (kit-less)` exits before pytest. Both are
CI plumbing, fixed here.
# Description
## 1. Build Base Docker Image
#7405 added a "Verify image invariants" step that runs `docker image
inspect` on the commit tag
after `ecr-build-push-pull`. The action leaves an image on the runner
only after a full build or an
exact-tag pull. On a deps-cache hit it just aliases the commit tag onto
the `deps-<hash>` manifest in
ECR and pulls nothing, so the inspect fails:
```
Error response from daemon: No such image: isaac-lab-ci:develop-8ca264bd8e5528d7d9b7b6f58a9d8cd051c5933a
```
develop runs 33733992373, 33740164999 and 33781460422 fail this way; PRs
pass only after a deps
change (full build) or on a re-run of an already built head (exact-tag
pull).
The step also ran too late to be a gate. The action pushes the commit
tag and the deps tag inside
itself, so an image that fails the check is already published. That is
not hypothetical: on
`newton-world-prefixes` (run 33735349483) `deps-6aebe721c56a91f7` was
pushed at 09:05:07 and the
invariant failed at 09:05:11, leaving an image whose Isaac Sim
extensions cannot load in the shared
cache. Any later run whose install inputs hash the same inherits it.
Fix: move the assertion into the action, before every push, behind a
`verify-command` input that the
base image job passes. A failure now publishes nothing, so the next run
rebuilds and fails again
rather than serving the bad image from the cache. Skipping it on a cache
hit is then correct by
construction — every published image passed when it was built, and the
deps hash covers exactly the
install inputs the invariant asserts against.
## 2. Multi-GPU training smoke (kit-less)
`run_tests.sh` bind-mounts nine writable runtime directories under
`/isaac-sim/…` into every
container, so `/isaac-sim` exists as a directory in the kit-less image
too, and the container script
then runs `ln -s /isaac-sim _isaac_sim` unconditionally. #7466 taught
`isaaclab.sh` to treat a
`_isaac_sim` directory without a source-build marker as a downloaded
Isaac Sim and to refuse it next
to an active venv; the kit-less image sets
`VIRTUAL_ENV=/opt/isaaclab-venv`, so every run stops with:
```
[ERROR] Downloaded Isaac Sim packages cannot be combined with a Python virtual environment.
```
#7405 reworked the same check (venv-on-bundled-Python exemption) and the
run-tests script without
exempting the kit-less lane. Every PR's smoke run since 11:43Z on
2026-09-03 fails identically.
Fix: plant the link only when `/isaac-sim/python.sh` exists. Kit images
are unchanged; the kit-less
image resolves its interpreter through `VIRTUAL_ENV`, as it did before
#7466.
## 3. Audit of every `_isaac_sim` site
Every place that creates the link, and how it interacts with the guard:
| Site | Creates | Verdict |
| --- | --- | --- |
| `run-tests/run_tests.sh` | `/isaac-sim` in every image | **Fixed
here**: guarded on `python.sh` |
| `multi-gpu/multi_gpu_host_launcher.sh` | `/isaac-sim`, unguarded |
Correct today: its only caller passes `CI_IMAGE_TAG`, the Kit image. The
kit-less lane goes through `run-tests` |
| `Dockerfile.base`, `Dockerfile.curobo` | `ISAACSIM_ROOT_PATH`,
alongside `VIRTUAL_ENV` | Correct:
`UV_PYTHON=${ISAACSIM_ROOT_PATH}/kit/python/bin/python3`, so the venv is
on the bundled Python and #7405's exemption applies |
| `Dockerfile.kitless` | never; asserts `test ! -e
"${ISAACLAB_PATH}/_isaac_sim"` | Correct |
| `cli/commands/misc.py` (`--isaacsim_source`) | source-build link |
Correct: writes `.isaaclab_source_build` |
| `docs/.../installation/index.rst` ×3 | user-run `ln -s` | Out of
scope, see below |
## 4. Verification
- `docker/test/test_container_profiles.py` gains two regression tests —
one pinning the kit-less
guard, one pinning that verification precedes both push steps. Each
fails without its fix. 15 pass
in that file, 20 with `test_carb_env_shim.py`.
- `yaml.safe_load` on both YAML files, `bash -n` on the script,
pre-commit on the changed files.
- This PR touches `.github/actions/{ecr-build-push-pull,run-tests}/` and
`build.yaml`, which the
detect-changes patterns of both workflows match, so its own CI exercises
both paths: the base image
job takes the deps-cache-hit path (no dependency change here) and the
kit-less smoke job runs.
## 5. Out of scope
Three checkers implement the same "downloaded Isaac Sim vs. virtual
environment" rule and #7405
updated only two, leaving user-facing divergences that this PR does not
touch:
- `isaaclab.bat` has neither the venv-on-bundled-Python exemption nor
the `--isaacsim_source`
escape, so on Windows the remedy the Linux error message recommends is
still rejected.
- `cli/commands/envs.py::_reject_downloaded_isaac_sim` has no exemption
either, so
`isaaclab -c/-u` rejects a venv created on the bundled Python that
`isaaclab.sh` accepts.
- The docs link a *source build* with a bare `ln -s`, which leaves no
`.isaaclab_source_build`
marker, so the same tree is treated as "downloaded" unless linked via
`--isaacsim_source`.
`isaaclab.sh` also lacks the `python.sh` precondition its Python twin
(`cli/utils.py`) has, but
that is deliberate:
`test_launcher_rejects_downloaded_isaac_sim_with_active_environment`
pins the
fail-closed behaviour on a bare directory. That is why the fix here is
in the CI script.
## Type of change
- Bug fix (non-breaking change which fixes an issue)
## Release backport
- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`
## Checklist
- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation (the action
README documents the new output)
- [x] My changes generate no new warnings
- [ ] Tests: CI-only change; the workflows' own runs on this PR are the
test
- [ ] Changelog: no source package changed
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there1 parent 8365c57 commit df44b19
6 files changed
Lines changed: 85 additions & 30 deletions
File tree
- .github
- actions
- ecr-build-push-pull
- run-tests
- workflows
- docker/test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
19 | 32 | | |
20 | 33 | | |
21 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
40 | 50 | | |
41 | 51 | | |
42 | 52 | | |
| |||
242 | 252 | | |
243 | 253 | | |
244 | 254 | | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
245 | 272 | | |
246 | 273 | | |
247 | 274 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
320 | 320 | | |
321 | 321 | | |
322 | 322 | | |
323 | | - | |
324 | | - | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
325 | 328 | | |
326 | 329 | | |
327 | 330 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
210 | 213 | | |
211 | 214 | | |
212 | 215 | | |
| |||
215 | 218 | | |
216 | 219 | | |
217 | 220 | | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
| 221 | + | |
232 | 222 | | |
233 | 223 | | |
234 | 224 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
288 | 289 | | |
289 | 290 | | |
290 | 291 | | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
291 | 327 | | |
292 | 328 | | |
293 | 329 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | 41 | | |
56 | 42 | | |
57 | 43 | | |
| |||
0 commit comments