Verify a freshly built image before it is published

The invariants step ran after ecr-build-push-pull returned, but the
action pushes the commit tag and the deps tag inside itself, so an image
that fails the check is already in the shared cache. That happened four
seconds apart on newton-world-prefixes (run 33735349483): the deps tag
deps-6aebe721c56a91f7 was pushed at 09:05:07 and the invariant failed at
09:05:11, leaving an image whose Isaac Sim extensions cannot load.

Any later run whose install inputs hash the same then gets a cache hit
and inherits it, so gating the check on whether this run built the image
would let the second push of the change that broke it go green.

Move the assertion into the action, before every push, behind a
verify-command input the base image job passes. A failure now publishes
nothing, so the next run rebuilds and fails again. The built output added
for the old gate is gone with it.

Author: hujc7

.github/actions/ecr-build-push-pull/README.md

@@ -16,11 +16,17 @@ ECR is also used as the BuildKit layer cache.
     ecr-url: (optional, complete url for ECR storage)
 ```
 
-## Outputs
+## Verifying a freshly built image
 
-- `built`: `'true'` when this run built the image, so it exists locally under `image-tag`.
-  An exact-tag hit pulls it as well; a deps-cache hit only creates the ECR tag unless
-  `pull-on-deps-hit` is set. Gate steps that need the image locally on this output.
+Pass `verify-command` to assert against the image before it is published:
+
+```yaml
+    verify-command: uv run --no-project --with pytest python -m pytest -q docker/test/test_image_invariants.py
+```
+
+It runs only on a full build, with `IMAGE_TAG` and `IMAGE_DIGEST` exported. A failure fails the
+action with nothing pushed, so the next run rebuilds rather than serving the bad image from the
+deps cache. Exact-tag and deps-cache hits skip it: that image passed the command when it was built.
 
 ## ECR URL resolution order
 

.github/actions/ecr-build-push-pull/action.yml

@@ -37,6 +37,16 @@ inputs:
     description: Tag used for the ECR layer cache image (e.g. "cache-base", "cache-curobo").
     required: false
     default: 'cache'
+  verify-command:
+    description: >
+      Command run against a freshly built image, before it is tagged or pushed, with IMAGE_TAG
+      and IMAGE_DIGEST exported. A failing command fails the action with nothing published, so
+      the next run rebuilds instead of inheriting the bad image from the cache.
+
+      Not run on an exact-tag or deps-cache hit: those serve an image that already passed this
+      command when it was built.
+    required: false
+    default: ''
   pull-on-deps-hit:
     description: >
       Pull the image locally after a deps-cache hit. Needed by jobs that run
@@ -50,15 +60,6 @@ inputs:
       that only need the tag pushed.
     required: false
     default: 'false'
-outputs:
-  built:
-    description: >
-      'true' when this run built the image, so it exists locally under image-tag.
-      An exact-tag hit pulls it as well; a deps-cache hit only creates the ECR tag
-      unless pull-on-deps-hit is set, so steps that need the image locally should
-      check this output.
-    value: ${{ steps.pull-exact.outputs.hit != 'true' && steps.deps-cache.outputs.deps-cache-hit != 'true' }}
-
 runs:
   using: composite
   steps:
@@ -251,6 +252,24 @@ runs:
         cache-to: ${{ steps.resolve-ecr.outputs.available == 'true' && format('type=registry,ref={0},mode=max', env.CACHE_IMAGE) || '' }}
         deps-hash: ${{ steps.deps-hash.outputs.hash }}
 
+    # Assert against the image while it is only local: the push steps below publish under both
+    # the commit tag and the deps tag, and a deps-cache hit later serves that image without
+    # rebuilding it, so anything published unverified stays unverified.
+    - name: Verify freshly built image
+      if: >
+        inputs.verify-command != '' &&
+        steps.pull-exact.outputs.hit != 'true' &&
+        steps.deps-cache.outputs.deps-cache-hit != 'true'
+      shell: bash
+      env:
+        IMAGE_TAG: ${{ inputs.image-tag }}
+        VERIFY_COMMAND: ${{ inputs.verify-command }}
+      run: |
+        set -euo pipefail
+        IMAGE_DIGEST="$(docker image inspect --format '{{.Id}}' "${IMAGE_TAG}")"
+        export IMAGE_DIGEST
+        bash -c "${VERIFY_COMMAND}"
+
     - name: Tag built image with ECR-prefixed name
       if: >
         steps.resolve-ecr.outputs.available == 'true' &&

.github/workflows/build.yaml

@@ -207,32 +207,18 @@ jobs:
       with:
         enable-cache: true
 
+    # #6329 aborts the pip install when it strands a prebundled package's __init__.py
+    # (nvbugs 6343978: 14 Isaac Sim extensions fail to load). The images install with
+    # ``uv sync``, which never runs that guard, so assert the same invariant on the image.
     - name: Build and push to ECR
-      id: image
       uses: ./.github/actions/ecr-build-push-pull
       with:
         image-tag: ${{ needs.config.outputs.ci_image_tag }}
         isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }}
         isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }}
         dockerfile-path: docker/Dockerfile.base
         cache-tag: cache-base
-
-    # #6329 aborts the pip install when it strands a prebundled package's __init__.py
-    # (nvbugs 6343978: 14 Isaac Sim extensions fail to load). The images install with
-    # ``uv sync``, which never runs that guard, so assert the same invariant on the image.
-    # Only a freshly built image is on this runner; a deps-cache hit reuses one that
-    # passed this check when it was built.
-    - name: Verify image invariants
-      if: steps.image.outputs.built == 'true'
-      shell: bash
-      env:
-        IMAGE_TAG: ${{ needs.config.outputs.ci_image_tag }}
-      run: |
-        set -euo pipefail
-        IMAGE_DIGEST="$(docker image inspect --format '{{.Id}}' "${IMAGE_TAG}")"
-        export IMAGE_DIGEST
-        uv run --no-project --with pytest \
-          python -m pytest -q docker/test/test_image_invariants.py
+        verify-command: uv run --no-project --with pytest python -m pytest -q docker/test/test_image_invariants.py
 
   build-curobo:
     name: Build cuRobo Docker Image

docker/test/test_container_profiles.py

@@ -289,6 +289,23 @@ def test_kitless_compose_service_has_no_isaac_sim_mounts():
     assert all("/kit/" not in mount["target"].lower() for mount in mounts)
 
 
+def test_image_is_verified_before_it_is_published():
+    """A published image must be a verified one.
+
+    The push steps publish under both the commit tag and the deps tag, and a later deps-cache hit
+    serves that image without rebuilding it, so anything published unverified stays unverified.
+    """
+    action = yaml.safe_load(
+        (REPO_ROOT / ".github" / "actions" / "ecr-build-push-pull" / "action.yml").read_text(encoding="utf-8")
+    )
+    names = [step["name"] for step in action["runs"]["steps"] if "name" in step]
+
+    assert names.index("Verify freshly built image") < names.index("Push to ECR") < names.index("Push deps tag")
+
+    build = (REPO_ROOT / ".github" / "workflows" / "build.yaml").read_text(encoding="utf-8")
+    assert "verify-command: uv run" in build, "the base image job must hand its invariants to the action"
+
+
 def test_run_tests_links_isaac_sim_only_where_kit_is_installed():
     """The kit-less image has no Kit under ``/isaac-sim``, which the runtime mounts create anyway.