Skip to content

ci: add gate, preflight and verification to testing image release - #718

Open
nvasiu wants to merge 2 commits into
mainfrom
gate-ecr-release
Open

nvasiu wants to merge 2 commits into
mainfrom
gate-ecr-release

Conversation

@nvasiu

@nvasiu nvasiu commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available:

Related to #716

Description of changes:

The ecr-release.yml action would previously run for every monorepo release, regardless if the release included a new version for the testing package or not. So it was possible for this action to publish the latest changes of the testing package before they were released. This PR updates the action to be safer.

.github/workflows/ecr-release.yml

  • Add a preflight job to parse the release tag, verify the tag matches the source, check if the version already exists in public ECR, and emit a plan.
  • Gates the build on the above preflight checks.
  • Add a post-publish verification that polls public ECR (polls 10 times with 15s wait = 150s total) and confirms that the new version was published.

.github/scripts/parse_testing_version.py

  • Separate script for parsing the testing version from a release tag.

.github/scripts/tests/test_parse_testing_version.py

  • Unit testing for the parsing script.

.github/workflows/test-parser.yml

  • Wire the new script and test into the script-test workflow.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@nvasiu
nvasiu deployed to ai-pr-review-runtime September 10, 2026 23:21 — with GitHub Actions Active
@nvasiu
nvasiu deployed to ai-pr-review-runtime September 10, 2026 23:29 — with GitHub Actions Active
Comment thread .github/workflows/ecr-release.yml Outdated
@github-actions

This comment has been minimized.

@nvasiu nvasiu changed the title ci: gate testing image publish on testing release ci: add gate, preflight and verification to testing image release Sep 11, 2026
@nvasiu
nvasiu deployed to ai-pr-review-runtime September 11, 2026 18:35 — with GitHub Actions Active
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/scripts/parse_testing_version.py Outdated
@github-actions

This comment has been minimized.

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preflight and verification confirm that an image tag exists. They do not confirm the image contains what the release names. The Dockerfile installs aws-durable-execution-sdk-python>=1.0.0, resolved at build time and unbounded, so two builds of one testing version can produce different images, and skip-if-exists assumes a version identifies one artifact.

(The other three points are on the relevant lines.)

Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/scripts/parse_testing_version.py Outdated
Comment thread .github/workflows/ecr-release.yml Outdated
@nvasiu
nvasiu deployed to ai-pr-review-runtime September 11, 2026 22:48 — with GitHub Actions Active
Comment thread .github/workflows/ecr-release.yml
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/workflows/ecr-release.yml Outdated
@github-actions

This comment has been minimized.

@nvasiu
nvasiu deployed to ai-pr-review-runtime September 14, 2026 20:51 — with GitHub Actions Active
Comment thread .github/workflows/ecr-release.yml Outdated
@github-actions

This comment has been minimized.

@nvasiu

nvasiu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@yaythomas
Re: #718 (review)

Yes, the preflight confirms that a tag exists, but it's not guaranteed that the image matches the release. And skip-if-exists assumes a version maps to one image, which is not true while we have an unbounded SDK dependency.

But this workflow's current policy is that if a version is already published to ECR, we won't ever overwrite it. So it shouldn't ever be a concern that the user will get different SDK versions from the same image version.

But for the sake of reproducing the image on the ECR in the future / visiblity into what SDK version the image is using, we could:

  • Pin the SDK dependency in the Dockerfile.
  • Or label images with which SDK version they are using (doesn't prevent different images per version, but lets us detect them).

I think either of these option would need some more discussion, and are out of scope for this PR. But if we want to implement either of those, they would be compatible with this PR.

@nvasiu
nvasiu force-pushed the gate-ecr-release branch 2 times, most recently from 5d8d07a to a4e387c Compare September 14, 2026 21:38
@nvasiu
nvasiu deployed to ai-pr-review-runtime September 14, 2026 21:45 — with GitHub Actions Active
@nvasiu
nvasiu deployed to ai-pr-review-runtime September 14, 2026 21:59 — with GitHub Actions Active
Comment thread .github/workflows/ecr-release.yml
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/workflows/ecr-release.yml Outdated
@github-actions

This comment has been minimized.

@nvasiu
nvasiu deployed to ai-pr-review-runtime September 18, 2026 20:31 — with GitHub Actions Active
Comment thread .github/workflows/ecr-release.yml Outdated
Comment thread .github/scripts/latest_testing_version.py Outdated
Comment thread .github/workflows/ecr-release.yml
@github-actions

This comment has been minimized.

@nvasiu
nvasiu deployed to ai-pr-review-runtime September 21, 2026 20:35 — with GitHub Actions Active
Comment on lines +254 to +257
docker manifest create "$base:v$VERSION" "$image_x86_64" "$image_arm64"
docker manifest annotate "$base:v$VERSION" "$image_arm64" --arch arm64 --os linux
docker manifest annotate "$base:v$VERSION" "$image_x86_64" --arch amd64 --os linux
docker manifest push "$base:v$VERSION"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_pvjgcb3ats4msus2hcvg553pvr

[P1] These commands also run when BUILT_THIS_RUN is false, recreating an already-published v$VERSION from mutable architecture tags. A concurrent retry can update those tags with different image digests and fail; the next recovery then silently rewrites the existing release manifest and verification passes against the rewritten digest. Only create v$VERSION after a fresh build. For recovery and latest, copy the exact existing top-level manifest instead, and test that this path never pushes v$VERSION.

Comment on lines +311 to +313
if printf '%s\n' $existing_tags | grep -qx "v$VERSION"; then
newest="$(python .github/scripts/latest_testing_version.py $existing_tags)"
break

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_otspk2nucjnfic2v5zdpbwiemx

[P1] Seeing this run's tag does not prove the complete tag list is fresh. After v3 has updated latest, a serialized v2 backport can receive a successful stale response containing v2 but omitting v3, select v2, and downgrade latest; verification only reports failure and never repairs it. Maintain a strongly consistent serialized high-water mark or use a queue/lock that preserves every updater, refuse downgrades, and reconcile before releasing the lock.

Comment on lines +20 to +23
if not tag.startswith("v"):
continue
try:
version = Version(tag[1:])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_bn7u2dvrf6j45ju4qaataf7xkx

[P2] Version() accepts abbreviated PEP 440 values such as 1 and 1.2, although this workflow only publishes versions with an X.Y.Z prefix. A stray registry tag such as v999 is therefore selected as newest, causing attempts to use nonexistent v999-x86_64 and v999-arm64 images and blocking releases. Full-match the supported tag grammar before parsing, and add regression cases for v1 and v1.2.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Found two high-impact release-state issues and one tag-validation bug. Tests do not cover the ECR recovery and concurrency paths involved.

Reviewed commit 0f4faa37a8847743aa2df07bd47bad2bf3d8885a. Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants