Skip to content

Commit 5b8790f

Browse files
authored
ci: publish the version-named layer parameter from the pipeline (#846)
* ci: publish the version-named layer parameter from the pipeline The second half of #843. That change removed the version-named SSM pointer from the layer stacks, because a parameter whose Name contains the package version is replaced when the version changes and — retained — collides with its own orphan when a version is re-used, which wedged every merge to main for two months. The pointer itself is still wanted, so it is now written after deployment with `aws ssm put-parameter --overwrite`, which is idempotent and cannot collide. Two scripts: publish-layer-parameter.sh reads the layer ARN from the stack's own output and writes /lambda-web-adapter/layer/<arch>/<version>. It refuses a version containing a slash before making any AWS call, since such a name would fall outside the granted path and could address a `latest` pointer. configure-layer-parameter-iam.sh grants the PipelineExecutionRole ssm:PutParameter on that path. Run once per role, with administrator credentials in the role's account. This is a prerequisite: SAM-managed pipeline roles do not have the permission, and until the bootstrap is run the publish steps fail with AccessDenied. 69 roles are in scope — 32 gamma, 32 prod, 2 China gamma, 2 China prod, 1 beta, all distinct. Three things about the grant are deliberate: * Region-agnostic. `put-role-policy` replaces the whole named policy, so a per-region document meant a second invocation for another region silently revoked the first. The Resource wildcards the region instead, so one run per role covers every region and re-running is idempotent. * No region argument. With the region wildcarded there is nothing for the partition parsed from the role ARN to disagree with; an `aws` role paired with `cn-north-1` previously produced a policy matching nothing while reporting success. * An explicit Deny on `/latest`. The Allow has to wildcard the version segment, which also covers the `latest` pointer CloudFormation owns — and which the e2e fixture resolves at deploy time. Without the Deny, a mistyped version could overwrite stack-managed state out of band, where nobody would look for drift. In the workflows, both architectures deploy before either pointer is written. Ordered the other way — publish x86, deploy arm64 — a missed bootstrap left the region with a new x86 layer and no arm64 one, failed deploy-prod, skipped publish-to-public-ecr, and stopped the release after prod had been partially mutated. Ordered this way the same failure leaves both layers published and the pointers stale. Each stack name is now defined once per job and consumed by both the deploy and the publish step, rather than written twice in ten places with nothing asserting the two agree. Exercised: the publish script's argument, architecture and slash-in-version guards; the IAM script's ARN validation against a non-ARN, a short account id, a user ARN and a nested role path; the rendered policy document; and both workflows parse. * ci: take the SSM grant from the infrastructure package, not a bootstrap script The PipelineExecutionRole is managed in LambdaWebAdapterInfrastructureCDK, so the grant belongs there rather than in an imperative script run by hand against 69 roles. lib/infra-template.yaml now carries it in PipelineExecutionRolePermissions, allowing ssm:PutParameter on parameter/lambda-web-adapter/layer/{x86_64,arm64}/* and denying .../latest, which CloudFormation owns. Deletes .github/scripts/configure-layer-parameter-iam.sh. Keeping it would have been worse than redundant: it wrote a separate inline policy under its own name, so the two would have coexisted on every role with no indication which was authoritative, and its region-wildcarded Resource was broader than the per-region scope the infra template can express — one pipeline-resources stack exists per region and its role only deploys in that region. The workflow comments and the publish script's header now say where the permission comes from, so the AccessDenied case points at a region whose pipeline-resources stack has not been updated rather than at a script nobody ran.
1 parent 428260f commit 5b8790f

5 files changed

Lines changed: 195 additions & 22 deletions

File tree

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Publishes the version-named SSM pointer after CloudFormation has deployed a layer.
4+
# The parameter deliberately lives outside the stack: retaining a
5+
# parameter whose Name contains the package version made CloudFormation collide with
6+
# the retained resource whenever a version was re-used.
7+
#
8+
# The PipelineExecutionRole's ssm:PutParameter grant for these names lives in
9+
# LambdaWebAdapterInfrastructureCDK (lib/infra-template.yaml,
10+
# PipelineExecutionRolePermissions), which also denies the `latest` pointer that
11+
# CloudFormation owns.
12+
#
13+
# Usage:
14+
# publish-layer-parameter.sh <stack-name> <x86_64|arm64> <version> <region>
15+
set -euo pipefail
16+
export AWS_PAGER=""
17+
18+
STACK_NAME="${1:?stack name required}"
19+
ARCHITECTURE="${2:?architecture required}"
20+
VERSION="${3:?version required}"
21+
REGION="${4:?region required}"
22+
23+
# A version containing a slash would write outside the path the grant covers — and, worse,
24+
# could name a `latest` pointer. Fail before the API call rather than on an opaque
25+
# AccessDenied.
26+
if [[ "$VERSION" != "${VERSION//\//}" || -z "$VERSION" ]]; then
27+
echo "Refusing to publish a parameter for version '$VERSION'" >&2
28+
exit 2
29+
fi
30+
31+
case "$ARCHITECTURE" in
32+
x86_64)
33+
OUTPUT_KEY="LambdaAdapterLayerX86Arn"
34+
DESCRIPTION_ARCH="X86_64"
35+
;;
36+
arm64)
37+
OUTPUT_KEY="LambdaAdapterLayerArm64Arn"
38+
DESCRIPTION_ARCH="Arm64"
39+
;;
40+
*)
41+
echo "Unsupported architecture: $ARCHITECTURE" >&2
42+
exit 2
43+
;;
44+
esac
45+
46+
LAYER_ARN="$(
47+
aws cloudformation describe-stacks \
48+
--stack-name "$STACK_NAME" \
49+
--region "$REGION" \
50+
--query "Stacks[0].Outputs[?OutputKey=='${OUTPUT_KEY}'].OutputValue | [0]" \
51+
--output text
52+
)"
53+
54+
if [[ -z "$LAYER_ARN" || "$LAYER_ARN" == "None" ]]; then
55+
echo "Stack $STACK_NAME has no $OUTPUT_KEY output in $REGION" >&2
56+
exit 1
57+
fi
58+
59+
PARAMETER_NAME="/lambda-web-adapter/layer/${ARCHITECTURE}/${VERSION}"
60+
PARAMETER_VERSION="$(
61+
aws ssm put-parameter \
62+
--name "$PARAMETER_NAME" \
63+
--description "Layer ARN for the Lambda Web Adapter ${DESCRIPTION_ARCH} Layer: ${VERSION}" \
64+
--type String \
65+
--value "$LAYER_ARN" \
66+
--overwrite \
67+
--region "$REGION" \
68+
--query Version \
69+
--output text
70+
)"
71+
72+
echo "Published $PARAMETER_NAME -> $LAYER_ARN (parameter version $PARAMETER_VERSION)"

.github/workflows/merge.yaml

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,13 +228,19 @@ jobs:
228228
run: |
229229
echo "CARGO_PKG_VERSION=$(cargo metadata --no-deps --format-version=1 | jq -r '.packages[0].version')" >> $GITHUB_ENV
230230
231+
- name: Resolve the layer stack names
232+
run: |
233+
# Once, from BETA_STACK_NAME, so the deploy and the publish below cannot drift.
234+
echo "X86_STACK=${BETA_STACK_NAME}-x86" >> $GITHUB_ENV
235+
echo "ARM64_STACK=${BETA_STACK_NAME}-arm64" >> $GITHUB_ENV
236+
231237
- uses: actions/download-artifact@v4
232238
with:
233239
name: packaged-beta-x86_64.yaml
234240

235241
- name: Deploy x86_64 layer to beta account
236242
run: |
237-
sam deploy --stack-name ${BETA_STACK_NAME}-x86 \
243+
sam deploy --stack-name ${X86_STACK} \
238244
--template packaged-beta-x86_64.yaml \
239245
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
240246
--capabilities CAPABILITY_IAM \
@@ -250,7 +256,7 @@ jobs:
250256

251257
- name: Deploy arm64 layer to beta account
252258
run: |
253-
sam deploy --stack-name ${BETA_STACK_NAME}-arm64 \
259+
sam deploy --stack-name ${ARM64_STACK} \
254260
--template packaged-beta-arm64.yaml \
255261
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
256262
--capabilities CAPABILITY_IAM \
@@ -260,6 +266,23 @@ jobs:
260266
--no-fail-on-empty-changeset \
261267
--role-arn ${BETA_CLOUDFORMATION_EXECUTION_ROLE}
262268
269+
# Both deploys complete before either pointer is written, deliberately. These steps
270+
# need ssm:PutParameter, granted to the PipelineExecutionRole by
271+
# LambdaWebAdapterInfrastructureCDK (lib/infra-template.yaml,
272+
# PipelineExecutionRolePermissions) — so a region whose pipeline-resources stack has
273+
# not been updated with that grant fails here with AccessDenied. Ordered this way,
274+
# that failure leaves both layers published and the version pointers stale; ordered
275+
# between the deploys, it left the region with a new x86 layer and no arm64 one.
276+
- name: Publish versioned x86_64 layer parameter
277+
run: |
278+
.github/scripts/publish-layer-parameter.sh \
279+
"$X86_STACK" x86_64 "${CARGO_PKG_VERSION}" "${BETA_REGION}"
280+
281+
- name: Publish versioned arm64 layer parameter
282+
run: |
283+
.github/scripts/publish-layer-parameter.sh \
284+
"$ARM64_STACK" arm64 "${CARGO_PKG_VERSION}" "${BETA_REGION}"
285+
263286
e2e-test-zip:
264287
needs: [deploy-beta]
265288
runs-on: ubuntu-24.04

.github/workflows/release.yaml

Lines changed: 91 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -428,6 +428,11 @@ jobs:
428428
runs-on: ubuntu-24.04
429429
strategy:
430430
matrix: ${{fromJSON(needs.load-matrices.outputs.gamma)}}
431+
env:
432+
# One definition each, used by the deploy and by the publish step
433+
# below, so the two cannot drift apart in a copy-paste.
434+
X86_STACK: lambda-adapter-gamma-x86-${{ matrix.region }}
435+
ARM64_STACK: lambda-adapter-gamma-arm64-${{ matrix.region }}
431436
steps:
432437
- uses: actions/checkout@v4
433438
- uses: actions/setup-python@v4
@@ -464,7 +469,7 @@ jobs:
464469

465470
- name: Deploy x86_64 Layer to all regions in gamma account
466471
run: |
467-
sam deploy --stack-name lambda-adapter-gamma-x86-${{ matrix.region }} \
472+
sam deploy --stack-name ${X86_STACK} \
468473
--template packaged-gamma-x86_64-${{ matrix.region }}.yaml \
469474
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
470475
--capabilities CAPABILITY_IAM \
@@ -474,14 +479,15 @@ jobs:
474479
--no-fail-on-empty-changeset \
475480
--role-arn ${{ matrix.cloudformation_execution_role }}
476481
482+
477483
- uses: actions/download-artifact@v4
478484
with:
479485
name: packaged-gamma-arm64-${{ matrix.region }}.yaml
480486

481487
- name: Deploy arm64 Layer to supported regions in gamma account
482488
if: ${{ matrix.arm64_supported }}
483489
run: |
484-
sam deploy --stack-name lambda-adapter-gamma-arm64-${{ matrix.region }} \
490+
sam deploy --stack-name ${ARM64_STACK} \
485491
--template packaged-gamma-arm64-${{ matrix.region }}.yaml \
486492
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
487493
--capabilities CAPABILITY_IAM \
@@ -491,12 +497,37 @@ jobs:
491497
--no-fail-on-empty-changeset \
492498
--role-arn ${{ matrix.cloudformation_execution_role }}
493499
500+
# Both deploys complete before either pointer is written, deliberately. These steps
501+
# need ssm:PutParameter, granted to the PipelineExecutionRole by
502+
# LambdaWebAdapterInfrastructureCDK (lib/infra-template.yaml,
503+
# PipelineExecutionRolePermissions) — so a region whose pipeline-resources stack has
504+
# not been updated with that grant fails here with AccessDenied. Ordered this way,
505+
# that failure leaves both layers published and the version pointers stale; ordered
506+
# between the deploys, it left the region with a new x86 layer and no arm64 one.
507+
- name: Publish versioned x86_64 layer parameter
508+
run: |
509+
.github/scripts/publish-layer-parameter.sh \
510+
"$X86_STACK" \
511+
x86_64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
512+
513+
- name: Publish versioned arm64 layer parameter
514+
if: ${{ matrix.arm64_supported }}
515+
run: |
516+
.github/scripts/publish-layer-parameter.sh \
517+
"$ARM64_STACK" \
518+
arm64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
519+
494520
deploy-prod:
495521
needs: [load-matrices, deploy-gamma, package-prod]
496522
runs-on: ubuntu-24.04
497523
environment: prod
498524
strategy:
499525
matrix: ${{fromJSON(needs.load-matrices.outputs.prod)}}
526+
env:
527+
# One definition each, used by the deploy and by the publish step
528+
# below, so the two cannot drift apart in a copy-paste.
529+
X86_STACK: lambda-adapter-prod-x86-${{ matrix.region }}
530+
ARM64_STACK: lambda-adapter-prod-arm64-${{ matrix.region }}
500531
steps:
501532
- uses: actions/checkout@v4
502533
- uses: actions/setup-python@v4
@@ -533,7 +564,7 @@ jobs:
533564

534565
- name: Deploy x86_64 Layer to all regions in prod account
535566
run: |
536-
sam deploy --stack-name lambda-adapter-prod-x86-${{ matrix.region }} \
567+
sam deploy --stack-name ${X86_STACK} \
537568
--template packaged-prod-x86_64-${{ matrix.region }}.yaml \
538569
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
539570
--capabilities CAPABILITY_IAM \
@@ -543,14 +574,15 @@ jobs:
543574
--no-fail-on-empty-changeset \
544575
--role-arn ${{ matrix.cloudformation_execution_role }}
545576
577+
546578
- uses: actions/download-artifact@v4
547579
with:
548580
name: packaged-prod-arm64-${{ matrix.region }}.yaml
549581

550582
- name: Deploy arm64 Layer to supported regions in prod account
551583
if: ${{ matrix.arm64_supported }}
552584
run: |
553-
sam deploy --stack-name lambda-adapter-prod-arm64-${{ matrix.region }} \
585+
sam deploy --stack-name ${ARM64_STACK} \
554586
--template packaged-prod-arm64-${{ matrix.region }}.yaml \
555587
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
556588
--capabilities CAPABILITY_IAM \
@@ -560,11 +592,29 @@ jobs:
560592
--no-fail-on-empty-changeset \
561593
--role-arn ${{ matrix.cloudformation_execution_role }}
562594
595+
- name: Publish versioned x86_64 layer parameter
596+
run: |
597+
.github/scripts/publish-layer-parameter.sh \
598+
"$X86_STACK" \
599+
x86_64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
600+
601+
- name: Publish versioned arm64 layer parameter
602+
if: ${{ matrix.arm64_supported }}
603+
run: |
604+
.github/scripts/publish-layer-parameter.sh \
605+
"$ARM64_STACK" \
606+
arm64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
607+
563608
deploy-china-gamma:
564609
needs: [load-matrices, package-china-gamma]
565610
runs-on: ubuntu-24.04
566611
strategy:
567612
matrix: ${{fromJSON(needs.load-matrices.outputs.china-gamma)}}
613+
env:
614+
# One definition each, used by the deploy and by the publish step
615+
# below, so the two cannot drift apart in a copy-paste.
616+
X86_STACK: lambda-adapter-gamma-x86-${{ matrix.region }}
617+
ARM64_STACK: lambda-adapter-gamma-arm64-${{ matrix.region }}
568618
steps:
569619
- uses: actions/checkout@v4
570620
- uses: actions/setup-python@v4
@@ -603,7 +653,7 @@ jobs:
603653

604654
- name: Deploy x86_64 Layer to all regions in china
605655
run: |
606-
sam deploy --stack-name lambda-adapter-gamma-x86-${{ matrix.region }} \
656+
sam deploy --stack-name ${X86_STACK} \
607657
--template packaged-china-gamma-x86_64-${{ matrix.region }}.yaml \
608658
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
609659
--capabilities CAPABILITY_IAM \
@@ -613,14 +663,15 @@ jobs:
613663
--no-fail-on-empty-changeset \
614664
--role-arn ${{ matrix.cloudformation_execution_role }}
615665
666+
616667
- uses: actions/download-artifact@v4
617668
with:
618669
name: packaged-china-gamma-arm64-${{ matrix.region }}.yaml
619670

620671
- name: Deploy arm64 Layer to supported china regions
621672
if: ${{ matrix.arm64_supported }}
622673
run: |
623-
sam deploy --stack-name lambda-adapter-gamma-arm64-${{ matrix.region }} \
674+
sam deploy --stack-name ${ARM64_STACK} \
624675
--template packaged-china-gamma-arm64-${{ matrix.region }}.yaml \
625676
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
626677
--capabilities CAPABILITY_IAM \
@@ -630,12 +681,30 @@ jobs:
630681
--no-fail-on-empty-changeset \
631682
--role-arn ${{ matrix.cloudformation_execution_role }}
632683
684+
- name: Publish versioned x86_64 layer parameter
685+
run: |
686+
.github/scripts/publish-layer-parameter.sh \
687+
"$X86_STACK" \
688+
x86_64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
689+
690+
- name: Publish versioned arm64 layer parameter
691+
if: ${{ matrix.arm64_supported }}
692+
run: |
693+
.github/scripts/publish-layer-parameter.sh \
694+
"$ARM64_STACK" \
695+
arm64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
696+
633697
deploy-china-prod:
634698
needs: [load-matrices, deploy-china-gamma, package-china-prod]
635699
runs-on: ubuntu-24.04
636700
environment: prod
637701
strategy:
638702
matrix: ${{fromJSON(needs.load-matrices.outputs.china-prod)}}
703+
env:
704+
# One definition each, used by the deploy and by the publish step
705+
# below, so the two cannot drift apart in a copy-paste.
706+
X86_STACK: lambda-adapter-prod-x86-${{ matrix.region }}
707+
ARM64_STACK: lambda-adapter-prod-arm64-${{ matrix.region }}
639708
steps:
640709
- uses: actions/checkout@v4
641710
- uses: actions/setup-python@v4
@@ -674,7 +743,7 @@ jobs:
674743

675744
- name: Deploy x86_64 Layer to all regions in china
676745
run: |
677-
sam deploy --stack-name lambda-adapter-prod-x86-${{ matrix.region }} \
746+
sam deploy --stack-name ${X86_STACK} \
678747
--template packaged-china-prod-x86_64-${{ matrix.region }}.yaml \
679748
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
680749
--capabilities CAPABILITY_IAM \
@@ -684,14 +753,15 @@ jobs:
684753
--no-fail-on-empty-changeset \
685754
--role-arn ${{ matrix.cloudformation_execution_role }}
686755
756+
687757
- uses: actions/download-artifact@v4
688758
with:
689759
name: packaged-china-prod-arm64-${{ matrix.region }}.yaml
690760

691761
- name: Deploy arm64 Layer to supported china regions
692762
if: ${{ matrix.arm64_supported }}
693763
run: |
694-
sam deploy --stack-name lambda-adapter-prod-arm64-${{ matrix.region }} \
764+
sam deploy --stack-name ${ARM64_STACK} \
695765
--template packaged-china-prod-arm64-${{ matrix.region }}.yaml \
696766
--parameter-overrides CargoPkgVersion=${CARGO_PKG_VERSION} \
697767
--capabilities CAPABILITY_IAM \
@@ -701,6 +771,19 @@ jobs:
701771
--no-fail-on-empty-changeset \
702772
--role-arn ${{ matrix.cloudformation_execution_role }}
703773
774+
- name: Publish versioned x86_64 layer parameter
775+
run: |
776+
.github/scripts/publish-layer-parameter.sh \
777+
"$X86_STACK" \
778+
x86_64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
779+
780+
- name: Publish versioned arm64 layer parameter
781+
if: ${{ matrix.arm64_supported }}
782+
run: |
783+
.github/scripts/publish-layer-parameter.sh \
784+
"$ARM64_STACK" \
785+
arm64 "${CARGO_PKG_VERSION}" "${{ matrix.region }}"
786+
704787
publish-to-public-ecr:
705788
needs: [deploy-prod]
706789
runs-on: ubuntu-24.04

template-arm64.yaml

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,9 @@ Resources:
3939
Type: String
4040
Value: !Ref LambdaAdapterLayerArm64
4141

42-
# A per-version parameter (/lambda-web-adapter/layer/arm64/<CargoPkgVersion>) used to
43-
# live here. See the matching comment in template-x86_64.yaml for why it was removed:
44-
# a Name change replaces the parameter, and UpdateReplacePolicy: Retain orphaned the
45-
# old name, so re-deploying an already-used version failed with AlreadyExists.
42+
# The version-named parameter (/lambda-web-adapter/layer/arm64/<CargoPkgVersion>)
43+
# is published after deployment with `aws ssm put-parameter --overwrite`. See the
44+
# matching comment in template-x86_64.yaml for why it cannot live in this stack.
4645

4746
Outputs:
4847
LambdaAdapterLayerArm64Arn:

template-x86_64.yaml

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -39,14 +39,10 @@ Resources:
3939
Type: String
4040
Value: !Ref LambdaAdapterLayerX86
4141

42-
# A per-version parameter (/lambda-web-adapter/layer/x86_64/<CargoPkgVersion>) used
43-
# to live here. It was removed: because CloudFormation replaces a parameter when its
44-
# Name changes, and UpdateReplacePolicy was Retain, every version bump orphaned the
45-
# previous name. Deploying a version whose name had already been orphaned — after the
46-
# v1.1.0 revert, or on any re-release — then failed with AlreadyExists and wedged
47-
# every merge to main. Existing per-version parameters are retained, and the layer's
48-
# Description still records the version, so `aws lambda list-layer-versions` maps a
49-
# version to its layer ARN.
42+
# The version-named parameter (/lambda-web-adapter/layer/x86_64/<CargoPkgVersion>)
43+
# is published after deployment with `aws ssm put-parameter --overwrite`. It cannot
44+
# live in this stack: a Name change replaces the resource, while retaining old
45+
# versions leaves names that CloudFormation collides with when a version is re-used.
5046

5147
Outputs:
5248
LambdaAdapterLayerX86Arn:

0 commit comments

Comments
 (0)