[belaboe]

Problem

API Gateway allows one DeleteRestApi request every 30 seconds per account. This quota is fixed and cannot be raised. When several stacks in one account tear down at once, CloudFormation's own SDK retries run out, and the stack ends DELETE_FAILED:

DefaultCompute/API ... Too Many Requests (Service: ApiGateway, Status Code: 429, ...) (SDK Attempt Count: 5)

runDestroyWithRetries, the retry loop behind both sandbox:destroy and production destroy, then retried on a fixed 60s/120s schedule:

• Parallel destroys retried in lockstep and collided again.
• The loop gave up after 3 attempts, about 13 minutes after the first failure.

Data:

• E2E logs: 35 failed or sampled E2E logs from 2026-10-19 to 10-22 contain the 429. 21 of them are the Hosting matrix (9 variants that tear down together), plus Telemetry, Production, Sandbox, Compute Smoke and Amplify.
• Recovery time: when the throttle persisted, it cleared 9 to 32 minutes after the first 429. Most cases needed a second destroy call from the test harness or cleanup step. Two never cleared within the job.
• Live stacks: the stacks left DELETE_FAILED in the E2E accounts today show the same resource reason.

Change

After a failed attempt, runDestroyWithRetries asks isThrottled whether this attempt was throttled. The real implementation is destroyWasThrottled, which uses DescribeStackEvents:

• It reads each stack's events newest-first, back to this attempt's own stack-level DELETE_IN_PROGRESS, and ignores anything up to the previous attempt's failure.
• A resource DELETE_FAILED whose reason carries a throttle token counts as a throttle. The tokens are Too Many Requests, Status Code: 429, TooManyRequestsException, ThrottlingException, Error Code: Throttling, Rate exceeded, RequestLimitExceeded and Error Code: SlowDown. Matching is case-sensitive, and stack-level summary and nested-stack events are skipped.

How each kind of failure is retried:

Failure	Retry schedule
Throttled	Exponential backoff with equal jitter: steps of 1, 2, 4, then 5 minutes, each wait a random point in [step/2, step]. Continues while a wait would end within 20 minutes of the first failure.
Any other failure	Unchanged: at most 60s then 120s per call. No ordinary retry once throttle retries have used the budget.
Events cannot be read (permission, Region, all stacks gone)	Prints a warning and uses the ordinary schedule.

Why these numbers:

• 1 min base: many throttles cleared on the first 60s retry.
• 5 min cap: about one throttled CloudFormation attempt, and about 10 queued deletes at 1 per 30s.
• 20 min budget: in a single call, it covers about 7 of the 13 persistent throttle windows observed. Today's schedule covers 2.
• Worst case per call: about 30 minutes with 5-minute attempts (two throttled failures, then ordinary ones: the last throttle wait ends inside the budget, then one ordinary 120 s retry). All-throttled is about 27 minutes. Today it is about 18 minutes. This only happens when the throttle persists, and today that case ends in a failure anyway. The changeset states this. The budget is fixed: no environment override is added here (an open question for maintainers, since a new variable is customer-facing).
• Mixed failures: an attempt counts as throttled when any resource delete in it was throttled, so another resource that failed for a different reason in the same attempt gets the throttle schedule too. It is retried later, not hidden, and still fails the call once the budget is spent. The runDestroyWithRetries JSDoc says so.
• When it gives up: if the last attempt was throttled, destroy prints a one-line hint that AWS throttling was the cause, to wait a few minutes and run destroy again, and to check the stack events if it fails again. If an ordinary failure gets no retry only because throttle retries used up the budget, it says that instead.

Other details:

• Public API: none. The new helpers, including the per-path dependency builders sandboxDestroyDeps and productionDestroyDeps, are not exported from @aws-blocks/core/scripts.
• Edge stack: throttle detection reads the stack names from cdk ls with a default-Region client, so a stack in another Region (the Lambda@Edge stack) is not read and falls back to the ordinary schedule. The per-Region stack list in fix(core): destroy and watch every top-level stack of a multi-stack app #968 fixes that for this detection too; this PR does not duplicate it.
• Dependencies: no new dependency; @aws-sdk/client-cloudformation is already used here.
• Changeset: @aws-blocks/core patch.

Not changed here (load spreading)

The quota is per account. A full PR Checks run deletes about 15 REST APIs from 12 stacks within about 20 to 30 minutes. Parallel PRs that share a pool account add to that. The Hosting matrix is the hot spot. Spreading the load would remove more of the 429s than any retry can:

• staggering Hosting teardown, or capping its parallelism
• fewer native stacks per run
• REGIONAL instead of EDGE endpoints for test apps, which raises the CreateRestApi quota from 1 per 30s to 1 per 3s but leaves DeleteRestApi unchanged

test-apps/compute-smoke/test/sandbox-destroy.ts calls raw cdk destroy --all with no retry, so it does not benefit from this change. Its stacks are the live leaks.

Tests

• Red, then green: packages/core/src/scripts/sandbox-destroy-throttle.test.ts.
  • On main: a 429 destroy gets 3 attempts (the test expects 4), and the budget and attempt-time tests fail.
  • Review follow-ups, each checked against the mutant it targets: the per-path builders wire in destroyWasThrottled and emptySandboxBuckets (deleting either line fails a test); the pending wait counts against the budget; buckets are emptied before each throttle retry; slow attempts that were never throttled keep both ordinary retries; both exhaustion messages.
• Local lanes (AL2023 container): npm run build, lint:deps, test:e2e:local and lint pass. npm test passes except core's telemetry.test.js, which hung only while two lane containers ran at once; core's suite re-run alone passes (0 failures), and CI's unit job passes.
• E2E: the head commit has no E2E: trailer, so this is a full PR Checks run.
• Status (head 8bb0af6, full PR Checks run): Build, Unit Tests, E2E Local; E2E Sandbox; E2E Production; E2E Hosting (all 9); Telemetry E2E; E2E Templates (Linux and Windows); E2E VPC Smoke; E2E Supabase; E2E Amplify Interop; Native SDK E2E; Require changeset: all pass.
• Not caused by this change, not re-run:
  • E2E Compute Smoke failed in "Ensure persistent compute test VPC", before any test: the shared BlocksComputeTestVpc stack update was refused ("Cannot delete export ... as it is in use by BlocksComputeTestVpcEndpoints"), a test-infrastructure mismatch in the pool account.
  • Build and Test aggregates that job. bench / summary fails its floor with no scored cell (mean composite 0.0), as on earlier runs; not a required check.

[changeset-bot]

🦋 Changeset detected

Latest commit: f51c712

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@aws-blocks/core	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[belaboe]

On the review body: the mixed throttle-plus-real-failure behaviour is intended and now stated in the runDestroyWithRetries JSDoc (a9c188a3), and head 8bb0af6 has no E2E: trailer: its full PR Checks run passed every E2E job, including Hosting (all 9) and Telemetry, except Compute Smoke, which failed before the tests ran because the shared BlocksComputeTestVpc stack update was refused (an export in use by BlocksComputeTestVpcEndpoints). The budget knob, telemetry counter and cross-layer budget questions are listed as open for maintainers in the description.

[osama-rizk]

Since the last review: 4 resolved · 0 new · 0 still open

This makes throttled destroys back off with jitter inside a bounded budget, and after the follow-ups it's ready from my side. I re-checked each item against the code at 8bb0af6, not just the replies. The history was rewritten on the same base, but 680dff2 has the same tree as the 4db0cf5 I reviewed (only the commit messages changed), so the earlier review still applies to that code.

Resolved:

• Worst case documented. The changeset, the runDestroyWithRetries JSDoc and the description now say a destroy that stays throttled can run about 30 minutes (about 18 before). My fake-clock run at this head gives 31 / 27 / 18 minutes (mixed, all-throttled, all-ordinary) with 5-minute attempts, which matches. Keeping the budget fixed and leaving the override to maintainers is a sound answer.
• Wiring guarded. sandboxDestroyDeps / productionDestroyDeps are pinned by the new call-site tests. Deleting isThrottled: destroyWasThrottled from either compiled path now fails a test. The destroyWasThrottled([], …) case calls no AWS API.
• Budget and bucket mutants killed. Dropping + next from the budget check, and skipping emptyBuckets on a throttle retry, both fail the suite now. I also tried four mutants on the new code: no post-budget guard, letting the budget cut ordinary retries when there was no throttle, no throttle hint, and always printing the budget-used hint. Each one fails a test.
• Failure hint. Both exhaustion messages print in the right cases and are tested. The "used up the retry time" branch can only be reached when throttle retries spent the budget.

What I verified:

• Root build OK. sandbox-destroy-throttle + sandbox-empty-buckets: 42/42 pass.
• The public API is unchanged. scripts/index.ts still exports only destroy and destroySandbox.
• No E2E: trailer on any commit. The full PR Checks run 39970264690 on 8bb0af6 passed every E2E (end-to-end) job, including Sandbox, Production, Hosting (all 9) and Telemetry, except Compute Smoke. Compute Smoke failed in cdk deploy BlocksComputeTestVpc ("Cannot delete export … in use by BlocksComputeTestVpcEndpoints") before any test ran. Runs 39974123347 and 39980152214 on unrelated PRs fail the same way the same day, and this PR touches no test-infra/ or workflow files, so the failure is unrelated. Build and Test only aggregates that job. bench / summary is not required (9 of 10 cells isolation_unavailable).

Non-blocking follow-ups (maintainer decisions, not for this PR):

• An optional budget override for customers or CI. It's customer-facing, so it should be a product decision.
• A throttleRetries telemetry counter, so we can see how often customers hit this.
• CI still never ran the throttle branch: no 429 showed up in this run's destroys. The ordinary 60s/120s path did run in Hosting (nuxt-web-function) and recovered. The first real throttle will be the real test, and the logs will show it.

Blocking: none. Open questions for the author: none.