feat: add bin/smoke for post-promotion checks - #1023
Conversation
Deploys promote one prebuilt artifact across dev, staging and production, so the failure worth catching is an app serving another environment's data or configuration. Nothing checked for it: CI only runs the Elixir suite against containers, and the health-check job in heroku.yaml fires on GitHub deployment events, which promotions stopped producing in November 2025 -- its last records for orcasite and orcasite-beta are from 2025-11-15. Even when it ran it fetched / and accepted any success status, which would have passed while production listed development's hydrophones. The checks are self-configuring rather than hardcoded, so one command works against any environment: ask the app's own API which feeds it has, then confirm /listen lists exactly those, that one of them renders, and that feeds the API does not have return 404. A page rendered against a different environment disagrees with the API of the one serving it. Also checks that the injected runtime config reports a bucket, that the config script reaches the served HTML, that no absolute API host is baked into the client bundle, and that the seed page is absent unless EXPECT_SEED_PAGE=1. Verified against all three environments. Dev passes. Beta and production currently fail only on the seed page, correctly: #1022 is merged but not yet promoted, so they still serve a build that baked its visibility in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 31 minutes Limit details: You’ve used all 1 included review currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Review found the script passed on the bug it was written for. The feed comparison only asserted that every feed the API reports appears on /listen. The failure in #1019 was the opposite shape: production served a page built from development's feed list, which is a superset of its own, since dev seeds from prod and adds test hydrophones. Nothing was missing, so the check reported ok. It only fires today because dev happens to lack two feeds prod has -- drift, not detection. Now compares both ways by extracting slugs from the page's own links, so extra feeds fail too. Also fixes substring matching: grep for a bare slug matched inside a longer one, and interpolated it as a regex. The bundle scan truncated to `head -12`. Sorted order puts the app chunks last, so it kept twelve numeric vendor chunks and dropped _app, main, webpack and the page chunk -- exactly where inlined NEXT_PUBLIC_* values land. It now scans all 24, fails when a chunk cannot be fetched rather than counting it clean, and looks for baked bucket names as well as API hosts. Also: retries and redirect following, since this runs while dynos restart and an http:// argument otherwise failed every check with no explanation; /listen fetched once instead of three times; an empty feed list distinguished from an unreachable API; colour gated on a tty and NO_COLOR; and the two piped `grep -q` calls on page HTML replaced with variable matching, which would have failed spuriously once the page outgrew the pipe buffer and pipefail saw curl's EPIPE. The hardcoded dev-only slugs stay, now overridable via ABSENT_SLUGS and documented as deliberate operator knowledge rather than presented as self-configuring: a nonsense slug cannot catch detail pages prerendered from another environment, having never been prerendered anywhere. Verified against all three environments. Dev passes. Beta and production still fail only the seed page, correctly, pending promotion of #1022. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Pushed The feed comparison was one-directional. It asserted only that every feed the API reports appears on It only fired today because dev happens to lack The bundle scan was truncated to the wrong chunks. Also fixed: retries and redirect following (this runs while dynos restart, and an The hardcoded dev-only slugs stay, but now overridable via Verified against all three environments. Dev passes fully; beta and production still fail only the seed page, correctly, pending promotion of #1022. 🤖 |
2 participants
Deploys promote one prebuilt artifact across dev, staging and production, so the failure worth catching is an app serving another environment's data or configuration — the bug fixed in #1019, which reached production and needed a rollback.
Nothing checks for it today. CI only runs the Elixir suite against containers. The
health-checkjob inheroku.yamlfires on GitHub deployment events, and promotions stopped producing those:Review apps still create them, so the workflow works — it just never fires for the environments that matter. And even when it did, it fetched
/and accepted any success status, which would have passed happily while production listed development's hydrophones.Approach
The checks are self-configuring rather than hardcoded, so one command works against any environment with no per-app expectations to maintain:
It asks the app's own API which feeds it has, then confirms
/listenlists exactly those, that one of them renders, and that feeds the API does not have return 404. A page rendered against a different environment disagrees with the API of the one serving it — which is precisely the bug, expressed as an invariant instead of a fixture.It also checks that the injected runtime config reports a bucket, that the config script reaches the served HTML, that no absolute API host is baked into the client bundle, and that
/seedis absent unlessEXPECT_SEED_PAGE=1.Pure bash and curl, no dependencies, shellcheck clean, exits non-zero on failure.
Verified against all three environments
Dev passes fully. Beta and production pass everything except the seed page:
That is correct, not a bug in the script: #1022 is merged but not yet promoted, so beta and production still serve a build that baked the page's visibility in. It should go green once you promote — which makes it a decent first exercise of the thing.
Not automated
Deliberately a script rather than a workflow, since promotions emit no events a workflow could hang off. Run it by hand after promoting. If you would rather have it wired into CI against a fixed URL, or run on a schedule, happy to add that.
🤖