Skip to content

Commit 96f8bb0

Browse files
rainheadclaude
rainhead
and
claude
authored
feat: add bin/smoke for post-promotion checks (#1023)
* feat: add bin/smoke for post-promotion checks Deploys promote one prebuilt artifact across dev, staging and production, so the failure worth catching is an app serving another environment's data or configuration. Nothing checked for it: CI only runs the Elixir suite against containers, and the health-check job in heroku.yaml fires on GitHub deployment events, which promotions stopped producing in November 2025 -- its last records for orcasite and orcasite-beta are from 2025-11-15. Even when it ran it fetched / and accepted any success status, which would have passed while production listed development's hydrophones. The checks are self-configuring rather than hardcoded, so one command works against any environment: ask the app's own API which feeds it has, then confirm /listen lists exactly those, that one of them renders, and that feeds the API does not have return 404. A page rendered against a different environment disagrees with the API of the one serving it. Also checks that the injected runtime config reports a bucket, that the config script reaches the served HTML, that no absolute API host is baked into the client bundle, and that the seed page is absent unless EXPECT_SEED_PAGE=1. Verified against all three environments. Dev passes. Beta and production currently fail only on the seed page, correctly: #1022 is merged but not yet promoted, so they still serve a build that baked its visibility in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: compare feeds in both directions, scan every chunk Review found the script passed on the bug it was written for. The feed comparison only asserted that every feed the API reports appears on /listen. The failure in #1019 was the opposite shape: production served a page built from development's feed list, which is a superset of its own, since dev seeds from prod and adds test hydrophones. Nothing was missing, so the check reported ok. It only fires today because dev happens to lack two feeds prod has -- drift, not detection. Now compares both ways by extracting slugs from the page's own links, so extra feeds fail too. Also fixes substring matching: grep for a bare slug matched inside a longer one, and interpolated it as a regex. The bundle scan truncated to `head -12`. Sorted order puts the app chunks last, so it kept twelve numeric vendor chunks and dropped _app, main, webpack and the page chunk -- exactly where inlined NEXT_PUBLIC_* values land. It now scans all 24, fails when a chunk cannot be fetched rather than counting it clean, and looks for baked bucket names as well as API hosts. Also: retries and redirect following, since this runs while dynos restart and an http:// argument otherwise failed every check with no explanation; /listen fetched once instead of three times; an empty feed list distinguished from an unreachable API; colour gated on a tty and NO_COLOR; and the two piped `grep -q` calls on page HTML replaced with variable matching, which would have failed spuriously once the page outgrew the pipe buffer and pipefail saw curl's EPIPE. The hardcoded dev-only slugs stay, now overridable via ABSENT_SLUGS and documented as deliberate operator knowledge rather than presented as self-configuring: a nonsense slug cannot catch detail pages prerendered from another environment, having never been prerendered anywhere. Verified against all three environments. Dev passes. Beta and production still fail only the seed page, correctly, pending promotion of #1022. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent ac18b03 commit 96f8bb0

2 files changed

Lines changed: 259 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,34 @@ Because one artifact serves every environment, anything captured while it is bui
137137
138138
Local development builds and runs in a single environment, where inlining is harmless, so [`ui/.env.development`](ui/.env.development) sets `NEXT_PUBLIC_*` normally.
139139
140+
### Smoke checks
141+
142+
Run [`bin/smoke`](bin/smoke) against an app after promoting to it:
143+
144+
```shell
145+
bin/smoke https://live.orcasound.net
146+
```
147+
148+
It exits non-zero if anything fails. Most checks are self-configuring: they ask
149+
the app's own API what it should be showing and confirm the pages agree, so the
150+
same command works against any environment. An app serving another environment's
151+
data disagrees with its own API, which is the failure this deployment model
152+
invites and the reason these exist. The feed comparison runs in both directions,
153+
because the failure that prompted this showed up as a page listing *extra*
154+
feeds — production serving a page built from development's list.
155+
156+
Two checks carry knowledge the app cannot supply, and both take an override:
157+
158+
- `EXPECT_SEED_PAGE=1` where seeding is deliberately enabled; elsewhere `/seed`
159+
is expected to be absent.
160+
- `ABSENT_SLUGS` lists feeds that exist only in development, so elsewhere they
161+
must not resolve. A nonsense slug cannot cover this: a slug that never existed
162+
anywhere was never prerendered either.
163+
164+
Nothing runs these automatically. The `health-check` job in
165+
[`heroku.yaml`](.github/workflows/heroku.yaml) fires on GitHub deployment
166+
events, which slug promotions no longer produce.
167+
140168
### Console
141169
142170
```shell

‎bin/smoke‎

Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
#!/usr/bin/env bash
2+
# Post-deploy checks for a running Orcasite app.
3+
#
4+
# bin/smoke https://live.orcasound.net
5+
#
6+
# Deploys promote one prebuilt artifact across dev, staging and production, so
7+
# the failure worth catching is an app serving another environment's data or
8+
# configuration. Most checks are self-configuring: they ask the app's own API
9+
# what it should be showing and confirm the pages agree, so the same command
10+
# works against any environment. Two checks carry operator knowledge instead,
11+
# each marked below, because the app's public surface cannot supply it.
12+
#
13+
# Exits non-zero if any check fails.
14+
15+
set -uo pipefail
16+
17+
BASE="${1:-}"
18+
19+
if [ -z "$BASE" ]; then
20+
echo "usage: bin/smoke <base-url> e.g. bin/smoke https://live.orcasound.net" >&2
21+
exit 64
22+
fi
23+
24+
BASE="${BASE%/}"
25+
TIMEOUT=45
26+
failures=0
27+
28+
# Retries matter: this is meant to run right after a promotion, while dynos are
29+
# still restarting and the router returns transient 503s.
30+
CURL_OPTS=(-sS -L --max-time "$TIMEOUT" --retry 3 --retry-connrefused)
31+
32+
if [ -t 1 ] && [ -z "${NO_COLOR:-}" ]; then
33+
C_OK=$'\033[32m'
34+
C_BAD=$'\033[31m'
35+
C_OFF=$'\033[0m'
36+
else
37+
C_OK="" C_BAD="" C_OFF=""
38+
fi
39+
40+
pass() { printf ' %sok%s %s\n' "$C_OK" "$C_OFF" "$1"; }
41+
42+
fail() {
43+
printf ' %sFAIL%s %s\n' "$C_BAD" "$C_OFF" "$1"
44+
if [ $# -gt 1 ]; then
45+
printf ' %s\n' "$2"
46+
fi
47+
failures=$((failures + 1))
48+
}
49+
50+
get() { curl "${CURL_OPTS[@]}" "$BASE$1"; }
51+
status() { curl "${CURL_OPTS[@]}" -o /dev/null -w '%{http_code}' "$BASE$1"; }
52+
53+
expect_status() {
54+
local path="$1" want="$2" got
55+
got="$(status "$path")"
56+
if [ "$got" = "$want" ]; then
57+
pass "$path -> $got"
58+
else
59+
fail "$path -> $got (expected $want)"
60+
fi
61+
}
62+
63+
echo "Smoke checking $BASE"
64+
65+
# --- Reachability -----------------------------------------------------------
66+
echo
67+
echo "Routes"
68+
expect_status / 200
69+
expect_status /listen 200
70+
71+
# Fetched once and reused: mid-restart, separate requests can land on different
72+
# dynos and disagree with each other.
73+
listen_html="$(get /listen)"
74+
75+
# --- Per-environment configuration -----------------------------------------
76+
# Injected per request, so it must describe the app answering the request. If
77+
# this reports another environment's bucket, the artifact is serving build-time
78+
# configuration and audio URLs point at the wrong place.
79+
echo
80+
echo "Runtime configuration"
81+
config="$(get /api/runtime-config)"
82+
83+
if [ -z "$config" ]; then
84+
fail "/api/runtime-config returned nothing"
85+
bucket=""
86+
else
87+
bucket="$(printf '%s' "$config" | grep -o '"s3Bucket":"[^"]*"' | cut -d'"' -f4)"
88+
89+
if [ -n "$bucket" ]; then
90+
pass "s3Bucket = $bucket"
91+
else
92+
# An empty bucket is the deliberate last-resort fallback in
93+
# ui/src/utils/runtimeConfig.ts, and means injection failed.
94+
fail "no s3Bucket in /api/runtime-config" "$config"
95+
fi
96+
fi
97+
98+
# Matched against a variable rather than through a pipe: `curl | grep -q` lets
99+
# grep exit on first match and kills curl with EPIPE, which pipefail would
100+
# report as a failed check once the page outgrows the pipe buffer.
101+
if [[ "$listen_html" == *'<script src="/api/runtime-config"'* ]]; then
102+
pass "config script present in served HTML"
103+
else
104+
fail "config script missing from /listen"
105+
fi
106+
107+
# --- Pages must agree with this app's own API -------------------------------
108+
echo
109+
echo "Feeds match this environment's API"
110+
feeds_response="$(curl "${CURL_OPTS[@]}" -X POST "$BASE/graphql" \
111+
-H 'Content-Type: application/json' \
112+
-d '{"query":"query { feeds { slug } }"}')"
113+
114+
api_slugs="$(printf '%s' "$feeds_response" | grep -o '"slug":"[^"]*"' | cut -d'"' -f4 | sort -u)"
115+
116+
if [ -z "$feeds_response" ]; then
117+
fail "no response from $BASE/graphql" "same-origin GraphQL may be unreachable"
118+
elif [ -z "$api_slugs" ]; then
119+
# An empty list is a real answer; distinguish it from an unreachable API so
120+
# a freshly reset database does not look like an outage.
121+
fail "API returned no feeds" "$(printf '%s' "$feeds_response" | head -c 200)"
122+
else
123+
api_count="$(printf '%s\n' "$api_slugs" | wc -l | tr -d ' ')"
124+
pass "API reports $api_count feeds"
125+
126+
# Compared both ways. A page missing feeds is the obvious failure, but the
127+
# bug this script exists for showed up as the opposite: production served a
128+
# page baked from development's feed list, which is a superset of its own.
129+
# Checking only that the API's feeds appear would have passed on it.
130+
page_slugs="$(printf '%s' "$listen_html" |
131+
grep -o 'href="/listen/[^"]*"' |
132+
sed 's|.*/listen/||; s|"$||' | sort -u)"
133+
134+
missing="$(comm -23 <(printf '%s\n' "$api_slugs") <(printf '%s\n' "$page_slugs") | tr '\n' ' ')"
135+
extra="$(comm -13 <(printf '%s\n' "$api_slugs") <(printf '%s\n' "$page_slugs") | tr '\n' ' ')"
136+
137+
if [ -z "${missing// /}" ] && [ -z "${extra// /}" ]; then
138+
pass "/listen lists exactly the feeds this API reports"
139+
else
140+
[ -n "${missing// /}" ] && fail "/listen is missing feeds the API reports: $missing" \
141+
"the page may have been rendered against a different environment"
142+
[ -n "${extra// /}" ] && fail "/listen lists feeds the API does not have: $extra" \
143+
"the page was rendered against a different environment"
144+
fi
145+
146+
first_slug="$(printf '%s\n' "$api_slugs" | head -1)"
147+
expect_status "/listen/$first_slug" 200
148+
fi
149+
150+
expect_status /listen/definitely-not-a-real-feed 404
151+
152+
# OPERATOR KNOWLEDGE, not derived from the app. A nonsense slug only proves
153+
# unknown feeds 404; it cannot catch detail pages prerendered from another
154+
# environment's feed list, because a slug that never existed anywhere was never
155+
# prerendered either. These are real feeds that exist only in development, so
156+
# outside development they must not resolve. Override for an environment that
157+
# legitimately has them.
158+
ABSENT_SLUGS="${ABSENT_SLUGS:-paul-test rpi-steve-test}"
159+
160+
for absent in $ABSENT_SLUGS; do
161+
if printf '%s\n' "$api_slugs" | grep -qxF -- "$absent"; then
162+
pass "/listen/$absent skipped (this environment really has it)"
163+
else
164+
expect_status "/listen/$absent" 404
165+
fi
166+
done
167+
168+
# --- Nothing environment-specific baked into the bundle ---------------------
169+
# Endpoints resolve same-origin, and the bucket is injected at request time,
170+
# precisely so one artifact can serve every app. Anything absolute here means a
171+
# build-time capture crept back in.
172+
echo
173+
echo "Client bundle"
174+
chunks="$(printf '%s' "$listen_html" | grep -o '/_next/static/chunks/[^"]*\.js' | sort -u)"
175+
176+
if [ -z "$chunks" ]; then
177+
fail "found no client chunks to inspect"
178+
else
179+
# Every chunk, deliberately. Sorted order puts the app chunks (_app, main,
180+
# webpack, the page chunk) last, so truncating this list drops exactly the
181+
# files where inlined values land.
182+
baked="" unreadable="" scanned=0
183+
while IFS= read -r chunk; do
184+
[ -z "$chunk" ] && continue
185+
body="$(get "$chunk")"
186+
187+
if [ -z "$body" ]; then
188+
unreadable="$unreadable $chunk"
189+
continue
190+
fi
191+
192+
scanned=$((scanned + 1))
193+
found="$(printf '%s' "$body" |
194+
grep -oh 'https\?://[a-z0-9.-]*/graphql\|wss\?://[a-z0-9.:-]*/socket\|[a-z-]*-orcasound-net' |
195+
sort -u | tr '\n' ' ')"
196+
[ -n "$found" ] && baked="$baked $chunk:$found"
197+
done <<<"$chunks"
198+
199+
if [ -n "$unreadable" ]; then
200+
# Silently treating these as clean would report a pass having inspected
201+
# nothing.
202+
fail "could not fetch client chunks:$unreadable"
203+
fi
204+
205+
if [ -z "$baked" ]; then
206+
pass "no absolute API host or bucket in $scanned chunks"
207+
else
208+
fail "environment-specific value baked into client bundle:$baked"
209+
fi
210+
fi
211+
212+
# --- Developer tooling should not be public ---------------------------------
213+
# OPERATOR KNOWLEDGE. Visibility follows ENABLE_SEED_FROM_PROD for the serving
214+
# app, which cannot be read from the app's public surface without trusting the
215+
# thing under test. Set EXPECT_SEED_PAGE=1 where seeding is deliberately on.
216+
echo
217+
echo "Seed page"
218+
if [ "${EXPECT_SEED_PAGE:-0}" = "1" ]; then
219+
expect_status /seed 200
220+
else
221+
expect_status /seed 404
222+
fi
223+
224+
echo
225+
if [ "$failures" -eq 0 ]; then
226+
echo "All checks passed."
227+
exit 0
228+
fi
229+
230+
echo "$failures check(s) failed."
231+
exit 1

0 commit comments

Comments
 (0)