Skip to content

Commit a06ae5a

Browse files
mpstatonclaude
mpstaton
and
claude
committed
ci(deploy-watch): a watchdog for the half of the gap CI structurally cannot see
ci.yml catches a broken build before merge. Nothing caught a deploy that failed after it, and that is not an oversight in monitoring — it is invisible by construction. A failed Railway deploy leaves the previously-built container serving, the health check passing and the domain answering 200. A broken deploy and a healthy one are externally identical. The only record is a red row in a dashboard, and dashboards are pull-based: somebody has to decide to look. Nobody did, for twelve days. Polls every service's latest deployment every 15 minutes and then SHOUTS — the run fails, and a GitHub issue opens on failure and closes on recovery, so the signal lands in the same work trail as everything else rather than in a tab. Three options were weighed and the reasoning is in the context-v issue. Railway native notifications were rejected because the config would live in a dashboard, invisible to the repo and un-reviewable. A Railway webhook has better latency but needs a hosted receiver to translate its payload into a GitHub event. The scheduled poll is self-contained, versioned, and reviewable, and its only weakness — latency up to one interval — is irrelevant against a failure mode that lasted twelve days. Authenticates with a PROJECT token, scoped to one environment of one project. An account token would hand this workflow every project in the workspace. Note project tokens use the `Project-Access-Token` header, not `Authorization: Bearer`. The workflow resolves its own environmentId from the token, so no Railway IDs are committed and repointing it is a secret swap. Transient states (BUILDING, DEPLOYING, QUEUED, INITIALIZING) are deliberately not failures — a run landing mid-deploy must stay quiet, or the watchdog becomes noise and stops being read. Verified before committing: the GraphQL query and both jq/awk filters were run against the live production environment (11 services, all SUCCESS), and the failure branch was exercised with synthetic FAILED/CRASHED/BUILDING rows to confirm it catches the first two and ignores the rest. Requires one manual step that cannot be automated from here: minting the project token and adding it as the RAILWAY_TOKEN repo secret. Files changed: - .github/workflows/deploy-watch.yml (new) - context-v/issues/A-Failed-Deploy-Is-Silent-Nothing-Watches-Production-After-Merge.md (new) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZJvZWco3z7SR2dEhFqEjA
1 parent 14f82d2 commit a06ae5a

2 files changed

Lines changed: 290 additions & 0 deletions

File tree

‎.github/workflows/deploy-watch.yml‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# Deploy watchdog — the half of the twelve-day gap that CI structurally cannot see.
2+
#
3+
# ci.yml catches a broken build BEFORE merge. Nothing catches a deploy that
4+
# fails AFTER it, because a failed Railway deploy is invisible from outside:
5+
# the previously-built container keeps serving, the health check keeps passing,
6+
# and the domain keeps answering 200. A broken deploy and a healthy one are
7+
# externally identical. The only record is a red row in a dashboard, and
8+
# dashboards are pull-based — somebody has to decide to look. Nobody did, for
9+
# twelve days.
10+
#
11+
# So this polls and then SHOUTS: a failed run, plus a GitHub issue that opens on
12+
# failure and closes on recovery.
13+
#
14+
# See context-v/issues/A-Failed-Deploy-Is-Silent-Nothing-Watches-Production-After-Merge.md
15+
name: Deploy watch
16+
17+
on:
18+
schedule:
19+
# Every 15 minutes. Two API calls per run = 192 requests/day, comfortable
20+
# against Railway's 1000/hour Hobby limit. Do not tighten below ~10 min
21+
# without checking the plan's rate limit.
22+
- cron: "*/15 * * * *"
23+
workflow_dispatch:
24+
25+
# Never let two watchdog runs race on the same issue.
26+
concurrency:
27+
group: deploy-watch
28+
cancel-in-progress: false
29+
30+
permissions:
31+
contents: read
32+
issues: write
33+
34+
jobs:
35+
check:
36+
name: Railway deploy status
37+
runs-on: ubuntu-latest
38+
steps:
39+
- name: Query Railway for every service's latest deployment
40+
id: check
41+
env:
42+
# A PROJECT token — scoped to one environment of one project, and the
43+
# least-privilege choice: an account token would hand this workflow
44+
# every project in the workspace. Project tokens authenticate with the
45+
# `Project-Access-Token` header, NOT `Authorization: Bearer`.
46+
RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }}
47+
run: |
48+
set -euo pipefail
49+
50+
if [ -z "${RAILWAY_TOKEN:-}" ]; then
51+
echo "::error::RAILWAY_TOKEN secret is not set. See the context-v issue for how to mint a project token."
52+
exit 1
53+
fi
54+
55+
api() {
56+
curl -sSf -X POST https://backboard.railway.com/graphql/v2 \
57+
-H "Project-Access-Token: ${RAILWAY_TOKEN}" \
58+
-H "Content-Type: application/json" \
59+
-d "$1"
60+
}
61+
62+
# The token knows which environment it belongs to, so no Railway IDs
63+
# are committed here. Swapping the secret repoints the whole workflow.
64+
ENV_ID=$(api '{"query":"query { projectToken { environmentId } }"}' \
65+
| jq -r '.data.projectToken.environmentId')
66+
67+
if [ -z "$ENV_ID" ] || [ "$ENV_ID" = "null" ]; then
68+
echo "::error::Could not resolve environmentId from the token. Is it a PROJECT token?"
69+
exit 1
70+
fi
71+
72+
api "$(jq -nc --arg id "$ENV_ID" '{
73+
query: "query($id: String!) { environment(id: $id) { serviceInstances { edges { node { serviceName latestDeployment { status createdAt } } } } } }",
74+
variables: {id: $id}
75+
}')" > status.json
76+
77+
# FAILED/CRASHED are the states that mean "this did not ship."
78+
# Transient states (BUILDING, DEPLOYING, INITIALIZING, QUEUED) are not
79+
# failures — a run that happens to land mid-deploy must stay quiet.
80+
jq -r '
81+
.data.environment.serviceInstances.edges[]
82+
| .node
83+
| "\(.serviceName)\t\(.latestDeployment.status // "NONE")\t\(.latestDeployment.createdAt // "-")"
84+
' status.json | sort > all.tsv
85+
86+
echo "### Latest deployment per service" >> "$GITHUB_STEP_SUMMARY"
87+
echo '```' >> "$GITHUB_STEP_SUMMARY"
88+
cat all.tsv >> "$GITHUB_STEP_SUMMARY"
89+
echo '```' >> "$GITHUB_STEP_SUMMARY"
90+
cat all.tsv
91+
92+
awk -F'\t' '$2=="FAILED" || $2=="CRASHED"' all.tsv > bad.tsv || true
93+
94+
if [ -s bad.tsv ]; then
95+
{
96+
echo "broken=true"
97+
echo "detail<<EOF"
98+
cat bad.tsv
99+
echo "EOF"
100+
} >> "$GITHUB_OUTPUT"
101+
else
102+
echo "broken=false" >> "$GITHUB_OUTPUT"
103+
fi
104+
105+
- name: Open or update the failure issue
106+
if: steps.check.outputs.broken == 'true'
107+
env:
108+
GH_TOKEN: ${{ github.token }}
109+
DETAIL: ${{ steps.check.outputs.detail }}
110+
run: |
111+
set -euo pipefail
112+
BODY=$(printf '%s\n\n```\n%s\n```\n\n[Run](%s/%s/actions/runs/%s) · [Why this watchdog exists](%s/%s/blob/rebuild/turbo-rsbuild/context-v/issues/A-Failed-Deploy-Is-Silent-Nothing-Watches-Production-After-Merge.md)\n' \
113+
"One or more Railway services report a failed latest deployment. The site may still be serving an older container, which is exactly why this is easy to miss." \
114+
"$DETAIL" \
115+
"${GITHUB_SERVER_URL}" "${GITHUB_REPOSITORY}" "${GITHUB_RUN_ID}" \
116+
"${GITHUB_SERVER_URL}" "${GITHUB_REPOSITORY}")
117+
118+
EXISTING=$(gh issue list --label deploy-failure --state open --limit 1 --json number -q '.[0].number // empty')
119+
120+
if [ -n "$EXISTING" ]; then
121+
gh issue comment "$EXISTING" --body "$BODY"
122+
echo "Updated issue #$EXISTING"
123+
else
124+
gh issue create \
125+
--title "Deploy failing on Railway (production)" \
126+
--label deploy-failure \
127+
--body "$BODY"
128+
fi
129+
130+
- name: Close the failure issue on recovery
131+
if: steps.check.outputs.broken == 'false'
132+
env:
133+
GH_TOKEN: ${{ github.token }}
134+
run: |
135+
set -euo pipefail
136+
EXISTING=$(gh issue list --label deploy-failure --state open --limit 1 --json number -q '.[0].number // empty')
137+
if [ -n "$EXISTING" ]; then
138+
gh issue close "$EXISTING" \
139+
--comment "Every service reports a healthy latest deployment as of run ${GITHUB_RUN_ID}. Closing automatically."
140+
fi
141+
142+
- name: Fail the run so the red mark is visible
143+
if: steps.check.outputs.broken == 'true'
144+
run: |
145+
echo "::error::Failed Railway deployments:"
146+
printf '%s\n' "${{ steps.check.outputs.detail }}"
147+
exit 1

‎context-v/issues/A-Failed-Deploy-Is-Silent-Nothing-Watches-Production-After-Merge.md‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
title: "A failed deploy is silent, so nothing watches production after merge"
3+
lede: "CI now catches a broken build before merge. Nothing catches a deploy that fails after it — the old container keeps serving, the health check keeps passing, and the only record is a red row in a dashboard nobody has open. That is how twelve days passed unnoticed."
4+
date_created: 2026-08-15
5+
date_modified: 2026-08-15
6+
authors:
7+
- Michael Staton
8+
augmented_with:
9+
- Claude Code on Claude Opus 5
10+
semantic_version: 0.0.1.0
11+
tags:
12+
- Issue
13+
- Augment-It
14+
- Deployment
15+
- Railway
16+
- CI-CD
17+
- Observability
18+
status: Open
19+
---
20+
21+
# A failed deploy is silent, so nothing watches production after merge
22+
23+
## Why Care?
24+
25+
On 2026-08-15 we found that every frontend Docker build had been failing since
26+
2026-08-03 — twelve days — on a single missing `COPY tsconfig.base.json`. The
27+
same investigation found five of ten services deploying from an abandoned
28+
branch (`feature/workspace-auth`, tipped 2026-07-28), which meant production had
29+
been running three-week-old code for a month.
30+
31+
Neither was noticed, and neither was *noticeable*. That is the actual defect.
32+
33+
A failed Railway deploy does not take the site down. The previously-built
34+
container keeps serving, the health check keeps passing, the domain keeps
35+
answering 200. **A broken deploy and a healthy one are externally
36+
indistinguishable.** The only difference is a red row in a dashboard, and
37+
dashboards are pull-based — somebody has to decide to look.
38+
39+
`.github/workflows/ci.yml` (shipped 2026-08-15) closes the *pre-merge* half:
40+
sixteen images build on every pull request, so the class of bug that caused the
41+
outage now fails loudly and early. It structurally cannot close the other half.
42+
A PR workflow runs before the merge; the deploy happens after it, on Railway's
43+
infrastructure, and nothing reports back.
44+
45+
## The gap, precisely
46+
47+
| Moment | Watched by | Fails loudly? |
48+
|---|---|---|
49+
| Code authored | nothing | — |
50+
| Pull request opened | `ci.yml` — 16 image builds + unit groups | ✅ since 2026-08-15 |
51+
| Merged to trunk | `ci.yml` on push | ✅ |
52+
| **Railway builds the image** | **nothing** | ❌ |
53+
| **Railway starts the container** | **nothing** | ❌ |
54+
| Container serving stale code | nothing | ❌ |
55+
56+
Rows 4 and 5 are this issue. Row 6 is harder and deliberately out of scope
57+
(see *Not in scope*).
58+
59+
## Why the obvious answers do not work
60+
61+
**"Add a health check."** Railway already has one and it passed throughout the
62+
twelve days. It was checking the *old, healthy* container. A health check
63+
answers "is something serving?", never "is it serving what we merged?"
64+
65+
**"Watch the site."** Same failure. `augment.didi.sh` returned 200 the entire
66+
time. Uptime monitoring cannot see staleness.
67+
68+
**"Look at the dashboard."** This is the current design, and it is what failed.
69+
Any fix that still requires someone to remember to look has not fixed anything.
70+
71+
**"CI can check it."** A pull-request workflow finishes before Railway starts
72+
building. It has nothing to report on yet.
73+
74+
The property we need is **push, not pull**: something must interrupt us when a
75+
deploy ends badly.
76+
77+
## Decision — poll the Railway API from a scheduled workflow
78+
79+
Three options were considered.
80+
81+
| Option | Mechanism | Verdict |
82+
|---|---|---|
83+
| **A. Railway native notifications** | Project settings → Slack/Discord/webhook | Rejected for now — no Slack/Discord surface is wired for this tree, and the config lives in a dashboard, so it is invisible to the repo and un-reviewable |
84+
| **B. Railway webhook → `repository_dispatch`** | Railway POSTs on deploy status change | Best latency, but needs a public receiver to translate Railway's payload into a GitHub event. Real infrastructure to host and maintain |
85+
| **C. Scheduled GitHub Actions poll** ✅ | Cron workflow queries the Railway GraphQL API | Chosen |
86+
87+
**C wins on this tree's constraints.** It is self-contained in the repo, so the
88+
config is versioned and reviewable rather than buried in a dashboard. It needs
89+
no hosted receiver. And it can open a GitHub issue on failure, which is already
90+
the house convention for a visible work trail.
91+
92+
Its weakness is latency — up to the poll interval. That is acceptable: the
93+
failure mode being fixed lasted twelve days, so detecting within fifteen minutes
94+
is not the constraint that matters.
95+
96+
### Shape
97+
98+
- `.github/workflows/deploy-watch.yml`, `schedule` every 15 minutes plus
99+
`workflow_dispatch` for manual runs.
100+
- One GraphQL call for every service's latest deployment status:
101+
`environment(id:) { serviceInstances { … latestDeployment { status } } }`.
102+
- `FAILED` or `CRASHED` on any service → the run fails **and** a GitHub issue is
103+
opened (or an existing one updated). Recovery closes it.
104+
- Auth by **project token**, scoped to a single environment of a single project.
105+
It uses the `Project-Access-Token` header rather than `Authorization: Bearer`,
106+
and it is the least-privilege choice — an account token would grant the
107+
workflow every project in the workspace.
108+
- The workflow derives `projectId` / `environmentId` from the token itself via
109+
`query { projectToken { projectId environmentId } }`, so **no Railway IDs are
110+
committed to the repo** and the file is portable to another environment by
111+
swapping the secret alone.
112+
113+
### Rate limits
114+
115+
Railway allows 100 requests/hour on Free, 1000 on Hobby, 10000 on Pro. At two
116+
calls per run, every 15 minutes, this costs **192 requests/day** — comfortable
117+
on Hobby and above, and it should not be tightened below ~10 minutes without
118+
checking the plan.
119+
120+
## Not in scope
121+
122+
**Detecting a *stale but green* deploy** — a service whose last deployment
123+
succeeded but which is not running trunk. That is the `feature/workspace-auth`
124+
failure, and it is a different check: compare each service's deployed commit
125+
against the trunk's `HEAD`. Worth doing, deliberately separated so this issue
126+
ships. Both trigger branches and the deployed SHA are readable from the same
127+
API.
128+
129+
**Group I (e2e) in CI** — needs NATS + SurrealDB service containers. Tracked
130+
separately.
131+
132+
## Done when
133+
134+
1. `deploy-watch.yml` exists and runs green against a healthy environment.
135+
2. A deliberately failed deploy produces a GitHub issue within one poll interval.
136+
3. Recovery closes the issue automatically.
137+
4. `DEPLOYMENT.md` documents the token, its scope, and how to rotate it.
138+
139+
## Related
140+
141+
- `changelog/2026-08-15_01_Every-Frontend-Deploy-Had-Been-Failing-For-Twelve-Days-On-One-Missing-COPY.md` — the outage that surfaced this
142+
- [[Rename-Strategy-Curator-To-Corpora-Curator]] — the work in progress when it was found
143+
- [[Move-Remaining-Remotes-To-Remote-Hosting-Prod-Falls-Back-To-Localhost]] — the earlier "prod silently falls back" issue, same family

0 commit comments

Comments
 (0)