Skip to content

Commit e898504

Browse files
Denis YermakouAxonOS-BCI
authored andcommitted
monitor: calibrate staleness detection to the 3h scan cadence
The map stopped moving at 2026-08-28T00:29Z and every signal this repository publishes stayed green for the next seven hours: all seven workflow badges passing, no alert issue, no red run. That is the exact silent-failure mode health.yml was written to prevent, and it did not fire because the monitor is calibrated to a round number instead of to the cadence of the thing it monitors. The arithmetic: the engine publishes every 3h, the monitor runs every 12h ("43 6,18 * * *") against a 12h threshold. Data that freezes just after a check is 11.9h old at the next one — under the limit — and is not reported until the check after that. Worst-case detection latency is ~24h for a pipeline whose whole claim is a 3h refresh. Three changes, no new dependencies: * MAX_AGE_HOURS 12 -> 7, overridable via RADAR_MAX_AGE_HOURS. One dropped scheduled run is normal on GitHub and must not page anyone; two consecutive misses is a stalled pipeline. 7h is 6h of missed scans plus grace for engine runtime and sync lag. * health.yml cron "43 6,18 * * *" -> "43 */3 * * *", still offset from the engine (:17) and the sync (:37). The check reads two small JSON files over HTTPS; running it eight times a day costs seconds on a public repository and cuts worst-case detection from ~24h to ~10h. * sync.yml gains a freshness readout after the contract gate. The gate proves the payload is well-formed; it cannot prove it is recent. A frozen engine keeps serving last night's valid JSON, the sync finds nothing to commit and exits 0. The readout prints the payload age on every run and raises a ::warning:: past the threshold, so a stalled upstream is visible in the run log itself rather than only in an issue that may not open for hours. This does not fix the outage. The sync path is healthy: it pulled the 00:24Z payload, the contract gate passed, and there has been nothing new upstream since. The engine (private) is what stopped publishing. This change is about the second defect the outage exposed — that nothing here said so.
1 parent 50e67bf commit e898504

3 files changed

Lines changed: 40 additions & 3 deletions

File tree

.github/workflows/health.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,12 @@ name: Health monitor
77

88
on:
99
schedule:
10-
- cron: "43 6,18 * * *" # twice a day, offset from the radar cron
10+
# Every 3h, offset from the engine (:17) and the sync (:37). A monitor
11+
# that runs every 12h against a 12h threshold cannot see an outage shorter
12+
# than ~24h — which is how a 7h freeze stayed green with every badge
13+
# passing. The check reads two small JSON files; running it 8x a day costs
14+
# seconds on a public repo and cuts worst-case detection to ~10h.
15+
- cron: "43 */3 * * *"
1116
workflow_dispatch:
1217

1318
permissions:

.github/workflows/sync.yml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,32 @@ jobs:
8181
print("contract gate: schema + feed well-formed")
8282
GATE
8383
84+
- name: Freshness readout
85+
# The contract gate proves the payload is well-formed. It cannot prove
86+
# the payload is recent: a frozen engine keeps serving last night's
87+
# valid JSON, this workflow finds nothing to commit, exits 0, and the
88+
# badge stays green while the public map quietly stops moving. That is
89+
# the failure this repository already paid for once. Read the age out
90+
# loud on every run so a stalled upstream is visible in the run log
91+
# rather than only in an issue the monitor may not open for hours.
92+
run: |
93+
python3 - <<'FRESH'
94+
import json, os
95+
from datetime import datetime, timezone
96+
limit = float(os.environ.get("RADAR_MAX_AGE_HOURS", "7"))
97+
at = json.load(open("data/last_run.json"))["at"]
98+
dt = datetime.fromisoformat(at.replace("Z", "+00:00"))
99+
age = (datetime.now(timezone.utc) - dt).total_seconds() / 3600
100+
line = f"last engine publish {at} — {age:.1f}h old (limit {limit:.0f}h)"
101+
print(line)
102+
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as fh:
103+
fh.write(f"- {'STALE' if age > limit else 'fresh'}: {line}\n")
104+
if age > limit:
105+
print(f"::warning title=Engine data is stale::{line}. The sync "
106+
"path is healthy; the upstream engine has not published. "
107+
"Check the engine repository's Actions quota and schedule.")
108+
FRESH
109+
84110
- name: Redeploy the site with whatever arrived
85111
env:
86112
GH_TOKEN: ${{ github.token }}

scripts/health_check.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@
66
radar is healthy:
77
88
* STALE — status.generated_at older than MAX_AGE_HOURS (scans run every
9-
3 hours; 12h of silence means several consecutive failures).
9+
3 hours, so 7h of silence means two consecutive scans are gone;
10+
one dropped scheduled run is tolerated, two are not). Override
11+
with RADAR_MAX_AGE_HOURS when the upstream cadence changes.
1012
* FAILED — the last recorded run ended with ok=false.
1113
1214
On a problem it opens (or updates) a single, marker-identified alert issue.
@@ -32,7 +34,11 @@
3234
PAGES = f"https://{OWNER.lower()}.github.io/{NAME}/"
3335
API = f"https://api.github.com/repos/{REPO}"
3436
MARKER = "<!-- axonos-radar-health-alert f3a91c2e -->"
35-
MAX_AGE_HOURS = 12
37+
# Calibrated to the cadence of the thing being monitored, not to a round
38+
# number. The engine publishes every 3h; GitHub routinely drops one scheduled
39+
# run, so a single miss must not page anyone. Two consecutive misses is a
40+
# stalled pipeline and has to be visible the same day, not a day later.
41+
MAX_AGE_HOURS = float(os.environ.get("RADAR_MAX_AGE_HOURS", "7"))
3642
BOT_LOGINS = ("github-actions[bot]", "github-actions")
3743

3844
HEADERS = {"Accept": "application/vnd.github+json", "User-Agent": "axonos-radar-health",

0 commit comments

Comments
 (0)