Stagger the hourly refreshes and shorten their budget

grab_concurrency is per process, not global, and hourly refresh took the
concurrent process count from three to six - so the worker ceiling went from 3x8
to 6x8. Measured after eight hours: load average 50.89 on two vCPUs, memory at
2615 MB of 3801 with 217 MB free, against ~5 and 1336 MB before. Nothing had
skipped, but that is one bad hour from an OOM kill, which would take a whole run's
unwritten results with it.

Staggering to :00/:20/:40 only works if the budget fits in the gap. At 3300s the
three runs still overlap two and three deep and the peak is unchanged - the
stagger on its own does nothing, which is what I had recommended. At 1100s exactly
one refresh is live at any moment, so the ceiling is 3x8 + 1x8 = 32.

The cost is coverage: roughly a third of the hosts per protocol per hour, so
cycles stretch from ~2.5 to ~7.2 days for https and ~1.6 to ~4.6 for ssh. Still
far short of the months it was heading for when refresh ran daily, and the trade
buys back the headroom that made the box unsafe to leave overnight.

Scan durations held through all of it - no skips in eight hours, ssh at
3375-3410s against a 2737s nominal, so its shortened window absorbed a +638s
overrun that would have breached at the old 3128s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: joocer

terraform/user_data.sh.tftpl

@@ -347,9 +347,20 @@ systemctl enable --now ichnos-web
 # Hourly, not daily, because refresh's cost is fixed by its budget rather than by how
 # many hosts exist, and once daily it was reaching under 2% of the known set per run -
 # a full cycle measured in months while discovery adds tens of thousands of hosts a
-# day. Hourly turns that into roughly two days. It starts at :00 and the scans at :06,
-# so they overlap by design; the scan budgets are sized for that overlap rather than
-# pretending it is not there.
+# day. It overlaps the scans by design; their budgets are sized for that.
+#
+# Staggered :00/:20/:40, and the budget shortened to match, because grab_concurrency is
+# per *process*, not global. Hourly refresh took the concurrent process count from three
+# (the scans) to six, so the worker ceiling went from 3x8 to 6x8 - measured at load
+# average 50 on two vCPUs with 217 MB free, which is one bad hour from an OOM kill that
+# would take a whole run's unwritten results with it.
+#
+# The stagger only works with a budget under 20 minutes. At the previous 55 minutes,
+# runs starting :00/:20/:40 still overlap two and three deep and the peak is unchanged;
+# at 18 minutes exactly one refresh is ever live, so the ceiling is 3x8 + 1x8 = 32.
+# The cost is coverage - roughly a third of the hosts per protocol per hour, so cycles
+# stretch from ~2.5 to ~7 days for https and ~1.6 to ~4.6 for ssh. Still far short of
+# the months it was heading for when this ran daily.
 #
 # Scans start at :06 rather than :00, which is about where the results land rather
 # than about the scanning. A run takes ~3253s, so starting at :06 it finishes around
@@ -364,8 +375,8 @@ cat > /etc/cron.d/ichnos <<CRONEOF
 6 * * * * root flock -n /var/lock/ichnos-scan-${proto}.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos scan --protocol ${proto} --candidates ${budget.candidates} --rate-pps ${budget.rate_pps} --store dynamodb >> /var/log/ichnos/scan-${proto}.log 2>&1'
 %{ endfor ~}
 0 * * * * root flock -n /var/lock/ichnos-refresh-http.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos refresh --protocol http --store dynamodb >> /var/log/ichnos/refresh-http.log 2>&1'
-0 * * * * root flock -n /var/lock/ichnos-refresh-https.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos refresh --protocol https --store dynamodb >> /var/log/ichnos/refresh-https.log 2>&1'
-0 * * * * root flock -n /var/lock/ichnos-refresh-ssh.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos refresh --protocol ssh --store dynamodb >> /var/log/ichnos/refresh-ssh.log 2>&1'
+20 * * * * root flock -n /var/lock/ichnos-refresh-https.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos refresh --protocol https --store dynamodb >> /var/log/ichnos/refresh-https.log 2>&1'
+40 * * * * root flock -n /var/lock/ichnos-refresh-ssh.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos refresh --protocol ssh --store dynamodb >> /var/log/ichnos/refresh-ssh.log 2>&1'
 30 3 * * * root flock -n /var/lock/ichnos-responsive.lock bash -c 'set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos responsive-refresh --store dynamodb >> /var/log/ichnos/responsive.log 2>&1'
 5 * * * * root set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos publish >> /var/log/ichnos/publish.log 2>&1
 0 3 * * 0 root set -a; . /etc/ichnos/env; set +a; /opt/ichnos-venv/bin/ichnos jurisdiction-refresh --source rir --countries "${jurisdiction_countries}" >> /var/log/ichnos/jurisdiction.log 2>&1

terraform/variables.tf

@@ -81,9 +81,9 @@ variable "refresh_rate_per_second" {
 }
 
 variable "refresh_duration_seconds" {
-  description = "Wall-clock budget for one `refresh` run. Makes refresh cost duration x rate rather than scaling with how many hosts discovery has found; coverage becomes a rolling cycle rather than a daily sweep. 0 means unbounded."
+  description = "Wall-clock budget for one `refresh` run. Makes refresh cost duration x rate rather than scaling with how many hosts discovery has found; coverage becomes a rolling cycle rather than a daily sweep. Held under 20 minutes so the three staggered hourly runs (:00/:20/:40 in user_data.sh.tftpl) never overlap each other - grab_concurrency is per process, so concurrent refreshes multiply the worker ceiling, and at 55 minutes they overlapped two and three deep for a measured load average of 50 on two vCPUs. 0 means unbounded."
   type        = number
-  default     = 3300
+  default     = 1100
 }
 
 variable "grab_timeout_seconds" {
@@ -136,14 +136,6 @@ variable "scan_protocol_budgets" {
     rate together changes only how much work happens inside an unchanged window, which
     is not what was short.
 
-    Historic note, since it was got wrong twice: Refresh does not cost the three
-    protocols equally - measured at +23s for http, +38s for https and +349s for ssh -
-    and ssh was already the tightest, so with refresh running hourly it would breach the
-    window every hour. Cutting its candidates while holding the rate at 80pps is what
-    buys margin: dropping both together keeps the nominal pinned at 3128s and changes
-    nothing. At 200000@80pps the nominal is 2503s, so even ssh's worst measured overrun
-    plus refresh lands near 3000s.
-
     650000 was tried and is too much. It was sized by scaling that 40% linearly to an
     expected 65%; measured, it ran at 91.4% average and 96.1% peak, and every run
     overran by ~8% - enough to eat the 472s buffer and make the following tick skip.