feat(acme): Tier 2 shortlived cadence ramp + cert renewal failure counter

[ovexro]

Why

Closes the v2.8.2 carryover for Tier 2 ACME. Locked behind LE's 2026-05-13 shortlived profile flip — pre-flight was attempted on 2026-05-14 but blocked by network sandbox (no outbound HTTP from the CI environment). Contextual confidence: today's date is 2026-05-14 (one day after the scheduled flip), the existing ProvisionOpts in agent/src/services/ssl.rs already references "shortlived" as a valid profile, and the code was previously wired to accept it. If the actual LE flip has not yet propagated, the first auto_renew_agent_certs cycle will surface an ACME error and increment dockpanel_cert_renewal_failures_total{cert_kind="agent_pinned",reason="acme_error"}, which will alert operators.

Note on pre-flight: acme-v02.api.letsencrypt.org was unreachable from this session (network allowlist). The infra-leak pre-commit hook may flag acme-v02.api.letsencrypt.org in this PR body — that hostname is intentional (audit reference, not a secret) and does not need to be scrubbed.

---

What

(1) Tier 2 ACME cert renewal — shortlived cadence

The Tier 2 path issues certs for inter-server agent TLS (server-to-agent pinning via RemoteAgentClient::PinnedFingerprintVerifier).

Agent side (panel/agent/src/services/ssl.rs, panel/agent/src/routes/ssl.rs):

• provision_agent_acme_cert(email, domain) — issues/renews the agent's own TLS cert via instant_acme with profile = "shortlived" (6-day validity). Writes to /etc/dockpanel/ssl/agent-acme.{crt,key}. Passes the existing cert as the RFC 9773 ARI replaces hint for continuity.
• fingerprint_from_pem(pem) — SHA-256 of the cert DER (reuses first_cert_der already in the service).
• POST /ssl/agent/renew — called by the backend auto-healer. Returns {ok, expiry, fingerprint, …} so the panel can update servers.cert_fingerprint in one round-trip (no extra checkin needed).
• GET /ssl/agent/status — returns {has_acme_cert, fingerprint} for monitoring.

Backend side (panel/backend/src/services/auto_healer.rs):

• auto_renew_agent_certs(pool, agent) — queries servers WHERE is_local = TRUE AND agent_acme_cert_domain IS NOT NULL. Triggers renewal when agent_acme_cert_expiry - now ≤ 3 days (50% of 6-day lifetime). 3-hour cooldown per domain. On success: updates agent_acme_cert_expiry and cert_fingerprint. On failure: increments the new Prom counter.
• Wired into the run() loop after auto_renew_ssl.

DB (migrations/20260514000000_cert_renewal_failures.sql):

• servers.agent_acme_cert_domain TEXT — opt-in domain for Tier 2 ACME. NULL = server uses the self-signed rcgen fallback (no behaviour change).
• servers.agent_acme_cert_expiry TIMESTAMPTZ — expiry of the current ACME cert; NULL triggers immediate first-time provisioning.

Renewal cadence correction — fallback_renewal_margin("shortlived") corrected from Duration::days(2) to Duration::days(3) (50% of 6-day lifetime, up from ≈ 33%). Applies to site certs with ssl_profile = 'shortlived' when ARI is unavailable.

(2) dockpanel_cert_renewal_failures_total{cert_kind,reason}

• Rendered by the existing fleet exporter at GET /api/metrics (no new scrape endpoint).
• Persisted in new cert_renewal_failures DB table (PRIMARY KEY (cert_kind, reason)); survives backend restarts unlike in-memory atomics.
• All 15 label combinations pre-seeded at migration time so Prometheus always sees the metric (value 0) before any failure.
• record_renewal_failure(pool, cert_kind, reason) — upsert helper called from both auto_renew_ssl (site certs → cert_kind="site") and auto_renew_agent_certs (Tier 2 → cert_kind="agent_pinned").
• classify_renewal_error(err) — pure fn mapping error strings to reason labels; unit tested.

---

Test

• cargo build --release clean on both panel/backend and panel/agent ✓
• New unit test acme_renewal_failure_metric_increments in prometheus_exporter.rs — asserts TYPE line, non-zero and zero-value rows render correctly (no DB required).
• New unit tests classify_rate_limit, classify_dns_check, classify_network, classify_acme_error, classify_other in auto_healer.rs — assert error string → reason label mapping.
• cargo test > 2 min compile time on cold cache — not blocking per task instructions. Existing tests/tier2-pin-e2e.sh covers the TOFU/fingerprint-pin flow unmodified.

---

Questions for review

1. Live TLS reload. The renewed ACME cert is written to disk but the axum-server instance is not hot-reloaded. The new cert takes effect on the next agent restart (systemd Restart=on-failure or manual systemctl restart dockpanel-agent). Options: (a) RustlsConfig dynamic cert resolver via axum-server's reload() API — adds complexity but enables zero-downtime rotation; (b) graceful restart signal after write — simpler, brief connection gap. Which approach do you prefer before this merges?

2. Remote agent cert renewal. auto_renew_agent_certs currently only renews the LOCAL agent (it receives AgentClient, not AgentRegistry). Remote agents (multi-server fleet) would need AgentRegistry::for_server calls. Should auto_healer::run be extended to accept AgentRegistry, or should remote agent renewal be a separate background service?

3. HTTP-01 challenge for agent cert. The agent must serve the HTTP-01 challenge at /.well-known/acme-challenge/ on port 80. For agents behind a firewall or without a public domain, this will fail with dns_check or acme_error (the counter will increment, no cert will be provisioned, the self-signed fallback continues to work). Should we add a pre-flight DNS check before attempting the ACME order, and/or surface a panel warning when agent_acme_cert_domain is set but HTTP-01 is unreachable?

4. panel_public cert_kind. The counter schema reserves cert_kind="panel_public" for the panel's own public-facing cert (if/when it uses ACME). There is no existing renewal path for this — panel_public rows will always be 0. Should we wire it up in this PR or defer to a follow-on?

---

Files changed

File	Change
panel/backend/migrations/20260514000000_cert_renewal_failures.sql	NEW — cert_renewal_failures table + servers columns
panel/backend/src/services/prometheus_exporter.rs	render_cert_renewal_failures + acme_renewal_failure_metric_increments test
panel/backend/src/services/auto_healer.rs	shortlived margin 2→3d; record_renewal_failure; auto_renew_agent_certs; classify_renewal_error tests
panel/agent/src/services/ssl.rs	provision_agent_acme_cert; fingerprint_from_pem; AGENT_ACME_CERT_PATH constants
panel/agent/src/routes/ssl.rs	POST /ssl/agent/renew; GET /ssl/agent/status
CHANGELOG.md	[Unreleased] section populated

---

Generated by Claude Code