fix(ha): ride out a Home Assistant restart instead of stopping after 10 errors

[chalfontchubby]

Opened by Claude on behalf of @chalfontchubby.

Fixes #5437
Fixes #5354
Addresses the start-up half of #5134 (a companion change for the 10 minute wait is separate).

Problem

A Home Assistant restart takes minutes to serve its API again, but Predbat treated the outage as fatal in several places that each trip within seconds:

• HAInterface.api_call() counted every timeout, connection error and non-JSON reply (a 502/404 HTML page) in a counter with no time decay, and stopped the whole interface at 10.
• The Web Socket loop stopped after 10 failed reconnects, about 50 seconds.
• HAInterface.initialize() made one /api/services check and raised on any failure, leaving the HA interface missing for the whole run.

The reporter's log on #5437 shows all three in sequence: Web Socket exception in startup: 502, Too many API errors, stopping, Cannot initialise Home Assistant Interface.

Change

• New OutageTimer (ha.py). An outage is fatal only when failures run unbroken for HA_OUTAGE_LIMIT_SECONDS (15 minutes); any good reply ends it, and failures more than HA_OUTAGE_GAP_SECONDS (10 minutes) apart are separate outages. REST calls and the Web Socket each have their own timer, so one working cannot hide the other being down.
• initialize() retries /api/services with backoff (5s doubling to 30s) for the same period before raising.
• Counted as an outage: timeouts, connection errors, non-JSON replies, other requests errors (e.g. a reset part way through a reply), and a gateway status even with a JSON body: 502/503/504, plus Cloudflare's origin-down 520-524 and 530 (HA_RESTARTING_STATUSES, shared by REST and the Web Socket).
• initialize() also keeps retrying on a gateway status with a JSON body, rather than taking the gateway's error body as Home Assistant being ready.
• Not an outage either: a malformed ha_url/ha_key (InvalidURL, MissingSchema, InvalidSchema, InvalidHeader, URLRequired), a redirect loop (TooManyRedirects) or a body that cannot be encoded (InvalidJSONError, e.g. a NaN attribute). initialize() fails at once on these, naming the error; at runtime the call is logged and dropped without touching the outage timer.
• Not an outage: a 401. initialize() fails at once on it (waiting will not fix a bad key), and 10 in a row at runtime is still fatal. A Web Socket handshake refused with any other status (404, 401, 500, 501, Cloudflare's 525/526 certificate failures...) is still counted and stops after 10. Supervisor calls (core=False) no longer count against Home Assistant.
• tools/debug-journal.md: the "Too many API errors" row now describes the new behaviour (including what is still counted to 10), cites this PR, and keeps the old mechanism for older logs; the GH#5134 start-up stall in the component-health row is framed as pre-fix behaviour. Main's GH#5437 paragraph (one outage walking both this mechanism and Failed HAInterface init blocks startup for 10 minutes in wait_api_started, so an HA restart costs ~12.5 min downtime #5134's) is merged in and framed as pre-fix behaviour.

Tests

New and updated tests in test_hainterface_api.py, _websocket.py, _lifecycle.py (fake clock via patch("ha.time", ...), so asyncio and threading keep the real one). Covered: a burst of failures is tolerated, fatal exactly at the limit, success ends an outage, a long gap starts a new one, REST success does not hide a down Web Socket, auth_ok ends the Web Socket outage, retries then success (including a JSON-bodied 503 during the initialize() retries), gives up after the limit, 401 fails fast and 10 in a row is fatal, gateway handshakes (502-504, 520, 530) versus counted ones (404, 401, 500, 501, 526), a JSON-bodied 503/530 on REST, and errors that are not an outage (malformed URL/header, redirect loop, unencodable body) raise from initialize(), are dropped at runtime, and never start an outage. ./run_all --quick and pre-commit pass. I checked the main new tests fail if the fix they cover is removed.

Known trade-offs

• initialize() blocks start-up for up to 15 minutes while Home Assistant is unreachable (longer if each attempt hangs to its 300s timeout), so a wrong ha_url that still connects now takes that long to report. Components started after the HA interface, including the web component, wait too, so the UI and /api/ping are unreachable for that time. A degraded start where the Web Socket loop does the waiting would avoid that, but is a bigger change.
• A Web Socket that authenticates and is then dropped repeatedly is never fatal (same as before).
• A plain non-JSON 404/403 reply counts as an outage.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The recovery logic matches the stated requirements and is covered across relevant failure modes.

0 open findings

What changed in this PR

Improves resilience during Home Assistant restarts by replacing rapid failure counters with duration-based outage handling.

Changes:

• Adds independent REST and WebSocket outage timers.
• Retries startup API checks with bounded backoff.
• Expands tests and updates troubleshooting guidance.

File	Description
apps/​predbat/​ha.py	Implements outage tracking, retries, and error classification.
apps/​predbat/​tests/​test_hainterface_api.py	Tests REST and initialization recovery behavior.
apps/​predbat/​tests/​test_hainterface_websocket.py	Tests WebSocket outage and handshake handling.
apps/​predbat/​tests/​test_hainterface_lifecycle.py	Adapts lifecycle tests for startup retries.
apps/​predbat/​tests/​test_hainterface_common.py	Adds deterministic fake-time support.
tools/​debug-journal.md	Documents current and legacy outage behavior.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.