feat: reconnect the live stream with bounded backoff

[KristofferRisa]

Pull Request

Related Issue

Closes #22

Goal

powerctl live should survive a transient network blip instead of exiting, so it can be left running as a monitor. Follows the plan in #22, including the settled decision there: reconnect on by default, --no-reconnect to opt out.

One deviation from the plan, called out below: a small adjacent fix to the Ctrl+C path that the tests exposed.

Changes Made

• internal/api/reconnect.go (new) — ReconnectPolicy, the retry loop, the error taxonomy.
  • Subscribe stays a single-attempt primitive; SubscribeWithReconnect layers retry above it, so the connection code stays testable without a loop threaded through it.
  • Backoff: 5s doubling to a 5-minute ceiling, with equal jitter (half fixed, half random) so several clients dropped by one outage don't retry in lockstep.
  • Budget: at most 10 connection attempts per rolling hour. Tibber allows 20, and a stream that keeps failing must not spend the allowance a manual restart needs. Hitting the cap ends the stream with ErrReconnectBudget.
  • Reset: a connection that delivered data and stayed up a minute returns the backoff to 5s.
  • The budget is checked against the clock as it will read when the wait ends, so we don't pause five minutes before reporting that there was never a slot to wait for.
• internal/api/websocket.go — fatal errors are tagged at the point they're raised (markFatal) rather than matched on message text later. Covers a refused handshake (401/403), connection_error in place of connection_ack, GraphQL error payloads, and handler errors.
• internal/commands/live.go — --no-reconnect; reconnect notices to stderr.
• Docs — README (usage + troubleshooting), ARCHITECTURE (a new Reconnection section carrying the taxonomy table, plus the command/error tables and file trees).

Retryable vs fatal

Getting this wrong is expensive in both directions — dropping a recoverable stream, or burning the hourly budget on a request that can never succeed.

Condition	Behaviour
Read error, abnormal or server-initiated close	Retry
Handshake refused with HTTP 401/403	Fatal, exit 1
Close code 4403 Invalid token	Fatal, exit 1
connection_error instead of connection_ack	Fatal, exit 1
Unknown home ID, other GraphQL errors (#13)	Fatal, exit 1
Ack timeout (10s internal deadline)	Retry
ctx cancelled by SIGINT/SIGTERM	Clean exit 0

The ack timeout is why cancellation is detected by checking ctx in the loop rather than by matching context.DeadlineExceeded on the error — that deadline is internal, and a timed-out ack is exactly the transient case worth retrying.

Adjacent fix, outside the issue's scope

The cancellation test sat at exactly 5.00s, which turned out to be a real bug rather than slow testing: on Ctrl+C the deferred close handshake waits up to five seconds for a reply the server may never send. Subscribe now drops the socket with CloseNow when the context is already cancelled. Pre-existing, two lines, and it directly undermines the headline behaviour here (a monitor you leave running and then interrupt) — but say the word and I'll split it out.

Type of Change

• New feature (non-breaking change that adds functionality)
• Bug fix (non-breaking change that fixes an issue) — the Ctrl+C delay above
• Documentation update

On "non-breaking": the default changes. live previously exited 1 on the first stream error; it now retries transient ones. Anything scripted around that prompt non-zero exit needs --no-reconnect. This was the open question in #22 and was settled there in favour of default-on.

Tasks

Mirroring #22:

• Classify close codes and errors into retryable vs fatal
• Backoff with jitter, capped delay, capped attempts per rolling hour
• Reset backoff after a connection has been healthy for a while
• --no-reconnect flag
• Report attempts on stderr, keeping stdout clean
• Tests added or updated
• README.md / ARCHITECTURE.md updated

Left as scoped out in the issue: heartbeat/ping, backfilling measurements missed during a gap, any change to consumption or the fail-fast HTTP client.

Testing

• Unit tests added/updated
• go test ./... passes locally
• Manual testing completed

Two layers. reconnect_test.go drives the loop on a faked clock — sleeping advances time, so the rolling-hour budget is exercised in microseconds. livestream_test.go runs the whole path over a real socket: an httptest server speaking graphql-transport-ws, dropping the connection mid-stream (CloseNow) and asserting the client reconnects and keeps delivering measurements.

LiveClient gained one unexported endpoint field so tests can point it at that server; NewLiveClient still sets WebSocketEndpoint.

Tests were checked for teeth. Three mutations of the implementation, each caught:

Mutation	Result
retryable ignores the fatal classification	3 tests fail
Hourly budget never trips	3 tests fail
Backoff doesn't grow	4 tests fail

The budget mutation originally hung rather than failing — an uncapped loop spins forever on a faked clock — so the always-failing attempt helper now aborts past a bound. With that, the same mutation fails in 3ms instead of stalling until the go test timeout.

Verified against the real Tibber endpoint, which is the part a mock can't prove — that 4403 is genuinely what the server sends:

$ TIBBER_TOKEN=invalid-token-for-test ./powerctl live --home-id <id>
Connecting to live stream...
Error: Stream error: failed to read ack: failed to get reader: received close frame:
status = StatusCode(4403) and reason = "Invalid token"

exit code: 1   elapsed: 0s   stdout: empty

Fatal, immediate, no retry loop, and nothing on stdout.

make check                                   # fmt, vet, mod verify, race tests
go test -race -count=1 ./internal/api/
./powerctl live --help

Checklist

• gofmt -s -l . reports nothing
• go vet ./... is clean
• Self-review of code completed
• Comments added for non-obvious logic
• Documentation updated
• --format json output remains stable — reconnect notices go to stderr precisely to keep the stdout contract intact
• No new warnings or errors introduced

AI Assistance

• AI tooling was used for part or all of this change

Written with Claude Code. Reviewed, tested and mutation-checked as described above.

Additional Notes

Two things worth a reviewer's attention:

1. The default change is the only behavioural break, and it's deliberate (Reconnect the live stream with bounded backoff #22).
2. The 10/hour cap is a guess. It leaves 10 connections of headroom against Tibber's 20 and is spent over ~20 minutes of continuous failure. If you'd rather it were tunable by flag or config, that's an easy follow-up — I kept the surface to --no-reconnect since that's all Reconnect the live stream with bounded backoff #22 called for.

⚡ Built on gilfoyle with Claude Code