|
2 | 2 |
|
3 | 3 | All notable changes to Docker Dash are documented here. |
4 | 4 |
|
| 5 | +## [6.17.2] - 2026-04-22 — "HA Phase 4 — Leader election (multi-replica now safe)" |
| 6 | + |
| 7 | +Cron jobs, Docker event stream, and git polling now run **on the leader replica only** in HA mode. Multi-replica HA deploy finally becomes safe: no more duplicate daily backups, no more concurrent `VACUUM` (DB corruption risk), no more N× GitHub API rate-limit hits from git polling. |
| 8 | + |
| 9 | +### How it works |
| 10 | + |
| 11 | +Redis `SET NX PX` with TTL 30s + heartbeat 10s: |
| 12 | + |
| 13 | +- **Startup**: first call to `cluster.isLeader()` in HA mode lazily starts the election loop. Attempt `SET NX` to claim the `leader` key. Success → become leader. Failure → become reader. Standalone mode: always leader (return `true` without Redis traffic). |
| 14 | +- **Leader heartbeat**: every 10s, extend the lock with `SET XX PX` (refreshes TTL only if we still own it). If extension fails (TTL expired, someone else grabbed it), transition to reader and fire `onBecomeReader` callbacks. |
| 15 | +- **Reader poll**: every 10s, try `SET NX` — on leader death (or graceful `shutdown()`), a reader wins and transitions to leader. |
| 16 | +- **Graceful shutdown**: leader releases the lock proactively via a Lua script that only DELs if we still own it. Another replica picks it up within milliseconds instead of waiting out the 30s TTL. |
| 17 | +- **Internal-reset recovery**: if `_leaderState` is lost (e.g. module reset in tests) while Redis still holds our NODE_ID, `_electOnce` detects this via GET + comparison and re-claims leader without spurious role transition. |
| 18 | + |
| 19 | +### Wiring — what runs on the leader only |
| 20 | + |
| 21 | +**Cron jobs via `_m(name, fn)`** — now leader-aware. Any reader replica calling a `_m`-wrapped job returns immediately (silent skip, no metric increment). Opt-out via `_m(name, fn, { everywhere: true })` for idempotent jobs; none qualify today but the escape hatch exists. |
| 22 | + |
| 23 | +All 13 cron jobs now leader-only: |
| 24 | +`stats-aggregate-1m` · `stats-aggregate-1h` · `alert-evaluate` · `session-mfa-cleanup` · `security-alert-windowed` · `purge-old-data` · `vacuum-db` · `certificate-scan` · `secret-rotation-scan` · `daily-backup` · `schedule-executor` · `s3-backup` · `sandbox-ttl-sweep` |
| 25 | + |
| 26 | +**Docker event stream** — gated via `cluster.onBecomeLeader` / `onBecomeReader` in `src/ws/index.js`. On leader transition: `_startAllEventStreams()` subscribes to Docker for every active host. On reader transition: `_stopAllEventStreams()` destroys all streams. Readers still deliver events to their local clients via Redis pub/sub (shipped in v6.17.1). |
| 27 | + |
| 28 | +**Git polling** — gated via the same callbacks in `src/jobs/index.js`. `gitPolling.startAll()` / `stopAll()` fire on role transition. Previously running per-replica would have N×-multiplied the GitHub API rate-limit hit. |
| 29 | + |
| 30 | +### What still runs on every replica |
| 31 | + |
| 32 | +- **SSH tunnels** (`src/services/ssh-tunnel.js`) — readers need them to serve HTTP reads (container list, stats, inspect). Not gated. Remote hosts see N SSH connections; acceptable for v6.17.2. Future v7.x may proxy read-path SSH through the leader. |
| 33 | +- **Stats service** (`statsService.start()`) — per-replica stats collection feeds local metrics endpoint. Aggregation (which writes to DB) is leader-only via the cron gate. |
| 34 | + |
| 35 | +### Safety checks |
| 36 | + |
| 37 | +- **Standalone completely unaffected.** `cluster.isLeader()` short-circuits to `true` without touching Redis. `onBecomeLeader(fn)` fires `fn` synchronously at registration — cron jobs start immediately. |
| 38 | +- **Rollback safe.** If you unset `DD_MODE` and restart, standalone path takes over. The `leader` key in Redis is orphaned (harmless) and expires via TTL. |
| 39 | +- **Throwing role-transition callbacks don't block siblings.** Each callback runs in its own try/catch. |
| 40 | + |
| 41 | +### Tests — 8 new leader-election tests (879 total) |
| 42 | + |
| 43 | +- `isLeader()` acquires the lock on first call (fresh Redis → become leader) |
| 44 | +- A second "replica" cannot acquire while held (NX returns null) |
| 45 | +- `onBecomeLeader` fires callbacks on role transition |
| 46 | +- `_forceRole` test helper — verifies callback sequence across multiple transitions |
| 47 | +- Idempotent transitions don't fire callbacks twice |
| 48 | +- A throwing callback doesn't prevent siblings from firing |
| 49 | +- Standalone mode: `onBecomeLeader` fires synchronously at registration |
| 50 | +- Standalone mode: `onBecomeReader` never fires |
| 51 | + |
| 52 | +### Tests / Lint |
| 53 | + |
| 54 | +- **879 passing + 4 skipped / 57 suites** (was 871 / 57; +8 Phase 4 tests) |
| 55 | +- Lint: 0 warnings / 0 errors |
| 56 | + |
| 57 | +### Files touched |
| 58 | + |
| 59 | +- `src/services/cluster.js` — +80 LOC: leader election loop, heartbeat, role transitions, callback registration, graceful lock release via Lua DEL-if-owned |
| 60 | +- `src/jobs/index.js` — `_m(name, fn, opts)` leader-aware, gitPolling start/stop wired to role callbacks |
| 61 | +- `src/ws/index.js` — Docker event stream start/stop on role transition |
| 62 | +- `src/__tests__/cluster.test.js` — +8 Phase 4 tests + standalone callback tests |
| 63 | + |
| 64 | +### HA mode v6.17.x complete |
| 65 | + |
| 66 | +v6.17.0 foundation + v6.17.1 pub/sub + v6.17.2 leader election = **multi-replica HA is safe**. v7.0.0 will bring the failover runbook, sticky-session LB docs, and a real multi-replica staging soak before promoting to "production-grade HA". |
| 67 | + |
| 68 | +--- |
| 69 | + |
5 | 70 | ## [6.17.1] - 2026-04-22 — "HA Phase 3 — WebSocket pub/sub via Redis" |
6 | 71 |
|
7 | 72 | Cross-replica WebSocket events now work. User connected to replica A **now receives** events emitted by replica B (alerts, container state changes, log lines) through Redis pub/sub. Before this, multi-replica HA deploys had silent event delivery gaps. |
|
0 commit comments