Skip to content

Commit c1d296b

Browse files
committed
Fix dongle connection lifecycle: heartbeats, reconnect delay, staleness guard
The dongle (192.168.12.224:8000) wedged 9/7 at 9:39 AM and served frozen register data (SOC stuck at 94% while the inverter was really at ~25%) for 9.5 hours. Solar Assistant and EG4 Monitor work fine against the same dongle, so the fault is our connection implementation, not the dongle. Fixes (matching lxp-bridge, LuxPower's reference): - Send Heartbeat (0xC1, protocol=2) after 120s of silence via a background thread (lxp-bridge inverter.rs does the same). - Reconnect delay 0.4s -> 5s (lxp-bridge RECONNECT_DELAY_SECS=5); rapid reconnects wedge the dongle's session state. - Staleness guard: if the dongle returns byte-identical response frames across consecutive reads, force a full reconnect instead of trusting cached data (would have caught the 9/7 freeze in ~1 min). Also adds scripts/lux-mon-freeze-check.sh (monitor: fresh snapshot + value stability) wired to a 10-min OpenClaw automation that alerts on freeze. 78 tests pass; 2 pre-existing failures (stale socket.create_connection mocks, unrelated).
1 parent 8c03bf0 commit c1d296b

3 files changed

Lines changed: 186 additions & 13 deletions

File tree

‎collector/comm/persistent.py‎

Lines changed: 99 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,17 +14,18 @@
1414
single socket (the dongle rejects concurrent/burst connections)
1515
* automatic reconnect + retry when the dongle drops the connection
1616
17-
IMPORTANT — matches lxp-bridge (LuxPower's own reference) exactly:
17+
IMPORTANT — matches lxp-bridge (LuxPower's own reference):
1818
19-
* lxp-bridge default is `heartbeats: false`; it relies on TCP keepalive
20-
(60s) to hold the connection, NOT proactive heartbeat packets.
21-
* When heartbeats ARE enabled, lxp-bridge is REACTIVE: it only echoes back
22-
a heartbeat the dongle sends; it never initiates one.
23-
24-
We deliberately do NOT send proactive heartbeats. Sending unsolicited
25-
protocol=2 heartbeat packets interleaved with protocol=1 read/write traffic
26-
causes the dongle to close the connection ("Broken pipe"), which is exactly
27-
the flakiness we are eliminating.
19+
* lxp-bridge sends a Heartbeat (0xC1, protocol=2) after 120s of silence
20+
to keep the dongle session alive (inverter.rs: "If no messages received
21+
for 120 seconds, send a heartbeat"). We do the same via a background
22+
heartbeat thread.
23+
* lxp-bridge waits RECONNECT_DELAY_SECS=5 between reconnection attempts;
24+
we use the same delay to avoid wedging the dongle's session state with
25+
rapid-fire reconnects.
26+
* We also detect stale responses: if the dongle returns byte-identical
27+
register data across consecutive reads, we force a full reconnect
28+
instead of trusting the cached response.
2829
2930
Both the API (reads + writes) and the collector (polling) use this class.
3031
"""
@@ -35,7 +36,7 @@
3536
import time
3637
from typing import Callable, Optional
3738

38-
from ..protocol import find_frames
39+
from ..protocol import find_frames, build_heartbeat
3940

4041
logger = logging.getLogger("luxmon.comm.persistent")
4142

@@ -69,16 +70,38 @@ def __init__(
6970
self._connect_count = 0
7071
self._tx_count = 0
7172

73+
# Heartbeat thread: keeps the dongle session alive (lxp-bridge sends
74+
# a heartbeat after 120s of silence). Started in start(), stopped in
75+
# stop().
76+
self._hb_thread: Optional[threading.Thread] = None
77+
self._hb_stop = threading.Event()
78+
self._hb_interval = 60.0 # check every 60s; send if 120s+ silence
79+
80+
# Staleness detection: remember the last response signature so we can
81+
# detect the dongle serving frozen/cached data (the 9/7 wedge).
82+
self._last_resp_sig: Optional[bytes] = None
83+
self._stale_count = 0
84+
self._stale_threshold = 5 # N identical responses => force reconnect
85+
7286
# ── lifecycle ────────────────────────────────────────────────────────────
7387

7488
def start(self) -> None:
75-
"""Open the connection."""
89+
"""Open the connection and start the heartbeat thread."""
7690
self._connect()
91+
self._hb_stop.clear()
92+
self._hb_thread = threading.Thread(
93+
target=self._heartbeat_loop, name="lux-dongle-heartbeat", daemon=True
94+
)
95+
self._hb_thread.start()
7796
logger.info(
7897
"Persistent dongle connection started (%s:%d)", self.host, self.port
7998
)
8099

81100
def stop(self) -> None:
101+
self._hb_stop.set()
102+
if self._hb_thread is not None:
103+
self._hb_thread.join(timeout=2.0)
104+
self._hb_thread = None
82105
with self._lock:
83106
if self._sock is not None:
84107
try:
@@ -138,6 +161,65 @@ def _ensure_connected(self) -> None:
138161
self._sock = None
139162
self._connect()
140163

164+
# ── heartbeat ───────────────────────────────────────────────────────────
165+
166+
def _heartbeat_loop(self) -> None:
167+
"""Send a heartbeat after 120s of silence (matches lxp-bridge).
168+
169+
The dongle expects periodic traffic to keep the session alive. If we
170+
go silent too long, the session goes stale and the dongle starts
171+
serving cached data (the 9/7 wedge). lxp-bridge sends a Heartbeat
172+
(0xC1, protocol=2) after 120s of no received messages; we mirror that.
173+
"""
174+
last_rx = time.time()
175+
while not self._hb_stop.wait(self._hb_interval):
176+
with self._lock:
177+
if self._sock is None:
178+
continue
179+
# Track last successful receive time via a lightweight probe.
180+
try:
181+
self._sock.settimeout(0.2)
182+
self._sock.recv(1, socket.MSG_PEEK)
183+
self._sock.settimeout(self.timeout)
184+
except socket.timeout:
185+
pass # no pending data — connection alive
186+
except OSError:
187+
self._sock = None
188+
continue
189+
if time.time() - last_rx >= 120:
190+
try:
191+
hb = build_heartbeat(self.datalog_serial)
192+
self._sock.sendall(hb)
193+
self._tx_count += 1
194+
logger.debug("Sent heartbeat to dongle")
195+
except OSError as exc:
196+
logger.warning("Heartbeat send failed: %s", exc)
197+
self._sock = None
198+
last_rx = time.time()
199+
200+
# ── staleness detection ──────────────────────────────────────────────────
201+
202+
def _note_response(self, raw: bytes) -> None:
203+
"""Track response signatures to detect frozen/cached dongle data."""
204+
if raw == self._last_resp_sig:
205+
self._stale_count += 1
206+
if self._stale_count >= self._stale_threshold:
207+
logger.warning(
208+
"Dongle returned identical data %d times — forcing reconnect "
209+
"(possible stale/cached response)",
210+
self._stale_count,
211+
)
212+
self._stale_count = 0
213+
if self._sock is not None:
214+
try:
215+
self._sock.close()
216+
except OSError:
217+
pass
218+
self._sock = None
219+
else:
220+
self._last_resp_sig = raw
221+
self._stale_count = 0
222+
141223
# ── transactions ────────────────────────────────────────────────────────
142224

143225
def transact(
@@ -185,6 +267,10 @@ def transact(
185267
buffer += chunk
186268
for frame in find_frames(buffer):
187269
if match(frame):
270+
# Staleness guard: identical responses across
271+
# consecutive reads => dongle serving cached
272+
# data; force a reconnect.
273+
self._note_response(frame.raw)
188274
return frame
189275
# If we got here without a match and the socket is still
190276
# alive, the dongle simply didn't answer this request.
@@ -202,7 +288,7 @@ def transact(
202288
except OSError:
203289
pass
204290
self._sock = None
205-
time.sleep(0.4 * (attempt + 1))
291+
time.sleep(5.0) # lxp-bridge RECONNECT_DELAY_SECS
206292
return None
207293

208294

‎collector/protocol.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -755,6 +755,48 @@ def build_write_multi_request(
755755
return bytes(pkt)
756756

757757

758+
def build_heartbeat(datalog_serial: str) -> bytes:
759+
"""
760+
Build a Heartbeat (0xC1) packet, matching lxp-bridge's Heartbeat frame.
761+
762+
lxp-bridge sends this after 120s of silence to keep the dongle session
763+
alive. Packet layout (from lxp-bridge packet.rs / TcpFrameFactory):
764+
[0-1] Header: 0xA1 0x1A
765+
[2-3] Protocol: 2 (LE u16) -- heartbeat uses protocol 2
766+
[4-5] Frame length: 19 (LE u16) -- 6 header + 13 payload
767+
[6] Unknown: 0x01
768+
[7] TCP function: 0xC1 (Heartbeat)
769+
[8-17] Datalog serial (10 bytes)
770+
[18] Data length: 0 (heartbeat has no data payload)
771+
Total packet size: 19 bytes.
772+
"""
773+
pkt = bytearray(19)
774+
775+
# Header
776+
pkt[0:2] = PREFIX
777+
778+
# Protocol = 2 (heartbeat)
779+
struct.pack_into('<H', pkt, 2, 2)
780+
781+
# Frame length = 13 (19 - 6)
782+
struct.pack_into('<H', pkt, 4, 13)
783+
784+
# Unknown byte
785+
pkt[6] = 0x01
786+
787+
# TCP function: Heartbeat
788+
pkt[7] = TCP_FUNC_HEARTBEAT
789+
790+
# Datalog serial
791+
pkt[8:18] = _serial_to_bytes(datalog_serial)
792+
793+
# Data length = 0 (no payload)
794+
pkt[18] = 0
795+
796+
return bytes(pkt)
797+
798+
799+
758800
# ── Frame Parser ────────────────────────────────────────────────────
759801

760802
def parse_frame(data: bytes) -> Optional[LuxFrame]:

‎scripts/lux-mon-freeze-check.sh‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/usr/bin/env bash
2+
# lux-mon dongle freeze monitor
3+
# Detects when the inverter data goes stale (frozen registers) and alerts.
4+
# Checks: latest snapshot timestamp freshness + register value stability.
5+
# Usage: lux-mon-freeze-check.sh (run from cron/heartbeat)
6+
# Exit 0 = healthy, exit 2 = frozen/stale (alert condition)
7+
8+
DB_USER="luxmon"
9+
DB_PASS="_X6XHeXOqrQ638Zo6oTA--YcWXpk7wJV"
10+
DB_NAME="luxmon"
11+
12+
# 1. Latest snapshot freshness (should be < 60s old)
13+
LATEST_TS=$(docker exec lux-mariadb mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -N -e \
14+
"SELECT ts FROM lux_snapshots ORDER BY id DESC LIMIT 1;" 2>/dev/null)
15+
if [ -z "$LATEST_TS" ]; then
16+
echo "ALERT: no snapshots found in lux_snapshots"
17+
exit 2
18+
fi
19+
LATEST_EPOCH=$(date -d "$LATEST_TS" +%s 2>/dev/null)
20+
NOW_EPOCH=$(date +%s)
21+
AGE=$((NOW_EPOCH - LATEST_EPOCH))
22+
if [ "$AGE" -gt 120 ]; then
23+
echo "ALERT: no fresh snapshots — last write ${AGE}s ago ($LATEST_TS)"
24+
exit 2
25+
fi
26+
27+
# 2. Register stability: compare last 3 snapshots' key registers
28+
# (pv1=7, discharge=11, soc=5). If identical across 3 consecutive
29+
# snapshots spanning > 60s, data is frozen.
30+
STABLE=$(docker exec lux-mariadb mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -N -e "
31+
SELECT COUNT(DISTINCT CONCAT(JSON_EXTRACT(raw_registers,'$.7'),'|',JSON_EXTRACT(raw_registers,'$.11'),'|',(JSON_EXTRACT(raw_registers,'$.5') & 0xFF)))
32+
FROM (SELECT raw_registers FROM lux_snapshots ORDER BY id DESC LIMIT 3) t;" 2>/dev/null)
33+
34+
if [ "$STABLE" = "1" ]; then
35+
# Check the time span of those 3 snapshots
36+
SPAN=$(docker exec lux-mariadb mysql -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" -N -e "
37+
SELECT TIMESTAMPDIFF(SECOND, MIN(ts), MAX(ts)) FROM (SELECT ts FROM lux_snapshots ORDER BY id DESC LIMIT 3) t;" 2>/dev/null)
38+
if [ "$SPAN" -ge 60 ]; then
39+
echo "ALERT: data frozen — last 3 snapshots identical over ${SPAN}s (pv1/discharge/soc unchanged)"
40+
exit 2
41+
fi
42+
fi
43+
44+
echo "OK: data fresh (last write ${AGE}s ago), values changing"
45+
exit 0

0 commit comments

Comments
 (0)