Skip to content

fix(gateway): send each hub command once and wait for its ack - #5231

Merged
springfall2008 merged 11 commits into
mainfrom
fix/gateway-wait-for-ack
Sep 26, 2026
Merged

springfall2008 merged 11 commits into
mainfrom
fix/gateway-wait-for-ack

Conversation

@mgazza

@mgazza mgazza commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

PredBat's generic write_and_poll_* loop checks the read-back every 2 s and, until it matches, calls the gateway's select_event / number_event / switch_event again, up to 10 times. GatewayMQTT published the command on every call. When the inverter's dongle takes several seconds to accept a Modbus write, each re-send queues another full write on the hub. The first write usually lands, but only after PredBat's ~20 s window. By then PredBat has already logged Inverter 0 write to <x> failed, and the hub is still working through the duplicates.

The hub already publishes an ack for every command on predbat/devices/<id>/ack/<command_id>, with ok, the refusal reason, and the values it applied. PredBat never subscribed to it, so it could not tell "slow but succeeding" from "refused".

Solution (gateway component only; the shared inverter.py loop is unchanged)

  • Subscribe to predbat/devices/<id>/ack/+ alongside status and online.
  • Track control writes (slot times, rates, reserve, target SoC, charge/discharge enable) per entity. Once an ack has been matched on the current connection:
    • An identical command (same entity, command and payload) is published at most once per 30 s window (_COMMAND_ACK_WINDOW). This covers waiting for the ack, after an ok ack while telemetry catches up, and after a refusal. The generic loop keeps polling in the meantime, so each write normally sends the command once.
    • A different value is published immediately and replaces the tracked one.
    • A command with no ack after the window is re-sent once, which covers a dropped QoS 0 message. If the re-send is not acked either, acks stop being relied on until the next one arrives.
  • ok ack: the cached slot start/end selects and enable switches are updated from applied, so the read-back matches straight away. Staged or recorded EMS slot endpoints and number entities still wait for telemetry.
  • Refusal: logs Warn: GatewayMQTT: <command> for <serial> refused by hub: <error> and does not re-send that value within the window. The write is still reported as failed, but the hub is no longer hammered.
  • replay rejection: the next attempt goes out immediately under a new id. The command id counter is now seeded from the clock, so ids no longer restart at PBAT1 after a restart. A restart was the main way to hit the hub's replay check, or to match a late ack meant for the previous process.
  • Outcomes are recorded per command id. An ok for any send of a value counts. When one command reaches several units, a refusal from any of them outranks an ok, and it undoes an optimistic cache update by restoring the last telemetry. Every refusal restarts the window.
  • Shared state (the id counter and the tracked-command map) is guarded by a lock, because acks arrive on the MQTT listener's loop while writes come from the engine thread.

Compatibility

  • Receiving acks depends on the broker ACL allowing the PredBat instance token to subscribe to ack/+.
  • If the subscription is denied or fails, the gateway logs one warning and control writes behave exactly as before.
  • The same applies if no ack is ever seen, for example with hub firmware that does not publish acks: nothing is suppressed until an ack has been matched on the connection.

Testing

  • New TestCommandAck cases in apps/predbat/tests/test_gateway.py:
    • slow ack (arrives after 10 s of polling): exactly one publish, then a matching read-back;
    • multi-unit acks: a refusal after an ok restores the cache; a staged first ok does not block a later usable one; refusals restart the window;
    • refusal: no re-send within the window, reason logged, new attempt after the window;
    • no ack: one re-send after the window, then fallback to publishing every call;
    • new value while a command is pending: published immediately, and the stale ack is ignored;
    • ok ack updates enable switches and slot times; staged/recorded/number acks leave the cache alone;
    • an ack for an earlier send of the same value; a refusal after an ok from another unit; the replay re-send;
    • no subscription, and subscribed but no ack yet (unchanged behaviour); failed publish; ack topic routing; denied subscription warns once; id seeding.
  • python3 ../apps/predbat/unit_test.py -t gateway and --quick pass. pre-commit (ruff, black, cspell) passes.

🤖 Generated with Claude Code

mgazza and others added 5 commits September 25, 2026 15:46
The generic write_and_poll loop re-calls the gateway's select/number/switch
handlers every 2 s until the read-back matches, and each call published the
command again. A slow dongle write therefore queued up to ten Modbus writes
on the hub, and PredBat reported writes as failed that the hub was still
working through.

The hub already publishes an ack per command on
predbat/devices/<id>/ack/<command_id>. Subscribe to it and track control
writes per entity: once an ack has been matched on this connection, an
identical command is published once per 30 s window (while waiting for the
ack, after an ok ack, and after a refusal), a different value goes out
immediately, and an unanswered command is re-sent once before acks stop
being relied on. Ok acks update cached slot times and enable switches from
the applied values; refusals are logged with the hub's reason; a "replay"
rejection re-sends under a new id. Without the ack subscription, or before
any ack is seen, behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
… command ids

Review follow-ups:
- keep the recent ids of a tracked value, so an ack for an earlier send
  (before suppression is on) confirms it instead of being ignored;
- a refusal after an ok from another unit is still reported and kept, and
  only the first ok ack updates the cached slot times;
- seed the command id counter from the clock so ids do not restart at
  PBAT1 after a restart (the hub's replay check, and a late ack for the
  previous process's id);
- time the ack window with a monotonic clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
Review follow-ups:
- record ok/refused per id: an ok for any send of a value means it was
  applied, while within one id (several units) a refusal outranks an ok;
- a failed publish drops only its own id, so earlier sends stay matchable;
- derive the slot entities from the entity's own suffix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
Review follow-ups:
- only mark the cache as updated when an ack actually carried usable
  values, so a staged/recorded first ok does not block a later plain one;
- a refusal for the id whose ok updated the cache restores the last
  telemetry instead of leaving the optimistic value in place;
- every refusal restarts the cooldown window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
…refusal

Review follow-ups:
- a fresh attempt for an already-answered value keeps its earlier ids, so a
  late ack from another unit is still logged and acted on; a failed publish
  only removes its own id;
- every refusal for an id is logged and restarts the window, not just the
  first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
@springfall2008 springfall2008 added the BOT_REVIEW Trigger an autotriage label Sep 25, 2026
Comment thread apps/predbat/gateway.py
Comment thread apps/predbat/gateway.py
Comment thread apps/predbat/gateway.py
Comment thread apps/predbat/gateway.py
Comment thread apps/predbat/gateway.py
Comment thread apps/predbat/gateway.py
Comment thread apps/predbat/gateway.py
@springfall2008 springfall2008 removed the BOT_REVIEW Trigger an autotriage label Sep 25, 2026
mgazza and others added 6 commits September 25, 2026 22:09
… on publish errors

Review follow-ups:
- remember every id whose ok updated the cache, across fresh attempts, so a
  late refusal from another unit still restores telemetry;
- a publish error no longer removes an id whose ack already arrived while
  the publish awaited the broker's confirmation;
- a replay rejection is logged as "refused by hub: replay" (and still
  re-sent under a new id).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
Review follow-ups:
- a fresh attempt keeps earlier ids' outcomes (pruned with the ids), so a
  refusal from an earlier attempt still outranks that id's late ok;
- a replay arriving after the command was answered is treated as another
  unit's refusal: logged and restarts the window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
…s the entry

A late replay for an earlier id (e.g. from another unit after a fresh
attempt) is handled as a refusal instead of discarding the current attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
…back on publish errors

Review follow-ups:
- a replay for the newest unanswered id drops the entry unless another send
  in the same attempt already applied the value, in which case only that id
  is forgotten (acks from earlier attempts do not count);
- a failed publish restores the id history as it was, including an id
  pruned to make room.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
…, time replays

Review follow-ups:
- prune the per-attempt id list with the kept ids;
- roll back a failed publish only if no ack of any kind arrived meanwhile
  (an ack counter, not an outcome comparison);
- a replay of the newest id after another send applied the value still
  restarts the refusal window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
A failing publish then never loses an earlier id, even when the history is
full and acks for other ids arrive while it awaits the broker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018BaJjLpuMEtQYfvmTRcFJf
@springfall2008
springfall2008 merged commit d0aa73a into main Sep 26, 2026
2 checks passed
@springfall2008
springfall2008 deleted the fix/gateway-wait-for-ack branch September 26, 2026 08:21
springfall2008 added a commit that referenced this pull request Sep 27, 2026
…9-27

docs(debug-journal): fold in the 2026-09-27 queue slice (15 candidates); correct entries overtaken by the #5126/#5251/#5220/#5231/#5216/#5247/#5256 merge batch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants