Feature: Repeated sending for DIRECT-route packets with echo-cancellation (improves delivery, reduces floods)

[usrflo]

Relates to #1342.

Motivation

Direct messages over multi-hop paths fail silently when a single hop misses a packet due to temporary radio interference or channel congestion. The sending node then falls back to a flood — which is both unreliable under load and consumes significant airtime. In my tests most incoming paths with a length of 3 hops couldn't be reused in back traces: a far too low number that should be increased by an error correction.

What this PR does

This branch adds conditional repeated sending for DIRECT-route packets at the repeater level:

• After forwarding a DIRECT packet, a repeater schedules up to max_resend_attempts re-transmissions (default: 2, configurable 0–3 via set max.resend).
• Each pending retransmit is cancelled immediately when the repeater overhears the downstream relay successfully forwarding the same packet (echo-detection via hash comparison in the outbound queue).
• The RX loop was changed to drain all pending packets per loop() call, minimising the race window between echo detection and retransmit timer expiry.
• ACK packets use a dedicated deduplication table (_acks[] in SimpleMeshTables) to keep deduplication cheap (4-byte compare, no SHA-256) and to prevent ACK entries from evicting long-lived flood-packet hashes.

Comparison with PR #2367 (HALO)

This implementation shares the same goal as #2367 — improve direct-path reliability — but uses a different strategy. Both branches were applied to mcsim (including fixes to make the simulator timing-accurate for retransmit scenarios, see simulator commits) and run against the same standardised test scenario.

Test topology: 3 repeaters between Alice and Bob, all links of marginal/bad quality
Test: 10 random seeds × 20 DMs each = 200 deliveries total

Branch	Avg delivered	Avg TX (direct)	Avg collisions
halo-direct-path-retries (#2367)	~15.9 / 20	~420	~15
repeated-sending-2 (this PR)	~19.4 / 20	~187	~3

Key advantages of this approach:

• Higher delivery rate (~97 % vs ~80 % in the test scenario)
• 55 % fewer TX packets — echo-cancellation prevents unnecessary retransmits before they fire
• ~80 % fewer collisions — significantly less channel saturation
• Simpler and more resource-efficient — no SNR-gated retry logic, no neighbor-table lookup; just overhear and cancel

Delivery rate vs. resend attempts (mcsim, retry_showcase topology)

max_resend_attempts	Delivery rate
0 (stock MeshCore)	71 %
1	91 %
2 (default)	95.5 %
3	97 %

A value of 2–3 is a good compromise between reliable delivery and channel efficiency.

CLI

No new commands are needed — max_resend_attempts is already exposed:

get max.resend # current value (default: 2)
set max.resend <0–3> # 0 = disabled

[usrflo]

Some information about the 3 commits:

1. the first commit MESHCORE_SIMULATOR patch contains common changes to use "mcsim" (see https://github.com/Brent-A/mcsim and https://github.com/usrflo/mcsim) or any other simulation that requires simulation hooks. This commit is not required for this feature but is a requirement to run simulated comparisons.
2. the second commit Implemented repeated sending for error correction contains changes to the MeshCore implementation (dev branch)
3. the third commit Added comments to explain changes related to the repeated-sending feature is meant for the code review. This commit might be removed if this pull request is accepted.

[cwichura]

This seems very similar to #2367.

[usrflo]

After mcsim-simulation runs and internal tests on real hardware I released firmware packages named "ufo_0.1" for repeaters, companions and room-servers at https://github.com/usrflo/MeshCore/releases. This firmware is based on MeshCore-dev v1.16.0 with application of this pull request.

It would be great to get some feedback on this feature. @wlockwood?

[usrflo]

To recognize propagated TRACE packages a special detection was required as TRACE packages differ in using path and body fields. In the simulation with a relatively stable chain of repeaters (TOPOLOGY, BEHAVIOR) I got the following results:

Without repeated sending (max.resend 0)
averages over 10 seeds:
sent per run: 8.0
received total per run: 6.2
collisions per run: 0.0
overall delivery rate: 77.5%

With repeated sending (max.resend 2)
averages over 10 seeds:
sent per run: 8.0
received total per run: 8.0
collisions per run: 3.6
overall delivery rate: 100.0%

[usrflo]

After the last bugfix I tested the firmware on the following hardware device chain:
C1 -> R1 -> R2 -> R3 -> C2 with weak links (-80 / -90 / -100 RSSI) between each of these nodes (strongly isolated devices with weak antennas). A successful delivery was counted when C2 sent out an ACK.

Without repeated sending: 81 of 100 packages were delivered
With repeated sending: 96 of 100 packages were delivered 🚀

Side note:
These tests were executed with

1. the noise_floor calculation fix from https://github.com/usrflo/MeshCore/tree/fix/noise-floor-ratchet, see Fix/noise floor ratchet, adapt noise to the real floor with recovery in both directions #2933
2. the non blocking waiting for silence before TX from https://github.com/usrflo/MeshCore/tree/feature/quiet-dwell, see README-quiet-dwell.md

See the according firmware release ufo_0.5 at https://github.com/usrflo/MeshCore/releases

---

To give an idea about the measurement: I used 2 observers to check the details on the two separated locations.
The SNR/RSSI values in the following excerpt are those of the observer. The DUP at the end of the line marks a repeated package sending.

One sample:

13:03:24.523  Δ=  7.508s  Direct·TXT_MSG sender=MeshL1New origin=MeshL1New dest=WL1_C2  path=ND1_R1→BB1_R4→SB1_R2  hop 1/4  SNR= +3.2 RSSI= -72  871CD76C34937AEC ⚠noi
┃ ⚠ INTERFERENCE END    noise      13:03:18  dur=6.5s peak=+40  pkts=0 rx_errs=0
13:03:25.299  Δ=  0.776s  Direct·TXT_MSG sender=ND1_R1 origin=MeshL1New dest=WL1_C2  path=BB1_R4→SB1_R2  hop 2/4  SNR= +9.5 RSSI= -59  871CD76C34937AEC
13:03:26.329  Δ=  1.030s  Direct·TXT_MSG sender=ND1_R1 origin=MeshL1New dest=WL1_C2  path=BB1_R4→SB1_R2  hop 2/4  SNR=+13.2 RSSI= -59  871CD76C34937AEC DUP ⟳1.030s
13:03:26.847  Δ=  0.518s  Direct·TXT_MSG sender=BB1_R4 origin=MeshL1New dest=WL1_C2  path=SB1_R2  hop 3/4  SNR=+13.2 RSSI= -75  871CD76C34937AEC
13:03:27.398  Δ=  0.551s  Direct·TXT_MSG sender=SB1_R2 origin=MeshL1New dest=WL1_C2  path=∅  hop 4/4  SNR=+12.0 RSSI= -49  871CD76C34937AEC
13:03:27.915  Δ=  0.517s  Direct·ACK   sender=direct    path=∅  hop 1/1  SNR=+11.8 RSSI= -76  70E804018D91907B
┃ ⚠ INTERFERENCE START  contention 13:03:33  nf=-108 rssi=-76 margin=+32 (dev_ms=8340913)
  └─ chain MeshL1New → ND1_R1 → BB1_R4 → SB1_R2 → WL1_C2  | 6 RX | SNR +3.2..+13.2 RSSI -76..-49 | TXT_MSG

[usrflo]

Today's merges were a setback; I will try to find the reason. So long this pull request is set back to draft.

[usrflo]

Good numbers again, I fixed the timing issues of the last commits.

The following problems are addressed in the changes of the last days:

• schedule the sending after noise detection to 3 different wait buckets based on the residual path length (mod 3) of the direct message; so multiple nodes nearby don't send at the same time after noise detection
• don't resend packages if the sending queue becomes filled up too much (prevent congestion)
• appropriate timing for package cancellations before a resend. This depends on the airtime and constant wait intervals for the processing on the nodes.

Last results with the test chain Companion1->Rep1->Rep2->Rep3->Companion2, 50 direct messages in a 10 second interval, up to 2 package repetitions (default). Every ACK is counted as success:
a) sending over very weak links (-70 ... -100 RSSI) with an interferer node (an advert every 6 seconds): 46 ACKs
b) sending over very weak links (-70 ... -100 RSSI) without an interferer node: 49 ACKs
c) sending over stable links: 50 ACKs

[usrflo]

Last update: the max wait time limit - used as a window to cancel packages prepared to be resended - needs to be calculated according to the airtime / package size.

[usrflo]

Repeated-sending - current state and the safeguards that keep it stable

A resend is a redundant retransmission of a DIRECT (acknowledged-path) packet, queued by the originator and auto-cancelled when a downstream relay's forward — or the destination's ACK — is overheard. The logic only ever adds a retry that can be called off, and sheds/collisions are guarded as follows.

Safeguards

• Scoped eligibility + attempt cap. Only direct-routed packets still carrying ≥1 relay hash are resendable (there must be a downstream forward that can cancel it). The final hop has no downstream forward, so it gets exactly one resend, cancelled by the returning ACK. Retries are capped at getMaxResendAttempts() (default 2).

• Non-invasive resend LBT. Resends deliberately skip CAD so the radio stays in RX and can still overhear the cancelling forward. Instead isResendChannelActive() gates on isReceivingPacket() || (currentRSSI − noiseFloor ≥ RESEND_INTERFERENCE_MARGIN), margin fixed at 12 dB (as long as the prefs int.threshold discussion goes on). The RSSI-energy term detects an ongoing transmission at any point — preamble or payload — so a long packet is not overlaid by a retry. First sends keep normal CAD carrier-sense, but it's recommended to set cad off as it seems to increase deafness.

• Cancel window sized to packet length / SF. The wait before a resend fires is C0 + K·airtime + margin + (attempt−1)·jitter, under an airtime-proportional ceiling (CAP_BASE + CAP_AIRTIME_X·airtime). The window therefore always outlasts the downstream forward for any length or spreading factor. The earlier flat 1500 ms cap under-covered SF10 long messages, so every long packet fired its resend before the forward could cancel it — a redundant, colliding resend on each one.

• Per-hop busy-recheck bucket (resenders stay out of each other's way). When the channel is busy, deferred resenders retry at (pathHashCount % 3 + 1) × 120 ms instead of all firing on the same loop tick. The hash count drops by exactly 1 per forwarding hop, so any two mutually-in-range chain neighbours (≤2 hops apart) always land in different buckets and never collide on retry; same-bucket pairs are ≥3 hops apart and don't interfere. This is a re-check offset only — the cancel window is not inflated.

• Pool-shedding (queue-overflow protection). A resend shares a packet-pool slot with RX, so it is only queued when the free pool is healthy (freeCount > 6). Under sustained load the redundant retry is dropped rather than exhausting the pool and deafening the node — the primary direct forward has already been transmitted.

---

If you like to test this PR it's recommended to use:

set cad off
set agc.reset.interval 0     # I only tested with 0, so better to do so
set max.resend 2

This PR ist part of the ufo firmware, latest build at time of writing: v0.6

[usrflo]

With the latest checkin statistics on package resending can be viewed, sample:

> get max.resend
> 2, resends 64/173 (36%)

[m0urs]

I have been testing this PR (and their modifications) since a few weeks now, and I confirm that this is a HUGE improvement.

I have two recipients, both 4 Hops away and all 4 repeaters have this PR installed. Sind I have that PR in place, I do have a 100% delivery and ACK rate to both recipients (and vice versa). I never had that before with the standard firmware without that modification.

So I really would like to have this PR included in the next official release.

Thanks @usrflo for that great improvement!

[Alain2019]

Are there reasons to keep the conflicts unsolved?

I see this as a huge improvement to meshcore.

[wlockwood]

Are there reasons to keep the conflicts unsolved?

I see this as a huge improvement to meshcore.

There's a competing approach by mikecarper that solves similar problems in a slightly different way: https://github.com/mikecarper/MeshCore/releases/tag/v1.17.1.5-halo-keymind-cascade-dev-26303793

[Alain2019]

Are there reasons to keep the conflicts unsolved?
I see this as a huge improvement to meshcore.

There's a competing approach by mikecarper that solves similar problems in a slightly different way: https://github.com/mikecarper/MeshCore/releases/tag/v1.17.1.5-halo-keymind-cascade-dev-26303793

Not much info about the mike carper version. As I understand it it's a simple repeat if not heard. This should be used with quite some care with flood messages. A repeater with 32 neighbours. Is it sufficient that only one resend it or all 32?

Is there some more info?

[wlockwood]

I'm not the developer, but there's a lot going on with HALO/Keymind:

• DMs: Keymind doesn’t merely repeat unless it hears any duplicate. It listens for the intended downstream node forwarding the packet—effectively a hop-level ACK—and cancels retries only on confirmed path progress. The final relay sends one extra copy because the recipient produces no forwarding echo.

• Floods: Keymind retries until it hears the packet from a qualifying downstream repeater, not just anyone. In bridge mode it can require echoes from every relevant direction/bucket. Upstream or ignored repeaters don’t prematurely cancel retries.

[Alain2019]

I'm not the developer, but there's a lot going on with HALO/Keymind:

* **DMs**: Keymind doesn’t merely repeat unless it hears any duplicate. It listens for the intended downstream node forwarding the packet—effectively a hop-level ACK—and cancels retries only on confirmed path progress. The final relay sends one extra copy because the recipient produces no forwarding echo.

* **Floods**: Keymind retries until it hears the packet from a qualifying downstream repeater, not just anyone. In bridge mode it can require echoes from every relevant direction/bucket. Upstream or ignored repeaters don’t prematurely cancel retries.

Seems also nice and also bigger than this PR. But it seems completely out of the official code tree. Also I didn't find any information about how it works.

Ideally the two should be merged or better the best of both should be merged into the official code tree.

[wlockwood]

Ideally, yes. I'd love to see retries of some flavor make it into the official firmware. 2 per-hop retries and 2 end-to-end retries makes a huge difference for the usable number of hops a client can communicate over:

In the above table, each cell indicates the number of hops a message can span while maintaining a delivery certainty of 99% assuming a link reliability along the top.

[Alain2019]

Ideally, yes. I'd love to see retries of some flavor make it into the official firmware. 2 per-hop retries and 2 end-to-end retries makes a huge difference for the usable number of hops a client can communicate over: In the above table, each cell indicates the number of hops a message can span while maintaining a delivery certainty of 99% assuming a link reliability along the top.

Yes. There should be a rather good reliability for a rather low nr of hops, lets say 5-6 hops. This is now NOT case. It's simply not reliable enough to do really useful stuff with meshcore.

[wlockwood]

For routed messages, you're not wrong. Flood messages are fairly reliable in a dense network like we have in the PNW.

[usrflo]

Conflict with current dev is resolved - the branch is up to date again.

Referring to HALO/Keymind: I tested that fork side by side (mcsim + hardware); my evaluation is here: #2367 (comment).

For DMs, HALO waits for the intended downstream forward - essentially the same principle used by this PR "repeated-sending": an implicit hop-level ACK, cancelled when the forward is overheard. I don't see a fundamentally better approach there. The gains also remained modest: in mikecarper's own 30-seed mcsim DM test, delivery improved from 150→153 and ACKs from 38→39.

Where I deliberately stopped with the repeated sending:

• No retries on flood. Per-hop flood retries would turn every non-echoing listener into a retransmitter. The 32-neighbour amplification @Alain2019 sketches is real, and as @wlockwood points out, floods are already fairly reliable in dense networks. Repetition is much more useful where a lost packet has exactly one intended next hop - i.e. direct/routed messages.

• Retries are bounded at 2. Long-running hardware tests on a lossy reference chain showed 2 retries to be the sweet spot. Beyond roughly 3 potential retries, packet collisions start to outweigh the gains. There is only one configuration knob: max.resend, defaulting to 2 and allowing 0–3. Keeping the configuration simple is a feature here, not a limitation.

• Resend LBT. The resend LBT (12 dB margin above a median-estimated noise floor, without CAD) also came out of that hardware testing. The underlying noise-floor fix is tracked in Fix/noise floor ratchet, adapt noise to the real floor with recovery in both directions #2933. The RX-focused LBT approach maximizes the likelihood to hear a downstream package repetition while CAD blocked RX too much and decreased deliverability in my tests.

Recap: mcsim improved from 77.5%→100%; the hardware reference chain showed an improvement from 81/100→96/100; and ACKs reached 46–50/50 depending on interference.

If you see the requirement: I'm happy to merge the best of both approaches. What exactly do you see as an advantage of the HALO implementation?

[mikecarper]

No retries on flood. I limit this to hops 0-2; after that it doesn't try. This is the biggest win for user experience actually; like this is the main reason people install the firmware I put out from this branch https://github.com/mikecarper/MeshCore/tree/keymindCascade.

Retries are bounded at 2. We're dealing with a repeaters with 200+ neighbors and 2 tries doesn't work.

Resend LBT. I need to look into this as it might help the situation but the big repeaters can be locked up for 8+ seconds here (the queue depth stays + for the entire time)

[Alain2019]

As far as I can see it I think that @usrflo it's forwarding of DM is further tested and thought over.

I do see a benefit for flood's also, but with limited resends. If I read @mikecarper his comment it's limited to the first hops. That could be a solution, but maybe also the situation of a busy "air" (lot of traffic) and not a single resend. I see flood's failing on a few hops and that's not nice. I have also a few "high", busy and "strong" repeaters in the neighbourhood, maybe not the best situation.

Making meshcore more reliable is the most important.

[Alain2019]

And a big thank you @usrflo and @mikecarper for the work.

[m0urs]

As far as I can see it I think that @usrflo it's forwarding of DM is further tested and thought over.

I do see a benefit for flood's also, but with limited resends. If I read @mikecarper his comment it's limited to the first hops. That could be a solution, but maybe also the situation of a busy "air" (lot of traffic) and not a single resend. I see flood's failing on a few hops and that's not nice. I have also a few "high", busy and "strong" repeaters in the neighbourhood, maybe not the best situation.

Making meshcore more reliable is the most important.

I fully agree. I am using @usrflo modification now for several weeks and I can confirm that it is really working. I have an almost 100% success rate with DMs now between two nodes via 4 hops which was never possible with the original firmware.

However I am also experiencing similar issues with flood traffic as @Alain2019 does. Often my channel messages will be sent out by my repeater in the garden but will never spread further. I need to send it one or two times again until it really gets out into the mesh. Unfortuately you will not see that a message is not spread until you double check with an observer.

So maybe having some kind of resend for flood traffic would solve this issue too.