Skip to content

mesh_gate: split by concern, guard the ARP table, add introspection - #2

Open
aldwinhermanudin wants to merge 5 commits into
mainfrom
refactor/mesh-gate-split
Open

aldwinhermanudin wants to merge 5 commits into
mainfrom
refactor/mesh-gate-split

Conversation

@aldwinhermanudin

Copy link
Copy Markdown
Contributor

Follow-up to #1. Makes the mesh-gate example navigable before deciding whether its reusable parts should become a component.

app_main.c was 1013 lines, ~75% of it infrastructure identical for any gate. Split along that seam:

File Lines Concern
app_main.c 182 bring-up order, status loop
gate_netif.c 252 AP-side esp_netif, local delivery, addressing
gate_bridge.c 342 the datapath — three receive callbacks
gate_arp.c 375 proxy ARP
gate.h 111 the seam

Behaviour

No change intended. Re-verified against the unmodified Rimba fixture test-mesh-gate-sta on the same three-board rig:

TEST|STEP|dhcp|PASS|lease=10.9.9.2
TEST|STEP|delivery|PASS|replies=14/15 timeouts=1
TEST|STEP|ttl|PASS|observed_ttl=64 expected=64
TEST|RESULT|PASS

The original scored 13/15. That difference is within the bench's RX-overload variance and is not claimed as an improvement.

Three real changes

The proxy-ARP table is now guarded. It has three concurrent accessors — both receive tasks snoop into it, the announce task ages and reads it — and was unlocked, so two receive contexts could interleave on a slot and publish a mismatched IP/MAC pair. The proactive push transmits, so it snapshots under the lock and sends afterwards rather than holding it across the loop.

Introspection. The gate could not report what it had learned without editing source. It now logs associated clients, mesh peers, cross-bridge mappings and known gates:

uptime=90s  ap_clients=1  mesh_peers=1  arp_mappings=2  gates_known=0  heap=8573968

A _Static_assert couples EXAMPLE_AP_MAX_STAS to GATE_MAX_CLIENTS. Only the mesh→AP direction consults the client table, so a client past the end of it would associate, take a lease, send into the mesh, and silently never receive a reply. The Kconfig range already prevents this; the assert keeps them in step if someone widens that range. Verified it fires by temporarily widening the range.

Incidental

Four open-coded copies of "alloc mmpkt, append, tag VIF_AP, transmit" folded into gate_netif_tx_ap(). The netif hook keeps its blocking tx_wait_until_ready because it runs on the lwIP task; receive-context callers must not block, and that distinction is now explicit.

app_main.c was 1013 lines, about three quarters of which was infrastructure
identical for any gate. Split along the seams so each concern reads on its
own, and so the reusable parts are visibly separable if they are ever lifted
into a component:

    app_main.c     182  bring-up order and a status loop
    gate_netif.c   252  AP-side esp_netif, local delivery, addressing
    gate_bridge.c  342  the datapath -- three receive callbacks
    gate_arp.c     375  proxy ARP
    gate.h         111  the seam between them

No behavioural change was intended. Verified against the unmodified Rimba
fixture test-mesh-gate-sta on the same three-board rig: DHCP lease, 14/15
replies to a mesh node, ttl=64 (pure L2 bridge). The original scored 13/15;
the difference is within the bench's RX-overload variance and is not claimed
as an improvement.

Three things did change:

- The proxy-ARP table is now guarded. It has three concurrent accessors --
  both receive tasks snoop into it and the announce task ages and reads it --
  and was previously unlocked, so two receive contexts could interleave on a
  slot and publish a mismatched IP/MAC pair. The proactive push transmits, so
  it snapshots the table under the lock and sends afterwards rather than
  holding it across the loop.

- Introspection. The gate could not previously report what it had learned
  without editing the source. It now logs associated clients, mesh peers,
  cross-bridge ARP mappings and known gates, backed by small accessors.

- A _Static_assert couples EXAMPLE_AP_MAX_STAS to GATE_MAX_CLIENTS. Only the
  mesh->AP direction consults the client table, so a client past the end of it
  would associate, take a DHCP lease, send into the mesh, and silently never
  receive a reply. The Kconfig range already prevents this; the assert is what
  keeps the two in step if someone widens that range.

Also folded four open-coded copies of "alloc mmpkt, append, tag VIF_AP,
transmit" into gate_netif_tx_ap(). The netif hook keeps its blocking
tx_wait_until_ready because it runs on the lwIP task; receive-context callers
must not block, and that distinction is now explicit rather than incidental.
The target hardware address sits at ARP-payload offset 18, which is absolute 32
once the 14-byte Ethernet header is counted. The builder wrote it at 34, so:

  out[32..33]  never written -- uninitialised stack went on air
  out[34..39]  the requester MAC, shifted two bytes late
  out[38..39]  written twice, the target protocol address winning

Every proxy-ARP reply the gate sent to an AP client therefore carried a
malformed target hardware address and leaked two bytes of stack.

It resolved correctly anyway, which is why it survived every bridge test: lwIP's
etharp_input decides "for us" from the target PROTOCOL address, which was
correct, and populates its cache from the sender hardware/protocol pair, which
was also correct. Nothing on the receive path reads the field that was wrong.

The mesh-side SNAP builder in the same file always had this right (a + 18), and
that disagreement between two builders of the same structure is what exposed it.
…l comment

Brings the example in step with the shipped Rimba gate, which raised its client
ceiling and made the proxy-ARP push event-driven.

Client ceiling 8 -> 16, with the arithmetic recorded where the constant is
defined. The proactive push is O(clients x mesh hosts) in BOTH directions at
~4.85 ms per push, so with 5 mesh nodes it costs 5.2% of airtime at 16 clients
on a 15 s period, 25.8% on a 3 s period, and 51.7% at 32 clients on 3 s. Two
ceilings outside the example bind before the array does, and both are noted:
MESH_MPP_MAX (32), which lives on every mesh node and which 16 clients on one
gate already half-fills, and the flat /24's ~98 usable addresses. None of them
is fixed by a faster PHY.

ARP_TABLE_MAX is now derived (AP_MAX_STAS + 32) rather than a flat 24. It holds
BOTH sides, so at 24 with 5 mesh nodes it saturated near 19 clients and then
thrashed its LRU -- silently ceasing to teach ARP rather than failing.

The push period default moves 3 s -> 15 s, which is only safe because the push
is now event-driven on top of it: arp_learn() wakes the announce task on a
genuinely new mapping, and the task waits with ulTaskNotifyTake(pdTRUE, ...) so
notifications collapse. A joining host is taught in milliseconds; the period is
upkeep. The task is now started BEFORE the receive callbacks are registered,
because the wake is by handle and a host learned while that handle is NULL
drops its wake silently and waits out a full period.

Also corrects an actively wrong comment in gate_netif.c.
CONFIG_LWIP_DHCPS_MAX_STATION_NUM bounds the lease-record list, NOT the address
range: the range is start = server + 1 = .2 through start + DHCPS_MAX_LEASE
(100), so the pool is .2-.101 whatever that Kconfig says, and its top two
addresses do overlap the mesh nodes' .100 upwards. Unreachable below ~100
clients, and pinning it risks the DHCP_NOT_STOPPED boot loop, so it is recorded
rather than changed. The old comment made raising the ceiling look dangerous
when it is not, and made the real overlap look impossible.
The "Verified on hardware" section listed association, DHCP, reply counts and
TTL -- all read from serial logs and ping -- under a heading that reads as a
stronger claim than the evidence supports. Per the project rule that the
verification tier is recorded per frame, the section now says what the evidence
is: log-level, no capture, no byte-level comparison against a Linux
transmission.

The interoperability paragraph had the same problem in a different form. Those
runs, including the monitor capture byte-diffing the announcement and beacon
gate bit against a live Linux gate, were performed on the Rimba implementation
this example derives from and were not re-run here. Same code, different
repository's evidence -- now stated rather than implied.

The distinction is not pedantry. Log-level evidence shows the datapath works end
to end but cannot show that an emitted frame is well-formed, and a malformed
field that no receiver happens to read passes every test on that page. That is
precisely the proxy-ARP target-hardware-address defect fixed two commits ago,
which survived every bridge test for exactly that reason -- so it is now cited
there as the concrete example.

Also refreshes the defaults changed in the previous commit: the proxy-ARP push
interval (3 s -> 15 s, with a note that it is upkeep and a new host is pushed
immediately) and the client ceiling's new maximum.
@aldwinhermanudin

Copy link
Copy Markdown
Contributor Author

Review

Verdict: the split is sound and the locking work is the most valuable part of it. Three commits added on top — one of them a real bug this review found.

The split

The seam is the right one. gate.h exposes state and entry points rather than internals, gate_arp.c has no knowledge of the bridge's client table, and gate_netif.c owns everything esp_netif-shaped. Folding the four open-coded "alloc mmpkt, append, tag VIF_AP, transmit" copies into gate_netif_tx_ap() is a real reduction, and keeping the blocking tx_wait_until_ready only on the lwIP-task path — with that distinction stated — is the correct call rather than a uniform one.

The _Static_assert here is better than the equivalent in Rimba: AP_MAX_STAS <= GATE_MAX_CLIENTS is a genuine check across two independently-set values, not a tautology.

The ARP table lock is a fix Rimba did not have

The description undersells this. Three concurrent accessors and no guard was one problem; the announce loop reading entries live across its transmit calls was a second, and arguably worse — a slot could be evicted and refilled mid-push, teaching a host an IP/MAC pair that never coexisted. Snapshotting under the lock and sending afterwards fixes both.

This has now been backported to the Rimba gate, which still had the unguarded version in shipped code.

Bug found: the proxy-ARP reply was malformed

arp_build_reply() wrote the target hardware address at absolute offset 34. It belongs at 32 — ARP-payload offset 18, plus the 14-byte Ethernet header. So:

  • out[32..33] were never written — uninitialised stack went on air
  • out[34..39] held the requester MAC, shifted two bytes late
  • out[38..39] were written twice, the target protocol address winning

Every proxy-ARP reply sent to an AP client carried a malformed target hardware address.

It resolved correctly anyway, which is why it survived every bridge test on record: lwIP's etharp_input decides for us from the target protocol address (correct here) and populates its cache from the sender hardware/protocol pair (also correct). Nothing on the receive path reads the field that was wrong.

What gave it away was this file disagreeing with itselfarp_build_snap_reply() puts the same field at a + 18, which is right. Two builders of one structure, off by two.

Same defect was present in both Rimba gate apps and is fixed there too.

Also pushed

  • Ceiling 8 → 16 with the airtime arithmetic recorded at the constant, and the two limits that bind before the array does (MESH_MPP_MAX = 32, which 16 clients on one gate already half-fills mesh-wide; the flat /24's ~98 usable addresses). ARP_TABLE_MAX is now derived rather than a flat 24 — it holds both sides, so at 24 with 5 mesh nodes it saturated near 19 clients and then thrashed its LRU, silently ceasing to teach ARP.
  • Push period 3 s → 15 s, made safe by being event-driven. arp_learn() wakes the announce task on a genuinely new mapping; notifications collapse. The task is now started before the receive callbacks, because the wake is by handle and a host learned while that handle is NULL drops its wake silently.
  • A comment in gate_netif.c that was actively wrong. CONFIG_LWIP_DHCPS_MAX_STATION_NUM bounds the lease-record list, not the address range — the pool is .2-.101 regardless and does overlap the mesh nodes' .100 upwards. Unreachable below ~100 clients and risky to pin, so recorded rather than changed.
  • The README's verification tier. "Verified on hardware" covered evidence that is entirely log-level, and the interoperability paragraph implied captures that were taken against the Rimba implementation, not this example. Both now say so. The proxy-ARP defect above is cited there as the concrete reason the distinction matters: a malformed field no receiver reads passes every log-level test.

Not addressed

mesh_gate still has no teardown — no way to unregister the receive callbacks, destroy the second netif, or stop the announce task. That remains the largest obstacle to lifting the reusable parts into a component, and it grows more expensive the longer the example is copied.

The locking added in this branch was correct but used a portMUX critical
section, which disables interrupts -- and it sat on the PER-FRAME path, scanning
the table with interrupts off on every received frame.

That costs real delivery. The Rimba gate carries the same code, and an A/B there
on one night, all three arms under the same RF conditions:

    critical sections   28/30 pings, then 28/30    median RTT 46 ms
    no locking at all   30/30                      median RTT 43 ms
    mutex               30/30                      median RTT 43 ms

The two 28/30 runs looked like bench variance -- they ran hours after the
baseline with the median RTT up -- so the no-locking control was built and run
rather than argued about.

A mutex is legal here because every accessor is a task: both receive callbacks
are dispatched from the one MAC event loop and the announce task is its own, so
no interrupt context ever touches the table. It costs a take/give and blocks
nothing at the driver level.

The lock is still never held across a transmit; the push snapshots under it and
sends afterwards, or the datapath would block for the whole sweep.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant