Repository navigation
node, router: members hold two routers, routers send members away when they drain or fill, and a scalability guide - #647
Merged
Conversation
The retry gave up after eight answers, and the control plane names a wait of two seconds on every 429, so a member stopped trying after about twenty seconds: on the testnet 1500 members starting at once through the limiter of ten a second needed 150, and a third of them gave up. The budget is now three minutes of elapsed time whatever the control plane answers, which admits a fleet of about 1500 members that start together.
What each component spends per member, which setting bounds what, how to size routers, the join and a service for a fleet of a given size, the kernel limits a host running many nodes needs, and the metrics that say a bound is near. The numbers are the testnet's: 3000 members resident on three routers of 2 vCPU twenty minutes after the first started, 150 KiB of router memory per connection, ten enrollments a second at the control plane. The guide also states the two facts that surprise: every member connects to every router, so the high watermark of one router bounds the mesh and more routers add relay capacity rather than members; and the low watermark is a floor to keep above the fleet, since trimming below it disconnects most of the mesh at once.
The connection manager closes connections down to the low watermark once the high one is crossed, and the defaults were libp2p's, 1000 and 4000: on a hub whose every connection is a member session, the first connection over 4000 cost 3000 members their router at once. The low watermark now follows the high one at 15% below it unless set, and a low mark at or above the high one is refused.
A member chose among routers from a flat list of addresses. /info now lists each router with a live lease as RouterInfo: its peer ID and addresses, the labels the control plane attested at its enrollment, and its load from its last lease, open inbound connections against the number it accepts. A router declares labels with --label, the same labels as a node's, signed only if mesh:role:router allows them in allowed_labels and refused at enrollment otherwise. The lease carries the load; the labels come from the enrollment record. router_addresses stays, flattened from the same routers, for members that only need something to dial.
A node attached to every router the control plane listed, so a mesh of N members put N connections on every router and the high watermark of one router bounded the mesh; on the testnet three routers each held a connection from all 3000 members. A node now keeps sessions with --routers of them (default 2, every router when there are fewer), chosen from /info: those attesting every pair of --router-selector, ordered by the pairs of --router-prefer they attest and then by the room they reported in their last lease. The connection manager protects the attached ones; AutoRelay is offered those and no others, since a reservation needs an authenticated session. A router that drops is redialled as before, and one that does not come back, or that refuses the credential, is replaced from the list and left alone for a while; the monitor tops the count back up on its tick. A selector no router attests is retried the way an unanswered router is, with a warning that names the selector and the labels seen, and the control plane warns when a refresh drops a label the policy no longer allows. agentmesh_node_routers and agentmesh_node_router_candidates say where a node stands.
AutoRelay publishes a /p2p-circuit address only for a relay on a public IP; a router on a private or loopback address yields none. A node that is reachable through its routers alone then publishes no address at all, the router's DHT drops its provider records as address-less, and nobody finds its services. The testnet has public routers and never saw this; an on-premises mesh and the integration tests do. The node now adds <router>/p2p/<id>/p2p-circuit for each address of each router AutoRelay holds a reservation on, read from the connection manager tag AutoRelay protects a reserved relay with, skipping any AutoRelay already published. GET /debug/mesh-info lists the routers the node is attached to.
A router relays only between peers it has admitted. With a node holding two of the routers, the destination's router is often one the caller has no session with, and the circuit is refused as "src not authenticated". Before dialing, the node runs the credential handshake with every router that one of the destination's circuit addresses goes through and that it is not yet authenticated to, on the connection the circuit will use. A relay that is not a router (a node with --enable-relay) is left alone.
Two routers labelled by site, node A selecting site=a and node B site=b, neither publishing a direct address. Each node attaches to the router its selector names and no other, read from /debug/mesh-info; router B federates with router A; B discovers A's service through the one DHT and calls it over a circuit through router A, which admits B on the handshake B runs there first. Afterwards B is connected to both routers and still attached to one, and A to its own only. Disabling the transit handshake fails the call with 502.
…aded
A router that stopped closed its connections and every member found out
when its session died, redialled a router that was gone, and only then
turned to the others; a router at its high watermark had the connection
manager close connections blindly and the members dialled straight back,
so a router over capacity stayed over capacity.
The router now tells a member it will no longer hold it, on a new stream
protocol /mesh/goaway/1.0.0 carrying one RouterGoAway{reason,
retry_after}: DRAINING on shutdown, with the lease ttl it withdrew with as
the retry time, and OVERLOADED every 5 seconds while its inbound
connections are at the high watermark, for as many members at random as
take it back to the low watermark, with a 5 minute retry time. It forgets
them as authenticated and closes their connections after a 5 second grace,
which also covers a member that does not speak the protocol. Peer routers
are never sent away and are protected from the connection manager's trim:
losing one splits the DHT.
A node that receives the message, from a router it holds, shuns that
router for the retry time (capped at an hour), detaches from it and tops
up from the others at once, so it does not redial the router when the
connection then closes. The SDKs do not handle the message yet and see
the connection close as before.
Metrics: agentmesh_router_goaway_sent_total{reason,result} and
agentmesh_node_router_goaway_received_total{reason}.
The connection manager trims by tag value, lowest first, and every connection carried the same value, so above the high watermark it closed members as readily as strangers. A peer that passes the handshake is now tagged, and the tag goes when its credential lapses or the router sends it away, so a trim closes untagged connections first: a scanner, a control plane publisher, a member attached to another router that ran a DHT query here. Those are all connections that come back on demand; a member's session is not. A deadline that closed connections for not authenticating was considered and not done: the control plane's event publisher and a member's DHT queries from another router are unauthenticated by design, and closing them on a timer would only have them redial.
Three routers, a node holding two with its monitor off; one of the two gets SIGTERM. The node is on the third router within the second, on the router's go-away alone: without it the node redials the stopped router and, with no monitor to top it up, holds one router until the test gives up. The cross-router test shares the control plane setup.
The scalability guide stated the model it was written against: every member on every router, so the high watermark of one router bounded the mesh and the low watermark was a floor to keep above the fleet. The guide now gives the model the node implements: a member holds K routers (2), a router holds F x K / N members, a mesh of N routers holds N x H / K, and the high watermark is sized for the day one router is down (1.2 x F x K / (N - 1)). It adds how labels place members on routers and what a selector does and does not do, the router-to-router connection count, the go-away on shutdown and overload, the node's router gauges, and an autoscaler rule that follows from the model. The Kubernetes guide, the architecture and networking concepts and the node reference say the same in their one or two sentences on routers.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a robust router load-shedding and selection mechanism, allowing nodes to dynamically attach to a subset of routers based on labels and load. It implements the /mesh/goaway/1.0.0 protocol, enabling routers to gracefully drain members during shutdown or shed excess load when exceeding high watermarks. Nodes now maintain a router catalog, support selector and preference labels, and automatically failover to alternative routers when sent away or disconnected. Additionally, client retry logic has been refactored from attempt-based to time-budget-based, and database schema migrations have been added to track router connections and limits. I have no feedback to provide.
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A member holds two routers instead of every router, so adding routers adds members; a router tells its members to go elsewhere when it stops or fills up; and the scalability guide states the model and how to size for it.
Refs #277. Closes #193.
Why
Ladder run 3 on
bananas(#277) put 3248 members on four routers and stopped there: every member attached to every router, so each router sat at 3200–3500 of its 4000 inbound and the next step would have trimmed the whole mesh to the low watermark (1000). Members that lost a router redialled the same one. The enrollment retry gave up after ~20 s where a fleet of 1500 at 10/s needs 150 s.What changes
Each commit stands alone and builds alone.
Backoff.Budget(3 m) replaces an attempt count; a fleet turned away with429keeps retrying withRetry-Afteruntil it is in.--low-watermarkdefaults to 15% below the high mark (3400 of 4000), not 25% of it./infowith labels and loadRouterInfo{peer_id, addresses, labels, connections, connection_limit}onControlPlaneInfoResponse.routers; the lease carries the load; routers take--label(checked against the router role'sallowed_labelsat enrollment).--routers(2),--router-selector(every pair required),--router-prefer(ordering, then load). The node keeps a catalog from/info, tops up from it when a router drops, shuns one that refuses it./debug/mesh-inforeportsattached_routers; gaugesagentmesh_node_routers{state},agentmesh_node_router_candidates{match}./p2p-circuitaddresses only for relays on public IPs; a node whose routers are on private or loopback addresses published no address at all. The node adds one per reserved router./mesh/goaway/1.0.0carryingRouterGoAway{reason, retry_after}.DRAININGto every member on shutdown;OVERLOADEDat the high watermark to random members down to the low one, 5 min retry. Peer routers are never shed and are protected from the trim. The node, on a go-away from a router it holds, shuns it, detaches and tops up at once. SDKs do not handle it yet and see the connection close as before (noted insdk/README.md).SIGTERMone: the node is on the third in ~200 ms. Fails with the drain removed.--high-watermark≥ 1.2×F×K/(N−1) so a stopped router's members fit on the others. Labels place members (selector, prefer; one value per key; hierarchy = several keys); a selector does not isolate. Router-to-router connections are N(N−1)/2. Metrics and an autoscaler rule. Kubernetes guide, concepts and references aligned.Verified
go test ./...andtests/integration(110 s) green;golangci-lint0 issues;hack/verify-generated.shandverify-sdk-generated.shup to date; each commit builds and vets on its own.Next
Rerun the member ladder on
bananasonce this is deployed: 5000 members on four routers (expected ≈2500 inbound each), then six routers for 10k. #277 closes with those numbers; label-scoped federation is tracked separately per the comment on #193.