Skip to content

fix(metrics): count inbound sync stream shedding by reason (#1307) - #1317

Open
MavenRain wants to merge 6 commits into
mainfrom
1307-sync-stream-shed-metric
Open

fix(metrics): count inbound sync stream shedding by reason (#1307)#1317
MavenRain wants to merge 6 commits into
mainfrom
1307-sync-stream-shed-metric

Conversation

@MavenRain

Copy link
Copy Markdown
Contributor

Closes #1307.

Problem

WorkerNetwork::shed_inbound_sync_stream refuses inbound sync streams in two ways, both on stream open before any request frame is read: when the admission caps hit it spawns a shed task that writes Deny(AtCapacity), and when the shed budget (MAX_CONCURRENT_SHED_TASKS, #1265) is exhausted it drops the stream with no reply. Neither path reaches a counter, and the only trace is a debug! line the default info filter hides. On the requester side the budget-exhausted drop reads as a generic failed to read sync ack frame I/O error, so backpressure is indistinguishable from a transport fault.

Fix

  • crates/consensus/worker/src/metrics.rs: SyncShedReason { Denied, BudgetExhausted } with a const fn label, and WorkerMetrics::record_sync_stream_shed recording tn_worker.sync_streams_shed_total{worker, reason} (same labeled-counter pattern as ForwardDropReason / record_forward_dropped).
  • crates/consensus/worker/src/network/mod.rs: WorkerNetwork gains a metrics: WorkerMetrics field built from the worker id in new. No signature change: the registry keys series by name and labels, so this instance and the RequestHandler's record into the same per-worker series.
  • admit_shed_and_record wraps try_admit_shed: a reserved slot records denied, no slot records budget_exhausted. shed_inbound_sync_stream calls it at the decision point, before the spawn, so the count stays exact when the best-effort, SYNC_REQUEST_READ_TIMEOUT-bounded deny write later fails or the task never runs.
  • Why both arms: the shed budget has no per-peer sub-cap, while the admission path it guards has one. A nonzero budget_exhausted rate while denied stays low is the signature of shed slots pinned by peers that never read their deny reply (the squatting case from the issue). The pair, not the sum, is the signal. The series carry no peer label; attribution needs the per-peer debug! lines.

Threat model: the counter adds one labeled increment per refused stream, the same cost as the existing quorum_failures_total / forwarded_txns_dropped_total counters, and only on a path that already dropped the stream, so a flooding peer cannot make the responder do more work than before. No admission bound changes: MAX_CONCURRENT_BATCH_STREAMS, MAX_PENDING_REQUESTS_PER_PEER, and MAX_CONCURRENT_SHED_TASKS are untouched. The primary's PrimaryMetrics mirror is out of scope, per the issue.

Testing

  • gateledger run -- cargo +nightly fmt -p tn-worker -- --check: green.
  • metrics::tests::test_metrics_register_and_update extended: two Denied and one BudgetExhausted refusal produce one series per reason under the worker label with counts 2 and 1. Confirmed by mutation (collapsing the budget_exhausted label into denied fails the test).
  • New network::tests::shed_admit_records_reason_per_arm: fills the shed budget through admit_shed_and_record (every slot records denied), then one more call is refused and records budget_exhausted. Confirmed by mutation (swapping the two reasons in admit_shed_and_record fails the test).
  • gateledger run -- cargo +1.94 test -p tn-worker --lib in an isolated target dir: green.
  • CI runs on push; the pinned cargo +1.94 attest runs on the second machine.

Both shed arms in WorkerNetwork::shed_inbound_sync_stream refuse a
stream before any request frame is read, so no counter saw them and
the only trace was a debug! line hidden by the default info filter.

Add tn_worker.sync_streams_shed_total{worker, reason} with reasons
denied (admission caps hit, deny task spawned) and budget_exhausted
(shed budget full, stream dropped with no reply), recorded at the
admission decision through admit_shed_and_record, before the
best-effort deny write. The pair is the signal: a nonzero
budget_exhausted rate while denied stays low is the signature of
shed slots pinned by peers that never read their deny reply.

Extend the WorkerMetrics unit test and add a wiring test for the
per-arm reason; both confirmed by mutation.

Closes #1307.

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
@MavenRain MavenRain self-assigned this Aug 30, 2026
…-metric

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…-metric

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…-metric

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…-metric

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metric: Inbound sync stream shedding

1 participant