Update grafana json

[grantkee]

Problem

The protocol has several new metrics, but the grafana dashboard was not edited to track these. etc/grafana/telcoin-node-metrics.json

[grantkee]

Two more dashboard candidates, from the review of #1306 (perf(batch-builder): read sender balances through one state provider, closes #1303).

Neither series exists yet, so there is nothing to add to etc/grafana/telcoin-node-metrics.json today. Recording them here as nice-to-have panels if the review's observability findings land, so they do not get lost the way the current batch did.

tn_reth.batch_balance_read_failures_total

Counts builds where WorkerTxPool::get_account_balances could not acquire a state provider at all, so every sender of the batch just mined was reported at U256::ZERO in the optimistic pool update and its remaining transactions stayed parked until the engine's authoritative update landed the same round.

Counts events, one per failed build. That is deliberately different from tn_reth.canon_state_resync_read_failures_total, which counts individual address reads that stay in the dirty set for retry. The batch-builder path has no dirty set and no retry, so the two should not share a series or a panel: a rate() over both would mix retryable per-address reads with unretried whole-set degradations.

A counter for degradation after a successful acquisition

The narrower gap the same review turned up. The existing warn! guards only the acquisition step, so a read that fails after the provider is in hand degrades the whole set to zero with no log line and no metric.

Reth's realistic post-acquisition failure modes are all address independent (DatabaseError in LatestStateProviderRef::basic_account, StateAtBlockPruned and AccountChangesetNotFound in HistoricalStateProviderRef), so any of them fails for every address in the set rather than one sender. A per-address log would emit one line per sender for a single underlying fault; a post-loop count of failed reads targets it better and is the more useful panel.

Alerting shape for both

Alert on a sustained rate, not on any nonzero value. Zero balance is the fail-safe direction: it can only keep a sender's transactions parked, never promote an unfunded one, and the engine's authoritative update corrects it within the same consensus round. An isolated increment is the intended absorption of a transient provider fault. A rate that does not fall back to zero means every build is losing its optimistic nonce and balance advance, and throughput has quietly fallen back to the engine's update alone.

Full detail, including the proposed RethEnvMetrics field and its doc comment, is in the review notes on #1306.
A trap for whichever panel tracks tn_network.peer_penalties_total, found while investigating #1338. This one is about an existing series rather than a proposed one, so it affects the dashboard as it is built today.

The counter increments before the exemption is applied, so it counts penalties that were never enforced:

// crates/network-libp2p/src/peers/manager.rs:499-512
pub(crate) fn process_penalty(&mut self, peer_id: PeerId, penalty: Penalty) {
    if self.is_local_peer(&peer_id) {
        // ...
        return;
    }
    self.metrics.record_penalty(&penalty);              // :508  counted here
    let action = self.peers.process_penalty(&peer_id, penalty);  // :509  exemption applied in here

AllPeers::process_penalty (all_peers.rs:264) resolves a TrustBasis and hands it to Peer::apply_penalty (peer.rs:288-312), which bypasses the score model entirely for exempt peers. Exempt covers the operator allowlist and committee validators, with TrustBasis spanning the previous, current, and next committees. The only guard ahead of the metric is is_local_peer, so self-connections are excluded but exempt peers are not.

The result is that severity="fatal" counts increments attributable to committee validators that were never disconnected or banned. That is the severity an operator would alert on to find a genuinely faulty committee member, so as built the alert fires for honest peers.

#1338 is a concrete producer of exactly this: a committee validator that broadcasts an empty batch has every peer classify it Penalty::Fatal, suppress the ban because the proposer is exempt, log warn! "skipping severe/fatal penalty for exempt peer", and increment the fatal counter anyway.

Two ways to handle it, depending on appetite for a code change:

Dashboard only: treat peer_penalties_total{severity="fatal"} as "penalties computed" rather than "peers punished", and read it alongside tn_network.peers_banned_total, which only increments on real enforcement. A fatal count with a flat banned count means suppression, not misbehavior.

With a code change: either move record_penalty after the exemption resolves, or keep it where it is and add an exempt label (or a separate suppressed counter) so one panel can separate enforced from suppressed. Splitting them is more useful than moving the call, since a suppressed fatal against a committee member is worth seeing rather than dropping. The warn! at peer.rs:302-308 already exists for this, but a log line is not something a dashboard can graph.