[QA-01] Close essential operator diagnostics and fail-closed cases

[krisoshea-eth]

Why

Advanced edge hardening follows the happy path, but the retained operator failures must be understandable and demonstrably fail closed before handoff.

Planning

• Lane: Shared
• Target: Week 16
• Effort: M, approximately 3–5 ideal engineering days
• Ready after: E2E-01, OUT-01, and DATA-01
• Owner: TBD
• Reviewer: TBD

Tasks

• Cover invalid or locked key, wrong chain, inactive Builder, unsupported fork, missing preferences, and BN/EL syncing.
• Cover Builder status/balance and insufficient balance with an external-top-up warning; do not claim predictive low-balance alerting.
• Cover malformed candidate, commitment mismatch, missing reveal material, publication rejection, and late reveal.
• Cover Engine INVALID responses without wedging fork choice, and keep Builder-versus-BN fault ownership explicit, using Lodestar #9332 as the upstream watch.
• Prove that attestations and aggregates do not continue supporting an EL-invalidated Gloas payload, using the final disposition of Lodestar #9637.
• Reuse the Deathstar proposer-equivocation feature and consensus_and_equivocation behavior from feat: implement consensus_and_equivocation for blocks and payload envelopes ChainSafe/lodestar#9757, then run the stored Lodestar Builder fixture and assert that the BN refuses envelope publication.
• Cover basic duplicate event, request, and publication idempotency.
• Add bounded BlockObserver stream, retry, and terminal-failure diagnostics or metrics once selection is active; reuse existing API-client metrics and avoid identity labels.
• Document Builder or BN offline after selection as paid-without-reveal without implementing HA.
• Map every retained core failure to a named test or deterministic assertion.

Done when

The working loop is diagnosable, essential fail-closed cases including BN-owned equivocation rejection are tested, and advanced reliability work is explicitly deferred.

Evidence

• Failure-to-test index:
• Deathstar fixture: https://github.com/krisoshea-eth/lodestar-eip-7732-builder-docs/blob/main/docs/test-plans/pr-9757-builder-equivocation.yaml
• Operator diagnostics:
• CI/run logs:
• Reviewer:

Upstream watch

• Merged Lodestar #9972 implements the fail-closed block_hash != parent_block_hash invariant from consensus-specs #5594. Reuse the upstream validation and refreshed fixtures rather than adding another Builder-only guard.
• Draft consensus-specs #5573 proposes model-generated Gloas state-transition compliance vectors, including execution payload bids, parent execution payloads, payload attestations, pending payments, and PTC windows. Reuse accepted vectors when available instead of duplicating them as Builder-only unit tests.
• Draft consensus-specs #5590 is a non-normative ReqResp test-format exploration. Track its ownership decision for envelope transport evidence, but do not treat it as a current Builder contract.
• Merged Lodestar #9986 makes expected-error spec vectors fail when Lodestar accepts invalid input and exposes Gloas Builder and validator sweep-index divergences. Track the resulting fixes and refreshed fixtures here; do not duplicate the shared harness change in packages/builder.
• Open consensus-specs #5572 adds randomized equivocation delay to fork-choice compliance generation so should_apply_proposer_boost is tested when a sibling exists but is not PTC-timely. Reuse the accepted vectors for proposer-equivocation and PTC-timing evidence rather than creating a separate Builder-only generator.

Import-timeliness watch

• Approved Lodestar #9968 records whether a canonical block was imported late and adds case-specific logging and a metric. Reuse it to distinguish late Builder selection or reveal handling caused by local BN import delay from Builder-runtime delay. It is diagnostic evidence for QA-01, not a replacement for API-02, REL-01, or a new Builder service.
• Open Lodestar #9994 guards known genesis and pre-Gloas roots from impossible payload-envelope sync. Reuse its eligibility rule in negative-path QA and keep unknown post-Gloas roots eligible for real recovery.

[linear-code]

LOD-23