refactor: select experimental native state transition during setup - #10014
Draft
wemeetagain wants to merge 18 commits into
Draft
refactor: select experimental native state transition during setup#10014wemeetagain wants to merge 18 commits into
wemeetagain wants to merge 18 commits into
Conversation
…10002) lodestar-geth-2 on glamsterdam-devnet-9 was stuck at head 16192 for 20+ minutes after a restart, every range sync batch starting at 16192 failed with `BLOCK_ERROR_PARENT_PAYLOAD_UNKNOWN` at 16193 (~1000 errors, 190+ sync chain restarts) until a Prysm peer happened to serve the batch without the 16192 envelope. the payload of 16192 was revealed late and orphaned, 16193 builds on the EMPTY variant of 16192 (its `bid.parent_block_hash` is the 16191 payload hash). block 16192 was already imported as target of the previous finalized sync chain, so the batch is processed as `[16193, ...]` with parent 16192 EMPTY from fork choice, which is correct. but peers still serve the orphaned 16192 envelope in `execution_payload_envelopes_by_range` and `assertLinearChainSegment` unconditionally takes an envelope for the parent slot as the parent EL head, so the first block check fails against the orphaned payload hash and the batch is retried forever. only take the parent envelope as EL head if the first block actually builds on it (the checkpoint sync anchor case this was added for), otherwise keep the fork choice parent and report the envelope as orphaned, same as in-segment orphans. replayed the real 16192/16193 blocks and 16192 envelope from the devnet through `assertLinearChainSegment`, unstable throws `PARENT_PAYLOAD_UNKNOWN`, with this change it passes and reports slot 16192 as orphaned.
**Motivation** - gossip handler functions contains 2 parts: validation part and handler part - right now, the handler part was not awaited, however it's still in the same event loop. As we can see in #9988, we run state transition before forwarding an ACCEPT block **Description** - use `callInNextEventLoop()` util to forward ACCEPT gossip message asap. The more forwarding messages, the more peer score/mesh peers we can gain - remove the `callInNextEventLoop()` in NetworkProcessor, otherwise we still forward block after we run state transition Closes #9988
Empty array values break older VCs (e.g. Nimbus) that only allowlist blob_schedule as non-string in getSpec. **Motivation** Lodestar `v1.47.0` always returns `GAS_LIMIT_SCHEDULE` from `GET /eth/v1/config/spec`, including as `[]` on mainnet. That breaks Nimbus VC `v26.8.0` (and similarly strict clients), which fail to decode the array and mark the BN Offline in a ~2s Online/Offline flap. Emitting the field before Gloas is scheduled has no operational value under EIP-8261. **Description** Omit `GAS_LIMIT_SCHEDULE` from `renderJsonSpec()` when `GLOAS_FORK_EPOCH === Infinity` or the schedule is empty, mirroring the existing `BLOB_SCHEDULE` interop guard. Include it only when Gloas is scheduled and the schedule has entries. Adds unit coverage for omit/include behavior.
fix `writting` typo in a comment in `persistedKeys.ts`
…le (#10003) after a restart from a db or checkpoint state at an epoch boundary whose boundary slot was skipped, the head block's post-state does not exist anywhere. the anchor state is the boundary state (slot > head block slot) and is only cached under its own root, regen walks ancestors only and the anchor has none. every `head` state request (`getStateFinalityCheckpoints`, `getEpochCommittees`, `postStateValidators`, ...) fails with `REGEN_ERROR_NO_SEED_STATE` until the first new block is imported, seen on glamsterdam-devnet-9 after restarts with anchor state slot 16000 and head block 15999. serve the closest head state in that case, same as `chain.getHeadState()` does internally. the cached head post-state is still preferred when it exists.
…sync (#10005) 19 of 30 lodestar nodes on glamsterdam-devnet-9 had their head parked for hours on a block whose payload was orphaned (e.g. 7264, 7584, 7745) while range sync kept importing thousands of blocks behind it. the validator client logs `Node is syncing` the whole time, `notifyForkchoiceUpdate` goes out with the stale head and the EL answers `Too deep reorg`. range sync imports the orphaned envelope as an in-segment orphan, which adds a FULL fork choice variant without descendants. during range sync no attestations are imported (`AttestationImportOpt.Skip` on the finalized chain, blocks older than `FORK_CHOICE_ATT_EPOCH_LIMIT` otherwise) so the EMPTY variant that carries the whole chain has weight 0 as well, same root, and `getPayloadStatusTiebreaker` picks FULL for any block older than the previous slot, exactly as in the spec `get_head`. the spec never sees this tie because it feeds block attestations into `get_weight`. the head only moves again once a block with a newer justification is imported and the leaf fails `nodeIsViableForHead`, on devnet-9 that was 270 epochs later. the chain did not build on an orphaned payload, so there is nothing to gain from importing it: fork choice does not need the variant, the EL does not need `newPayload` for it and lighthouse/prysm do not even serve those envelopes by range. skip DA verification and import for the envelopes `assertLinearChainSegment` reports as orphaned, including the parent one from #10002 which otherwise creates the same dead leaf at the batch boundary, whenever the block attestations are not imported (`importsBlockAttestations()`, same condition as `importBlock()`). orphaned payloads of recent blocks are still imported, there weight decides the variant and a reorg of the child could still build on the payload. skipped envelopes were never validated, they are pruned from `seenPayloadEnvelopeInputCache` so they are not served to peers, by-root sync reloads the entry from db if the payload is ever needed. the batch's own map is left untouched. **verified on devnet-9** - lodestar-erigon-1 (head parked at 7745 for 2.5h): restarted with this change it skipped the orphaned envelopes at 7264 and 7392 and the head followed the synced chain through 7264, 7392 and 7584, the slots other unpatched nodes were parked at - lodestar-reth-1 (head parked at 7264 for 3.5h, then looping on `PARENT_PAYLOAD_UNKNOWN` at 16542 after unparking) restarted with #10002 + this change: skipped the orphaned envelopes at 16541/16548/16839/16856/16944, head tracked the imported tip the whole way and reached head - 14 more nodes that had unparked and were looping at 16065 restarted with both changes: 0 `PARENT_PAYLOAD_UNKNOWN` since, orphaned envelopes at 16065/16096 skipped, heads followed the import and all reached head the attestation gate and the cache prune were added in review and are covered by unit tests only, the finalized sync chain behaves the same with and without them **known gap**: an orphaned payload of the last block in a batch (or of a count=1 target batch) has no child in the segment and is still imported, in a non-finalizing chain that FULL leaf parks the head the same way until the justified checkpoint moves. deferring the last envelope of a batch to the next one (where #10002 already classifies the parent payload) or resolving parent payloads from the seen cache would close it, left as follow-up
**Motivation** - monitor the `PersistentCheckpointStateCache` persisted epochs/states **Description** - as monitored on `glamsterdam-devnet-9`: <img width="1394" height="462" alt="Screenshot 2026-09-04 at 16 21 40" src="https://github.com/user-attachments/assets/3d7a01ec-a866-4a5f-806f-4940495e8a2a" /> Co-authored-by: twoeths <twoeths@users.noreply.github.com>
…10006) lodestar-geth-5 on glamsterdam-devnet-9 (running #10002 + #10005) got stuck at head 17533 looping on `BLOCK_ERROR_PARENT_PAYLOAD_UNKNOWN` for 17547. the first peer served block 17533 without its envelope, so it was imported with a PENDING payload. the next peer served the same batch including the envelope and the finalized sync chain re-downloaded a 1-slot batch `[17533]` with it as well, but with every block already known `processBlocks` returns early on "no relevant blocks" and drops the envelopes with them. 17547 builds on the FULL variant of 17533, which never gets created, so `verifyBlocksSanityChecks` throws on every retry and range sync never advances. import the envelopes of already known blocks in that early return if the block has no FULL variant yet. if our head descends from the block's EMPTY variant the chain built on EMPTY and the payload is orphaned, importing it would only add the dead FULL leaf #10005 removes, so it is skipped and pruned from the seen cache under the same condition as there: only while the block attestations are not imported (`importsBlockAttestations()`), a recent envelope is imported anyway so a reorg building on it does not have to fetch it through unknown block sync. a fork child on EMPTY does not block the import, the chain being synced may still build on the payload. `importExecutionPayload()` verifies the envelope against the block state, its signature and the EL as for any other envelope. the head is recomputed afterwards, the cached one would not see the new FULL variants until the next block import. lodestar-geth-5 restarted with this change synced through 17547 to head, but the restart re-anchored it at 16960 so 17533 came in as a new block with its envelope and the early-return path was not exercised live, it is covered by the unit tests same known gap as #10005: the envelope of the sync target block itself (no children yet) is imported even if its payload later turns out to be orphaned builds on #10005 (merged)
**Motivation**
- log peer clients when it serves orphaned payloads, so that we can
monitor and work with them to avoid it
- in the future, we can consider to also penalize peers in this
situation
**Description**
- remove the log inside `BeaconChain.processBlocks()`
- new ProcessBlocksResult:
```typescript
export type ProcessBlocksResult = {
orphaned: OrphanedPayloadEnvelope[];
skipped: boolean;
};
```
and return it from `processChainSegments`
- then range sync pick it up and log
read the parent slot from `latest_block_header` instead of the bid as per ethereum/consensus-specs#5554 also updates to [v1.7.0-beta.0](https://github.com/ethereum/consensus-specs/releases/tag/v1.7.0-beta.0): - update the spec test vectors and specrefs - rename the payload timeliness committee SSZ types - update `MAX_SIGNED_INCLUSION_LIST_SIZE` the voluntary exit gossip changes are left for a follow-up
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This follows up on the squashed Zig integration in #9632 and selects the state-transition implementation once during node or historical-worker setup. TypeScript remains the default;
--chain.nativeStateView=trueenables the experimental native implementation and requires a restart to change. Factories, descendants, reloads, and archived-block replay retain that choice.The native adapter handles full and blinded block serialization, while the TypeScript path keeps using block objects without additional serialization. Startup checks reject a mismatched anchor backend, native transitions reject Gloas and later forks, and native/historical metrics have distinct names. Native rewards APIs remain unsupported and are documented as an experimental limitation.
Depends on lodestar-z #648, pinned to
d7e17aa0eb05f0b6e5ad71b9a5cf8b0d6ffce29d. That PR provides immutable setup configuration, worker resource isolation, input validation, and native memory accounting. Tree pools allocate on first state construction so default TypeScript/BLS workers avoid roughly 440 MB of unused memory per environment. It also honors canonical milliseconds over deprecated slot-duration defaults, fixing an actual six-second devnet startup failure. Sync committee lookups return the complete cache promised by their declarations, fixing a devnet failure that prevented native validators from obtaining sync duties. This branch merges currentunstableinto the #9632 squash; the foundation changes follow that merge in separate commits.Validation:
Repeated observer replay in this two-peer topology hit range-request rate limits; checkpoint recovery restored agreement. Both backends also logged publication failures where no peer subscribed to an attestation subnet. These test-environment limitations are recorded without changing peer scoring.
This is a pre-Gloas foundation for controlled fleet testing, with no automatic implementation fallback or full API parity claim.