Audit pre-Cancun GeneralStateTest coverage in execution-specs

[danceratopz]

Top-level objective

Our end goal is to archive ethereum/tests and ethereum/legacytests once the execution-specs test release provides complete replacement coverage. This issue is one audit subtask under that meta-goal: establish whether all relevant pre-Cancun GeneralStateTest scenarios are covered, identify gaps, and make the result reproducible.

Blockchain-only tests and other legacy suites will need their own audit or migration subtasks before either repository can be archived.

Problem

execution-spec-tests PR #1442 imported the active GeneralStateTests from an ethereum/tests snapshot and filled them from Cancun onward. Tests that had already become legacy-only were outside that input set. The Weld subsequently moved the resulting static-test collection into execution-specs, preserving that historical boundary.

We now have concrete evidence of a missing consensus-regression case:

• RevertPrecompiledTouch_d0g0v0.json was generated from a GeneralStateTest, so this is not a blockchain-test-format limitation.
• The pre-Cancun source test was removed from the active ethereum/tests set in May 2024 and remained available through ethereum/legacytests.
• The similarly named test now in execution-specs is valid from Cancun and funds the precompile accounts, so it cannot exercise deletion of the empty RIPEMD-160 account under the historical Spurious Dragon behavior.
• EELS explicitly implements the block 2,675,119 RIPEMD-160 touched-account exception, but the unified fixture release does not currently contain the historical regression scenario that would exercise it.

This example also shows why filename matching is insufficient: a current test can share the legacy test's name and broad control flow without covering the same consensus condition.

Goal

Produce a complete and reviewable mapping from pre-Cancun GeneralStateTest scenarios to their coverage in execution-specs, then create focused follow-up work for every confirmed gap. The resulting evidence should satisfy the GeneralStateTests portion of the top-level archival checklist.

Coverage model

Use an authored scenario combined with each materially distinct fork-semantic regime as the unit of coverage. Do not count every generated data/gas/value (d/g/v) expansion as an independent authored test, and do not require a one-to-one file port when a modern test demonstrably exercises the same rule and assertions.

A scenario should be considered covered only when the current test reaches the same consensus behavior with equivalent material preconditions and post-state assertions on a relevant fork. Renamed or consolidated tests may qualify with evidence. Obsolete, redundant, or intentionally unsupported cases should receive an explicit rationale rather than silently disappearing from the inventory.

Proposed work

1. Establish a canonical pre-Cancun GeneralStateTest inventory from the last complete pre-removal ethereum/tests source snapshot, cross-checked against the corresponding ethereum/legacytests state and generated blockchain fixtures.
2. Normalize each source scenario into the coverage unit above and record its relevant fork-semantic regimes, provenance, and legacy fixture references.
3. Map each unit to execution-specs Python tests, ported static tests, or another demonstrably equivalent current test using provenance markers first and semantic review where provenance is absent.
4. Assign every unit one disposition: directly ported, semantically covered, missing, intentionally not ported with rationale, or unresolved.
5. Emit both a machine-readable ledger and a human-readable report, including evidence for every non-trivial equivalence decision.
6. Open focused follow-up issues or PRs for confirmed gaps, starting with the historical RIPEMD-160 touched-account deletion case.
7. Make the audit reproducible so it can be rerun as current tests change and used as an archival gate rather than a one-time spreadsheet.

Deliverables

• A versioned inventory of pre-Cancun GeneralStateTest scenario/fork-semantic units with source provenance.
• A machine-readable mapping from every inventory unit to its execution-specs disposition and supporting evidence.
• A summarized coverage report listing confirmed gaps, justified exclusions, and unresolved cases.
• Focused follow-up issues for confirmed missing coverage.
• A documented, rerunnable command or workflow that refreshes the report.

Acceptance criteria

• Every pre-Cancun GeneralStateTest unit in the canonical inventory has exactly one disposition and a supporting source or coverage reference.
• Semantic matches are reviewed against material preconditions, executed behavior, fork validity, and post-state assertions rather than accepted by filename alone.
• The known RevertPrecompiledTouch empty-RIPEMD-account scenario is reported as missing unless equivalent historical-fork coverage is added before the audit lands.
• Every confirmed gap has a focused follow-up issue or linked implementation PR; every intentional exclusion has a written rationale.
• Any unresolved unit remains an explicit blocker for the GeneralStateTests archival gate.
• The top-level repository-archival tracker is updated with the report and the resulting GeneralStateTests gate status.

Out of scope

• Auditing blockchain-only scenarios or non-GeneralStateTest suites such as transaction, difficulty, RLP, and VM tests; these require sibling subtasks.
• Porting every confirmed gap within this audit issue.
• Archiving either legacy repository before all suite-level coverage gates are complete.
• Requiring byte-for-byte fixture identity when semantic equivalence can be demonstrated.

Acknowledgements

H/t @taratorio and @chfast for flagging the broader risk of missing pre-Cancun coverage.

Related evidence

• Historical legacy fixture
• GeneralStateTests integration in execution-spec-tests
• Yellow Paper Appendix K.1

[chfast]

There is big gap in execution-specs: it cannot have empty accounts in the state. This prevents porting a lot of state clearing scenarios.

For the K1, I would suggest implement it only for one fork. This is enough to sync Mainnet and will not polute newer forks.

[taratorio]

Another gap between EEST<>legacytests I stumbled across is related to YP+EIP-684 storage clearing (without EIP-7610):

• nonce=0,code=0,storage!=0
• create collision circumvents EIP-684 and initialises a new account with empty storage (as per yellow paper)

The corresponding test fixtures without EIP-7610 can be found at this PR ethereum/legacytests#18
With them our serial execution passed successfully but our parallel execution failed, meaning we hadn't handled that properly in our parallel exec flow for Glamsterdam. EEST latest releases didn't catch this.

More information can be found at https://gist.github.com/taratorio/8e584cf8a678b57df6282d5fff069877

[spencer-tb]

Will prio this tomorrow, we want to discuss 7610 and 8253 on ACDT next week :)