Fix LFS restore stalls caused by genesis placeholders

[spreston8]

Six defects on the LFS restore path, all reproduced on the RGB testnet shard and most of them the same failure mode: state written when the truth was not yet knowable, then trusted later.

A validator bonded but never proposing gets the genesis hash in its latest-message slot, network-uniform by design so every node agrees on a validator with no blocks. Four consumers mistake that seed for testimony:

• Parent selection. Genesis is held, so it survives the unheld-slot abstention and becomes a parent candidate. Every ancestry walk bounded by a height-0 candidate runs to the bottom of the DAG — below a restored node's horizon. Two joiners sat 31 hours making zero proposals, the propose check demanding a block at 140077 they could never hold. own_testimony states the rule once (held, and signed by that validator); floor_of_view's inline copy is removed in favour of it.
• The LFS acceptance window. A joiner seeds latest from the approved block's justifications, so genesis arrives at height 0 and the window lowers to -1, after which the gate admits the whole chain. A join that should take 75 blocks walked 3,098 heights and never reached Running. The lowering still applies to a validator's own latest message, whose parents the restore needs.
• Mergeable-channel GC. Genesis has no main parent, so counting a placeholder slot collapses the common-ancestor search and the pass refuses every deletion — the GC collects nothing for as long as a bonded validator stays silent.
• The estimator. Its tips feed the read API, which takes the first off a set, so a placeholder there lets show_main_chain return a chain rooted at height 0.

Two more on the same path, found while verifying the first:

• The finalizer froze on a restored node. CliqueOracle::get_corresponding_weight_map reads the target's MAIN PARENT's weight map, so holding a block is not enough to judge it, and a lagging validator's tip one hop above the horizon held the LFB for every validator. A tip at or below the current floor is no longer a candidate — its own floor and its frontier are both bounded by its height, so it cannot raise a maximum — and a tip that still will not derive abstains, as fork_choice_floor already does. Confined to floor_of_view: the consensus derivation used by propose and validate is untouched, since a restored node quietly deriving a different floor than a full node is a fork, not a stall.
• Shipped floor-cache entries pointed below the horizon. They were written without checking that the floor and frontier they name are held — the rule seed_floor_caches applies to the anchor's seed and documents. The solicited set is every restored block, the anchor included, and put_cached_floor overwrites, so a peer's unvalidated entry landed on top of the seed verified on the line above.

Separately, the repeat-deploy carrier watermark was written at startup, where an empty database cannot tell a node about to run a genesis ceremony from one about to restore. It wrote 0 — claiming the index is complete over every height — so a restored node read an index miss as proof a deploy had no carrier and admitted a repeat the full node rejects. It is now written where the answer is known: 0 at ceremony completion, and at restore completion the lowest height above which the node holds every block it will need. Not the DAG minimum: a shipped genesis sits at 0 while the node holds nothing between it and the restore band. Coverage also stops at the acceptance bound, because below it a block is fetched only as a parent of an accepted one, so contiguity there is not completeness.

Also here: parent collapsing is an optimisation, so a coverage walk that leaves held history is now unproven rather than fatal to the snapshot; and both missing-block log sites emit the backtrace they already capture, at debug under f1r3.trace.absence. That backtrace is what identified the parent-selection walk after code reading had failed to.

Verified on the shard, fixed build against unfixed controls on the same shard:

	before	after
LFS window	Minimum block height is -1, 3,098 heights, never ready	approved - 75, 77 heights, ready in ~2 min
propose	0 in 31h, 7,670 stall warnings	proposing on a 10s cadence, 0 warnings
coverage	coverage_from: 0 over 157,000 unheld heights	coverage_from: 157520 while the DAG minimum is 0
finalizer	LFB frozen at 118 while the shard reached 133, 50 holds	derives past the lagging validator

Reviewer notes:

• Block shape changes on shards with a silent validator. Its placeholder slot no longer adds genesis as a declared parent, so multi_parent_casper_merge_spec and limited_parent_depth_spec now expect 2 parents where they asserted 3. Genesis is an ancestor of every block, so it adds nothing to a merge. Parent selection is proposer-local and validators replay declared parents, so blocks from unfixed proposers remain valid.
• One finalizer contract changed. a_walk_crossing_the_retention_edge_holds_the_cycle asserted AbsenceHold; a lone undecidable tip now abstains and the answer is NoAdvance, so it is renamed ..._abstains_the_tip. The abstention logs the tip and the missing block.
• carrier_watermark_initializes_once_per_database had its expectations changed — the old assertion ("an empty database is complete from the first insert") is the defect stated as a contract. ensure_carrier_watermark now returns Option<i64>, meaning "the watermark if known". The M1 defect is silent in production, with no error or stall, and was caught only by reading the log line the fix added.

Co-Authored-By: Claude noreply@anthropic.com

[spreston8]

Reviewed at 4b682a1 against dev. Suggestions below.

High

1. Worth fixing the finalizer hold here rather than after. On a shard carrying both idle shapes — validator5 bonded with --heartbeat-disabled from genesis, validator4 paused after proposing — dev's placeholder drags the window to -1, the joiner downloads to genesis and finalizes normally (119 → 147, 0 holds). At this head the placeholder correctly stops lowering the window, and the same joiner reaches Running in 30s and then never finalizes: LFB frozen at 118 while the shard reached 133, 50× "the floor walk needs a block this node does not hold", missing block at height 6. The defect is dev's, but the lowering was hiding it, so this branch turns a visible stall into a silent one.

   The absence is raised inside derive_floor's frontier resolution, on the warm path. Capturing a backtrace in held_meta and printing it where floor_of_view catches BlockNotHeld gives the same frames on all 50 holds: CliqueOracle ← derive_floor ← floor_of_view ← compute_last_finalized_block. A horizon guard in cold_parent_frontier never fires, so the cold descent is not the path. Fits: floor_of_block on the stale tip succeeds and persists a frontier pivot, and the next derivation walks the warm path from that pivot into history the restore never fetched. The exact line is open — held_number on the cached pivot, or on a spine candidate around ft_witnessed_exact.

   Two things it is not. Not the seeding path: the joiner logs "Seeded the anchor's finalized floor and frontier from the approved block" (floor_number 115) with neither warning present, and seed_floor_caches caches a floor for the anchor alone, so a tip ~110 below the anchor has nothing in its ancestry passing through it. Not per-slot abstention either: floor_of_view abstaining a slot whose floor does not derive, the way fork_choice_floor does, was built and run and never fires — per-tip floor_of_block succeeds for all four tips, so the hold is unchanged.

2. Worth clamping coverage to max(contiguous_coverage_start, min_height - 1). Rows below the window are fetched only as parents of accepted blocks, so a deep secondary parent makes a lower row contiguous without it being complete; a run above logged coverage_from: 5 against min_height: 8. In the differential harness, with carrier c26 absent and sibling Z26 held, a repeat at height 29 gives Valid at W=26 and Undecidable at W=27 — same deploy, two verdicts, against InvalidRepeatDeploy on a full node.

Medium

3. Worth applying own_testimony at the remaining consumers, or saying why not. mergeable_channels_gc.rs:311 passes every latest-message hash to deepest_common_strict_ancestor, whose main_parent collect short-circuits on genesis (probe: with_seed=None against without_seed=Some(4)), and :225 then refuses every deletion — GC collects nothing while any bonded validator has never proposed.

Low

4. Worth carrying justification.validator past lfs_block_requester.rs:893-896 so the LFS check is sender == validator rather than sender non-empty, and matching the missing formatting across the two new absence log sites (PrettyPrinter in one, raw hex in the other).

Co-Authored-By: Claude noreply@anthropic.com

[spreston8]

All four addressed. Every fix has a test that was red first, and each was red-checked by reverting the fix.

1. The finalizer hold — neither candidate line was the one.

CliqueOracle::get_corresponding_weight_map reads the target's MAIN PARENT's weight map, not the target's own (clique_oracle.rs:133-143). lookup_unsafe on an unheld main parent yields MissingBlock, which From<KvStoreError> for CasperError turns into BlockNotHeld. That is the CliqueOracle frame in the backtrace with no held_meta frame beneath it: the absence is raised in the oracle's own lookup. Holding a block is not enough — its main parent must be held too, so a tip one hop above the horizon fails.

The abstention patch counted 0 for a reason that turned out to matter: floor_of_block on validator4's stale tip succeeded from a shipped floor-cache entry, so the condition never saw it, and the failure came one step later when that same entry was used as a frontier pivot. With entries validated (below), the abstention stops being a no-op and is in.

Three parts, each its own red test:

• floor_of_view builds its candidate tips from slots above the current floor. Both of a tip's candidates are bounded by its height, so one at or below the floor cannot raise the maximum; deriving it anyway is what walked a lagging validator's history. The slot still counts as agreement.
• A tip whose floor or frontier will not derive abstains, as fork_choice_floor already does.
• Confined to floor_of_view. finalized_floor as called from propose and validate is untouched, so the consensus-visible floor stays node-identical — a restored node quietly deriving a different floor than a full node is a fork, not a stall.

Contract change worth noting: a_walk_crossing_the_retention_edge_holds_the_cycle asserted AbsenceHold; a lone undecidable tip now abstains and the answer is NoAdvance. Renamed to ..._abstains_the_tip. The abstention logs the tip and the missing block, so the diagnosis the hold carried is not lost.

Found while verifying that, and fixed here: apply_floor_cache_entries wrote shipped entries without checking that the floor and frontier they name are held — the rule seed_floor_caches applies to the anchor's seed and documents. Worse, the solicited set is every block in the restored DAG, put_cached_floor is an unconditional overwrite, and request_floor_cache runs on the line after seed_floor_caches: the peer's unvalidated entry for the anchor landed on top of the seed just verified. Entries whose values are not held are now refused, and an entry never replaces a floor the node already has.

2. Coverage clamp — min_height - 1 is correct, the counter-argument was not.

The case raised against it was a block at min_height - 1 whose only in-cone child sits below the bound. That cannot exist: a block's number exceeds its parents', so every child of a block at min_height - 1 sits at or above the bound, is accepted, and requests all of its parents. The row is complete. Clamped at min_height - 1, with the induction in the comment since it is not obvious from the code.

3. GC consumer. common_strict_main_chain_ancestors now filters to own testimony. It also returns Result instead of swallowing storage errors: dropping a slot there raises the anchor and collects MORE, so a silent error would have been unsafe in the deletion direction.

4. The requester carries justification.validator, so the check is sender == validator rather than sender non-empty — genesis is the only unsigned block today, so the weaker test was an accident of what the requester kept. Both new absence log sites use PrettyPrinter.

Co-Authored-By: Claude noreply@anthropic.com

[spreston8]

Re-reviewed at 7f9e5de against dev. All four addressed. Three smaller suggestions.

Medium

1. Worth deriving before probing in floor_of_view. Each tip now resolves twice a cycle: the probe runs floor_of_block + parent_frontier, then finalized_floor runs both again with the same arguments. floor_of_block is a warm cache read, but parent_frontier caches nothing — incremental_frontier re-collects the band [tip..pivot], reads a weight map per band block and repeats every ft_witnessed_exact call. That band is the tip-to-frontier distance, so the doubling is largest exactly when finality is already lagging. Calling finalized_floor first and probing only on BlockNotHeld gives the identical answer — it errors precisely when some tip is undecidable — with nothing added to the healthy path. On the shard that produced the original hold the probe abstained nothing across 120s: the stale tip sits near height 11 against a floor of 116, so the height filter removes it and the abstention never fires.

Low

2. Worth applying own_testimony in estimator too, or saying why not. snapshot.rs:539-548 filters only on invalid blocks, so the placeholder stays in the tip set, and block_api.rs:1402 takes tip_hashes.first() off a HashSet: show_main_chain can return a chain rooted at genesis. Pre-existing, not introduced here.

3. Worth refreshing the description. It opens "Three defects on the LFS restore path" and its before/after table predates the finalizer fix, the floor-entry validation and the GC fix.

Co-Authored-By: Claude noreply@anthropic.com