blockvolume reports Ready=True after SmartWAL recovery skips CRC-failed committed record

[pingqiu]

Summary

Phase 34 D4 SmartWAL corruption gate now reaches the product assertion and fails for a real product reason: after a real SmartWAL WAL-record CRC corruption, blockvolume detects the CRC mismatch during recovery, skips the record, recovers to a later frontier, and the read-only status surface still reports the volume as Ready=True reason=first_volume_verified.

This is at minimum a status-surface correctness bug. The durability safety of skipping a mid-history committed record needs engine-owner review.

Evidence

QA run: 20260529-232752-b23c
Scenario: testops/scenarios/helm-smartwal-corrupt-restart-chain.yaml
Source commit under test: 03fe9ae testops: keep smartwal corruption gate scoped to volume
Finding doc: internal/docs/qa-assignments/phase34-d4-smartwal-corrupt-finding.md

Corruption landed in the real SmartWAL WAL region:

target_record_offset=6016
mutated_offset=6047
target_offset_inside_wal=true
target_offset_inside_extent=false

Recovery log shows the product detected the corruption:

smartwal: recovery CRC mismatch LSN=45 LBA=0 expected=80824a23 actual=91e6685b — skipping
smartwal: recovery: 40 LBAs verified, 1 torn, frontier=59
blockvolume: durable recovered: recovered LSN=59

Status surface after corruption:

smartwal_corruption_status_surface=failed
ready_true_after_corruption=true
blocked_true_after_corruption=false
reason_after_corruption=first_volume_verified

operator-snapshot.json reports the same volume as:

{
  "status": "ready",
  "reason_code": "first_volume_verified",
  "conditions": [{"type": "Ready", "status": "True"}]
}

Problem

Layer 1: confirmed status-surface bug

A volume with a detected WAL CRC failure is indistinguishable from a clean first-volume success. Per the Phase 32 negative-first status contract, corrupted or contradictory evidence must not surface as Ready=True reason=first_volume_verified.

Expected behavior should be one of:

• Ready=False / Blocked=True with a stable reason such as wal_record_crc_failed, wal_integrity_fault, or durable_recovery_failed.
• Ready=Unknown / EvidenceStale=True if the system cannot classify the recovery outcome safely.
• A Recovered/Degraded condition if engine-owner review concludes the skip is safe but still materially different from a clean recovery.

Layer 2: engine-owner review required

The corrupted record is LSN=45; recovery reached frontier=59. The system skipped LSN 45 and kept later records 46..59.

This may be safe only if LSN 45 had already been checkpointed to stable extents before replay. If it was committed-but-not-flushed, this is silent loss of an acknowledged write from the middle of the log while later writes are kept.

Torn-tail discard is normally safe; mid-history CRC-failure skip is not obviously safe and needs explicit durability semantics.

Acceptance criteria

• The D4 gate must no longer observe Ready=True reason=first_volume_verified after a detected SmartWAL CRC mismatch.
• The product must expose a stable condition/reason for WAL integrity/recovery faults, or intentionally classify the evidence as Unknown rather than clean Ready.
• Engine-owner review documents whether mid-history CRC-failed committed records can be skipped safely. If not safe, recovery must refuse/fence/mark degraded instead of silently continuing.
• testops/scenarios/helm-smartwal-corrupt-restart-chain.yaml should remain red until the product behavior is fixed or explicitly re-scoped.

Non-goals

• Do not weaken the D4 gate to pass by accepting false Ready.
• Do not classify this as a scenario failure; the gate now reaches the product assertion.
• Do not broaden this into a full WAL chaos matrix before resolving the single mid-history CRC case.