Add local data lifecycle runner drill

Author: lemon-mint

RELEASE_SCOPE.md

@@ -93,7 +93,7 @@ This matrix is the current simulation/local-loopback fault-tolerance envelope. A
 | Observability | `examples/kv/cmd/kvnode/main.go`; `examples/kv/cmd/kvnode/main_test.go`; admin `/metrics` exposes low-cardinality storage-fault, transport-drop, EPaxos instance/executed, and send-queue gauges without new dependencies; `go test -tags kvnode ./examples/kv/cmd/kvnode -run 'TestHandleMetricsReportsLowCardinalityAdminState|TestAPIMuxSeparationRoutesOnlyPlaneEndpoints' -count=1`; `go test -tags kvnode ./examples/kv/cmd/kvnode -count=1` |
 | Production limits documentation | `README.md` support boundary names the production library surface, example-service boundary, cluster-size limits, timing limits, quorum non-claims, finite-model limits, HTTP body/scan limits, binary-value routes, latest-read scope, checksum fail-fast behavior, and checkpoint-backed corruption-recovery limits; `EPAXOS.MD`; `MODEL_EQ_REPORT.MD`; `bash tests/audit_repo.sh`; `bash tests/release_scope_audit.sh` |
 | API contracts documentation | `README.md` API contracts plus `EPAXOS.MD` document `RawNode`/`Ready`/`Advance`, persistence-before-send/apply, idempotent committed application, checksum errors, transport codec ownership, exact-byte conflict keys, zero-copy proposal ownership, scan consistency scope, and binary-value behavior; `bash tests/audit_repo.sh`; `bash tests/release_scope_audit.sh` |
-| Operations readiness artifact audit | Example/operator artifacts exist and are audit-gated, not production-proven: `deploy/systemd/kvnode@.service`; `deploy/systemd/kvnode.env.example`; `examples/kv/cmd/kvcheckpoint`; `examples/kv/cmd/kvcheckpoint/main_test.go`; `tests/kvnode_systemd_manifest_audit.sh`; `tests/kvnode_incident_tabletop_drill.sh`; `tests/kvnode_local_capacity_drill.sh`; `tests/kvnode_local_runner.go`; `docs/operations/kvnode-data-lifecycle-incident-runbook.md`; `docs/operations/kvnode-upgrade-rollback.md`; `tests/kvnode_capacity_envelope.sh`; `tests/operations_readiness_audit.sh`; `bash tests/operations_readiness_audit.sh`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`; `tests/ci.sh` now runs the operations readiness audit before release-scope and repository audits. |
+| Operations readiness artifact audit | Example/operator artifacts exist and are audit-gated, not production-proven: `deploy/systemd/kvnode@.service`; `deploy/systemd/kvnode.env.example`; `examples/kv/cmd/kvcheckpoint`; `examples/kv/cmd/kvcheckpoint/main_test.go`; `tests/kvnode_systemd_manifest_audit.sh`; `tests/kvnode_incident_tabletop_drill.sh`; `tests/kvnode_local_capacity_drill.sh`; `tests/kvnode_local_runner.go`; `docs/operations/kvnode-data-lifecycle-incident-runbook.md`; `docs/operations/kvnode-upgrade-rollback.md`; `tests/kvnode_capacity_envelope.sh`; `tests/operations_readiness_audit.sh`; `bash tests/operations_readiness_audit.sh`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode data`; `tests/ci.sh` now runs the operations readiness audit before release-scope and repository audits. The local Go runner data mode writes `data-lifecycle-summary.txt`, runs offline checkpoint/verify/restore/repair on a stopped local node, restarts it, verifies catch-up canaries on all nodes, and preserves `none-target-environment-data-lifecycle-drill-still-required`. |
 | Evidence bundle and go/no-go workflow | `release/EPAXOS_READINESS_EVIDENCE.md`; `tests/go_no_go_workflow.sh`; `tests/ci.sh`; `bash tests/go_no_go_workflow.sh` returns the current `No-go.` decision and lists open release items; `bash tests/release_scope_audit.sh` checks the evidence/workflow paths. |
 
 TryPreAccept message-path coverage note: `tests/tla_model_check.sh` now runs `tla/EPaxosTryPreAcceptMessagePath.cfg`, `tla/EPaxosTryPreAcceptMessagePathFive.cfg`, and `tla/EPaxosTryPreAcceptMessagePathSeven.cfg`; each finite 3/5/7 config covers follower `MsgTryPreAccept` commit-only, stale/conflict reject, duplicate matching re-ack without durable rewrite, fresh durable ack, and coordinator `MsgTryPreAcceptResp` stale restart, older/duplicate OK ignore, first OK below quorum, pre-seeded quorum immediate accept, and OK slow-quorum accept.
@@ -124,7 +124,7 @@ Config chain recovery note: `tests/tla_model_check.sh` now runs `tla/EPaxosConfi
 | --- | --- |
 | Broader formal model coverage | Finite configured TLC models are closed above, including bounded prepare branch-priority/try-witness coverage, finite 3-, 5-, and 7-replica Accept-Deps optimized-recovery evidence coverage, finite 3-, 5-, and 7-replica abstract TryPreAccept response branch-slice coverage, finite 3-, 5-, and 7-replica TryPreAccept message-path coverage, finite 3-, 5-, and 7-replica committed-conflict evidence-query guard/fail-closed coverage, one finite three-voter committed-conflict evidence-staleness request-scoping slice (`tla/EPaxosEvidenceStaleness.cfg` generated `6/6` states), finite 3-, 5-, and 7-replica uncommitted-conflict force/defer quorum coverage, finite configuration-barrier/add/remove/chain pinning coverage, one finite normal configuration-transition retry-timer slice (`tla/EPaxosConfigTransitionRetry.cfg` generated `8/8` states), one finite normal configuration-transition response de-duplication slice (`tla/EPaxosConfigTransitionDedup.cfg` generated `16/16` states), one finite durable configuration replay slice, finite config recovery-after-removal, recovery-after-addition, lost/duplicate response de-duplication, and recovery retry-timer slices (`tla/EPaxosConfigRecovery.cfg` generated `44/30` states, `tla/EPaxosConfigAddRecovery.cfg` generated `15/15` states, `tla/EPaxosConfigRecoveryDedup.cfg` generated `11/11` states, and `tla/EPaxosConfigRecoveryRetry.cfg` generated `8/8` states), a finite rollback-allocation next-instance/skip/apply-order check, and a finite `TOQClockDiscipline.tla` bounded-skew/bounded-delay contract. Remaining open: arbitrary/general recovery under configuration changes beyond the finite recovery slices, arbitrary membership histories, arbitrary durable histories, joint consensus, arbitrary message loss and retry/rebroadcast behavior, complete optimized-recovery branch parity, unbounded proofs, external target proof, synchronized-clock implementation, one-way-delay measurement, and operational clock-discipline proof. |
 | Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract and keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`; these artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim. |
-| Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, checksum-mismatch, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. |
+| Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, `tests/kvnode_local_runner.go --mode data` stops one local loopback node and runs offline checkpoint/verify/restore/repair on a stopped local node before restart/catch-up verification, `data-lifecycle-summary.txt` records `data_lifecycle=offline-checkpoint-verify-restore-repair` plus `none-target-environment-data-lifecycle-drill-still-required`, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, checksum-mismatch, local data-lifecycle drill, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. |
 | Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape and records bounded write/read/scan latency plus admin metric samples. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, and peer_count=3, and a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, and `status=local-go-runner-only`. This is workstation harness evidence only; measured target-environment capacity results remain open. |
 | Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster; `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open. |
 
@@ -168,7 +168,7 @@ The review baseline is the current repository state after the EPaxos recovery, R
 - The chaos campaign is a named local CI gate; it closes only simulation/local-loopback checkpoint-backed single-node corruption recovery. It does not close in-place Pebble/WAL repair, checksum recomputation, corrupt-record deletion, synthesized reconstruction without a verified checkpoint, production deployment proof, target capacity proof, or incident drill evidence listed as open.
 - Request deadline budgets use explicit Go HTTP and handler deadlines; they are not OS-clock mutation evidence.
 - Clock pause handling covers deterministic logical RawNode tick pause/resume, not OS scheduler suspension or wall-clock pause.
-- VM rollback restart catch-up covers an in-memory RawNode/storage rollback and quorum catch-up without duplicate apply plus a finite TLA rollback-allocation check for one learned own ref and one defensive skip state; it does not close in-place bit-level disk repair, synthesized reconstruction without a verified checkpoint, multi-replica/quorum-loss recovery, unbounded rollback histories, target-environment deployment proof, or checkpoint-restore data-lifecycle drills.
+- VM rollback restart catch-up covers an in-memory RawNode/storage rollback and quorum catch-up without duplicate apply plus a finite TLA rollback-allocation check for one learned own ref and one defensive skip state; it does not close in-place bit-level disk repair, synthesized reconstruction without a verified checkpoint, multi-replica/quorum-loss recovery, unbounded rollback histories, target-environment deployment proof, or target-environment checkpoint-restore data-lifecycle drills.
 - API separation covers distinct client, peer-replication, and administrative HTTP listener/route planes; peer replication is POST-only and rejects other methods before body decode; TLS evidence is tracked separately under Transport security and does not add authentication or authorization.
 - Transport security covers optional TLS listeners plus CA-configured peer HTTPS client verification; it does not implement client-certificate authentication, HTTP authentication, or authorization.
 - Request size limits cover HTTP request body bytes on client, peer, and administrative planes; they do not bound scan result cardinality or keyspace traversal.

docs/operations/kvnode-data-lifecycle-incident-runbook.md

@@ -316,6 +316,26 @@ Evidence to retain:
 - Per-process `kvnode` logs showing stop, storage remove, restore, restart, and health check.
 
 
+## Local data lifecycle drill
+
+The build-tagged Go runner includes a local loopback data-lifecycle drill for the offline helper path. It starts a disposable three-node `kvnode` cluster, writes a pre-checkpoint canary, stops node 2 before opening its Pebble directory, runs `kvcheckpoint checkpoint`, `kvcheckpoint verify`, `kvcheckpoint restore`, restarts node 2, writes a post-restore canary, stops node 2 again, runs `kvcheckpoint repair`, restarts node 2, and verifies both canaries from all three client listeners.
+
+Local data lifecycle command:
+
+```sh
+KVNODE_GO_RUNNER_RUN=yes KVNODE_GO_RUNNER_MODE=data \
+  go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode data \
+  2>&1 | tee "${EVIDENCE_DIR}/go-runner-data-lifecycle-local.txt"
+```
+
+This is local loopback evidence only. The generated `data-lifecycle-summary.txt`, `checkpoint.log`, `verify.log`, `restore.log`, `repair.log`, and final `summary.txt` should be retained with the transcript. The summary includes `status=local-go-runner-only`, `data_lifecycle=offline-checkpoint-verify-restore-repair`, and `release_claim=none-target-environment-data-lifecycle-drill-still-required`; it does not replace a reviewed target-environment backup/restore/disaster-recovery drill.
+
+Evidence to retain:
+
+- `metadata.env`, `data-lifecycle-summary.txt`, `summary.txt`, and the four helper logs from the script evidence directory.
+- The full runner transcript, including the preserved `run_dir`.
+- Confirmation that the drill used the disposable runner data directory only and stopped the selected node before each offline `kvcheckpoint` operation.
+
 ## Local incident tabletop drill
 
 The repository includes a local loopback tabletop harness for the test-fault branches of the storage-failure and network-partition procedures. It starts a disposable three-node `kvnode` cluster, captures admin evidence files, injects and clears one storage fault plus bidirectional transport drops around one node, and verifies post-clear client canaries on all nodes.