Add runner checkpoint reports

Author: lemon-mint

RELEASE_SCOPE.md

@@ -93,7 +93,7 @@ This matrix is the current simulation/local-loopback fault-tolerance envelope. A
 | Observability | `examples/kv/cmd/kvnode/main.go`; `examples/kv/cmd/kvnode/main_test.go`; admin `/metrics` exposes low-cardinality storage-fault, transport-drop, EPaxos instance/executed, and send-queue gauges without new dependencies; `go test -tags kvnode ./examples/kv/cmd/kvnode -run 'TestHandleMetricsReportsLowCardinalityAdminState|TestAPIMuxSeparationRoutesOnlyPlaneEndpoints' -count=1`; `go test -tags kvnode ./examples/kv/cmd/kvnode -count=1` |
 | Production limits documentation | `README.md` support boundary names the production library surface, example-service boundary, cluster-size limits, timing limits, quorum non-claims, finite-model limits, HTTP body/scan limits, binary-value routes, latest-read scope, checksum fail-fast behavior, and checkpoint-backed corruption-recovery limits; `EPAXOS.MD`; `MODEL_EQ_REPORT.MD`; `bash tests/audit_repo.sh`; `bash tests/release_scope_audit.sh` |
 | API contracts documentation | `README.md` API contracts plus `EPAXOS.MD` document `RawNode`/`Ready`/`Advance`, persistence-before-send/apply, idempotent committed application, checksum errors, transport codec ownership, exact-byte conflict keys, zero-copy proposal ownership, scan consistency scope, and binary-value behavior; `bash tests/audit_repo.sh`; `bash tests/release_scope_audit.sh` |
-| Operations readiness artifact audit | Example/operator artifacts exist and are audit-gated, not production-proven: `deploy/systemd/kvnode@.service`; `deploy/systemd/kvnode.env.example`; `examples/kv/cmd/kvcheckpoint`; `examples/kv/cmd/kvcheckpoint/main_test.go`; `tests/kvnode_systemd_manifest_audit.sh`; `tests/kvnode_incident_tabletop_drill.sh`; `tests/kvnode_local_capacity_drill.sh`; `tests/kvnode_local_runner.go`; `docs/operations/kvnode-data-lifecycle-incident-runbook.md`; `docs/operations/kvnode-upgrade-rollback.md`; `tests/kvnode_capacity_envelope.sh`; `tests/operations_readiness_audit.sh`; `bash tests/operations_readiness_audit.sh`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode data`; `tests/ci.sh` now runs the operations readiness audit before release-scope and repository audits. The local Go runner data mode writes `data-lifecycle-summary.txt`, runs offline checkpoint/verify/restore/repair on a stopped local node, restarts it, verifies catch-up canaries on all nodes, and preserves `none-target-environment-data-lifecycle-drill-still-required`. |
+| Operations readiness artifact audit | Example/operator artifacts exist and are audit-gated, not production-proven: `deploy/systemd/kvnode@.service`; `deploy/systemd/kvnode.env.example`; `examples/kv/cmd/kvcheckpoint`; `examples/kv/cmd/kvcheckpoint/main_test.go`; `tests/kvnode_systemd_manifest_audit.sh`; `tests/kvnode_incident_tabletop_drill.sh`; `tests/kvnode_local_capacity_drill.sh`; `tests/kvnode_local_runner.go`; `docs/operations/kvnode-data-lifecycle-incident-runbook.md`; `docs/operations/kvnode-upgrade-rollback.md`; `tests/kvnode_capacity_envelope.sh`; `tests/operations_readiness_audit.sh`; `bash tests/operations_readiness_audit.sh`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode data`; `tests/ci.sh` now runs the operations readiness audit before release-scope and repository audits. The local Go runner data mode writes `data-lifecycle-summary.txt`, runs offline checkpoint/verify/restore/repair on a stopped local node, writes and validates distinct `checkpoint-report.env`, `verify-report.env`, `restore-report.env`, and `repair-report.env` helper reports before summary, restarts the node, verifies catch-up canaries on all nodes, and preserves `none-target-environment-data-lifecycle-drill-still-required`. |
 | Evidence bundle and go/no-go workflow | `release/EPAXOS_READINESS_EVIDENCE.md`; `tests/go_no_go_workflow.sh`; `tests/ci.sh`; `bash tests/go_no_go_workflow.sh` returns the current `No-go.` decision and lists open release items; `bash tests/release_scope_audit.sh` checks the evidence/workflow paths. |
 
 TryPreAccept message-path coverage note: `tests/tla_model_check.sh` now runs `tla/EPaxosTryPreAcceptMessagePath.cfg`, `tla/EPaxosTryPreAcceptMessagePathFive.cfg`, and `tla/EPaxosTryPreAcceptMessagePathSeven.cfg`; each finite 3/5/7 config covers follower `MsgTryPreAccept` commit-only, stale/conflict reject, duplicate matching re-ack without durable rewrite, fresh durable ack, and coordinator `MsgTryPreAcceptResp` stale restart, older/duplicate OK ignore, first OK below quorum, pre-seeded quorum immediate accept, and OK slow-quorum accept.
@@ -124,7 +124,7 @@ Config chain recovery note: `tests/tla_model_check.sh` now runs `tla/EPaxosConfi
 | --- | --- |
 | Broader formal model coverage | Finite configured TLC models are closed above, including bounded prepare branch-priority/try-witness coverage, finite 3-, 5-, and 7-replica Accept-Deps optimized-recovery evidence coverage, finite 3-, 5-, and 7-replica abstract TryPreAccept response branch-slice coverage, finite 3-, 5-, and 7-replica TryPreAccept message-path coverage, finite 3-, 5-, and 7-replica committed-conflict evidence-query guard/fail-closed coverage, one finite three-voter committed-conflict evidence-staleness request-scoping slice (`tla/EPaxosEvidenceStaleness.cfg` generated `6/6` states), finite 3-, 5-, and 7-replica uncommitted-conflict force/defer quorum coverage, finite configuration-barrier/add/remove/chain pinning coverage, one finite normal configuration-transition retry-timer slice (`tla/EPaxosConfigTransitionRetry.cfg` generated `8/8` states), one finite normal configuration-transition response de-duplication slice (`tla/EPaxosConfigTransitionDedup.cfg` generated `16/16` states), one finite durable configuration replay slice, finite config recovery-after-removal, recovery-after-addition, lost/duplicate response de-duplication, and recovery retry-timer slices (`tla/EPaxosConfigRecovery.cfg` generated `44/30` states, `tla/EPaxosConfigAddRecovery.cfg` generated `15/15` states, `tla/EPaxosConfigRecoveryDedup.cfg` generated `11/11` states, and `tla/EPaxosConfigRecoveryRetry.cfg` generated `8/8` states), a finite rollback-allocation next-instance/skip/apply-order check, and a finite `TOQClockDiscipline.tla` bounded-skew/bounded-delay contract. The TOQ operational-clock boundary is now documented in `EPAXOS.MD` and `MODEL_EQ_REPORT.MD`: the core consumes embedder-provided clock, one-way-delay, and sync-group values, but does not implement synchronization, measurement, drift monitoring, or target-environment validation. Remaining open: arbitrary/general recovery under configuration changes beyond the finite recovery slices, arbitrary membership histories, arbitrary durable histories, joint consensus, arbitrary message loss and retry/rebroadcast behavior, complete optimized-recovery branch parity, unbounded proofs, external target proof, synchronized-clock implementation, one-way-delay measurement, runtime drift enforcement, and operational clock-discipline proof. |
 | Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract, emits `release_claim=none-target-environment-deployment-manifest-still-required`, and keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`; these artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim, so target-environment deployment execution remains open. |
-| Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, `KVNODE_CHECKPOINT_REPORT=/path/report.env` lets successful helper operations write `status=example-operator-report` plus `release_claim=none-target-environment-data-lifecycle-drill-still-required`, `tests/kvnode_local_runner.go --mode data` stops one local loopback node and runs offline checkpoint/verify/restore/repair on a stopped local node before restart/catch-up verification, `data-lifecycle-summary.txt` records `data_lifecycle=offline-checkpoint-verify-restore-repair` plus `none-target-environment-data-lifecycle-drill-still-required`, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, helper reports, checksum-mismatch, local data-lifecycle drill, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. |
+| Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, `KVNODE_CHECKPOINT_REPORT=/path/report.env` lets successful helper operations write `status=example-operator-report` plus `release_claim=none-target-environment-data-lifecycle-drill-still-required`, `tests/kvnode_local_runner.go --mode data` stops one local loopback node and runs offline checkpoint/verify/restore/repair on a stopped local node before restart/catch-up verification, the runner uses distinct `checkpoint-report.env`, `verify-report.env`, `restore-report.env`, and `repair-report.env` paths under `data-lifecycle/*-report.env` and validates each report's `status`, `operation`, and `release_claim` before writing `data-lifecycle-summary.txt`, those reports record `operation=checkpoint`, `operation=verify`, `operation=restore`, `operation=repair`, and `result=success`, `data-lifecycle-summary.txt` records `data_lifecycle=offline-checkpoint-verify-restore-repair`, `reports=checkpoint-report.env,verify-report.env,restore-report.env,repair-report.env`, and `none-target-environment-data-lifecycle-drill-still-required`, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, helper reports, checksum-mismatch, local data-lifecycle drill, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open; target-environment backup/restore/disaster-recovery drill remains open. |
 | Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; its `metadata.env` and `summary.md` emit `release_claim=none-target-environment-capacity-results-still-required` plus bounded single-line `environment_label` and `workload_label` provenance fields; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs while preserving the same release-claim non-claim and defaulting provenance to `environment_label=local-loopback` and `workload_label=local-capacity-drill`; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape and records bounded write/read/scan latency plus admin metric samples. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, a non-claim metadata sample with 1 op, 16-byte values, scan limit 1, peer_count=3, and `release_claim=none-target-environment-capacity-results-still-required`, and a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, and `status=local-go-runner-only`. This is workstation harness evidence only; measured target-environment capacity results remain open because target-environment capacity measurement remains open. |
 | Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster and writes `release_claim=none-target-environment-operator-review-still-required` into raw tabletop evidence; `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open, so target-environment incident-response operator review remains open. |
 

docs/operations/kvnode-data-lifecycle-incident-runbook.md

@@ -330,11 +330,11 @@ KVNODE_GO_RUNNER_RUN=yes KVNODE_GO_RUNNER_MODE=data \
   2>&1 | tee "${EVIDENCE_DIR}/go-runner-data-lifecycle-local.txt"
 ```
 
-This is local loopback evidence only. The generated `data-lifecycle-summary.txt`, `checkpoint.log`, `verify.log`, `restore.log`, `repair.log`, and final `summary.txt` should be retained with the transcript. The summary includes `status=local-go-runner-only`, `data_lifecycle=offline-checkpoint-verify-restore-repair`, and `release_claim=none-target-environment-data-lifecycle-drill-still-required`; it does not replace a reviewed target-environment backup/restore/disaster-recovery drill.
+This is local loopback evidence only. The generated `data-lifecycle-summary.txt`, `checkpoint.log`, `verify.log`, `restore.log`, `repair.log`, distinct `checkpoint-report.env`, `verify-report.env`, `restore-report.env`, `repair-report.env`, and final `summary.txt` should be retained with the transcript. The runner sets a unique `KVNODE_CHECKPOINT_REPORT` path for each helper operation, validates each report's `status=example-operator-report`, `operation`, and `release_claim` fields before writing the data-lifecycle summary, and records `reports=checkpoint-report.env,verify-report.env,restore-report.env,repair-report.env`. The summary includes `status=local-go-runner-only`, `data_lifecycle=offline-checkpoint-verify-restore-repair`, and `release_claim=none-target-environment-data-lifecycle-drill-still-required`; it does not replace a reviewed target-environment backup/restore/disaster-recovery drill.
 
 Evidence to retain:
 
-- `metadata.env`, `data-lifecycle-summary.txt`, `summary.txt`, and the four helper logs from the script evidence directory.
+- `metadata.env`, `data-lifecycle-summary.txt`, `summary.txt`, the four helper logs, and the four helper report files from the script evidence directory.
 - The full runner transcript, including the preserved `run_dir`.
 - Confirmation that the drill used the disposable runner data directory only and stopped the selected node before each offline `kvcheckpoint` operation.