Add local Go kvnode runner

Author: lemon-mint

RELEASE_SCOPE.md

@@ -10,7 +10,7 @@ A release claim is allowed only when the item is listed in **Closed release item
 
 No-go.
 
-The implemented EPaxos core, explicit TOQ core mode, deterministic timing subset, HTTP KV service-hardening changes, local fault campaign, local mixed-version binary drill, operations artifacts, evidence bundle, and go/no-go workflow are ready for reviewer assessment under a simulation/local-loopback release scope. The broader production-readiness request is not complete because broader formal proof, target-environment deployment and data-lifecycle drills, target capacity measurements, and incident-readiness exercises remain open below. External multi-host Jepsen histories are outside the current release evidence.
+The implemented EPaxos core, explicit TOQ core mode, deterministic timing subset, HTTP KV service-hardening changes, local fault campaign, local mixed-version binary drill, operations artifacts, local Go runner, evidence bundle, and go/no-go workflow are ready for reviewer assessment under a simulation/local-loopback release scope. The broader production-readiness request is not complete because broader formal proof, target-environment deployment and data-lifecycle drills, target capacity measurements, and incident-readiness exercises remain open below. External multi-host Jepsen histories are outside the current release evidence.
 
 ## Fault-tolerance target and evidence matrix
 
@@ -81,7 +81,7 @@ This matrix is the current simulation/local-loopback fault-tolerance envelope. A
 | Observability | `examples/kv/cmd/kvnode/main.go`; `examples/kv/cmd/kvnode/main_test.go`; admin `/metrics` exposes low-cardinality storage-fault, transport-drop, EPaxos instance/executed, and send-queue gauges without new dependencies; `go test -tags kvnode ./examples/kv/cmd/kvnode -run 'TestHandleMetricsReportsLowCardinalityAdminState|TestAPIMuxSeparationRoutesOnlyPlaneEndpoints' -count=1`; `go test -tags kvnode ./examples/kv/cmd/kvnode -count=1` |
 | Production limits documentation | `README.md` support boundary names the production library surface, example-service boundary, cluster-size limits, timing limits, quorum non-claims, finite-model limits, HTTP body/scan limits, binary-value routes, latest-read scope, checksum fail-fast behavior, and checkpoint-backed corruption-recovery limits; `EPAXOS.MD`; `MODEL_EQ_REPORT.MD`; `bash tests/audit_repo.sh`; `bash tests/release_scope_audit.sh` |
 | API contracts documentation | `README.md` API contracts plus `EPAXOS.MD` document `RawNode`/`Ready`/`Advance`, persistence-before-send/apply, idempotent committed application, checksum errors, transport codec ownership, exact-byte conflict keys, zero-copy proposal ownership, scan consistency scope, and binary-value behavior; `bash tests/audit_repo.sh`; `bash tests/release_scope_audit.sh` |
-| Operations readiness artifact audit | Example/operator artifacts exist and are audit-gated, not production-proven: `deploy/systemd/kvnode@.service`; `deploy/systemd/kvnode.env.example`; `examples/kv/cmd/kvcheckpoint`; `examples/kv/cmd/kvcheckpoint/main_test.go`; `tests/kvnode_systemd_manifest_audit.sh`; `tests/kvnode_incident_tabletop_drill.sh`; `tests/kvnode_local_capacity_drill.sh`; `docs/operations/kvnode-data-lifecycle-incident-runbook.md`; `docs/operations/kvnode-upgrade-rollback.md`; `tests/kvnode_capacity_envelope.sh`; `tests/operations_readiness_audit.sh`; `bash tests/operations_readiness_audit.sh`; `tests/ci.sh` now runs the operations readiness audit before release-scope and repository audits. |
+| Operations readiness artifact audit | Example/operator artifacts exist and are audit-gated, not production-proven: `deploy/systemd/kvnode@.service`; `deploy/systemd/kvnode.env.example`; `examples/kv/cmd/kvcheckpoint`; `examples/kv/cmd/kvcheckpoint/main_test.go`; `tests/kvnode_systemd_manifest_audit.sh`; `tests/kvnode_incident_tabletop_drill.sh`; `tests/kvnode_local_capacity_drill.sh`; `tests/kvnode_local_runner.go`; `docs/operations/kvnode-data-lifecycle-incident-runbook.md`; `docs/operations/kvnode-upgrade-rollback.md`; `tests/kvnode_capacity_envelope.sh`; `tests/operations_readiness_audit.sh`; `bash tests/operations_readiness_audit.sh`; `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`; `tests/ci.sh` now runs the operations readiness audit before release-scope and repository audits. |
 | Evidence bundle and go/no-go workflow | `release/EPAXOS_READINESS_EVIDENCE.md`; `tests/go_no_go_workflow.sh`; `tests/ci.sh`; `bash tests/go_no_go_workflow.sh` returns the current `No-go.` decision and lists open release items; `bash tests/release_scope_audit.sh` checks the evidence/workflow paths. |
 
 TryPreAccept message-path coverage note: `tests/tla_model_check.sh` now runs `tla/EPaxosTryPreAcceptMessagePath.cfg`, `tla/EPaxosTryPreAcceptMessagePathFive.cfg`, and `tla/EPaxosTryPreAcceptMessagePathSeven.cfg`; each finite 3/5/7 config covers follower `MsgTryPreAccept` commit-only, stale/conflict reject, duplicate matching re-ack without durable rewrite, fresh durable ack, and coordinator `MsgTryPreAcceptResp` stale restart, older/duplicate OK ignore, first OK below quorum, pre-seeded quorum immediate accept, and OK slow-quorum accept.
@@ -95,8 +95,8 @@ TryConflict force/defer quorum note: `tests/tla_model_check.sh` now runs `tla/EP
 | Broader formal model coverage | Finite configured TLC models are closed above, including bounded prepare branch-priority/try-witness coverage, finite 3-, 5-, and 7-replica Accept-Deps optimized-recovery evidence coverage, finite 3-, 5-, and 7-replica abstract TryPreAccept response branch-slice coverage, finite 3-, 5-, and 7-replica committed-conflict evidence-query guard/fail-closed coverage, finite configuration-barrier coverage, one finite add-voter configuration-transition pinning check, one finite remove-voter configuration-transition pinning check, one finite add-then-remove configuration-chain pinning check, a finite rollback-allocation next-instance/skip/apply-order check, and a finite `TOQClockDiscipline` bounded-skew/bounded-delay `ProcessAt` contract; unbounded proof, complete optimized-recovery branch parity beyond the finite TryPreAccept branch slice, even-size optimized-quorum proof, operational synchronized-clock/OWD-measurement implementation proof for TOQ deployments, arbitrary membership-change proof, arbitrary multi-step reconfiguration chains, joint consensus, recovery under configuration changes, durable replay, full rollback-history proof, and arbitrary application/state-machine semantics remain open. |
 | Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract and keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`; these artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim. |
 | Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, checksum-mismatch, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. |
-| Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs; `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample and a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, and peer_count=3. This is workstation harness evidence only; measured target-environment capacity results remain open. |
-| Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open. |
+| Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape and records bounded write/read/scan latency plus admin metric samples. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, and peer_count=3, and a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, and `status=local-go-runner-only`. This is workstation harness evidence only; measured target-environment capacity results remain open. |
+| Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster; `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open. |
 
 TryPreAccept response branch-slice note: `tla/EPaxosTryPreAcceptBranches.tla` is only a finite abstract scenario/stage model for stale restart, committed evidence ignore/fail-closed, direct/forced accept, one uncommitted deferral with duplicate suppression, and OK slow-quorum accept. Its only quorum detail is `okVotes >= SlowQuorum`, it runs for 3/5/7, and complete optimized-recovery branch parity beyond this finite slice remains open.
 

release/EPAXOS_READINESS_EVIDENCE.md

@@ -114,6 +114,8 @@ Non-claims remain explicit: No target-environment remote claim, no in-place Pebb
 - `go test ./examples/kv/cmd/kvcheckpoint -count=1` passed after adding the maintained offline checkpoint helper; the CLI verifies checkpoints before `restore` and `repair` replaces a data directory.
 - `KVNODE_INCIDENT_TABLETOP_RUN=yes bash tests/kvnode_incident_tabletop_drill.sh` passed on local loopback after adding the incident tabletop harness; the generated summary reported `status=local-tabletop-only`, `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, and post-clear canaries visible on all nodes.
 - `KVNODE_LOCAL_CAPACITY_RUN=yes bash tests/kvnode_local_capacity_drill.sh` passed on local loopback after adding the three-node capacity wrapper; it ran `tests/kvnode_capacity_envelope.sh` with `peer_count=3`, 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, PID/data-dir resource sampling, and `status=local-loopback-only`.
+- `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help` passed in this session and compiled the build-tagged custom local Go runner.
+- `KVNODE_GO_RUNNER_RUN=yes KVNODE_GO_RUNNER_OPS_PER_PHASE=2 KVNODE_GO_RUNNER_VALUE_BYTES=16 KVNODE_GO_RUNNER_SCAN_LIMITS=1 go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode all` passed on local loopback after adding the custom Go runner; archived sample files are `local://kvnode-go-runner-20260709-summary.txt`, `local://kvnode-go-runner-20260709-incident-summary.txt`, `local://kvnode-go-runner-20260709-capacity-summary.txt`, `local://kvnode-go-runner-20260709-latency.csv`, and `local://kvnode-go-runner-20260709-resources.csv`; generated summaries reported `status=local-go-runner-only`, storage/transport faults exercised and cleared, post-clear canaries visible on all nodes, 5 local capacity latency rows, p50 0.033010833s, p95 0.036474708s, p99 0.036474708s, and no target-environment capacity or operator-review claim.
 
 ### New or updated artifacts in this evidence bundle scope
 
@@ -149,6 +151,8 @@ Non-claims remain explicit: No target-environment remote claim, no in-place Pebb
   - Opt-in bounded capacity-envelope collection harness.
 - `tests/kvnode_local_capacity_drill.sh`
   - Local loopback capacity wrapper only: starts a disposable three-node `kvnode` cluster, runs `tests/kvnode_capacity_envelope.sh` against all three client/admin listeners with PIDs and data directories supplied, and emits `status=local-loopback-only`.
+- `tests/kvnode_local_runner.go`
+  - Custom local Go runner only: build-tagged with `kvnode_local_runner`, starts a disposable three-node `kvnode` loopback cluster, exercises storage-fault and transport-fault administrative branches, verifies post-clear client canaries, records bounded write/read/scan latency and admin metric samples, emits `status=local-go-runner-only`, and preserves target-environment capacity/operator-review non-claims.
 - `tests/kvnode_mixed_version_drill.sh`
   - Local loopback old/new binary rolling-upgrade and binary-rollback harness. It requires explicit `KVNODE_UPGRADE_OLD_REF`, builds old/new binaries from clean git archives with `-trimpath -buildvcs=false`, records source-tree hashes and binary SHA-256s, rejects identical refs/source/binaries unless `KVNODE_UPGRADE_SMOKE_ONLY=yes`, exercises one-node-at-a-time upgrade and rollback, and verifies each upgraded/rolled node with a 204 write plus latest GET and barrier scan from all nodes. Binary rollback uses the node's current data; checkpoint restore is documented as a separate data-lifecycle fallback, not this mixed-version drill. The currently archived old/new refs differ only by timeout-outcome wording, so this is drill-mechanics evidence for distinct binaries, not broad protocol/storage compatibility evidence.
 - `tests/kvnode_incident_tabletop_drill.sh`
@@ -218,8 +222,8 @@ The following blockers are still listed in `RELEASE_SCOPE.md` and prevent a go d
 - `tla/EPaxosTryConflictForce.tla` adds finite 3/5/7 quorum-arithmetic coverage for uncommitted-conflict force/defer decisions, but arbitrary recovery histories and complete optimized-recovery branch parity remain open.
 - Deployment manifest artifacts are example/operator material only; `tests/kvnode_systemd_manifest_audit.sh` renders and audits the example `ExecStart` contract, but reviewed execution under a target system manager, container, or orchestration environment remains open.
 - Data lifecycle now has a maintained local offline `kvcheckpoint` helper plus runbook/audit evidence, but a reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. The mixed-version drill's binary rollback keeps current data and does not exercise checkpoint restore.
-- Target-environment capacity-envelope measurements remain open; local evidence now includes a single-node workstation sample plus a three-node loopback wrapper sample, but neither is a measured target-environment capacity result.
-- Incident readiness has runbook/audit evidence plus a local loopback tabletop harness for storage-failure and network-partition test-fault branches, but operator-reviewed target-environment tabletop or live drill evidence remains open.
+- Target-environment capacity-envelope measurements remain open; local evidence now includes a single-node workstation sample, a three-node loopback shell wrapper sample, and a custom local Go runner sample, but none is a measured target-environment capacity result.
+- Incident readiness has runbook/audit evidence, a local loopback tabletop shell harness, and a custom local Go runner for storage-failure and network-partition test-fault branches, but operator-reviewed target-environment tabletop or live drill evidence remains open.
 
 ## Final workflow command