Add manifest-derived deployment rehearsal

Author: lemon-mint

RELEASE_SCOPE.md

@@ -134,7 +134,7 @@ Config chain recovery lost-response retry note: `tests/tla_model_check.sh` now r
 | Item | Current state |
 | --- | --- |
 | Broader formal model coverage | Finite configured TLC models are closed above, including bounded prepare branch-priority/try-witness coverage, finite 3-, 5-, and 7-replica Accept-Deps optimized-recovery evidence coverage, finite 3-, 5-, and 7-replica abstract TryPreAccept response branch-slice coverage, finite 3-, 5-, and 7-replica TryPreAccept message-path coverage, finite 3-, 5-, and 7-replica committed-conflict evidence-query guard/fail-closed coverage, one finite three-voter committed-conflict evidence-staleness request-scoping slice (`tla/EPaxosEvidenceStaleness.cfg` generated `6/6` states), finite 3-, 5-, and 7-replica uncommitted-conflict force/defer quorum coverage, finite F<=3 Accept-Deps optimized-recovery decision-tree branch parity (`tla/EPaxosOptimizedRecoveryDecisionTree.cfg` generated `78/78` states), finite configuration-barrier/add/remove/chain pinning coverage, one finite add-then-remove configuration-chain retry-timer slice (`tla/EPaxosConfigChainTransitionRetry.cfg` generated `8/8` states), one finite add-then-remove configuration-chain lost-response-before-retry slice (`tla/EPaxosConfigChainTransitionLostResponseRetry.cfg` generated `20/20` states), one finite normal configuration-transition retry-timer slice (`tla/EPaxosConfigTransitionRetry.cfg` generated `8/8` states), one finite normal configuration-transition lost-response-before-retry slice (`tla/EPaxosConfigTransitionLostResponseRetry.cfg` generated `20/20` states), one finite normal configuration-transition response de-duplication slice (`tla/EPaxosConfigTransitionDedup.cfg` generated `16/16` states), one finite durable configuration replay slice, finite config recovery-after-removal, recovery-after-addition, lost/duplicate response de-duplication, recovery retry-timer, explicit lost-response-before-retry, and mid-chain lost-response-before-retry slices (`tla/EPaxosConfigRecovery.cfg` generated `44/30` states, `tla/EPaxosConfigAddRecovery.cfg` generated `15/15` states, `tla/EPaxosConfigRecoveryDedup.cfg` generated `11/11` states, `tla/EPaxosConfigRecoveryRetry.cfg` generated `8/8` states, `tla/EPaxosConfigRecoveryLostResponseRetry.cfg` generated `11/11` states, and `tla/EPaxosConfigChainRecoveryLostResponseRetry.cfg` generated `32/22` states), a finite rollback-allocation next-instance/skip/apply-order check, and a finite `TOQClockDiscipline.tla` bounded-skew/bounded-delay contract. The TOQ operational-clock boundary is now documented in `EPAXOS.MD` and `MODEL_EQ_REPORT.MD`: the core consumes embedder-provided clock, one-way-delay, and sync-group values, but does not implement synchronization, measurement, drift monitoring, or target-environment validation. Remaining open: arbitrary/general recovery under configuration changes beyond the finite recovery slices, arbitrary membership histories, arbitrary durable histories, joint consensus, arbitrary message loss beyond the named finite pre-retry loss slices, arbitrary retry/rebroadcast behavior, unbounded proofs, external target proof, synchronized-clock implementation, one-way-delay measurement, runtime drift enforcement, and operational clock-discipline proof. |
-| Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract, emits `release_claim=none-target-environment-deployment-manifest-still-required`, keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`, and supports `KVNODE_SYSTEMD_MANIFEST_REPORT=/path/report.env` for a machine-readable example/operator report. The report writes `status=example-operator-report`, `artifact=systemd-manifest-audit`, `rendered_exec=...`, `systemd_analyze=skipped|verified`, and `release_claim=none-target-environment-deployment-manifest-still-required`; the manifest report remains a local static-render non-claim and does not prove a target deployment. `KVNODE_GO_RUNNER_RUN=yes go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode deployment` now adds local static-render plus loopback process evidence: it runs the static manifest audit, starts a local direct-args loopback cluster with the manifest example default deadline/body/scan settings, writes `deployment-manifest-summary.txt`, `systemd-manifest-report.env`, and `systemd-manifest-audit.log`, records `systemd_manifest_audit=passed`, `launch_path=direct-local-runner-args`, `launch_defaults=request_deadline_ms=5000,peer_deadline_ms=2000,max_client_body_bytes=1048576,max_peer_body_bytes=1048576,max_admin_body_bytes=65536,max_scan_limit=1000`, `deployment_manifest_ran=`, and the same deployment non-claim. These artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim, so target-environment deployment execution remains open. |
+| Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract, emits `release_claim=none-target-environment-deployment-manifest-still-required`, keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`, and supports `KVNODE_SYSTEMD_MANIFEST_REPORT=/path/report.env` for a machine-readable example/operator report. The report writes `status=example-operator-report`, `artifact=systemd-manifest-audit`, `rendered_exec=...`, `systemd_analyze=skipped|verified`, and `release_claim=none-target-environment-deployment-manifest-still-required`; the manifest report remains a local static-render non-claim and does not prove a target deployment. `KVNODE_GO_RUNNER_RUN=yes go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode deployment` now adds manifest-derived local substitution evidence: it runs the static manifest audit, writes `deployment-manifest-summary.txt`, `systemd-manifest-report.env`, `systemd-manifest-audit.log`, and `deployment-manifest-local-launch.env`, starts a local loopback cluster from per-node substituted EnvironmentFile values rather than ad hoc direct args, substitutes the built temp binary, temp data directories, and loopback listeners for target paths, records `systemd_manifest_audit=passed`, `manifest_report=systemd-manifest-report.env`, `local_launch_report=deployment-manifest-local-launch.env`, `artifact=systemd-manifest-local-launch`, `systemd_exec_contract=rendered-then-substituted`, `node_1_exec_argv_json=...`, `launch_path=manifest-derived-local-substitution`, `local_substitution=temp-binary-temp-data-loopback-listeners`, `launch_defaults=request_deadline_ms=5000,peer_deadline_ms=2000,max_client_body_bytes=1048576,max_peer_body_bytes=1048576,max_admin_body_bytes=65536,max_scan_limit=1000`, `non_claim=local-static-render-plus-manifest-derived-loopback-process-check-only`, `deployment_manifest_ran=`, and the same deployment release non-claim. These artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim, so target-environment deployment execution remains open. |
 | Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, `KVNODE_CHECKPOINT_REPORT=/path/report.env` lets successful helper operations write `status=example-operator-report` plus `release_claim=none-target-environment-data-lifecycle-drill-still-required`, `tests/kvnode_local_runner.go --mode data` stops one local loopback node and runs offline checkpoint/verify/restore/repair on a stopped local node before restart/catch-up verification, the runner uses distinct `checkpoint-report.env`, `verify-report.env`, `restore-report.env`, and `repair-report.env` paths under `data-lifecycle/*-report.env` and validates each report's `status`, `operation`, and `release_claim` before writing `data-lifecycle-summary.txt`, those reports record `operation=checkpoint`, `operation=verify`, `operation=restore`, `operation=repair`, and `result=success`, `data-lifecycle-summary.txt` records `data_lifecycle=offline-checkpoint-verify-restore-repair`, `reports=checkpoint-report.env,verify-report.env,restore-report.env,repair-report.env`, and `none-target-environment-data-lifecycle-drill-still-required`, and `KVNODE_GO_RUNNER_DATA_LIFECYCLE_REPORT=/path/report.env` lets the runner write a 0600 machine-readable example/operator report after a successful local data drill. The runner report writes `status=example-operator-report`, `artifact=data-lifecycle-drill`, `data_lifecycle=offline-checkpoint-verify-restore-repair`, `checkpoint=verified`, `reports=checkpoint-report.env,verify-report.env,restore-report.env,repair-report.env`, restore/repair/canary result fields, and `release_claim=none-target-environment-data-lifecycle-drill-still-required`; report paths `.` and `/` are rejected before the local cluster starts, and final `summary.txt` records `data_lifecycle_report=...` when set. `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, helper reports, the consolidated local runner report, checksum-mismatch, local data-lifecycle drill, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open; target-environment backup/restore/disaster-recovery drill remains open. |
 | Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; its `metadata.env` and `summary.md` emit `release_claim=none-target-environment-capacity-results-still-required` plus bounded single-line `environment_label` and `workload_label` provenance fields, and it supports `KVNODE_CAPACITY_REPORT=/path/report.env` for a machine-readable example/operator capacity report. The report writes `status=example-operator-report`, `artifact=capacity-envelope-sample`, `environment_label`, `workload_label`, `peer_count`, `operation_count=...`, `throughput_ops_per_second=...`, `latency_file=latency.csv`, `resources_file=resources.csv`, and `release_claim=none-target-environment-capacity-results-still-required`; report paths `.` and `/` are rejected before traffic, and successful reports are written with mode 0600. `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs while preserving the same release-claim non-claim, defaulting provenance to `environment_label=local-loopback` and `workload_label=local-capacity-drill`, defaulting the capacity report to `capacity/capacity-report.env`, and writing the resolved `capacity_report` path to wrapper `summary.txt`; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape, records bounded write/read/scan latency plus admin metric samples, accepts `KVNODE_GO_RUNNER_ENVIRONMENT_LABEL` and `KVNODE_GO_RUNNER_WORKLOAD_LABEL`, defaults them to `environment_label=local-loopback` and `workload_label=local-go-runner`, validates both as non-empty single-line values without `=` and with maximum length 128, and writes the labels to `metadata.env`, `capacity-summary.txt`, and final `summary.txt` when capacity mode runs. These checks state that the custom Go runner capacity labels document `KVNODE_GO_RUNNER_ENVIRONMENT_LABEL` and `KVNODE_GO_RUNNER_WORKLOAD_LABEL`; the defaulting custom Go runner provenance to `environment_label=local-loopback` and `workload_label=local-go-runner` behavior remains explicit; custom Go runner validates label values as non-empty, single-line, without `=`, and at most 128 characters; custom Go runner writes `environment_label` and `workload_label` to `metadata.env`, `capacity-summary.txt`, and capacity `summary.txt` when `capacity_ran=true`. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, a non-claim metadata sample with 1 op, 16-byte values, scan limit 1, peer_count=3, `release_claim=none-target-environment-capacity-results-still-required`, `environment_label=local-loopback`, and `workload_label=local-capacity-drill`, plus a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, `status=local-go-runner-only`, a custom Go runner provenance sample with `environment_label=local-loopback`, `workload_label=local-go-runner-capacity`, and `latency_rows=3`, and a capacity report sample with `environment_label=local-loopback`, `workload_label=local-capacity-report`, `operation_count=3`, and report mode 0600. This is workstation harness evidence only; measured target-environment capacity results remain open because target-environment capacity measurement remains open. |
 | Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster, writes `release_claim=none-target-environment-operator-review-still-required` into raw tabletop evidence, and supports `KVNODE_INCIDENT_TABLETOP_REPORT=/path/report.env` for a machine-readable example/operator incident report. The report writes `status=example-operator-report`, `artifact=incident-tabletop-drill`, `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, `canaries=baseline-and-after-clear-visible-on-all-nodes`, `operator_review=not-performed`, and `release_claim=none-target-environment-operator-review-still-required`; this remains local loopback/tabletop evidence and does not replace operator review. `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open, so target-environment incident-response operator review remains open. |