Add incident tabletop report output

Author: lemon-mint

RELEASE_SCOPE.md

@@ -126,7 +126,7 @@ Config chain recovery note: `tests/tla_model_check.sh` now runs `tla/EPaxosConfi
 | Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract, emits `release_claim=none-target-environment-deployment-manifest-still-required`, keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`, and supports `KVNODE_SYSTEMD_MANIFEST_REPORT=/path/report.env` for a machine-readable example/operator report. The report writes `status=example-operator-report`, `artifact=systemd-manifest-audit`, `rendered_exec=...`, `systemd_analyze=skipped|verified`, and `release_claim=none-target-environment-deployment-manifest-still-required`; the manifest report remains a local static-render non-claim and does not prove a target deployment. `KVNODE_GO_RUNNER_RUN=yes go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode deployment` now adds local static-render plus loopback process evidence: it runs the static manifest audit, starts a local direct-args loopback cluster with the manifest example default deadline/body/scan settings, writes `deployment-manifest-summary.txt`, `systemd-manifest-report.env`, and `systemd-manifest-audit.log`, records `systemd_manifest_audit=passed`, `launch_path=direct-local-runner-args`, `launch_defaults=request_deadline_ms=5000,peer_deadline_ms=2000,max_client_body_bytes=1048576,max_peer_body_bytes=1048576,max_admin_body_bytes=65536,max_scan_limit=1000`, `deployment_manifest_ran=`, and the same deployment non-claim. These artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim, so target-environment deployment execution remains open. |
 | Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, `KVNODE_CHECKPOINT_REPORT=/path/report.env` lets successful helper operations write `status=example-operator-report` plus `release_claim=none-target-environment-data-lifecycle-drill-still-required`, `tests/kvnode_local_runner.go --mode data` stops one local loopback node and runs offline checkpoint/verify/restore/repair on a stopped local node before restart/catch-up verification, the runner uses distinct `checkpoint-report.env`, `verify-report.env`, `restore-report.env`, and `repair-report.env` paths under `data-lifecycle/*-report.env` and validates each report's `status`, `operation`, and `release_claim` before writing `data-lifecycle-summary.txt`, those reports record `operation=checkpoint`, `operation=verify`, `operation=restore`, `operation=repair`, and `result=success`, `data-lifecycle-summary.txt` records `data_lifecycle=offline-checkpoint-verify-restore-repair`, `reports=checkpoint-report.env,verify-report.env,restore-report.env,repair-report.env`, and `none-target-environment-data-lifecycle-drill-still-required`, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, helper reports, checksum-mismatch, local data-lifecycle drill, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open; target-environment backup/restore/disaster-recovery drill remains open. |
 | Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; its `metadata.env` and `summary.md` emit `release_claim=none-target-environment-capacity-results-still-required` plus bounded single-line `environment_label` and `workload_label` provenance fields; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs while preserving the same release-claim non-claim and defaulting provenance to `environment_label=local-loopback` and `workload_label=local-capacity-drill`; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape, records bounded write/read/scan latency plus admin metric samples, accepts `KVNODE_GO_RUNNER_ENVIRONMENT_LABEL` and `KVNODE_GO_RUNNER_WORKLOAD_LABEL`, defaults them to `environment_label=local-loopback` and `workload_label=local-go-runner`, validates both as non-empty single-line values without `=` and with maximum length 128, and writes the labels to `metadata.env`, `capacity-summary.txt`, and final `summary.txt` when capacity mode runs. These checks state that the custom Go runner capacity labels document `KVNODE_GO_RUNNER_ENVIRONMENT_LABEL` and `KVNODE_GO_RUNNER_WORKLOAD_LABEL`; the defaulting custom Go runner provenance to `environment_label=local-loopback` and `workload_label=local-go-runner` behavior remains explicit; custom Go runner validates label values as non-empty, single-line, without `=`, and at most 128 characters; custom Go runner writes `environment_label` and `workload_label` to `metadata.env`, `capacity-summary.txt`, and capacity `summary.txt` when `capacity_ran=true`. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, a non-claim metadata sample with 1 op, 16-byte values, scan limit 1, peer_count=3, `release_claim=none-target-environment-capacity-results-still-required`, `environment_label=local-loopback`, and `workload_label=local-capacity-drill`, plus a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, `status=local-go-runner-only`, and a custom Go runner provenance sample with `environment_label=local-loopback`, `workload_label=local-go-runner-capacity`, and `latency_rows=3`. This is workstation harness evidence only; measured target-environment capacity results remain open because target-environment capacity measurement remains open. |
-| Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster and writes `release_claim=none-target-environment-operator-review-still-required` into raw tabletop evidence; `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open, so target-environment incident-response operator review remains open. |
+| Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster, writes `release_claim=none-target-environment-operator-review-still-required` into raw tabletop evidence, and supports `KVNODE_INCIDENT_TABLETOP_REPORT=/path/report.env` for a machine-readable example/operator incident report. The report writes `status=example-operator-report`, `artifact=incident-tabletop-drill`, `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, `canaries=baseline-and-after-clear-visible-on-all-nodes`, `operator_review=not-performed`, and `release_claim=none-target-environment-operator-review-still-required`; this remains local loopback/tabletop evidence and does not replace operator review. `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open, so target-environment incident-response operator review remains open. |
 
 TryPreAccept response branch-slice note: `tla/EPaxosTryPreAcceptBranches.tla` is only a finite abstract scenario/stage model for stale restart, committed evidence ignore/fail-closed, direct/forced accept, one uncommitted deferral with duplicate suppression, and OK slow-quorum accept. Its only quorum detail is `okVotes >= SlowQuorum`, it runs for 3/5/7, and complete optimized-recovery branch parity beyond this finite slice remains open.
 

docs/operations/kvnode-data-lifecycle-incident-runbook.md

@@ -345,16 +345,18 @@ The repository includes a local loopback tabletop harness for the test-fault bra
 Local tabletop command:
 
 ```sh
-KVNODE_INCIDENT_TABLETOP_RUN=yes bash tests/kvnode_incident_tabletop_drill.sh \
+KVNODE_INCIDENT_TABLETOP_REPORT="${EVIDENCE_DIR}/incident-tabletop-report.env" \
+  KVNODE_INCIDENT_TABLETOP_RUN=yes bash tests/kvnode_incident_tabletop_drill.sh \
   2>&1 | tee "${EVIDENCE_DIR}/incident-tabletop-local.txt"
 ```
 
 This is local tabletop evidence only. It does not replace operator review, target-environment execution, real host/network fault handling, credential rotation, or disaster-recovery sign-off.
 
 Evidence to retain:
 
-- `metadata.env`, `summary.txt`, and per-node `/livez`, `/readyz`, `/metrics`, `/faults/storage`, and `/faults/transport` captures from the script evidence directory.
-- The full tabletop transcript, including `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, and `status=local-tabletop-only`.
+- `metadata.env`, `summary.txt`, optional `incident-tabletop-report.env`, and per-node `/livez`, `/readyz`, `/metrics`, `/faults/storage`, and `/faults/transport` captures from the script evidence directory.
+- The full tabletop transcript, including `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, `status=local-tabletop-only`, and `release_claim=none-target-environment-operator-review-still-required`.
+  If the optional report path is used, retain its `status=example-operator-report`, `artifact=incident-tabletop-drill`, `operator_review=not-performed`, and `release_claim=none-target-environment-operator-review-still-required` fields.
 - Confirmation that no storage directories were wiped or restored during the network-partition branch.
 
 ## Incident response: storage failure

release/EPAXOS_READINESS_EVIDENCE.md

@@ -197,6 +197,7 @@ Non-claims remain explicit: No target-environment remote claim, no in-place Pebb
 - `go test ./examples/kv/cmd/kvcheckpoint -count=1` passed after adding the maintained offline checkpoint helper; the CLI verifies checkpoints before `restore` and `repair` replaces a data directory.
 - `go test ./examples/kv/cmd/kvcheckpoint -count=1` passed after adding `KVNODE_CHECKPOINT_REPORT` coverage; the tests prove successful `checkpoint`, `verify`, `restore`, and `repair` operations write `status=example-operator-report`, quoted paths, and `release_claim=none-target-environment-data-lifecycle-drill-still-required`, and that report-write failures do not mask underlying command failures.
 - `KVNODE_INCIDENT_TABLETOP_RUN=yes KVNODE_INCIDENT_BASE_PORT=30080 KVNODE_INCIDENT_PEER_BASE_PORT=30180 KVNODE_INCIDENT_ADMIN_BASE_PORT=30280 KVNODE_INCIDENT_OUT_DIR=/tmp/kvnode-incident-nonclaim-20260709 bash tests/kvnode_incident_tabletop_drill.sh` passed on local loopback after adding the incident tabletop harness non-claim to generated metadata; archived sample files are `local://kvnode-incident-nonclaim-20260709-metadata.env` and `local://kvnode-incident-nonclaim-20260709-summary.txt`; generated evidence reported `status=local-tabletop-only`, `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, post-clear canaries visible on all nodes, and `release_claim=none-target-environment-operator-review-still-required`.
+- `KVNODE_INCIDENT_TABLETOP_RUN=yes KVNODE_INCIDENT_BASE_PORT=35080 KVNODE_INCIDENT_PEER_BASE_PORT=35180 KVNODE_INCIDENT_ADMIN_BASE_PORT=35280 KVNODE_INCIDENT_OUT_DIR=/tmp/kvnode-incident-report-20260709 KVNODE_INCIDENT_TABLETOP_REPORT=/tmp/kvnode-incident-report-20260709/report.env bash tests/kvnode_incident_tabletop_drill.sh` passed on local loopback after adding the incident tabletop report output; archived sample files are `local://kvnode-incident-report-20260709-report.env`, `local://kvnode-incident-report-20260709-summary.txt`, and `local://kvnode-incident-report-20260709-metadata.env`; generated evidence reported `status=example-operator-report`, `artifact=incident-tabletop-drill`, `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, `canaries=baseline-and-after-clear-visible-on-all-nodes`, `operator_review=not-performed`, `release_claim=none-target-environment-operator-review-still-required`, and report mode `0600`. This remains local loopback/tabletop evidence only; operator-reviewed target-environment tabletop or live drill evidence remains open.
 - `KVNODE_LOCAL_CAPACITY_RUN=yes bash tests/kvnode_local_capacity_drill.sh` passed on local loopback after adding the three-node capacity wrapper; it ran `tests/kvnode_capacity_envelope.sh` with `peer_count=3`, 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, PID/data-dir resource sampling, and `status=local-loopback-only`.
 - `KVNODE_LOCAL_CAPACITY_RUN=yes KVNODE_LOCAL_CAPACITY_BASE_PORT=29080 KVNODE_LOCAL_CAPACITY_PEER_BASE_PORT=29180 KVNODE_LOCAL_CAPACITY_ADMIN_BASE_PORT=29280 KVNODE_CAPACITY_OPS_PER_PHASE=1 KVNODE_CAPACITY_VALUE_BYTES=16 KVNODE_CAPACITY_SCAN_LIMITS=1 bash tests/kvnode_local_capacity_drill.sh` passed on local loopback after adding the machine-readable capacity non-claim; archived sample files are `local://kvnode-local-capacity-nonclaim-20260709-wrapper-metadata.env`, `local://kvnode-local-capacity-nonclaim-20260709-wrapper-summary.txt`, `local://kvnode-local-capacity-nonclaim-20260709-capacity-metadata.env`, `local://kvnode-local-capacity-nonclaim-20260709-capacity-summary.md`, `local://kvnode-local-capacity-nonclaim-20260709-latency.csv`, and `local://kvnode-local-capacity-nonclaim-20260709-resources.csv`; generated wrapper and harness metadata both include `release_claim=none-target-environment-capacity-results-still-required`, the sample recorded 3 HTTP operations against peer_count=3, and target-environment capacity measurement remains open.
 - `KVNODE_LOCAL_CAPACITY_RUN=yes KVNODE_LOCAL_CAPACITY_BASE_PORT=31080 KVNODE_LOCAL_CAPACITY_PEER_BASE_PORT=31180 KVNODE_LOCAL_CAPACITY_ADMIN_BASE_PORT=31280 KVNODE_LOCAL_CAPACITY_OUT_DIR=/tmp/kvnode-capacity-provenance-20260709 KVNODE_CAPACITY_OPS_PER_PHASE=1 KVNODE_CAPACITY_VALUE_BYTES=16 KVNODE_CAPACITY_SCAN_LIMITS=1 bash tests/kvnode_local_capacity_drill.sh` passed on local loopback after adding validated capacity provenance labels; archived sample files are `local://kvnode-capacity-provenance-20260709-wrapper-metadata.env`, `local://kvnode-capacity-provenance-20260709-wrapper-summary.txt`, `local://kvnode-capacity-provenance-20260709-capacity-metadata.env`, `local://kvnode-capacity-provenance-20260709-capacity-summary.md`, `local://kvnode-capacity-provenance-20260709-latency.csv`, and `local://kvnode-capacity-provenance-20260709-resources.csv`; wrapper and harness evidence both report `environment_label=local-loopback`, `workload_label=local-capacity-drill`, and `release_claim=none-target-environment-capacity-results-still-required`.
@@ -246,6 +247,7 @@ Non-claims remain explicit: No target-environment remote claim, no in-place Pebb
   - Local loopback old/new binary rolling-upgrade and binary-rollback harness. It requires explicit `KVNODE_UPGRADE_OLD_REF`, builds old/new binaries from clean git archives with `-trimpath -buildvcs=false`, records source-tree hashes and binary SHA-256s, rejects identical refs/source/binaries unless `KVNODE_UPGRADE_SMOKE_ONLY=yes`, exercises one-node-at-a-time upgrade and rollback, and verifies each upgraded/rolled node with a 204 write plus latest GET and barrier scan from all nodes. Binary rollback uses the node's current data; checkpoint restore is documented as a separate data-lifecycle fallback, not this mixed-version drill. The currently archived old/new refs differ only by timeout-outcome wording, so this is drill-mechanics evidence for distinct binaries, not broad protocol/storage compatibility evidence.
 - `tests/kvnode_incident_tabletop_drill.sh`
   - Local loopback tabletop harness only: starts a disposable three-node `kvnode` cluster, captures admin evidence, exercises and clears storage-fault and transport-fault test branches, verifies client canaries after clearing faults, and emits `status=local-tabletop-only` plus `release_claim=none-target-environment-operator-review-still-required` in generated evidence.
+  - Optional `KVNODE_INCIDENT_TABLETOP_REPORT=/path/report.env` writes a 0600 example/operator incident report after a successful local run with `status=example-operator-report`, `artifact=incident-tabletop-drill`, storage/transport fault results, canary results, `operator_review=not-performed`, and `release_claim=none-target-environment-operator-review-still-required`.
 - `tests/operations_readiness_audit.sh`
   - Audit for the operations artifacts above.
 - `tests/toolchain.env`, `.github/workflows/ci.yml`, and `tests/toolchain_audit.sh`

tests/kvnode_incident_tabletop_drill.sh

@@ -27,6 +27,8 @@ Optional inputs:
   KVNODE_INCIDENT_READY_ATTEMPTS   Readiness polling attempts. Default: 120
   KVNODE_INCIDENT_CURL_TIMEOUT     curl per-request timeout seconds. Default: 5
   KVNODE_INCIDENT_OUT_DIR          Evidence/run directory. Default: <tmp>/kvnode-incident-tabletop-<timestamp>
+  KVNODE_INCIDENT_TABLETOP_REPORT   Optional success report path. When set,
+                                   writes a 0600 example/operator report.
 
 Example:
   KVNODE_INCIDENT_TABLETOP_RUN=yes tests/kvnode_incident_tabletop_drill.sh
@@ -83,6 +85,8 @@ require_command mkdir
 require_command rm
 require_command cat
 require_command grep
+require_command dirname
+require_command chmod
 
 BASE_PORT="${KVNODE_INCIDENT_BASE_PORT:-24080}"
 PEER_BASE_PORT="${KVNODE_INCIDENT_PEER_BASE_PORT:-24180}"
@@ -292,6 +296,25 @@ assert_metrics_contains() {
   fi
 }
 
+write_report() {
+  local report_path="${KVNODE_INCIDENT_TABLETOP_REPORT:-}"
+  [[ -n "$report_path" ]] || return 0
+  [[ "$report_path" != "." && "$report_path" != "/" ]] || fail "KVNODE_INCIDENT_TABLETOP_REPORT-must-name-a-file"
+  mkdir -p "$(dirname "$report_path")"
+  {
+    echo "status=example-operator-report"
+    echo "artifact=incident-tabletop-drill"
+    printf 'evidence_dir=%q\n' "$EVIDENCE_DIR"
+    echo "storage_fault=exercised-and-cleared"
+    echo "transport_fault=exercised-and-cleared"
+    echo "canaries=baseline-and-after-clear-visible-on-all-nodes"
+    echo "operator_review=not-performed"
+    echo "release_claim=none-target-environment-operator-review-still-required"
+  } > "$report_path"
+  chmod 0600 "$report_path"
+  printf 'report=%q\n' "$report_path"
+}
+
 cat > "$EVIDENCE_DIR/metadata.env" <<EOF
 status=local-tabletop-only
 run_id=$run_id
@@ -360,5 +383,6 @@ canaries=baseline-and-after-clear-visible-on-all-nodes
 release_claim=none-target-environment-operator-review-still-required
 EOF
 
+write_report
 cat "$EVIDENCE_DIR/summary.txt"
 echo "kvnode-incident-tabletop status=pass evidence_dir=$EVIDENCE_DIR"