Skip to content

Commit 9af071f

Browse files
committed
Record capacity harness nonclaim
1 parent b610bcb commit 9af071f

6 files changed

Lines changed: 15 additions & 4 deletions

RELEASE_SCOPE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ Config chain recovery note: `tests/tla_model_check.sh` now runs `tla/EPaxosConfi
125125
| Broader formal model coverage | Finite configured TLC models are closed above, including bounded prepare branch-priority/try-witness coverage, finite 3-, 5-, and 7-replica Accept-Deps optimized-recovery evidence coverage, finite 3-, 5-, and 7-replica abstract TryPreAccept response branch-slice coverage, finite 3-, 5-, and 7-replica TryPreAccept message-path coverage, finite 3-, 5-, and 7-replica committed-conflict evidence-query guard/fail-closed coverage, one finite three-voter committed-conflict evidence-staleness request-scoping slice (`tla/EPaxosEvidenceStaleness.cfg` generated `6/6` states), finite 3-, 5-, and 7-replica uncommitted-conflict force/defer quorum coverage, finite configuration-barrier/add/remove/chain pinning coverage, one finite normal configuration-transition retry-timer slice (`tla/EPaxosConfigTransitionRetry.cfg` generated `8/8` states), one finite normal configuration-transition response de-duplication slice (`tla/EPaxosConfigTransitionDedup.cfg` generated `16/16` states), one finite durable configuration replay slice, finite config recovery-after-removal, recovery-after-addition, lost/duplicate response de-duplication, and recovery retry-timer slices (`tla/EPaxosConfigRecovery.cfg` generated `44/30` states, `tla/EPaxosConfigAddRecovery.cfg` generated `15/15` states, `tla/EPaxosConfigRecoveryDedup.cfg` generated `11/11` states, and `tla/EPaxosConfigRecoveryRetry.cfg` generated `8/8` states), a finite rollback-allocation next-instance/skip/apply-order check, and a finite `TOQClockDiscipline.tla` bounded-skew/bounded-delay contract. Remaining open: arbitrary/general recovery under configuration changes beyond the finite recovery slices, arbitrary membership histories, arbitrary durable histories, joint consensus, arbitrary message loss and retry/rebroadcast behavior, complete optimized-recovery branch parity, unbounded proofs, external target proof, synchronized-clock implementation, one-way-delay measurement, and operational clock-discipline proof. |
126126
| Deployment manifest | Example systemd artifacts now exist (`deploy/systemd/kvnode@.service`, `deploy/systemd/kvnode.env.example`) plus `tests/kvnode_systemd_manifest_audit.sh`, which renders the example EnvironmentFile into the `ExecStart` contract and keeps `systemd-analyze verify` opt-in via `KVNODE_SYSTEMD_ANALYZE=yes`; these artifacts are checked by `tests/operations_readiness_audit.sh`. A reviewed and exercised target deployment under systemd/container/orchestration remains open before this can be a production manifest claim. |
127127
| Data lifecycle | Local destructive-storage remove/restore evidence exists, the KV example has exercised Pebble checkpoint/whole-directory restore plus offline and live-source checkpoint-backed repair tests for checksum-detected bit-level corruption, `examples/kv/cmd/kvcheckpoint` provides a maintained offline checkpoint/verify/verified-restore/repair helper, `TestRestoreRejectsCorruptCheckpointWithoutReplacingLiveData` verifies restore fails closed before replacement, `tests/kvnode_local_runner.go --mode data` stops one local loopback node and runs offline checkpoint/verify/restore/repair on a stopped local node before restart/catch-up verification, `data-lifecycle-summary.txt` records `data_lifecycle=offline-checkpoint-verify-restore-repair` plus `none-target-environment-data-lifecycle-drill-still-required`, and `docs/operations/kvnode-data-lifecycle-incident-runbook.md` documents checkpoint, verification, repair, restore, checksum-mismatch, local data-lifecycle drill, and evidence-capture procedures. A reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. |
128-
| Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape and records bounded write/read/scan latency plus admin metric samples. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, and peer_count=3, and a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, and `status=local-go-runner-only`. This is workstation harness evidence only; measured target-environment capacity results remain open. |
128+
| Capacity envelope | `tests/kvnode_capacity_envelope.sh` is an opt-in bounded harness for throughput, latency, memory RSS, disk growth, queue depth, value size, scan limit, and peer-count samples; its `metadata.env` and `summary.md` emit `release_claim=none-target-environment-capacity-results-still-required`; `tests/kvnode_local_capacity_drill.sh` starts a disposable three-node loopback cluster and runs that harness against all three client/admin listeners with PIDs and data dirs while preserving the same release-claim non-claim; `tests/kvnode_local_runner.go` is a custom Go runner that starts the same local-only three-node loopback shape and records bounded write/read/scan latency plus admin metric samples. `bash -n tests/kvnode_capacity_envelope.sh`, `bash tests/kvnode_capacity_envelope.sh --help`, `bash tests/kvnode_local_capacity_drill.sh --help`, `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help`, and `tests/operations_readiness_audit.sh` pass. Local loopback samples have passed, including the earlier single-node workstation sample, a three-node local wrapper sample with 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, a non-claim metadata sample with 1 op, 16-byte values, scan limit 1, peer_count=3, and `release_claim=none-target-environment-capacity-results-still-required`, and a custom Go runner sample with `KVNODE_GO_RUNNER_OPS_PER_PHASE=2`, `KVNODE_GO_RUNNER_VALUE_BYTES=16`, `KVNODE_GO_RUNNER_SCAN_LIMITS=1`, and `status=local-go-runner-only`. This is workstation harness evidence only; measured target-environment capacity results remain open because target-environment capacity measurement remains open. |
129129
| Incident readiness | `docs/operations/kvnode-data-lifecycle-incident-runbook.md` now covers storage failure, network partition, peer compromise, replay/checksum suspicion, and recovery stalls, with evidence-capture steps and non-claims; `tests/kvnode_incident_tabletop_drill.sh` locally rehearses the storage-failure and network-partition test-fault branches on a disposable loopback cluster; `tests/kvnode_local_runner.go` also locally exercised `/faults/storage`, `/faults/transport`, `/readyz`, `/metrics`, and post-clear canaries with `status=local-go-runner-only`; `tests/operations_readiness_audit.sh` checks those artifacts. Operator-reviewed target-environment tabletop or live drill evidence remains open. |
130130

131131
TryPreAccept response branch-slice note: `tla/EPaxosTryPreAcceptBranches.tla` is only a finite abstract scenario/stage model for stale restart, committed evidence ignore/fail-closed, direct/forced accept, one uncommitted deferral with duplicate suppression, and OK slow-quorum accept. Its only quorum detail is `okVotes >= SlowQuorum`, it runs for 3/5/7, and complete optimized-recovery branch parity beyond this finite slice remains open.

release/EPAXOS_READINESS_EVIDENCE.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,7 @@ Non-claims remain explicit: No target-environment remote claim, no in-place Pebb
193193
- `go test ./examples/kv/cmd/kvcheckpoint -count=1` passed after adding the maintained offline checkpoint helper; the CLI verifies checkpoints before `restore` and `repair` replaces a data directory.
194194
- `KVNODE_INCIDENT_TABLETOP_RUN=yes bash tests/kvnode_incident_tabletop_drill.sh` passed on local loopback after adding the incident tabletop harness; the generated summary reported `status=local-tabletop-only`, `storage_fault=exercised-and-cleared`, `transport_fault=exercised-and-cleared`, and post-clear canaries visible on all nodes.
195195
- `KVNODE_LOCAL_CAPACITY_RUN=yes bash tests/kvnode_local_capacity_drill.sh` passed on local loopback after adding the three-node capacity wrapper; it ran `tests/kvnode_capacity_envelope.sh` with `peer_count=3`, 5 ops per value-size phase, 64/1024-byte values, scan limits 1/8, PID/data-dir resource sampling, and `status=local-loopback-only`.
196+
- `KVNODE_LOCAL_CAPACITY_RUN=yes KVNODE_LOCAL_CAPACITY_BASE_PORT=29080 KVNODE_LOCAL_CAPACITY_PEER_BASE_PORT=29180 KVNODE_LOCAL_CAPACITY_ADMIN_BASE_PORT=29280 KVNODE_CAPACITY_OPS_PER_PHASE=1 KVNODE_CAPACITY_VALUE_BYTES=16 KVNODE_CAPACITY_SCAN_LIMITS=1 bash tests/kvnode_local_capacity_drill.sh` passed on local loopback after adding the machine-readable capacity non-claim; archived sample files are `local://kvnode-local-capacity-nonclaim-20260709-wrapper-metadata.env`, `local://kvnode-local-capacity-nonclaim-20260709-wrapper-summary.txt`, `local://kvnode-local-capacity-nonclaim-20260709-capacity-metadata.env`, `local://kvnode-local-capacity-nonclaim-20260709-capacity-summary.md`, `local://kvnode-local-capacity-nonclaim-20260709-latency.csv`, and `local://kvnode-local-capacity-nonclaim-20260709-resources.csv`; generated wrapper and harness metadata both include `release_claim=none-target-environment-capacity-results-still-required`, the sample recorded 3 HTTP operations against peer_count=3, and target-environment capacity measurement remains open.
196197
- `go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help` passed in this session and compiled the build-tagged custom local Go runner.
197198
- `KVNODE_GO_RUNNER_RUN=yes KVNODE_GO_RUNNER_OPS_PER_PHASE=2 KVNODE_GO_RUNNER_VALUE_BYTES=16 KVNODE_GO_RUNNER_SCAN_LIMITS=1 KVNODE_GO_RUNNER_BASE_PORT=28080 KVNODE_GO_RUNNER_PEER_BASE_PORT=28180 KVNODE_GO_RUNNER_ADMIN_BASE_PORT=28280 go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --mode all` passed on local loopback after adding the custom Go runner data-lifecycle mode; archived sample files are `local://kvnode-go-runner-all-20260710-verified-summary.txt`, `local://kvnode-go-runner-all-20260710-verified-incident-summary.txt`, `local://kvnode-go-runner-all-20260710-verified-capacity-summary.txt`, `local://kvnode-go-runner-all-20260710-verified-data-lifecycle-summary.txt`, `local://kvnode-go-runner-all-20260710-verified-latency.csv`, `local://kvnode-go-runner-all-20260710-verified-resources.csv`, `local://kvnode-go-runner-all-20260710-verified-checkpoint.log`, `local://kvnode-go-runner-all-20260710-verified-verify.log`, `local://kvnode-go-runner-all-20260710-verified-restore.log`, and `local://kvnode-go-runner-all-20260710-verified-repair.log`; generated summaries reported `mode=all`, `incident_ran=true`, `capacity_ran=true`, `data_lifecycle_ran=true`, storage/transport faults exercised and cleared, post-clear canaries visible on all nodes, 5 local capacity latency rows with p50 0.034385750s, p95 0.035291167s, p99 0.035291167s, `data_lifecycle=offline-checkpoint-verify-restore-repair`, `checkpoint=verified`, stopped-node restore and repair restarts, pre-checkpoint and post-restore canaries visible on all nodes after repair, and the target-environment capacity/operator-review/data-lifecycle non-claim strings.
198199

@@ -227,9 +228,9 @@ Non-claims remain explicit: No target-environment remote claim, no in-place Pebb
227228
- `docs/operations/kvnode-upgrade-rollback.md`
228229
- Rolling upgrade and rollback plan.
229230
- `tests/kvnode_capacity_envelope.sh`
230-
- Opt-in bounded capacity-envelope collection harness.
231+
- Opt-in bounded capacity-envelope collection harness; `metadata.env` and `summary.md` now emit `release_claim=none-target-environment-capacity-results-still-required` so raw samples carry the same target-environment non-claim as the release docs.
231232
- `tests/kvnode_local_capacity_drill.sh`
232-
- Local loopback capacity wrapper only: starts a disposable three-node `kvnode` cluster, runs `tests/kvnode_capacity_envelope.sh` against all three client/admin listeners with PIDs and data directories supplied, and emits `status=local-loopback-only`.
233+
- Local loopback capacity wrapper only: starts a disposable three-node `kvnode` cluster, runs `tests/kvnode_capacity_envelope.sh` against all three client/admin listeners with PIDs and data directories supplied, emits `status=local-loopback-only`, and preserves `release_claim=none-target-environment-capacity-results-still-required` in wrapper metadata and summary.
233234
- `tests/kvnode_local_runner.go`
234235
- Custom local Go runner only: build-tagged with `kvnode_local_runner`, starts a disposable three-node `kvnode` loopback cluster, exercises storage-fault and transport-fault administrative branches, verifies post-clear client canaries, records bounded write/read/scan latency and admin metric samples, stops one local node for offline `kvcheckpoint` checkpoint/verify/restore/repair in data mode, verifies catch-up canaries after restore and repair, emits `status=local-go-runner-only`, and preserves target-environment capacity/operator-review/data-lifecycle non-claims.
235236
- `tests/kvnode_mixed_version_drill.sh`
@@ -355,7 +356,7 @@ The following blockers are still listed in `RELEASE_SCOPE.md` and prevent a go d
355356
- `tla/EPaxosConfigAddRecovery.tla` adds one finite staged old-instance recovery-after-addition slice and generated `15/15` states, and `TestOldConfigRecoveryUsesPinnedVotersAfterAddition` covers the matching Go prepare/accept quorum path with added-voter responses rejected for old quorum progress, but arbitrary recovery under configuration changes, joint consensus, message loss, arbitrary membership histories, and unbounded proof remain open.
356357
- Deployment manifest artifacts are example/operator material only; `tests/kvnode_systemd_manifest_audit.sh` renders and audits the example `ExecStart` contract, but reviewed execution under a target system manager, container, or orchestration environment remains open.
357358
- Data lifecycle now has a maintained local offline `kvcheckpoint` helper, runbook/audit evidence, and a local Go runner data-lifecycle drill that stops one loopback node, runs offline checkpoint/verify/restore/repair, restarts it, and verifies catch-up canaries, but a reviewed operator backup/restore/disaster-recovery drill in the target environment remains open. The mixed-version drill's binary rollback keeps current data and does not exercise checkpoint restore.
358-
- Target-environment capacity-envelope measurements remain open; local evidence now includes a single-node workstation sample, a three-node loopback shell wrapper sample, and a custom local Go runner sample, but none is a measured target-environment capacity result.
359+
- Target-environment capacity-envelope measurements remain open; local evidence now includes a single-node workstation sample, a three-node loopback shell wrapper sample, a machine-readable capacity non-claim sample, and a custom local Go runner sample, but none is a measured target-environment capacity result, so target-environment capacity measurement remains open.
359360
- Incident readiness has runbook/audit evidence, a local loopback tabletop shell harness, and a custom local Go runner for storage-failure and network-partition test-fault branches, but operator-reviewed target-environment tabletop or live drill evidence remains open.
360361

361362
## Final workflow command

tests/kvnode_capacity_envelope.sh

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ kvnode capacity-envelope harness (opt-in, bounded)
1111
Status: harness only. This script gathers small, operator-chosen samples from an
1212
already-running kvnode cluster. It does not start nodes, does not change release
1313
scope, and does not produce production capacity evidence by itself.
14+
It emits release_claim=none-target-environment-capacity-results-still-required
15+
in metadata and summary output.
1416
1517
Required opt-in:
1618
KVNODE_CAPACITY_RUN=yes
@@ -179,6 +181,7 @@ printf 'sample,resource,target,value,unit\n' > "$RESOURCES_CSV"
179181
peer_count="${KVNODE_PEER_COUNT:-${#CLIENT_URLS[@]}}"
180182
cat > "$METADATA_ENV" <<EOF
181183
status=harness-only
184+
release_claim=none-target-environment-capacity-results-still-required
182185
run_id=$run_id
183186
client_urls=$client_urls_raw
184187
admin_urls=$admin_urls_raw
@@ -311,6 +314,7 @@ cat > "$SUMMARY_MD" <<EOF
311314
# kvnode capacity-envelope harness sample
312315
313316
Status: harness output only. These numbers are samples from the requested run and are not production capacity evidence unless the environment, workload, peer count, and operator procedure have been separately approved and recorded.
317+
Release claim: release_claim=none-target-environment-capacity-results-still-required.
314318
315319
## Inputs
316320

tests/kvnode_local_capacity_drill.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ kvnode cluster, then runs tests/kvnode_capacity_envelope.sh against all three
1313
client/admin listeners with PIDs and data directories supplied for resource
1414
sampling. It does not assert production capacity, target-environment coverage,
1515
or operator acceptance.
16+
The generated metadata and summary retain release_claim=none-target-environment-capacity-results-still-required.
1617
1718
Required opt-in:
1819
KVNODE_LOCAL_CAPACITY_RUN=yes
@@ -226,6 +227,7 @@ run_id=$run_id
226227
run_dir=$RUN_DIR
227228
capacity_dir=$CAPACITY_DIR
228229
non_claim=not_target_environment_capacity_evidence
230+
release_claim=none-target-environment-capacity-results-still-required
229231
EOF
230232

231233
echo "kvnode-local-capacity phase=build"

tests/operations_readiness_audit.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,7 @@ require_text "$capacity" "resources.csv before/after RSS, disk
255255
require_text "$capacity" "summary.md Machine-generated sample summary with no readiness claim."
256256
require_text "$capacity" "not production capacity evidence"
257257
require_text "$capacity" "status=harness-only"
258+
require_text "$capacity" "release_claim=none-target-environment-capacity-results-still-required"
258259
require_text "$capacity" "latency_summary="
259260
require_text "$capacity" "Throughput sample:"
260261
require_text "$capacity" "Memory RSS samples:"
@@ -271,4 +272,5 @@ require_text "$local_capacity" "KVNODE_CAPACITY_RUN=yes"
271272
require_text "$local_capacity" "KVNODE_PEER_COUNT=3"
272273
require_text "$local_capacity" "status=local-loopback-only"
273274
require_text "$local_capacity" "not_target_environment_capacity_evidence"
275+
require_text "$local_capacity" "release_claim=none-target-environment-capacity-results-still-required"
274276
bash -n "$local_capacity"

tests/release_scope_audit.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,8 @@ require_text "custom Go runner sample"
262262
require_text "status=local-go-runner-only"
263263
require_text "go run -tags kvnode_local_runner ./tests/kvnode_local_runner.go --help"
264264
require_text "measured target-environment capacity results remain open"
265+
require_text "release_claim=none-target-environment-capacity-results-still-required"
266+
require_text "target-environment capacity measurement remains open"
265267
require_text "also locally exercised \`/faults/storage\`, \`/faults/transport\`, \`/readyz\`, \`/metrics\`"
266268
require_text "three-node local wrapper sample"
267269
require_text "locally rehearses the storage-failure and network-partition"

0 commit comments

Comments
 (0)