docs(ops): record r12 preflight failure

Author: JuliaEdom

ci-soak-r12-preflight-fail-20260821-01.md

@@ -0,0 +1,92 @@
+# CI-soak r12 preflight failure — 2026-08-21
+
+## Goal and authority
+
+Execute one freshly authorized post-fix `--count 2000` rehearsal from exact
+source HEAD `bfb82ecb6c66e5490db2d99bbdaf8b9da55f2082`, using new r12
+snapshot/project/output identities and unconditional restoration of the four
+exact protected co-tenants.
+
+The rehearsal never reached a stop or controller invocation. The exact
+classification is `PREFLIGHT_FAIL_BEFORE_STOP`.
+
+## Local and read-only gates
+
+The fresh local architecture gate passed:
+
+```text
+ARCHITECTURE_READY=PASS blockers=0 head=bfb82ecb6c66e5490db2d99bbdaf8b9da55f2082
+```
+
+The exact `git archive` was `17,981,440` bytes with SHA-256
+`1654503d7a1cae3a95baf5f87e43a0411c93fb18db14250dd58a44a35a5252b7`.
+Its local and remote hashes matched. The r12 preflight wrapper passed local and
+native macOS `bash -n`; its SHA-256 was
+`e3835c7a18d0ebba2464a0a08a413bf09d3b6428d85c2295f25201b3c427b946`.
+
+Before any remote path was created, a strict read-only preflight passed. It
+proved Docker API range `1.44..1.53`, four CPUs, `7,269,556,224` bytes of
+memory, no r12 snapshot/control/output/preflight/owner-lock identity, project
+resources `0/0/0`, no r12 probes or controller, and the expected Mac checkout
+HEAD/status. All four protected exact IDs were running with restart count zero;
+MinIO, Iceberg, both ClickHouse routes, Kind `/livez`, and the one-apiserver
+check passed. The retained rollback ClickHouse remained exited cleanly and
+disconnected.
+
+The preceding r11 authorization was consumed by read-only orchestration only.
+Its first probe exposed an unsupported Docker template and its corrected probe
+then stopped on the expected no-match exit from `pgrep` under `pipefail`. No
+r11 remote identity, lock, project resource, probe, stop, or controller was
+created. The r12 probe carried the bounded `pgrep || true` correction and
+passed completely.
+
+## Terminal r12 preflight failure
+
+The preflight wrapper was invoked exactly once. It extracted the exact source,
+verified the protected source hashes including `verify_coschedule.py`, captured
+the source visibility hash, and then failed before the output visibility
+container or Compose validation:
+
+```text
+PREFLIGHT_RESULT=FAIL
+REASON=output_marker_hash_mismatch
+ATTEMPT_ID=ci-soak-bfb82ec-r12-preflight
+PROJECT_RESOURCES=0/0/0
+OWNER_LOCK_RELEASE=PASS
+STOP_COMMAND=NOT_INVOKED
+CONTROLLER=NOT_INVOKED
+```
+
+The terminal preflight result SHA-256 is
+`9a47518e73a724f7f105beaa79e7c7ad794da3d633694d2e2cb69e6fb7c59bb1`.
+The wrapper contained output marker text `ci-soak-output-bfb82ec-r12` but
+retained the r11 marker hash
+`a451640563c229772e9d8fa5287f6a20bace28a11f1b62b8086017b6a35b62a8`.
+The correct SHA-256 for the r12 text is
+`9853a9344b1378f968eb4f5c808c6541275746d6f5682a507b6d3294d4bfb6f2`.
+No correction or rerun used the consumed r12 identities.
+
+## Independent safety postflight
+
+The independent read-only postflight exited `0` and proved:
+
+- the owner lock and r12 controller were absent;
+- r12 project resources were `0/0/0` and all four r12 probe counts were zero;
+- the four protected exact IDs were still running with restart count zero;
+- MinIO/Iceberg, host and workload ClickHouse routes, Kind `/livez`, and the
+  one-apiserver check passed;
+- the rollback ClickHouse and original Mac checkout were unchanged;
+- the r12 snapshot, control directory, empty output identity, and preflight
+  evidence remain retained.
+
+No co-tenant stop, controller, traffic, full soak, rollback exercise,
+production action, cleanup of retained evidence, fetch, or push occurred.
+Grok was not used.
+
+## Resume boundary
+
+Treat the r11 and r12 local control identities and all r12 remote paths as
+consumed evidence. Do not rerun, overwrite, adopt, or clean them. A later
+attempt starts at r13 or later with a fresh exact-HEAD gate, new archive,
+snapshot/project/output/control/attempt identities, a corrected marker hash,
+fresh read-only preflight, and fresh explicit authorization.

ci-soak-runtime-harness.md

@@ -1367,3 +1367,38 @@ control identity, attempt name, and authorization as consumed. A later
 external attempt must use fresh r11-or-later identities, a new exact-HEAD gate
 and preflight, and fresh authorization. Full soak, rollback, retained-evidence
 cleanup, production action, fetch, and push remain unauthorized.
+
+## r11 read-only stop and r12 preflight failure before stop — 2026-08-21
+
+The r11 authorization was consumed without creating a remote identity. Its
+strict read-only probe first exposed an unsupported Docker Go-template field;
+the one corrected probe then stopped when the expected no-match exit from
+`pgrep` propagated through `set -e` / `pipefail`. No lock, probe, project
+resource, co-tenant stop, or controller existed.
+
+A separately authorized r12 attempt used exact source/gate HEAD
+`bfb82ecb6c66e5490db2d99bbdaf8b9da55f2082`. The fresh architecture gate
+passed, and the corrected strict read-only probe proved fresh r12 identities,
+Docker API/capacity, protected identities, restart-zero state, checkout state,
+and all health/routes before any remote path creation. The exact archive
+SHA-256 was
+`1654503d7a1cae3a95baf5f87e43a0411c93fb18db14250dd58a44a35a5252b7`.
+
+The r12 preflight wrapper ran exactly once and failed closed at
+`output_marker_hash_mismatch`. It contained r12 marker text but the r11 marker
+hash. The correct r12 marker SHA-256 is
+`9853a9344b1378f968eb4f5c808c6541275746d6f5682a507b6d3294d4bfb6f2`.
+The terminal result hash is
+`9a47518e73a724f7f105beaa79e7c7ad794da3d633694d2e2cb69e6fb7c59bb1`.
+It records owner-lock release PASS, project resources `0/0/0`, and both stop
+and controller `NOT_INVOKED`.
+
+Independent postflight found no lock, controller, project resource, or probe;
+all four protected exact IDs remained running with restart count zero and all
+readiness/routes passed. The rollback ClickHouse and Mac checkout were
+unchanged. The r12 snapshot/control/output/preflight paths remain retained
+evidence. Canonical detail is
+`ci-soak-r12-preflight-fail-20260821-01.md`. Do not reuse or clean r11/r12.
+A later external attempt starts at r13 or later with fresh authorization,
+identities, gate, archive, and preflight. No rehearsal, traffic, full soak,
+rollback, production action, fetch, or push occurred.

docs/operations/ci-soak-next-session-runbook.md

@@ -15,11 +15,11 @@ not authorize an external action.
 | --- | --- |
 | Closing implementation commit | `726d171322dc8933d8788e7388f1bcd5d0d8e32e` — `fix(ops): reconcile ci soak verifier contract` |
 | Required ancestry | The current `HEAD` must contain the closing implementation commit before any post-fix validation |
-| Latest local architecture gate | `ARCHITECTURE_READY=PASS blockers=0 head=1fc959efcc1c871fd3057f27a8aef60db44fc878` before the stopped r10 delegation |
+| Latest local architecture gate | `ARCHITECTURE_READY=PASS blockers=0 head=bfb82ecb6c66e5490db2d99bbdaf8b9da55f2082` before the r12 preflight |
 | Local verification | Focused gate: `129 passed`; Ruff check/format, `py_compile`, `git diff --check`, strict UTF-8/LF/no-BOM/no-NUL, and the clean-HEAD architecture gate passed |
-| Post-fix external rehearsal | **Not run**; r10 orchestration stopped before external mutation |
+| Post-fix external rehearsal | **Not run**; r12 preflight failed before stop/controller mutation |
 | Last external evidence | Retained r9 attempt: **FAIL** against older source `7e8ec87c25bbdc8f8aa58c116ded9914470789cb` |
-| Latest orchestration evidence | [r10 stopped before mutation](../../ci-soak-r10-orchestration-stop-20260821-01.md); no rehearsal verdict exists |
+| Latest orchestration evidence | [r12 preflight failed before stop](../../ci-soak-r12-preflight-fail-20260821-01.md); no rehearsal verdict exists |
 | Push/fetch | Neither was performed for the closing slice; push remains unauthorized |
 
 The local correction is closed. Counts below `1_440_000` use
@@ -30,8 +30,9 @@ and then delegates to the byte-pinned verifier. The same exact verifier ID,
 labels, name, restart count, terminal state, exit code, logs, phase-specific
 JSON, and cleanup/no-replacement state are checked fail-closed.
 
-This is local contract evidence only. It does not prove current
-Docker/Colima, Kind, Flink, ClickHouse, Iceberg, or MinIO behavior.
+The r12 read-only gate proved the current protected Docker/Colima identities
+and basic readiness only. It did not run the corrected Compose controller or
+prove current Flink/exactness behavior.
 
 ## Source-of-truth order
 
@@ -41,9 +42,12 @@ Use this order when sources disagree:
 2. The first handoff block in local `AGENT_STATE.md`, when that ignored file is
    present.
 3. This tracked runbook for the resume sequence and authorization boundary.
-4. [`ci-soak-runtime-harness.md`](../../ci-soak-runtime-harness.md) for the
+4. The retained
+   [r12 preflight-failure report](../../ci-soak-r12-preflight-fail-20260821-01.md)
+   for the latest external facts and consumed identities.
+5. [`ci-soak-runtime-harness.md`](../../ci-soak-runtime-harness.md) for the
    accumulated harness history and detailed contracts.
-5. The immutable
+6. The immutable
    [r9 rehearsal report](../../ci-soak-r9-rehearsal-20260821-01.md) and
    [r1-r7 architecture audit](../../ci-soak-r1-r7-architecture-audit.md) for
    historical evidence.
@@ -73,8 +77,9 @@ The ancestry command must exit `0`. Then:
    new relevant change, a fresh failure, or a newly authorized exact-HEAD
    attempt.
 5. The next actionable runtime slice is at most one fresh post-fix
-   `--count 2000` rehearsal using r11-or-later identities, and only after fresh
-   explicit authorization. The stopped r10 orchestration must not be resumed.
+   `--count 2000` rehearsal using r13-or-later identities, and only after fresh
+   explicit authorization. The consumed r10/r11/r12 orchestration must not be
+   resumed.
 
 If a new local change legitimately requires the architecture gate, run it from
 a clean tracked checkout:
@@ -115,8 +120,28 @@ co-tenants remained healthy under their exact IDs with restart count zero.
 Treat the r10 attempt name, prompt, local control directory, and authorization
 as consumed evidence. The exact record is the
 [r10 orchestration-stop report](../../ci-soak-r10-orchestration-stop-20260821-01.md).
-Do not resume r10 or launch a duplicate. A future attempt starts at r11 or
-later with fresh identities, gate, preflight, and explicit authorization.
+Do not resume r10 or launch a duplicate; the later r11/r12 outcomes below now
+supersede that historical continuation boundary.
+
+## Consumed r11 and r12 orchestration — no rehearsal ran
+
+The r11 authorization was consumed by read-only probe diagnostics only. Its
+corrected probe stopped because an expected no-match `pgrep` exit propagated
+through `pipefail`; no r11 remote identity or mutation was created.
+
+The fresh r12 exact-HEAD architecture gate and strict read-only runtime gate
+passed. The r12 preflight wrapper was then invoked exactly once and failed at
+`output_marker_hash_mismatch` before output-probe creation, Compose validation,
+co-tenant stop, or controller invocation. It released the owner lock and left
+project resources `0/0/0`. Independent postflight proved all four protected
+exact IDs still running with restart count zero and every readiness route
+green. The exact record is the
+[r12 preflight-failure report](../../ci-soak-r12-preflight-fail-20260821-01.md).
+
+Treat all r11/r12 local identities and retained r12 remote paths as consumed.
+Do not rerun, overwrite, adopt, or clean them. A future attempt starts at r13
+or later with a corrected marker hash, fresh identities, gate, preflight, and
+explicit authorization.
 
 ## Historical runtime safety snapshot
 
@@ -145,14 +170,14 @@ gate; do not silently substitute a new container.
 | Run local checks after a relevant local change | Allowed within that scoped change; avoid evidence-refresh repeats |
 | SSH, Docker/Colima inspection, or creation of a fresh Mac snapshot | Requires explicit authorization for the new external slice |
 | Stop or restore protected co-tenants; run `--count 2000` traffic | Requires the same fresh, bounded rehearsal authorization and a green preflight |
-| Reuse, rerun, overwrite, or clean r9 evidence | Forbidden |
+| Reuse, rerun, overwrite, or clean r9/r11/r12 evidence | Forbidden |
 | Full soak, rollback exercise, retained-evidence cleanup, production action | Separate future authorization required |
 | Fetch or push | Not authorized |
 
 Authorization for one item does not imply authorization for the next row.
 Project-scoped fail-closed cleanup and exact protected-co-tenant restoration are
-mandatory parts of any newly authorized rehearsal; cleanup of retained r9 or
-unrelated resources is outside that scope.
+mandatory parts of any newly authorized rehearsal; cleanup of retained
+evidence or unrelated resources is outside that scope.
 
 ## Requirements for one future post-fix rehearsal
 
@@ -185,5 +210,6 @@ or push gates.
 - [Runtime harness history](../../ci-soak-runtime-harness.md)
 - [Immutable r9 failure report](../../ci-soak-r9-rehearsal-20260821-01.md)
 - [r10 orchestration-stop report](../../ci-soak-r10-orchestration-stop-20260821-01.md)
+- [r12 preflight-failure report](../../ci-soak-r12-preflight-fail-20260821-01.md)
 - [Architecture audit and readiness contract](../../ci-soak-r1-r7-architecture-audit.md)
 - [Compose foundation context](ci-soak-compose-foundation.md)