Skip to content

Commit 1fc959e

Browse files
JuliaEdomJuliaEdom
committed
docs(ops): add ci soak resume runbook
1 parent 726d171 commit 1fc959e

3 files changed

Lines changed: 179 additions & 2 deletions

File tree

‎ci-soak-runtime-harness.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# CI Soak Runtime Harness
22

3+
> **Next-session entry point:** use the tracked
4+
> [CI-soak next-session runbook](docs/operations/ci-soak-next-session-runbook.md)
5+
> for current status, consumed identities, authorization boundaries, and the
6+
> exact cold-start sequence. The chronological material below is supporting
7+
> history.
8+
39
## Goal
410

511
Add a fail-closed local Compose runtime contract and Kubernetes-pods compatibility shim without changing any byte in the tracked source pack or claiming a soak result.

‎docs/operations/ci-soak-next-session-runbook.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
# CI-soak next-session runbook
2+
3+
**Updated:** 2026-08-21
4+
5+
**Purpose:** one unambiguous entry point for resuming the CI-soak work without
6+
repeating completed local work or reusing consumed runtime evidence.
7+
8+
Live repository facts from the commands below override copied chat context and
9+
older status sections. This runbook records the operational boundary; it does
10+
not authorize an external action.
11+
12+
## Status at a glance
13+
14+
| Surface | Exact state |
15+
| --- | --- |
16+
| Closing implementation commit | `726d171322dc8933d8788e7388f1bcd5d0d8e32e` — `fix(ops): reconcile ci soak verifier contract` |
17+
| Required ancestry | The current `HEAD` must contain the closing implementation commit before any post-fix validation |
18+
| Local architecture gate | `ARCHITECTURE_READY=PASS blockers=0 head=726d171322dc8933d8788e7388f1bcd5d0d8e32e` |
19+
| Local verification | Focused gate: `129 passed`; Ruff check/format, `py_compile`, `git diff --check`, strict UTF-8/LF/no-BOM/no-NUL, and the clean-HEAD architecture gate passed |
20+
| Post-fix external rehearsal | **Not run** |
21+
| Last external evidence | Retained r9 attempt: **FAIL** against older source `7e8ec87c25bbdc8f8aa58c116ded9914470789cb` |
22+
| Push/fetch | Neither was performed for the closing slice; push remains unauthorized |
23+
24+
The local correction is closed. Counts below `1_440_000` use
25+
`canary` / `kind_residual_20`; the exact full-soak count uses
26+
`soak` / `dual_mean_90`. The controller starts and identity-checks a named
27+
verifier before producer traffic, waits for stable producer-final evidence,
28+
and then delegates to the byte-pinned verifier. The same exact verifier ID,
29+
labels, name, restart count, terminal state, exit code, logs, phase-specific
30+
JSON, and cleanup/no-replacement state are checked fail-closed.
31+
32+
This is local contract evidence only. It does not prove current
33+
Docker/Colima, Kind, Flink, ClickHouse, Iceberg, or MinIO behavior.
34+
35+
## Source-of-truth order
36+
37+
Use this order when sources disagree:
38+
39+
1. Current `git status`, `git rev-parse HEAD`, and commit ancestry.
40+
2. The first handoff block in local `AGENT_STATE.md`, when that ignored file is
41+
present.
42+
3. This tracked runbook for the resume sequence and authorization boundary.
43+
4. [`ci-soak-runtime-harness.md`](../../ci-soak-runtime-harness.md) for the
44+
accumulated harness history and detailed contracts.
45+
5. The immutable
46+
[r9 rehearsal report](../../ci-soak-r9-rehearsal-20260821-01.md) and
47+
[r1-r7 architecture audit](../../ci-soak-r1-r7-architecture-audit.md) for
48+
historical evidence.
49+
50+
`docs/SESSION_HANDOFF.md` is an ignored local mirror. It is useful on this
51+
machine, but it must not override current Git facts or this tracked boundary.
52+
53+
## Cold-start sequence
54+
55+
Run only these read-only checks before selecting work:
56+
57+
```powershell
58+
git status --short --branch
59+
git rev-parse HEAD
60+
git merge-base --is-ancestor 726d171322dc8933d8788e7388f1bcd5d0d8e32e HEAD
61+
```
62+
63+
The ancestry command must exit `0`. Then:
64+
65+
1. Read the first block of `AGENT_STATE.md`, if present, and this runbook.
66+
2. Treat every pre-existing untracked or ignored path as protected user state.
67+
Never bulk-stage with `git add .` or `git add -A`.
68+
3. If tracked files are unexpectedly dirty, stop before editing overlapping
69+
files and identify their owner and scope.
70+
4. Do not rerun the 129-test focused gate, the architecture gate, or an
71+
external rehearsal merely to refresh timestamps. Repeat a gate only for a
72+
new relevant change, a fresh failure, or a newly authorized exact-HEAD
73+
attempt.
74+
5. The next actionable runtime slice is at most one fresh post-fix
75+
`--count 2000` rehearsal, and only after fresh explicit authorization.
76+
77+
If a new local change legitimately requires the architecture gate, run it from
78+
a clean tracked checkout:
79+
80+
```powershell
81+
python scripts/golden_soak/architecture_gate.py
82+
```
83+
84+
Require one exact terminal line with the current 40-character `HEAD`. A PASS
85+
does not authorize runtime work.
86+
87+
## Consumed r9 evidence — never reuse or clean
88+
89+
| Identity | Consumed value |
90+
| --- | --- |
91+
| Attempt | `ci-soak-7e8ec87-r9-rehearsal-20260821-01` |
92+
| Source/gate HEAD | `7e8ec87c25bbdc8f8aa58c116ded9914470789cb` |
93+
| Snapshot | `/Users/julia/agentflow-fc5-7113966/ci-soak-rehearsal-7e8ec87-r9` |
94+
| Compose project | `agentflow-ci-soak-7e8ec87-r9` |
95+
| Output | `.artifacts/soak-rehearsal-2000-7e8ec87-r9` under the snapshot |
96+
| Result | `FAIL`: producer delivered `2000/2000`; the old sequential `dual_mean_90` verification failed closed |
97+
98+
The snapshot, project, output, wrapper, and attempt names are immutable failure
99+
evidence. Do not rerun the controller from them, overwrite them, or remove
100+
them. Do not weaken the `90 eps` full-soak floor to make a short rehearsal
101+
pass.
102+
103+
## Historical runtime safety snapshot
104+
105+
These identities were healthy after the r9 postflight, but they are historical
106+
observations, not standing authority for a later mutation:
107+
108+
| Surface | Last recorded identity/state |
109+
| --- | --- |
110+
| MinIO | `f51db9e3ee0715bbfc91c2a715a4fc114f80fdbe82e156e89db74626b729aa42` |
111+
| Iceberg REST | `1e80588ca8fb0859934ada5635731027ca6692d9db597105504b1f7c2d761211` |
112+
| Current ClickHouse | `f0f0b82817bb87ec522f16426795df021e8d249fdc0c07a9474ac34717488c61` |
113+
| Kind | `0545702c4bc4ffdb5402b324af5dd51af71bed57ca7078707c931eae8aee365b` |
114+
| Old rollback ClickHouse | `a8cc630eedb5d116d605449771ea080c400ced742dc0540c44225897330c15b9`; exited and disconnected |
115+
| Mac checkout | `ae9fb69db7de737b469f868f218e8d623c206959` |
116+
117+
Revalidate every identity, restart count, readiness endpoint, network
118+
attachment, owner lock, and Mac checkout state in a fresh preflight before any
119+
authorized stop or mutation. A mismatch is new input and closes the mutation
120+
gate; do not silently substitute a new container.
121+
122+
## Authorization boundary
123+
124+
| Action | Current boundary |
125+
| --- | --- |
126+
| Read local status and tracked docs | Allowed as read-only inspection |
127+
| Run local checks after a relevant local change | Allowed within that scoped change; avoid evidence-refresh repeats |
128+
| SSH, Docker/Colima inspection, or creation of a fresh Mac snapshot | Requires explicit authorization for the new external slice |
129+
| Stop or restore protected co-tenants; run `--count 2000` traffic | Requires the same fresh, bounded rehearsal authorization and a green preflight |
130+
| Reuse, rerun, overwrite, or clean r9 evidence | Forbidden |
131+
| Full soak, rollback exercise, retained-evidence cleanup, production action | Separate future authorization required |
132+
| Fetch or push | Not authorized |
133+
134+
Authorization for one item does not imply authorization for the next row.
135+
Project-scoped fail-closed cleanup and exact protected-co-tenant restoration are
136+
mandatory parts of any newly authorized rehearsal; cleanup of retained r9 or
137+
unrelated resources is outside that scope.
138+
139+
## Requirements for one future post-fix rehearsal
140+
141+
Before the first mutation, require and preserve:
142+
143+
1. A clean tracked checkout whose exact `HEAD` contains
144+
`726d171322dc8933d8788e7388f1bcd5d0d8e32e`, plus an exact-HEAD
145+
architecture-gate PASS.
146+
2. A new archive whose source and critical-file hashes match that exact HEAD.
147+
3. Fresh snapshot, Compose project, output, wrapper, and attempt identities;
148+
prove all are absent before creation.
149+
4. Current Docker context, capacity, bind visibility, ports, and Compose config.
150+
5. Current exact protected-container IDs, restart counts, network attachments,
151+
health/API checks, Mac checkout identity, and an exclusive owner lock.
152+
6. One controller invocation with `--count 2000`; no raw retry after a failure.
153+
7. Terminal wrapper/controller results, runtime state, component logs,
154+
phase-specific JSON, exact verifier identity/exit evidence, and hashes.
155+
8. Zero candidate containers, networks, volumes, runtime directories, or owner
156+
locks after exit, plus exact restoration and readiness of every protected
157+
co-tenant.
158+
159+
Classify the result exactly as emitted. A failed rehearsal remains useful
160+
evidence; preserve it and stop. A passing short rehearsal closes only the
161+
capacity-independent rehearsal gate, not the full soak, rollback, production,
162+
or push gates.
163+
164+
## Detailed references
165+
166+
- [Golden-soak source-pack README](../../scripts/golden_soak/README.md)
167+
- [Runtime harness history](../../ci-soak-runtime-harness.md)
168+
- [Immutable r9 failure report](../../ci-soak-r9-rehearsal-20260821-01.md)
169+
- [Architecture audit and readiness contract](../../ci-soak-r1-r7-architecture-audit.md)
170+
- [Compose foundation context](ci-soak-compose-foundation.md)

‎scripts/golden_soak/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,9 @@ manifests are not a Docker Compose runtime contract.
77

88
The current architecture verdict and finding register are documented in
99
[`ci-soak-r1-r7-architecture-audit.md`](../../ci-soak-r1-r7-architecture-audit.md).
10-
The exact local verification evidence and authoritative next-session resume
11-
checkpoint are in
10+
The authoritative next-session entry point is the tracked
11+
[`ci-soak-next-session-runbook.md`](../../docs/operations/ci-soak-next-session-runbook.md);
12+
the detailed local verification history is in
1213
[`ci-soak-runtime-harness.md`](../../ci-soak-runtime-harness.md). The older
1314
[`ci-soak-compose-foundation.md`](../../docs/operations/ci-soak-compose-foundation.md)
1415
is historical topology context, not current resume state.

0 commit comments

Comments
 (0)