Skip to content

Commit 7ddfddf

Browse files
committed
docs: record GitHub CI recovery blocker
1 parent bed05b9 commit 7ddfddf

1 file changed

Lines changed: 132 additions & 0 deletions

File tree

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# GitHub CI Recovery Plan — 2026-09-08
2+
3+
Status: **BLOCKED — upstream Actions reruns require repository administrator permission**
4+
5+
## Execution record — 2026-09-09
6+
7+
- Prepared durable Agent Run `ci-recovery-20260908` against local source
8+
`bed05b9947bb10959cbf112b8051b80450145304`.
9+
- Confirmed PR #2115 includes commit `92cb3b9`, which bounds the activation
10+
fixture's readiness/reap waits. That commit is not present in this local
11+
checkout, so this worktree cannot supply source-level acceptance for it.
12+
- Confirmed the current `apt.llvm.org` Noble LLVM 22 index contains `clang-22`,
13+
`libclang-rt-22-dev`, and `llvm-22`; the MSan package outage is no longer
14+
reproducible from the configured source.
15+
- A focused local `scripts/test.sh --suites cli` run on `bed05b99` remained
16+
incomplete after 204 seconds and was stopped. It made no worktree changes.
17+
- GitHub rejected reruns of failed workflow runs `34288397854` and
18+
`34282381014`: `Must have admin rights to Repository`.
19+
- Direct GitHub CLI verification confirms `ngallodev` has `pull` only (no
20+
push, maintain, or admin) on `DeusData/codebase-memory-mcp`; both affected
21+
PR branches are owned by `DeusData`.
22+
23+
Required upstream action: an administrator must rerun the failed jobs (or push
24+
a new commit to each PR) and inspect the fresh `MSan`, Windows guards, ARM
25+
Unix shard, and `ci-ok` results. Until those runs pass on their current merge
26+
SHAs, this plan is not complete.
27+
28+
## Scope and evidence
29+
30+
This plan resolves the failures seen in the required `PR` workflows, then
31+
verifies fresh GitHub Actions results. It does not weaken or bypass a test.
32+
33+
| Failure | Evidence | Classification |
34+
| --- | --- | --- |
35+
| Windows concurrent cold start | Runs [34288397854](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34288397854) and [34282381014](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34282381014) both fail `test-windows-guards`: a cold-storm client reports `secure CLI coordination could not be created (endpoint)`. | Shared product/coordination defect; the same failure crosses unrelated PRs. |
36+
| ARM CLI suite | Run [34288397854](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34288397854) exhausts the 900-second `cli` suite limit after `ASSERT(host_serving)` in `cli_install_into_host_namespace_still_drains_host_cohort`. | Product or test-contract regression specific to PR #2115's activation-guard change. |
37+
| MSan image build | Run [34282381014](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34282381014) cannot install `clang-22`, `libclang-rt-22-dev`, or `llvm-22` from its configured LLVM apt source. | CI image/toolchain provisioning defect, before product tests run. |
38+
39+
`ci-ok` is an aggregate failure and requires no separate repair.
40+
41+
## Order of work
42+
43+
1. Record a fresh failure matrix before editing: workflow run, merge SHA, failed
44+
job/step, runner image, and exact first error. Keep this evidence with the
45+
repair PR. Do not use historical logs as pass evidence.
46+
2. Repair MSan provisioning first because it is isolated from product behavior.
47+
3. Repair the Windows cold-start race on a branch based on current `main`.
48+
4. Rebase the activation-guard PR onto that baseline, then resolve its ARM CLI
49+
hang without changing the suite timeout.
50+
5. Rerun required checks on the resulting merge SHAs and inspect each job,
51+
rather than treating a successful workflow dispatch as success.
52+
53+
## 1. MSan provisioning
54+
55+
1. In the exact failing Docker build context, capture `apt-get update` output
56+
and `apt-cache policy` for all three LLVM 22 packages. Confirm the
57+
repository suite, key, architecture, and package names actually published
58+
by apt.llvm.org for the runner's Ubuntu release.
59+
2. Replace the unavailable source with one reproducible LLVM 22 provisioning
60+
route already compatible with the MSan image. Prefer the repository's
61+
existing source-built runtime path if it supplies the required compiler and
62+
runtimes; otherwise use the correct published LLVM repository. Pin the
63+
selected compiler/runtime version together.
64+
3. Run `scripts/ci/msan-lane.sh build`, then `scripts/ci/msan-lane.sh run`.
65+
Retain the build output proving the compiler and MSan runtimes are present.
66+
67+
Acceptance: the MSan image builds without retries masking a missing package,
68+
and the MSan test job reaches and passes its intended suite.
69+
70+
## 2. Windows concurrent cold start
71+
72+
1. Reproduce the existing six-client scenario from
73+
`tests/windows/test_daemon_stability.py` on the Windows runner or the
74+
authoritative Windows host. Repeat it enough to distinguish deterministic
75+
failure from a timing-only flake.
76+
2. Instrument only the shared endpoint/coordination creation failure path:
77+
capture the platform error code, operation, and safe runtime-path category.
78+
Do not log endpoint secrets, user paths, or broaden the test's deadlines.
79+
3. Trace endpoint construction, startup-lock acquisition, generation probing,
80+
spawn handoff, and cleanup as one state transition. Fix the shared state
81+
handoff that permits a racing client to observe an unusable endpoint; do not
82+
add per-client retries that merely conceal a failed handoff.
83+
4. Add or extend the smallest deterministic regression assertion at the shared
84+
boundary if the current cold-storm test cannot distinguish the broken state.
85+
86+
Acceptance: repeated cold storms succeed, all clients either share the valid
87+
ephemeral daemon or receive the defined recoverable state, and the daemon
88+
retires as the existing guard requires. Run the focused Windows guard followed
89+
by the complete Windows test matrix.
90+
91+
## 3. ARM activation-guard hang
92+
93+
1. Run the `cli` suite alone with its normal 900-second ceiling, preserving its
94+
captured output. On the stall, collect the host/target daemon status, cohort
95+
claims, startup-lock state, and child-process tree before cleanup.
96+
2. Compare the install target's `HOME`/runtime namespace with the host namespace
97+
used by `cli_install_into_host_namespace_still_drains_host_cohort`. Establish
98+
whether the test's expectation is still valid or the activation guard now
99+
fails to drain a legitimate host cohort.
100+
3. Repair the lifecycle ownership at the common activation/cohort boundary. A
101+
sandbox install must not drain an unrelated host daemon, while an explicit
102+
host-target install must complete its expected drain and restore service.
103+
4. Add a bounded regression test for both namespaces. Do not solve the failure
104+
by skipping the test, extending its timeout, or serializing unrelated suites.
105+
106+
Acceptance: the focused `cli` suite completes under its current limit on ARM
107+
and x86 Linux; the exact host and sandbox cases pass; the full Unix shard matrix
108+
has no timed-out suite.
109+
110+
## Integration and evidence gate
111+
112+
1. Keep MSan, shared Windows coordination, and activation-guard changes in
113+
reviewable commits. Rebase dependent PRs after the shared repair merges.
114+
2. Before push, run the smallest relevant local checks:
115+
116+
- `scripts/ci/msan-lane.sh all` for the image repair;
117+
- the focused Windows daemon stability guard on Windows;
118+
- `scripts/test.sh --suites cli` for the activation path.
119+
120+
3. For each changed branch, inspect the new `PR` workflow's individual jobs:
121+
MSan, Windows guards/matrix, Unix ARM shards, diagnostics, lint, security,
122+
and final `ci-ok`.
123+
4. Declare the work complete only when every required job is `success` on the
124+
post-repair merge SHA. Attach the URLs, SHAs, command output, and any
125+
remaining environmental limitation to the PR or a durable handoff.
126+
127+
## Explicit non-solutions
128+
129+
- Do not rerun until green without identifying the failing transition.
130+
- Do not suppress Windows cold-storm coverage.
131+
- Do not raise the ARM suite timeout or mark the CLI suite optional.
132+
- Do not make `ci-ok` ignore a failed `test` job.

0 commit comments

Comments
 (0)