|
| 1 | +# GitHub CI Recovery Plan — 2026-09-08 |
| 2 | + |
| 3 | +Status: **BLOCKED — upstream Actions reruns require repository administrator permission** |
| 4 | + |
| 5 | +## Execution record — 2026-09-09 |
| 6 | + |
| 7 | +- Prepared durable Agent Run `ci-recovery-20260908` against local source |
| 8 | + `bed05b9947bb10959cbf112b8051b80450145304`. |
| 9 | +- Confirmed PR #2115 includes commit `92cb3b9`, which bounds the activation |
| 10 | + fixture's readiness/reap waits. That commit is not present in this local |
| 11 | + checkout, so this worktree cannot supply source-level acceptance for it. |
| 12 | +- Confirmed the current `apt.llvm.org` Noble LLVM 22 index contains `clang-22`, |
| 13 | + `libclang-rt-22-dev`, and `llvm-22`; the MSan package outage is no longer |
| 14 | + reproducible from the configured source. |
| 15 | +- A focused local `scripts/test.sh --suites cli` run on `bed05b99` remained |
| 16 | + incomplete after 204 seconds and was stopped. It made no worktree changes. |
| 17 | +- GitHub rejected reruns of failed workflow runs `34288397854` and |
| 18 | + `34282381014`: `Must have admin rights to Repository`. |
| 19 | +- Direct GitHub CLI verification confirms `ngallodev` has `pull` only (no |
| 20 | + push, maintain, or admin) on `DeusData/codebase-memory-mcp`; both affected |
| 21 | + PR branches are owned by `DeusData`. |
| 22 | + |
| 23 | +Required upstream action: an administrator must rerun the failed jobs (or push |
| 24 | +a new commit to each PR) and inspect the fresh `MSan`, Windows guards, ARM |
| 25 | +Unix shard, and `ci-ok` results. Until those runs pass on their current merge |
| 26 | +SHAs, this plan is not complete. |
| 27 | + |
| 28 | +## Scope and evidence |
| 29 | + |
| 30 | +This plan resolves the failures seen in the required `PR` workflows, then |
| 31 | +verifies fresh GitHub Actions results. It does not weaken or bypass a test. |
| 32 | + |
| 33 | +| Failure | Evidence | Classification | |
| 34 | +| --- | --- | --- | |
| 35 | +| Windows concurrent cold start | Runs [34288397854](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34288397854) and [34282381014](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34282381014) both fail `test-windows-guards`: a cold-storm client reports `secure CLI coordination could not be created (endpoint)`. | Shared product/coordination defect; the same failure crosses unrelated PRs. | |
| 36 | +| ARM CLI suite | Run [34288397854](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34288397854) exhausts the 900-second `cli` suite limit after `ASSERT(host_serving)` in `cli_install_into_host_namespace_still_drains_host_cohort`. | Product or test-contract regression specific to PR #2115's activation-guard change. | |
| 37 | +| MSan image build | Run [34282381014](https://github.com/DeusData/codebase-memory-mcp/actions/runs/34282381014) cannot install `clang-22`, `libclang-rt-22-dev`, or `llvm-22` from its configured LLVM apt source. | CI image/toolchain provisioning defect, before product tests run. | |
| 38 | + |
| 39 | +`ci-ok` is an aggregate failure and requires no separate repair. |
| 40 | + |
| 41 | +## Order of work |
| 42 | + |
| 43 | +1. Record a fresh failure matrix before editing: workflow run, merge SHA, failed |
| 44 | + job/step, runner image, and exact first error. Keep this evidence with the |
| 45 | + repair PR. Do not use historical logs as pass evidence. |
| 46 | +2. Repair MSan provisioning first because it is isolated from product behavior. |
| 47 | +3. Repair the Windows cold-start race on a branch based on current `main`. |
| 48 | +4. Rebase the activation-guard PR onto that baseline, then resolve its ARM CLI |
| 49 | + hang without changing the suite timeout. |
| 50 | +5. Rerun required checks on the resulting merge SHAs and inspect each job, |
| 51 | + rather than treating a successful workflow dispatch as success. |
| 52 | + |
| 53 | +## 1. MSan provisioning |
| 54 | + |
| 55 | +1. In the exact failing Docker build context, capture `apt-get update` output |
| 56 | + and `apt-cache policy` for all three LLVM 22 packages. Confirm the |
| 57 | + repository suite, key, architecture, and package names actually published |
| 58 | + by apt.llvm.org for the runner's Ubuntu release. |
| 59 | +2. Replace the unavailable source with one reproducible LLVM 22 provisioning |
| 60 | + route already compatible with the MSan image. Prefer the repository's |
| 61 | + existing source-built runtime path if it supplies the required compiler and |
| 62 | + runtimes; otherwise use the correct published LLVM repository. Pin the |
| 63 | + selected compiler/runtime version together. |
| 64 | +3. Run `scripts/ci/msan-lane.sh build`, then `scripts/ci/msan-lane.sh run`. |
| 65 | + Retain the build output proving the compiler and MSan runtimes are present. |
| 66 | + |
| 67 | +Acceptance: the MSan image builds without retries masking a missing package, |
| 68 | +and the MSan test job reaches and passes its intended suite. |
| 69 | + |
| 70 | +## 2. Windows concurrent cold start |
| 71 | + |
| 72 | +1. Reproduce the existing six-client scenario from |
| 73 | + `tests/windows/test_daemon_stability.py` on the Windows runner or the |
| 74 | + authoritative Windows host. Repeat it enough to distinguish deterministic |
| 75 | + failure from a timing-only flake. |
| 76 | +2. Instrument only the shared endpoint/coordination creation failure path: |
| 77 | + capture the platform error code, operation, and safe runtime-path category. |
| 78 | + Do not log endpoint secrets, user paths, or broaden the test's deadlines. |
| 79 | +3. Trace endpoint construction, startup-lock acquisition, generation probing, |
| 80 | + spawn handoff, and cleanup as one state transition. Fix the shared state |
| 81 | + handoff that permits a racing client to observe an unusable endpoint; do not |
| 82 | + add per-client retries that merely conceal a failed handoff. |
| 83 | +4. Add or extend the smallest deterministic regression assertion at the shared |
| 84 | + boundary if the current cold-storm test cannot distinguish the broken state. |
| 85 | + |
| 86 | +Acceptance: repeated cold storms succeed, all clients either share the valid |
| 87 | +ephemeral daemon or receive the defined recoverable state, and the daemon |
| 88 | +retires as the existing guard requires. Run the focused Windows guard followed |
| 89 | +by the complete Windows test matrix. |
| 90 | + |
| 91 | +## 3. ARM activation-guard hang |
| 92 | + |
| 93 | +1. Run the `cli` suite alone with its normal 900-second ceiling, preserving its |
| 94 | + captured output. On the stall, collect the host/target daemon status, cohort |
| 95 | + claims, startup-lock state, and child-process tree before cleanup. |
| 96 | +2. Compare the install target's `HOME`/runtime namespace with the host namespace |
| 97 | + used by `cli_install_into_host_namespace_still_drains_host_cohort`. Establish |
| 98 | + whether the test's expectation is still valid or the activation guard now |
| 99 | + fails to drain a legitimate host cohort. |
| 100 | +3. Repair the lifecycle ownership at the common activation/cohort boundary. A |
| 101 | + sandbox install must not drain an unrelated host daemon, while an explicit |
| 102 | + host-target install must complete its expected drain and restore service. |
| 103 | +4. Add a bounded regression test for both namespaces. Do not solve the failure |
| 104 | + by skipping the test, extending its timeout, or serializing unrelated suites. |
| 105 | + |
| 106 | +Acceptance: the focused `cli` suite completes under its current limit on ARM |
| 107 | +and x86 Linux; the exact host and sandbox cases pass; the full Unix shard matrix |
| 108 | +has no timed-out suite. |
| 109 | + |
| 110 | +## Integration and evidence gate |
| 111 | + |
| 112 | +1. Keep MSan, shared Windows coordination, and activation-guard changes in |
| 113 | + reviewable commits. Rebase dependent PRs after the shared repair merges. |
| 114 | +2. Before push, run the smallest relevant local checks: |
| 115 | + |
| 116 | + - `scripts/ci/msan-lane.sh all` for the image repair; |
| 117 | + - the focused Windows daemon stability guard on Windows; |
| 118 | + - `scripts/test.sh --suites cli` for the activation path. |
| 119 | + |
| 120 | +3. For each changed branch, inspect the new `PR` workflow's individual jobs: |
| 121 | + MSan, Windows guards/matrix, Unix ARM shards, diagnostics, lint, security, |
| 122 | + and final `ci-ok`. |
| 123 | +4. Declare the work complete only when every required job is `success` on the |
| 124 | + post-repair merge SHA. Attach the URLs, SHAs, command output, and any |
| 125 | + remaining environmental limitation to the PR or a durable handoff. |
| 126 | + |
| 127 | +## Explicit non-solutions |
| 128 | + |
| 129 | +- Do not rerun until green without identifying the failing transition. |
| 130 | +- Do not suppress Windows cold-storm coverage. |
| 131 | +- Do not raise the ARM suite timeout or mark the CLI suite optional. |
| 132 | +- Do not make `ci-ok` ignore a failed `test` job. |
0 commit comments