Skip to content

Commit 73f4826

Browse files
committed
Close out Windows RC qualification
1 parent 88d4dfb commit 73f4826

15 files changed

Lines changed: 1267 additions & 224 deletions

‎docs/CLI_MIGRATION_STATUS.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,3 +383,36 @@ Status: **IMPLEMENTED / REQUIRES EXTERNAL RERUN**
383383
- **RECONCILED:** `uninstall --help` now explicitly limits removal to CLI-owned agent assets/hooks and states that MCP registrations/assets are preserved.
384384
- **CLARIFIED:** legacy MCP SessionStart/SubagentStart payload constants are named and documented as byte-for-byte recognition fingerprints only; current CLI hook generation continues to invoke `codebase-memory-cli hook-augment`.
385385
- **REVIEWED:** release-facing `codebase-memory-mcp` repository URLs remain intentional historical-repository-slug references; compatibility cache/config paths retaining the historical name are likewise unchanged.
386+
### CP83 — integration version reconciliation (static)
387+
388+
**COMPLETE (static only; no compile/test run):** reconciled qualification metadata with the current SpecGen/Agent-Workflow integration set: SpecGen `0.2.1`, Agent-Workflow `0.10.0`, and released `specgen-agent-workflow-contracts` `0.2.1`. The benchmark corpus now records these as expected release versions while retaining immutable commit SHAs as the authoritative frozen-corpus identity. The contracts repository `master` metadata still declares `0.2.0`; codebase-memory-cli intentionally follows the published `v0.2.1` release consumed by both applications rather than rewriting upstream metadata.
389+
390+
**CP84 — COMPLETE (static only; no compile/test run):** tightened integration-corpus initialization after the CP83 version reconciliation. SpecGen `0.2.1` and Agent-Workflow `0.10.0` now declare `master` as their pre-freeze initialization ref; `agent-workflow-spec-contracts` `0.2.1` declares release tag `v0.2.1` because repository `master` still reports project version `0.2.0`. Qualification instructions now require operators to verify the declared version evidence before recording the immutable frozen commit SHA.
391+
392+
**CP85 — COMPLETE (static only; no compile/test run):** made the CP84 corpus initialization contract machine-enforceable. Added `scripts/qualification/validate-corpus-checkouts.py`, a stdlib-only validator for repository origin, cleanliness, initialization ref, version evidence, and frozen commit identity. The corpus manifest now carries structured `version_check` metadata: `pyproject.toml` project versions for SpecGen `0.2.1` and Agent-Workflow `0.10.0`, and exact tag `v0.2.1` for the shared contracts repository. Qualification instructions require validation once with `--allow-unpinned` during initialization and again without it after immutable commit SHAs are recorded. No compilation or test execution was performed for CP85.
393+
394+
395+
**CP86 — STATIC REPAIR / WINDOWS BASELINE PREP:** rebased from the authoritative post-CP85 source and reconciled the current benchmark integration set to SpecGen `0.2.1`, Agent-Workflow `0.10.0`, and shared contracts `0.2.1`. Added native PowerShell benchmark and corpus-orchestration scripts plus `docs/WINDOWS_BENCHMARK_RUNBOOK.md`. Began static reconciliation of the 64 CLI-suite failures reported after CP85: updated CLI skill namespaces, CLI instruction markers, current hook filenames, MCP-preservation expectations, and explicit hook-surface test calls where the tests were still exercising pre-CP78 combined-install semantics. No compile or test run was performed; the next external suite run is the validation source for remaining failures.
396+
397+
**CP87 — STATIC CLI TEST CONTRACT CLOSEOUT:** consumed the external CP86 full-suite evidence (7,391 passed / 22 failed / 3 skipped; all 137 non-CLI suites green) and statically reconciled every reported CLI failure location with the CP78 split lifecycle. Hook-oriented integration tests now use the explicit hook plan/install surface; current CLI hook/script identities use the `codebase-memory-cli` namespace; MCP-era Gemini/Claude hook commands/scripts are asserted foreign and preserved rather than migrated; OpenClaw compaction is treated as an explicit CLI hook surface while MCP compaction remains foreign; and activation ordering tests now use an asset-owned OpenClaw instruction path rather than a hook/config path no longer touched by ordinary `install`. No compilation or test execution was performed for CP87; the next external `scripts/test.sh` run is authoritative validation.
398+
399+
### CP86 — Windows benchmark automation and CLI contract repair
400+
401+
- **COMPLETE (static implementation):** added native Windows benchmark execution (`scripts/benchmark-agent-workflows.ps1`) and five-repository corpus orchestration (`scripts/qualification/run-windows-benchmark-corpus.ps1`) while retaining exact candidate-byte provenance.
402+
- **COMPLETE:** reconciled the benchmark integration set to SpecGen `0.2.1`, Agent-Workflow `0.10.0`, shared contracts `v0.2.1`, and the selected `herdr` scale corpus.
403+
- **EVIDENCE:** external full-suite execution reduced CLI failures from 64 to 22; every non-CLI suite remained green.
404+
405+
### CP87 — CLI test-contract closeout
406+
407+
- **COMPLETE (static repair):** reconciled the 22 residual CLI assertions with the CP78 explicit-hook/MCP-preservation contract without restoring MCP ownership or implicit hook installation.
408+
- **EVIDENCE:** external targeted CLI execution completed 236 passes with only two Claude hook ownership/mutation assertions remaining.
409+
410+
### CP88 — Windows RC qualification closeout
411+
412+
- **COMPLETE (static implementation; no local compile/test run):** adds `scripts/qualification/run-windows-rc-qualification.ps1`, a top-level native-Windows RC qualification orchestrator covering exact artifact acquisition/hash verification, machine-state capture, portable smoke/canonical workflow, retained Windows guards against supplied candidate bytes, explicit recovery cycles, isolated installed-product/CLI-hook/MCP-coexistence lifecycle checks, five-repository benchmark execution, promotion-compatible evidence generation, evidence self-verification, and optional draft-release upload.
413+
- **COMPLETE:** `scripts/setup-windows.ps1` accepts `-Binary`, `-InstallDir`, and `-NoPathPrompt` so qualification can install byte-identical RC candidates without rebuilding or substituting the public `latest` release.
414+
- **COMPLETE:** Windows benchmark workloads are repository-specific and frozen in `BENCHMARK_CORPUS.json`; the validator rejects missing workload files/fields before measurement begins.
415+
- **FIXED (benchmark correctness):** the PowerShell benchmark harness now fails a repository run on any required warm-up/measured operation failure and emits explicit `RESULT`/`FAILURES.txt` evidence instead of silently excluding failures from performance summaries.
416+
- **FIXED (evidence isolation):** corpus initialization writes the frozen manifest into the qualification evidence tree rather than dirtying the exact RC source checkout.
417+
- **RECONCILED:** the last two known Claude-hook tests now enforce exact current CLI command ownership while preserving MCP-era hook identities as foreign, matching the CP78 side-by-side contract.
418+
- **PENDING EXTERNAL EVIDENCE:** run the external CLI/full suite after CP88 and then execute the Windows RC orchestrator on `luigi.home.arpa` against the immutable draft artifact. No compile or test execution was performed locally for CP88.

‎docs/RELEASE_QUALIFICATION_PLAN.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,8 @@ The first Windows benchmark generation (`windows-corpus-1`) uses five fixed repo
166166
- `specgen-aw` (`https://github.com/ngallodev-software/specgen-aw.git`);
167167
- `agent-workflow` (`https://github.com/ngallodev-software/agent-workflow.git`);
168168
- `agent-workflow-spec-contracts` (`https://github.com/ngallodev-software/agent-workflow-spec-contracts.git`);
169+
170+
For the current integration set, qualification expects SpecGen `0.2.1`, Agent-Workflow `0.10.0`, and the released `specgen-agent-workflow-contracts` package `0.2.1`. The corpus manifest records both the expected version and the pre-freeze initialization ref: `master` for SpecGen and Agent-Workflow, and release tag `v0.2.1` for the shared contracts repository because its current `master` project metadata still reports `0.2.0`. Frozen commit SHAs remain the authoritative benchmark identity after initialization. The machine-verifiable `version_check` object is enforced by `scripts/qualification/validate-corpus-checkouts.py`; initialization uses `--allow-unpinned`, then the validator is rerun without that flag after exact commit SHAs are recorded.
169171
- `herdr` (`https://github.com/herdrdev/herdr.git`).
170172

171173
Exact commit SHAs are not selected from remote HEAD during planning. They are recorded from the actual clean checkouts frozen for qualification on `luigi.home.arpa`. After that freeze, any commit change creates a new corpus generation and requires new baseline capture.

‎docs/WINDOWS_BENCHMARK_PROTOCOL.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ Use the following stable five-repository corpus for `windows-corpus-1`:
5050
2. `specgen-aw` — realistic specification-authoring application from the user's development workflow: `https://github.com/ngallodev-software/specgen-aw.git`.
5151
3. `agent-workflow` — realistic companion workflow application: `https://github.com/ngallodev-software/agent-workflow.git`.
5252
4. `agent-workflow-spec-contracts` — the shared contract dependency used by SpecGen/Agent-Workflow integration: `https://github.com/ngallodev-software/agent-workflow-spec-contracts.git`.
53+
54+
Current integration-version expectations for the first frozen corpus are SpecGen `0.2.1`, Agent-Workflow `0.10.0`, and the released shared contracts package `0.2.1`. Before freeze, initialize SpecGen and Agent-Workflow from their declared `master` integration refs, and initialize `agent-workflow-spec-contracts` from release tag `v0.2.1`; its current `master` metadata still reports `0.2.0` and must not be substituted for the released contract package consumed by the two applications. These initialization refs are pre-freeze selection constraints, not substitutes for the immutable commit SHA captured from each frozen checkout. If upstream moves again before the corpus is frozen, update the expected version/ref and the resulting pinned commit before baseline capture. Before recording the frozen SHAs, run `python scripts/qualification/validate-corpus-checkouts.py --allow-unpinned`; after replacing the commit placeholders, rerun the validator without `--allow-unpinned`.
5355
5. `herdr` — larger open-source, multi-language workload used to add scale and language diversity: `https://github.com/herdrdev/herdr.git`.
5456

5557
The repository identities are fixed for this corpus generation. The exact commit for each entry is deliberately captured from the actual frozen checkout used to initialize `luigi.home.arpa`; do not substitute current remote HEAD during planning. Once those commits are recorded, changing any one of them creates a new corpus generation and requires a new baseline.
@@ -75,7 +77,7 @@ Do not reuse caches between baseline and candidate.
7577

7678
## Workload semantics
7779

78-
Use `scripts/benchmark-agent-workflows.sh` (or its Windows wrapper when added) with identical workload parameters for baseline and candidate.
80+
Use `scripts/benchmark-agent-workflows.ps1` on native Windows (the Linux counterpart is `scripts/benchmark-agent-workflows.sh`) with the repository-specific frozen workload parameters in `BENCHMARK_CORPUS.json`. Identical parameters are required for baseline and candidate.
7981

8082
Measure real agent workflows:
8183

0 commit comments

Comments
 (0)