Consume the harness as a plugin instead of carrying a copy of it - #343
Open
pysnooLab wants to merge 8 commits into
Open
Consume the harness as a plugin instead of carrying a copy of it#343pysnooLab wants to merge 8 commits into
pysnooLab wants to merge 8 commits into
Conversation
The harness now lives in marmelab/AIHarness and ships as the `aiharness` Claude Code
plugin. Keeping a second copy here is the drift problem the extraction existed to solve,
and it had already started: the plugin carries fixes this repo does not.
Removed, because the plugin owns them: .claude/{hooks,agents,rules,commands,scripts,
styles,adapters} and the 13 generic skills. 14 568 lines out of the working tree.
Kept, because they are this project's, not the harness's:
- harness.config.json, the contract the plugin reads
- the five domain skills (frontend-dev, backend-dev, shadcn-customization,
delete-initial-resource, update-branding)
- .claude/settings.json permissions and env, minus the 30 hook registrations the plugin
now supplies through its own hooks/hooks.json
- .claude/launch.json, unrelated to the harness
Two config changes are required rather than cosmetic:
`containers.allow: ["supabase"]`. The plugin's docker guard stopped hardcoding a vendor
and reads this list, empty by default so every container launch is blocked. Without the
line the local Supabase stack is refused.
The `e2e` validation step is dropped and `extraForbidden` becomes ["build", "e2e"], the
one part of PR #341 that survives this migration. That step ran Playwright in the
base-branch checkout on every ticket stop, testing code the ticket had not touched.
The suite now runs once at end of feature, launched by a plugin hook. Everything else in
#341 is already in the plugin, so that PR can be closed once this lands rather than merged
and immediately deleted.
.mcp.json is removed: the plugin declares the Playwright MCP server itself, and its
launcher script needed a fix to resolve the consuming project rather than its own
location (marmelab/AIHarness, fix/plugin-mcp-wiring).
The "claude" vitest project goes with the tests it ran: those 387 hook tests live in the
plugin's repo now, with their own CI. The app and functions projects are untouched.
Verified with the harness fully removed: typecheck clean, build clean, app project 143
passed / 1 skipped, functions project 108 passed.
This was referenced Aug 5, 2026
Declare the Playwright MCP server, and resolve it against the consuming project
marmelab/AIHarness#2
Merged
The typecheck script only covered tsconfig.app.json, whose include is src, demo and vitest-browser.d.ts, so e2e/ was never typechecked: the test runner transpiled the specs without checking them. Add tsconfig.e2e.json covering e2e/ and the runner config, chain it into the typecheck script, and give the harness its own validation step so a broken spec fails at ticket time rather than in the end-of-feature suite.
A harness hook auto-committed binary failure screenshots left behind by a test run in an in-progress worktree, which later broke a rebase. Ignore .vitest-attachments/ and test-results/ unanchored, so they are skipped wherever a runner drops them rather than only at the repo root.
e2e specs are loaded by the test runner's Node ESM loader, where the CJS papaparse module arrives wrapped: `import * as Papa from "papaparse"` leaves Papa.parse undefined and the spec fails at runtime. Under Vite the same form works, so src/ keeps it and only e2e/ is restricted. The compiler cannot catch this, since @types/papaparse declares ESM-shaped named exports, so both import forms typecheck. Enforce the default import with no-restricted-imports scoped to e2e/**/*.ts instead.
Profiling one harness run showed where a ticket's turns actually go. The developer on a 3-file ticket spent 4 tool calls on the feature and about 50 on its e2e spec, ~35 of them grepping node_modules to answer four questions this repo already knows: what the toast says after a create, how to open the create form, what the mobile nav contains, and how the create button is labelled. Those greps cost the most expensive turns of the run, because context is largest at the end. Worse, the spec it eventually wrote asserted on the list's sort control unconditionally. The suite runs two projects and the Pixel 5 one has no list toolbar, so it failed, but not until end-of-feature: the per-ticket validation chain does not run the suite. Fixing it there cost another developer, another review and another merge on an already-green feature. Both come down to facts that live in this repo and were written down nowhere an agent reads. So write them down. The new project skill states the two projects and what differs between them (no sort, bulk-select or column header in mobile), the per-resource create labels, the notification texts and which resources have no override, the available fixtures, and the login sequence. Every value in it is checked against playwright.config.ts, englishCrmMessages.ts, MobileNavigation.tsx and the existing specs. It complements the generic aiharness:e2e-conventions skill rather than repeating it: the generic one now says to read the projects array before asserting on anything viewport-dependent, and this one says what that array actually contains. NOTIFICATION and CREATE_BUTTON put the two label tables in fixtures.ts, so a spec imports them instead of re-deriving them. Both patterns for handling the viewport split already exist in the suite (test.skip(isMobile, ...) in bulkContactTags, the if/else in userAddingATask); the skill points at them rather than inventing a third. No helper beyond the constants: a fixture that no spec uses yet cannot be exercised, and this branch cannot run the suite to exercise it.
Two errors in the same sentence, and the first one caused the symptom: harness runs
sometimes asked scoping questions and sometimes did not, with nothing to explain which.
The trigger was conditional ("for a vague or broad request"), so the call was made before
the skill was ever opened, on a judgment nobody wrote down and nobody could review. The
plugin's grill-me now decides for itself in a Step 0 that answers out loud, so this file
should invoke it unconditionally and let it exit in one line when the scope is already
precise. That is the difference between "no questions because it was clear" and "no
questions because nobody ran it", which were previously indistinguishable.
The second error: "the planner never questions" is false. planner.md flags ambiguity
before producing tickets, reports open questions in its output, and stops for one
clarifying question when a request is too vague to decompose safely. The true statement is
narrower and still supports grilling first: one question, at the decomposition threshold,
by which point the dispatch prompt is already written.
Needs AIHarness 0.4.7 (marmelab/AIHarness#32), which ships the Step 0 the first paragraph
now relies on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The agent harness now lives in marmelab/AIHarness and ships as the
aiharnessClaude Code plugin. Keeping a second copy here is exactly the drift problem the extraction existed to solve, and it had already started: the plugin carries fixes this repo does not.That drift is no longer hypothetical. With both copies installed, every hook and every agent is registered twice, and the stale local copy still enforces rules the plugin has since fixed:
.claude/hooks/bash-guard.mjsonmainrefuses any Playwright command lacking--headless, a flag no Playwright subcommand accepts. It blocked two legitimate commands during this session while the plugin's own guard, which checks--headed/--ui/--debug, allowed them.This branch is also the honest test of the extraction. Until a real project runs on the plugin alone, the plugin is theoretical.
Solution
Removed, because the plugin owns them:
.claude/{hooks,agents,rules,commands,scripts,styles,adapters}and the 13 generic skills. 14 568 lines out of the working tree.Kept, because they are this project's and not the harness's:
harness.config.json, the contract the plugin readsfrontend-dev,backend-dev,shadcn-customization,delete-initial-resource,update-branding.claude/settings.jsonpermissions and env, minus the 30 hook registrations the plugin now supplies through its ownhooks/hooks.json.claude/launch.json, unrelated to the harnessTwo config changes that are required, not cosmetic:
containers.allow: ["supabase"]. The plugin's docker guard stopped hardcoding a vendor and reads this list, empty by default so every container launch is blocked. Without the line, the local Supabase stack is refused.e2evalidation step is dropped andextraForbiddenbecomes["build", "e2e"]. That step ran Playwright in the base-branch checkout on every ticket stop, testing code the ticket had not touched; the suite now runs once at end of feature, launched by a plugin hook..mcp.jsonis removed: the plugin declares the Playwright MCP server itself. Its launcher script needed a fix to resolve the consuming project rather than its own location, AIHarness#2, which has landed.The
claudevitest project goes with the tests it ran: those hook tests live in the plugin's repo now (441 today), with their own CI. Theappandfunctionsprojects are untouched.How to test
The app must be provably unaffected, since this only removes tooling:
Verified here with the harness fully removed: typecheck clean, build clean,
app143 passed / 1 skipped,functions108 passed.Then the part that actually matters, the harness itself:
/plugin marketplace add marmelab/AIHarnessthen/plugin install aiharness.node scripts/check-config-sync.mjs --app <this repo>. Green against this branch on plugin 0.2.9: 7 role matchers, all declared.#harnessrequest end to end and confirm the worktree is created, the validation chain runs on the developer's stop, and the merger is refused without a review verdict.mcp__plugin_aiharness_playwright__browser_*, notmcp__playwright__browser_*: a plugin exposes an MCP server namespaced, and the agents declaring the bare name is precisely why the browser leg was silently dead for weeks (fixed in AIHarness#10).Run it from a checkout of this branch, not from
main: onmainthe plugin and the local copy are both registered, so hooks fire twice and the stale copy wins some of the arguments.Note on #341
#341 becomes redundant with this branch. Every fix in it is already in the plugin, that is where they came from, and this branch deletes the files it modifies. The one part that survives is its
harness.config.jsonchange, carried over here. Close it referencing this PR rather than merging then deleting, but keep it open until this lands somainis not left with the broken per-ticket e2e in the meantime.