fix(tests): make the test loop trustworthy and type-checked for agents (EveryInc#1835)

Author: tmchow

.agents/skills/ce-skill-work/SKILL.md

@@ -36,7 +36,7 @@ Each block starts from the same possible elements, in this order: the result and
 - **Long-running skills state their pacing.** A skill that owns a long loop or dispatches workers instructs batching (independent calls and dispatches issued in one response; serialize only real dependencies), narration (what the user hears before, during, and at close), and completion honesty (a step is done only when actually performed; the turn does not end on described-but-undone work). A skill that runs a few calls and returns omits all three — see the guide's long-running execution section.
 - **Autonomy is one compact policy.** Name safe local actions and let in-scope work that follows from the user's request proceed, including an external write that is the requested job or named in the skill's authority envelope. Confirm only when an external write, destructive action, purchase, or material scope expansion is outside that envelope, or when only the user can supply the input. Do not repeat "ask first", "do not mutate", or "wait for approval" at each step.
 - **Match freedom to fragility.** High freedom for many valid approaches; medium freedom for a preferred pattern or parameterized command; low freedom when one known-good command or sequence exists and agents fail if they invent it. Write a fragile command once; collapse easy variants to one skeleton plus deltas.
-- **Validate to the risk.** Mechanical contracts (frontmatter, paths, greppable invariants) go in `bun test`. Behavior-bearing prose changes get a targeted eval per `references/evaluate.md`, on Claude and Codex, or an explicit skip reason in the report. Never ship an untested behavior change as "reference".
+- **Validate to the risk.** After any edit, `bun run test:skill-guards` runs the mechanical guards in about a second, including the 8,000-byte SKILL.md budget in `tests/codex-skill-prompt-budget.test.ts`. Mechanical contracts (frontmatter, paths, greppable invariants) go in `bun test`. Behavior-bearing prose changes get a targeted eval per `references/evaluate.md`, on Claude and Codex, or an explicit skip reason in the report. Never ship an untested behavior change as "reference".
 
 ## Modes
 

.github/workflows/ci.yml

@@ -132,6 +132,10 @@ jobs:
       - name: Run tests
         run: bun run test
 
+      - name: Typecheck
+        if: ${{ !cancelled() }}
+        run: bun run typecheck
+
   # Focused Windows gate for the skill scripts that reach for POSIX process and
   # file primitives: peer-job-runner (#1243) and ce-babysit-pr's pr-snapshot
   # (#1280). Not a full-suite matrix — POSIX lifecycle tests (fork/killpg/ps)

AGENTS.md

@@ -13,6 +13,7 @@ It also contains:
 ```bash
 bun install
 bun run test              # full test suite (also runs in CI; `--parallel` across worker processes)
+bun run typecheck         # tsc over src, scripts, and tests (also runs in CI)
 bun run release:validate  # plugin/marketplace consistency (also runs in CI)
 bun run plugin:validate   # Claude marketplace + plugin schema (also runs in CI; needs `claude` on PATH)
 ```
@@ -37,7 +38,7 @@ bun run codex:dev -- remove   # remove both supported CE installation surfaces
 - **Contribution gate (non-maintainers):** If you are not a repository maintainer or admin, do not open a PR without a linked issue — file the issue first and reference it from the PR. Adding a **new skill** has a stricter gate: non-maintainers and non-admins must raise a discussion in an issue and get explicit maintainer approval **before** starting the work; do not open a new-skill PR that has not been approved this way. Maintainers and admins are exempt from both gates but still follow the merge policy above.
 - **PR disclosure:** `.github/pull_request_template.md` ends with `## Security Disclosure` and `## Agent Disclosure` sections. Fill both when opening a PR — including PRs authored via `gh pr create --body`/`--body-file`, which bypass the template so nothing pre-fills them. State any security-relevant changes (or "No security-relevant changes"), and the model that did the bulk of the work — your harness plus the most specific model identity your own context gives you, e.g. `Claude Code · claude-opus-4-8` or `Codex CLI · GPT-5`. Copy an exact model ID verbatim when your harness states one; when it exposes only a generic family, report the family and stop. Measured 2026-07-24: Codex and Cursor agents cannot see their running model at all (Codex's "based on GPT-5" is fixed boilerplate), so do not upgrade a family to a version, and do not read config files for one — the configured default is often not the model actually running. Never invent a version or variant. The body above those sections stays freeform — add whatever sections best explain the change.
 - **Safety:** Do not delete or overwrite user data. Avoid destructive commands.
-- **Testing:** Run `bun run test` after changes that affect parsing, conversion, output, skill conventions, or other mechanical guards. Local `bun run test` is the same suite CI runs — there is no separate local-only unit-test lane. Prefer it over bare `bun test`: the package script carries `--parallel`, which is where the suite's speed comes from. Bare `bun test <file>` is still the right tool for iterating on one file. The suite uses disposable fixtures and has no production access. Run it, fix failures caused by the requested change, and rerun affected tests without asking for approval at each step.
+- **Testing:** Run `bun run test` after changes that affect parsing, conversion, output, skill conventions, or other mechanical guards. Local `bun run test` is the same suite CI runs — there is no separate local-only unit-test lane. Prefer it over bare `bun test`: the package script carries `--parallel`, which is where the suite's speed comes from. Bare `bun test <file>` is still the right tool for iterating on one file; for a subset, `bun run test <paths>` keeps `--parallel` (bare `bun test <dir>` runs serially). After editing a skill, `bun run test:skill-guards` runs the skill-convention and contract guards in about a second. The suite uses disposable fixtures and has no production access. Run it, fix failures caused by the requested change, and rerun affected tests without asking for approval at each step.
 - **Compounding learnings:** After a solved, verified problem, automatically invoke the `ce-compound` skill with `mode:non-interactive` at the completion checkpoint only when the work produced durable project reasoning that is not readily recoverable from the final code, tests, types, comments, or existing documentation, and losing it would plausibly cause recurrence, material risk, or substantial rediscovery. Apply this counterfactual: if the learning document disappeared, would a future engineer reading the final implementation still be likely to repeat the mistake or redo substantial investigation? If not, do not invoke it. Completion, effort, and diff size alone are not enough. Capture at the checkpoint so a qualifying learning can ship in the PR that produced it, and only where the repository treats captured learnings as tracked, committed knowledge. This repository does: `docs/solutions/` is tracked (see *Repository Docs Convention*). If `ce-compound` is not callable in the current harness (a checkout without the plugin installed or linked), do not block and do not skip silently: say in the completion report that a qualifying learning was left uncaptured, so the author can run it from a plugin-enabled session.
 - **Release versioning:** Releases are prepared by release automation, not normal feature PRs. The repo has one root plugin/package release component (`compound-engineering`) plus marketplace components (`marketplace`, `cursor-marketplace`). GitHub release PRs and GitHub Releases are the canonical release-notes surface for new releases; root `CHANGELOG.md` is only a pointer to that history. Use conventional titles such as `feat:` and `fix:` so release automation can classify change intent, but do not hand-bump release-owned versions or hand-author release notes in routine PRs.
 - **Output Paths:** Keep OpenCode output at `opencode.json` and `.opencode/{agents,skills,plugins}`. For OpenCode, commands go to `~/.config/opencode/commands/<name>.md`; `opencode.json` is deep-merged (never overwritten wholesale).
@@ -106,6 +107,8 @@ This repository authors each skill once and distributes it across multiple agent
 
 **Before creating, editing, reviewing, or acting on review feedback for anything under `skills/**`, invoke the repo-local `ce-skill-work` skill** (`.agents/skills/ce-skill-work/`, which Codex and Cursor discover directly; `.claude/skills` is a symlink to `.agents/skills` for Claude Code). It carries the procedures for each of those four activities, the audit questions, the provenance rule for removals, and the validation contract. The same routing applies when a skill-authoring best practice itself changes or is newly learned — a prompt-guide lesson, a tuning that demonstrably worked, a standard-level correction: invoke `ce-skill-work` and record the practice in `docs/solutions/skill-design/portable-agent-skill-authoring.md` (and here when it must be always-loaded), never only in one skill's prose. This file states only what must be always-loaded; when the two disagree, fix the disagreement rather than following the shorter one.
 
+A SKILL.md under `skills/` must stay within Codex's 8,000-byte skill prompt budget; `tests/codex-skill-prompt-budget.test.ts` enforces it as a ratchet, and its `OVER_BUDGET` set lists the current exemptions.
+
 Three rules that hold regardless of whether the skill was invoked:
 
 - **State conditions, not procedures or cases.** When a block keeps absorbing "add the case we just found" — in authoring, in a review round, or in your own fix to a finding — the representation is wrong. Delete the additions and restate the goal, then re-verify against every path the additions served; a restatement that no longer names a path is a new defect, not a simplification.
@@ -163,7 +166,7 @@ Behavioral changes to a plugin skill or skill-local persona (anything under `ski
 
 ## CI and Quality Gates
 
-PR CI (`.github/workflows/ci.yml`) is the merge gate. It runs, in order: PR-title lint (PRs only), `bun run release:validate`, `bun run plugin:validate`, and `bun run test`. A separate `docs-audit` job runs compound-cli's `compound audit` over `docs/solutions/`; it stays out of `bun run test` and `release:validate` so the local lane never needs compound-cli or network access. Do not invent a parallel local-only mechanical suite — if a check is deterministic and should block merges, put it in one of those steps (usually `bun run test`).
+PR CI (`.github/workflows/ci.yml`) is the merge gate. It runs, in order: PR-title lint (PRs only), `bun run release:validate`, `bun run plugin:validate`, `bun run test`, and `bun run typecheck`. A separate `docs-audit` job runs compound-cli's `compound audit` over `docs/solutions/`; it stays out of `bun run test` and `release:validate` so the local lane never needs compound-cli or network access. Do not invent a parallel local-only mechanical suite — if a check is deterministic and should block merges, put it in one of those steps (usually `bun run test`).
 
 The `test` script is `scripts/run-tests.ts`: one `bun test --parallel` pass (implies `--isolate`), then, only if every first-pass failure is a `TimeoutError`, one serial re-run of those files in a fresh bun process. A pass whose bun worker leaves an exited child unreaped for a minute is killed and repeated once in full. An assertion failure or error anywhere keeps the first result with no re-run. Do not add `retry` counts for this signature — a retry runs inside the same worker. A test file may not depend on another file's leftovers, and any test that writes outside its own `mktemp` directory is a latent flake. A test that runs a bundled script which inspects the repository must point that script at a throwaway repo, never this checkout. **Do not pin a worker count.** Size a test file by its measured time, not its line count. CI wall-time measurements and the per-file timing recipe: `docs/solutions/developer-experience/always-on-agents-md.md`. The TimeoutError re-run exists because bun can lose a child-exit inside a worker (oven-sh/bun#34069); details: `docs/solutions/developer-experience/bun-parallel-worker-loses-subprocess-exit.md`.
 

bun.lock

@@ -0,0 +1,2 @@
+[test]
+pathIgnorePatterns = ["tests/skill-eval-cell/fixtures/**"]