Skip to content

Commit 926d68d

Browse files
darrenhindeclaude
andauthored
fix(evals): repair deterministic eval baseline (behavioral regressions + tsc build) (#334)
* fix(evals): repair six eval-framework behavioral regressions Restore the deterministic eval baseline by fixing false-negatives and identity/discovery bugs across the framework: - error-handling: correct severity-boundary fixtures so mixed-severity cases actually cross the >1000 warning threshold (deterministic error/warning/info), plus explicit 1000-vs-1001 boundary tests. - context-loading-evaluator: emit `no-context-loaded` and `context-loaded-after-execution` as errors so missing/late context fails (was a passing warning); update focused test accordingly. - tool-usage-evaluator: enforce dedicated tools — bash `cat`/`ls`/`ls -la` now fail as `bash-antipattern-*` errors (ls de-allowlisted); npm/git/ piped commands still pass. - test-case-loader: canonicalize legacy agent ids via normalizeAgentId (`openagent` -> `core/openagent`), preserving canonical/unknown ids. - suite-validator: accept category-based agent ids, resolve test paths under the canonical dir, add requested-vs-declared conflict detection; new suite-validator.test.ts. - validate-suites-cli: recursive suite discovery (nested <cat>/<agent>/ config), deterministic order, fail-closed on zero suites; new validate-suites-cli.test.ts. Deterministic Vitest allowlist green; validate:suites:all non-vacuous and exits 0. No model/network/eval:sdk execution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(evals): clear pre-existing TypeScript build errors (tsc 39 -> 0) `npm run build` (tsc) failed on 39 pre-existing errors in files outside the behavioral-repair scope, blocking the baseline gate. Root causes: - glob: drop obsolete @types/glob ^8 so glob v13's own types (globSync) resolve (fixes run-sdk-tests.ts). - task-type-detector: introduce a local DetectedTaskType union for this (dead/unused) detector; shared TaskType union left unchanged. - behavior-evaluator: the expectedResponse check filtered a nonexistent `message` event type and was silently dead — route through base-evaluator.getAssistantMessages (assistant_message/text). Adds three deterministic tests exercising the now-live check. - integration tests: remove unused `expectedOutcome` fixture metadata (not on TestCase, never asserted) and add the required `category` field the excess-property error had masked. tsc now exits 0. Vitest allowlist 112/112. Lint remains unconfigured in evals/framework (no ESLint config has ever existed) and is deferred to a separate follow-up; it is not part of this change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(evals): sync bun.lock after @types/glob removal The previous commit removed @types/glob via npm, updating package-lock.json but leaving the primary bun.lock stale, which broke CI's `bun install --frozen-lockfile`. Regenerate bun.lock so both lockfiles agree (drops @types/glob and its transitive @types/minimatch only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(evals): add evals/framework/package-lock.json for build-check The pr-checks `build-check` job caches on and runs `npm ci` in evals/framework, but that workspace had no local lockfile, so the job failed at Setup Node.js ("Some specified paths were not resolved") and would fail `npm ci`. Add a standalone, in-sync package-lock.json (lockfileVersion 3, no @types/glob) so the job's cache path resolves and `npm ci` succeeds. Validated locally with `npm ci --dry-run`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 0c99ddc commit 926d68d

19 files changed

Lines changed: 5013 additions & 1090 deletions

bun.lock

Lines changed: 0 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)