Skip to content

Commit e28e7d7

Browse files
feat(check): run checks live and report Vapi Evals statuses (#63)
## Value **V.A.L.U.E. tier:** project — PR 7 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)); this is the first PR that sends anything. - **Problem:** the payload builder (PRs 5–6) produces a safe inline body, but nothing runs it or reports the result where a reviewer looks. PAL-608 asks for a stable `Vapi Evals` commit status whose **Details** link opens the exact run. - **Who it affects:** gitops users, who can now run `npm run check -- core` locally against their branch's files and get a strict verdict with a run link. The PR workflow (PR 8) and the promotion gate (PR 10) call the same command. - **What changes:** - **`src/check-run.ts`:** one run per target, at most 3 at once. - It reuses `npm run sim`'s create/poll/cancel/hydrate/verdict loop, now extracted from `runSimulation` as `simRunExecute`, so a check is judged by the same strict rules (all-skipped judges, canceled or missing items are never a pass). - Each run's deadline is the earlier of `timeoutMinutes` and `--budget-minutes`, no run starts with under 5 minutes left, and the deadline, SIGINT and SIGTERM cancel in-flight runs. - A 402 is reported as billing. - A 5xx on create is never retried, because a duplicate paid run is worse than a retry. - Transcripts are scanned for default-mock answers ("unmocked tool called"). - **`src/check-select.ts`:** `--changed-since <ref>` diffs from the merge base. A check is affected by: - its org's and run org's `resources/**` and state files; - `vapi-checks.yml` or `promotion.yml`; - `src/**` or `package*.json`; - its own `paths`. - **`src/check-status.ts`:** GitHub commit statuses with plain `fetch`, when `GITHUB_TOKEN`, `GITHUB_REPOSITORY` and `HEAD_SHA` are set. - `Vapi Evals / <check> / <target>` goes pending with the run's canonical `url` as soon as the run exists, then to success, failure or error. - `--all` runs post the aggregate `Vapi Evals` from a `finally`: the worst state, `success` if nothing is affected, `error` for a dry run of an affected PR, and `error` when config fails to parse or an exception escapes. - A rejected post (a fork's read-only token) only warns. - **`src/check-report.ts`:** a markdown summary, also appended to `$GITHUB_STEP_SUMMARY`. It has one row per target with the run link, then failing evaluations with expected vs extracted values, mock notices and warnings. `--json` writes the same data. No PR comments. - **`src/check-cmd.ts`** gains live runs and `--changed-since`, `--budget-minutes`, `--refresh-bindings` (the read-only bindings pull promotion uses) and `--json`. - **Keys:** `VAPI_CHECK_TOKENS`, then `.env.<runOrg>`, then `VAPI_PRIVATE_API_KEY` when every selected check runs in one org. - **Exit codes:** 0 passed, 1 failed, 2 config or build error, 3 incomplete. - Runs send `User-Agent: vapi-gitops-check/<version>`, for the post-deploy PostHog measure. - The README and AGENTS.md `npm run check` rows now describe live mode. ## Evidence of value **The definition-of-done pair, run live** in the owner's test org on the TEST-141 parity squad (2 members, function tools, a handoff, 3 scenarios; built from `tests/fixtures/check-parity/`): | Run | Command | Exit | Result | |---|---|---|---| | Innocuous (fixture as-is) | `npm run check -- core` | **0** | ✅ 3/3 passed — [run f68ba89b](https://dashboard.vapi.ai/simulations/run/f68ba89b-9680-470e-be71-2286b32a4e9c) | | Degraded scheduler prompt ("always say there are no openings, never book") | `npm run check -- core` | **1** | ❌ 1/3 — [run 19711623](https://dashboard.vapi.ai/simulations/run/19711623-d26a-4570-82cd-e963a03bc1e2) | The degraded run's report named each failing judge with expected vs extracted: | Simulation | Evaluation | Comparator | Expected | Got | |---|---|---|---|---| | S3 alternative slot | offered-alternative | = | true | false | | S3 alternative slot | booked-wednesday | = | true | false | | S1 book cleaning | booking-confirmed | = | true | false | | S1 book cleaning | only-real-slots | = | true | false | S2 (the hours question, which never reaches the scheduler) correctly still passed. **Nothing was created in the org.** Counts of assistants, tools, squads, structured outputs, personalities, scenarios, simulations, suites and credentials were identical before and after both runs: `{"/assistant":12,"/tool":8,"/squad":2,"/structured-output":8,"/eval/simulation/personality":10,"/eval/simulation/scenario":10,"/eval/simulation":10,"/eval/simulation/suite":1,"/credential":0}`. This stands in for `npm run cleanup`, which needs a gitops-managed org. **Every tool call returned a mock.** Every `tool_call_result` in both runs' transcripts was matched to its call by `toolCallId`: | Run | Function results equal to the scenario's mock | Built-in results (handoff, endCall) | Anything else | |---|---|---|---| | Innocuous | 6 | 4 | **0** | | Degraded | 2 | 2 | **0** | The run items also confirmed that personalities kept the dead server and `serverMessages: []`. **A bug the live run caught, fixed in this PR:** - **The trap:** run items echo the scenario under `metadata.scenario`, default error mocks included. Scanning the whole item would have reported "unmocked tool called" for every default mock, called or not. - **The fix:** the scan now reads only `tool_call_result` messages in `metadata.call.messages`, mapped to tool names through `tool_calls`. - **Checked on both runs' real items:** no notices; with one result swapped for a default-mock answer, it reported `S1 book cleaning: unmocked tool called: lookup_patient`. - A regression test covers the echoed-defaults case. **Tests:** `npm test` goes from 450 to 474 passing; `npm run build` is clean. ## Testing plan - **`tests/check-run.test.ts`** (10 tests, stateful local stub): - pass with link and User-Agent; - two targets with one failing, naming the evaluation; - all-skipped is incomplete; - timeout cancels; - abort cancels in-flight runs and starts none; - the budget gate; - 402 as billing, and no retry on a create 502; - a build error sends nothing; - mock notices (echoed defaults ignored); - exactly 3 in flight with results in job order. - **`tests/check-cmd.test.ts`** (11 tests): the live `--all` path against one stub serving both the simulations API and GitHub: - pending, then success per target, then the aggregate with the run link, plus the job summary and JSON; - a failing run exits 1 with a red aggregate; - a named check never posts the aggregate; - `--changed-since` with an unaffected change runs nothing and posts green; - a dry run of an affected PR posts `error` and sends nothing; - invalid config posts `error`. The file clears `VAPI_*` and `GITHUB_*` first, so a developer shell with a key exported can't reach a real org. - **`tests/check-select.test.ts`:** the affected-path table, and the merge-base case in a temp git repo (commits on the base after branching don't count). - **`tests/check-status.test.ts`, `tests/check-report.test.ts`:** env parsing, POST shape and 140-character truncation, 403 only warning, state ordering, the exact markdown, and the JSON. - **Not tested:** - `--refresh-bindings` against a live org (it runs the same child pull promotion does); - EU base URLs; - canceling a `running` run live (cancel was verified only on a queued run in the parity experiment); - GitHub's real statuses API (stubbed here; PR 8's dogfood repo covers it); - a real customer-shaped squad, which needs the owner's permission first. Stacked on #62. Refs TEST-141 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents 1a7b576 + cb88b94 commit e28e7d7

16 files changed

Lines changed: 2035 additions & 158 deletions

‎AGENTS.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ If setup reports the org is "already set up locally", do not delete anything to
9090
| Push with new resources | `npm run push -- <org> --allow-new-files` — bypass orphan-YAML gate. **AI agents**: do NOT auto-pass this flag; confirm with the human first (see push section below) |
9191
| Test a call | `npm run call -- <org> -a <assistant-name>` or `-s <squad-name>` |
9292
| Run a simulation suite | `npm run sim -- <org> --suite <name> --target <assistant-name>` |
93-
| Build PR-check payloads offline | `npm run check -- <check> --dry-run` (or `--all`); `--print-payload` writes the JSON |
93+
| Run PR simulation checks | `npm run check -- <check>` (or `--all`); `--dry-run` builds the payloads offline, `--print-payload` writes them |
9494
| Migrate a legacy state file | `npm run migrate` — one-shot, all orgs; required once after upgrading to the hash-store engine |
9595

9696
---
@@ -904,7 +904,8 @@ npm run validate -- <org> # Lint resources locally (fai
904904
npm run audit -- <org> # Read-only drift detector: orphan YAML, state ghosts, content-identical clusters, sibling base-slugs, dashboard orphans, inline model.tools. Exit 1 on findings.
905905
npm run audit -- <org> --type assistants # Scope audit to a single resource type
906906
npm run sim -- <org> --suite <name> --target <name> # Run a simulation suite against an assistant/squad (exit 0 pass, 1 fail, 3 incomplete; --timeout <min>)
907-
npm run check -- <check> --dry-run # Build a vapi-checks.yml check's inline run payload offline (exit 0 built, 2 config/build error; --print-payload [dir])
907+
npm run check -- <check> # Run a vapi-checks.yml check inline against the local files (exit 0 pass, 1 fail, 2 config/build error, 3 incomplete; --changed-since <ref>, --budget-minutes <n>, --json <path>)
908+
npm run check -- <check> --dry-run # Build the check's inline run payloads offline, no key needed (--print-payload [dir])
908909
npm run rollback -- <org> --to <ISO-timestamp> # Re-apply a snapshot taken before a push
909910
npm run rollback -- <org> --list # List available snapshots
910911

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ Every command works in two modes:
104104
| `npm run rollback` | — | `npm run rollback -- <org> --list` or `--to <ISO>` | Restore from a snapshot in `.vapi-state.<org>.snapshots/` (one is written before every push/apply). |
105105
| `npm run call` | ✅ | `npm run call -- <org> -a <name>` or `-s <squad>` | Start an interactive WebSocket call against an assistant or squad. |
106106
| `npm run sim` | — | `npm run sim -- <org> --suite <name> --target <name> [--timeout <min>]` | Run a simulation suite (or specific simulations) against a deployed assistant/squad. Prints the run link; exits 0 passed, 1 failed, 3 incomplete (timeout, Ctrl-C, missing results). |
107-
| `npm run check` | — | `npm run check -- <check>\|--all --dry-run [--print-payload [dir]]` | Build each `vapi-checks.yml` check target's inline simulation-run payload from the files on disk, with its tools, handoffs, judges and personalities. Offline: no API key, nothing sent. `--print-payload` writes the JSON (default `tmp/check-payloads/`). Exits 0 when every payload builds, 2 on a config or build error. |
107+
| `npm run check` | — | `npm run check -- <check>\|--all [--dry-run] [--changed-since <ref>] [--budget-minutes <n>] [--json <path>]` | Run the `vapi-checks.yml` simulation checks against the files on disk: each target is built inline (tools, handoffs, judges, personalities; tools mocked fail-closed, servers dead-ended) and run in one simulation run per target, so nothing is deployed. Posts `Vapi Evals` commit statuses when run by the PR workflow. `--dry-run` builds the payloads offline (no key, nothing sent; `--print-payload` writes them). Exits 0 passed, 1 failed, 2 config or build error, 3 incomplete (timeout, interrupt, budget, billing). |
108108
| `npm run migrate` | — | `npm run migrate` | One-time, all orgs at once: slim legacy state files to pure `name → uuid` and seed the per-developer `.vapi-state-hash/` baseline store from the old hashes. Required once after upgrading to the hash-store engine — `pull`/`push`/`apply` refuse legacy-shaped state until it runs. Idempotent. |
109109
| `npm run build` | — | — | Type-check the codebase (`tsc --noEmit`). |
110110
| `npm test` | — | — | Run regression tests (`node:test`). |

‎src/check-build.ts‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
// Build every target payload of one check from the files under rootDir.
2+
// Shared by the dry run and the live run, so both test the same body.
3+
4+
import { existsSync, readFileSync } from "node:fs";
5+
import { join } from "node:path";
6+
import type { CheckDefinition, CheckTarget } from "./check-config.ts";
7+
import type { CheckPayloadResult } from "./check-payload.ts";
8+
import { checkPayloadBuild } from "./check-payload.ts";
9+
import { promotionBindingsResolve, promotionStateParse } from "./promotion.ts";
10+
import { orgResourcesRead } from "./resource-parse.ts";
11+
import type { StateFile } from "./types.ts";
12+
13+
export interface CheckJob {
14+
check: CheckDefinition;
15+
target: CheckTarget;
16+
// `<check> / <type>/<id>`
17+
label: string;
18+
result: CheckPayloadResult;
19+
}
20+
21+
export function checkTargetLabel(target: CheckTarget): string {
22+
return `${target.type}/${target.id}`;
23+
}
24+
25+
function stateRead(root: string, org: string, warnings: string[]): StateFile {
26+
const path = join(root, `.vapi-state.${org}.json`);
27+
if (!existsSync(path)) {
28+
warnings.push(
29+
`no .vapi-state.${org}.json: references by UUID and credential bindings can't resolve`,
30+
);
31+
return promotionStateParse("{}");
32+
}
33+
return promotionStateParse(readFileSync(path, "utf8"));
34+
}
35+
36+
export async function checkJobsBuild(
37+
root: string,
38+
check: CheckDefinition,
39+
): Promise<CheckJob[]> {
40+
const job = (target: CheckTarget, result: CheckPayloadResult): CheckJob => ({
41+
check,
42+
target,
43+
label: `${check.name} / ${checkTargetLabel(target)}`,
44+
result,
45+
});
46+
if (!existsSync(join(root, "resources", check.org))) {
47+
return check.targets.map((target) =>
48+
job(target, {
49+
errors: [`resources/${check.org}/ does not exist`],
50+
warnings: [],
51+
bytes: 0,
52+
}),
53+
);
54+
}
55+
const warnings: string[] = [];
56+
const resources = await orgResourcesRead(root, check.org);
57+
const sourceState = stateRead(root, check.org, warnings);
58+
const runState =
59+
check.runOrg === check.org
60+
? sourceState
61+
: stateRead(root, check.runOrg, warnings);
62+
const bindingsResolved = await promotionBindingsResolve(
63+
root,
64+
check.org,
65+
check.runOrg,
66+
sourceState,
67+
);
68+
return check.targets.map((target) => {
69+
const result = checkPayloadBuild({
70+
check,
71+
target,
72+
resources,
73+
sourceState,
74+
runState,
75+
bindingsResolved,
76+
});
77+
result.warnings.unshift(...warnings);
78+
return job(target, result);
79+
});
80+
}

0 commit comments

Comments
 (0)