Repository navigation
Commit e28e7d7
authored
feat(check): run checks live and report Vapi Evals statuses (#63)
## Value
**V.A.L.U.E. tier:** project — PR 7 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)); this is the first PR that sends anything.
- **Problem:** the payload builder (PRs 5–6) produces a safe inline body, but nothing runs it or reports the result where a reviewer looks. PAL-608 asks for a stable `Vapi Evals` commit status whose **Details** link opens the exact run.
- **Who it affects:** gitops users, who can now run `npm run check -- core` locally against their branch's files and get a strict verdict with a run link. The PR workflow (PR 8) and the promotion gate (PR 10) call the same command.
- **What changes:**
- **`src/check-run.ts`:** one run per target, at most 3 at once.
- It reuses `npm run sim`'s create/poll/cancel/hydrate/verdict loop, now extracted from `runSimulation` as `simRunExecute`, so a check is judged by the same strict rules (all-skipped judges, canceled or missing items are never a pass).
- Each run's deadline is the earlier of `timeoutMinutes` and `--budget-minutes`, no run starts with under 5 minutes left, and the deadline, SIGINT and SIGTERM cancel in-flight runs.
- A 402 is reported as billing.
- A 5xx on create is never retried, because a duplicate paid run is worse than a retry.
- Transcripts are scanned for default-mock answers ("unmocked tool called").
- **`src/check-select.ts`:** `--changed-since <ref>` diffs from the merge base. A check is affected by:
- its org's and run org's `resources/**` and state files;
- `vapi-checks.yml` or `promotion.yml`;
- `src/**` or `package*.json`;
- its own `paths`.
- **`src/check-status.ts`:** GitHub commit statuses with plain `fetch`, when `GITHUB_TOKEN`, `GITHUB_REPOSITORY` and `HEAD_SHA` are set.
- `Vapi Evals / <check> / <target>` goes pending with the run's canonical `url` as soon as the run exists, then to success, failure or error.
- `--all` runs post the aggregate `Vapi Evals` from a `finally`: the worst state, `success` if nothing is affected, `error` for a dry run of an affected PR, and `error` when config fails to parse or an exception escapes.
- A rejected post (a fork's read-only token) only warns.
- **`src/check-report.ts`:** a markdown summary, also appended to `$GITHUB_STEP_SUMMARY`. It has one row per target with the run link, then failing evaluations with expected vs extracted values, mock notices and warnings. `--json` writes the same data. No PR comments.
- **`src/check-cmd.ts`** gains live runs and `--changed-since`, `--budget-minutes`, `--refresh-bindings` (the read-only bindings pull promotion uses) and `--json`.
- **Keys:** `VAPI_CHECK_TOKENS`, then `.env.<runOrg>`, then `VAPI_PRIVATE_API_KEY` when every selected check runs in one org.
- **Exit codes:** 0 passed, 1 failed, 2 config or build error, 3 incomplete.
- Runs send `User-Agent: vapi-gitops-check/<version>`, for the post-deploy PostHog measure.
- The README and AGENTS.md `npm run check` rows now describe live mode.
## Evidence of value
**The definition-of-done pair, run live** in the owner's test org on the TEST-141 parity squad (2 members, function tools, a handoff, 3 scenarios; built from `tests/fixtures/check-parity/`):
| Run | Command | Exit | Result |
|---|---|---|---|
| Innocuous (fixture as-is) | `npm run check -- core` | **0** | ✅ 3/3 passed — [run f68ba89b](https://dashboard.vapi.ai/simulations/run/f68ba89b-9680-470e-be71-2286b32a4e9c) |
| Degraded scheduler prompt ("always say there are no openings, never book") | `npm run check -- core` | **1** | ❌ 1/3 — [run 19711623](https://dashboard.vapi.ai/simulations/run/19711623-d26a-4570-82cd-e963a03bc1e2) |
The degraded run's report named each failing judge with expected vs extracted:
| Simulation | Evaluation | Comparator | Expected | Got |
|---|---|---|---|---|
| S3 alternative slot | offered-alternative | = | true | false |
| S3 alternative slot | booked-wednesday | = | true | false |
| S1 book cleaning | booking-confirmed | = | true | false |
| S1 book cleaning | only-real-slots | = | true | false |
S2 (the hours question, which never reaches the scheduler) correctly still passed.
**Nothing was created in the org.** Counts of assistants, tools, squads, structured outputs, personalities, scenarios, simulations, suites and credentials were identical before and after both runs: `{"/assistant":12,"/tool":8,"/squad":2,"/structured-output":8,"/eval/simulation/personality":10,"/eval/simulation/scenario":10,"/eval/simulation":10,"/eval/simulation/suite":1,"/credential":0}`. This stands in for `npm run cleanup`, which needs a gitops-managed org.
**Every tool call returned a mock.** Every `tool_call_result` in both runs' transcripts was matched to its call by `toolCallId`:
| Run | Function results equal to the scenario's mock | Built-in results (handoff, endCall) | Anything else |
|---|---|---|---|
| Innocuous | 6 | 4 | **0** |
| Degraded | 2 | 2 | **0** |
The run items also confirmed that personalities kept the dead server and `serverMessages: []`.
**A bug the live run caught, fixed in this PR:**
- **The trap:** run items echo the scenario under `metadata.scenario`, default error mocks included. Scanning the whole item would have reported "unmocked tool called" for every default mock, called or not.
- **The fix:** the scan now reads only `tool_call_result` messages in `metadata.call.messages`, mapped to tool names through `tool_calls`.
- **Checked on both runs' real items:** no notices; with one result swapped for a default-mock answer, it reported `S1 book cleaning: unmocked tool called: lookup_patient`.
- A regression test covers the echoed-defaults case.
**Tests:** `npm test` goes from 450 to 474 passing; `npm run build` is clean.
## Testing plan
- **`tests/check-run.test.ts`** (10 tests, stateful local stub):
- pass with link and User-Agent;
- two targets with one failing, naming the evaluation;
- all-skipped is incomplete;
- timeout cancels;
- abort cancels in-flight runs and starts none;
- the budget gate;
- 402 as billing, and no retry on a create 502;
- a build error sends nothing;
- mock notices (echoed defaults ignored);
- exactly 3 in flight with results in job order.
- **`tests/check-cmd.test.ts`** (11 tests): the live `--all` path against one stub serving both the simulations API and GitHub:
- pending, then success per target, then the aggregate with the run link, plus the job summary and JSON;
- a failing run exits 1 with a red aggregate;
- a named check never posts the aggregate;
- `--changed-since` with an unaffected change runs nothing and posts green;
- a dry run of an affected PR posts `error` and sends nothing;
- invalid config posts `error`.
The file clears `VAPI_*` and `GITHUB_*` first, so a developer shell with a key exported can't reach a real org.
- **`tests/check-select.test.ts`:** the affected-path table, and the merge-base case in a temp git repo (commits on the base after branching don't count).
- **`tests/check-status.test.ts`, `tests/check-report.test.ts`:** env parsing, POST shape and 140-character truncation, 403 only warning, state ordering, the exact markdown, and the JSON.
- **Not tested:**
- `--refresh-bindings` against a live org (it runs the same child pull promotion does);
- EU base URLs;
- canceling a `running` run live (cancel was verified only on a queued run in the parity experiment);
- GitHub's real statuses API (stubbed here; PR 8's dogfood repo covers it);
- a real customer-shaped squad, which needs the owner's permission first.
Stacked on #62.
Refs TEST-141
🤖 Generated with [Claude Code](https://claude.com/claude-code)16 files changed
Lines changed: 2035 additions & 158 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| |||
904 | 904 | | |
905 | 905 | | |
906 | 906 | | |
907 | | - | |
| 907 | + | |
| 908 | + | |
908 | 909 | | |
909 | 910 | | |
910 | 911 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | | - | |
| 107 | + | |
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
0 commit comments