Skip to content

Commit d92189b

Browse files
feat(check): build inline simulation-run payloads and add a dry-run CLI (#61)
## Value **V.A.L.U.E. tier:** project — PR 5 of 10 for inline simulation PR checks ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)); this PR adds an offline dry run, and nothing is sent. - **Problem:** to test a PR's changes without deploying them, the check has to turn the branch's files into one inline `POST /eval/simulation/run` body: the target with every tool, handoff and structured output, plus every scenario, judge and personality. It has to assemble that body the way push and the runtime would, or the check tests something other than what ships. - **Who it affects:** gitops users, who can now see exactly what a check would send (`npm run check -- core --dry-run --print-payload`) before any minutes are spent. Every later PR (mock policy, live runs, workflow, promotion gate) builds on this payload. - **What changes:** - **`src/check-payload.ts`** (with `check-payload-assistant.ts` and `check-payload-refs.ts`) builds the body from `orgResourcesRead` and the org states. It's pure, makes no network calls, and collects every problem so one dry run reports them all. - **Tools:** - runtime tool order: `model.tools`, then `toolIds`, then `toolRefs`. This is what `callAssistantsGet` does, and the parity run's likely `endCall` difference came from getting it wrong; - `##` comments stripped, and UUID references resolved through state; - a `toolRefs` pin wins over a duplicate `toolIds` entry, with a warning that the version pin is ignored; - `knowledgeBase` tools are kept by run-org UUID, since the API refuses them inline. - **Squads:** members are inlined (from a file, by UUID, or inline), and handoffs to members switch from ID to member name. - **Assistants:** - hook `do[].toolId` becomes an inline tool; - `artifactPlan.structuredOutputIds`, plus structured outputs that link the assistant through their own `assistant_ids`, are inlined; - linkage and server fields are stripped. - **Simulations:** - suites and simulations are deduplicated into entries with unique names of at most 80 characters; - judges' `structuredOutputId`s are inlined; - local personalities are inlined, and stock ones pass by ID. - **Credentials** are bound by name to the run org's UUIDs using promotion's binding policy (`bind` / `omit`). - **Fails the build, naming the field:** - a missing file; - a handoff leaving the target, or an assistant target that hands off; - legacy `assistantDestinations` by ID; - members without a name, or with the same name; - two inlined tools with the same type and name, which stored would keep but inline would silently drop; - tools referenced by ID inside overrides (under strict mocks); - over chat: audio judges, scenario hooks, or no required text judge; - any reference still a name after the build (the backstop for #31's silent drops); - a body over 4.5 MB. - **Warns** when text mentions `handoff_to_…` but a handoff is auto-named, because generated names differ inline and stored. - **`src/check-cmd.ts`:** `npm run check -- <check>|--all --dry-run [--print-payload [dir]]`. - Offline: no API key needed, and it honours `VAPI_GITOPS_ROOT`. - Exits 0 when every payload builds, and 2 on a usage, config or build error. - Without `--dry-run` it exits 2; live runs land in PR 7. - `package.json` gets a `check` script, and the README and AGENTS.md command tables get `npm run check` rows. - New fixture `tests/fixtures/check-parity/`: the TEST-141 parity squad written as gitops files. - **Deliberately not here:** the fail-closed mock policy (dead servers, default error mocks, the tool-type allowlist), which is the next PR. Until then the payload carries tools as written, and there's no live path. ## Evidence of value **The builder reproduces the payload that scored 15/15 in the parity run.** - **How:** the dental squad from the 2026-10-01 parity experiment was written as gitops files (`.md` assistants with `toolIds`, a handoff tool by `assistantId`, judges by `structuredOutputId`). `npm run check -- core --dry-run --print-payload` was run on it, and the output was diffed against the inline body the experiment sent, rebuilt from `parity.mjs`. - **Result:** the only differences are these. | Path | Experiment sent | Built from files | Why | |---|---|---|---| | `members[*].assistant.model.tools` order | `[lookup_patient, handoff, endCall]`, `[check_availability, endCall]` | `[endCall, lookup_patient, handoff]`, `[endCall, check_availability]` | **Intended:** the runtime order of the stored arm (`model.tools` then `toolIds`). Same tools, byte-for-byte, order aside | | `squad.name`, `personality.name` | `inline`, `caller` | `Bright Smile Dental`, `Dental caller` | Fixture names | | `iterations`, `transport` | 5, (API default) | 1, `vapi.webchat` | `vapi-checks.yml` defaults | - Every prompt, judge, tool mock, tool definition and handoff destination is identical. The handoff by `assistantId: scheduler` came out as `assistantName: "Scheduler"`, exactly what the experiment sent. **Tests:** `npm test` goes from 404 to 430 passing (26 new), and `npm run build` is clean. ## Testing plan - **`tests/check-payload.test.ts`** (21 tests, temp-dir fixtures): - the parity fixture end to end: tool order, member-name handoffs, `.md` prompt, inline judges, entries, transport; - `.md` body as the only system message; - missing `toolIds`, and `toolIds` by UUID with server fields stripped; - the `toolRefs` pin and warning; - `knowledgeBase` kept by UUID, and failing with no UUID; - duplicate tool names; - hook tools; - structured outputs from `artifactPlan` and from `assistant_ids` (by slug and by UUID); - an assistant target that hands off; - squad members by file, UUID and inline; - every squad problem reported at once; - strict vs off overrides; - credentials bound, omitted and missing; - a leftover reference, with free-form `parameters` ignored; - the chat rules, and voice allowing them; - stock and local personalities, name truncation and dedupe; - missing suites, simulations, scenarios, personalities and targets; - the auto-handoff warning; - the 4.5 MB limit. - **`tests/check-cmd.test.ts`** (5 tests): - a dry run with `--print-payload`; - `--all` with one broken check exiting 2; - no config exiting 0; - usage, selection, config and live-mode errors exiting 2; - a missing state file as a warning. - **Not tested:** - **Any live run:** nothing is sent until PR 7, so API acceptance of a built body is shown only for the parity fixture (the experiment's 201). - **Real customer squads:** override merge semantics and integration tool shapes are untested. - Phone-number bindings beyond the unit-tested promotion helper. - EU base URLs. Stacked on #60. Refs TEST-141 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents 8da229f + 680a03d commit d92189b

31 files changed

Lines changed: 2132 additions & 0 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ If setup reports the org is "already set up locally", do not delete anything to
9090
| Push with new resources | `npm run push -- <org> --allow-new-files` — bypass orphan-YAML gate. **AI agents**: do NOT auto-pass this flag; confirm with the human first (see push section below) |
9191
| Test a call | `npm run call -- <org> -a <assistant-name>` or `-s <squad-name>` |
9292
| Run a simulation suite | `npm run sim -- <org> --suite <name> --target <assistant-name>` |
93+
| Build PR-check payloads offline | `npm run check -- <check> --dry-run` (or `--all`); `--print-payload` writes the JSON |
9394
| Migrate a legacy state file | `npm run migrate` — one-shot, all orgs; required once after upgrading to the hash-store engine |
9495

9596
---
@@ -903,6 +904,7 @@ npm run validate -- <org> # Lint resources locally (fai
903904
npm run audit -- <org> # Read-only drift detector: orphan YAML, state ghosts, content-identical clusters, sibling base-slugs, dashboard orphans, inline model.tools. Exit 1 on findings.
904905
npm run audit -- <org> --type assistants # Scope audit to a single resource type
905906
npm run sim -- <org> --suite <name> --target <name> # Run a simulation suite against an assistant/squad (exit 0 pass, 1 fail, 3 incomplete; --timeout <min>)
907+
npm run check -- <check> --dry-run # Build a vapi-checks.yml check's inline run payload offline (exit 0 built, 2 config/build error; --print-payload [dir])
906908
npm run rollback -- <org> --to <ISO-timestamp> # Re-apply a snapshot taken before a push
907909
npm run rollback -- <org> --list # List available snapshots
908910

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ Every command works in two modes:
104104
| `npm run rollback` | — | `npm run rollback -- <org> --list` or `--to <ISO>` | Restore from a snapshot in `.vapi-state.<org>.snapshots/` (one is written before every push/apply). |
105105
| `npm run call` | ✅ | `npm run call -- <org> -a <name>` or `-s <squad>` | Start an interactive WebSocket call against an assistant or squad. |
106106
| `npm run sim` | — | `npm run sim -- <org> --suite <name> --target <name> [--timeout <min>]` | Run a simulation suite (or specific simulations) against a deployed assistant/squad. Prints the run link; exits 0 passed, 1 failed, 3 incomplete (timeout, Ctrl-C, missing results). |
107+
| `npm run check` | — | `npm run check -- <check>\|--all --dry-run [--print-payload [dir]]` | Build each `vapi-checks.yml` check target's inline simulation-run payload from the files on disk, with its tools, handoffs, judges and personalities. Offline: no API key, nothing sent. `--print-payload` writes the JSON (default `tmp/check-payloads/`). Exits 0 when every payload builds, 2 on a config or build error. |
107108
| `npm run migrate` | — | `npm run migrate` | One-time, all orgs at once: slim legacy state files to pure `name → uuid` and seed the per-developer `.vapi-state-hash/` baseline store from the old hashes. Required once after upgrading to the hash-store engine — `pull`/`push`/`apply` refuse legacy-shaped state until it runs. Idempotent. |
108109
| `npm run build` | — | — | Type-check the codebase (`tsc --noEmit`). |
109110
| `npm test` | — | — | Run regression tests (`node:test`). |

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
"validate": "tsx src/validate-cmd.ts",
2020
"audit": "tsx src/audit-cmd.ts",
2121
"sim": "tsx src/sim-cmd.ts",
22+
"check": "tsx src/check-cmd.ts",
2223
"rollback": "tsx src/rollback-cmd.ts",
2324
"promote": "tsx src/promote-cmd.ts",
2425
"build": "tsc --noEmit",

‎src/check-cmd.ts‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
// CLI entry: `npm run check -- <check>|--all --dry-run [--print-payload [dir]]`
2+
//
3+
// Builds each check target's inline simulation-run payload from the files
4+
// on disk (see check-payload.ts) and reports what would run. Config-free:
5+
// it needs no API key and makes no network calls in --dry-run.
6+
//
7+
// Exit codes: 0 every payload built, 2 usage, config or build error.
8+
9+
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
10+
import { join, relative, resolve } from "node:path";
11+
import { fileURLToPath } from "node:url";
12+
import type { CheckDefinition } from "./check-config.ts";
13+
import { CHECKS_CONFIG_FILE, checksConfigLoad } from "./check-config.ts";
14+
import type { CheckPayloadResult } from "./check-payload.ts";
15+
import { checkPayloadBuild } from "./check-payload.ts";
16+
import { promotionBindingsResolve, promotionStateParse } from "./promotion.ts";
17+
import { orgResourcesRead } from "./resource-parse.ts";
18+
import type { StateFile } from "./types.ts";
19+
20+
const USAGE = [
21+
"Usage:",
22+
" npm run check -- <check> --dry-run [--print-payload [dir]]",
23+
" npm run check -- --all --dry-run [--print-payload [dir]]",
24+
"",
25+
"Options:",
26+
` <check> A check name from ${CHECKS_CONFIG_FILE}`,
27+
" --all Every check",
28+
" --dry-run Build the payloads offline; nothing is sent",
29+
" --print-payload [dir] Write each payload as JSON (default: tmp/check-payloads)",
30+
"",
31+
"Exit codes: 0 payloads built, 2 usage, config or build error",
32+
].join("\n");
33+
34+
const DEFAULT_PAYLOAD_DIR = "tmp/check-payloads";
35+
36+
interface CheckArgs {
37+
check?: string;
38+
all: boolean;
39+
dryRun: boolean;
40+
payloadDir?: string;
41+
help: boolean;
42+
}
43+
44+
class UsageError extends Error {}
45+
46+
function argsParse(args: string[]): CheckArgs {
47+
const parsed: CheckArgs = { all: false, dryRun: false, help: false };
48+
for (let index = 0; index < args.length; index++) {
49+
const arg = args[index]!;
50+
if (arg === "--help" || arg === "-h") parsed.help = true;
51+
else if (arg === "--all") parsed.all = true;
52+
else if (arg === "--dry-run") parsed.dryRun = true;
53+
else if (arg === "--print-payload") {
54+
const next = args[index + 1];
55+
parsed.payloadDir =
56+
next && !next.startsWith("--") ? args[++index] : DEFAULT_PAYLOAD_DIR;
57+
} else if (arg.startsWith("-"))
58+
throw new UsageError(`Unknown option: ${arg}`);
59+
else if (parsed.check) throw new UsageError(`Unexpected argument: ${arg}`);
60+
else parsed.check = arg;
61+
}
62+
if (parsed.help) return parsed;
63+
if (parsed.all === Boolean(parsed.check))
64+
throw new UsageError("Name one check, or pass --all");
65+
return parsed;
66+
}
67+
68+
function emptyState(): StateFile {
69+
return promotionStateParse("{}");
70+
}
71+
72+
function stateRead(root: string, org: string, warnings: string[]): StateFile {
73+
const path = join(root, `.vapi-state.${org}.json`);
74+
if (!existsSync(path)) {
75+
warnings.push(
76+
`no .vapi-state.${org}.json: references by UUID and credential bindings can't resolve`,
77+
);
78+
return emptyState();
79+
}
80+
return promotionStateParse(readFileSync(path, "utf8"));
81+
}
82+
83+
function payloadFileName(check: string, target: string): string {
84+
return `${check}--${target.replace(/\//g, "--")}.json`;
85+
}
86+
87+
function resultPrint(
88+
label: string,
89+
result: CheckPayloadResult,
90+
check: CheckDefinition,
91+
): void {
92+
console.log(`\n${label}`);
93+
if (result.body) {
94+
const count = result.body.simulations.length;
95+
console.log(
96+
` ✅ ${count} simulation${count === 1 ? "" : "s"} × ${check.iterations} iteration${check.iterations === 1 ? "" : "s"} over ${check.transport}, ${(result.bytes / 1024).toFixed(1)} KB`,
97+
);
98+
} else {
99+
console.log(
100+
` ❌ ${result.errors.length} problem${result.errors.length === 1 ? "" : "s"}`,
101+
);
102+
}
103+
for (const warning of result.warnings) console.log(` ⚠️ ${warning}`);
104+
for (const error of result.errors) console.log(` ❌ ${error}`);
105+
}
106+
107+
async function checkDryRun(
108+
root: string,
109+
check: CheckDefinition,
110+
payloadDir: string | undefined,
111+
): Promise<boolean> {
112+
if (!existsSync(join(root, "resources", check.org))) {
113+
console.log(`\n${check.name}\n ❌ resources/${check.org}/ does not exist`);
114+
return false;
115+
}
116+
const warnings: string[] = [];
117+
const resources = await orgResourcesRead(root, check.org);
118+
const sourceState = stateRead(root, check.org, warnings);
119+
const runState =
120+
check.runOrg === check.org
121+
? sourceState
122+
: stateRead(root, check.runOrg, warnings);
123+
const bindingsResolved = await promotionBindingsResolve(
124+
root,
125+
check.org,
126+
check.runOrg,
127+
sourceState,
128+
);
129+
let ok = true;
130+
for (const target of check.targets) {
131+
const targetLabel = `${target.type}/${target.id}`;
132+
const result = checkPayloadBuild({
133+
check,
134+
target,
135+
resources,
136+
sourceState,
137+
runState,
138+
bindingsResolved,
139+
});
140+
result.warnings.unshift(...warnings);
141+
resultPrint(`${check.name} / ${targetLabel}`, result, check);
142+
if (!result.body) {
143+
ok = false;
144+
continue;
145+
}
146+
if (payloadDir) {
147+
const dir = resolve(root, payloadDir);
148+
mkdirSync(dir, { recursive: true });
149+
const file = join(dir, payloadFileName(check.name, targetLabel));
150+
writeFileSync(file, `${JSON.stringify(result.body, null, 2)}\n`);
151+
console.log(` 📝 ${relative(root, file)}`);
152+
}
153+
}
154+
return ok;
155+
}
156+
157+
export async function checkCommandRun(
158+
args = process.argv.slice(2),
159+
root = resolve(
160+
process.env.VAPI_GITOPS_ROOT ??
161+
fileURLToPath(new URL("..", import.meta.url)),
162+
),
163+
): Promise<number> {
164+
let parsed: CheckArgs;
165+
try {
166+
parsed = argsParse(args);
167+
} catch (error) {
168+
if (!(error instanceof UsageError)) throw error;
169+
console.error(`❌ ${error.message}\n\n${USAGE}`);
170+
return 2;
171+
}
172+
if (parsed.help) {
173+
console.log(USAGE);
174+
return 0;
175+
}
176+
let config;
177+
try {
178+
config = checksConfigLoad(root);
179+
} catch (error) {
180+
console.error(`❌ ${CHECKS_CONFIG_FILE}: ${(error as Error).message}`);
181+
return 2;
182+
}
183+
if (!config) {
184+
console.log(
185+
`No ${CHECKS_CONFIG_FILE} at the repository root; nothing to check.`,
186+
);
187+
return 0;
188+
}
189+
const names = parsed.all ? Object.keys(config.checks) : [parsed.check!];
190+
const missing = names.filter((name) => !config.checks[name]);
191+
if (missing.length > 0) {
192+
console.error(
193+
`❌ No check named ${missing.join(", ")} in ${CHECKS_CONFIG_FILE} (checks: ${Object.keys(config.checks).join(", ")})`,
194+
);
195+
return 2;
196+
}
197+
if (!parsed.dryRun) {
198+
console.error(
199+
"❌ Live runs aren't available yet: pass --dry-run to build the payloads offline.",
200+
);
201+
return 2;
202+
}
203+
let ok = true;
204+
for (const name of names) {
205+
if (!(await checkDryRun(root, config.checks[name]!, parsed.payloadDir)))
206+
ok = false;
207+
}
208+
console.log(
209+
ok ? "\n✅ Every payload built." : "\n❌ Some payloads could not be built.",
210+
);
211+
return ok ? 0 : 2;
212+
}
213+
214+
const isMainModule =
215+
resolve(process.argv[1] ?? "") === resolve(fileURLToPath(import.meta.url));
216+
if (isMainModule) {
217+
checkCommandRun().then(
218+
(code) => process.exit(code),
219+
(error: unknown) => {
220+
console.error(
221+
`❌ Check failed: ${error instanceof Error ? error.message : String(error)}`,
222+
);
223+
process.exit(2);
224+
},
225+
);
226+
}

0 commit comments

Comments
 (0)