Skip to content

Commit 2bd9f32

Browse files
test(promotion): pin the exact files promoting an assistant writes to the target (#68)
## Value **V.A.L.U.E. tier:** small — a tests-only baseline under the remaining promotion changes for TEST-141 ([TEST-141](https://linear.app/vapi/issue/TEST-141/gitops-run-simulation-suites-against-pr-changes-inline-as-ci-checks)). It changes no behaviour. It's small rather than micro because it guards a blast-radius path: promotion into production. - **Problem:** the code that moves assistants from a lower org into production is the riskiest path in gitops. The existing tests check pieces of it (canonicalization, bindings, deletions) but never the full set of files a promotion writes. So a refactor could change what lands in prod without any test failing. - **Who it affects:** teams that promote resources between orgs (dev → staging → prod). It's a baseline for future changes to promotion's plan/apply core. The stacked PRs don't touch that core: #65's commit behaviour is covered by its own `promote-cmd` and workflow-commit tests, and #66 only adds the `check` field. - **What changes:** new `tests/promotion-golden.test.ts`. It runs one staging → prod promotion through `promotionPlanBuild` and `promotionPlanApply`, then asserts the plan and every file left in the target org: - source UUIDs canonicalized to names (`toolIds`, a handoff destination); - referenced tools and assistants pulled in as dependencies, and an unreferenced staging-only tool left behind; - a credential bound by name, and a phone number bound to the target's; - a target file the source no longer has deleted; - an unrelated target file left untouched; - the markdown prompt body preserved; - the source org never written. A second test re-plans right after the apply and expects no changes. ## Evidence of value **Mutation check:** each deliberate break of `src/promotion.ts` turns the test red, and it's green again once the break is reverted. | Break | Golden test | |---|---| | Phone binding returns the source phone | ❌ fails | | Credential binding returns the source UUID | ❌ fails | | UUID → name canonicalization disabled | ❌ fails | | None (`main`) | ✅ passes | The expected output was read line by line against promotion's documented behaviour before it was pinned. **Tests:** `npm test` goes from 474 to 476 passing. ## Testing plan - `tests/promotion-golden.test.ts`: the golden promotion, and idempotence on re-plan. - **Not tested:** - the child `pull`/`apply` processes (this exercises the file-writing core, not the CLI); - a real two-org promotion (manual QA before #65 and #66 merge). Base of the promotion stack: #68 ← #65 ← #66. Refs TEST-141 ## After review Chris's mutation-tested suggestions, each verified to fail the golden when the code it guards is broken: - `.vapi-ignore`d prod files are left alone (an ignored, prod-only `assistants/hotline.yml`). - The whole repository is snapshotted, so a stray write to `.env.*` or a state file fails it. - Billing arrives only as a dependency of a dependency (intake → `handoff-to-billing` → billing). - A file promotion doesn't rewrite is copied byte for byte (it keeps a comment). - The prompt body has a heading, blank lines and a `---` separator. Not taken: the `omit` phone case (`promotion.test.ts` already covers it), the unused state entries, and the input/output formatting pass, which is a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents fbbf64c + e5ba491 commit 2bd9f32

1 file changed

Lines changed: 255 additions & 0 deletions

File tree

‎tests/promotion-golden.test.ts‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
import assert from "node:assert/strict";
2+
import {
3+
mkdir,
4+
mkdtemp,
5+
readdir,
6+
readFile,
7+
rm,
8+
writeFile,
9+
} from "node:fs/promises";
10+
import { tmpdir } from "node:os";
11+
import { dirname, join, relative } from "node:path";
12+
import test from "node:test";
13+
import type { PromotionBindings } from "../src/promotion.ts";
14+
import {
15+
promotionPlanApply,
16+
promotionPlanBuild,
17+
promotionStateParse,
18+
} from "../src/promotion.ts";
19+
20+
// Golden test for moving an assistant from a lower org into production: the
21+
// exact files promotion writes into the target org. Each piece is covered
22+
// elsewhere; this pins the whole written output, so a refactor that drifts
23+
// any of it (reference canonicalization, transitive dependency selection,
24+
// credential or phone bindings, deletions, .vapi-ignore'd and unrelated files
25+
// left alone, verbatim copies, markdown bodies, writes outside the target org)
26+
// fails here. Update the expected output only for an intended change.
27+
28+
const FILES: Record<string, string> = {
29+
"resources/staging/assistants/intake.md":
30+
"---\nname: Intake\nfirstMessage: Thanks for calling, how can I help?\nmodel:\n provider: openai\n model: gpt-4.1\n toolIds:\n - 11111111-1111-4111-8111-111111111111\n - handoff-to-billing\n tools:\n - type: endCall\nvoice:\n provider: 11labs\n voiceId: sarah\n credentialId: 22222222-2222-4222-8222-222222222222\nphoneNumberIds:\n - staging-phone\n---\n\n# Intake\n\nYou are the intake agent.\n\n---\n\nAsk for the caller's account number first.\n",
31+
"resources/staging/assistants/billing.yml":
32+
"# Owned by the billing team.\nname: Billing\nmodel:\n provider: openai\n model: gpt-4.1\n",
33+
"resources/staging/tools/lookup.yml":
34+
"type: function\nfunction:\n name: lookup_account\n parameters:\n type: object\n properties:\n accountNumber:\n type: string\nserver:\n url: https://crm.example.com/vapi\n credentialId: crm\n",
35+
"resources/staging/tools/handoff-to-billing.yml":
36+
"type: handoff\ndestinations:\n - type: assistant\n assistantId: 33333333-3333-4333-8333-333333333333\n description: Billing questions\n",
37+
"resources/staging/tools/staging-only-debug.yml":
38+
"type: function\nfunction:\n name: debug_dump\n",
39+
"resources/prod/assistants/intake.md":
40+
"---\nname: Intake\nmodel:\n provider: openai\n model: gpt-4o\n---\n\nOld prompt.\n",
41+
"resources/prod/assistants/retired.yml": "name: Retired\n",
42+
"resources/prod/squads/keep.yml": "name: Unrelated squad\n",
43+
// Kept out of gitops in prod: the patterns select it and staging has no
44+
// such file, but .vapi-ignore must stop promotion deleting it.
45+
"resources/prod/.vapi-ignore": "assistants/hotline\n",
46+
"resources/prod/assistants/hotline.yml": "name: Prod-only hotline\n",
47+
".env.staging": "VAPI_PHONE_NUMBER_MAIN=staging-phone\n",
48+
".env.prod": "VAPI_PHONE_NUMBER_MAIN=prod-phone\n",
49+
};
50+
51+
const SOURCE_STATE = {
52+
tools: {
53+
lookup: {
54+
uuid: "11111111-1111-4111-8111-111111111111",
55+
},
56+
},
57+
assistants: {
58+
billing: {
59+
uuid: "33333333-3333-4333-8333-333333333333",
60+
},
61+
retired: {
62+
uuid: "44444444-4444-4444-8444-444444444444",
63+
},
64+
},
65+
credentials: {
66+
eleven: {
67+
uuid: "22222222-2222-4222-8222-222222222222",
68+
},
69+
crm: {
70+
uuid: "55555555-5555-4555-8555-555555555555",
71+
},
72+
},
73+
};
74+
75+
const TARGET_STATE = {
76+
assistants: {
77+
intake: {
78+
uuid: "66666666-6666-4666-8666-666666666666",
79+
},
80+
retired: {
81+
uuid: "77777777-7777-4777-8777-777777777777",
82+
},
83+
},
84+
credentials: {
85+
eleven: {
86+
uuid: "prod-eleven",
87+
},
88+
crm: {
89+
uuid: "prod-crm",
90+
},
91+
},
92+
};
93+
94+
const BINDINGS: PromotionBindings = {
95+
credentials: {
96+
default: "bind",
97+
aliases: {},
98+
},
99+
phoneNumbers: {
100+
default: "omit",
101+
aliases: {
102+
main: "bind",
103+
},
104+
},
105+
};
106+
107+
const EXPECTED_TARGET: Record<string, string> = {
108+
// Copied byte for byte: a file promotion doesn't rewrite keeps its comments.
109+
"assistants/billing.yml":
110+
"# Owned by the billing team.\nname: Billing\nmodel:\n provider: openai\n model: gpt-4.1\n",
111+
"assistants/intake.md": [
112+
"---",
113+
"name: Intake",
114+
"firstMessage: Thanks for calling, how can I help?",
115+
"model:",
116+
" provider: openai",
117+
" model: gpt-4.1",
118+
" toolIds:",
119+
" - lookup",
120+
" - handoff-to-billing",
121+
" tools:",
122+
" - type: endCall",
123+
"voice:",
124+
" provider: 11labs",
125+
" voiceId: sarah",
126+
" credentialId: eleven",
127+
"phoneNumberIds:",
128+
" - prod-phone",
129+
"---",
130+
"",
131+
"# Intake",
132+
"",
133+
"You are the intake agent.",
134+
"",
135+
"---",
136+
"",
137+
"Ask for the caller's account number first.",
138+
"",
139+
].join("\n"),
140+
"squads/keep.yml": "name: Unrelated squad\n",
141+
".vapi-ignore": "assistants/hotline\n",
142+
"assistants/hotline.yml": "name: Prod-only hotline\n",
143+
"tools/handoff-to-billing.yml": [
144+
"type: handoff",
145+
"destinations:",
146+
" - type: assistant",
147+
" assistantId: billing",
148+
" description: Billing questions",
149+
"",
150+
].join("\n"),
151+
"tools/lookup.yml": [
152+
"type: function",
153+
"function:",
154+
" name: lookup_account",
155+
" parameters:",
156+
" type: object",
157+
" properties:",
158+
" accountNumber:",
159+
" type: string",
160+
"server:",
161+
" url: https://crm.example.com/vapi",
162+
" credentialId: crm",
163+
"",
164+
].join("\n"),
165+
};
166+
167+
async function fixtureWrite(): Promise<string> {
168+
const root = await mkdtemp(join(tmpdir(), "vapi-promotion-golden-"));
169+
for (const [path, content] of Object.entries(FILES)) {
170+
await mkdir(dirname(join(root, path)), { recursive: true });
171+
await writeFile(join(root, path), content);
172+
}
173+
return root;
174+
}
175+
176+
async function filesRead(dir: string): Promise<Record<string, string>> {
177+
const out: Record<string, string> = {};
178+
const walk = async (current: string): Promise<void> => {
179+
for (const entry of await readdir(current, { withFileTypes: true })) {
180+
const full = join(current, entry.name);
181+
if (entry.isDirectory()) await walk(full);
182+
else out[relative(dir, full)] = await readFile(full, "utf8");
183+
}
184+
};
185+
await walk(dir);
186+
return Object.fromEntries(
187+
Object.entries(out).sort(([a], [b]) => a.localeCompare(b)),
188+
);
189+
}
190+
191+
function planArgs(root: string) {
192+
return {
193+
rootDir: root,
194+
source: "staging",
195+
target: "prod",
196+
// Billing isn't named: it must arrive as a dependency of a dependency
197+
// (intake -> handoff-to-billing -> billing).
198+
patterns: ["assistants/intake", "assistants/retired", "assistants/hotline"],
199+
sourceState: promotionStateParse(JSON.stringify(SOURCE_STATE)),
200+
targetState: promotionStateParse(JSON.stringify(TARGET_STATE)),
201+
bindings: BINDINGS,
202+
};
203+
}
204+
205+
test("promoting an assistant writes exactly the expected target files", async () => {
206+
const root = await fixtureWrite();
207+
try {
208+
const plan = await promotionPlanBuild(planArgs(root));
209+
const changes = plan.changes.map((change) => [change.kind, change.path]);
210+
await promotionPlanApply(plan);
211+
assert.deepEqual(
212+
{
213+
changes,
214+
files: await filesRead(root),
215+
},
216+
{
217+
changes: [
218+
["create", "assistants/billing.yml"],
219+
["update", "assistants/intake.md"],
220+
["delete", "assistants/retired.yml"],
221+
["create", "tools/handoff-to-billing.yml"],
222+
["create", "tools/lookup.yml"],
223+
],
224+
// Only resources/prod changes; the source org, .env files and every
225+
// other file in the repo are left byte-for-byte as they were.
226+
files: {
227+
...Object.fromEntries(
228+
Object.entries(FILES).filter(
229+
([path]) => !path.startsWith("resources/prod/"),
230+
),
231+
),
232+
...Object.fromEntries(
233+
Object.entries(EXPECTED_TARGET).map(([path, content]) => [
234+
`resources/prod/${path}`,
235+
content,
236+
]),
237+
),
238+
},
239+
},
240+
);
241+
} finally {
242+
await rm(root, { recursive: true, force: true });
243+
}
244+
});
245+
246+
test("re-planning right after an apply finds nothing left to change", async () => {
247+
const root = await fixtureWrite();
248+
try {
249+
await promotionPlanApply(await promotionPlanBuild(planArgs(root)));
250+
const again = await promotionPlanBuild(planArgs(root));
251+
assert.deepEqual(again.changes, []);
252+
} finally {
253+
await rm(root, { recursive: true, force: true });
254+
}
255+
});

0 commit comments

Comments
 (0)