Skip to content

Commit a32bc5c

Browse files
fix(cleanup): never delete resources matched by .vapi-ignore
.vapi-ignore marks platform resources a repository must not manage, and pull never writes them to state. cleanup treats every platform resource missing from state as an orphan and didn't read .vapi-ignore, so a destructive cleanup deleted exactly the resources a team had excluded, such as another team's assistants in a shared org. Push already orphan-protects them. cleanup now loads the org's ignore patterns and keeps every orphan that matches, checking the IDs pull would give the resource (its name slug, with and without the UUID suffix). Kept resources are listed as retained in both dry and destructive runs. tests/cleanup-ignore.test.ts runs the real cleanup command against a stub API: before this change a destructive run deleted an ignored assistant and tool along with the true orphan; now only the orphan. Docs: AGENTS.md, the workflows guide and .vapi-ignore.example describe the fixed behaviour; improvements.md #36 is resolved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d58a518 commit a32bc5c

6 files changed

Lines changed: 257 additions & 28 deletions

File tree

‎AGENTS.md‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -229,13 +229,9 @@ The engine resolves IDs and credential names to each org's UUIDs on push.
229229

230230
`resources/<org>/.vapi-ignore` lists platform resources this repo must not
231231
manage, as gitignore-style patterns (see `resources/.vapi-ignore.example`).
232-
Matched resources are skipped on pull and push, and push never deletes them.
233-
A resource that references an ignored one is a validation error.
234-
235-
**`npm run cleanup` does not read `.vapi-ignore`.** Ignored resources are
236-
never in the state file, so cleanup lists them as orphans and a destructive
237-
run would delete them. Check its dry-run list against `.vapi-ignore` with the
238-
human before any `--force` run.
232+
Matched resources are skipped on pull and push, and neither push nor
233+
`npm run cleanup` deletes them. A resource that references an ignored one is a
234+
validation error.
239235

240236
---
241237

‎docs/guides/workflows.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -186,11 +186,10 @@ npm run cleanup -- <org>
186186
npm run cleanup -- <org> --force --confirm <org>
187187
```
188188

189-
**Check the dry-run list before a destructive run.** Cleanup treats every
190-
platform resource that isn't in `.vapi-state.<org>.json` as an orphan, and it
191-
does not read `.vapi-ignore`: resources you've excluded with `.vapi-ignore`
192-
are never in the state file, so they appear in the list and `--force` would
193-
delete them.
189+
Cleanup treats every platform resource that isn't in
190+
`.vapi-state.<org>.json` as an orphan, except those matched by
191+
`resources/<org>/.vapi-ignore`: it lists those as retained and never deletes
192+
them. Still read the dry-run list before a destructive run.
194193

195194
**When the list includes Vapi's built-in fixtures** (for example the stock
196195
simulation personalities, which can't be deleted — see

‎improvements.md‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ you which stack PR closes the row.**
8787
| 33 | `npm run sim` reported every run as passed | A failing suite exited 0 — false green | None | RESOLVED 2026-10-01 |
8888
| 34 | No pre-merge simulation signal; simulations only tested what was deployed | A PR that breaks an agent merges green | #33 | RESOLVED 2026-10-01 |
8989
| 35 | A failed promotion pushed nothing, not even state | git lost track of resources already on the platform | None | RESOLVED 2026-10-01 |
90-
| 36 | `cleanup` deletes resources excluded by `.vapi-ignore` | A destructive cleanup can delete resources another team owns | None | Open |
90+
| 36 | `cleanup` deletes resources excluded by `.vapi-ignore` | A destructive cleanup can delete resources another team owns | None | RESOLVED 2026-10-03 |
9191

9292
**Active backlog after cleanup:** `#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.
9393

@@ -1888,6 +1888,8 @@ None needed once the fix below lands.
18881888

18891889
## 36. `cleanup` deletes resources excluded by `.vapi-ignore`
18901890

1891+
**[RESOLVED 2026-10-03]**
1892+
18911893
**Discovered:** 2026-10-03, while reviewing the agent instructions for the public release.
18921894

18931895
### Problem
@@ -1914,19 +1916,20 @@ another team deletes that team's assistants, tools or squads.
19141916

19151917
### Current mitigation
19161918

1917-
The default run is a dry run, and the destructive run needs `--confirm <org>`.
1918-
The docs and `AGENTS.md` now tell people and agents to check the dry-run list
1919-
against `.vapi-ignore` first.
1919+
None needed once the fix below lands.
19201920

1921-
### Possible fix
1921+
### Possible fix (landed)
19221922

1923-
Load the org's ignore patterns in `cleanup.ts` and exclude matches from the
1924-
deletion list (printing them as retained, as push does), with a test that a
1925-
destructive cleanup keeps an ignored resource.
1923+
`src/cleanup.ts` loads the org's ignore patterns and keeps every orphan that
1924+
matches one, checking the IDs pull would give the resource (its name slug,
1925+
with and without the UUID suffix). Kept resources are listed as retained.
1926+
`tests/cleanup-ignore.test.ts` runs a destructive cleanup against a stub API:
1927+
before the fix it deleted an ignored assistant and tool along with the true
1928+
orphan; after it, only the orphan.
19261929

19271930
### Status
19281931

1929-
Open.
1932+
**RESOLVED 2026-10-03.**
19301933

19311934
---
19321935

‎resources/.vapi-ignore.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
# id matches `.vapi-ignore`.
1111
# - `--force` on push bypasses the load-filter (so a deliberate override
1212
# can flow through) BUT the orphan-protect still applies.
13+
# - `npm run cleanup -- <org>` never deletes a matched resource, even with
14+
# `--force --confirm <org>`; it lists it as retained.
1315
# - A resource that references an ignored resource (e.g. a squad
1416
# pointing at `assistants/foo` while `assistants/foo` is ignored) is
1517
# a validation ERROR — `--strict` push aborts before any API call.

‎src/cleanup.ts‎

Lines changed: 86 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,16 @@
11
import { resolve } from "path";
22
import { fileURLToPath } from "url";
3-
import { VAPI_BASE_URL, VAPI_ENV, VAPI_TOKEN } from "./config.ts";
3+
import {
4+
loadIgnorePatterns,
5+
matchesIgnore,
6+
VAPI_BASE_URL,
7+
VAPI_ENV,
8+
VAPI_TOKEN,
9+
} from "./config.ts";
10+
import { FOLDER_MAP } from "./resource-parse.ts";
11+
import { slugify } from "./slug-utils.ts";
412
import { loadState } from "./state.ts";
13+
import type { ResourceType } from "./types.ts";
514

615
// ─────────────────────────────────────────────────────────────────────────────
716
// Dangerous Sync - Delete everything NOT in state file
@@ -68,6 +77,29 @@ async function vapiDelete(endpoint: string): Promise<void> {
6877
interface VapiResource {
6978
id: string;
7079
name?: string;
80+
function?: { name?: string };
81+
}
82+
83+
// The .vapi-ignore pattern a platform resource matches, or null. Checks the
84+
// ids pull would give it — its name slug with the UUID suffix (what pull
85+
// writes for an untracked resource) and without it — so a pattern written
86+
// against either form protects the resource.
87+
function ignoredBy(
88+
folder: string,
89+
resource: VapiResource,
90+
patterns: string[],
91+
): string | null {
92+
if (patterns.length === 0) return null;
93+
const name = resource.name ?? resource.function?.name;
94+
const shortId = resource.id.slice(0, 8);
95+
const ids = name
96+
? [`${slugify(name)}-${shortId}`, slugify(name)]
97+
: [`resource-${shortId}`];
98+
for (const id of ids) {
99+
const matched = matchesIgnore(folder, id, patterns);
100+
if (matched) return matched;
101+
}
102+
return null;
71103
}
72104

73105
function readConfirmToken(argv: string[]): string | undefined {
@@ -149,43 +181,75 @@ async function main(): Promise<void> {
149181
}[] = [];
150182

151183
// Fetch and compare each resource type
152-
const resourceTypes = [
184+
const resourceTypes: Array<{
185+
type: ResourceType;
186+
name: string;
187+
endpoint: string;
188+
deleteEndpoint: string;
189+
}> = [
153190
{
191+
type: "assistants",
154192
name: "assistants",
155193
endpoint: "/assistant",
156194
deleteEndpoint: "/assistant",
157195
},
158-
{ name: "tools", endpoint: "/tool", deleteEndpoint: "/tool" },
159196
{
197+
type: "tools",
198+
name: "tools",
199+
endpoint: "/tool",
200+
deleteEndpoint: "/tool",
201+
},
202+
{
203+
type: "structuredOutputs",
160204
name: "structured outputs",
161205
endpoint: "/structured-output",
162206
deleteEndpoint: "/structured-output",
163207
},
164-
{ name: "squads", endpoint: "/squad", deleteEndpoint: "/squad" },
165208
{
209+
type: "squads",
210+
name: "squads",
211+
endpoint: "/squad",
212+
deleteEndpoint: "/squad",
213+
},
214+
{
215+
type: "personalities",
166216
name: "personalities",
167217
endpoint: "/eval/simulation/personality",
168218
deleteEndpoint: "/eval/simulation/personality",
169219
},
170220
{
221+
type: "scenarios",
171222
name: "scenarios",
172223
endpoint: "/eval/simulation/scenario",
173224
deleteEndpoint: "/eval/simulation/scenario",
174225
},
175226
{
227+
type: "simulations",
176228
name: "simulations",
177229
endpoint: "/eval/simulation",
178230
deleteEndpoint: "/eval/simulation",
179231
},
180232
{
233+
type: "simulationSuites",
181234
name: "simulation suites",
182235
endpoint: "/eval/simulation/suite",
183236
deleteEndpoint: "/eval/simulation/suite",
184237
},
185-
{ name: "evals", endpoint: "/eval", deleteEndpoint: "/eval" },
238+
{
239+
type: "evals",
240+
name: "evals",
241+
endpoint: "/eval",
242+
deleteEndpoint: "/eval",
243+
},
186244
];
187245

188-
for (const { name, endpoint, deleteEndpoint } of resourceTypes) {
246+
// Resources this repo must not manage (.vapi-ignore) are never written to
247+
// state, so every one of them would look like an orphan here. Keep them,
248+
// as push's orphan-protection does.
249+
const ignorePatterns = loadIgnorePatterns();
250+
let retained = 0;
251+
252+
for (const { type, name, endpoint, deleteEndpoint } of resourceTypes) {
189253
console.log(`📥 Fetching ${name}...`);
190254
try {
191255
// Enable debug for structured outputs to see response format
@@ -202,7 +266,16 @@ async function main(): Promise<void> {
202266
continue;
203267
}
204268

205-
const orphans = resources.filter((r) => !stateIds.has(r.id));
269+
const orphans: VapiResource[] = [];
270+
for (const r of resources.filter((r) => !stateIds.has(r.id))) {
271+
const matched = ignoredBy(FOLDER_MAP[type], r, ignorePatterns);
272+
if (matched) {
273+
console.log(
274+
` 🚫 ${r.name || r.function?.name || r.id} retained (matched .vapi-ignore: ${matched})`,
275+
);
276+
retained++;
277+
} else orphans.push(r);
278+
}
206279

207280
if (orphans.length > 0) {
208281
console.log(
@@ -228,6 +301,12 @@ async function main(): Promise<void> {
228301
"\n═══════════════════════════════════════════════════════════════",
229302
);
230303

304+
if (retained > 0) {
305+
console.log(
306+
`\n🚫 ${retained} resource(s) not in state were kept because they match .vapi-ignore`,
307+
);
308+
}
309+
231310
if (toDelete.length === 0) {
232311
console.log("✅ Nothing to delete - all resources match state file\n");
233312
return;

‎tests/cleanup-ignore.test.ts‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
import assert from "node:assert/strict";
2+
import { spawn } from "node:child_process";
3+
import {
4+
cpSync,
5+
mkdirSync,
6+
mkdtempSync,
7+
rmSync,
8+
symlinkSync,
9+
writeFileSync,
10+
} from "node:fs";
11+
import { createServer } from "node:http";
12+
import type { AddressInfo } from "node:net";
13+
import { tmpdir } from "node:os";
14+
import { join } from "node:path";
15+
import test from "node:test";
16+
import { fileURLToPath } from "node:url";
17+
18+
// `npm run cleanup` deletes platform resources that aren't in the state
19+
// file. Resources excluded by .vapi-ignore are never written to state, so
20+
// cleanup must keep them, as push's orphan-protection does.
21+
22+
const REPO = fileURLToPath(new URL("..", import.meta.url));
23+
const ORG = "test-cleanup-org";
24+
const TRACKED = "11111111-1111-4111-8111-111111111111";
25+
const IGNORED_ASSISTANT = "22222222-2222-4222-8222-222222222222";
26+
const IGNORED_TOOL = "33333333-3333-4333-8333-333333333333";
27+
const ORPHAN = "44444444-4444-4444-8444-444444444444";
28+
29+
const PLATFORM: Record<string, unknown[]> = {
30+
"/assistant": [
31+
{ id: TRACKED, name: "Front Desk" },
32+
{ id: IGNORED_ASSISTANT, name: "Legacy Bot" },
33+
{ id: ORPHAN, name: "Old Experiment" },
34+
],
35+
"/tool": [
36+
{ id: IGNORED_TOOL, type: "function", function: { name: "legacy_lookup" } },
37+
],
38+
};
39+
40+
async function cleanupRun(
41+
args: string[],
42+
): Promise<{ code: number | null; output: string; deletes: string[] }> {
43+
const deletes: string[] = [];
44+
const server = createServer((req, res) => {
45+
res.setHeader("content-type", "application/json");
46+
if (req.method === "DELETE") {
47+
deletes.push(req.url ?? "");
48+
res.end("{}");
49+
return;
50+
}
51+
const path = (req.url ?? "").split("?")[0]!;
52+
res.end(JSON.stringify(PLATFORM[path] ?? []));
53+
});
54+
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
55+
const baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
56+
const dir = mkdtempSync(join(tmpdir(), "vapi-cleanup-ignore-"));
57+
try {
58+
cpSync(join(REPO, "src"), join(dir, "src"), { recursive: true });
59+
cpSync(join(REPO, "package.json"), join(dir, "package.json"));
60+
symlinkSync(join(REPO, "node_modules"), join(dir, "node_modules"), "dir");
61+
writeFileSync(join(dir, `.env.${ORG}`), "VAPI_TOKEN=fake-token-not-used\n");
62+
writeFileSync(
63+
join(dir, `.vapi-state.${ORG}.json`),
64+
JSON.stringify({ assistants: { "front-desk": { uuid: TRACKED } } }),
65+
);
66+
mkdirSync(join(dir, "resources", ORG), { recursive: true });
67+
writeFileSync(
68+
join(dir, "resources", ORG, ".vapi-ignore"),
69+
"# owned by another team\nassistants/legacy-*\ntools/legacy-*\n",
70+
);
71+
const child = spawn(
72+
process.execPath,
73+
["--import", "tsx", "src/cleanup.ts", ORG, ...args],
74+
{
75+
cwd: dir,
76+
env: {
77+
...process.env,
78+
VAPI_BASE_URL: baseUrl,
79+
VAPI_TOKEN: "fake-token-not-used",
80+
},
81+
},
82+
);
83+
let output = "";
84+
child.stdout.on("data", (chunk) => (output += chunk));
85+
child.stderr.on("data", (chunk) => (output += chunk));
86+
const code = await new Promise<number | null>((resolve) =>
87+
child.on("close", resolve),
88+
);
89+
return { code, output, deletes };
90+
} finally {
91+
rmSync(dir, { recursive: true, force: true });
92+
await new Promise<void>((resolve) => server.close(() => resolve()));
93+
}
94+
}
95+
96+
test(
97+
"a destructive cleanup deletes only true orphans and keeps .vapi-ignore matches",
98+
{ timeout: 60_000 },
99+
async () => {
100+
const result = await cleanupRun(["--force", "--confirm", ORG]);
101+
assert.deepEqual(
102+
{
103+
code: result.code,
104+
deletes: result.deletes,
105+
retainedAssistant: result.output.includes(
106+
"Legacy Bot retained (matched .vapi-ignore: assistants/legacy-*)",
107+
),
108+
retainedTool: result.output.includes(
109+
"legacy_lookup retained (matched .vapi-ignore: tools/legacy-*)",
110+
),
111+
},
112+
{
113+
code: 0,
114+
deletes: [`/assistant/${ORPHAN}`],
115+
retainedAssistant: true,
116+
retainedTool: true,
117+
},
118+
);
119+
},
120+
);
121+
122+
test(
123+
"a dry run lists ignored resources as kept, not as orphans",
124+
{ timeout: 60_000 },
125+
async () => {
126+
const result = await cleanupRun([]);
127+
assert.deepEqual(
128+
{
129+
code: result.code,
130+
deletes: result.deletes,
131+
orphanListed: result.output.includes(
132+
`assistants: Old Experiment (${ORPHAN})`,
133+
),
134+
ignoredListed:
135+
result.output.includes(IGNORED_ASSISTANT) &&
136+
result.output.includes("🗑️ assistants: Legacy Bot"),
137+
keptSummary: result.output.includes(
138+
"2 resource(s) not in state were kept because they match .vapi-ignore",
139+
),
140+
},
141+
{
142+
code: 0,
143+
deletes: [],
144+
orphanListed: true,
145+
ignoredListed: false,
146+
keptSummary: true,
147+
},
148+
);
149+
},
150+
);

0 commit comments

Comments
 (0)