Skip to content

Commit 7d143c8

Browse files
fix(cleanup): never delete resources matched by .vapi-ignore (#73)
## Value **V.A.L.U.E. tier:** small — a behavior change in a destructive command (`npm run cleanup`). Not micro: it changes what the command deletes. - **Problem:** `npm run cleanup -- <org> --force --confirm <org>` deletes every platform resource that isn't in `.vapi-state.<org>.json`. That included resources matched by `resources/<org>/.vapi-ignore`, which is how users tell this repo "don't manage these". `push` already respected the ignore file; cleanup didn't. So the file you write to protect a resource was the one way to get it deleted (`improvements.md` #36). - **Who it affects:** anyone who runs cleanup in an org that also holds resources managed outside this repo (dashboard-owned assistants, legacy tools, another team's squad). - **What changes:** - Cleanup loads `.vapi-ignore` and checks each orphan against it, using the file IDs `pull` would give it (`<slug>-<uuid8>`, and the bare slug), so a pattern written against your files matches the platform resource too. - Matched resources are listed as `🚫 <name> retained (matched .vapi-ignore: <pattern>)` and never deleted, in dry runs and destructive runs. The summary counts them. - `AGENTS.md`, the workflows guide and `.vapi-ignore.example` say that neither push nor cleanup touches ignored resources. `improvements.md` #36 is marked resolved. ## Evidence of value New `tests/cleanup-ignore.test.ts` runs the real `src/cleanup.ts` against a stub API holding a tracked assistant, an ignored assistant (`assistants/legacy-*`), an ignored tool (`tools/legacy-*`) and one genuine orphan. | Run | Before this PR | After | | --- | --- | --- | | Destructive cleanup: DELETE requests | 3 — the orphan **plus the ignored assistant and tool** | 1 — the orphan only | | Dry run output | Lists the ignored resources as "would delete" | Lists them as retained, naming the matching pattern | The test was run against the old `cleanup.ts` first and failed on the extra deletes, then passed with the fix. ## Testing plan - `npm test` and `npx tsc --noEmit` pass. - **Not tested:** a live cleanup against a real org. The stub mirrors the list and delete endpoints cleanup calls; no live run was needed to show the bug or the fix. Refs TEST-141 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents 66e1f33 + 232bfb8 commit 7d143c8

6 files changed

Lines changed: 264 additions & 28 deletions

File tree

‎AGENTS.md‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -229,13 +229,9 @@ The engine resolves IDs and credential names to each org's UUIDs on push.
229229

230230
`resources/<org>/.vapi-ignore` lists platform resources this repo must not
231231
manage, as gitignore-style patterns (see `resources/.vapi-ignore.example`).
232-
Matched resources are skipped on pull and push, and push never deletes them.
233-
A resource that references an ignored one is a validation error.
234-
235-
**`npm run cleanup` does not read `.vapi-ignore`.** Ignored resources are
236-
never in the state file, so cleanup lists them as orphans and a destructive
237-
run would delete them. Check its dry-run list against `.vapi-ignore` with the
238-
human before any `--force` run.
232+
Matched resources are skipped on pull and push, and neither push nor
233+
`npm run cleanup` deletes them. A resource that references an ignored one is a
234+
validation error.
239235

240236
---
241237

‎docs/guides/workflows.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -186,11 +186,10 @@ npm run cleanup -- <org>
186186
npm run cleanup -- <org> --force --confirm <org>
187187
```
188188

189-
**Check the dry-run list before a destructive run.** Cleanup treats every
190-
platform resource that isn't in `.vapi-state.<org>.json` as an orphan, and it
191-
does not read `.vapi-ignore`: resources you've excluded with `.vapi-ignore`
192-
are never in the state file, so they appear in the list and `--force` would
193-
delete them.
189+
Cleanup treats every platform resource that isn't in
190+
`.vapi-state.<org>.json` as an orphan, except those matched by
191+
`resources/<org>/.vapi-ignore`: it lists those as retained and never deletes
192+
them. Still read the dry-run list before a destructive run.
194193

195194
**When the list includes Vapi's built-in fixtures** (for example the stock
196195
simulation personalities, which can't be deleted — see

‎improvements.md‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ you which stack PR closes the row.**
8787
| 33 | `npm run sim` reported every run as passed | A failing suite exited 0 — false green | None | RESOLVED 2026-10-01 |
8888
| 34 | No pre-merge simulation signal; simulations only tested what was deployed | A PR that breaks an agent merges green | #33 | RESOLVED 2026-10-01 |
8989
| 35 | A failed promotion pushed nothing, not even state | git lost track of resources already on the platform | None | RESOLVED 2026-10-01 |
90-
| 36 | `cleanup` deletes resources excluded by `.vapi-ignore` | A destructive cleanup can delete resources another team owns | None | Open |
90+
| 36 | `cleanup` deletes resources excluded by `.vapi-ignore` | A destructive cleanup can delete resources another team owns | None | RESOLVED 2026-10-03 |
9191

9292
**Active backlog after cleanup:** `#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.
9393

@@ -1888,6 +1888,8 @@ None needed once the fix below lands.
18881888

18891889
## 36. `cleanup` deletes resources excluded by `.vapi-ignore`
18901890

1891+
**[RESOLVED 2026-10-03]**
1892+
18911893
**Discovered:** 2026-10-03, while reviewing the agent instructions for the public release.
18921894

18931895
### Problem
@@ -1914,19 +1916,20 @@ another team deletes that team's assistants, tools or squads.
19141916

19151917
### Current mitigation
19161918

1917-
The default run is a dry run, and the destructive run needs `--confirm <org>`.
1918-
The docs and `AGENTS.md` now tell people and agents to check the dry-run list
1919-
against `.vapi-ignore` first.
1919+
None needed once the fix below lands.
19201920

1921-
### Possible fix
1921+
### Possible fix (landed)
19221922

1923-
Load the org's ignore patterns in `cleanup.ts` and exclude matches from the
1924-
deletion list (printing them as retained, as push does), with a test that a
1925-
destructive cleanup keeps an ignored resource.
1923+
`src/cleanup.ts` loads the org's ignore patterns and keeps every orphan that
1924+
matches one, checking the IDs pull would give the resource (its name slug,
1925+
with and without the UUID suffix). Kept resources are listed as retained.
1926+
`tests/cleanup-ignore.test.ts` runs a destructive cleanup against a stub API:
1927+
before the fix it deleted an ignored assistant and tool along with the true
1928+
orphan; after it, only the orphan.
19261929

19271930
### Status
19281931

1929-
Open.
1932+
**RESOLVED 2026-10-03.**
19301933

19311934
---
19321935

‎resources/.vapi-ignore.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
# id matches `.vapi-ignore`.
1111
# - `--force` on push bypasses the load-filter (so a deliberate override
1212
# can flow through) BUT the orphan-protect still applies.
13+
# - `npm run cleanup -- <org>` never deletes a matched resource, even with
14+
# `--force --confirm <org>`; it lists it as retained.
1315
# - A resource that references an ignored resource (e.g. a squad
1416
# pointing at `assistants/foo` while `assistants/foo` is ignored) is
1517
# a validation ERROR — `--strict` push aborts before any API call.

‎src/cleanup.ts‎

Lines changed: 87 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,16 @@
11
import { resolve } from "path";
22
import { fileURLToPath } from "url";
3-
import { VAPI_BASE_URL, VAPI_ENV, VAPI_TOKEN } from "./config.ts";
3+
import {
4+
loadIgnorePatterns,
5+
matchesIgnore,
6+
VAPI_BASE_URL,
7+
VAPI_ENV,
8+
VAPI_TOKEN,
9+
} from "./config.ts";
10+
import { FOLDER_MAP } from "./resource-parse.ts";
11+
import { slugify } from "./slug-utils.ts";
412
import { loadState } from "./state.ts";
13+
import type { ResourceType } from "./types.ts";
514

615
// ─────────────────────────────────────────────────────────────────────────────
716
// Dangerous Sync - Delete everything NOT in state file
@@ -68,6 +77,30 @@ async function vapiDelete(endpoint: string): Promise<void> {
6877
interface VapiResource {
6978
id: string;
7079
name?: string;
80+
function?: { name?: string };
81+
}
82+
83+
// The .vapi-ignore pattern a platform resource matches, or null. Checks the
84+
// ids pull would give it — its name slug with the UUID suffix (what pull
85+
// writes for an untracked resource) and without it — so a pattern written
86+
// against either form protects the resource.
87+
function ignoredBy(
88+
folder: string,
89+
resource: VapiResource,
90+
patterns: string[],
91+
): string | null {
92+
if (patterns.length === 0) return null;
93+
// Truthy, not ??: pull treats an empty name as missing (extractName).
94+
const name = resource.name || resource.function?.name;
95+
const shortId = resource.id.slice(0, 8);
96+
const ids = name
97+
? [`${slugify(name)}-${shortId}`, slugify(name)]
98+
: [`resource-${shortId}`];
99+
for (const id of ids) {
100+
const matched = matchesIgnore(folder, id, patterns);
101+
if (matched) return matched;
102+
}
103+
return null;
71104
}
72105

73106
function readConfirmToken(argv: string[]): string | undefined {
@@ -149,43 +182,75 @@ async function main(): Promise<void> {
149182
}[] = [];
150183

151184
// Fetch and compare each resource type
152-
const resourceTypes = [
185+
const resourceTypes: Array<{
186+
type: ResourceType;
187+
name: string;
188+
endpoint: string;
189+
deleteEndpoint: string;
190+
}> = [
153191
{
192+
type: "assistants",
154193
name: "assistants",
155194
endpoint: "/assistant",
156195
deleteEndpoint: "/assistant",
157196
},
158-
{ name: "tools", endpoint: "/tool", deleteEndpoint: "/tool" },
159197
{
198+
type: "tools",
199+
name: "tools",
200+
endpoint: "/tool",
201+
deleteEndpoint: "/tool",
202+
},
203+
{
204+
type: "structuredOutputs",
160205
name: "structured outputs",
161206
endpoint: "/structured-output",
162207
deleteEndpoint: "/structured-output",
163208
},
164-
{ name: "squads", endpoint: "/squad", deleteEndpoint: "/squad" },
165209
{
210+
type: "squads",
211+
name: "squads",
212+
endpoint: "/squad",
213+
deleteEndpoint: "/squad",
214+
},
215+
{
216+
type: "personalities",
166217
name: "personalities",
167218
endpoint: "/eval/simulation/personality",
168219
deleteEndpoint: "/eval/simulation/personality",
169220
},
170221
{
222+
type: "scenarios",
171223
name: "scenarios",
172224
endpoint: "/eval/simulation/scenario",
173225
deleteEndpoint: "/eval/simulation/scenario",
174226
},
175227
{
228+
type: "simulations",
176229
name: "simulations",
177230
endpoint: "/eval/simulation",
178231
deleteEndpoint: "/eval/simulation",
179232
},
180233
{
234+
type: "simulationSuites",
181235
name: "simulation suites",
182236
endpoint: "/eval/simulation/suite",
183237
deleteEndpoint: "/eval/simulation/suite",
184238
},
185-
{ name: "evals", endpoint: "/eval", deleteEndpoint: "/eval" },
239+
{
240+
type: "evals",
241+
name: "evals",
242+
endpoint: "/eval",
243+
deleteEndpoint: "/eval",
244+
},
186245
];
187246

188-
for (const { name, endpoint, deleteEndpoint } of resourceTypes) {
247+
// Resources this repo must not manage (.vapi-ignore) are never written to
248+
// state, so every one of them would look like an orphan here. Keep them,
249+
// as push's orphan-protection does.
250+
const ignorePatterns = loadIgnorePatterns();
251+
let retained = 0;
252+
253+
for (const { type, name, endpoint, deleteEndpoint } of resourceTypes) {
189254
console.log(`📥 Fetching ${name}...`);
190255
try {
191256
// Enable debug for structured outputs to see response format
@@ -202,7 +267,16 @@ async function main(): Promise<void> {
202267
continue;
203268
}
204269

205-
const orphans = resources.filter((r) => !stateIds.has(r.id));
270+
const orphans: VapiResource[] = [];
271+
for (const r of resources.filter((r) => !stateIds.has(r.id))) {
272+
const matched = ignoredBy(FOLDER_MAP[type], r, ignorePatterns);
273+
if (matched) {
274+
console.log(
275+
` 🚫 ${r.name || r.function?.name || r.id} retained (matched .vapi-ignore: ${matched})`,
276+
);
277+
retained++;
278+
} else orphans.push(r);
279+
}
206280

207281
if (orphans.length > 0) {
208282
console.log(
@@ -228,6 +302,12 @@ async function main(): Promise<void> {
228302
"\n═══════════════════════════════════════════════════════════════",
229303
);
230304

305+
if (retained > 0) {
306+
console.log(
307+
`\n🚫 ${retained} resource(s) not in state were kept because they match .vapi-ignore`,
308+
);
309+
}
310+
231311
if (toDelete.length === 0) {
232312
console.log("✅ Nothing to delete - all resources match state file\n");
233313
return;

0 commit comments

Comments
 (0)