Skip to content

Commit 49d0272

Browse files
feat(validate): catch broken references and show findings on the PR
A reference that names no file and no state entry failed three different ways depending on the field: silently dropped (toolIds, structuredOutputIds), sent raw and rejected mid-push (squad members, hook tools, personalityId, scenarioId), or deferred (improvements.md #31). validate never checked it, so the new CI check couldn't either. - src/validate-refs.ts, run by validate (so by apply and CI) and by push: - dangling-reference (error): a name with no local file and no state entry; - malformed-reference (error): an empty or non-name list entry, which used to crash validate with no file named; - override-tool-by-name (error): toolIds names inside assistantOverrides, membersOverrides or targetOverrides, which push never resolves; the fix points to tools:append, since model.tools there replaces the member's tool set; - unresolved-credential (warning): a credential name not in state, naming the org's bootstrap pull; - reference-by-uuid (warning): only for a UUID this repo tracks, naming the file to use; dashboard-owned UUIDs aren't reported. - One collector (referencesCollect in resolver.ts) feeds both these rules and reference-to-ignored, so judge references to ignored structured outputs are caught; ID cleaning tolerates non-string entries. - validate skips .vapi-ignore'd files, as push does, and now also runs reference-to-ignored. It reads the committed state file offline. - On GitHub Actions, validate prints each finding as an annotation, so it shows on the file in the PR, warnings included. - The validate header no longer prints an API URL for an offline command. - Docs: a rule table in troubleshooting (safe advice for ignored and override references), the commands row, AGENTS.md (never edit state or .vapi-ignore to make a reference resolve), improvements.md #31 marked mitigated with its remaining gaps, and #38 for promotion and stock personalities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 4979ccd commit 49d0272

13 files changed

Lines changed: 879 additions & 60 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,8 @@ precisely.
135135
run it locally for the org it names and fix the errors. Without that org's
136136
`.env.<org>`, run `VAPI_PRIVATE_API_KEY=validate-only npm run validate -- <org>`;
137137
never ask for a real key just to validate. Don't weaken the check or the
138-
workflow to get past it.
138+
workflow to get past it, and never edit the state file or `.vapi-ignore`
139+
to make a reference resolve: fix the name, or pull.
139140
4. **Build PR checks offline** if `vapi-checks.yml` exists:
140141
`npm run check -- --all --dry-run`. Fix anything it reports.
141142
5. **Deploy only with a yes** (safety rule 1): `npm run apply -- <org>`, or

‎docs/guides/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ The other commands are direct only.
1010
| Command | Usage | What it does |
1111
| --- | --- | --- |
1212
| `npm run setup` | `npm run setup [-- <org>]` | Connect an org: creates `.env.<org>` and `resources/<org>/`. |
13-
| `npm run validate` | `npm run validate -- <org>` | Check resource files offline. Run it before every `apply`. |
13+
| `npm run validate` | `npm run validate -- <org>` | Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. |
1414
| `npm run apply` | `npm run apply -- <org> [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). |
1515
| `npm run pull` | `npm run pull -- <org> [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. |
1616
| `npm run push` | `npm run push -- <org> [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. |

‎docs/guides/troubleshooting.md‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22

33
## "Reference not found" warnings
44

5-
The referenced resource doesn't exist. Check:
5+
The referenced resource doesn't exist. `npm run validate` reports these as
6+
`dangling-reference` errors before you deploy. Check:
67

78
1. File exists in correct folder
89
2. Filename matches exactly (case-sensitive)
@@ -30,7 +31,9 @@ bypassed).
3031

3132
## "Credential with ID not found" errors
3233

33-
The credential UUID doesn't exist in the target org. Fix:
34+
The credential UUID doesn't exist in the target org. `npm run validate`
35+
warns about a credential name that isn't in the state file
36+
(`unresolved-credential`). Fix:
3437

3538
1. Run `npm run pull -- <org>` to fetch credentials into the state file
3639
2. If the credential doesn't exist, create it in the Vapi dashboard with the same name
@@ -96,11 +99,25 @@ VAPI_PRIVATE_API_KEY=validate-only npm run validate -- <org>
9699
a placeholder is enough. You don't need that org's real key or its
97100
`.env.<org>`.
98101

99-
Each error names the resource (`assistants/<id>`), field and rule. Plain `push` only warns about
100-
these errors, so a repository that has been deploying with `push` can carry
101-
some from before the check existed; they show up on the next pull request,
102-
whatever it changes. Fix them in that PR or a separate one first. `apply`
103-
refuses to deploy until they're fixed anyway.
102+
Each finding names the resource (`assistants/<id>`), the rule and, where it
103+
can, the field. On GitHub it's also shown on the file in the pull request.
104+
Plain `push` only warns about these errors (unless `--strict`), so a
105+
repository that has been deploying with `push` can carry some from before
106+
the check existed; they show up on the next pull request, whatever it
107+
changes. Fix them in that PR or a separate one first. `apply` refuses to
108+
deploy until they're fixed anyway.
109+
110+
| Rule | Severity | What to do |
111+
| --- | --- | --- |
112+
| `dangling-reference` | error | A reference names no local file and no state entry. Fix the name (it's the file name without extension, including any folder), or run `npm run pull -- <org>` if the resource was created in the dashboard. Don't add a state entry by hand. |
113+
| `malformed-reference` | error | A reference list holds an empty entry or something that isn't a name, often a `- ` left while editing. Remove it or write the name. |
114+
| `override-tool-by-name` | error | References inside `assistantOverrides`, `membersOverrides` and `targetOverrides` aren't resolved. Put the tool inline under the override's `tools:append`. `model.tools` there replaces the member's whole tool set; see [squads](../learnings/squads.md). |
115+
| `reference-to-ignored` | error | The referenced resource matches `.vapi-ignore`, so this repo never deploys it. Remove the reference, or reference it by UUID if it must stay dashboard-owned. Don't edit `.vapi-ignore` to get past this without the resource owner's sign-off: un-ignoring gives the resource to gitops (see [YAML conventions](../learnings/yaml-conventions.md)). |
116+
| `name-length` | error | Shorten the name to 40 characters or fewer. |
117+
| `voice-provider-schema` | error | Move the setting to where that voice provider expects it; the message says where. |
118+
| `unresolved-credential` | warning | The credential name isn't in the state file. Run `npm run pull -- <org> --bootstrap` and commit the state file, or create the credential in the dashboard first. |
119+
| `reference-by-uuid` | warning | A UUID references a resource this repo tracks, which only works in one org and breaks promotion. Use the name the warning gives. UUIDs of resources the repo doesn't track (dashboard-owned or ignored ones) aren't reported. |
120+
| `so-assistant-lockstep`, `prompt-duplicate-*`, `max-tokens-floor` | warning | Follow the message; see [structured outputs](../learnings/structured-outputs.md) and [writing prompts](writing-prompts.md). |
104121

105122
A `Failed to import TypeScript resource … is not set` error from this check
106123
means a `.ts` resource reads a variable from `.env.<org>`, which CI doesn't

‎improvements.md‎

Lines changed: 77 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,13 +82,14 @@ you which stack PR closes the row.**
8282
| 28 | Handoff tools 400 on first push into an empty org | Push aborts before the assistant-linking pass runs | None | RESOLVED 2026-08-01 |
8383
| 29 | SO linking sent filtered `assistantIds` arrays | Silent unlink of live-but-untracked assistants | None | RESOLVED 2026-08-03 (#51) |
8484
| 30 | Tool-linking pass could PATCH a raw assistant slug | Mid-push 400 naming the wrong resource | None | RESOLVED 2026-08-03 (#51) |
85-
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | Open |
85+
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | Mitigated 2026-10-03 (#77): validate/apply/CI block it; plain push only warns |
8686
| 32 | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56) |
8787
| 33 | `npm run sim` reported every run as passed | A failing suite exited 0 — false green | None | RESOLVED 2026-10-01 |
8888
| 34 | No pre-merge simulation signal; simulations only tested what was deployed | A PR that breaks an agent merges green | #33 | RESOLVED 2026-10-01 |
8989
| 35 | A failed promotion pushed nothing, not even state | git lost track of resources already on the platform | None | RESOLVED 2026-10-01 |
9090
| 36 | `cleanup` deletes resources excluded by `.vapi-ignore` | A destructive cleanup can delete resources another team owns | None | RESOLVED 2026-10-03 |
9191
| 37 | Resource validation ran only at deploy time, after merge | A config `apply` refuses could merge and block deploys and promotion | #32 | RESOLVED 2026-10-03 (#76) |
92+
| 38 | Promotion can't carry a simulation that uses a stock personality by UUID | Gated orgs, and any promoted tests, need local personality files | None | Open |
9293

9394
**Active backlog after cleanup:** `#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.
9495

@@ -1568,6 +1569,8 @@ the repo and a subsequent push runs.
15681569

15691570
## 31. Unresolved references are handled three different ways depending on the field, and `validate.ts` has no dangling-reference check
15701571

1572+
**[Mitigated 2026-10-03] (#77)**: validate, apply and CI block it; plain `push` only warns, and the three runtime behaviours remain.
1573+
15711574
**Discovered:** while fixing #29 and #30 — those two entries close the
15721575
loudest and quietest failure modes for their specific fields, but the
15731576
underlying question ("what happens when a reference resolves to nothing")
@@ -1646,9 +1649,42 @@ conditions; it doesn't require picking one runtime behavior (filter vs.
16461649
defer vs. 400) for every field, since it stops the push before any of those
16471650
three behaviors gets a chance to run.
16481651

1652+
### Possible fix (landed)
1653+
1654+
`src/validate-refs.ts` uses `referencesCollect` (`src/resolver.ts`): the
1655+
`extractReferencedIds` walk plus scenario judges'
1656+
`evaluations[].structuredOutputId`, the same collector `reference-to-ignored`
1657+
uses. It reports an error for any
1658+
name that matches no local file and no state entry (`dangling-reference`).
1659+
Names matched by `.vapi-ignore` are left to `reference-to-ignored`, which
1660+
`npm run validate` now runs too. Alongside it: `override-tool-by-name`
1661+
(an error: push never resolves `toolIds` inside overrides),
1662+
`malformed-reference` (an error: an empty or non-name list entry, which used
1663+
to crash `validate`), `unresolved-credential` and `reference-by-uuid`
1664+
(warnings; only for a UUID this repo tracks, naming the file to use).
1665+
`validate` skips `.vapi-ignore`d files, as push does. `validate`
1666+
reads the committed state file, so the check runs offline and in CI, and
1667+
`apply` stops on it before its pull. `push` runs the same checks with its
1668+
other validators: warnings by default, blocking under `--strict`.
1669+
16491670
### Status
16501671

1651-
**Open.**
1672+
**Mitigated 2026-10-03 (#77).** `validate`, `apply` and the Validate
1673+
resources check stop an unresolved reference before any of that runs. Still
1674+
open:
1675+
1676+
- Plain `push` only warns (unless `--strict`), then filters, defers or sends
1677+
raw names exactly as before.
1678+
- A reference whose file was deleted, while its state entry remains, passes
1679+
validation. Under `push --force` the orphan pass deletes the target and its
1680+
state entry before the apply pass, and the reference then hits the
1681+
resolver's silent drop.
1682+
- Names push never resolves still go unchecked: `hooks[].do[].toolId` and
1683+
`artifactPlan.structuredOutputIds` inside overrides, `model.toolRefs` at any
1684+
depth, and `toolIds` in a squad's inline `members[].assistant`. Only
1685+
`toolIds` inside overrides are checked.
1686+
- Annotations carry a file but no line, so GitHub shows them at the top of
1687+
the file, and it keeps at most 10 per type per step.
16521688

16531689
---
16541690

@@ -1986,6 +2022,45 @@ against fixture orgs.
19862022

19872023
---
19882024

2025+
## 38. Promotion can't carry a simulation that uses a stock personality by UUID
2026+
2027+
**Discovered:** 2026-10-03, while testing the promotion check gate (#66).
2028+
2029+
### Problem
2030+
2031+
Every org has Vapi's stock simulation personalities, with fixed UUIDs
2032+
(`a0000000-0000-4000-8000-00000000000<n>`). A simulation can reference one by
2033+
that UUID, and `npm run check` accepts it. Promotion's dependency check
2034+
doesn't: it treats the UUID as a managed dependency that must exist as a file
2035+
in the source org.
2036+
2037+
### Current behavior (Verified)
2038+
2039+
Promoting a simulation with `personalityId: a0000000-…` fails with
2040+
"Referenced managed dependency is missing from source:
2041+
personalities/a0000000-…", and nothing is promoted.
2042+
2043+
### Risk
2044+
2045+
Teams that use stock personalities can't promote their tests, and can't gate
2046+
an org on a check whose tests use them.
2047+
2048+
### Current mitigation
2049+
2050+
Use personality files under `simulations/personalities/` in any org you
2051+
promote out of. The promotion guide says so.
2052+
2053+
### Possible fix
2054+
2055+
Treat stock personality UUIDs as present in every org in promotion's
2056+
dependency check, as `check-payload.ts` already does.
2057+
2058+
### Status
2059+
2060+
**Open.**
2061+
2062+
---
2063+
19892064
## Out of scope (intentionally not improvements)
19902065

19912066
- **State file is identity-only and not git-ignored.** It's intentionally

‎src/push.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ import {
3939
validateNoIgnoredReferences,
4040
validateResources,
4141
} from "./validate.ts";
42+
import { validateReferences } from "./validate-refs.ts";
4243

4344
// Map a resource label to its state-file key. Used for snapshotting —
4445
// snapshot directories are keyed by the same names the state file uses.
@@ -1714,6 +1715,12 @@ async function main(): Promise<void> {
17141715
// a config that references an ignored resource is a contradiction the
17151716
// operator should see.
17161717
...validateNoIgnoredReferences(loadedResources, loadIgnorePatterns()),
1718+
...validateReferences({
1719+
loaded: loadedResources,
1720+
org: VAPI_ENV,
1721+
state,
1722+
ignorePatterns: loadIgnorePatterns(),
1723+
}),
17171724
];
17181725
if (findings.length > 0) {
17191726
console.log(summarizeFindings(findings));

‎src/resolver.ts‎

Lines changed: 56 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import type { ResourceState, StateFile } from "./types.ts";
1+
import type { ResourceState, ResourceType, StateFile } from "./types.ts";
22

33
// ─────────────────────────────────────────────────────────────────────────────
44
// ID Resolution - Convert resource IDs to Vapi UUIDs
@@ -358,8 +358,10 @@ export function extractReferencedIds(
358358
const scenarios: string[] = [];
359359
const simulations: string[] = [];
360360

361-
// Helper to clean IDs (remove comments)
362-
const cleanId = (id: string) => id.split("##")[0]?.trim() ?? "";
361+
// Helper to clean IDs (remove comments). A non-string entry (an empty
362+
// `- ` list item, an object) comes back as "" instead of throwing.
363+
const cleanId = (id: unknown) =>
364+
typeof id === "string" ? (id.split("##")[0]?.trim() ?? "") : "";
363365

364366
// Check root level toolIds
365367
if (Array.isArray(data.toolIds)) {
@@ -455,3 +457,54 @@ export function extractReferencedIds(
455457
simulations,
456458
};
457459
}
460+
461+
// The reference fields push resolves by name, by the type they name, and the
462+
// resource types that can carry them.
463+
export const REFERENCE_TYPES: Array<{
464+
refKey: keyof ExtractedReferences;
465+
refType: ResourceType;
466+
}> = [
467+
{ refKey: "tools", refType: "tools" },
468+
{ refKey: "structuredOutputs", refType: "structuredOutputs" },
469+
{ refKey: "assistants", refType: "assistants" },
470+
{ refKey: "personalities", refType: "personalities" },
471+
{ refKey: "scenarios", refType: "scenarios" },
472+
{ refKey: "simulations", refType: "simulations" },
473+
];
474+
475+
export const RESOURCE_TYPES_WITH_REFS: ResourceType[] = [
476+
"tools",
477+
"structuredOutputs",
478+
"assistants",
479+
"squads",
480+
"personalities",
481+
"scenarios",
482+
"simulations",
483+
"simulationSuites",
484+
"evals",
485+
];
486+
487+
// Every reference push resolves, by type: `extractReferencedIds` plus scenario
488+
// judges' `evaluations[].structuredOutputId`, which resolveReferences handles
489+
// separately. Names are cleaned of `##` comments; "" marks an entry that is
490+
// empty or isn't a name. One collector for every reference rule, so a field
491+
// can't be checked by one rule and missed by another.
492+
export function referencesCollect(
493+
data: Record<string, unknown>,
494+
): Map<ResourceType, string[]> {
495+
const extracted = extractReferencedIds(data);
496+
const refs = new Map<ResourceType, string[]>();
497+
for (const { refKey, refType } of REFERENCE_TYPES)
498+
refs.set(refType, [...extracted[refKey]]);
499+
if (Array.isArray(data.evaluations))
500+
for (const evaluation of data.evaluations) {
501+
if (!evaluation || typeof evaluation !== "object") continue;
502+
if (!("structuredOutputId" in evaluation)) continue;
503+
const id = (evaluation as { structuredOutputId?: unknown })
504+
.structuredOutputId;
505+
refs
506+
.get("structuredOutputs")!
507+
.push(typeof id === "string" ? (id.split("##")[0]?.trim() ?? "") : "");
508+
}
509+
return refs;
510+
}

‎src/state.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ function migrateSection(
4848
// State Management
4949
// ─────────────────────────────────────────────────────────────────────────────
5050

51-
function createEmptyState(): StateFile {
51+
export function createEmptyState(): StateFile {
5252
return {
5353
credentials: {},
5454
assistants: {},

0 commit comments

Comments
 (0)