You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 49d0272
Browse filesBrowse the repository at this point in the historyBrowse files
feat(validate): catch broken references and show findings on the PR
A reference that names no file and no state entry failed three different
ways depending on the field: silently dropped (toolIds,
structuredOutputIds), sent raw and rejected mid-push (squad members, hook
tools, personalityId, scenarioId), or deferred (improvements.md #31).
validate never checked it, so the new CI check couldn't either.
- src/validate-refs.ts, run by validate (so by apply and CI) and by push:
- dangling-reference (error): a name with no local file and no state
entry;
- malformed-reference (error): an empty or non-name list entry, which
used to crash validate with no file named;
- override-tool-by-name (error): toolIds names inside
assistantOverrides, membersOverrides or targetOverrides, which push
never resolves; the fix points to tools:append, since model.tools
there replaces the member's tool set;
- unresolved-credential (warning): a credential name not in state,
naming the org's bootstrap pull;
- reference-by-uuid (warning): only for a UUID this repo tracks, naming
the file to use; dashboard-owned UUIDs aren't reported.
- One collector (referencesCollect in resolver.ts) feeds both these rules
and reference-to-ignored, so judge references to ignored structured
outputs are caught; ID cleaning tolerates non-string entries.
- validate skips .vapi-ignore'd files, as push does, and now also runs
reference-to-ignored. It reads the committed state file offline.
- On GitHub Actions, validate prints each finding as an annotation, so it
shows on the file in the PR, warnings included.
- The validate header no longer prints an API URL for an offline command.
- Docs: a rule table in troubleshooting (safe advice for ignored and
override references), the commands row, AGENTS.md (never edit state or
.vapi-ignore to make a reference resolve), improvements.md #31 marked
mitigated with its remaining gaps, and #38 for promotion and stock
personalities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/guides/commands.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ The other commands are direct only.
10
10
| Command | Usage | What it does |
11
11
| --- | --- | --- |
12
12
|`npm run setup`|`npm run setup [-- <org>]`| Connect an org: creates `.env.<org>` and `resources/<org>/`. |
13
-
|`npm run validate`|`npm run validate -- <org>`| Check resource files offline. Run it before every `apply`. |
13
+
|`npm run validate`|`npm run validate -- <org>`| Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. |
14
14
|`npm run apply`|`npm run apply -- <org> [types or paths]`|**The default deploy:** pull, merge, then push. See [workflows](workflows.md). |
15
15
|`npm run pull`|`npm run pull -- <org> [--force] [--bootstrap]`| Sync platform changes down; never overwrites local edits unless `--force`. |
16
16
|`npm run push`|`npm run push -- <org> [--dry-run] [--strict]`| Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. |
Copy file name to clipboardExpand all lines: docs/guides/troubleshooting.md
+24-7Lines changed: 24 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,8 @@
2
2
3
3
## "Reference not found" warnings
4
4
5
-
The referenced resource doesn't exist. Check:
5
+
The referenced resource doesn't exist. `npm run validate` reports these as
6
+
`dangling-reference` errors before you deploy. Check:
6
7
7
8
1. File exists in correct folder
8
9
2. Filename matches exactly (case-sensitive)
@@ -30,7 +31,9 @@ bypassed).
30
31
31
32
## "Credential with ID not found" errors
32
33
33
-
The credential UUID doesn't exist in the target org. Fix:
34
+
The credential UUID doesn't exist in the target org. `npm run validate`
35
+
warns about a credential name that isn't in the state file
36
+
(`unresolved-credential`). Fix:
34
37
35
38
1. Run `npm run pull -- <org>` to fetch credentials into the state file
36
39
2. If the credential doesn't exist, create it in the Vapi dashboard with the same name
@@ -96,11 +99,25 @@ VAPI_PRIVATE_API_KEY=validate-only npm run validate -- <org>
96
99
a placeholder is enough. You don't need that org's real key or its
97
100
`.env.<org>`.
98
101
99
-
Each error names the resource (`assistants/<id>`), field and rule. Plain `push` only warns about
100
-
these errors, so a repository that has been deploying with `push` can carry
101
-
some from before the check existed; they show up on the next pull request,
102
-
whatever it changes. Fix them in that PR or a separate one first. `apply`
103
-
refuses to deploy until they're fixed anyway.
102
+
Each finding names the resource (`assistants/<id>`), the rule and, where it
103
+
can, the field. On GitHub it's also shown on the file in the pull request.
104
+
Plain `push` only warns about these errors (unless `--strict`), so a
105
+
repository that has been deploying with `push` can carry some from before
106
+
the check existed; they show up on the next pull request, whatever it
107
+
changes. Fix them in that PR or a separate one first. `apply` refuses to
108
+
deploy until they're fixed anyway.
109
+
110
+
| Rule | Severity | What to do |
111
+
| --- | --- | --- |
112
+
|`dangling-reference`| error | A reference names no local file and no state entry. Fix the name (it's the file name without extension, including any folder), or run `npm run pull -- <org>` if the resource was created in the dashboard. Don't add a state entry by hand. |
113
+
|`malformed-reference`| error | A reference list holds an empty entry or something that isn't a name, often a `- ` left while editing. Remove it or write the name. |
114
+
|`override-tool-by-name`| error | References inside `assistantOverrides`, `membersOverrides` and `targetOverrides` aren't resolved. Put the tool inline under the override's `tools:append`. `model.tools` there replaces the member's whole tool set; see [squads](../learnings/squads.md). |
115
+
|`reference-to-ignored`| error | The referenced resource matches `.vapi-ignore`, so this repo never deploys it. Remove the reference, or reference it by UUID if it must stay dashboard-owned. Don't edit `.vapi-ignore` to get past this without the resource owner's sign-off: un-ignoring gives the resource to gitops (see [YAML conventions](../learnings/yaml-conventions.md)). |
116
+
|`name-length`| error | Shorten the name to 40 characters or fewer. |
117
+
|`voice-provider-schema`| error | Move the setting to where that voice provider expects it; the message says where. |
118
+
|`unresolved-credential`| warning | The credential name isn't in the state file. Run `npm run pull -- <org> --bootstrap` and commit the state file, or create the credential in the dashboard first. |
119
+
|`reference-by-uuid`| warning | A UUID references a resource this repo tracks, which only works in one org and breaks promotion. Use the name the warning gives. UUIDs of resources the repo doesn't track (dashboard-owned or ignored ones) aren't reported. |
120
+
|`so-assistant-lockstep`, `prompt-duplicate-*`, `max-tokens-floor`| warning | Follow the message; see [structured outputs](../learnings/structured-outputs.md) and [writing prompts](writing-prompts.md). |
104
121
105
122
A `Failed to import TypeScript resource … is not set` error from this check
106
123
means a `.ts` resource reads a variable from `.env.<org>`, which CI doesn't
Copy file name to clipboardExpand all lines: improvements.md
+77-2Lines changed: 77 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -82,13 +82,14 @@ you which stack PR closes the row.**
82
82
| 28 | Handoff tools 400 on first push into an empty org | Push aborts before the assistant-linking pass runs | None | RESOLVED 2026-08-01 |
83
83
| 29 | SO linking sent filtered `assistantIds` arrays | Silent unlink of live-but-untracked assistants | None | RESOLVED 2026-08-03 (#51) |
84
84
| 30 | Tool-linking pass could PATCH a raw assistant slug | Mid-push 400 naming the wrong resource | None | RESOLVED 2026-08-03 (#51) |
85
-
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None |Open |
85
+
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None |Mitigated 2026-10-03 (#77): validate/apply/CI block it; plain push only warns|
86
86
| 32 | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56) |
87
87
| 33 |`npm run sim` reported every run as passed | A failing suite exited 0 — false green | None | RESOLVED 2026-10-01 |
88
88
| 34 | No pre-merge simulation signal; simulations only tested what was deployed | A PR that breaks an agent merges green |#33| RESOLVED 2026-10-01 |
89
89
| 35 | A failed promotion pushed nothing, not even state | git lost track of resources already on the platform | None | RESOLVED 2026-10-01 |
90
90
| 36 |`cleanup` deletes resources excluded by `.vapi-ignore`| A destructive cleanup can delete resources another team owns | None | RESOLVED 2026-10-03 |
91
91
| 37 | Resource validation ran only at deploy time, after merge | A config `apply` refuses could merge and block deploys and promotion |#32| RESOLVED 2026-10-03 (#76) |
92
+
| 38 | Promotion can't carry a simulation that uses a stock personality by UUID | Gated orgs, and any promoted tests, need local personality files | None | Open |
92
93
93
94
**Active backlog after cleanup:**`#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.
94
95
@@ -1568,6 +1569,8 @@ the repo and a subsequent push runs.
1568
1569
1569
1570
## 31. Unresolved references are handled three different ways depending on the field, and `validate.ts` has no dangling-reference check
1570
1571
1572
+
**[Mitigated 2026-10-03] (#77)**: validate, apply and CI block it; plain `push` only warns, and the three runtime behaviours remain.
1573
+
1571
1574
**Discovered:** while fixing #29 and #30 — those two entries close the
1572
1575
loudest and quietest failure modes for their specific fields, but the
1573
1576
underlying question ("what happens when a reference resolves to nothing")
@@ -1646,9 +1649,42 @@ conditions; it doesn't require picking one runtime behavior (filter vs.
1646
1649
defer vs. 400) for every field, since it stops the push before any of those
1647
1650
three behaviors gets a chance to run.
1648
1651
1652
+
### Possible fix (landed)
1653
+
1654
+
`src/validate-refs.ts` uses `referencesCollect` (`src/resolver.ts`): the
1655
+
`extractReferencedIds` walk plus scenario judges'
1656
+
`evaluations[].structuredOutputId`, the same collector `reference-to-ignored`
1657
+
uses. It reports an error for any
1658
+
name that matches no local file and no state entry (`dangling-reference`).
1659
+
Names matched by `.vapi-ignore` are left to `reference-to-ignored`, which
1660
+
`npm run validate` now runs too. Alongside it: `override-tool-by-name`
1661
+
(an error: push never resolves `toolIds` inside overrides),
1662
+
`malformed-reference` (an error: an empty or non-name list entry, which used
1663
+
to crash `validate`), `unresolved-credential` and `reference-by-uuid`
1664
+
(warnings; only for a UUID this repo tracks, naming the file to use).
1665
+
`validate` skips `.vapi-ignore`d files, as push does. `validate`
1666
+
reads the committed state file, so the check runs offline and in CI, and
1667
+
`apply` stops on it before its pull. `push` runs the same checks with its
1668
+
other validators: warnings by default, blocking under `--strict`.
1669
+
1649
1670
### Status
1650
1671
1651
-
**Open.**
1672
+
**Mitigated 2026-10-03 (#77).**`validate`, `apply` and the Validate
1673
+
resources check stop an unresolved reference before any of that runs. Still
1674
+
open:
1675
+
1676
+
- Plain `push` only warns (unless `--strict`), then filters, defers or sends
1677
+
raw names exactly as before.
1678
+
- A reference whose file was deleted, while its state entry remains, passes
1679
+
validation. Under `push --force` the orphan pass deletes the target and its
1680
+
state entry before the apply pass, and the reference then hits the
1681
+
resolver's silent drop.
1682
+
- Names push never resolves still go unchecked: `hooks[].do[].toolId` and
1683
+
`artifactPlan.structuredOutputIds` inside overrides, `model.toolRefs` at any
1684
+
depth, and `toolIds` in a squad's inline `members[].assistant`. Only
1685
+
`toolIds` inside overrides are checked.
1686
+
- Annotations carry a file but no line, so GitHub shows them at the top of
1687
+
the file, and it keeps at most 10 per type per step.
1652
1688
1653
1689
---
1654
1690
@@ -1986,6 +2022,45 @@ against fixture orgs.
1986
2022
1987
2023
---
1988
2024
2025
+
## 38. Promotion can't carry a simulation that uses a stock personality by UUID
2026
+
2027
+
**Discovered:** 2026-10-03, while testing the promotion check gate (#66).
2028
+
2029
+
### Problem
2030
+
2031
+
Every org has Vapi's stock simulation personalities, with fixed UUIDs
2032
+
(`a0000000-0000-4000-8000-00000000000<n>`). A simulation can reference one by
2033
+
that UUID, and `npm run check` accepts it. Promotion's dependency check
2034
+
doesn't: it treats the UUID as a managed dependency that must exist as a file
2035
+
in the source org.
2036
+
2037
+
### Current behavior (Verified)
2038
+
2039
+
Promoting a simulation with `personalityId: a0000000-…` fails with
2040
+
"Referenced managed dependency is missing from source:
2041
+
personalities/a0000000-…", and nothing is promoted.
2042
+
2043
+
### Risk
2044
+
2045
+
Teams that use stock personalities can't promote their tests, and can't gate
2046
+
an org on a check whose tests use them.
2047
+
2048
+
### Current mitigation
2049
+
2050
+
Use personality files under `simulations/personalities/` in any org you
2051
+
promote out of. The promotion guide says so.
2052
+
2053
+
### Possible fix
2054
+
2055
+
Treat stock personality UUIDs as present in every org in promotion's
2056
+
dependency check, as `check-payload.ts` already does.
2057
+
2058
+
### Status
2059
+
2060
+
**Open.**
2061
+
2062
+
---
2063
+
1989
2064
## Out of scope (intentionally not improvements)
1990
2065
1991
2066
-**State file is identity-only and not git-ignored.** It's intentionally
0 commit comments