You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b0f37f5
Browse filesBrowse the repository at this point in the historyBrowse files
feat(validate): catch broken references and show findings on the PR (#77)
## Value
**V.A.L.U.E. tier:** small — a behavior change: `validate`, and so `apply` and the Validate resources check, now fail on configs they used to pass.
- **Problem:** a reference that names no file fails in one of three ways, depending on the field (`improvements.md` #31):
- it's **silently dropped**: `model.toolIds`, `artifactPlan.structuredOutputIds`;
- it's **sent raw and rejected partway through a push**: squad members, hook tools, `personalityId`, `scenarioId`;
- or it's **deferred** to a linking pass.
`validate` never checked references, so the Validate resources check from #76 couldn't catch a typo'd tool name either. Warnings were also invisible in CI: they don't fail the check, and nobody reads the job log.
- **Who it affects:** everyone who edits resource files by hand or with a coding agent, and reviewers of their PRs.
- **What changes:**
- **New `src/validate-refs.ts`**, run by `validate` (so by `apply` and CI) and by `push`:
| Rule | Severity | Catches |
| --- | --- | --- |
| `dangling-reference` | error | A name with no local file and no state entry. Uses the shared reference walk, plus scenario judges' `evaluations[].structuredOutputId`. |
| `malformed-reference` | error | An empty or non-name entry in a reference list (it used to crash `validate`). |
| `override-tool-by-name` | error | A tool name in `toolIds` inside `assistantOverrides`, `membersOverrides` or `targetOverrides`, where push never resolves names. The fix it gives is `tools:append`, because `model.tools` there replaces the member's tool set. |
| `unresolved-credential` | warning | A credential name missing from the state file. The message names the org's bootstrap pull. |
| `reference-by-uuid` | warning | A UUID for a resource this repo tracks, which breaks promotion; it names the file to use. UUIDs the repo doesn't track (dashboard-owned, stock personalities) aren't reported. |
- **`validate` now also runs `reference-to-ignored`,** as `push` already did. It reads the committed state file and stays offline.
- **On GitHub Actions, every finding becomes an annotation,** so it shows on the file in the PR, warnings included.
- **`push`** reports the new rules alongside its existing validators: warnings by default, blocking under `--strict`.
- **Docs:**
- a rule-by-rule table in troubleshooting;
- the `validate` row in the commands guide;
- `AGENTS.md`: never edit the state file to make a reference resolve;
- `improvements.md` #31 marked resolved by validation.
## Evidence of value
The starter example with two typos, `scheduler` → `schedular` in the squad and `booking-confirmed` → `booking-confirmd` in a judge:
| | Before (#76) | After |
| --- | --- | --- |
| `npm run validate` | `0 error(s)` — ✅ Validation passed | `2 error(s)`, one `dangling-reference` per typo, naming the file and the missing name |
- **New `tests/validate-refs.test.ts`** covers:
- names that resolve to local files and to state;
- a typo in each reference field;
- ignored references;
- UUIDs and stock personalities;
- all three override keys;
- credentials as names, as UUIDs, and known to state.
- **Annotation format:** escaping of `%`, newlines, `:` and `,` is tested in `tests/validate.test.ts`.
- **End to end:** `tests/ci-validate-workflow.test.ts` runs the CI step with `GITHUB_ACTIONS=true` on the typo'd squad. The step fails, and the `::error` points at `resources/clinic/squads/front-desk.yml`.
- **Mutation:** dropping the judge collection or the override walk fails two tests.
- **Every example org still passes.** The cross-org promotion example (which has no state files) now shows an `unresolved-credential` warning, which is accurate.
## Testing plan
- `npm test` (523 tests) and `npx tsc --noEmit` pass.
- **Behavior to expect:** `apply` validates before it pulls. So a reference to a resource created in the dashboard and never pulled now stops `apply`; the message says to pull first. Before, `apply` went on to pull and push, and the reference resolved only if the pull happened to produce that exact name.
- **Not tested:** a live `apply` or `push`. Neither code path changed except for the added findings.
- **Not in this PR:** a check for secrets committed in resource files. It goes in its own PR, because a false positive there would block a customer's merge.
Refs TEST-141
## After review
- **Ignored files:** `validate` skips `.vapi-ignore`d files, as push does, so ignored files can't fail CI or `apply`.
- **One collector:** `referencesCollect` in `resolver.ts` feeds both `reference-to-ignored` and these rules, so a judge that references an ignored structured output is caught.
- **Lookups:** use `?.uuid`, so `constructor` and friends don't resolve.
- **Troubleshooting:** the table gives safe advice for ignored and override references, and AGENTS.md says never to edit the state file or `.vapi-ignore` to make a reference resolve.
- **improvements.md:** #31 is marked mitigated, with the remaining gaps listed (unchecked override paths, `toolRefs`, inline members, annotation lines). #38 records the promotion and stock-personality limitation found in #66.
Follow-ups: line numbers on annotations, the wider override walk, and deduplicating the credential walkers.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Copy file name to clipboardExpand all lines: docs/guides/commands.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ The other commands are direct only.
10
10
| Command | Usage | What it does |
11
11
| --- | --- | --- |
12
12
|`npm run setup`|`npm run setup [-- <org>]`| Connect an org: creates `.env.<org>` and `resources/<org>/`. |
13
-
|`npm run validate`|`npm run validate -- <org>`| Check resource files offline. Run it before every `apply`. |
13
+
|`npm run validate`|`npm run validate -- <org>`| Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. |
14
14
|`npm run apply`|`npm run apply -- <org> [types or paths]`|**The default deploy:** pull, merge, then push. See [workflows](workflows.md). |
15
15
|`npm run pull`|`npm run pull -- <org> [--force] [--bootstrap]`| Sync platform changes down; never overwrites local edits unless `--force`. |
16
16
|`npm run push`|`npm run push -- <org> [--dry-run] [--strict]`| Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. |
Copy file name to clipboardExpand all lines: docs/guides/troubleshooting.md
+24-7Lines changed: 24 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,8 @@
2
2
3
3
## "Reference not found" warnings
4
4
5
-
The referenced resource doesn't exist. Check:
5
+
The referenced resource doesn't exist. `npm run validate` reports these as
6
+
`dangling-reference` errors before you deploy. Check:
6
7
7
8
1. File exists in correct folder
8
9
2. Filename matches exactly (case-sensitive)
@@ -30,7 +31,9 @@ bypassed).
30
31
31
32
## "Credential with ID not found" errors
32
33
33
-
The credential UUID doesn't exist in the target org. Fix:
34
+
The credential UUID doesn't exist in the target org. `npm run validate`
35
+
warns about a credential name that isn't in the state file
36
+
(`unresolved-credential`). Fix:
34
37
35
38
1. Run `npm run pull -- <org>` to fetch credentials into the state file
36
39
2. If the credential doesn't exist, create it in the Vapi dashboard with the same name
@@ -96,11 +99,25 @@ VAPI_PRIVATE_API_KEY=validate-only npm run validate -- <org>
96
99
a placeholder is enough. You don't need that org's real key or its
97
100
`.env.<org>`.
98
101
99
-
Each error names the resource (`assistants/<id>`), field and rule. Plain `push` only warns about
100
-
these errors, so a repository that has been deploying with `push` can carry
101
-
some from before the check existed; they show up on the next pull request,
102
-
whatever it changes. Fix them in that PR or a separate one first. `apply`
103
-
refuses to deploy until they're fixed anyway.
102
+
Each finding names the resource (`assistants/<id>`), the rule and, where it
103
+
can, the field. On GitHub it's also shown on the file in the pull request.
104
+
Plain `push` only warns about these errors (unless `--strict`), so a
105
+
repository that has been deploying with `push` can carry some from before
106
+
the check existed; they show up on the next pull request, whatever it
107
+
changes. Fix them in that PR or a separate one first. `apply` refuses to
108
+
deploy until they're fixed anyway.
109
+
110
+
| Rule | Severity | What to do |
111
+
| --- | --- | --- |
112
+
|`dangling-reference`| error | A reference names no local file and no state entry. Fix the name (it's the file name without extension, including any folder), or run `npm run pull -- <org>` if the resource was created in the dashboard. Don't add a state entry by hand. |
113
+
|`malformed-reference`| error | A reference list holds an empty entry or something that isn't a name, often a `- ` left while editing. Remove it or write the name. |
114
+
|`override-tool-by-name`| error | References inside `assistantOverrides`, `membersOverrides` and `targetOverrides` aren't resolved. Put the tool inline under the override's `tools:append`. `model.tools` there replaces the member's whole tool set; see [squads](../learnings/squads.md). |
115
+
|`reference-to-ignored`| error | The referenced resource matches `.vapi-ignore`, so this repo never deploys it. Remove the reference, or reference it by UUID if it must stay dashboard-owned. Don't edit `.vapi-ignore` to get past this without the resource owner's sign-off: un-ignoring gives the resource to gitops (see [YAML conventions](../learnings/yaml-conventions.md)). |
116
+
|`name-length`| error | Shorten the name to 40 characters or fewer. |
117
+
|`voice-provider-schema`| error | Move the setting to where that voice provider expects it; the message says where. |
118
+
|`unresolved-credential`| warning | The credential name isn't in the state file. Run `npm run pull -- <org> --bootstrap` and commit the state file, or create the credential in the dashboard first. |
119
+
|`reference-by-uuid`| warning | A UUID references a resource this repo tracks, which only works in one org and breaks promotion. Use the name the warning gives. UUIDs of resources the repo doesn't track (dashboard-owned or ignored ones) aren't reported. |
120
+
|`so-assistant-lockstep`, `prompt-duplicate-*`, `max-tokens-floor`| warning | Follow the message; see [structured outputs](../learnings/structured-outputs.md) and [writing prompts](writing-prompts.md). |
104
121
105
122
A `Failed to import TypeScript resource … is not set` error from this check
106
123
means a `.ts` resource reads a variable from `.env.<org>`, which CI doesn't
Copy file name to clipboardExpand all lines: improvements.md
+77-2Lines changed: 77 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -82,13 +82,14 @@ you which stack PR closes the row.**
82
82
| 28 | Handoff tools 400 on first push into an empty org | Push aborts before the assistant-linking pass runs | None | RESOLVED 2026-08-01 |
83
83
| 29 | SO linking sent filtered `assistantIds` arrays | Silent unlink of live-but-untracked assistants | None | RESOLVED 2026-08-03 (#51) |
84
84
| 30 | Tool-linking pass could PATCH a raw assistant slug | Mid-push 400 naming the wrong resource | None | RESOLVED 2026-08-03 (#51) |
85
-
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None |Open |
85
+
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None |Mitigated 2026-10-03 (#77): validate/apply/CI block it; plain push only warns|
86
86
| 32 | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56) |
87
87
| 33 |`npm run sim` reported every run as passed | A failing suite exited 0 — false green | None | RESOLVED 2026-10-01 |
88
88
| 34 | No pre-merge simulation signal; simulations only tested what was deployed | A PR that breaks an agent merges green |#33| RESOLVED 2026-10-01 |
89
89
| 35 | A failed promotion pushed nothing, not even state | git lost track of resources already on the platform | None | RESOLVED 2026-10-01 |
90
90
| 36 |`cleanup` deletes resources excluded by `.vapi-ignore`| A destructive cleanup can delete resources another team owns | None | RESOLVED 2026-10-03 |
91
91
| 37 | Resource validation ran only at deploy time, after merge | A config `apply` refuses could merge and block deploys and promotion |#32| RESOLVED 2026-10-03 (#76) |
92
+
| 38 | Promotion can't carry a simulation that uses a stock personality by UUID | Gated orgs, and any promoted tests, need local personality files | None | Open |
92
93
93
94
**Active backlog after cleanup:**`#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.
94
95
@@ -1568,6 +1569,8 @@ the repo and a subsequent push runs.
1568
1569
1569
1570
## 31. Unresolved references are handled three different ways depending on the field, and `validate.ts` has no dangling-reference check
1570
1571
1572
+
**[Mitigated 2026-10-03] (#77)**: validate, apply and CI block it; plain `push` only warns, and the three runtime behaviours remain.
1573
+
1571
1574
**Discovered:** while fixing #29 and #30 — those two entries close the
1572
1575
loudest and quietest failure modes for their specific fields, but the
1573
1576
underlying question ("what happens when a reference resolves to nothing")
@@ -1646,9 +1649,42 @@ conditions; it doesn't require picking one runtime behavior (filter vs.
1646
1649
defer vs. 400) for every field, since it stops the push before any of those
1647
1650
three behaviors gets a chance to run.
1648
1651
1652
+
### Possible fix (landed)
1653
+
1654
+
`src/validate-refs.ts` uses `referencesCollect` (`src/resolver.ts`): the
1655
+
`extractReferencedIds` walk plus scenario judges'
1656
+
`evaluations[].structuredOutputId`, the same collector `reference-to-ignored`
1657
+
uses. It reports an error for any
1658
+
name that matches no local file and no state entry (`dangling-reference`).
1659
+
Names matched by `.vapi-ignore` are left to `reference-to-ignored`, which
1660
+
`npm run validate` now runs too. Alongside it: `override-tool-by-name`
1661
+
(an error: push never resolves `toolIds` inside overrides),
1662
+
`malformed-reference` (an error: an empty or non-name list entry, which used
1663
+
to crash `validate`), `unresolved-credential` and `reference-by-uuid`
1664
+
(warnings; only for a UUID this repo tracks, naming the file to use).
1665
+
`validate` skips `.vapi-ignore`d files, as push does. `validate`
1666
+
reads the committed state file, so the check runs offline and in CI, and
1667
+
`apply` stops on it before its pull. `push` runs the same checks with its
1668
+
other validators: warnings by default, blocking under `--strict`.
1669
+
1649
1670
### Status
1650
1671
1651
-
**Open.**
1672
+
**Mitigated 2026-10-03 (#77).**`validate`, `apply` and the Validate
1673
+
resources check stop an unresolved reference before any of that runs. Still
1674
+
open:
1675
+
1676
+
- Plain `push` only warns (unless `--strict`), then filters, defers or sends
1677
+
raw names exactly as before.
1678
+
- A reference whose file was deleted, while its state entry remains, passes
1679
+
validation. Under `push --force` the orphan pass deletes the target and its
1680
+
state entry before the apply pass, and the reference then hits the
1681
+
resolver's silent drop.
1682
+
- Names push never resolves still go unchecked: `hooks[].do[].toolId` and
1683
+
`artifactPlan.structuredOutputIds` inside overrides, `model.toolRefs` at any
1684
+
depth, and `toolIds` in a squad's inline `members[].assistant`. Only
1685
+
`toolIds` inside overrides are checked.
1686
+
- Annotations carry a file but no line, so GitHub shows them at the top of
1687
+
the file, and it keeps at most 10 per type per step.
1652
1688
1653
1689
---
1654
1690
@@ -1986,6 +2022,45 @@ against fixture orgs.
1986
2022
1987
2023
---
1988
2024
2025
+
## 38. Promotion can't carry a simulation that uses a stock personality by UUID
2026
+
2027
+
**Discovered:** 2026-10-03, while testing the promotion check gate (#66).
2028
+
2029
+
### Problem
2030
+
2031
+
Every org has Vapi's stock simulation personalities, with fixed UUIDs
2032
+
(`a0000000-0000-4000-8000-00000000000<n>`). A simulation can reference one by
2033
+
that UUID, and `npm run check` accepts it. Promotion's dependency check
2034
+
doesn't: it treats the UUID as a managed dependency that must exist as a file
2035
+
in the source org.
2036
+
2037
+
### Current behavior (Verified)
2038
+
2039
+
Promoting a simulation with `personalityId: a0000000-…` fails with
2040
+
"Referenced managed dependency is missing from source:
2041
+
personalities/a0000000-…", and nothing is promoted.
2042
+
2043
+
### Risk
2044
+
2045
+
Teams that use stock personalities can't promote their tests, and can't gate
2046
+
an org on a check whose tests use them.
2047
+
2048
+
### Current mitigation
2049
+
2050
+
Use personality files under `simulations/personalities/` in any org you
2051
+
promote out of. The promotion guide says so.
2052
+
2053
+
### Possible fix
2054
+
2055
+
Treat stock personality UUIDs as present in every org in promotion's
2056
+
dependency check, as `check-payload.ts` already does.
2057
+
2058
+
### Status
2059
+
2060
+
**Open.**
2061
+
2062
+
---
2063
+
1989
2064
## Out of scope (intentionally not improvements)
1990
2065
1991
2066
-**State file is identity-only and not git-ignored.** It's intentionally
0 commit comments