Skip to content

Commit 62fe579

Browse files
feat(engine): pin each org folder to its Vapi org (#84)
## Value **V.A.L.U.E. tier:** small. It adds a check before every command's first request, a new committed file per org folder, and a new way for commands to stop. Stacked on [#83](#83). - **Problem:** nothing ties `resources/<org>/` to a Vapi org. The key alone decides where a command reads and writes. #83 removes one way to get the wrong key (an exported generic variable), but not the others: a mistyped per-org variable, a `.env` file copied between orgs, a CI secret holding the wrong org's key. With any of these, `apply` pushes the folder into the wrong org, `cleanup` deletes from it, and `pull` fills the folder with another org's resources. Each one runs without an error. - **Who it affects:** teams with more than one org, especially dev and prod, where one wrong key sends a prod deploy to dev or dev's configuration to prod. - **What changes:** - **Pinning:** the first command that can see the key's org records it in `resources/<org>/.vapi-org.json`, which is committed with the folder. Promotion skips dotfiles, so the pin never travels to another org. - **Checking:** every command then checks the key before its first request: `pull`, `push`, `apply`, `promote` (through its child commands), `cleanup` and `rollback`. It stops when the key belongs to another org, naming both org IDs and where the key came from, never the key. It also stops when the key's org has no resources but the folder is pinned to an org that had some, since pushing would fill a fresh org with this folder. - **How it finds the key's org:** a private key can't ask which org it belongs to (`/org` and `/org/me` return 401), so the org ID comes from one-item list calls: `/assistant?limit=1`, then other types until one lists something. A matching key costs one extra request per command. If the probes fail (5xx, network), the command warns that the key wasn't checked and continues. A folder with no pin and no files makes no extra request. - **Escape hatch:** delete `.vapi-org.json` in a reviewed commit. There's no flag or environment variable to skip the check. The test preload sets an internal switch so tests that count requests stay deterministic, and `org-pin.test.ts` turns it back on. - **Docs:** README, AGENTS.md and the configuration guide describe the file. AGENTS.md tells agents to stop and ask, and never to edit or delete the pin to get past a stop. - **Not covered yet:** `call`, `sim` and `check` don't check the pin. They don't change configuration, but they could run against the wrong org. - **Known gap:** an org with no resources can't be identified, so it can't be pinned until something exists in it. A backend endpoint that tells a private key its org would close this, and would let setup show the org's name. ## Evidence of value | `resources/acme/` pinned to org A | Before | After | | --- | --- | --- | | Command with org B's key | Reads and writes org B | Stops after one probe request: "belongs to Vapi org B, but resources/acme/ is pinned to org A" | | Command with an empty org's key | Pushes the folder into it | Stops: "belongs to an org with no resources" | | Command with org A's key | Runs | Runs (one extra GET) | | Probes fail (5xx, network) | n/a | Warns the key wasn't checked, then runs | - **Tests (13 new, 575 total):** probe parsing (array and `{ results }` pages, empty, failed, 401), every check outcome, a corrupt pin, and no requests for an unpinned org with no folder. An end-to-end test runs `pull` against a stub API returning org B for a folder pinned to org A: exit 1, and only `/assistant?limit=1` reached the API. - **Live run** against a production org (read-only `pull --bootstrap --bindings-only`): the first run pinned the folder to the right org. With a fake org ID planted in the pin, the next run stopped at its first request, before listing anything. With the pin restored, it ran with no extra output. ## Testing plan - `npm run build` and `npm test` (575) pass. - **After merge:** in the dogfood repo, run `npm run pull -- <org>` once per org and commit the new `.vapi-org.json` files. Then run `apply --dry-run` with another org's key exported as that org's per-org variable, and check that it stops. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents 27836bc + 2a45f8e commit 62fe579

10 files changed

Lines changed: 515 additions & 6 deletions

File tree

‎AGENTS.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,12 @@ wizard needs a terminal. Instead:
110110
Add `--region eu` for EU orgs if auto-detection picks the wrong one.
111111
4. Verify: `npm run validate -- <org>` passes, and `resources/<org>/` plus
112112
`.vapi-state.<org>.json` exist.
113-
5. Commit `resources/<org>/` and `.vapi-state.<org>.json`. Never commit `.env.<org>`.
113+
5. Commit `resources/<org>/` (including `.vapi-org.json`) and `.vapi-state.<org>.json`. Never commit `.env.<org>`.
114+
115+
If a command stops because the key "belongs to Vapi org …, but resources/<org>/
116+
is pinned to org …", the key is for another org. Stop and tell the human.
117+
Never delete or edit `.vapi-org.json` to get past it: that file is the only
118+
thing tying the folder to its org.
114119

115120
If setup says the org is "already set up locally", don't delete anything to
116121
work around it — run `npm run pull -- <org>`, or ask the human.
@@ -138,8 +143,8 @@ precisely.
138143
run it locally for the org it names and fix the errors. Without that org's
139144
`.env.<org>`, run `VAPI_PRIVATE_API_KEY=validate-only npm run validate -- <org>`;
140145
never ask for a real key just to validate. Don't weaken the check or the
141-
workflow to get past it, and never edit the state file or `.vapi-ignore`
142-
to make a reference resolve: fix the name, or pull.
146+
workflow to get past it, and never edit the state file, `.vapi-ignore` or
147+
`.vapi-org.json` to make a reference resolve: fix the name, or pull.
143148
4. **Build PR checks offline** if `vapi-checks.yml` exists:
144149
`npm run check -- --all --dry-run`. Fix anything it reports.
145150
5. **Deploy only with a yes** (safety rule 1): `npm run apply -- <org>`, or

‎README.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,13 +182,18 @@ To test every pull request automatically, set up [PR checks](docs/guides/pr-chec
182182
- **Secrets stay out of git.** API keys live in `.env.<org>` (gitignored) or
183183
CI secrets. Credentials are referenced by name and bound to each org's own
184184
credentials.
185+
- **Each folder is pinned to its org.** The first command that can see which
186+
org a key belongs to records it in `resources/<org>/.vapi-org.json`. After
187+
that, every command checks the key against it before any request, and stops
188+
if the key belongs to a different org, so a wrong key can't pull another
189+
org's resources into the folder or push the folder into another org.
185190
- **`apply` is the default deploy.** It pulls first, so it won't overwrite
186191
changes made in the dashboard since your last pull. `pull` only syncs down;
187192
raw `push` skips the pull and is rarely what you want.
188193

189194
| Path | Committed? | What it is |
190195
| --- | --- | --- |
191-
| `resources/<org>/` | Yes | Your resources |
196+
| `resources/<org>/` | Yes | Your resources, and `.vapi-org.json`, the org the folder belongs to |
192197
| `.vapi-state.<org>.json` | Yes | Name → UUID mappings for the org |
193198
| `.env.<org>` | **No** | API key and generated binding IDs |
194199
| `.vapi-state-hash/` | No | Your local baseline for detecting dashboard edits |

‎docs/guides/configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ Set these in GitHub under **Settings → Secrets and variables → Actions**.
4949
| --- | --- |
5050
| `promotion.yml` | Promotion pipelines between orgs. Start from `promotion.example.yml`. |
5151
| `vapi-checks.yml` | PR checks. Start from `vapi-checks.example.yml`. |
52+
| `resources/<org>/.vapi-org.json` | The Vapi org ID the folder belongs to. Written by the first command that can see the key's org, then checked by every command before any request: a key for another org stops the command. If a folder genuinely moves to another org, delete the file in a reviewed commit. |
5253
| `resources/<org>/.vapi-ignore` | Platform resources this repo shouldn't manage, as glob patterns. See `resources/.vapi-ignore.example` and [YAML conventions](../learnings/yaml-conventions.md). |
5354

5455
## Other environment variables

‎src/api.ts‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,12 @@
1-
import { DRY_RUN, VAPI_BASE_URL, VAPI_TOKEN } from "./config.ts";
1+
import {
2+
BASE_DIR,
3+
DRY_RUN,
4+
VAPI_BASE_URL,
5+
VAPI_ENV,
6+
VAPI_TOKEN,
7+
VAPI_TOKEN_SOURCE,
8+
} from "./config.ts";
9+
import { ORG_PIN_FILE, OrgPinMismatchError, orgPinCheck } from "./org-pin.ts";
210
import type { VapiResponse } from "./types.ts";
311
import { userAgentGet } from "./user-agent.ts";
412
import {
@@ -22,6 +30,47 @@ import {
2230

2331
const DRY_RUN_COUNTS = { POST: 0, PATCH: 0, DELETE: 0 };
2432

33+
let orgPinChecked: Promise<void> | undefined;
34+
35+
/**
36+
* Check the key against resources/<org>/.vapi-org.json once per process,
37+
* before the first request (see org-pin.ts). A mismatch exits instead of
38+
* throwing, because several callers catch request errors and retry or carry
39+
* on, and none of them should reach the wrong org.
40+
*/
41+
export function orgPinGuard(): Promise<void> {
42+
orgPinChecked ??= (async () => {
43+
try {
44+
const outcome = await orgPinCheck({
45+
rootDir: BASE_DIR,
46+
org: VAPI_ENV,
47+
connection: {
48+
token: VAPI_TOKEN ?? "",
49+
baseUrl: VAPI_BASE_URL,
50+
userAgent: userAgentGet(),
51+
},
52+
keySource: VAPI_TOKEN_SOURCE,
53+
});
54+
if (outcome.kind === "pinned")
55+
console.log(
56+
`📌 Pinned resources/${VAPI_ENV}/ to Vapi org ${outcome.orgId} (${ORG_PIN_FILE}). Commit it with the folder.`,
57+
);
58+
if (outcome.kind === "unverified")
59+
console.warn(
60+
`⚠️ Couldn't confirm which Vapi org the key belongs to, so it wasn't checked against resources/${VAPI_ENV}/${ORG_PIN_FILE}.`,
61+
);
62+
} catch (error) {
63+
if (error instanceof OrgPinMismatchError) {
64+
console.error(`❌ ${error.message}`);
65+
process.exit(1);
66+
}
67+
throw error;
68+
}
69+
})();
70+
return orgPinChecked;
71+
}
72+
73+
2574
export function getDryRunCounts(): {
2675
POST: number;
2776
PATCH: number;
@@ -75,6 +124,7 @@ export async function vapiRequest<T = VapiResponse>(
75124
endpoint: string,
76125
body: Record<string, unknown>,
77126
): Promise<T> {
127+
await orgPinGuard();
78128
const url = `${VAPI_BASE_URL}${endpoint}`;
79129

80130
if (DRY_RUN) {
@@ -134,6 +184,7 @@ export async function vapiRequest<T = VapiResponse>(
134184
}
135185

136186
export async function vapiGet<T = unknown>(endpoint: string): Promise<T> {
187+
await orgPinGuard();
137188
const url = `${VAPI_BASE_URL}${endpoint}`;
138189

139190
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
@@ -177,6 +228,7 @@ export async function vapiGet<T = unknown>(endpoint: string): Promise<T> {
177228
}
178229

179230
export async function vapiDelete(endpoint: string): Promise<void> {
231+
await orgPinGuard();
180232
const url = `${VAPI_BASE_URL}${endpoint}`;
181233

182234
if (DRY_RUN) {

‎src/cleanup.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import {
77
VAPI_ENV,
88
VAPI_TOKEN,
99
} from "./config.ts";
10+
import { orgPinGuard } from "./api.ts";
1011
import { FOLDER_MAP } from "./resource-parse.ts";
1112
import { slugify } from "./slug-utils.ts";
1213
import { loadState } from "./state.ts";
@@ -28,6 +29,7 @@ function isRecord(value: unknown): value is Record<string, unknown> {
2829
}
2930

3031
async function vapiGet<T>(endpoint: string, debug = false): Promise<T> {
32+
await orgPinGuard();
3133
await sleep(REQUEST_DELAY_MS);
3234
const response = await fetch(`${VAPI_BASE_URL}${endpoint}`, {
3335
headers: {
@@ -68,6 +70,7 @@ async function vapiGet<T>(endpoint: string, debug = false): Promise<T> {
6870
}
6971

7072
async function vapiDelete(endpoint: string): Promise<void> {
73+
await orgPinGuard();
7174
await sleep(REQUEST_DELAY_MS);
7275
const response = await fetch(`${VAPI_BASE_URL}${endpoint}`, {
7376
method: "DELETE",

‎src/config.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,8 @@ const API_KEY = resolveOrgApiKey(VAPI_ENV, SHELL_ENV, ENV_FILE_VARS);
289289

290290
// API configuration
291291
export const VAPI_TOKEN = API_KEY?.token;
292+
/** Where VAPI_TOKEN came from, for messages that must never show the key. */
293+
export const VAPI_TOKEN_SOURCE = API_KEY?.source ?? "the environment";
292294
export const VAPI_BASE_URL = process.env.VAPI_BASE_URL || "https://api.vapi.ai";
293295

294296
if (!VAPI_TOKEN) {

‎src/org-pin.ts‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
// Pins each org folder to the Vapi org it was set up from.
2+
//
3+
// Nothing else ties resources/<org>/ to an org: the key alone decides where a
4+
// command reads and writes. A key for the wrong org (an exported variable, a
5+
// copied .env file, a mislabelled CI secret) would pull another org's
6+
// resources into the folder, or push this folder into another org. The first
7+
// command that can see the key's org records it in resources/<org>/.vapi-org.json,
8+
// which is committed with the folder. Every later command checks the key
9+
// against it and refuses a mismatch before any request.
10+
//
11+
// A private key can't ask which org it belongs to (/org and /org/me reject
12+
// private keys), so the org ID comes from the first resource the key can
13+
// list. Promotion skips dotfiles, so the pin never travels to another org.
14+
//
15+
// Config-free so rollback and tests can check any org.
16+
17+
import { existsSync, readFileSync, writeFileSync } from "node:fs";
18+
import { join } from "node:path";
19+
import { orgApiKeyVar } from "./api-key.ts";
20+
import {
21+
type VapiConnection,
22+
VapiApiError,
23+
vapiFetchJson,
24+
} from "./vapi-client.ts";
25+
26+
export const ORG_PIN_FILE = ".vapi-org.json";
27+
28+
// Cheap one-item list calls, most likely non-empty first. Every item carries
29+
// the orgId it belongs to.
30+
export const ORG_PROBE_ENDPOINTS = [
31+
"/assistant",
32+
"/tool",
33+
"/squad",
34+
"/phone-number",
35+
"/credential",
36+
"/structured-output",
37+
];
38+
39+
export type OrgProbeFetch = (
40+
connection: VapiConnection,
41+
endpoint: string,
42+
) => Promise<unknown>;
43+
44+
/** What the key's resources say about its org. */
45+
export type KeyOrg =
46+
| { kind: "found"; orgId: string }
47+
// Every probe answered and none listed anything: the key's org is empty.
48+
| { kind: "empty" }
49+
// A probe failed (network, 5xx, 401), so the org can't be told.
50+
| { kind: "unknown" };
51+
52+
export type OrgPinOutcome =
53+
| { kind: "match" }
54+
| { kind: "pinned"; orgId: string }
55+
| { kind: "unverified" }
56+
| { kind: "unpinned" };
57+
58+
export class OrgPinMismatchError extends Error {
59+
constructor(message: string) {
60+
super(message);
61+
this.name = "OrgPinMismatchError";
62+
}
63+
}
64+
65+
// A probe is a safety check, not the command: don't spend a minute of
66+
// backoff on a flaky endpoint before the real requests start.
67+
const probeFetchDefault: OrgProbeFetch = (connection, endpoint) =>
68+
vapiFetchJson(connection, "GET", endpoint, undefined, {
69+
retry: "rate-limit-only",
70+
maxRetries: 2,
71+
});
72+
73+
export function orgPinPath(rootDir: string, org: string): string {
74+
return join(rootDir, "resources", org, ORG_PIN_FILE);
75+
}
76+
77+
/** The org ID this folder is pinned to, if any. A corrupt pin file throws. */
78+
export function orgPinRead(rootDir: string, org: string): string | undefined {
79+
const path = orgPinPath(rootDir, org);
80+
if (!existsSync(path)) return undefined;
81+
let parsed: unknown;
82+
try {
83+
parsed = JSON.parse(readFileSync(path, "utf8"));
84+
} catch (error) {
85+
const reason = error instanceof Error ? error.message : String(error);
86+
throw new Error(
87+
`Can't read resources/${org}/${ORG_PIN_FILE}: ${reason}. Restore it from git.`,
88+
);
89+
}
90+
const orgId = (parsed as { orgId?: unknown } | null)?.orgId;
91+
return typeof orgId === "string" && orgId ? orgId : undefined;
92+
}
93+
94+
function itemsOf(data: unknown): unknown[] {
95+
if (Array.isArray(data)) return data;
96+
const results = (data as { results?: unknown } | null)?.results;
97+
return Array.isArray(results) ? results : [];
98+
}
99+
100+
/** Find the key's org from the first resource it can list. */
101+
export async function keyOrgFetch(
102+
connection: VapiConnection,
103+
probeFetch: OrgProbeFetch = probeFetchDefault,
104+
): Promise<KeyOrg> {
105+
let failed = false;
106+
for (const endpoint of ORG_PROBE_ENDPOINTS) {
107+
let data: unknown;
108+
try {
109+
data = await probeFetch(connection, `${endpoint}?limit=1`);
110+
} catch (error) {
111+
// A rejected key fails every probe; the command reports it properly.
112+
if (error instanceof VapiApiError && error.statusCode === 401)
113+
return { kind: "unknown" };
114+
failed = true;
115+
continue;
116+
}
117+
for (const item of itemsOf(data)) {
118+
const orgId = (item as { orgId?: unknown } | null)?.orgId;
119+
if (typeof orgId === "string" && orgId) return { kind: "found", orgId };
120+
}
121+
}
122+
return failed ? { kind: "unknown" } : { kind: "empty" };
123+
}
124+
125+
export interface OrgPinCheckArgs {
126+
rootDir: string;
127+
/** The org slug: the folder under resources/. */
128+
org: string;
129+
connection: VapiConnection;
130+
/** Where the key came from, for the error message (never the key). */
131+
keySource: string;
132+
probeFetch?: OrgProbeFetch;
133+
}
134+
135+
/**
136+
* Check the key against resources/<org>/.vapi-org.json. Pins an unpinned
137+
* folder to the key's org on first sight. Throws OrgPinMismatchError when the
138+
* key belongs to another org, or to an empty org while the folder is pinned
139+
* to one that had resources.
140+
*/
141+
export async function orgPinCheck(args: OrgPinCheckArgs): Promise<OrgPinOutcome> {
142+
const { rootDir, org, connection, keySource } = args;
143+
// Set only by tests/no-vapi-api.ts: tests that stub fetch count requests.
144+
if (process.env.VAPI_GITOPS_ORG_PIN === "off") return { kind: "unpinned" };
145+
const pinned = orgPinRead(rootDir, org);
146+
// No folder and no pin: nothing to protect or record (a CI run org, or an
147+
// org whose first pull hasn't created the folder yet). Skip the probes.
148+
const folderExists = existsSync(join(rootDir, "resources", org));
149+
if (!pinned && !folderExists) return { kind: "unpinned" };
150+
const keyOrg = await keyOrgFetch(connection, args.probeFetch);
151+
const folder = `resources/${org}/`;
152+
const fix =
153+
` Use ${org}'s own key, for example by exporting ${orgApiKeyVar(org)}.\n` +
154+
` If this folder really moved to another org, delete ${folder}${ORG_PIN_FILE}\n` +
155+
` in a reviewed commit and run the command again.`;
156+
157+
if (keyOrg.kind === "found") {
158+
if (pinned === keyOrg.orgId) return { kind: "match" };
159+
if (pinned)
160+
throw new OrgPinMismatchError(
161+
`The API key from ${keySource} belongs to Vapi org ${keyOrg.orgId}, but ${folder} is pinned to org ${pinned} (${ORG_PIN_FILE}).\n` +
162+
` Stopped before reading or changing anything.\n${fix}`,
163+
);
164+
if (!folderExists) return { kind: "unpinned" };
165+
writeFileSync(
166+
orgPinPath(rootDir, org),
167+
`${JSON.stringify({ orgId: keyOrg.orgId }, null, 2)}\n`,
168+
);
169+
return { kind: "pinned", orgId: keyOrg.orgId };
170+
}
171+
172+
if (!pinned) return { kind: "unpinned" };
173+
// The folder was pinned from an org that listed resources. A key whose org
174+
// lists none is far more likely a fresh, different org than the same org
175+
// emptied out, and pushing would fill it with this folder's resources.
176+
if (keyOrg.kind === "empty")
177+
throw new OrgPinMismatchError(
178+
`The API key from ${keySource} belongs to an org with no resources, but ${folder} is pinned to org ${pinned} (${ORG_PIN_FILE}).\n` +
179+
` Stopped before reading or changing anything.\n${fix}`,
180+
);
181+
return { kind: "unverified" };
182+
}

0 commit comments

Comments
 (0)