Skip to content

Commit a67a4b1

Browse files
feat: identify every gitops API request with a User-Agent (#78)
## Value **V.A.L.U.E. tier:** small — a behavior change: every API request now carries a new header. No blast-radius path. - **Problem:** we can't measure how gitops is used beyond simulations. Only `npm run sim` and `npm run check` identify themselves. Setup, pull, push, apply, promote, cleanup, rollback, call and audit send Node's default User-Agent, `node`, which was 126k of 728k `api.vapi.ai` requests in one hour this morning. So "how many deploys come from gitops, from CI or from laptops" has no answer. - **Who it affects:** the Vapi team measuring adoption after the public launch. Customers see no change in behavior. - **What changes:** - **`src/user-agent.ts`** builds `vapi-gitops-<command>/<version>`, plus ` (ci)` when `CI` or `GITHUB_ACTIONS` is set: - `<command>` comes from a fixed list of this repo's commands, so a fork's own npm script names are never sent. The first process pins its label in `VAPI_GITOPS_COMMAND`, which child processes inherit: `npm run apply` labels its pull and push `apply`, a promotion's applies are `promote`, and the PR check's bindings pull is `check`. Promotion gate runs are labelled `promote`, apart from PR check runs. - When a script is run directly, as the PR check workflow does, it falls back to the entry script's name. - `sim` and `check` keep their fixed labels, which existing simulation analytics already counts by. - **Every `fetch` to the Vapi API sends it:** `api.ts` (push, pull, apply, promote), cleanup, setup, the interactive pickers, rollback, call, and push's direct fetch. - **Fixes two tests that called production.** `cleanup-safety` and `new-file-gate` sent about 12 requests per `npm test` to `api.vapi.ai`, with a fake key, getting 401s. That breaks the repo's own rule that tests never call the real API, and with this PR it would have counted every fork's CI run as gitops usage. They now point at a dead local address. - **Docs:** `how-it-works.md` says exactly what the API sees (and that there is no other telemetry). `AGENTS.md` says every request must send the header. ## Evidence of value **Live, through the Cloudflare request logs in Axiom (`cloudflare-logpush`):** a read-only `npm run setup -- ua-check --resources none` against the test org, run from a scratch copy: | User-Agent | Requests | Status | | --- | --- | --- | | `vapi-gitops-setup/1.0.0` | 13 GET | 200 | | `vapi-gitops-test/1.0.0` | 24 GET | 401 — the two leaky tests, from the two `npm test` runs before the fix | Before this PR, both rows would have been indistinguishable `node` traffic. **Tests:** - **`tests/user-agent-coverage.test.ts`** reads every `fetch(` in `src/` and requires a `User-Agent`. On the parent branch it lists 10 call sites without one; here it lists none. It also runs `api.ts` against a local server with `npm_lifecycle_event=apply`. - **`tests/user-agent.test.ts`** pins the format: fixed sim and check labels, npm script vs. entry script vs. `cli`, label cleaning, and the CI marker for `GITHUB_ACTIONS=true`, `CI=true` and `CI=1` (but not `false`, `0` or empty). - **No more production calls:** running the full suite with a `fetch` trap that records any request to `vapi.ai` caught 12 requests before the test fix and none after. ## Testing plan - `npm test` (527 tests) and `npx tsc --noEmit` pass. - **Not covered:** - **Distinct-org counts for non-simulation commands.** The request logs carry the User-Agent but not the org. Only simulation runs get an org ID, through PostHog. Joining the two needs an API-side change, outside this repo. - **The version** is still `1.0.0`; bumping it is deliberately left out of this PR. - **The `call` command's WebSocket audio connection** isn't a Vapi REST request and doesn't carry the header. Refs TEST-141 ## After review - **Labels:** come only from the known command list, and are pinned for child processes (Chris's 🟠: the PR check's bindings pull was counted as a CI pull). `npx` and fork script names fall through to the entry script, then to `cli`. - **GitHub status call:** now sends the User-Agent too, so every `fetch` in `src/` does. The coverage test checks each call on its own and scans subfolders. - **`tests/no-vapi-api.ts`:** loads before every test file, with an unroutable base URL and no inherited real key. With a pretend key exported and a fetch trap on, the suite makes zero requests to vapi.ai. - **Docs:** `how-it-works.md` matches the code on `(ci)` and `<command>`. - **Formatting:** the unrelated Prettier changes are reverted. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents b0f37f5 + 6f0c59f commit a67a4b1

19 files changed

Lines changed: 303 additions & 20 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,8 @@ For changes under `src/`, `tests/` or `.github/`:
321321
table in the same change.
322322
- Changing an example under `examples/`? Doc snippets that start with
323323
`# examples/<path>` must match the file exactly (`npm test` checks).
324+
- Every request to the Vapi API sends `"User-Agent": userAgentGet()` from
325+
`src/user-agent.ts`; `npm test` fails on a `fetch` without it.
324326
- Commit messages follow Conventional Commits (`fix(pull): …`, `docs: …`).
325327
- When you hit engine friction ("this should be better"), add or update an
326328
entry in `improvements.md` in the same change. Upstream's log collects

‎docs/guides/how-it-works.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,17 @@ Tracks resource ID ↔ Vapi UUID mappings per org:
145145

146146
Every resource type has a section. Keys are sorted, so diffs stay readable.
147147

148+
## What Vapi sees
149+
150+
Every API request uses the org's private key and identifies the tool with a
151+
User-Agent: `vapi-gitops-<command>/<version>`, where `<command>` is the gitops
152+
command you ran (`apply`, `push`, `pull`, …; `cli` if it can't be told). It
153+
adds ` (ci)` when `GITHUB_ACTIONS=true` or `CI` is set to anything other than
154+
`false` or `0`. For example, `npm run apply` in a GitHub workflow sends
155+
`vapi-gitops-apply/1.0.0 (ci)`. Your own npm script names are never sent.
156+
Vapi uses it to count how the tool is used. Nothing else is sent beyond the
157+
requests themselves; there is no separate telemetry.
158+
148159
## Where things live
149160

150161
| Path | What it is |

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
"rollback": "tsx src/rollback-cmd.ts",
2424
"promote": "tsx src/promote-cmd.ts",
2525
"build": "tsc --noEmit",
26-
"test": "node --import tsx --test tests/*.test.ts"
26+
"test": "node --import tsx --import ./tests/no-vapi-api.ts --test tests/*.test.ts"
2727
},
2828
"devDependencies": {
2929
"@types/node": "^22.0.0",

‎src/api.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { DRY_RUN, VAPI_BASE_URL, VAPI_TOKEN } from "./config.ts";
22
import type { VapiResponse } from "./types.ts";
3+
import { userAgentGet } from "./user-agent.ts";
34
import {
45
INITIAL_DELAY_MS,
56
MAX_RETRIES,
@@ -97,6 +98,7 @@ export async function vapiRequest<T = VapiResponse>(
9798
headers: {
9899
"Content-Type": "application/json",
99100
Authorization: `Bearer ${VAPI_TOKEN}`,
101+
"User-Agent": userAgentGet(),
100102
},
101103
body: JSON.stringify(body),
102104
});
@@ -140,6 +142,7 @@ export async function vapiGet<T = unknown>(endpoint: string): Promise<T> {
140142
method: "GET",
141143
headers: {
142144
Authorization: `Bearer ${VAPI_TOKEN}`,
145+
"User-Agent": userAgentGet(),
143146
},
144147
});
145148

@@ -188,6 +191,7 @@ export async function vapiDelete(endpoint: string): Promise<void> {
188191
method: "DELETE",
189192
headers: {
190193
Authorization: `Bearer ${VAPI_TOKEN}`,
194+
"User-Agent": userAgentGet(),
191195
},
192196
});
193197

‎src/apply.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ import { execSync } from "child_process";
22
import { dirname, join, resolve } from "path";
33
import { fileURLToPath } from "url";
44
import { assertStateMigrated } from "./migrate-hash-store.ts";
5+
// Pins this command's User-Agent label for the pull and push it spawns.
6+
import "./user-agent.ts";
57

68
// ─────────────────────────────────────────────────────────────────────────────
79
// Apply: Pull → Merge → Push (safe bidirectional sync)

‎src/call.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import { dirname, join, resolve } from "path";
66
import * as readline from "readline";
77
import { fileURLToPath } from "url";
88
import type { Environment, StateFile } from "./types.ts";
9+
import { userAgentGet } from "./user-agent.ts";
910

1011
const require = createRequire(import.meta.url);
1112

@@ -362,6 +363,7 @@ async function createCall(
362363
headers: {
363364
"Content-Type": "application/json",
364365
Authorization: `Bearer ${config.token}`,
366+
"User-Agent": userAgentGet(),
365367
},
366368
body: JSON.stringify(body),
367369
});

‎src/check-status.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
// `Vapi Evals` (a stable, documented name) is what branch protection
66
// requires: per-target statuses only exist on PRs that touch a check.
77

8+
import { userAgentGet } from "./user-agent.ts";
9+
810
export const AGGREGATE_CONTEXT = "Vapi Evals";
911

1012
export type CommitState = "pending" | "success" | "failure" | "error";
@@ -84,6 +86,8 @@ export async function commitStatusPost(
8486
Accept: "application/vnd.github+json",
8587
"Content-Type": "application/json",
8688
"X-GitHub-Api-Version": "2022-11-28",
89+
// GitHub asks API clients to name themselves.
90+
"User-Agent": userAgentGet("check"),
8791
},
8892
body: JSON.stringify({
8993
context: status.context,

‎src/cleanup.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import { FOLDER_MAP } from "./resource-parse.ts";
1111
import { slugify } from "./slug-utils.ts";
1212
import { loadState } from "./state.ts";
1313
import type { ResourceType } from "./types.ts";
14+
import { userAgentGet } from "./user-agent.ts";
1415

1516
// ─────────────────────────────────────────────────────────────────────────────
1617
// Dangerous Sync - Delete everything NOT in state file
@@ -29,7 +30,10 @@ function isRecord(value: unknown): value is Record<string, unknown> {
2930
async function vapiGet<T>(endpoint: string, debug = false): Promise<T> {
3031
await sleep(REQUEST_DELAY_MS);
3132
const response = await fetch(`${VAPI_BASE_URL}${endpoint}`, {
32-
headers: { Authorization: `Bearer ${VAPI_TOKEN}` },
33+
headers: {
34+
Authorization: `Bearer ${VAPI_TOKEN}`,
35+
"User-Agent": userAgentGet(),
36+
},
3337
});
3438
if (!response.ok) {
3539
throw new Error(`GET ${endpoint} failed: ${response.status}`);
@@ -67,7 +71,10 @@ async function vapiDelete(endpoint: string): Promise<void> {
6771
await sleep(REQUEST_DELAY_MS);
6872
const response = await fetch(`${VAPI_BASE_URL}${endpoint}`, {
6973
method: "DELETE",
70-
headers: { Authorization: `Bearer ${VAPI_TOKEN}` },
74+
headers: {
75+
Authorization: `Bearer ${VAPI_TOKEN}`,
76+
"User-Agent": userAgentGet(),
77+
},
7178
});
7279
if (!response.ok && response.status !== 404) {
7380
throw new Error(`DELETE ${endpoint} failed: ${response.status}`);

‎src/interactive.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import searchableCheckbox, { BACK_SENTINEL } from "./searchableCheckbox.js";
99
// the launcher, which runs before any org/token is selected.
1010
import { isBackupCopyFile } from "./slug-utils.ts";
1111
import type { StateFile } from "./types.ts";
12+
import { userAgentGet } from "./user-agent.ts";
1213

1314
// ─────────────────────────────────────────────────────────────────────────────
1415
// Constants
@@ -223,7 +224,10 @@ async function apiGet(
223224
): Promise<unknown> {
224225
const response = await fetch(`${baseUrl}${endpoint}`, {
225226
method: "GET",
226-
headers: { Authorization: `Bearer ${token}` },
227+
headers: {
228+
Authorization: `Bearer ${token}`,
229+
"User-Agent": userAgentGet(),
230+
},
227231
});
228232
if (!response.ok) {
229233
const text = await response.text();

‎src/promotion-gate.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,8 @@ export async function promotionGateRun(
153153
connectionFor: () => ({
154154
token: connection.token,
155155
baseUrl: connection.baseUrl ?? DEFAULT_BASE_URL,
156-
userAgent: userAgentGet("check"),
156+
// Gate runs are counted apart from PR check runs.
157+
userAgent: userAgentGet("promote"),
157158
}),
158159
deadline: gateDeadline(check, Date.now()),
159160
signal: controller.signal,

0 commit comments

Comments
 (0)