Skip to content

Commit 2340843

Browse files
leggetterclaude
andcommitted
docs: add an agent skill for reviewing CLI changes
Adds .agents/skills/hookdeck-cli-review/SKILL.md, alongside hookdeck-cli-release, plus a pointer in AGENTS.md § Agent skills. The skill is the review delta over AGENTS.md: what to look at in a diff, and the traps a green pull request hides. - Acceptance runs on every PR to main/next with no path filter, but is not a required check, so a red or skipped run still merges -- and then blocks the next release tag, since release.yml gates its builds on acceptance. The one case it genuinely does not run is a head branch updated by merging another PR into it; workflow_dispatch is the only remedy. - A new acceptance build tag must be registered in five hand-maintained lists -- run_parallel.sh, acceptance.yml, and three in test/acceptance/README.md -- with //go:build manual as a deliberate exception. - REFERENCE.md is generated and no CI job checks it; CLITelemetry JSON tags are a wire contract; acceptance logs are world-readable on a public repo. - What not to raise: CHANGELOG entries, and the machine-written package.json version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 5c6948c commit 2340843

2 files changed

Lines changed: 237 additions & 0 deletions

File tree

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
---
2+
name: hookdeck-cli-review
3+
description: >-
4+
Reviews changes to the Hookdeck CLI: why a green pull request still needs its
5+
acceptance run read, where an acceptance-test build tag must be registered,
6+
which generated files and wire contracts break silently, which docs move with
7+
the code, and what not to raise. Use when reviewing a pull request, a diff or
8+
a branch in hookdeck-cli, or when deciding whether a change is safe to merge.
9+
---
10+
11+
# Reviewing changes to the Hookdeck CLI
12+
13+
`AGENTS.md` is the reference for building, testing and linting this repository,
14+
and for the rules it already states. Read it first. This file is the review
15+
delta: what to look at in a diff, and the traps that a green pull request
16+
hides. Where a rule already lives in `AGENTS.md`, this points at it rather than
17+
restating it.
18+
19+
## Acceptance runs on the pull request, but nothing enforces it
20+
21+
`.github/workflows/test-acceptance.yml` triggers on every `pull_request`
22+
targeting `main` or `next`, with no path filter. So the acceptance suite — the
23+
three matrix slices plus the separate telemetry job — **does run on the pull
24+
request in front of you**. Assuming it did not is the wrong call.
25+
26+
What it is not is *required*. Branch protection on `main` requires exactly:
27+
28+
```
29+
unit-test build-linux build-mac build-windows
30+
```
31+
32+
Acceptance is not in that list. A pull request with a **red or skipped**
33+
acceptance run is still mergeable, and the merge button will not tell you.
34+
35+
**So read `gh pr checks` and flag a red or skipped acceptance run**, because
36+
nothing else in the process will. That is the single most useful thing this
37+
review does that the required checks do not.
38+
39+
### The one case where it genuinely does not run
40+
41+
`pull_request` does not fire when a pull request's head branch is updated by
42+
**merging another pull request into it**. `test-acceptance.yml` states this in
43+
a comment above its triggers. A stacked branch — a release branch that collects
44+
other pull requests — can therefore reach `main` having never run acceptance at
45+
all, and `gh pr checks` shows that as an absence rather than as a failure.
46+
47+
The remedy is `workflow_dispatch`, which the same workflow accepts against any
48+
ref. One catch, also in that comment: GitHub reads the workflow definition from
49+
the ref you dispatch, so a branch can only be dispatched once it already
50+
carries the `workflow_dispatch` block. A branch cut before that block landed
51+
has to be rebased first.
52+
53+
Dispatch it on its own, though. `test/acceptance/README.md` § Rate limits
54+
explains why: a push to the branch also triggers the `pull_request` run, and
55+
two concurrent runs against the same projects exhaust the API rate limit and
56+
fail at the timeout with no assertion failures at all — which reads exactly
57+
like a code failure.
58+
59+
So "acceptance never ran" is a finding with an action attached, not just an
60+
observation.
61+
62+
### Why a reviewer should care, not just the release engineer
63+
64+
`.github/workflows/release.yml` calls the same acceptance workflow as a gate:
65+
`build-mac`, `build-linux` and `build-windows` each declare
66+
`needs: [acceptance]`, and `publish-npm` needs those three builds. A red
67+
acceptance therefore blocks the whole release pipeline.
68+
69+
Merging a red acceptance does not fail loudly at merge time. It fails at the
70+
next release tag, on someone else's change, and the bisect starts from the
71+
wrong commit.
72+
73+
### Running it yourself
74+
75+
`go test ./...` does not include the acceptance suite — see `AGENTS.md`
76+
§ Running `go test` after code changes. A local run needs real credentials in
77+
`test/acceptance/.env`, which is gitignored. Without them, "not verified" is the
78+
honest report, not a pass.
79+
80+
## A new acceptance tag has to be registered in five places
81+
82+
Every `*_test.go` under `test/acceptance/` carries exactly one feature build
83+
tag, and untagged files leak into every tag build — `AGENTS.md` § Acceptance
84+
tests and feature tags states that rule; do not repeat it in a review, check it.
85+
86+
`test/acceptance/README.md` § Parallelisation covers the rest of the mechanism:
87+
which tags are in which slice, that each slice has its own test account, and
88+
that telemetry runs separately. Read it rather than relying on this file.
89+
90+
What no file states is that the same set of tags is maintained **by hand in
91+
five places**, and they drift independently:
92+
93+
- `test/acceptance/run_parallel.sh` — `SLICE0_TAGS`, `SLICE1_TAGS`,
94+
`SLICE2_TAGS`, which is how the suite runs locally
95+
- `.github/workflows/acceptance.yml` — the `matrix.include` `tags` strings,
96+
which is how it runs in CI
97+
- `test/acceptance/README.md` § Running Tests — the "Run all automated tests"
98+
command, a single `-tags="..."` string that has to carry every tag
99+
- `test/acceptance/README.md` § Running Tests — the three per-slice commands
100+
- `test/acceptance/README.md` § Parallelisation — the `Slice N features` bullets
101+
102+
Registered in CI but not in the script, the tag never runs locally. In the
103+
script but not in CI, it never runs on a pull request. Left out of the "run all"
104+
command, the documented way to run everything quietly stops running everything —
105+
and that one is the worst of the five, because the command still exits 0.
106+
107+
All five agree today, on 21 distinct tags. Check all five whenever a diff adds
108+
one.
109+
110+
One thing the README gets wrong, before you trust it on this: it attributes the
111+
matrix to `.github/workflows/test-acceptance.yml`. That file only declares the
112+
triggers and calls `.github/workflows/acceptance.yml`, which is where the slices
113+
and their tags actually live.
114+
115+
### The `manual` tag is a deliberate exception
116+
117+
`test/acceptance/project_use_manual_test.go` is `//go:build manual`, and
118+
`manual` appears in **none** of those five lists. That is intentional: the test
119+
needs an interactive browser login, which no CI job can perform, and
120+
`test/acceptance/README.md` gives it its own `-tags=manual` command instead.
121+
Applying the registration rule literally will report it as drift. It is not.
122+
123+
### Which slice a tag lands in is a decision, not bookkeeping
124+
125+
The README describes the slices; what it does not say is that choosing between
126+
them is a judgement call. They are not interchangeable buckets:
127+
128+
- a different test account per slice means a test that assumes resources
129+
created by another slice will not find them
130+
- slice 0 already carries 12 tags against a 12-minute timeout, while slice 1
131+
carries 2
132+
133+
Adding a slow tag to slice 0 is how the suite starts timing out. If a diff adds
134+
a tag, the slice it went into is worth a sentence.
135+
136+
The telemetry job is separate for a reason worth checking against a diff: it
137+
forces `HOOKDECK_CLI_TELEMETRY_DISABLED=0`, where the matrix jobs set it to `1`.
138+
A telemetry-tagged test added to a slice will not behave as intended.
139+
140+
## Credentials, and logs anyone can read
141+
142+
This repository is public, so **every acceptance log is world-readable**.
143+
`.github/workflows/acceptance.yml` says so itself, which is why the test
144+
credentials have to belong to a test-only account rather than to a person: a key
145+
that reaches every org its owner belongs to must never be one a failing test can
146+
print.
147+
148+
The question worth asking of a diff is therefore: **does this print or commit a
149+
credential?**
150+
151+
- a new `t.Logf`, `fmt.Print` or error string that interpolates a key, a config
152+
file or a whole HTTP request
153+
- a test fixture or `.env`-shaped file added to the tree
154+
- a redaction helper whose pattern is narrower than the data it guards
155+
156+
That last one has a concrete edge here. **The CLI writes single-quoted TOML.**
157+
Viper's writer emits:
158+
159+
```toml
160+
profile = 'default'
161+
162+
[default]
163+
api_key = 'cli_secret_abc123'
164+
```
165+
166+
A redaction regex written against `api_key = "..."` — double quotes, as in the
167+
`pkg/config/testdata/*.toml` fixtures — matches nothing against a config the CLI
168+
actually wrote, and the key goes to the log verbatim. Check redaction patterns
169+
against real output, not against a fixture.
170+
171+
## Things that compile, pass, and are still broken
172+
173+
Three surfaces in this repository have no test or CI job guarding them, so a
174+
green run means nothing about them.
175+
176+
### `REFERENCE.md` is generated
177+
178+
It is produced by `tools/generate-reference`, and the generated regions are
179+
fenced by `<!-- GENERATE... -->` / `<!-- GENERATE_END -->` markers.
180+
181+
- `go run ./tools/generate-reference --check` regenerates to a temp file, diffs,
182+
and **exits 1 when `REFERENCE.md` is stale**
183+
- **no CI job runs it.** Nothing catches a command-surface change that leaves
184+
the file behind
185+
186+
So run it yourself when a diff touches commands, flags or examples. And flag a
187+
hand-edit *inside* a `GENERATE` block: it is not wrong today, it is deleted on
188+
the next regeneration, silently.
189+
190+
### The website is a second consumer
191+
192+
The same generator feeds the docs website, and it emits positional arguments
193+
only when `Annotations["cli.arguments"]` is set on the command (`AGENTS.md`
194+
§ Cobra Example and output for website docs). A new command that takes
195+
positional arguments without that annotation documents them nowhere.
196+
197+
The review consequence: a command-surface change is not finished inside this
198+
repository. Say so, so the docs change is not discovered after release.
199+
200+
### `CLITelemetry` JSON tags are a wire contract
201+
202+
`pkg/hookdeck/telemetry.go` defines `CLITelemetry`, sent on every API request
203+
in the `X-Hookdeck-CLI-Telemetry` header. Its `json:"..."` tags —
204+
`command_path`, `invocation_id`, `command_flags`, `mcp_client` and the rest —
205+
are field names that Hookdeck's usage analytics query by name.
206+
207+
Renaming a field compiles, passes unit tests, passes acceptance, and silently
208+
empties the dashboards built on the old name. Treat a change to those tags as a
209+
breaking change to a consumer outside this repository, and say so explicitly.
210+
Adding a field is safe; renaming or removing one is not.
211+
212+
## Docs that move with the code
213+
214+
A change to a command's surface is incomplete without:
215+
216+
- **`REFERENCE.md`** — regenerated, per above
217+
- **Cobra `Short` and `Long`**, via the shared helpers in `pkg/cmd/helptext.go`
218+
rather than per-command literals (`AGENTS.md` § Command help text)
219+
- **`KNOWN_ISSUES.md`**, when the change *fixes* user-visible behaviour. The
220+
file documents each issue by symptom, cause, affected versions and
221+
recommended fix. A fix that ships while its entry still says "Affected:
222+
v2.5.0 and earlier" with no resolution tells users to keep working around
223+
something that no longer exists.
224+
225+
## What not to raise
226+
227+
- **A `CHANGELOG.md` entry.** The file states plainly that it is no longer
228+
maintained; release notes come from the GitHub Releases page and are drafted
229+
at release time. Asking for an entry asks for something the repository
230+
deliberately stopped doing.
231+
- **The `version` field in `package.json`.** It is machine-written:
232+
`release.yml` sets it from the tag and commits it during `publish-npm`. It is
233+
expected to lag the tag in the tree, and a human editing it by hand is noise,
234+
not a finding.
235+
- **Style the repository does not enforce**, or a refactor nobody asked for. A
236+
review that returns twenty comments buries the one that mattered.

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -596,6 +596,7 @@ Summary for code and docs work:
596596
- **Never copy a skill into a second location.** `.agents/skills/` and a root `skills/` briefly held byte-identical copies, so a fix applied to one left the other wrong (see #336). One canonical directory, symlinks everywhere else.
597597
- **Windows:** Git must create symlinks correctly (`core.symlinks` / Developer Mode). If symlinks are missing after clone, recreate them (`mklink /D` on Windows, or copy `.agents/skills/` into `.cursor/skills` and `.claude/skills` as a fallback).
598598
- **Releases:** For cutting GitHub releases, tags, npm/beta publish flow, and drafting release notes, use **`.agents/skills/hookdeck-cli-release/SKILL.md`**; human-facing steps remain in **README.md § Releasing**.
599+
- **Reviewing:** For reviewing a pull request, diff or branch, use **`.agents/skills/hookdeck-cli-review/SKILL.md`**: why a green pull request still needs its acceptance run read, where a new acceptance-test build tag has to be registered, which generated files and wire contracts break silently, which docs move with a command's surface, and what not to raise.
599600
600601
---
601602

0 commit comments

Comments
 (0)