|
| 1 | +--- |
| 2 | +name: hookdeck-cli-review |
| 3 | +description: >- |
| 4 | + Reviews changes to the Hookdeck CLI: why a green pull request still needs its |
| 5 | + acceptance run read, where an acceptance-test build tag must be registered, |
| 6 | + which generated files and wire contracts break silently, which docs move with |
| 7 | + the code, and what not to raise. Use when reviewing a pull request, a diff or |
| 8 | + a branch in hookdeck-cli, or when deciding whether a change is safe to merge. |
| 9 | +--- |
| 10 | + |
| 11 | +# Reviewing changes to the Hookdeck CLI |
| 12 | + |
| 13 | +`AGENTS.md` is the reference for building, testing and linting this repository, |
| 14 | +and for the rules it already states. Read it first. This file is the review |
| 15 | +delta: what to look at in a diff, and the traps that a green pull request |
| 16 | +hides. Where a rule already lives in `AGENTS.md`, this points at it rather than |
| 17 | +restating it. |
| 18 | + |
| 19 | +## Acceptance runs on the pull request, but nothing enforces it |
| 20 | + |
| 21 | +`.github/workflows/test-acceptance.yml` triggers on every `pull_request` |
| 22 | +targeting `main` or `next`, with no path filter. So the acceptance suite — the |
| 23 | +three matrix slices plus the separate telemetry job — **does run on the pull |
| 24 | +request in front of you**. Assuming it did not is the wrong call. |
| 25 | + |
| 26 | +What it is not is *required*. Branch protection on `main` requires exactly: |
| 27 | + |
| 28 | +``` |
| 29 | +unit-test build-linux build-mac build-windows |
| 30 | +``` |
| 31 | + |
| 32 | +Acceptance is not in that list. A pull request with a **red or skipped** |
| 33 | +acceptance run is still mergeable, and the merge button will not tell you. |
| 34 | + |
| 35 | +**So read `gh pr checks` and flag a red or skipped acceptance run**, because |
| 36 | +nothing else in the process will. That is the single most useful thing this |
| 37 | +review does that the required checks do not. |
| 38 | + |
| 39 | +### The one case where it genuinely does not run |
| 40 | + |
| 41 | +`pull_request` does not fire when a pull request's head branch is updated by |
| 42 | +**merging another pull request into it**. `test-acceptance.yml` states this in |
| 43 | +a comment above its triggers. A stacked branch — a release branch that collects |
| 44 | +other pull requests — can therefore reach `main` having never run acceptance at |
| 45 | +all, and `gh pr checks` shows that as an absence rather than as a failure. |
| 46 | + |
| 47 | +The remedy is `workflow_dispatch`, which the same workflow accepts against any |
| 48 | +ref. One catch, also in that comment: GitHub reads the workflow definition from |
| 49 | +the ref you dispatch, so a branch can only be dispatched once it already |
| 50 | +carries the `workflow_dispatch` block. A branch cut before that block landed |
| 51 | +has to be rebased first. |
| 52 | + |
| 53 | +Dispatch it on its own, though. `test/acceptance/README.md` § Rate limits |
| 54 | +explains why: a push to the branch also triggers the `pull_request` run, and |
| 55 | +two concurrent runs against the same projects exhaust the API rate limit and |
| 56 | +fail at the timeout with no assertion failures at all — which reads exactly |
| 57 | +like a code failure. |
| 58 | + |
| 59 | +So "acceptance never ran" is a finding with an action attached, not just an |
| 60 | +observation. |
| 61 | + |
| 62 | +### Why a reviewer should care, not just the release engineer |
| 63 | + |
| 64 | +`.github/workflows/release.yml` calls the same acceptance workflow as a gate: |
| 65 | +`build-mac`, `build-linux` and `build-windows` each declare |
| 66 | +`needs: [acceptance]`, and `publish-npm` needs those three builds. A red |
| 67 | +acceptance therefore blocks the whole release pipeline. |
| 68 | + |
| 69 | +Merging a red acceptance does not fail loudly at merge time. It fails at the |
| 70 | +next release tag, on someone else's change, and the bisect starts from the |
| 71 | +wrong commit. |
| 72 | + |
| 73 | +### Running it yourself |
| 74 | + |
| 75 | +`go test ./...` does not include the acceptance suite — see `AGENTS.md` |
| 76 | +§ Running `go test` after code changes. A local run needs real credentials in |
| 77 | +`test/acceptance/.env`, which is gitignored. Without them, "not verified" is the |
| 78 | +honest report, not a pass. |
| 79 | + |
| 80 | +## A new acceptance tag has to be registered in five places |
| 81 | + |
| 82 | +Every `*_test.go` under `test/acceptance/` carries exactly one feature build |
| 83 | +tag, and untagged files leak into every tag build — `AGENTS.md` § Acceptance |
| 84 | +tests and feature tags states that rule; do not repeat it in a review, check it. |
| 85 | + |
| 86 | +`test/acceptance/README.md` § Parallelisation covers the rest of the mechanism: |
| 87 | +which tags are in which slice, that each slice has its own test account, and |
| 88 | +that telemetry runs separately. Read it rather than relying on this file. |
| 89 | + |
| 90 | +What no file states is that the same set of tags is maintained **by hand in |
| 91 | +five places**, and they drift independently: |
| 92 | + |
| 93 | +- `test/acceptance/run_parallel.sh` — `SLICE0_TAGS`, `SLICE1_TAGS`, |
| 94 | + `SLICE2_TAGS`, which is how the suite runs locally |
| 95 | +- `.github/workflows/acceptance.yml` — the `matrix.include` `tags` strings, |
| 96 | + which is how it runs in CI |
| 97 | +- `test/acceptance/README.md` § Running Tests — the "Run all automated tests" |
| 98 | + command, a single `-tags="..."` string that has to carry every tag |
| 99 | +- `test/acceptance/README.md` § Running Tests — the three per-slice commands |
| 100 | +- `test/acceptance/README.md` § Parallelisation — the `Slice N features` bullets |
| 101 | + |
| 102 | +Registered in CI but not in the script, the tag never runs locally. In the |
| 103 | +script but not in CI, it never runs on a pull request. Left out of the "run all" |
| 104 | +command, the documented way to run everything quietly stops running everything — |
| 105 | +and that one is the worst of the five, because the command still exits 0. |
| 106 | + |
| 107 | +All five agree today, on 21 distinct tags. Check all five whenever a diff adds |
| 108 | +one. |
| 109 | + |
| 110 | +One thing the README gets wrong, before you trust it on this: it attributes the |
| 111 | +matrix to `.github/workflows/test-acceptance.yml`. That file only declares the |
| 112 | +triggers and calls `.github/workflows/acceptance.yml`, which is where the slices |
| 113 | +and their tags actually live. |
| 114 | + |
| 115 | +### The `manual` tag is a deliberate exception |
| 116 | + |
| 117 | +`test/acceptance/project_use_manual_test.go` is `//go:build manual`, and |
| 118 | +`manual` appears in **none** of those five lists. That is intentional: the test |
| 119 | +needs an interactive browser login, which no CI job can perform, and |
| 120 | +`test/acceptance/README.md` gives it its own `-tags=manual` command instead. |
| 121 | +Applying the registration rule literally will report it as drift. It is not. |
| 122 | + |
| 123 | +### Which slice a tag lands in is a decision, not bookkeeping |
| 124 | + |
| 125 | +The README describes the slices; what it does not say is that choosing between |
| 126 | +them is a judgement call. They are not interchangeable buckets: |
| 127 | + |
| 128 | +- a different test account per slice means a test that assumes resources |
| 129 | + created by another slice will not find them |
| 130 | +- slice 0 already carries 12 tags against a 12-minute timeout, while slice 1 |
| 131 | + carries 2 |
| 132 | + |
| 133 | +Adding a slow tag to slice 0 is how the suite starts timing out. If a diff adds |
| 134 | +a tag, the slice it went into is worth a sentence. |
| 135 | + |
| 136 | +The telemetry job is separate for a reason worth checking against a diff: it |
| 137 | +forces `HOOKDECK_CLI_TELEMETRY_DISABLED=0`, where the matrix jobs set it to `1`. |
| 138 | +A telemetry-tagged test added to a slice will not behave as intended. |
| 139 | + |
| 140 | +## Credentials, and logs anyone can read |
| 141 | + |
| 142 | +This repository is public, so **every acceptance log is world-readable**. |
| 143 | +`.github/workflows/acceptance.yml` says so itself, which is why the test |
| 144 | +credentials have to belong to a test-only account rather than to a person: a key |
| 145 | +that reaches every org its owner belongs to must never be one a failing test can |
| 146 | +print. |
| 147 | + |
| 148 | +The question worth asking of a diff is therefore: **does this print or commit a |
| 149 | +credential?** |
| 150 | + |
| 151 | +- a new `t.Logf`, `fmt.Print` or error string that interpolates a key, a config |
| 152 | + file or a whole HTTP request |
| 153 | +- a test fixture or `.env`-shaped file added to the tree |
| 154 | +- a redaction helper whose pattern is narrower than the data it guards |
| 155 | + |
| 156 | +That last one has a concrete edge here. **The CLI writes single-quoted TOML.** |
| 157 | +Viper's writer emits: |
| 158 | + |
| 159 | +```toml |
| 160 | +profile = 'default' |
| 161 | + |
| 162 | +[default] |
| 163 | + api_key = 'cli_secret_abc123' |
| 164 | +``` |
| 165 | + |
| 166 | +A redaction regex written against `api_key = "..."` — double quotes, as in the |
| 167 | +`pkg/config/testdata/*.toml` fixtures — matches nothing against a config the CLI |
| 168 | +actually wrote, and the key goes to the log verbatim. Check redaction patterns |
| 169 | +against real output, not against a fixture. |
| 170 | + |
| 171 | +## Things that compile, pass, and are still broken |
| 172 | + |
| 173 | +Three surfaces in this repository have no test or CI job guarding them, so a |
| 174 | +green run means nothing about them. |
| 175 | + |
| 176 | +### `REFERENCE.md` is generated |
| 177 | + |
| 178 | +It is produced by `tools/generate-reference`, and the generated regions are |
| 179 | +fenced by `<!-- GENERATE... -->` / `<!-- GENERATE_END -->` markers. |
| 180 | + |
| 181 | +- `go run ./tools/generate-reference --check` regenerates to a temp file, diffs, |
| 182 | + and **exits 1 when `REFERENCE.md` is stale** |
| 183 | +- **no CI job runs it.** Nothing catches a command-surface change that leaves |
| 184 | + the file behind |
| 185 | + |
| 186 | +So run it yourself when a diff touches commands, flags or examples. And flag a |
| 187 | +hand-edit *inside* a `GENERATE` block: it is not wrong today, it is deleted on |
| 188 | +the next regeneration, silently. |
| 189 | + |
| 190 | +### The website is a second consumer |
| 191 | + |
| 192 | +The same generator feeds the docs website, and it emits positional arguments |
| 193 | +only when `Annotations["cli.arguments"]` is set on the command (`AGENTS.md` |
| 194 | +§ Cobra Example and output for website docs). A new command that takes |
| 195 | +positional arguments without that annotation documents them nowhere. |
| 196 | + |
| 197 | +The review consequence: a command-surface change is not finished inside this |
| 198 | +repository. Say so, so the docs change is not discovered after release. |
| 199 | + |
| 200 | +### `CLITelemetry` JSON tags are a wire contract |
| 201 | + |
| 202 | +`pkg/hookdeck/telemetry.go` defines `CLITelemetry`, sent on every API request |
| 203 | +in the `X-Hookdeck-CLI-Telemetry` header. Its `json:"..."` tags — |
| 204 | +`command_path`, `invocation_id`, `command_flags`, `mcp_client` and the rest — |
| 205 | +are field names that Hookdeck's usage analytics query by name. |
| 206 | + |
| 207 | +Renaming a field compiles, passes unit tests, passes acceptance, and silently |
| 208 | +empties the dashboards built on the old name. Treat a change to those tags as a |
| 209 | +breaking change to a consumer outside this repository, and say so explicitly. |
| 210 | +Adding a field is safe; renaming or removing one is not. |
| 211 | + |
| 212 | +## Docs that move with the code |
| 213 | + |
| 214 | +A change to a command's surface is incomplete without: |
| 215 | + |
| 216 | +- **`REFERENCE.md`** — regenerated, per above |
| 217 | +- **Cobra `Short` and `Long`**, via the shared helpers in `pkg/cmd/helptext.go` |
| 218 | + rather than per-command literals (`AGENTS.md` § Command help text) |
| 219 | +- **`KNOWN_ISSUES.md`**, when the change *fixes* user-visible behaviour. The |
| 220 | + file documents each issue by symptom, cause, affected versions and |
| 221 | + recommended fix. A fix that ships while its entry still says "Affected: |
| 222 | + v2.5.0 and earlier" with no resolution tells users to keep working around |
| 223 | + something that no longer exists. |
| 224 | + |
| 225 | +## What not to raise |
| 226 | + |
| 227 | +- **A `CHANGELOG.md` entry.** The file states plainly that it is no longer |
| 228 | + maintained; release notes come from the GitHub Releases page and are drafted |
| 229 | + at release time. Asking for an entry asks for something the repository |
| 230 | + deliberately stopped doing. |
| 231 | +- **The `version` field in `package.json`.** It is machine-written: |
| 232 | + `release.yml` sets it from the tag and commits it during `publish-npm`. It is |
| 233 | + expected to lag the tag in the tree, and a human editing it by hand is noise, |
| 234 | + not a finding. |
| 235 | +- **Style the repository does not enforce**, or a refactor nobody asked for. A |
| 236 | + review that returns twenty comments buries the one that mattered. |
0 commit comments