Skip to content

Commit 249bffc

Browse files
committed
test(expo): give the skill an e2e agent on openai/gpt-6-luna-fast through the Vercel AI Gateway
The composed e2e config sets agents.default to e2e's built-in agent on gateway('openai/gpt-6-luna-fast') when the machine has a Vercel AI Gateway key. With no key the config is what it was, so a run without one behaves as before. VERIFY_AGENT_MODEL names another language model on the gateway for one command. The replay cache is off when an agent is configured, so the model acts and judges on the screen of each run. The key comes from AI_GATEWAY_API_KEY, or from the file that AI_GATEWAY_API_KEY_FILE names, which only its owner may read. The CLI removes both variables from its own environment when it starts and holds the key in a Secret, so the log redaction and the evidence scan know its value. Only the e2e process receives it. A key in a file is handed over as the path, and e2e reads the file. doctor prints an agent line with the model and the variable the key came from. Its e2e-pins check now covers the ai package, pinned at 7.0.128. No golden spec uses an agent step, and no workflow passes the key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # .claude/skills/verify-clerk-expo/src/core/MANIFEST # .claude/skills/verify-clerk-expo/src/core/types.ts
1 parent a41755e commit 249bffc

14 files changed

Lines changed: 524 additions & 20 deletions

File tree

‎.claude/skills/verify-clerk-expo/SKILL.md‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,11 +99,12 @@ Run it first, and again whenever anything looks off. Without `--live` it only re
9999
| Checks | Pass when |
100100
| ------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
101101
| `backend` | Always. The line says which backend `doctor` chose and why |
102-
| `node`, `e2e-pins` | Node is 24.8.0 or newer on 24, and the installed `e2e`, `@e2e-dev/mobile`, and `@e2e-dev/github` are the versions that `package.json` pins. `e2e-pins` fails until `npm ci` has run in this worktree |
102+
| `node`, `e2e-pins` | Node is 24.8.0 or newer on 24, and the installed `e2e`, `@e2e-dev/mobile`, `@e2e-dev/github`, and `ai` are the versions that `package.json` pins. `e2e-pins` fails until `npm ci` has run in this worktree |
103103
| `xcode`, `template`, `proxy-trust`, `lane-ports` | Local iOS: `xcodebuild` runs. `Clerk Verify Template iOS` exists and is shut down. It trusts a custom CA if macOS has a system HTTPS proxy. Every booted `verify-ios-<n>` simulator has a live claim |
104104
| `jdk`, `template`, `lane-ports` | Local Android: a Java 21 is found. The SDK's emulator and adb run. Ports 5560 to 5562 hold only lanes that the CLI booted |
105105
| `remote-env`, `git-fetch`, `git-head`, `git-push`, `github-rest`, `remote-commit`, `tunnel-egress`, `clerk-egress` | With the remote backend, in place of the local checks: `npm` is installed, git can fetch and push the branch, HEAD is at or ahead of the branch on GitHub, GitHub's REST API answers for the repository, GitHub has HEAD, and this machine reaches the tunnel host and Clerk. `remote-sessions` lists the sessions of this checkout that are still running |
106106
| `instances`, `clerk-api`, `settings` | A Platform API credential reaches the verification workspace. Clerk's Backend API accepts the secret key of the application this worktree holds. That application shows the settings recorded for it, and every declaration under `specs/` is well formed |
107+
| `agent` | No gateway key is set, or one is set and readable. The line names the model that [agent steps](#agent-steps) use, or says `none` |
107108
| `build` | A build of the fixture matches the current tree |
108109
| `gh-attach` | `gh pr comment` has `--attach`. A `gh` without it prints `warn` and not `FAIL`, because only `attach` needs it |
109110
| `stale-claims` | No lane is claimed by a worktree that no longer exists |
@@ -195,6 +196,48 @@ A failing spec prints `FAIL`, the first assertion message, and the path of its f
195196

196197
With `--retries`, a test that fails and then passes prints `flaky` and the error of its failed attempt, and a `flaky` line under the results counts such tests. `run.json` records the test as `flaky` with its `attempts`, never as passed, and `run` exits 0. The failure page of the failed attempt and the files of every attempt stay in the run directory. A test that fails on every attempt prints `FAIL`, and `run` exits 1.
197198

199+
### Agent steps
200+
201+
e2e can also act on the app and judge a screen with a model, through `agent.act(...)` and `agent.assert(...)` on the `agent` fixture. This skill configures one agent, named `default`, and only when the machine has a Vercel AI Gateway key. It is e2e's built-in agent on `openai/gpt-6-luna-fast`. With no key the config has no agent and every spec without an agent step runs as it always has. A spec that calls `agent` then stops its run with `MODEL_UNAVAILABLE`.
202+
203+
```ts
204+
import { test, expect } from '../fixtures.ts';
205+
206+
test('the agent judges the sign-in screen', async ({ host, screen, agent }) => {
207+
await host.launch({ screen: 'auth', authMode: 'signIn' });
208+
await expect(screen.getByTestId('clerk.auth.start.identifier')).toBeVisible({ timeout: 20_000 });
209+
await agent.assert('A sign-in screen is showing, with a field for an email address and a Continue button', {
210+
vision: 'only',
211+
});
212+
});
213+
```
214+
215+
Supply the key in one of two ways.
216+
217+
- Set `AI_GATEWAY_API_KEY_FILE` to a file that holds the key and that only you can read (mode 0600), for example `~/.verify/ai-gateway-key`. The CLI hands e2e the path and e2e reads the file, so no process has the key in its environment. Do not replace the file while a run is in progress.
218+
- Set `AI_GATEWAY_API_KEY` to the key. The CLI hands it to the e2e process in that process's environment, where the processes that e2e starts, such as `agent-device`, can read it too. Prefer the file.
219+
220+
The CLI removes both variables from its own environment when it starts, so the processes that the CLI itself starts, such as the build, `git`, and `gh`, never see them. It redacts the key from the logs it writes, and `attach` refuses a run whose files hold it. Never put the key on a command line, in a spec, or in a workflow file. `doctor` prints an `agent` line with the model and the variable the key came from, or `none` when no key is set. A key file that is missing, empty, or readable by other users fails that check, and `run` then fails with `KEYS_MISSING`.
221+
222+
`VERIFY_AGENT_MODEL=<provider/model>` names another language model on the gateway for one command. `openai/gpt-6-luna` is the same family at half the listed price, without the fast tier. The model must accept tool calls and images. The gateway refuses a decision model such as `openai/gpt-6-luna-decisions` for this agent, and the step fails with `MODEL_PROVIDER_FAILED`.
223+
224+
e2e's replay cache is off when an agent is configured, so the model acts and judges on the screen of each run. The model receives the text of the screen, and a screenshot when a step asks for one.
225+
226+
Use an agent step in two cases.
227+
228+
- An agent explores the app live, in a spec under `specs/explored/`.
229+
- A claim has no fixed locator that can check it, such as how a screen looks. `agent.assert(claim, { vision: true })` adds a screenshot to the screen text, and `vision: 'only'` judges the screenshot alone.
230+
231+
Committed golden specs assert on what a user sees with fixed steps. Keep an exact assertion after each agent step, so that a wrong verdict fails the spec. A golden spec with an agent step needs the gateway key in CI. The workflows do not pass it today, and one agent step without a model stops every test of its run, so do not commit one until they do. Do not put a password or a code in an instruction, because the instruction goes to the model provider and into the report. A spec whose agent types a code or a password is a `form-entry` spec.
232+
233+
An agent step costs money and time. These figures are for `openai/gpt-6-luna-fast`, measured with the clerk-ios skill on a local iOS simulator. e2e prints the tokens, the cost, and the model calls of each run in its summary.
234+
235+
| Measured | Result |
236+
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
237+
| 2026-10-06, one test with three `act` steps and one `assert`, 38 live runs | 38 passed. The median test cost 0.003 USD and took 41 seconds. The same flow with fixed steps took 13 seconds |
238+
| 2026-10-06, per step | An `act` took about 3 model calls and 0.001 USD. An `assert` took 1 call and 0.0002 USD |
239+
| 2026-10-07, three tests with one `act` and two `assert` steps, one of them from a screenshot alone, 8 runs | 24 of 24 tests passed. A run cost 0.0006 to 0.0018 USD in 4 model calls, and a test took 5.5 to 9.8 seconds |
240+
198241
## Evidence
199242

200243
Every `run` writes `.verify/runs/<run-id>/` and prints its path.

‎.claude/skills/verify-clerk-expo/package-lock.json‎

Lines changed: 133 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.claude/skills/verify-clerk-expo/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
"@e2e-dev/github": "0.4.0",
1111
"@e2e-dev/mobile": "0.10.0",
1212
"@types/node": "24.19.1",
13+
"ai": "7.0.128",
1314
"e2e": "0.18.0",
1415
"typescript": "7.0.2"
1516
},

0 commit comments

Comments
 (0)