Skip to content

Commit 59d9815

Browse files
test(scenarios): give the external-driver crm run a measured 30s budget and a lighter stand-in (#31284)
Fixes #31281 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:`@objectstack/scenarios` 的外部驱动端到端用例改为显式 30 秒超时,并在用例旁写明实测数据;测试用的替身命令不再加载 `node:crypto`,key 指纹从 SHA-256 改为 FNV-1a。只改测试,不动 `src/`。 - 为什么改:这个用例每个回合启动一次替身进程,连续 45 次。本地 2.2–2.5 秒,负载高的 Test Core 分片上被 vitest 默认 5 秒超时在 5042ms 截断,把 main 合并队列里别人的 PR 弹出去(#31247 两次)。 - 风险与回滚:只改测试文件与其替身,不发布(skip-changeset)。回滚即 revert。 - 先看:`packages/qa/scenarios/test/runner/external-driver.test.ts` 末尾的超时和注释。 - 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run test/runner/external-driver.test.ts`,预期 8/8 通过。 - 需要你做的:无,席位落地。 ## What changes - `test/runner/external-driver.test.ts`, the case "a full crm run through the command…": an explicit 30 000 ms budget instead of vitest's 5 s default, with the measurement written beside it. - `test/runner/fixtures/fake-external-driver.mjs`: the stand-in no longer imports `node:crypto`. It fingerprints the key it read with FNV-1a (32-bit hex) as `keyPrint`. The test computes the same function and now also asserts that this run's keys have distinct fingerprints, so the membership check cannot pass by collision. **Root cause.** The case starts the stand-in once per turn, one after another: 45 starts (3 author calls; 42 asks over scratch run, scratch iterate-run and template iterate-run, measured from the call log). Each start costs about 50 ms locally, and node's own start-up is about 35 ms of that. On a Test Core shard one start took 212 ms (the `info` case in the failing job), and this case was cut off at 5042 ms. Both modes are kept: each mode boots its own app and issues its own keys. ## Tests - Per-start cost, 30 starts each, on this container: bare node about 35 ms; with `node:fs`/`node:path` about 39 ms; with `node:crypto` added 41 to 45 ms. That is the stand-in's start-up before and after. - The case, 5 local runs each: before 2374 to 2520 ms; after 2186 to 2425 ms. - `pnpm --filter @objectstack/scenarios exec vitest run test/runner/external-driver.test.ts`: Test Files 1 passed (1), Tests 8 passed (8). ## Acceptance notes - CI evidence: merge-queue run 42414930416, job Test Core (2/8). The case failed with `Test timed out in 5000ms` (5042 ms). The coordinator reported the same on #31256 and #31277, and #31247 was dequeued twice (runs 42414930416 and 42417438743). - In the same job, `test/runner/codex-driver.test.ts` took 7657 ms for 12 cases. No case failed, and per-case times were not printed for it. Not touched here (outside the claim's surface). ## Verification On `31dffb8d`: `pnpm --filter @objectstack/scenarios typecheck` exit 0; scoped lint `node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios/test/runner --no-inline-config` exit 0; `node scripts/check-nul-bytes.mjs` exit 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01FEkQ5mx64pxVGCbKfuxEwi Co-authored-by: Claude <noreply@anthropic.com>
1 parent ea053b4 commit 59d9815

2 files changed

Lines changed: 33 additions & 11 deletions

File tree

‎packages/qa/scenarios/test/runner/external-driver.test.ts‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
* in a 0600 file only, and no written artifact carrying it.
1010
*/
1111

12-
import { createHash } from 'node:crypto';
1312
import { existsSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
1413
import { join, dirname, resolve } from 'node:path';
1514
import { fileURLToPath } from 'node:url';
@@ -31,7 +30,12 @@ const COMMAND = `${process.execPath} ${FAKE}`;
3130

3231
/** A key no other text in a run could hold by chance. */
3332
const KEY = 'sk-scenario-e2e-key-9f27c1d4';
34-
const sha = (s: string) => createHash('sha256').update(s).digest('hex');
33+
/** The stand-in's fingerprint of the key it read (FNV-1a, 32-bit, hex): enough to tell this run's keys apart without logging one. */
34+
const keyPrint = (s: string) => {
35+
let h = 0x811c9dc5;
36+
for (let i = 0; i < s.length; i++) h = Math.imul(h ^ s.charCodeAt(i), 0x01000193) >>> 0;
37+
return h.toString(16).padStart(8, '0');
38+
};
3539

3640
const author = (prompt: string, cwd = tempDir('ext-project')): AuthorTask => ({ card: 'c', mode: 'scratch', phase: 'build', step: 'author', cwd, prompt, tier: 'starter' });
3741
const ask = (prompt: string): AskTask => ({ card: 'c', mode: 'scratch', phase: 'run', step: 'q1', persona: 'sdr', prompt, mcp: { url: 'http://127.0.0.1:9/api/v1/mcp', apiKey: KEY } });
@@ -85,7 +89,7 @@ describe('the external driver contract — one process per call, JSON in on stdi
8589
const { input, argv, seen } = echoed(turn);
8690
expect(input).toMatchObject({ contract: EXTERNAL_DRIVER_CONTRACT, kind: 'ask', step: { card: 'c', mode: 'scratch', phase: 'run', step: 'q1', persona: 'sdr' }, prompt: 'ECHO how many accounts?', limits: { timeoutMs: 5 * 60_000 } });
8791
expect(Object.keys(input.mcp as object).sort()).toEqual(['configFile', 'url']);
88-
expect(seen).toMatchObject({ fileMode: '600', urlMatches: true, runsInConfigDir: true, keySha: sha(KEY), keyOnArgv: false, keyInEnv: false, keyOnStdin: false });
92+
expect(seen).toMatchObject({ fileMode: '600', urlMatches: true, runsInConfigDir: true, keyPrint: keyPrint(KEY), keyOnArgv: false, keyInEnv: false, keyOnStdin: false });
8993
expect(argv.join(' ')).not.toContain(KEY);
9094
expect(existsSync(seen!.configFile as string)).toBe(false);
9195
expect(existsSync(dirname(seen!.configFile as string))).toBe(false);
@@ -185,10 +189,11 @@ describe('the persona keys, end to end over the external driver — the record,
185189
const calls = readFileSync(log, 'utf8').trim().split('\n').map((l) => JSON.parse(l) as { kind: string; seen?: Record<string, unknown> });
186190
const asks = calls.filter((c) => c.kind === 'ask');
187191
expect(asks.length).toBeGreaterThan(5);
188-
const keys = new Set([`${KEY}admin`, ...fx.scenario.card.run.personas.map((p) => `${KEY}${p.id}`)].map(sha));
192+
const keys = new Set([`${KEY}admin`, ...fx.scenario.card.run.personas.map((p) => `${KEY}${p.id}`)].map(keyPrint));
193+
expect(keys.size).toBe(fx.scenario.card.run.personas.length + 1);
189194
for (const a of asks) {
190195
expect(a.seen).toMatchObject({ fileMode: '600', keyOnArgv: false, keyInEnv: false, keyOnStdin: false });
191-
expect(keys.has(a.seen!.keySha as string)).toBe(true);
196+
expect(keys.has(a.seen!.keyPrint as string)).toBe(true);
192197
expect(existsSync(a.seen!.configFile as string)).toBe(false);
193198
}
194199

@@ -222,5 +227,12 @@ describe('the persona keys, end to end over the external driver — the record,
222227
const page = readFileSync(join(out, 'apps', fx.scenario.card.id, 'README.md'), 'utf8');
223228
expect(page).toContain('- Driver: external:fake-agent 1.2.3 (re-graded from a recording of that run)\n');
224229
expect(page).not.toContain(runs);
225-
});
230+
// The budget is measured, not vitest's 5 s default. The run starts the
231+
// stand-in once per turn, one after another: 45 starts (3 author calls,
232+
// 42 asks over scratch run, scratch iterate-run and template iterate-run).
233+
// Measured 2.2–2.5 s over 10 local runs at ~50 ms a start. On a Test Core
234+
// shard one start took 212 ms (the info case) and this case was cut off
235+
// at 5042 ms (merge-queue run 42414930416). 30 s is ~6x the slowest
236+
// observed.
237+
}, 30_000);
226238
});

‎packages/qa/scenarios/test/runner/fixtures/fake-external-driver.mjs‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,16 @@
1515
* prints JSON of another shape, anything else echoes the call it read.
1616
* - With `FAKE_DRIVER_LOG=FILE`, every call appends one line: its kind, its
1717
* step and, for an ask, what it saw of the key — the config file's mode, a
18-
* SHA-256 of the key it read there, and whether the key was also on argv,
19-
* in the environment or on stdin.
18+
* fingerprint of the key it read there (FNV-1a, not the key), and whether
19+
* the key was also on argv, in the environment or on stdin.
20+
*
21+
* Every turn of a card run starts this file once, one after another, so its
22+
* start-up is the e2e case's cost: it loads `node:fs` and `node:path` only.
23+
* `node:crypto` added ~7 ms a start (41–45 ms against 35–39 ms for a bare
24+
* node on the box that measured it), and the fingerprint only has to tell this
25+
* run's few keys apart.
2026
*/
2127

22-
import { createHash } from 'node:crypto';
2328
import { appendFileSync, readFileSync, statSync } from 'node:fs';
2429
import { dirname, join } from 'node:path';
2530

@@ -29,7 +34,12 @@ process.on('SIGTERM', () => {
2934
});
3035

3136
const out = (v) => process.stdout.write(`${JSON.stringify(v)}\n`);
32-
const sha = (s) => createHash('sha256').update(s).digest('hex');
37+
/** FNV-1a (32-bit) as hex; `keyPrint` in external-driver.test.ts is the same function. */
38+
const keyPrint = (s) => {
39+
let h = 0x811c9dc5;
40+
for (let i = 0; i < s.length; i++) h = Math.imul(h ^ s.charCodeAt(i), 0x01000193) >>> 0;
41+
return h.toString(16).padStart(8, '0');
42+
};
3343

3444
let raw = '';
3545
process.stdin.on('data', (d) => (raw += d));
@@ -63,7 +73,7 @@ process.stdin.on('end', () => {
6373
fileMode: (statSync(file).mode & 0o777).toString(8),
6474
urlMatches: config.mcpServers.app.url === call.mcp.url,
6575
runsInConfigDir: process.cwd() === dirname(file),
66-
keySha: sha(key),
76+
keyPrint: keyPrint(key),
6777
keyOnArgv: process.argv.some((a) => a.includes(key)),
6878
keyInEnv: Object.values(process.env).some((v) => typeof v === 'string' && v.includes(key)),
6979
keyOnStdin: raw.includes(key),

0 commit comments

Comments
 (0)