Skip to content

Commit 5c89d50

Browse files
authored
fix(ci): make the PR gate trustworthy and replay recorded model traffic (#1567)
* fix(ci): make the PR gate trustworthy and replay recorded model traffic Most red runs came from the gate, not the pull requests: - teamPlanRuntime.supervisor: the lead fixture shared the members' fake upstream and reply queue, so its late request became "the last request" or took a member's scripted reply. Requests are now attributed per agent, and the lead's call is delayed on every run to keep that timing covered. - parentProcessGuard: on dash a poll caught between fork and exec swallowed TERM, so about 1 in 1000 short commands waited out a full second. The trap now stops the poll with SIGKILL. - Regressions merged red (#1554 i18n tests, the video gallery mock) and a RawRequestTab race under coverage. - The full sweep never installed ripgrep, so agent creation returned 500. - CODEOWNER-authored PRs no longer need allow-cli-core-change (owners read from the base branch); coverage reports a failed suite's lines as not measured instead of 0% and ignores __fixtures__; failed server test files are re-run once to label them FLAKY or FAILED, still failing the gate. - release.ts commits on release/v<ver> for a PR and tags main with --tag. check:agent-e2e now also replays DeepSeek Flash traffic recorded into scripts/quality-gate/replay/cassettes: thinking, fragmented tool input, parallel tools, tool errors, large output, subagents, MCP, images, an agent team, auto and manual compaction, interrupts, and the OpenAI Chat proxy. The replay matches each agent loop's requests structurally and fails with a diff. quality:record-cassettes re-records locally; it never runs in CI. The desktop UI smoke gains a case that drives the real CLI against a recording. * test(compact): drop an unused import that fails check:policy * test(release): give the fixture repo its own git identity * fix(quality-gate): never let the replay lane outlive its verdict The agent-e2e job passed every scenario, then hung until its 20 minute timeout: the replay process did not exit on CI (it does locally and in a Linux container). Exit with the verdict 15s after finishing if something still holds the process, and log what is still running. Tool schemas carry the platform, so a macOS recording drifted on every Linux run (81 notes); compare the tool list and token budget only.
1 parent 0894dc8 commit 5c89d50

65 files changed

Lines changed: 23171 additions & 129 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/AGENTS.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,12 @@ These rules apply to `.github/` changes in addition to the root instructions.
55
- Treat workflow changes as product changes. Run `bun run check:policy`; run `actionlint` when available.
66
- `scripts/pr/change-policy.ts` is the source of truth for path-to-check routing. Do not duplicate the routing graph in advisory automation.
77
- Routing is path prefixes plus the import graph from `scripts/pr/module-graph.ts`. Prefixes decide areas and every blocking rule; the graph only widens which surface checks run. When the graph cannot be built the run selects every surface rather than silently falling back to prefixes.
8-
- `nightly-quality.yml` runs every deterministic lane unconditionally and re-proves the module graph. It exists because per-PR selection can only ever cover what a diff reaches; do not move its jobs into the required PR gate. It is `workflow_dispatch` only — when to spend ~90 minutes of CI is the maintainer's call, and `pr-quality-workflow.test.ts` fails if a `schedule:` is added back.
8+
- `nightly-quality.yml` runs every deterministic lane unconditionally and re-proves the module graph. It exists because per-PR selection can only ever cover what a diff reaches; do not move its jobs into the required PR gate. It runs daily at 19:00 UTC and on `workflow_dispatch`; the maintainer chose that cadence, `pr-quality-workflow.test.ts` pins it, and changing it is the maintainer's call. Every job that runs runtime suites must install ripgrep first (also pinned there): agent discovery lists files through it.
99
- Keep `pr-quality-gate` as the stable required status. Selected jobs must succeed and unselected jobs must be explicitly skipped; never convert failures into success.
1010
- Required PR jobs must remain offline and must not receive provider credentials or depend on paid/live services.
1111
- A `pull_request_target` workflow may inspect PR metadata using trusted base code, but must never execute the PR head, install PR-controlled dependencies, or expose secrets to contributor code.
1212
- Keep Bun aligned with `package.json#packageManager` and use frozen lockfile installs in required jobs.
1313
- Do not weaken coverage, test routing, CODEOWNERS, or branch protections to make another change pass. Maintainer override labels require an explicit, documented decision.
14+
- A PR authored by an individual CODEOWNER needs no `allow-cli-core-change` label (maintainer decision, 2026-10). Both `scope-plan` and `pr-triage.yml` read CODEOWNERS from the base branch, never the PR head. The other override labels are unchanged.
15+
- Never retry a failed test into a pass. `check:server` re-runs each failed file once to label it `FLAKY` or `FAILED`, and still fails the gate either way; fix the race, or quarantine with a reason and owner only as a last resort.
1416
- Repository settings, rulesets, secrets, and required-check changes require explicit user authorization; editing workflow files alone does not grant it.

‎.github/pull_request_template.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424

2525
## Risk
2626

27-
- [ ] This PR does not touch CLI core paths, or it has maintainer approval for `allow-cli-core-change`.
27+
- [ ] This PR does not touch CLI core paths, is authored by a CODEOWNER, or has maintainer approval for `allow-cli-core-change`.
2828
- [ ] Production code changes include matching tests, or have maintainer approval for `allow-missing-tests`.
2929
- [ ] Coverage baseline/threshold changes have maintainer approval for `allow-coverage-baseline-change`.
3030
- [ ] Any quarantine-policy change has maintainer approval; deterministic provider/chat contract tests were not quarantined.

‎.github/workflows/nightly-quality.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ jobs:
4242
- uses: oven-sh/setup-bun@v2
4343
with:
4444
bun-version-file: package.json
45+
# Agent and skill discovery lists files through ripgrep; without it the
46+
# runtime suites see every created agent as missing (HTTP 500).
47+
- name: Install ripgrep
48+
run: sudo apt-get update && sudo apt-get install -y ripgrep
4549
- name: Install root dependencies
4650
run: bun install --frozen-lockfile
4751
- name: Install desktop dependencies
@@ -57,7 +61,7 @@ jobs:
5761
run: bun run check:policy
5862
- name: Deterministic agent flow
5963
run: bun run check:agent-flow
60-
- name: Agent loop against a mock LLM
64+
- name: Agent loop against a mock LLM and recorded traffic
6165
run: bun run check:agent-e2e
6266
- name: Root runtime tests
6367
run: bun run check:server

‎.github/workflows/pr-quality.yml‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,10 +58,17 @@ jobs:
5858
REPOSITORY: ${{ github.repository }}
5959
PUSH_BEFORE: ${{ github.event.before }}
6060
PUSH_AFTER: ${{ github.sha }}
61+
BASE_REF: ${{ github.base_ref }}
6162
run: |
63+
: > author.txt
64+
: > codeowners.txt
6265
if [ "$EVENT_NAME" = "pull_request" ]; then
6366
gh api "repos/${REPOSITORY}/pulls/${PR_NUMBER}/files" --paginate --jq '.[].filename' > changed-files.txt
6467
jq -r '.pull_request.labels[].name' "$GITHUB_EVENT_PATH" > labels.txt
68+
# A CODEOWNER's own PR needs no maintainer-approval label. Owners come
69+
# from the base branch, so a PR cannot add its author to the list.
70+
jq -r '.pull_request.user.login' "$GITHUB_EVENT_PATH" > author.txt
71+
git show "origin/${BASE_REF}:.github/CODEOWNERS" > codeowners.txt 2>/dev/null || : > codeowners.txt
6572
else
6673
# A push to main: scope the run to what the push brought in. A new branch
6774
# or a force push has no usable "before", so fall back to the last commit.
@@ -75,7 +82,7 @@ jobs:
7582
7683
- name: Build deterministic scope plan
7784
id: policy
78-
run: bun run scripts/pr/change-policy.ts --files changed-files.txt --labels-file labels.txt --plan-only
85+
run: bun run scripts/pr/change-policy.ts --files changed-files.txt --labels-file labels.txt --author-file author.txt --codeowners-file codeowners.txt --plan-only
7986

8087
policy-enforcement:
8188
name: policy-enforcement
@@ -247,7 +254,7 @@ jobs:
247254
bun-version-file: package.json
248255
- name: Install root dependencies
249256
run: bun install --frozen-lockfile
250-
- name: Run the agent loop against a mock LLM
257+
- name: Run the agent loop against a mock LLM and recorded traffic
251258
run: bun run check:agent-e2e
252259
- name: Upload agent e2e artifacts
253260
if: always()

‎.github/workflows/pr-triage.yml‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,25 @@ jobs:
8888
}
8989
}
9090
91-
if (cliCoreFiles.length > 0 && !currentLabels.has('allow-cli-core-change')) {
91+
// A CODEOWNER's own PR is already maintainer-approved; owners come from the
92+
// base branch so the PR cannot add its author. Mirrors scripts/pr/change-policy.ts.
93+
let codeOwners = new Set()
94+
try {
95+
const { data } = await github.rest.repos.getContent({ owner, repo, path: '.github/CODEOWNERS', ref: context.payload.pull_request.base.ref })
96+
const text = Buffer.from(data.content || '', 'base64').toString('utf8')
97+
codeOwners = new Set(text.split(/\r?\n/)
98+
.map((line) => line.replace(/#.*$/, '').trim())
99+
.filter(Boolean)
100+
.flatMap((line) => line.split(/\s+/).slice(1))
101+
.filter((token) => /^@[A-Za-z0-9-]+$/.test(token))
102+
.map((token) => token.slice(1).toLowerCase()))
103+
} catch (error) {
104+
core.warning(`Could not read base CODEOWNERS: ${error.message}`)
105+
}
106+
const authorIsCodeOwner = codeOwners.has(String(context.payload.pull_request.user?.login || '').toLowerCase())
107+
const cliCoreApproved = currentLabels.has('allow-cli-core-change') || authorIsCodeOwner
108+
109+
if (cliCoreFiles.length > 0 && !cliCoreApproved) {
92110
areas.add('needs-maintainer-approval')
93111
}
94112
@@ -221,9 +239,11 @@ jobs:
221239
222240
const marker = '<!-- pr-quality-triage -->'
223241
const areaText = [...areas].filter((label) => label.startsWith('area:')).sort().join(', ') || 'none'
224-
const approvalText = cliCoreFiles.length > 0 && !currentLabels.has('allow-cli-core-change')
242+
const approvalText = cliCoreFiles.length > 0 && !cliCoreApproved
225243
? 'Blocked by policy until a maintainer applies `allow-cli-core-change` and approves the PR.'
226-
: 'No CLI-core policy block detected.'
244+
: cliCoreFiles.length > 0 && authorIsCodeOwner && !currentLabels.has('allow-cli-core-change')
245+
? 'CLI-core change authored by a CODEOWNER; no override label needed.'
246+
: 'No CLI-core policy block detected.'
227247
const missingTestText = missingTestSignals.length > 0 && !currentLabels.has('allow-missing-tests')
228248
? 'Blocked by policy until a maintainer applies `allow-missing-tests` or matching tests are added.'
229249
: 'No missing-test policy block detected.'

‎desktop/src/__tests__/untranslatedText.test.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,6 @@ const NON_PROSE_ELEMENTS = new Set(['kbd', 'Keycap', 'code', 'pre', 'style', 'sc
4040
*/
4141
const LANGUAGE_NEUTRAL: Record<string, string[]> = {
4242
'components/agentTeams/AgentTeamsCanvas.tsx': ['zZ'], // a sleeping agent's glyph
43-
'components/agentTeams/AgentTeamsWorkbench.tsx': ['Agent Teams ·'], // product name, kept in every locale
4443
'components/chat/MermaidRenderer.tsx': ['mermaid'], // the diagram language, as a code block labels it
4544
'components/chat/ToolCallBlock.tsx': ['bash', 'powershell'], // shell names, as a code block labels them
4645
'components/layout/H5ConnectionView.tsx': ['https://chat.example.com'], // example URL

‎desktop/src/components/chat/InlineVideoGallery.test.tsx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ vi.mock('../../lib/desktopRuntime', () => ({
77
getServerBaseUrl: () => 'http://127.0.0.1:4321',
88
}))
99

10-
const openPreviewLink = vi.hoisted(() => vi.fn())
10+
// The router reports whether it could place the link; a card the router places opens.
11+
const openPreviewLink = vi.hoisted(() => vi.fn(() => true))
1112
vi.mock('../../lib/openPreviewLink', () => ({ openPreviewLink }))
1213
vi.mock('../../i18n', () => ({ useTranslation: () => (key: string) => key }))
1314

‎desktop/src/components/trajectory/detail/RawRequestTab.test.tsx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,9 +95,10 @@ describe('RawRequestTab', () => {
9595
render(<RawRequestTab sessionId="s1" row={ROW} />)
9696

9797
expect(await screen.findByText(/Closest captured call by time/)).toBeInTheDocument()
98+
// The detail fetches the call in its own effect, so only its rendered body proves the request was made.
99+
expect(await screen.findByText('Response from the model')).toBeInTheDocument()
98100
// The record locator reaches calls beyond the capture's indexed window.
99101
expect(getTraceCall).toHaveBeenCalledWith('s1', 'match', '9000-12000')
100-
expect(await screen.findByText('Response from the model')).toBeInTheDocument()
101102
// Where the call went and with which model sit at the top.
102103
expect(screen.getByTestId('trajectory-request-endpoint')).toHaveTextContent(/POST\s*https:\/\/api\/v1\/messages/)
103104
expect(screen.getByTestId('trajectory-request-model')).toHaveTextContent('claude-x')

‎desktop/src/pages/settings/GeneralSettingsSubagents.test.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ describe('General settings ordinary SubAgent concurrency', () => {
4040
expect(input).toHaveAttribute('inputmode', 'numeric')
4141
expect(input).toHaveAttribute('placeholder', '不限(默认)')
4242
expect(screen.getByText(/限制同一会话的普通前台和后台 SubAgent/)).toHaveTextContent('达到上限时拒绝新启动,等已有任务完成后再启动')
43-
expect(screen.getByText(/主 Agent、Agent Teams、Workflow 不计入/)).toHaveTextContent('运行中的会话也会生效,无需重启')
43+
expect(screen.getByText(/主 Agent、智能体团队、Workflow 不计入/)).toHaveTextContent('运行中的会话也会生效,无需重启')
4444

4545
fireEvent.change(input, { target: { value: '37' } })
4646
expect(settingsApi.updateUser).not.toHaveBeenCalled()

0 commit comments

Comments
 (0)