Skip to content

Commit 3bd4bf6

Browse files
ci(ui): port the Test UI per-file timings reporter and its weekly dataset refresh (#29588)
<!-- claude-projects-attribution: {"github_login":"os-warren"} --> _Requested by **Warren** · [project thread](https://claude.ai/code/project/chan_01U5RdnAAmkSQmZbqh6FtqZ8?thread=cmsg_01U5RdnAAmkSQmZbqh6FtqZ8AE4cn5ty8rrc9mVmocckfL)_ Fixes #29517 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:`ci.yml` 的 Test UI 全量腿加挂逐文件耗时 reporter 并在全量电池事件上传每片报告;新增每周一的 `ui-test-timings-refresh.yml`,把最新合格运行的报告合并成 `ui/scripts/test-shard-timings.json` 并以 fleet App 开 PR。 - 为什么改:ui/ 并入时 reporter 与 refresh 都没搬过来,该数据集自 2026-10-19 冻结,CI 读不到 Test UI 各文件耗时,#25911 试点与 #29469 的四个折叠无法在 CI 复核,分片也按旧数据切。 - 风险与回滚:只报不判(按裁决无阈值、无新红门);上传步 `continue-on-error`,缺报告只是 warning,不会让分片转红;回滚即 revert 本 PR。 - 先看:`.github/workflows/ci.yml`(test-ui 的 `Run the ui tests` 与新的上传步)、`.github/workflows/ui-test-timings-refresh.yml`、`ui/scripts/test-file-timings.mjs`(`shardJobName` 改为 `Test UI (N/M)`)。 - 两分钟验证:`cd ui && pnpm exec vitest run scripts/__tests__/test-file-timings.test.ts`,预期 15 passed。 - 需要你做的:受管面(`.github/workflows/**`,Tier H),请你审批并合并;合并后首个全量电池跑完,在 Actions 里手动触发一次 UI Test Timings Refresh 即可拿到首个刷新 PR。 ## What changes - `.github/workflows/ci.yml`, job `test-ui`: - `Run the ui tests (shard N/16)`: on the whole-suite branch (`MODE=full`: `push`, `schedule`, `workflow_dispatch`, and a pull request that failed open) it now also passes `--reporter="${GITHUB_WORKSPACE}/ui/scripts/test-file-timings.mjs"` beside `default` and `github-actions`, and the step sets `OBJECTUI_TEST_TIMINGS_OUT` to `RUNNER_TEMP/test-file-timings/shard-N.json`. The affected speed passes no reporter. - New step `Publish this shard's per-file test timings`: `actions/upload-artifact@v7`, name `test-file-timings-N`, only when the event is neither `pull_request` nor `merge_group` (the file's own deny-list for the full battery), `continue-on-error: true`, `if-no-files-found: warn`, retention 7 days on `schedule` and 2 otherwise. - The comment that said the reporter and refresh were not ported is replaced by one that says what runs and why. - `.github/workflows/ui-test-timings-refresh.yml` (new), job `Regenerate the per-file test timings`: Mondays 05:47 UTC, `workflow_dispatch`, and `pull_request` on its own file and the three scripts it loads. It runs `test-file-timings.mjs select --shards 16 --limit 6`, downloads each run's `test-file-timings-*` artifacts with `gh run download`, drops a run whose download fails or comes back with fewer than 16 reports, runs `merge` into `ui/scripts/test-shard-timings.json`, and when the bytes changed pushes `claude/ui-test-timings-refresh-RUN` and opens the PR with `skip-changeset` as the fleet App. On `pull_request` every outward act is a dry run. With no eligible run (the state until a full-battery run uploads reports) it reads NOT MEASURED, warns, and stays green. It mirrors `shard-timings-refresh.yml` (Test Core, #28978) in shape, without that lane's ceilings, coverage accumulation and bin pins, which the ui dataset does not have. - `ui/scripts/test-file-timings.mjs`: `shardJobName` now spells the root job, `Test UI (N/M)` (it spelled objectui's `Test (shard N/M)`, so the selector could never find a shard here). `generated_by` names the workflow that exists. Header and one comment updated to the root paths. - `ui/scripts/__tests__/test-file-timings.test.ts`: the pins its retirement note promised back, now against the root `ci.yml` (parsed) and the refresh: the shard job name, the reporter on the whole suite and not on the affected speed, the upload name and path and its report-only shape, the refresh's `--shards` against the matrix width, its download pattern, and that the workflow `generated_by` names exists. - `scripts/pm/dispatch-gates.self-test.mjs`: the live population-marker census names every declaration by file and line, and its comment asks the author of a new one to add the row. The refresh's `no-check-families` declaration is the tenth row (count 9 to 10). Outside the claim's file surface, and the only file outside it. ## Why `ui/scripts/test-shard-timings.json` is what the Test UI sequencer splits on. On `main` `399df5bb` its `measured.at` is `2026-10-19T…`, its `generated_by` names `.github/workflows/test-shard-timings-refresh.yml`, which does not exist in this repository, and the `ci.yml` comment above `Run the ui tests` said the reporter and refresh were not ported. The writer, selector and generator were all present in `ui/scripts/test-file-timings.mjs`; only the CI wiring was missing, and the selector's job name still spelled objectui's. ## Tests - `cd ui && pnpm exec vitest run scripts/__tests__/test-file-timings.test.ts scripts/__tests__/select-affected-tests-wiring.test.ts`: 2 files, 25 tests passed (the wiring test runs the edited step as written). - The writer under vitest 4.1.12, through the exact CI spelling: `GITHUB_WORKSPACE=$PWD OBJECTUI_TEST_TIMINGS_OUT=… pnpm --dir ui run test --shard=1/1 --reporter=default --reporter=github-actions --reporter="$PWD/ui/scripts/test-file-timings.mjs" scripts/__tests__/test-file-timings.test.ts` exited 0 and wrote `{"schema":"objectui.test-file-timings/1","shard":"1/1",…,"files":[{"file":"scripts/__tests__/test-file-timings.test.ts","project":"unit",…}]}`. - Writer to generator: two files' report fed to `merge --runs-file … --out COPY --summary …` wrote a valid dataset (the reader's own `datasetWeights` check runs inside `buildDataset`) carrying the new `generated_by`. - No ablation: the diff changes CI wiring and a name, not a tested behaviour of a package. ## Acceptance notes - The committed `ui/scripts/test-shard-timings.json` still carries the old `generated_by` string. It is generator output and is never hand-edited; the first refresh PR rewrites it from the corrected generator, together with a measurement date after this card. That refresh PR is what closes the card's "Done when". - The reporter is spelled from the workspace root, not as `./scripts/test-file-timings.mjs` relative to `ui/` (objectui's spelling): `check-self-test-wired` keys a `./scripts/` path in a workflow to the root `scripts/`, so the relative spelling minted a key with no file behind it and redded its `--self-test` (measured on the first gate pass). - The refresh carries the `dispatch-gates: no-check-families` declaration: it is path-filtered and runs a generator, not a named check; the scripts it loads are pinned by Test UI's `test-file-timings.test.ts`. - Push runs feed the dataset as well as scheduled ones: unlike Test Core, a `push` run that starts Test UI runs the whole ui suite (`mode=full`), so it is a measurement. The selector already reads both. - The refresh's `pull_request` leg on this PR will read NOT MEASURED (no run has uploaded reports yet) and stay green. - `ui/scripts/dependabot-merge-gate.mjs` lists `Regenerate the per-file test timings` among its optional contexts; the new job carries that name. That file is otherwise not touched. ## Verification All readings at head `82d9d267` (branch off `399df5bb`, then `origin/main` merged after PR #29556 landed, since that PR also edits `scripts/pm/dispatch-gates.self-test.mjs`). <details><summary>dispatch-gates: 69 derived, 69 exit 0</summary> - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack`: 69 commands, each run with its exit code captured before any pipe. - 68 exited 0 on the first pass. `pnpm --dir ui exec vitest run scripts/` exited 1 on the first pass after the merge (5 cases in `check-doc-json-fences.test.ts`, `TypeError: fn is not a function` inside zod, a file this diff does not touch); after re-running the seven-package `@objectstack` build the ui jobs run (turbo replay, 1 s under the lock) it exited 0: `Test Files 170 passed | 2 skipped (172)`, `Tests 4976 passed | 2 skipped (4978)`. Read as stale build state after the merge. - `pnpm check:pm-dispatch-gates` exit 0: `dispatch-gates self-test (fast tier): 1945 cases pass`, including `the live tree carries the ten declarations measured for this census`. - `dispatch-gates --ran`: "69 derived famil(ies) accounted for — 69 run, 0 NOT-MEASURED", exit 0. - Two reds on the first gate pass (head `b117998e`) shaped the final diff: `check-self-test-wired --self-test` (the `./scripts/` reporter spelling) and the dispatch-gates coverage guard (the refresh's missing `no-check-families` declaration). Both are described under Acceptance notes. </details> <details><summary>outside the derived set</summary> - `pnpm --dir ui run type-check:scripts` (what Type Check UI runs for `ui/scripts`): exit 0, 0 errors. - `pnpm --dir ui exec eslint scripts/test-file-timings.mjs scripts/__tests__/test-file-timings.test.ts`: exit 0. - Both workflow files parse with the `yaml` package the ui pins use. </details> ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (not touched) - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes. (eslint on the two ui files: exit 0; repo-wide lint is CI's) - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset. (publishes nothing: CI wiring and a private ui script; `skip-changeset`) - [x] 6. `packages/spec` touched: n/a. - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. --- _Generated by [Claude Code](https://claude.ai/code/session_01JQwxbXf3xRiPQrLhrgsFzw)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent be3cee3 commit 3bd4bf6

5 files changed

Lines changed: 567 additions & 40 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 39 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3885,23 +3885,56 @@ jobs:
38853885
# paths relative to ui/, the vitest root, passed as positionals with no
38863886
# `--`, the only spelling ui/scripts/vitest-invocation-guard.mjs admits.
38873887
#
3888-
# objectui's full-battery runs also pass its per-file timings reporter
3889-
# and upload the result for the weekly dataset refresh. Both belong to
3890-
# that refresh workflow, which is in the governance-workflow audit
3891-
# (sub-issue 4 of the epic); until it is ported the committed dataset
3892-
# is what the sequencer reads.
3888+
# THE PER-FILE TIMINGS (objectstack#29517). The whole suite also passes
3889+
# ui/scripts/test-file-timings.mjs, the per-file timings reporter, beside
3890+
# the two reporters it would otherwise displace. It writes this shard's
3891+
# report to $OBJECTUI_TEST_TIMINGS_OUT, and the step after this one
3892+
# uploads it on the full battery's events as `test-file-timings-N`.
3893+
# .github/workflows/ui-test-timings-refresh.yml merges those reports
3894+
# weekly into ui/scripts/test-shard-timings.json, the dataset the shard
3895+
# sequencer splits on. Report-only, by ruling: the reporter never throws
3896+
# into vitest (a failure is one stderr line and no report), and nothing
3897+
# here or there grades a duration. The affected speed passes no reporter:
3898+
# a slice of the suite is not a measurement of it.
3899+
# ui/scripts/__tests__/test-file-timings.test.ts holds this step, the
3900+
# upload and the refresh's `--shards` to the names that script spells.
3901+
# The reporter is spelled from the workspace root, never `./scripts/…`
3902+
# relative to ui/: the root's workflow readers (check-self-test-wired)
3903+
# key a `./scripts/` path to the ROOT `scripts/`, where it is not.
38933904
- name: Run the ui tests (shard ${{ matrix.shard }}/16)
38943905
env:
38953906
MODE: ${{ steps.select.outputs.mode }}
3907+
OBJECTUI_TEST_TIMINGS_OUT: ${{ runner.temp }}/test-file-timings/shard-${{ matrix.shard }}.json
38963908
run: |
38973909
if [ "$MODE" = 'affected' ]; then
38983910
mapfile -t filters < "$RUNNER_TEMP/affected-tests.txt"
38993911
echo "Affected speed: ${#filters[@]} vitest filter(s), shard ${{ matrix.shard }} of 16."
39003912
pnpm --dir ui exec vitest run --shard=${{ matrix.shard }}/16 --reporter=default --reporter=github-actions "${filters[@]}"
39013913
else
3902-
pnpm --dir ui run test --shard=${{ matrix.shard }}/16 --reporter=default --reporter=github-actions
3914+
pnpm --dir ui run test --shard=${{ matrix.shard }}/16 --reporter=default --reporter=github-actions --reporter="${GITHUB_WORKSPACE}/ui/scripts/test-file-timings.mjs"
39033915
fi
39043916
3917+
# Diagnostics only, so it can never red a shard: `continue-on-error`, and
3918+
# a missing report is a warning, not an error (the refresh's selector
3919+
# then skips this run by name). Uploaded on the full battery's events
3920+
# only, the deny-list of THE TWO SPEEDS above `jobs:`; a pull request that
3921+
# failed open to the whole suite writes a report and uploads nothing.
3922+
# `overwrite: true` so a re-run shard replaces its report, as the
3923+
# shard-attestation uploads do. The refresh reads only the newest
3924+
# completed `push` / `schedule` runs of this workflow on `main`, as many
3925+
# as its selector's own cap examines, so the retention below is
3926+
# deliberately more than that window needs; it is not a measurement of it.
3927+
- name: Publish this shard's per-file test timings
3928+
if: ${{ always() && github.event_name != 'pull_request' && github.event_name != 'merge_group' }}
3929+
continue-on-error: true
3930+
uses: actions/upload-artifact@v7
3931+
with:
3932+
name: test-file-timings-${{ matrix.shard }}
3933+
path: ${{ runner.temp }}/test-file-timings/
3934+
if-no-files-found: warn
3935+
retention-days: ${{ github.event_name == 'schedule' && 7 || 2 }}
3936+
overwrite: true
3937+
39053938
test-ui-gate:
39063939
# Stable required-check name for the sharded Test UI matrix (#26460, ruling
39073940
# A, comment 6435318516), in test-gate's shape and for test-gate's reason

0 commit comments

Comments
 (0)