You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 4, 2026. It is now read-only.
Repository navigation
Commit 140916c
Browse filesBrowse the repository at this point in the historyBrowse files
authored
ci(test): widen the test matrix from 12 to 16 legs, and move every pin of the width (objectui#12384) (#12399)
Fixes#12384
## What this does
It widens `ci.yml`'s `test` matrix from 12 to 16 legs and moves every
pin of the width in the same diff. This is route A of the reading
`6181333653`, which the seat accepted in claim `6181366176`.
What stays: ⛔ `timeout-minutes: 20` is not raised, ⛔ `testTimeout` is
not raised, ⛔ no test is skipped or quarantined, ⛔ the relevance gate is
not widened, ⛔ the coverage matrix (8 legs, 40-minute cap) is not
touched, and ⛔ no runner is pinned.
It applies ruling `6121816480`'s own width rule: "the width derived from
the total measured duration at a target of about 10 minutes per shard".
Triage `6177117080` says: "A lighter shard plan, or a runner pin,
changes no limit and needs no ruling."
## Pins moved
Derived with `git grep` for `/12`, `twelve`, `--shards 12`, `shard
N/12`, `of 12` and `12-runner`, then read hit by hit.
- **`.github/workflows/ci.yml`**:
- the `shard:` list (1 to 16);
- the job name `Test (shard N/16)`;
- the step name `Run tests (shard N/16)` and its echo line;
- both `--shard=N/16` invocations;
- `test-aggregate`'s `check-test-shard-results.mjs --shards 16`;
- `shard-headroom.mjs --job 'Test (shard {n}/16)' --shards 16`;
- the full-battery echo line;
- the comments: a new "12 -> 16" paragraph beside the "4 -> 8" and "8 ->
12" ones, and the stale "8 runners" in the job's first paragraph, now
width-neutral.
- **`.github/workflows/test-shard-timings-refresh.yml`**:
`test-file-timings.mjs select --shards 16`, its NOT MEASURED warning
text, and the header. The header also gains one paragraph on what a
width move does to the selection (A3 below).
- **`.github/workflows/lint.yml`**: the one comment that names the legs.
- **`scripts/dependabot-merge-gate.mjs`**: the `NOT_A_GATE` rows, `Test
(shard 1/16)` through `Test (shard 16/16)`, and the comments and strings
that name the PR lane.
- **Usage headers**: `check-test-shard-results.mjs`,
`shard-headroom.mjs`, `test-file-timings.mjs` and
`select-affected-tests.mjs`.
- **Tests**:
- cases in `dependabot-merge-gate.test.ts`,
`select-affected-tests.test.ts` and `shard-headroom.test.ts`;
- comments in `check-eager-closure-budget.test.ts`,
`check-skill-examples.test.ts` and `check-test-path-roots.test.ts`.
- **`content/docs/guide/ci-cd-pipeline.md`**:
- the job-table row, whose "Appears as" `ci-cd-pipeline-doc.test.ts`
matches against ci.yml's `name:`;
- a 12 -> 16 history sentence;
- the sentences that state the shard count in the aggregator, refresh
and Dependabot sections.
- ⛔ The `OBJECTUI_ALLOW_MAJOR` sentences that objectui#12334 edits are
not touched.
- **`AGENTS.md`**: the `--shard=1/16` example, and the sentence two
lines below it that names the example's number. Nothing else in that
paragraph changes.
**One new pin, in the same family.** `test-file-timings.test.ts` now
holds the weekly refresh's `select --shards` to the `test` matrix width,
the way `check-test-shard-results.test.ts` already holds
`test-aggregate`'s `--shards`.
- Without it, a refresh left on an old width selects no run. It prints
NOT MEASURED and stays green, and the timings dataset quietly stops
refreshing.
- No test held this width pin before. The ablation is under
Verification.
**Left on purpose:**
- history: "objectui#12208 to 12" and "Balanced, the twelve legs ...";
- `test-file-timings.test.ts`'s writer case, which turns `shard: {
index: 3, count: 12 }` into `'3/12'`: an arbitrary reporter input and
output, not a pin of the matrix;
- `flowLaunch.translation-12075.test.tsx`'s "red in CI's shard 4/12": a
dated measurement.
## A2: the required-check set names the aggregate, not the legs
- **Command:** `node scripts/check-required-check-set.mjs --fixture
FILE`. FILE is a `gh api
repos/objectstack-ai/objectui/rules/branches/main` response read on
2026-10-15. The script's own live fetch answered HTTP 401 in this
container (exit 2, "could not take a reading").
- **Result:** exit 0, "✅ INTACT — 1 pinned and 6 watched context(s) all
present (7 required context(s) read)".
- **The 7 contexts:** `Lint`, `Type Check`, `Build & E2E`, `Build Docs`,
`Changeset Declaration`, `Test`, `Governed Surface Queue Guard`.
No `Test (shard N/W)` leg is required, so the width is a workflow edit,
not a ruleset change.
## A3: the timings refresh across the width change
`scripts/test-shard-timings.json` is keyed per file and records no
width, and the sequencer reads it per file, so it needs no change. The
refresh's selector judges a run by the leg names of one width:
`judgeRun` looks up `Test (shard n/M)` for each n.
**Measured on 2026-10-15** by running `selectRuns()` unchanged against
main's live runs, with its reads routed through `gh api`:
- `--shards 16`: it examined 30 push and schedule runs, its cap (limit ×
5). It skipped every one ("Test (shard 1/16) is not in the run") and
took 0. That is the CLI's NOT MEASURED, exit 3.
- Control, `--shards 12`: it took 6 of the first 6 runs examined.
**After landing:**
- The old 12-leg runs are skipped by name and are never mixed in.
- The first green 16-leg full battery is the first run taken: the
merge's own `push` run, then the hourly `schedule`.
- The refresh runs on Mondays at 06:47 UTC. If it fires before such a
run exists, it prints the NOT MEASURED warning, exits green and leaves
`test-shard-timings.json` untouched. It does not fail, and the next
week's run takes the 16-leg runs. Only a landing in the hour or so
before that slot hits this.
- This PR's own `pull_request` leg of the refresh workflow runs the same
selection at 16 and should read NOT MEASURED.
## A4: the per-leg estimate
These figures come from the reading's numbers. Actions history was not
re-measured.
Each leg carries 12/16 = 0.75 of today's load. The reading's 12-leg
figures, over 50 hours of finished PR and queue legs:
- median 11.3 min and p99 12.6 min;
- the full-battery maximum is 13.6 min;
- normal setup takes about 35 s;
- the reading's upper bound for a fixed per-leg vitest start cost is 1.2
min.
| | 12 legs (measured) | 16 legs, all cost scales | 16 legs, 1.8 min
fixed per leg |
|:--|--:|--:|--:|
| median leg | 11.3 min | 8.5 min | 8.9 min |
| p99 leg | 12.6 min | 9.45 min | 9.9 min |
| full-battery longest leg | 13.6 min | 10.2 min | 10.65 min |
| break-even slow factor, p99 leg (20 min / p99) | 1.59x | 2.12x | 2.02x
|
The last column is 1.8 + 0.75 × (leg − 1.8).
- **Headroom.** Slow VMs measured 1.6x to 2.0x against normal controls.
The p99 margin stays thin, at about 2.0x to 2.1x, as the reading said.
The median leg gets about 2.25x to 2.35x.
- **The nine legs named on the card.** Their projected `Run tests` of
1,168 to 1,383 s at 12 legs becomes 876 to 1,037 s at 16. With their own
slow setup (47 to 68 s) that is 15.6 to 18.3 min, all under 20.
- **Cost.** Four more legs at their fixed cost of 0.6 to 1.2 min each
come to about +2.5 to +5 runner-min per run. That is +2% to 4% of about
136 runner-min, or about +330 to +650 runner-min a day at about 134 runs
a day. Standard hosted runners on this public repository are not billed.
- **The ruling's acceptance.** It reads "ten consecutive full-suite runs
with the longest shard under 60 % of the cap", which is 12 min.
- The projected longest leg on a normal VM is 10.2 to 10.65 min, under
12.
- A VM slower than about 1.15x still crosses 12.
- NOT MEASURED until it runs: `test-aggregate`'s headroom step reads it
on every run.
## Verification (head `0f5a949e4`)
**Tests.** I ran them from the repository root, under the shared verify
lock.
- Command: `pnpm exec vitest run --maxWorkers=2 scripts/__tests__/`.
- Result: `Test Files 187 passed | 2 skipped (189)`, `Tests 5633 passed
| 2 skipped (5635)`, lock `VERDICT command-exit 0`.
- This covers every pin touched here: `check-test-shard-results`,
`shard-headroom`, `select-affected-tests`, `dependabot-merge-gate`,
`test-file-timings` and `ci-cd-pipeline-doc`. It also covers the other
workflow-reading suites under `scripts/__tests__`.
- None of the touched files carries a skip.
**Ablation of the new refresh pin.** I used `ablation-replace` in wrap
mode, under the lock, against the commit above.
- Mutation: `select --shards 16 --limit 6` became `select --shards 12
--limit 6` in the refresh workflow. The anchor count went 1 → 0, the
replacement count 0 → 1, and the blob `7198fa4f893c` became
`9bdb34cea68c`.
- Result: 1 failed and 14 passed, and the failure was the new case: "the
refresh's --shards and ci.yml's `test` matrix have drifted apart:
expected 12 to be 16".
- Restore: the blob after restore equals HEAD's (`7198fa4f893c`) and
`git diff HEAD` is empty. The green leg is the full run above.
**`check-test-shard-results.mjs --fixture`:**
| run | exit | reads |
|:--|--:|:--|
| `--shards 8` on the recorded 8-leg fixture
`run-36042402511-attempt-1.json` | 0 | "all 8 shards reported success" |
| `--shards 16` on that 8-leg fixture | 3 | every `Test (shard n/16)`
ABSENT |
| `--shards 16` on a 16-leg job list synthesized from that fixture (not
committed) | 0 | "all 16 shards reported success" |
| `--shards 12` on the synthesized list | 3 | every `Test (shard n/12)`
ABSENT |
**Gates that read these paths**, derived by hand from `lint.yml`,
`repo-gates.yml`, `line-citation-gate.yml` and `changeset-presence.yml`.
The objectstack `dispatch-gates` tool does not serve this repository.
Each exit code was captured before any pipe.
| gate | exit |
|:--|--:|
| `node scripts/check-control-bytes.mjs` | 0 |
| `node scripts/check-action-ref-convention.mjs` | 0 |
| `node scripts/check-doc-links.mjs` | 0 |
| `node scripts/check-shell-escape-residue.mjs` | 0 |
| `node scripts/check-doc-fence-languages.mjs --self-test` | 0 |
| `node scripts/check-doc-fence-languages.mjs` | 0 |
| `pnpm check:new-line-citations` ("0 new citation(s)") | 0 |
| `node scripts/check-changeset-presence.mjs` ("no changeset is owed") |
0 |
| `node scripts/check-test-path-roots.mjs` | 0 |
| `node scripts/check-lint-coverage.mjs` | 0 |
| `pnpm check:governed-queue-guard` | 0 |
| `pnpm check:required-check-set` | 0 |
The repository has no actionlint or other workflow linter, so the
workflow-shape readings are the `scripts/__tests__` suites above.
Repo-wide `pnpm lint` and the full test farm are CI's; this PR's own CI
is the first 16-leg run.
**Governed surface.** `node scripts/check-governed-queue-guard.mjs
--test` on the 17 changed paths exits 3:
```text
⛔ GOVERNED — 4 of 17 path(s) are on a governed surface:
AGENTS.md x1 — the repo-root agent instruction file
.github/workflows/** x3 — CI, deploy and release workflows: what runs on every push, what deploys, what gates the queue
- .github/workflows/ci.yml
- .github/workflows/lint.yml
- .github/workflows/test-shard-timings-refresh.yml
```
So this PR stays **draft** until an APPROVED review from a
`GOVERNED_APPROVERS` account. The claim also owes a
`CONTRACT_REVIEW_TIER` review.
## Acceptance notes
- **Changeset.** None is owed. `check-changeset-presence` reads 0
published source and 0 published contract changed, and CI config and
repository scripts publish nothing.
- **A stale comment, left alone.** `ci-cd-pipeline-doc.test.ts` has a
comment that still gives the matrix job's name with `/4`, stale since 4
-> 8. It is a comment that nothing reads, outside this card's surface.
Carrier: none, so it is noted here only.
- **A stale phrase, made width-neutral in place.** The header of
`select-affected-tests.mjs` said "the four-shard coverage lane", stale
since objectui#12256. It sat in the same sentence this diff edits, so it
now reads "the sharded coverage lane". It is a comment only, and the
coverage matrix itself is unchanged.
- **What holds each width pin now:**
- `--shards` in `test-aggregate`: `check-test-shard-results.test.ts`;
- `--shards` in `shard-headroom`: `shard-headroom.test.ts`;
- `--shards` in the refresh: `test-file-timings.test.ts`, new;
- the `NOT_A_GATE` rows: the partition test in
`dependabot-merge-gate.test.ts`;
- the job and step names: `select-affected-tests.test.ts` and
`test-file-timings.test.ts`;
- the doc row: `ci-cd-pipeline-doc.test.ts`.
- **The reading's out-of-scope finding stays on this card, its family
carrier.** Two `export-to-react-compiles` tests ran 17.3 s and 17.6 s
against `testTimeout: 15000` on a slow VM. This PR does nothing about
them, and `testTimeout` is unchanged.
## 维护者速读(草稿)
**改了什么**
- CI 的测试分片从 12 条腿加到 16 条,每条腿的负载降到原来的四分之三。
- 20 分钟超时上限不动,不跳过任何测试,不动覆盖率分片,也不锁定 runner。
- 所有写死 12 的地方一并改成 16:工作流、Dependabot 合并门、每周计时刷新、文档,以及 AGENTS.md 里的示例。
- 另加一条测试:以后再改宽度时,如果漏改了计时刷新,本地测试会直接变红。
**为什么改**
- 2026-10-14 有 16 条测试腿在 20 分钟上限被取消,全都没有挂死。
- 原因是部分虚拟机整体变慢,同样的文件要多花 1.6 到 2.0 倍的时间。
- 后果:测试全过的 PR 显示红色,要手动重跑;合并队列里的 PR 被踢出,后面排队的都要多等一轮。
- 12 条腿时,p99 腿只有约 1.59 倍余量,低于慢机器的系数;16 条腿约 2.0 到 2.1 倍。
- 这是执行既有裁决的宽度规则(每条腿约 10 分钟),不是新决定。
**风险与代价(含回滚)**
- 每次运行约多 2.5 到 5 个 runner 分钟(+2% 到 4%)。本仓是公开仓,标准 runner 不计费。
- p99 余量仍然偏薄:约 2.0 到 2.1 倍,而实测最慢约 2.0 倍。
- 测试套件继续增长会再吃掉余量。现有的只读 headroom 报告步骤每次运行都会显示,不新增门禁。
- 落地后第一次每周计时刷新,如果早于第一次 16 腿全绿运行触发,会报 NOT MEASURED 并保留原文件,不会变红。
- 回滚:revert 本 PR 即回到 12 条腿。没有数据迁移,也没有仓库设置改动,因为必需检查只认 `Test` 聚合。
**席位意见**
(留空)
**你要做的**
- 审阅后,在本 PR 上留下一条 APPROVED review。本 PR 改动了 `.github/workflows/**` 与
`AGENTS.md`,属于受管面。
- 之后由认领席翻 ready、挂入合并队列。
This change was implemented by the dispatched dev of the `domain:ui`
seat 1 PM, session
`https://claude.ai/code/session_01JZnppgHRp4sREcouhkQYDB`.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01JZnppgHRp4sREcouhkQYDB)_
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: os-zhuang <jack@objectstack.ai>
echo "::warning title=Test shard timings::NOT MEASURED: no completed full-battery run on main has all twelve test shards green and all twelve timing artifacts. scripts/test-shard-timings.json is untouched."
134
+
echo "::warning title=Test shard timings::NOT MEASURED: no completed full-battery run on main has all sixteen test shards green and all sixteen timing artifacts. scripts/test-shard-timings.json is untouched."
0 commit comments