Repository navigation
Commit 8852099
ci(test-core): start only the Test Core shards whose bin holds work (#26391)
<!-- claude-projects-attribution: {"github_login":"os-steve"} -->
_Requested by **Steve** · [project
thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9MGLtJ9JRr4ZgCPbNRAmMxS)_
Part of #26084
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:Test Core 不再固定起 8 个分片。新加一个 `Plan Test Core shards`
job,先把受影响包算一遍、按原规则切成 8 份,只为有包的那几份起 runner;没包的分片不再创建。分片数(8)、分片名、每片测的包都不变。
- 为什么改:最近 25 次 PR 运行里,Test Core 的 168 个分片有 72 个是空的(43%),每个空片照样花约 97 秒做
checkout 和安装,合计 7020 runner 秒。只改脚本、文档或工作流的 PR 通常只需要 1 到 3 片。
- 风险与回滚:代价是 Test Core 的分片要等规划 job 跑完再开始,大约晚 65 到 95 秒。规划 job 挂了就退回 8
片全跑(不会少测)。每个分片会自己重算一遍计划,和规划 job 对不上就直接变红。回滚就是 revert 这个 PR。
- 先看:`.github/workflows/ci.yml` 里的 `test-plan` job、`test` 的矩阵和
`test-gate` 的 verify 步骤;`scripts/check-shard-attestation.mjs` 的
`--planned` 计数;`scripts/ci/plan-test-core-shards.mjs`。
- 两分钟验证:打开本 PR 的 CI 运行,看 `Plan Test Core shards` 的日志列出哪些分片启动,再看 Test
Core 实际只出现那几个 `Test Core (k/8)`,并且必需检查 `Test Core` 是绿的。
- 需要你做的:这个 PR 改了 `.github/workflows/ci.yml`,属于 Tier H 受管面,需要 os-zhuang 或
hotlong 批准后才能落地。
## What changes
**Test Core starts only the shards whose bin holds work.** A new
`test-plan` job (`Plan Test Core shards`) runs the shards' own selection
(`scripts/ci/select-shard-packages.sh`, the same event fields) and the
same partition (`planRun()` from `scripts/partition-test-shards.mjs` at
`SHARD_COUNT` = 8, dogfood excluded) once, and publishes two outputs:
`shards`, the 1-based indices of the non-empty bins as a JSON array, and
`digest`, a sha256 of the whole partition.
- `test` builds its matrix from that list: `shard: ${{
fromJSON(needs.test-plan.outputs.shards || '[1, 2, 3, 4, 5, 6, 7, 8]')
}}`. Its `if:` gains `needs.test-plan.outputs.shards != '[]'`, so an
empty plan starts no shard runner.
- **The count does not change.** A started shard still runs `--shard
k/8`, keeps its name `Test Core (k/8)`, its cache key and its
attestation `test-k-of-8`. Only empty bins are never created. No package
moves to another shard, and no shard carries more than before.
- **Each shard checks the plan.** "Compute this shard's package set" now
ends with `plan-test-core-shards.mjs --verify`. The shard recomputes the
plan from its own package set and reds, naming both digests, if its
digest differs from the one the matrix was built from. Equal digests
mean equal bins, so the shards that were not started are empty by the
shard's own computation too.
- **It fails open.** If the plan job dies, it publishes nothing. The
matrix then falls back to the literal 1..8, `--verify` gets an empty
digest and says so, and the gate counts 1..8. If the planner's own
self-test fails, the job stops before it plans and the same fallback
applies. A broken planner costs runners, never coverage.
- **`test-gate` counts the planned roster.** It gets `--planned
"test=$OS_TEST_PLAN"` and `--plan-result "$OS_TEST_PLAN_RESULT"`.
`OS_TEST_PLAN` is byte-for-byte the expression the matrix reads,
fallback included. In `check-shard-attestation.mjs`:
- `rosterFor()` and `shardsNeedingCancelCause()` take the planned list,
so a cancel-cause read asks only about shards that actually existed.
- A skipped leg passes on `[]` only when the plan job concluded
`success`. Every other skip goes through the unchanged #4928 guard.
- An unreadable roster is refused with a reason (`parsePlannedRoster`).
- The static guard (`plannedMatrix()`) requires the planned matrix to
carry the full-roster fallback. It requires the gate's `--planned` value
to equal the matrix expression exactly, and `--plan-result` to read
`needs.test-plan.result`. Both jobs must list `test-plan` in `needs:`.
- Pins re-anchored onto the new matrix spelling:
- `partition-test-shards.mjs --self-test` check 4 now reads the fallback
literal.
- `check-shard-attestation.mjs`'s "grow the matrix" fixture now grows
the fallback literal.
- `check-aggregator-roster.mjs`'s two `test-gate` `needs:` anchors now
include `test-plan`. `test-plan` is declared a non-member
(`OS_AGGREGATOR_NON_MEMBERS: filter test-plan`): it supplies the roster
and carries no verdict.
**Required contexts: unchanged.** The reading of
`scripts/check-required-contexts.mjs` (`REQUIRED_CONTEXTS`, L388-470) at
this head is below. `Test Core` is the bare-named aggregate `test-gate`,
and its `name:`, `if: always()` and job id are untouched. The matrix
members `Test Core (k/8)` are not required contexts: rule (4) of that
script refuses a matrix on any required job. The new job `Plan Test Core
shards` carries no registered name. `Test UI (k/16)` is not in the
required set at all.
```
check-required-contexts: 8 required context name(s) pinned across 3 workflow(s)
lint.yml:lint → 'Lint & Repo Gates'
lint.yml:typecheck → 'TypeScript Type Check'
ci.yml:test-gate → 'Test Core'
ci.yml:build-core → 'Build Core'
governed-surface-guard.yml:governed-surface-guard → 'Governed Surface Queue Guard'
ci.yml:lint-ui → 'Lint UI'
ci.yml:build-e2e-ui → 'Build & E2E UI'
ci.yml:type-check-ui → 'Type Check UI'
```
**What this PR does not do, and why it says `Part of`:**
- **Test UI keeps 16 shards.** No Test UI shard is empty. Over the same
25 runs, 0 of 336 Test UI shard jobs ran an empty test step. In affected
mode the selector always adds the always-on population, 556 test files
(~25.8 file-minutes on `ui/scripts/test-shard-timings.json`).
`ui/scripts/test-shard-partition.mjs` gives every leg at least one file
once there are 16 or more. So skipping empty shards saves nothing there.
Running fewer, fuller shards would need the vitest denominator
(`--shard=k/16`) to change.
`ui/scripts/__tests__/select-affected-tests-wiring.test.ts` pins that
denominator: the step header, `--shard=1/16`, and a refusal of any new
expression in the step. That file is under `ui/`, which this card's
claim excludes. Measured on 6390fe5 with one `ui/packages/app-shell`
test file touched: 2552 of 5861 test files selected through 559 filters,
~220 file-minutes.
- **A diff touching one core package still starts all 8 shards.** That
is the right answer, not a miss. Every package diff on main reaches
`@objectstack/cli`, `spec`, `core`, `objectql` and `runtime` through the
dependency graph and the cross-package union. Ten of ten single-package
probes on 6390fe5 filled all eight bins, and the smallest was about
5600 s of predicted test window. Packing such a run into two bins would
roughly double its Test Core wall, so the plan removes only runners with
nothing to do.
## Tests
- `node scripts/ci/plan-test-core-shards.mjs --self-test`: 23 cases, 5
batteries. Covered: empty plan, single package, the bins-with-work rule,
digest movement, `--verify` matching, drift and the empty-digest
fallback, the output format, and the ci.yml wiring read from the live
file.
- `node scripts/check-shard-attestation.mjs --self-test`: 325
assertions, 24 batteries. Two batteries are new:
- `#26084 a planned roster` (16 cases): planned shards counted,
unplanned never named, an unexpected credential red, a failure veto, an
empty plan from a successful vs failed plan job, the filter-skip pass,
#3668 over a planned roster, the cancel-cause read limited to planned
shards, and `parsePlannedRoster` refusals.
- `#26084 the static guard` (10 cases): live-ci.yml fixtures that
diverge the gate's expression by one byte, drop `--planned`,
`--plan-result` or either `needs:` entry, break the fallback literal, or
add `--planned` beside a literal matrix. Every one goes red.
- `node scripts/partition-test-shards.mjs --self-test` and `node
scripts/check-aggregator-roster.mjs --self-test`: green after
re-anchoring.
- The 63 test files under `ui/scripts/__tests__/` that read the root
`ci.yml` are green against this `ci.yml` (1967 tests, `pnpm --dir ui
exec vitest run`, no `ui/` file changed).
`select-affected-tests-wiring.test.ts` and
`ci-setup-pnpm-wiring.test.ts` are among them.
## Acceptance notes
- **Card acceptance, Test Core leg.** "Three runs touching one core
package show at most 2 shard jobs" cannot hold on today's main, for the
reason above. What holds is narrower: a diff that reaches no package
starts 1 to 3 shards. Measured on 6390fe5 with this planner:
| One file changed | Shards started |
| --- | --- |
| `scripts/check-agent-model-declared.mjs` | `[1,2]` (spec, client) |
| `.github/workflows/ci.yml` | `[1]` |
| `content/docs/index.mdx` | `[1,2,3]` |
| `connector-imap`, `plugin-approvals` or `packages/spec` source | all 8
|
"A run touching `packages/spec` still shows the full count" holds.
- **The real-failure leg** ("test-gate verdict unchanged for a run with
a real failure") is shown with fixtures, not with a throwaway CI run:
the planned roster's red paths in `judge()` and the `--verify` CLI. A
missing planned shard and a `failure` result both red. No throwaway
branch run was made, so the gap is written here.
- **Test UI leg:** not done. It needs a `ui/` change to the wiring test,
and its premise of empty shards does not hold (0 of 336). A ui-lane card
would own it.
- **Cost:** one serial hop on every run that starts Test Core,
merge-queue entries included. Shards start after the plan job, ~65-95 s
later, the same checkout and install a shard pays before its own
selection (run 40426272873). The plan job runs once per run in place of
0 to 7 empty shards.
## Verification
<details><summary>Derived gate families (dispatch-gates --commands, 67
commands), all run locally at the head after merging main</summary>
Every derived command exited 0, plus the planner's self-test.
`check:dts-closure`, `check:dual-build-cjs-loads`,
`check:lean-entry-closure` and `check:sourcemap-no-sources-content` ran
after `turbo run build --filter='./packages/**'` under the verify lock
(exit 0). `dispatch-gates --ran` over the recorded exits: 67 derived, 67
run, 0 NOT-MEASURED. `check:self-test-wired` first went red: the
planner's self-test was not run by CI. The fix wires it into the plan
job.
</details>
<details><summary>The empty-shard reading</summary>
The 25 most recent completed, non-cancelled `pull_request` CI runs, read
2026-10-27 (runs 40415024961 to 40426272873). 21 of them started Test
Core, and 13 of those had at least one empty shard. An empty shard is
one whose "Run this shard's tests" step took 3 s or less. 72 of 168
shard jobs were empty, 7020 runner-seconds in total, 97.5 s each. Test
UI: 0 of 336 shard jobs had an empty "Run the ui tests" step.
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_01ADHkmNt4JnHedgMzAG9Xsy)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 80f267d commit 8852099
5 files changed
Lines changed: 744 additions & 28 deletions
File tree
- .github/workflows
- scripts
- ci
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
500 | 500 | | |
501 | 501 | | |
502 | 502 | | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
503 | 596 | | |
504 | 597 | | |
505 | 598 | | |
| |||
597 | 690 | | |
598 | 691 | | |
599 | 692 | | |
600 | | - | |
| 693 | + | |
601 | 694 | | |
602 | 695 | | |
603 | 696 | | |
604 | 697 | | |
605 | 698 | | |
606 | | - | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
607 | 705 | | |
608 | 706 | | |
609 | 707 | | |
| |||
676 | 774 | | |
677 | 775 | | |
678 | 776 | | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
679 | 785 | | |
680 | 786 | | |
681 | 787 | | |
682 | | - | |
| 788 | + | |
683 | 789 | | |
684 | 790 | | |
685 | 791 | | |
| |||
824 | 930 | | |
825 | 931 | | |
826 | 932 | | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
827 | 936 | | |
828 | 937 | | |
829 | 938 | | |
830 | 939 | | |
831 | 940 | | |
832 | 941 | | |
833 | 942 | | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
834 | 947 | | |
835 | 948 | | |
836 | 949 | | |
| |||
1656 | 1769 | | |
1657 | 1770 | | |
1658 | 1771 | | |
| 1772 | + | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
1659 | 1778 | | |
1660 | | - | |
| 1779 | + | |
1661 | 1780 | | |
1662 | 1781 | | |
1663 | 1782 | | |
| |||
1667 | 1786 | | |
1668 | 1787 | | |
1669 | 1788 | | |
1670 | | - | |
1671 | | - | |
1672 | | - | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
1673 | 1794 | | |
1674 | 1795 | | |
1675 | | - | |
| 1796 | + | |
1676 | 1797 | | |
1677 | 1798 | | |
1678 | 1799 | | |
| |||
1714 | 1835 | | |
1715 | 1836 | | |
1716 | 1837 | | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
1717 | 1843 | | |
1718 | 1844 | | |
1719 | 1845 | | |
| |||
1724 | 1850 | | |
1725 | 1851 | | |
1726 | 1852 | | |
1727 | | - | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
1728 | 1856 | | |
1729 | 1857 | | |
1730 | 1858 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
499 | 499 | | |
500 | 500 | | |
501 | 501 | | |
502 | | - | |
| 502 | + | |
503 | 503 | | |
504 | 504 | | |
505 | 505 | | |
| |||
528 | 528 | | |
529 | 529 | | |
530 | 530 | | |
531 | | - | |
| 531 | + | |
532 | 532 | | |
533 | 533 | | |
534 | 534 | | |
| |||
0 commit comments