Repository navigation
Commit 1d41037
ci(dogfood): prune the per-sha Turbo seed to the run's tasks before saving it (#30241)
<!-- claude-projects-attribution: {"github_login":"marchtian"} -->
_Requested by **田騏** · [project
thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSiNjyvCdXYygFqntMNR7S2dW)_
Part of #25452 (the card stays open for its after-reading: the pool
after ≥ 5 merges, read by an account that can read `actions/cache/*`)
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:把 Dogfood Regression Gate 三个分片的「Save Turbo cache (main
only)」接到现成的 `prune-turbo-cache.mjs` 后面。测试步骤的 turbo 命令加 `--summarize`
写运行摘要,新增的修剪步骤只留本次运行用到的任务条目,保存只在修剪答 `save=true`
时进行。不加脚本、不加门禁、不改恢复键、不抬池子上限。
- 为什么改:池子读数 6647111495 里,`Linux-turbo-dogfood-1-main-*` 每个 sha 约 3.69
GB,两份占池子 95%。整份保存会把之前每份种子都累积进来,把其他种子挤出 10 GB 池。裁决 6458731814 选 A「修剪,不删」。
- 风险与回滚:只动 `.github/workflows/ci.yml` 的 dogfood 作业。dogfood 只在 main 的
push/定时运行上跑,PR 上看不到效果。最坏情况是某次 push 不保存种子(没有运行摘要就是
`save=false`),下次继续恢复上一份种子。回滚 = revert 本 PR。
- 先看:`.github/workflows/ci.yml` dogfood 作业新增的「Prune the Turbo cache to
this run's tasks (main only)」步骤,以及其后保存步骤的 `if:`;同一作业测试步骤 turbo 命令上的
`--summarize`。
- 两分钟验证:`node scripts/ci/prune-turbo-cache.mjs --self-test`,预期最后一行
`prune-turbo-cache self-test: OK (25 case(s) across 5 batteries).`
- 需要你做的:作为 GOVERNED_APPROVERS 审批(或亲手合并)这个 Tier H draft PR。
## What changes
Ruling 6458731814 (#26957, letter A): "prune, do not drop", each
seed-save step "wired behind `scripts/ci/prune-turbo-cache.mjs --site …`
exactly as Test Core … [is] today, so a seed holds only the task entries
its own run used". "⛔ The pool limit is not raised here" stands.
One file, `.github/workflows/ci.yml`, job `dogfood` (`Dogfood Regression
Gate (N/3)`), +23 / -2:
- **Test step** ("Boot example apps and exercise real user flows"):
`pnpm turbo run test --filter=@objectstack/dogfood --log-order=stream`
gains `--summarize`, so the run writes `.turbo/runs/` (one JSON file per
run). The job had no run summary before: neither of its two turbo
invocations (the closure build and the test run) passed `--summarize`,
and the job sets no `TURBO_RUN_SUMMARY`.
- **New step** "Prune the Turbo cache to this run's tasks (main only)",
`id: prune-turbo-cache`, `if: always() && github.event_name == 'push'`:
`node scripts/ci/prune-turbo-cache.mjs --site 'Dogfood Regression Gate
(N/3)' --cache .turbo/cache --run-summaries .turbo/runs`. These are Test
Core's step and condition, without `--extra-summaries` (next bullet).
- **Save step** "Save Turbo cache (main only)": `if:` ANDs
`steps.prune-turbo-cache.outputs.save == 'true'`. The path and key are
unchanged.
- **Unchanged:** the restore step and its keys,
`upgrade-harness-optin.yml` (its restore at L96), the job's `if:`, its
timeout, and the two attestation steps, which stay last and
unconditioned.
Why one summary is enough, and why there is no `--extra-summaries`: the
test run plans the whole `^build` closure and replays it, so its summary
names every closure `build` task. Measured at base `f2a9007e93` with
`turbo run build --filter=@objectstack/dogfood --dry=json` against
`OS_TEST_SHARD=1/3 turbo run test --filter=@objectstack/dogfood
--dry=json`: 71 shared tasks, 71 hash-identical, 0 differing. The only
closure node missing from the test plan is `@objectstack/dogfood#build`,
which has no command and no cache entry. Test Core needs
`--extra-summaries` only because its closure summary is moved out of
`.turbo/runs/` by the remote-cache assertion, and the dogfood job runs
no such assertion.
No `--site` registry, self-test or gate lists sites. `git grep
prune-turbo-cache` outside the workflows finds only the script itself
and two `ui/scripts/__tests__` files about the ui jobs. So, as with
#27067, nothing else needs to change.
## Tests
<details><summary>Local seed-size estimate (closure only), at 4f58391
= base + this diff</summary>
The setup stands in for CI. A private `--cache-dir` was seeded with a
copy of this container's shared `.turbo/cache` (260 MB, 1059 files) as
the "restored seed". Then the job's closure command ran
(`--concurrency=2`): `Tasks: 69 successful, 69 total`, `Cached: 2
cached`, `6m39s`, on a shared 4-vCPU box. Then came the test step
exactly as this PR spells it, but with an out-of-range
`OS_TEST_SHARD=9/3`, so vitest refused at once. That run reported `69
cached, 70 total`, `Failed: @objectstack/dogfood#test`, and wrote its
summary anyway, so a red test step still leaves a summary to prune by.
Then the prune ran exactly as the new step does:
```
prune-turbo-cache: Dogfood Regression Gate (1/3): kept 69 task entries named by 1 run summary (70 task hash(es); 1 have no entry -- a failed task, or one turbo does not cache).
prune-turbo-cache: Dogfood Regression Gate (1/3): dropped 1053 file(s) the run did not use: 304.3 MB -> 35.7 MB on disk.
save=true
```
- Before: 304.3 MB. After: 35.7 MB (207 files = 69 entries x 3).
- The local "before" is not CI's. CI restores the accumulated ~3.69 GB
seed, and the first push run after this lands prunes that down to the
same closure, as Test Core's first run did (2912.4 MB -> 0.8 MB).
- The scope is the closure only. A real `dogfood#test` entry (`outputs:
[]`, a log archive) adds a small amount: NOT MEASURED, because a full
dogfood shard was not run here.
- Estimate: about 36 MB per shard seed, about 0.11 GB for the three
shards of one merge.
- `--summarize` moves no hash: the summary's 69 build hashes equal the
dry-run plan taken without it (69 identical, 0 differing).
</details>
- Upgrade-harness closure (`upgrade-harness-optin.yml` restores
`turbo-dogfood-1-main-*` and builds `--filter=@objectstack/dogfood`):
every one of the 69 closure tasks that has a command still has its
`.tar.zst` in the pruned seed (69 of 69, 0 missing).
- `node scripts/ci/prune-turbo-cache.mjs --self-test` (script
unchanged): `prune-turbo-cache self-test: OK (25 case(s) across 5
batteries).`, exit 0.
- The `dogfood` job's steps, parsed with the `yaml` package: ... 8
`Upload stall diagnostic reports`, 9 `Prune the Turbo cache to this
run's tasks (main only)` (`id=prune-turbo-cache`, `if="always() &&
github.event_name == 'push'"`), 10 `Save Turbo cache (main only)`
(`if="always() && github.event_name == 'push' &&
steps.prune-turbo-cache.outputs.save == 'true'"`), 11 `Attest this shard
ran and passed`, 12 `Publish this shard's attestation`.
- No ablation, because this is workflow config and not tested behaviour.
## Acceptance notes
- **Before** (pool reading 6647111495, main at `474c030b13`):
- First read: 10,925,870,698 B in 27 entries. Second read: 7,745,686,441
B in 16 entries.
- The two `Linux-turbo-dogfood-1-main-*` seeds are 3,685,866,556 B and
3,687,370,253 B, together 7.37 GB, or **95.2 %** of the second read.
`*-turbo-*` keys overall are 99.8 %.
- **Writer correction:** the reading names
`upgrade-harness-optin.yml:96` as the source, but that step is
`actions/cache/restore` and never saves. The writer is this job's save
step: `github.job` is `dogfood`, so its key is
`Linux-turbo-dogfood-SHARD-main-SHA`. The harness only restores shard
1's seed.
- **After:** owed after at least 5 merges to `main`. It is a maintainer
pool reading (`actions/cache/usage` plus the full `actions/caches`
listing, in the shares of 6647111495), because the seat token gets 403
on `actions/cache/*`. Until then, the first push run's three
`prune-turbo-cache: Dogfood Regression Gate (N/3): …` lines are the
measured post-prune seed size.
- **Not exercised on this PR:** the `dogfood` job does not run on
`pull_request` or `merge_group` (THE TWO SPEEDS), so this PR's own CI
cannot exercise the new step. The first run is the first push run on
`main` after merge.
## Verification
Derived gates (`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack`), run on `2a737765df` (this branch after
merging `origin/main` `ff12aed37`; the derived list was byte-identical
before the merge). Each exit code was captured before any pipe.
<details><summary>53 derived commands: 53 exit 0, plus the --ran
line</summary>
- 18 direct `node` gates (closing-keyword parity, comment-mask corpus,
self-test-wired, self-test-workflow-commands, step-collectors,
whole-set-label-write, assert-turbo-remote-cache / plan-test-core-shards
/ prune-turbo-cache / scheduled-full-run self-tests, drift-comment,
ci-failure self-test, dts-emitted self-test): exit 0.
- 35 `pnpm check:*`, including `check:ci-filter-parity`,
`check:required-contexts`, `check:shard-attestation`,
`check:stall-guard-budget`, `check:stall-guard-headroom`,
`check:workflow-status-functions`, `check:workflow-step-name-quoting`,
`check:agent-test-spelling`, `check:aggregator-roster`,
`check:nul-bytes`, `check:pm-dispatch-gates` (fast tier, 340 s) and
`check:type-check-debt` (under the verify lock, re-measure 366 s, 0 raw
tsc errors): exit 0.
- `check:dual-build-cjs-loads` first exited 3 (PREREQUISITE NOT MET: 7
packages outside the dogfood closure had no `dist/`). It was re-run
after `check:type-check-debt` built every package and then exited 0: 116
require entry points across 71 packages load.
```
Run reconciliation — 53 derived, 53 run, 0 NOT-MEASURED, 0 UNRUN.
✓ dispatch-gates --ran: 53 derived famil(ies) accounted for — 53 run, 0 NOT-MEASURED (a DERIVED zero — all 53 recorded an exit code and none of them is 3).
```
`actionlint` is not derived and not installed, so it was not run. CI's
Lint & Repo Gates runs the workflow lints.
</details>
Session: https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- signed: domain:devx#2 · marchtian ·
session_017ihL2ebbZkr2QAEG1x7UC8 -->
Co-authored-by: Claude <noreply@anthropic.com>1 parent 8eca18f commit 1d41037
1 file changed
Lines changed: 23 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2536 | 2536 | | |
2537 | 2537 | | |
2538 | 2538 | | |
| 2539 | + | |
| 2540 | + | |
| 2541 | + | |
2539 | 2542 | | |
2540 | 2543 | | |
2541 | 2544 | | |
2542 | | - | |
| 2545 | + | |
2543 | 2546 | | |
2544 | 2547 | | |
2545 | 2548 | | |
| |||
2574 | 2577 | | |
2575 | 2578 | | |
2576 | 2579 | | |
| 2580 | + | |
| 2581 | + | |
| 2582 | + | |
| 2583 | + | |
| 2584 | + | |
| 2585 | + | |
| 2586 | + | |
| 2587 | + | |
| 2588 | + | |
| 2589 | + | |
| 2590 | + | |
| 2591 | + | |
| 2592 | + | |
| 2593 | + | |
| 2594 | + | |
| 2595 | + | |
| 2596 | + | |
| 2597 | + | |
2577 | 2598 | | |
2578 | 2599 | | |
2579 | 2600 | | |
2580 | | - | |
| 2601 | + | |
2581 | 2602 | | |
2582 | 2603 | | |
2583 | 2604 | | |
| |||
0 commit comments