Skip to content

Commit 1d41037

Browse files
ci(dogfood): prune the per-sha Turbo seed to the run's tasks before saving it (#30241)
<!-- claude-projects-attribution: {"github_login":"marchtian"} --> _Requested by **田騏** · [project thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSiNjyvCdXYygFqntMNR7S2dW)_ Part of #25452 (the card stays open for its after-reading: the pool after ≥ 5 merges, read by an account that can read `actions/cache/*`) Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:把 Dogfood Regression Gate 三个分片的「Save Turbo cache (main only)」接到现成的 `prune-turbo-cache.mjs` 后面。测试步骤的 turbo 命令加 `--summarize` 写运行摘要,新增的修剪步骤只留本次运行用到的任务条目,保存只在修剪答 `save=true` 时进行。不加脚本、不加门禁、不改恢复键、不抬池子上限。 - 为什么改:池子读数 6647111495 里,`Linux-turbo-dogfood-1-main-*` 每个 sha 约 3.69 GB,两份占池子 95%。整份保存会把之前每份种子都累积进来,把其他种子挤出 10 GB 池。裁决 6458731814 选 A「修剪,不删」。 - 风险与回滚:只动 `.github/workflows/ci.yml` 的 dogfood 作业。dogfood 只在 main 的 push/定时运行上跑,PR 上看不到效果。最坏情况是某次 push 不保存种子(没有运行摘要就是 `save=false`),下次继续恢复上一份种子。回滚 = revert 本 PR。 - 先看:`.github/workflows/ci.yml` dogfood 作业新增的「Prune the Turbo cache to this run's tasks (main only)」步骤,以及其后保存步骤的 `if:`;同一作业测试步骤 turbo 命令上的 `--summarize`。 - 两分钟验证:`node scripts/ci/prune-turbo-cache.mjs --self-test`,预期最后一行 `prune-turbo-cache self-test: OK (25 case(s) across 5 batteries).` - 需要你做的:作为 GOVERNED_APPROVERS 审批(或亲手合并)这个 Tier H draft PR。 ## What changes Ruling 6458731814 (#26957, letter A): "prune, do not drop", each seed-save step "wired behind `scripts/ci/prune-turbo-cache.mjs --site …` exactly as Test Core … [is] today, so a seed holds only the task entries its own run used". "⛔ The pool limit is not raised here" stands. One file, `.github/workflows/ci.yml`, job `dogfood` (`Dogfood Regression Gate (N/3)`), +23 / -2: - **Test step** ("Boot example apps and exercise real user flows"): `pnpm turbo run test --filter=@objectstack/dogfood --log-order=stream` gains `--summarize`, so the run writes `.turbo/runs/` (one JSON file per run). The job had no run summary before: neither of its two turbo invocations (the closure build and the test run) passed `--summarize`, and the job sets no `TURBO_RUN_SUMMARY`. - **New step** "Prune the Turbo cache to this run's tasks (main only)", `id: prune-turbo-cache`, `if: always() && github.event_name == 'push'`: `node scripts/ci/prune-turbo-cache.mjs --site 'Dogfood Regression Gate (N/3)' --cache .turbo/cache --run-summaries .turbo/runs`. These are Test Core's step and condition, without `--extra-summaries` (next bullet). - **Save step** "Save Turbo cache (main only)": `if:` ANDs `steps.prune-turbo-cache.outputs.save == 'true'`. The path and key are unchanged. - **Unchanged:** the restore step and its keys, `upgrade-harness-optin.yml` (its restore at L96), the job's `if:`, its timeout, and the two attestation steps, which stay last and unconditioned. Why one summary is enough, and why there is no `--extra-summaries`: the test run plans the whole `^build` closure and replays it, so its summary names every closure `build` task. Measured at base `f2a9007e93` with `turbo run build --filter=@objectstack/dogfood --dry=json` against `OS_TEST_SHARD=1/3 turbo run test --filter=@objectstack/dogfood --dry=json`: 71 shared tasks, 71 hash-identical, 0 differing. The only closure node missing from the test plan is `@objectstack/dogfood#build`, which has no command and no cache entry. Test Core needs `--extra-summaries` only because its closure summary is moved out of `.turbo/runs/` by the remote-cache assertion, and the dogfood job runs no such assertion. No `--site` registry, self-test or gate lists sites. `git grep prune-turbo-cache` outside the workflows finds only the script itself and two `ui/scripts/__tests__` files about the ui jobs. So, as with #27067, nothing else needs to change. ## Tests <details><summary>Local seed-size estimate (closure only), at 4f58391 = base + this diff</summary> The setup stands in for CI. A private `--cache-dir` was seeded with a copy of this container's shared `.turbo/cache` (260 MB, 1059 files) as the "restored seed". Then the job's closure command ran (`--concurrency=2`): `Tasks: 69 successful, 69 total`, `Cached: 2 cached`, `6m39s`, on a shared 4-vCPU box. Then came the test step exactly as this PR spells it, but with an out-of-range `OS_TEST_SHARD=9/3`, so vitest refused at once. That run reported `69 cached, 70 total`, `Failed: @objectstack/dogfood#test`, and wrote its summary anyway, so a red test step still leaves a summary to prune by. Then the prune ran exactly as the new step does: ``` prune-turbo-cache: Dogfood Regression Gate (1/3): kept 69 task entries named by 1 run summary (70 task hash(es); 1 have no entry -- a failed task, or one turbo does not cache). prune-turbo-cache: Dogfood Regression Gate (1/3): dropped 1053 file(s) the run did not use: 304.3 MB -> 35.7 MB on disk. save=true ``` - Before: 304.3 MB. After: 35.7 MB (207 files = 69 entries x 3). - The local "before" is not CI's. CI restores the accumulated ~3.69 GB seed, and the first push run after this lands prunes that down to the same closure, as Test Core's first run did (2912.4 MB -> 0.8 MB). - The scope is the closure only. A real `dogfood#test` entry (`outputs: []`, a log archive) adds a small amount: NOT MEASURED, because a full dogfood shard was not run here. - Estimate: about 36 MB per shard seed, about 0.11 GB for the three shards of one merge. - `--summarize` moves no hash: the summary's 69 build hashes equal the dry-run plan taken without it (69 identical, 0 differing). </details> - Upgrade-harness closure (`upgrade-harness-optin.yml` restores `turbo-dogfood-1-main-*` and builds `--filter=@objectstack/dogfood`): every one of the 69 closure tasks that has a command still has its `.tar.zst` in the pruned seed (69 of 69, 0 missing). - `node scripts/ci/prune-turbo-cache.mjs --self-test` (script unchanged): `prune-turbo-cache self-test: OK (25 case(s) across 5 batteries).`, exit 0. - The `dogfood` job's steps, parsed with the `yaml` package: ... 8 `Upload stall diagnostic reports`, 9 `Prune the Turbo cache to this run's tasks (main only)` (`id=prune-turbo-cache`, `if="always() && github.event_name == 'push'"`), 10 `Save Turbo cache (main only)` (`if="always() && github.event_name == 'push' && steps.prune-turbo-cache.outputs.save == 'true'"`), 11 `Attest this shard ran and passed`, 12 `Publish this shard's attestation`. - No ablation, because this is workflow config and not tested behaviour. ## Acceptance notes - **Before** (pool reading 6647111495, main at `474c030b13`): - First read: 10,925,870,698 B in 27 entries. Second read: 7,745,686,441 B in 16 entries. - The two `Linux-turbo-dogfood-1-main-*` seeds are 3,685,866,556 B and 3,687,370,253 B, together 7.37 GB, or **95.2 %** of the second read. `*-turbo-*` keys overall are 99.8 %. - **Writer correction:** the reading names `upgrade-harness-optin.yml:96` as the source, but that step is `actions/cache/restore` and never saves. The writer is this job's save step: `github.job` is `dogfood`, so its key is `Linux-turbo-dogfood-SHARD-main-SHA`. The harness only restores shard 1's seed. - **After:** owed after at least 5 merges to `main`. It is a maintainer pool reading (`actions/cache/usage` plus the full `actions/caches` listing, in the shares of 6647111495), because the seat token gets 403 on `actions/cache/*`. Until then, the first push run's three `prune-turbo-cache: Dogfood Regression Gate (N/3): …` lines are the measured post-prune seed size. - **Not exercised on this PR:** the `dogfood` job does not run on `pull_request` or `merge_group` (THE TWO SPEEDS), so this PR's own CI cannot exercise the new step. The first run is the first push run on `main` after merge. ## Verification Derived gates (`node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack`), run on `2a737765df` (this branch after merging `origin/main` `ff12aed37`; the derived list was byte-identical before the merge). Each exit code was captured before any pipe. <details><summary>53 derived commands: 53 exit 0, plus the --ran line</summary> - 18 direct `node` gates (closing-keyword parity, comment-mask corpus, self-test-wired, self-test-workflow-commands, step-collectors, whole-set-label-write, assert-turbo-remote-cache / plan-test-core-shards / prune-turbo-cache / scheduled-full-run self-tests, drift-comment, ci-failure self-test, dts-emitted self-test): exit 0. - 35 `pnpm check:*`, including `check:ci-filter-parity`, `check:required-contexts`, `check:shard-attestation`, `check:stall-guard-budget`, `check:stall-guard-headroom`, `check:workflow-status-functions`, `check:workflow-step-name-quoting`, `check:agent-test-spelling`, `check:aggregator-roster`, `check:nul-bytes`, `check:pm-dispatch-gates` (fast tier, 340 s) and `check:type-check-debt` (under the verify lock, re-measure 366 s, 0 raw tsc errors): exit 0. - `check:dual-build-cjs-loads` first exited 3 (PREREQUISITE NOT MET: 7 packages outside the dogfood closure had no `dist/`). It was re-run after `check:type-check-debt` built every package and then exited 0: 116 require entry points across 71 packages load. ``` Run reconciliation — 53 derived, 53 run, 0 NOT-MEASURED, 0 UNRUN. ✓ dispatch-gates --ran: 53 derived famil(ies) accounted for — 53 run, 0 NOT-MEASURED (a DERIVED zero — all 53 recorded an exit code and none of them is 3). ``` `actionlint` is not derived and not installed, so it was not run. CI's Lint & Repo Gates runs the workflow lints. </details> Session: https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- signed: domain:devx#2 · marchtian · session_017ihL2ebbZkr2QAEG1x7UC8 --> Co-authored-by: Claude <noreply@anthropic.com>
1 parent 8eca18f commit 1d41037

1 file changed

Lines changed: 23 additions & 2 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2536,10 +2536,13 @@ jobs:
25362536
# The build step above still builds the closure once, so the closure's
25372537
# `build` tasks REPLAY here (identical hashes: nothing run-level is
25382538
# hashed into them any more) and only `dogfood#test` executes.
2539+
# `--summarize` writes `.turbo/runs/<id>.json` and moves no task hash;
2540+
# it names the closure's `build` tasks and `dogfood#test` alike, and
2541+
# the prune before the seed save keeps exactly those entries.
25392542
export OS_TEST_SHARD="${{ matrix.shard }}/3"
25402543
node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/dogfood.log" --stall-minutes 10 \
25412544
--report-dir "$RUNNER_TEMP/stall-reports" -- \
2542-
pnpm turbo run test --filter=@objectstack/dogfood --log-order=stream
2545+
pnpm turbo run test --filter=@objectstack/dogfood --summarize --log-order=stream
25432546
25442547
# Test leftovers, step 2 of 2 (#23880, ruling 乙): what the dogfood test
25452548
# run added to the working tree. Report only -- exits 0 on every path, no
@@ -2574,10 +2577,28 @@ jobs:
25742577
if-no-files-found: ignore
25752578
retention-days: 14
25762579

2580+
# Keeps only the entries the test step's run summary (`.turbo/runs/`)
2581+
# names, as Test Core's prune step does (#25452): the restore step takes
2582+
# the newest seed by prefix and turbo never evicts its local cache, so a
2583+
# seed saved whole is every seed before it plus this run (~3.69 GB per
2584+
# seed in #25452's pool reading). That one summary names the whole
2585+
# closure -- the test run plans the closure step's `build` tasks with the
2586+
# same hashes -- so no `--extra-summaries` is needed, and the closure
2587+
# `upgrade-harness-optin.yml` restores this seed for stays in it. No
2588+
# summary -- the test step was cancelled or never ran -- means no save,
2589+
# and the previous seed stays the newest. Exits 0 on every run-time
2590+
# path, so it cannot withhold the attestation below.
2591+
- name: Prune the Turbo cache to this run's tasks (main only)
2592+
id: prune-turbo-cache
2593+
if: always() && github.event_name == 'push'
2594+
run: |
2595+
node scripts/ci/prune-turbo-cache.mjs --site 'Dogfood Regression Gate (${{ matrix.shard }}/3)' \
2596+
--cache .turbo/cache --run-summaries .turbo/runs
2597+
25772598
# Seed the shared Turbo cache from main only (see the restore step
25782599
# above); shard-scoped key so the two matrix jobs don't collide.
25792600
- name: Save Turbo cache (main only)
2580-
if: always() && github.event_name == 'push'
2601+
if: always() && github.event_name == 'push' && steps.prune-turbo-cache.outputs.save == 'true'
25812602
uses: actions/cache/save@v6
25822603
with:
25832604
path: .turbo/cache

0 commit comments

Comments
 (0)