Skip to content

Commit e115841

Browse files
ci(test): check out Test Core and Test UI at depth 1 and fetch only the history they read (#31429)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XL3LLnsMifPUBTUagK6HWvv)_ domain:devx#3 · os-project-manager · session_01FEkQ5ocKVSrMoP2SVG2dfL Refs #22034 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:根 `ci.yml` 里 Test Core 规划(`test-plan`)、Test Core 分片(`test`)、Test UI 分片(`test-ui`)三个 job 不再用 `fetch-depth: 0`,改为深度 1 检出,再加一步只取"当前提交的完整祖先 + main(PR 目标分支不是 main 时再加它)",不取 tag、不取其它分支;`select-shard-packages.sh` 遇到浅克隆先自己补历史,补不到就报错退出;正文附全部 job 读 git 历史的清单。 - 为什么改:`fetch-depth: 0` 每次拉约 1.5k 个分支和 1.7 万个 tag,慢的是这批 ref 而不是 main 的深度;11-03 那次平台变慢时 19 个 Test UI 分片还没跑测试就在 30 分钟墙上被取消。按 #22034 的裁决,第一个 PR 只动这两条分片车道,读约 100 次运行后再动其它 job。 - 风险与回滚:风险是选包结果变化;已在三类事件(PR、merge queue、push)上用真实提交对比全历史克隆,包集合与规划摘要逐字节一致。新步骤三次拉取都失败会让该 job 变红而不是静默跑全量。回滚即还原本 PR。 - 先看:`.github/workflows/ci.yml` 的 `test` job 新步骤 "Fetch this commit's history and the base branch"(注释说明每个读者);`scripts/ci/select-shard-packages.sh` 新的浅克隆守卫段;正文的 Inventory 表。 - 两分钟验证:打开本 PR 的 CI,看 "Plan Test Core shards" 的新步骤日志打印 `shallow=false, no tags`,以及 "Plan this run's Test Core shards" 的 `Affected-set diff base:` 是本 PR 的 merge base。 - 需要你做的:合并本 PR(受管 `.github/workflows/**`,Tier H,须维护者手合)。 ## What changes Three jobs in the root `.github/workflows/ci.yml`, and nothing else in any workflow: 1. **`test-plan` ("Plan Test Core shards"), `test` ("Test Core (N/8)") and `test-ui` ("Test UI (N/16)") check out at depth 1.** Their `fetch-depth: 0` lines (L653, L882, L3825 on `main` `405a78eae`) are gone; `actions/checkout@v7` is left at its default. 2. **One new step in each, right after the checkout, byte-identical in all three: "Fetch this commit's history and the base branch".** It runs `git fetch --no-tags --unshallow origin HEAD_SHA +refs/heads/main:refs/remotes/origin/main`, adding `+refs/heads/BASE:refs/remotes/origin/BASE` when a pull request targets a branch other than `main`. That is HEAD's whole ancestry (the PR's merge base, the queue's `base_sha` and the push's `before` sha are all in it) and the base branch, with no tags and no other heads. Three attempts, then a red step with an `::error::`; it prints the commit count, `shallow=false` and the remote refs it holds. The `test` job's copy carries the comment naming each reader. 3. **`scripts/ci/select-shard-packages.sh` deepens a shallow checkout itself, or exits 1.** On `pull_request`, `merge_group` and `push`, a shallow checkout is deepened by HEAD's sha before any diff base is chosen; a deepen that fails prints an `::error::` and exits 1 before `turbo` runs. `schedule` and `workflow_dispatch` read no history and are left alone. This is the guard for a caller without the step above, because on a shallow clone this reader does not fail, it answers wrong (Inventory row 1). The three comments that said "`fetch-depth: 0` already makes this resolve" now name the fetch step. 4. **Pins moved:** `ui/scripts/__tests__/select-affected-tests.test.ts` asserted `fetch-depth: 0` on the `test-ui` checkout. It now asserts the fetch step sits between the checkout and the select step, deepens a shallow clone, and fetches `main` and the PR's base branch. `select-shard-packages.selftest.sh` gains four cases (below) and its floor moves 32/158 to 36/193. ⛔ Not in this PR, as the order rules: `type-check-ui` (still `fetch-depth: 0`, L3529) and every other job, any `run:` body beyond the fetch, the job graph, any timeout, any other workflow file, `ui/.github/**`. ## Why Card #22034, step 2, first PR, released to `domain:devx` by the maintainer's 「以上都同意」 (recorded on the card as 6718334937). `fetch-depth: 0` uses the refspec `+refs/heads/*` plus `+refs/tags/*`. The card measured that ref set, not main's depth, as the cost (473 s locally against 29-32 s for main alone; CI p99 553 s). `domain:devx#2`'s pointer 6674639434 measured the 2026-11-03 slowdown at 5.6-6.5x on the full-ref fetch against about 3x on shallow fetches, with 19 `Test UI (N/16)` shards cancelled at their 30-minute wall behind 1063-1799 s checkouts. After this lands the seat reads about 100 runs of these three jobs: the checkout cost is now the "Checkout repository" step plus the new fetch step. ## Inventory: every step in the root `ci.yml` that reads git history On `main` `405a78eae`. "Shallow today" is what the reader does on a depth-1 clone with nothing else fetched. Found by reading every `run:` step of all 18 jobs, then every script and package build they call (`scripts/**`, `ui/scripts/**`, `packages/*/scripts/**`), then every test file that runs `git` against the real checkout rather than a fixture repo. | # | Job · step | Reads | Needs | Shallow today | This PR | |---|---|---|---|---|---| | 1 | `test-plan` · Plan this run's Test Core shards; `test` · Compute this shard's package set → `scripts/ci/select-shard-packages.sh` | PR: `refs/remotes/origin/BASE` (fetched by ref if absent), `merge-base(origin/BASE, HEAD)`, `rev-list --count PINNED..MERGE_BASE` (log line only). Queue: `base_sha` (fetched by sha if absent). Push: `before` sha (same). Then `git diff --name-only BASE HEAD` and `turbo ls --affected` with `TURBO_SCM_BASE`. `schedule`/`workflow_dispatch`: nothing. | HEAD's ancestry down to the base, and the base branch tip | PR: base branch fetched, `merge-base` finds no path through the graft, `::warning::` and the FULL list (selects everything, loudly). Queue and push: base fetched by sha, `git diff` answers, and turbo prints `WARNING unable to detect git range, assuming all files have changed` and selects every package, exit 0, no annotation (measured below: 131 packages). **Silent.** | Fetch step deepens first; the script now deepens a shallow clone itself or exits 1 (`::error::`). | | 2 | `test` · Build this shard's dependency closure / Build the sliced package's dependency closure; `test-ui` · Build the @objectstack packages objectui consumes → spec's `build` → `gen:schema` (`packages/spec/scripts/build-schemas.ts`) | `merge-base HEAD origin/main` (fetches `origin/main` at `--depth=1` if absent); when that cannot answer, the committed `authorable-surface.base.json` anchor (its rev fetched at `--depth=1`); the release tag it dates tombstones by (fetched at `--depth=1`); trees at those revs. Only when turbo does not replay spec's build from cache. | HEAD's ancestry and `origin/main` for the merge-base path | Takes the committed-anchor path and prints `ℹ️ shallow history — merge-base HEAD origin/main cannot answer here` (a documented, deliberate fallback, already what every depth-1 job runs). | Fetch step keeps the merge-base path these jobs took under `fetch-depth: 0`. | | 3 | `test-ui` · Select the tests this change affects (inline) → `ui/scripts/select-affected-tests.mjs` | PR: `merge-base refs/remotes/origin/BASE HEAD` (no fetch). Queue: `base_sha`. Then `git diff --name-only --relative BASE...HEAD` (three-dot) and `git ls-files`. Other events: full suite, no history. | HEAD's ancestry and the base branch | PR: no `origin/BASE`, `::warning::` and the full suite. Queue: base absent, `git diff` exits 128, `::warning::` and the full suite. Loud, selects everything. | Fetch step deepens first. No guard added in the inline step: `select-affected-tests-wiring.test.ts` runs that step against the real checkout, and a shallow guard there would red it in every depth-1 checkout. | | 4 | `test` · Run this shard's tests; `test-ui` · Run the ui tests | Suites read the working tree (`git grep`, `git ls-files`) or build fixture repositories; `select-affected-tests-wiring.test.ts` stands in for `merge-base`; `check-lockfile-integrity.test.ts` uses `--base-ref HEAD`. None reads the real repository's history, tags or other branches. | nothing | unaffected | none | | 5 | `type-check-ui` · check:lockfile-integrity (PR only) → `ui/scripts/check-lockfile-integrity.mjs --base-ref origin/BASE` | `git merge-base origin/BASE HEAD`, `git show REV:pnpm-lock.yaml` | HEAD's ancestry and the base branch | Throws "could not find a merge base"; the step is `continue-on-error`, so the alarm reads nothing and the job stays green. **Silent in effect.** | PR 2 (`type-check-ui` keeps `fetch-depth: 0` here). | | 6 | `type-check-ui` · Verify changed en strings were followed by the nine translations → `ui/scripts/check-i18n-en-drift.mjs` | `merge-base HEAD origin/$GITHUB_BASE_REF`, then `origin/main`, then `main`; `git show`/`cat-file` at the base | HEAD's ancestry and `origin/main` (and the PR base) | Exits 1 naming the shallow clone. Loud. | PR 2 | | 7 | `type-check-ui` · Run type-check over the ui packages | `turbo run type-check --filter=...[BASE]`, BASE = the PR's payload `base.sha` or the queue's `base_sha` | HEAD's ancestry down to BASE | Not measured for the `--filter=[...]` form; the same SCM code measured silent-all for `ls --affected` (row 1), so expect every ui package type-checked. | PR 2 | | 8 | `type-check-ui` · Build the @objectstack packages objectui consumes | spec's build, as row 2 | as row 2 | as row 2 | PR 2 | | 9 | `filter` · `dorny/paths-filter` v4.0.3 (already depth 1) | PR: the GitHub API's file list. Push: `before` vs `after`. Queue: `base_sha` vs `head_sha`, base fetched at depth 1, two trees diffed. | two trees, no ancestry | Works; built for depth 1. | none | | 10 | `build-e2e-ui` · check:eager-closure — resolve the base-branch ceiling constants (PR only, already depth 1) | fetches the payload `base.sha` and `origin/BASE` tip at `--depth=1`, `git cat-file -p` the checker at each | two commits, no ancestry | Works; built for depth 1. Failure is `continue-on-error` and the checker exits 2 (loud). | none | | 11 | Every other job that builds spec (`build-core`, `dogfood`, `dogfood-verify`, `temporal-conformance`, `ui-deps`, `lint-ui`, `build-e2e-ui`, `live-e2e-ui`, `ui-published-artifacts`; all already depth 1) | spec's build, as row 2 | as row 2 | Committed-anchor path, as row 2. | none | | 12 | `build-e2e-ui` · The committed SDUI manifest is the one this tree generates | `git diff` of the working tree, `rev-parse HEAD` | nothing | unaffected | none | No step reads tags or another branch outside the ones above. No other script a `ci.yml` step calls reads history: `plan-test-core-shards.mjs`, `partition-test-shards.mjs`, `check-shard-attestation.mjs`, `check-test-completeness.mjs`, `report-test-leftovers.mjs` (`ls-files --others`), `report-test-timings.mjs`, `check-test-suite-ceilings.mjs`, `run-with-stall-guard.mjs`, `assert-turbo-remote-cache.mjs`, `prune-turbo-cache.mjs`, `check-dts-*`, `check-lean-entry-closure.mjs`, `check-sourcemap-no-sources-content.mjs`, `gen-sdui-manifest-node.mjs`, and every `pnpm --dir ui run` script in `lint-ui`/`type-check-ui` other than rows 5-6 (grep for `merge-base`, `log`, `rev-list`, `show REV:`, `cat-file`, `fetch` in each). ## Tests - `bash scripts/ci/select-shard-packages.selftest.sh`: `all 36 cases passed (193 checks)`. New and changed cases: - push and merge_group on a shallow clone: the run prints the deepen line, the clone ends `shallow=false`, and `merge-base(C0, HEAD)` answers (it did not before: a by-sha fetch of the base leaves HEAD grafted); - pull_request, merge_group and push on a shallow clone whose remote is unreachable: exit 1, the `::error::` line, `turbo` never called, no `turbo-ls.json`, no `::warning::`; - schedule on a shallow clone: no deepen, still shallow, the FULL list. - **Ablation** of the guard on the committed head (`node scripts/ablation-replace.mjs --file scripts/ci/select-shard-packages.sh --anchor 'pull_request | merge_group | push)' --replacement 'pull_request-ABLATED)'`): mutation landed (anchor 1 → 0), selftest `SELFTEST FAILED (36 cases, 193 checks)` with 24 FAIL lines; restored, blob equals HEAD, `git diff HEAD` empty. - `pnpm exec vitest run` (from `ui/`) on `select-affected-tests.test.ts`, `select-affected-tests-wiring.test.ts`, `test-file-timings.test.ts`, `check-lockfile-integrity.test.ts`, `check-changeset-presence.test.ts`: 128 passed. Before the pin moved, `select-affected-tests.test.ts` failed on exactly that `fetch-depth: 0` assertion. ## Acceptance notes **The selection does not change: measured on real commits, the new fetch against a full-history clone.** The control is a clone that ran the exact `fetch-depth: 0` refspec (`+refs/heads/*` and `+refs/tags/*`, 10300 refs). The candidate is a fresh depth-1 fetch of the same sha, as `actions/checkout` makes it, then this PR's fetch step extracted from `ci.yml` and run as written. Each runs the tree's own `select-shard-packages.sh` and `plan-test-core-shards.mjs` with the event's variables: | Commit | Event | Diff base | Packages | `turbo-ls.json` md5 | Plan digest | Shards | |---|---|---|---|---|---|---| | #31404's merge ref `71a88bc1b5` (8 files, `packages/rest`) | pull_request | `405a78eae` (merge-base) | 63 / 63 | `2e750807a6b6` both | `3c5802f6…` both | 1-8 both | | same | merge_group, base `405a78eae` | `405a78eae` | 63 / 63 | `2e750807a6b6` both | `3c5802f6…` both | 1-8 both | | `main` `383c35303` | push, before `405a78eae` | `405a78eae` | 122 / 122 | `aab238703101` both | `d97c92ae…` both | 1-8 both | | #31329's merge ref `1a6f01112` (`lint.yml` only) | pull_request and merge_group | `604a2b3b9` | 0 / 0 | `be1dac8b3858` both | `f4a1345e…` both | none | Test UI, same method: `select-affected-tests.mjs` on #31404's merge ref selects `691 of 6653` in both, filter files identical (md5 `062425c4fc43`). On #31329's merge ref it selects `691 of 6651` with diff base `604a2b3b9`, which is what CI's own full-history `Test UI (1/16)` printed on run 42437703944 (job 128600274602). This PR's own CI run is the second reading: its `Plan Test Core shards` and `Test UI` logs come from the new step, and the os-dev report on #22034 compares them with the same full-history control. **Without the fetch, Test Core's reader is silent.** On #31329's merge ref checked out at depth 1, a merge_group run of the old script fetched `base_sha` by sha and then `turbo ls --affected` printed `WARNING unable to detect git range, assuming all files have changed: Git error: fatal: no merge base found` and selected all 131 packages, exit 0 (turbo 2.11.10). That is why the script guard exists. **Fetch cost here.** Local readings on this container, not CI's: the depth-1 fetch took 8-24 s, the new step 63-71 s (index-pack of about 20.6k commits dominates), and the full refspec on top of main 47 s. This container's network is not CI's, so these are no forecast; the ~100-run reading after merge is the measurement. **Readers left for PR 2,** from the table: rows 5 and 7 are silent on a shallow clone, so PR 2 owes them the same fetch step before `type-check-ui` drops `fetch-depth: 0`. Row 7 also diffs from the PR payload's frozen `base.sha`, which `test`'s comment calls the base this selection must never use (#6195): an over-selection on long-open PRs, never a skip. Noted, not filed. **Comments not changed:** `test`'s "WHERE THE AFFECTED DIFF STARTS" block keeps its measurement "in a CI-shaped clone: `fetch-depth: 0` populates `refs/remotes/origin/*`" (a dated reading of why turbo's default base is unsafe, still true of the new clone). `type-check-ui`'s comment at L3587 belongs to PR 2. **Governed surface.** `node scripts/pm/check-governed-merges.mjs --branch HEAD` exit 3: "GOVERNED — a human merge is the review record … landing tier: H(人合)", `.github/workflows/** ×1`, 252 changed lines, under the 3000 threshold. `node scripts/pm/dispatch-gates.mjs --tier`: "no path-derived mandate". ## Verification Gate union derived by `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (70 families for this diff), run on head `65cf47ab`, each exit code captured before any pipe; `--ran`: "70 derived famil(ies) accounted for — 69 run, 1 NOT-MEASURED (1 DERIVED from a recorded exit 3)". - **67 exit 0**, among them `pnpm check:select-shard-packages` (the selftest above), `pnpm check:stall-guard-budget`, `pnpm check:stall-guard-headroom`, `pnpm check:shard-attestation`, `pnpm check:ci-filter-parity`, `node scripts/check-self-test-workflow-commands.mjs`, `node scripts/check-step-collectors.mjs`, `node scripts/ci/plan-test-core-shards.mjs --self-test`, `pnpm check:pm-dispatch-gates` ("2205 cases pass", fast tier) and `pnpm check:type-check-debt` (under `os-verify-lock.sh`, VERDICT command-exit 0). - **NOT MEASURED, no full build in this worktree** (Build Core runs them on this PR): `pnpm check:dual-build-cjs-loads` exit 3 ("Run `pnpm build` first"); `pnpm check:dts-closure` exit 1, its whole list is `missing packages/*/dist/index.d.ts` (the worktree built spec with `OS_SKIP_DTS=1` only). - `pnpm --dir ui exec vitest run scripts/` exit 1: 166 files pass; 4 fail on unbuilt dists (`Failed to resolve entry for package "@objectstack/formula"`, ui package builds, spec declarations). After building the six consumer packages under the lock, 2 of the 4 pass; the other 2 (`check-doc-snippet-types.test.ts`, `spec-symbol-ledger-plugin-detail-7265.test.ts`) read spec's `.d.ts`, which this worktree did not emit. None of the 4 reads `ci.yml` or a changed file; Test UI runs them on this PR. - Outside the derivation: `check-governed-merges --branch HEAD` exit 3 (GOVERNED, Tier H) and `dispatch-gates --tier` (no mandate), both above. - `pnpm lint`: the diff touches no package, so there is no scoped package lint. - `main` moved 12 commits after the branch point (to `0647dabb0`); none touches a file this PR changes, so no merge was taken. `skip-changeset`: `.github/**`, `scripts/ci/**` and `ui/scripts/__tests__/**` ship in no package. ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (not touched) - [x] 2. Every name I added uses the casing for its kind. (one step name, one shell variable) - [x] 3. No formatter ran; `pnpm lint` passes. (no package touched) - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. (publishes nothing: `skip-changeset`) - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. (not touched) - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. (the 维护者速读 block is the lane's required Chinese summary) - [x] The *Before Submitting* list in CONTRIBUTING.md is done. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01FEkQ5ocKVSrMoP2SVG2dfL --- _Generated by [Claude Code](https://claude.ai/code/session_01FEkQ5ocKVSrMoP2SVG2dfL)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 5ee59f5 commit e115841

4 files changed

Lines changed: 226 additions & 26 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 106 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -645,12 +645,33 @@ jobs:
645645
shards: ${{ steps.plan.outputs.shards }}
646646
digest: ${{ steps.plan.outputs.digest }}
647647
steps:
648+
# Depth 1, then the step below: the same history the shards fetch, because
649+
# the selection diffs against the PR's merge base, the queue's base_sha or
650+
# the push's before sha, and every shard re-derives this plan (#22034).
648651
- name: Checkout repository
649652
uses: actions/checkout@v7
650-
with:
651-
# The same history the shards check out: the selection diffs against
652-
# the PR's merge base, the queue's base_sha or the push's before sha.
653-
fetch-depth: 0
653+
654+
# The `test` job's step of the same name, verbatim (its comment says why).
655+
- name: Fetch this commit's history and the base branch
656+
env:
657+
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
658+
run: |
659+
refspecs=('+refs/heads/main:refs/remotes/origin/main')
660+
if [ -n "$PR_BASE_REF" ] && [ "$PR_BASE_REF" != main ]; then
661+
refspecs+=("+refs/heads/$PR_BASE_REF:refs/remotes/origin/$PR_BASE_REF")
662+
fi
663+
head=$(git rev-parse HEAD)
664+
for attempt in 1 2 3; do
665+
deepen=()
666+
if [ "$(git rev-parse --is-shallow-repository)" = true ]; then deepen=(--unshallow); fi
667+
if git fetch --no-tags "${deepen[@]}" origin "$head" "${refspecs[@]}"; then break; fi
668+
if [ "$attempt" = 3 ]; then
669+
echo "::error::Could not fetch $head's history and ${refspecs[*]} after 3 attempts. The package selection reads that history, so this job stops here rather than selecting from a shallow clone."
670+
exit 1
671+
fi
672+
sleep $((attempt * 15))
673+
done
674+
echo "History: $(git rev-list --count HEAD) commit(s) behind HEAD, shallow=$(git rev-parse --is-shallow-repository), no tags, refs: $(git for-each-ref --format='%(refname)' refs/remotes | tr '\n' ' ')"
654675
655676
# Restore only: the `test` shards save this key.
656677
- name: Setup workspace
@@ -875,11 +896,59 @@ jobs:
875896
shard: ${{ fromJSON(needs.test-plan.outputs.shards || '[1, 2, 3, 4, 5, 6, 7, 8]') }}
876897

877898
steps:
899+
# Depth 1 (the default), then the step below fetches the history this job
900+
# reads. NOT `fetch-depth: 0` (#22034).
878901
- name: Checkout repository
879902
uses: actions/checkout@v7
880-
with:
881-
# Full history so `turbo --affected` can diff against the PR base.
882-
fetch-depth: 0
903+
904+
# ⭐ THIS COMMIT'S HISTORY AND THE BASE BRANCH, NOT EVERY REF (#22034).
905+
# `fetch-depth: 0` fetched every head and every tag (`+refs/heads/*` and
906+
# `+refs/tags/*`: about 1.5k heads and 17k tags). That ref set, not the
907+
# depth of main, was the slow part: 473 s locally against 29-32 s for main
908+
# alone, a measured p99 of 553 s on CI, and a 2026-11-03 slowdown that hit
909+
# it 5.6-6.5x against about 3x for shallow fetches, cancelling 19 Test UI
910+
# shards at their wall before a test ran.
911+
#
912+
# What this job reads, and why each is here:
913+
# - HEAD's whole ancestry. The PR's merge base, the queue's `base_sha`
914+
# and the push's `before` sha are all in it, and `git merge-base`,
915+
# `git diff` and `turbo ls --affected` need the path from HEAD to
916+
# them. Without it turbo does not fail: measured on turbo 2.11.10 in
917+
# a depth-1 clone with the base fetched, it printed "unable to detect
918+
# git range, assuming all files have changed" and selected all 131
919+
# packages. select-shard-packages.sh now deepens or fails on a shallow
920+
# clone itself; this step is the normal path.
921+
# - refs/remotes/origin/main, with history: the PR selection's
922+
# merge-base, and spec's build (scripts/build-schemas.ts), which
923+
# anchors its authorable-surface checks on `merge-base HEAD
924+
# origin/main` when that answers and falls back to the committed
925+
# anchor when it does not. A PR into another branch fetches that
926+
# branch too.
927+
# - No tags. The one tag spec's build reads it fetches itself at depth 1.
928+
# `--unshallow` with HEAD's sha as the want deepens the checked-out
929+
# commit, not a branch tip. Three attempts, like the checkout's own fetch,
930+
# then a red step: a selection read from a shallow clone is the silent
931+
# all-packages answer above, never a fallback this job may take quietly.
932+
- name: Fetch this commit's history and the base branch
933+
env:
934+
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
935+
run: |
936+
refspecs=('+refs/heads/main:refs/remotes/origin/main')
937+
if [ -n "$PR_BASE_REF" ] && [ "$PR_BASE_REF" != main ]; then
938+
refspecs+=("+refs/heads/$PR_BASE_REF:refs/remotes/origin/$PR_BASE_REF")
939+
fi
940+
head=$(git rev-parse HEAD)
941+
for attempt in 1 2 3; do
942+
deepen=()
943+
if [ "$(git rev-parse --is-shallow-repository)" = true ]; then deepen=(--unshallow); fi
944+
if git fetch --no-tags "${deepen[@]}" origin "$head" "${refspecs[@]}"; then break; fi
945+
if [ "$attempt" = 3 ]; then
946+
echo "::error::Could not fetch $head's history and ${refspecs[*]} after 3 attempts. The package selection reads that history, so this job stops here rather than selecting from a shallow clone."
947+
exit 1
948+
fi
949+
sleep $((attempt * 15))
950+
done
951+
echo "History: $(git rev-list --count HEAD) commit(s) behind HEAD, shallow=$(git rev-parse --is-shallow-repository), no tags, refs: $(git for-each-ref --format='%(refname)' refs/remotes | tr '\n' ' ')"
883952
884953
- name: Setup workspace
885954
uses: ./.github/actions/setup-workspace
@@ -3817,12 +3886,38 @@ jobs:
38173886
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
38183887

38193888
steps:
3889+
# Depth 1, then the step below: the select step diffs against the base
3890+
# commit, which needs HEAD's ancestry and the base branch in the clone,
3891+
# and nothing else (#22034).
38203892
- name: Checkout repository
38213893
uses: actions/checkout@v7
3822-
with:
3823-
# The select step below diffs against the base commit, which needs
3824-
# the base's history in the clone.
3825-
fetch-depth: 0
3894+
3895+
# The `test` job's step of the same name, verbatim (its comment says why).
3896+
# Here the readers are the select step below (`git merge-base` against
3897+
# origin/<base ref>, then select-affected-tests.mjs's `git diff
3898+
# base...HEAD`) and spec's build. On a shallow clone the select step
3899+
# would not fail: it would warn and run the full suite.
3900+
- name: Fetch this commit's history and the base branch
3901+
env:
3902+
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
3903+
run: |
3904+
refspecs=('+refs/heads/main:refs/remotes/origin/main')
3905+
if [ -n "$PR_BASE_REF" ] && [ "$PR_BASE_REF" != main ]; then
3906+
refspecs+=("+refs/heads/$PR_BASE_REF:refs/remotes/origin/$PR_BASE_REF")
3907+
fi
3908+
head=$(git rev-parse HEAD)
3909+
for attempt in 1 2 3; do
3910+
deepen=()
3911+
if [ "$(git rev-parse --is-shallow-repository)" = true ]; then deepen=(--unshallow); fi
3912+
if git fetch --no-tags "${deepen[@]}" origin "$head" "${refspecs[@]}"; then break; fi
3913+
if [ "$attempt" = 3 ]; then
3914+
echo "::error::Could not fetch $head's history and ${refspecs[*]} after 3 attempts. The package selection reads that history, so this job stops here rather than selecting from a shallow clone."
3915+
exit 1
3916+
fi
3917+
sleep $((attempt * 15))
3918+
done
3919+
echo "History: $(git rev-list --count HEAD) commit(s) behind HEAD, shallow=$(git rev-parse --is-shallow-repository), no tags, refs: $(git for-each-ref --format='%(refname)' refs/remotes | tr '\n' ' ')"
3920+
38263921
38273922
- name: Setup Node.js
38283923
uses: actions/setup-node@v7

‎scripts/ci/select-shard-packages.selftest.sh‎

Lines changed: 67 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
# #10057 guard on a merge group is the one that matters most: an empty
1313
# docs-only group has to select nothing without complaint).
1414
#
15-
# Hermetic and offline: a throwaway upstream + clone (+ a shallow clone) under
15+
# Hermetic and offline: a throwaway upstream + clone (+ shallow clones) under
1616
# $TMPDIR, a fake `pnpm`/`turbo` pair on PATH that answers from control files
1717
# and records every call, and a stub `check-cross-package-test-inputs.mjs`
1818
# inside the fixture that records its argv and appends what it is told to.
@@ -172,6 +172,13 @@ git_q clone -q --depth 1 "file://$UP" "$SHALLOW" 2>/dev/null
172172
# removed from the script entirely.
173173
SHALLOW_PUSH="$FIX/shallow-push"
174174
git_q clone -q --depth 1 "file://$UP" "$SHALLOW_PUSH" 2>/dev/null
175+
# Two more, for the #22034 guard: one whose deepen cannot reach its remote (the
176+
# refusal), and one a full-battery event runs in (no deepen at all).
177+
SHALLOW_OFFLINE="$FIX/shallow-offline"
178+
git_q clone -q --depth 1 "file://$UP" "$SHALLOW_OFFLINE" 2>/dev/null
179+
git_q -C "$SHALLOW_OFFLINE" remote set-url origin "file://$FIX/no-such-upstream"
180+
SHALLOW_SCHEDULE="$FIX/shallow-schedule"
181+
git_q clone -q --depth 1 "file://$UP" "$SHALLOW_SCHEDULE" 2>/dev/null
175182

176183
ZEROS=0000000000000000000000000000000000000000
177184
# Well-formed, and in no fixture repository: the "resolvable shape, absent
@@ -262,7 +269,14 @@ expect_line() {
262269
expect_no_line() {
263270
if grep -qF -- "$1" "$RT/out.txt"; then bad "does not print: $1"; else ok "does not print: $1"; fi
264271
}
272+
# expect_shallow <true|false> <clone> -- what `--is-shallow-repository` says after the run
273+
expect_shallow() {
274+
local got
275+
got=$(git_q -C "$2" rev-parse --is-shallow-repository)
276+
if [ "$got" = "$1" ]; then ok "the clone is shallow=$1 after the run"; else bad "the clone is shallow=$1 after the run" "got: $got"; fi
277+
}
265278

279+
DEEPEN_LINE="This checkout is shallow: fetching HEAD's history (no tags) before choosing a diff base (#22034)."
266280
W_UNION='::warning::Could not union cross-package scans into the affected set; falling back to the full package list for this shard.'
267281
affected_call() { echo "affected argv=[ls --affected --output=json] TURBO_SCM_BASE=$1"; }
268282
full_call='full argv=[ls --output=json] TURBO_SCM_BASE=unset'
@@ -349,14 +363,26 @@ if git_q -C "$SHALLOW_PUSH" cat-file -e "$C0^{commit}" 2>/dev/null; then
349363
else
350364
ok 'precondition: the push shallow clone lacks C0 before the fetch case'
351365
fi
352-
run_case 'push: a before sha absent from a shallow checkout is fetched BY SHA' "$SHALLOW_PUSH" push '' '' '' "$C0"
366+
# #22034: the deepen of HEAD now brings the before sha in (it is HEAD's
367+
# parent here), so the by-sha fetch below it finds the object present. What is
368+
# pinned is that the run deepened, and that turbo diffed from a base the path
369+
# from HEAD reaches: a by-sha fetch alone left C1 grafted, and the real turbo
370+
# then answers "all files changed".
371+
run_case 'push: a shallow checkout is deepened, then diffed from the before sha (#22034)' "$SHALLOW_PUSH" push '' '' '' "$C0"
353372
expect_rc 0
354373
expect_warnings '' ''
374+
expect_line "$DEEPEN_LINE"
375+
expect_shallow false "$SHALLOW_PUSH"
355376
if git_q -C "$SHALLOW_PUSH" cat-file -e "$C0^{commit}" 2>/dev/null; then
356377
ok 'C0 is present after the run (the fetch happened)'
357378
else
358379
bad 'C0 is present after the run (the fetch happened)'
359380
fi
381+
if git_q -C "$SHALLOW_PUSH" merge-base "$C0" HEAD > /dev/null 2>&1; then
382+
ok 'merge-base(C0, HEAD) answers after the run (HEAD is no longer grafted)'
383+
else
384+
bad 'merge-base(C0, HEAD) answers after the run (HEAD is no longer grafted)'
385+
fi
360386
expect_turbo 'affected only, base = before sha' "$(affected_call "$C0")"
361387
expect_packages '1:a'
362388

@@ -538,9 +564,11 @@ if git_q -C "$SHALLOW" cat-file -e "$C0^{commit}" 2>/dev/null; then
538564
else
539565
ok 'precondition: the shallow clone lacks C0 before the fetch case'
540566
fi
541-
run_case 'merge_group: a base_sha absent from a shallow checkout is fetched BY SHA' "$SHALLOW" merge_group '' '' "$C0"
567+
run_case 'merge_group: a shallow checkout is deepened, then diffed from the base_sha (#22034)' "$SHALLOW" merge_group '' '' "$C0"
542568
expect_rc 0
543569
expect_warnings '' ''
570+
expect_line "$DEEPEN_LINE"
571+
expect_shallow false "$SHALLOW"
544572
if git_q -C "$SHALLOW" cat-file -e "$C0^{commit}" 2>/dev/null; then
545573
ok 'C0 is present after the run (the fetch happened)'
546574
else
@@ -586,14 +614,48 @@ expect_packages '1:a'
586614
expect_line "Affected-set diff base: $C1 (the merge group's base_sha)"
587615
expect_no_line "this push's before sha"
588616

617+
# ── a shallow checkout (#22034) ─────────────────────────────────────────────
618+
# The jobs check out at depth 1 and fetch HEAD's history in their own step; the
619+
# script's deepen is the guard for a caller without that step. A deepen that
620+
# cannot reach the remote is a red step that never reaches `turbo`, on every
621+
# diffing event: on a shallow clone the full-list fallback would be turbo's
622+
# own silent "all files changed", read as a decision.
623+
for ev in pull_request merge_group push; do
624+
reset_controls
625+
case "$ev" in
626+
pull_request) run_case "$ev: a shallow checkout whose deepen fails refuses, red, before turbo" "$SHALLOW_OFFLINE" "$ev" main "$C0" '' ;;
627+
merge_group) run_case "$ev: a shallow checkout whose deepen fails refuses, red, before turbo" "$SHALLOW_OFFLINE" "$ev" '' '' "$C0" ;;
628+
push) run_case "$ev: a shallow checkout whose deepen fails refuses, red, before turbo" "$SHALLOW_OFFLINE" "$ev" '' '' '' "$C0" ;;
629+
esac
630+
expect_rc 1
631+
expect_line "$DEEPEN_LINE"
632+
expect_line '::error::This checkout is shallow and HEAD'"'"'s history could not be fetched'
633+
expect_warnings '' ''
634+
expect_turbo 'never called' ''
635+
expect_union 'not invoked' ''
636+
if [ -e "$RT/turbo-ls.json" ]; then bad 'no turbo-ls.json is written'; else ok 'no turbo-ls.json is written'; fi
637+
expect_shallow true "$SHALLOW_OFFLINE"
638+
done
639+
640+
# The full battery reads no history, so a shallow checkout is left as it is.
641+
reset_controls
642+
run_case 'schedule: a shallow checkout is not deepened -- the FULL list reads no history' "$SHALLOW_SCHEDULE" schedule '' '' '' ''
643+
expect_rc 0
644+
expect_warnings '' ''
645+
expect_no_line "$DEEPEN_LINE"
646+
expect_shallow true "$SHALLOW_SCHEDULE"
647+
expect_turbo 'full only' "$full_call"
648+
expect_packages '3:a,b,c'
649+
589650
# ── Verdict ─────────────────────────────────────────────────────────────────
590651
# #4690: a battery that ran nothing is a failure, never a pass.
591652
# The floor RATCHETS: it was 20/80 when this battery covered three events, and
592653
# it is raised here to what push's own diff base and the two full-battery
593654
# events actually register (#16467). It is a floor, never a target -- a run
594655
# below it means cases stopped running, and the remedy is to find which,
595-
# never to lower the number.
596-
if [ "$cases" -lt 32 ] || [ "$checks" -lt 158 ]; then
656+
# never to lower the number. Raised again for the shallow-checkout guard
657+
# (#22034): 32/158 -> 36/193.
658+
if [ "$cases" -lt 36 ] || [ "$checks" -lt 193 ]; then
597659
echo "SELFTEST FAILED: only $cases case(s) / $checks check(s) ran -- the battery is short"
598660
exit 1
599661
fi

‎scripts/ci/select-shard-packages.sh‎

Lines changed: 37 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,10 @@
1313
# Environment in, `$RUNNER_TEMP/turbo-ls.json` out (a `turbo ls --output=json`
1414
# payload, ready for partition-test-shards.mjs), `::warning::` lines out for
1515
# every fallback. Exit 0 on every decided outcome -- a fallback to the full
16-
# list is a decision, not a failure; exit 2 on a usage error; and a failing
17-
# `turbo ls` exits with turbo's own status, exactly as it did under `bash -e`
18-
# when this text lived inline in ci.yml (see below).
16+
# list is a decision, not a failure; exit 2 on a usage error; exit 1 when the
17+
# checkout is shallow and HEAD's history cannot be fetched (#22034, below); and
18+
# a failing `turbo ls` exits with turbo's own status, exactly as it did under
19+
# `bash -e` when this text lived inline in ci.yml (see below).
1920
#
2021
# ## What decides the set, per event (#16453)
2122
#
@@ -99,13 +100,40 @@ PUSH_BEFORE_SHA="${OS_SHARD_PUSH_BEFORE_SHA:-}"
99100
# zeroes, and this value decides whether a run tests the workspace or a diff.
100101
ZERO_SHA='0000000000000000000000000000000000000000'
101102

103+
# ── A SHALLOW CHECKOUT IS DEEPENED FIRST, OR THE STEP FAILS (#22034) ───────
104+
# The jobs that run this script check out at depth 1 and fetch HEAD's history
105+
# and the base branch in their own step ("Fetch this commit's history and the
106+
# base branch" in ci.yml), instead of `fetch-depth: 0`'s every head and tag.
107+
# This is the guard for the day that step is missing. On a shallow clone the
108+
# readers below do not fail, they answer wrong: with the base present but the
109+
# path from HEAD to it cut by the graft, `turbo ls --affected` printed "unable
110+
# to detect git range, assuming all files have changed" and selected every
111+
# package, exit 0 (measured on turbo 2.11.10, a depth-1 merge ref, 131
112+
# packages), and the pull_request merge-base takes the full-list fallback.
113+
#
114+
# So every event that diffs deepens HEAD here, by its sha, without tags. A
115+
# deepen that fails exits 1 with an `::error::`, not the full-list fallback:
116+
# on a shallow clone that fallback would be the same wrong answer, read as a
117+
# decision. The full-battery events read no history and are left alone.
118+
case "$EVENT_NAME" in
119+
pull_request | merge_group | push)
120+
if [ "$(git rev-parse --is-shallow-repository)" = true ]; then
121+
echo "This checkout is shallow: fetching HEAD's history (no tags) before choosing a diff base (#22034)."
122+
if ! git fetch --no-tags --quiet --unshallow origin "$(git rev-parse HEAD)"; then
123+
echo "::error::This checkout is shallow and HEAD's history could not be fetched, so no diff base can be trusted here: on a shallow clone \`turbo ls --affected\` selects every package without failing. Refusing to select (#22034)."
124+
exit 1
125+
fi
126+
fi
127+
;;
128+
esac
129+
102130
SCM_BASE=''
103131
if [ "$EVENT_NAME" = "pull_request" ]; then
104132
if [ -z "$BASE_REF" ]; then
105133
echo "::warning::This pull_request event carries no base branch, so the affected-set diff base cannot be computed."
106134
else
107-
# `fetch-depth: 0` above already makes this resolve — the fetch is
108-
# the guard for the day that changes, not the normal path.
135+
# The job's own fetch step already makes this resolve (#22034) — the
136+
# fetch is the guard for the day that changes, not the normal path.
109137
#
110138
# `git cat-file -e` rather than the more idiomatic strict
111139
# `git rev-parse --verify` spelling. That is history rather than
@@ -151,8 +179,9 @@ if [ "$EVENT_NAME" = "merge_group" ]; then
151179
if [ -z "$MERGE_GROUP_BASE_SHA" ]; then
152180
echo "::warning::This merge_group event carries no base_sha, so the affected-set diff base cannot be computed."
153181
else
154-
# The checkout's `fetch-depth: 0` already makes this resolve -- the fetch
155-
# is the guard for the day that changes, exactly as it is above. Fetched
182+
# HEAD's history (the job's fetch step, or the deepen above) already
183+
# holds it -- the fetch is the guard for the day that changes, exactly as
184+
# it is above. Fetched
156185
# by sha, not by ref: a merge group's base is a commit, and the queue
157186
# branch it sits on is transient.
158187
if ! git cat-file -e "$MERGE_GROUP_BASE_SHA^{commit}" 2>/dev/null; then
@@ -184,7 +213,7 @@ if [ "$EVENT_NAME" = "push" ]; then
184213
elif [ "$PUSH_BEFORE_SHA" = "$ZERO_SHA" ]; then
185214
echo "::warning::This push event's before sha is all zeroes -- the branch's first push, or a force-push that rewrote history -- so there is no previous commit to diff from."
186215
else
187-
# `fetch-depth: 0` already makes this resolve; the fetch is the guard for
216+
# HEAD's history already holds it (#22034); the fetch is the guard for
188217
# the day that changes. By sha, as on merge_group: after a force-push the
189218
# previous tip may be unreachable from any ref.
190219
if ! git cat-file -e "$PUSH_BEFORE_SHA^{commit}" 2>/dev/null; then

0 commit comments

Comments
 (0)