Skip to content

[Decision] Merge queue: stop the remaining test shards once one shard is red (merge_group only), which narrows ruling 6139629628 A1 inside the queue #31517

Closed

Description

@objectstack-fleet

coordinator#1 · os-steve · session_01ADHkmRWWb3MUYRQZiCKQcX

Filing gate ②, a decision only the maintainer can make. It comes from recommendation 2 of the 2026-11-08 CI assessment (window 2026-11-01..07). The maintainer answered that assessment in the project thread, verbatim: 「同意,立卡」 · 「高优先」 (2026-11-08T12:30Z). The assessment did not cite ruling 6139629628 (#23177, letter B, item A1), and this change narrows A1 inside the merge queue, so the card asks for that one point instead of reading the 「同意」 as covering it. ⛔ Not a claim.

维护者速读

  • 问题:合并队列里一个测试分片已经红了,其余分片还会跑完(Test Core 最慢片约 19 分钟),聚合检查要等它们跑完才报红,排在后面的条目也就晚这么久才重建。
  • 冲突:评估建议只在合并队列(merge_group)上给 Test Core、Test UI、Dogfood 三个矩阵开 fail-fast。[Decision] 门禁过严拖慢发版:一轮 CI 暴露全部红、成本型棘轮加容差带、门禁瘦身审计 #23177 的裁决 A1(「一轮 CI 暴露全部红」)要求一处红不挡住其它检查。开了 fail-fast,队列里就只看得到第一个红分片。PR 上的运行不变,仍然一轮看全。
  • 要你选:A 只在队列里开 fail-fast(推荐);B 保持现状。
  • 收益有上限:它只管 CI 的测试矩阵。11-06 最大的一次级联红(65 次队列构建)出在 Lint 的 Test-source alias gate,A 帮不上那一次。

Background (measured, Actions REST, all CI and Lint runs 2026-11-01..07)

  • 1,840 merge_group runs of each workflow for 1,415 merged commits. Red: CI 125, Lint 121. Of the 225 red queue builds, 66 were roots and 157 cascades (a build whose parent group was already red).
  • Test Core shard medians on a 2026-11-06 sample: 1/8 14.6, 2/8 10.9, 3/8 15.1, 4/8 19.0, 5/8–8/8 9.7–10.2 min. An early red in a fast shard waits for the slowest one before Test Core reads red.

Governing text

Premises, each with a re-check

  1. A real red still reads red under fail-fast. Re-check: the ci.yml note above and run 30271824408.
  2. The mechanism must be the matrix's own fail-fast, never a run-level cancel: a matrix that is cancelled with zero attestations passes its aggregate (CI: Dogfood Regression Gate 把 cancelled 当失败 —— 每次连续推送都产生一条假红 #3668). Re-check: node scripts/check-shard-attestation.mjs --self-test.
  3. PR runs stay complete: the expression is false for pull_request, push and schedule. Re-check: the event list in the PR diff.

Options

Option What it does What people notice Work
A. Queue-only fail-fast fail-fast: ${{ github.event_name == 'merge_group' }} on test, test-ui, dogfood A red queue entry is ejected minutes to 10+ minutes sooner, and the entries behind it rebuild sooner. In the queue run the author sees the first red shard only; their PR run already showed every red. domain:devx · 1 Tier H PR on ci.yml (plus self-test fixtures if they pin fail-fast: false) · no precedent
B. Keep Nothing Every red queue entry holds the queue until its slowest shard ends none

In business terms. A: when the smoke alarm goes off in one room, evacuate now instead of waiting for every room's inspection. B: finish every inspection first, then evacuate.

os-decision-facets
① 项目长远合理性: A 把「一轮看全」留给 PR(作者需要它的地方),队列只负责尽快判出能不能合;主流合并队列(GitHub merge queue、Bors、Mergify)都把队列构建当作快速判定。
② 实际业务拉动: 一周 125 次 CI 队列红,每次后面最多 10 个条目等它;今天就有人在等。
③ 防 AI 犯错: 真红仍是响亮的红(主导聚合检查);风险是有人把它改成取消整次运行,那会静默放行(#3668),所以执行时只能用矩阵自带的 fail-fast。
④ 创业阶段不扩散: 改一个表达式,三处,不加门禁。
Prior rulings read: "fail-fast" (105 hits), "一轮 CI 暴露全部红" → #23177 / 6139629628; "cancelled" → #3668; ADR none; thread: the 2026-11-08 assessment thread.

Recommendation: A. Only facet ① was used to pick it: the queue's job is a fast mergeable/not-mergeable answer, and the complete red list belongs to the PR run. Do ②③④ flip it: no. Confidence gap: the minutes saved per red were not measured (they depend on which shard reds first); the first week after landing is the reading.

After the ruling

A → the domain:devx seat files nothing new: this card becomes the execution card (pm:queue, p1). One Tier H PR, GOVERNED_APPROVERS, and the maintainer's approval for ci.yml. The PR re-anchors any self-test fixture that pins fail-fast: false (check-shard-attestation.mjs, check-aggregator-roster.mjs, partition-test-shards.mjs) and amends the A1 wording in the ci.yml comments to say the queue is the exception. Reading after landing: queue time from first red shard to Test Core red, 7 days, on this card.
B → close as not planned.

Dedupe

Search 2026-11-08 12:3xZ: "fail-fast merge_group" 8 hits, "fail-fast matrix" 28, ""fail-fast"" 105, all unrelated or closed (#23177 and #23191 are the ruling and its execution). New.

Activity

  1. added
    toolingFix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface
    priority:p1High: required for production / M2
    needs-user-decisionBlocked on a maintainer decision — do not dispatch; decision cards only, never on a PR
    domain:devxDomain lane — seat card indexed by label:pm:seat
    area:devpathThe road — create, dev, verify, publish/install, connect an agent, iterate
    on Nov 8, 2026
  2. objectstack-fleet commented on Nov 9, 2026

    @objectstack-fleet
    ContributorAuthor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Ruling: #31517 · letter A · maintainer 「31517 … 同意」 2026-11-09T01:23Z

    The maintainer asked triage for an explanation (「具体解释 #31517…」), and then answered 「31517 31115 29471 31505 同意」 to triage's recommendation, A.


    Generated by Claude Code

  3. added
    pm:queueReady for the PM dispatch loop
    pm:dispatchedDispatched to a dev agent by /pm-dispatch
    and removed
    needs-user-decisionBlocked on a maintainer decision — do not dispatch; decision cards only, never on a PR
    pm:queueReady for the PM dispatch loop
    on Nov 9, 2026
  4. objectstack-fleet commented on Nov 9, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:devx#2 · zhuangjianguo · session_01QsR2jTo666XXx9e1Gm21C6

    Claim: PM loop round 9 · 2026-11-09T01:27Z
    Session: session_01QsR2jSkaG5iii6WC7w4qSX (card thread, dev) · seat session_01QsR2jTo666XXx9e1Gm21C6
    Account: zhuangjianguo
    Branch: claude/project-thread-c9oqlr
    Worktree: objectstack-issue-31517
    Domain: domain:devx
    Seat: domain:devx#2
    File surface: .github/workflows/ci.yml (strategy.fail-fast on test, test-ui, dogfood, and the A1 comments), and the self-tests and fixtures that read them (scripts/check-shard-attestation.mjs, check-aggregator-roster.mjs, partition-test-shards.mjs, and any other a grep finds). Tier H, hand-merged. ⛔ No run-level cancel, no concurrency change, no other workflow, no ui/.github/**.
    Container & model: S, mode:thread, model: default tier (no path-derived mandate), effort: high
    Clause-②: no
    Metadata-protocol: none
    Responsibility: merge-queue test matrices | a red shard waits for the slowest sibling (Test Core 4/8 median 19.0 min) before the aggregate reds | every queue entry behind a red one
    Thread-read: 6730013883
    Premise: holds; ruling A recorded in 6730013883, and fail-fast: false stands on the three matrices on main
    Serial constraints cleared: PR #31611 (#31519) and PR #31644 (#22034) landed in ci.yml. The dev confirms that a fail-fast-cancelled queue run is never read as green by --queue-verdict or the borrowed Test UI decision.

    Scope: one PR as ruled. The PR says Fixes #31517. Brief: /mnt/project-files/issue-31517/brief.md. Summons: ruling 6730013883 and mailbox notice 6730034988.

  5. objectstack-fleet commented on Nov 9, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:devx#2 · zhuangjianguo · session_01QsR2jSkaG5iii6WC7w4qSX
    os-dev-report

    {
      "issue": 31517,
      "status": "done",
      "branch": "claude/project-thread-c9oqlr",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/31753",
      "head": "0a383d677bd8bd31bdb92f4778c644ced0bce6f9",
      "session": "session_01QsR2jSkaG5iii6WC7w4qSX",
      "premise_still_valid": true,
      "summary": "Ruling A executed: strategy.fail-fast is ${{ github.event_name == 'merge_group' }} on test, test-ui and dogfood in ci.yml; pull_request, push, schedule and workflow_dispatch keep every shard running. The A1 comments now say the queue is the exception across shards and why. check-shard-attestation's self-test gains an 8-case battery for the fail-fast shape (floor 25 -> 26) using the measured cancel annotation of run 30271824408 (check run 89995981782: 'The strategy configuration was canceled because \"matrix._1\" failed'); ci-shard-matrix's fixture spells the new expression. Confirmed without change: queueRunVerdict covers only a success queue run and testUiFullGreen counts only success shards, so a fail-fast queue run is never borrowed by a push.",
      "files_changed": [".github/workflows/ci.yml", "scripts/check-shard-attestation.mjs", "scripts/ci-shard-matrix.mjs"],
      "line_budget": "97 changed lines (+90 / -7), 3 files, under the 3000 threshold",
      "tests": "At 0a383d67: check-shard-attestation --self-test exit 0 (350 assertions), live exit 0; ci-shard-matrix --self-test exit 0 (27 cases); check-aggregator-roster --self-test and live exit 0; partition-test-shards --self-test exit 0; scheduled-full-run --self-test exit 0; ui/scripts/__tests__ select-affected-tests-wiring, select-affected-tests, test-file-timings: 3 files, 95 tests passed. Mutation probes on the committed head 6903a73d: classifier reading the fail-fast text as a supersession -> exit 1 (2 #31517 cases red); judge routing failure into the cancelled branch -> exit 1 (1 #31517 case red); both restored, blob hash equal to HEAD.",
      "gates": "dispatch-gates --commands at 0a383d67: 63 derived, 63 run, all exit 0; --ran: '63 derived famil(ies) accounted for — 63 run, 0 NOT-MEASURED'. First run at 6903a73d: 59 exit 0, 4 exit 3 (dts-closure, dual-build-cjs-loads, lean-entry-closure, sourcemap-no-sources-content: no dist); type-check-debt built the workspace there, so the final run reads a dist built at 6903a73d. Body gates on the PR body: check-changeset-no-major, check-adr-0087-registration, check-closing-keyword-parity, check-partof-closing-keyword, all exit 0. PR check-runs: in_progress at report time.",
      "dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
      "merge_tree": "git merge-tree HEAD origin/main exit 0 before merging 90cea476; no parallel fail-fast change on main in the touched files",
      "deviations": "none",
      "mcp_calls": "6 — issue_read x3 (get, get_comments x2), actions_list list_workflow_jobs x1, get_check_run x1, pull_request_read get x1; all reads, no MCP write",
      "api_writes": "2 relay writes — pr_create (fleet-write run 42551655351, read back identical) and this post-stamped comment; plus git push x2; 2 REST reads of check-run annotations (89995981782, 89995981757)",
      "open_questions": [],
      "security_findings": "none",
      "out_of_scope_findings": ["carrier: none · noted, not filed: a matrix whose every shard is killed by its own timeout before any attests reads cancelled with zero attestations and passes by #3668 (ruled not extended on #22653); fail-fast does not widen it"],
      "owed_after_landing": "devx seat: 7-day reading on #31517 of merge_group queue time from the first red shard to the aggregate red, against the 7 days before"
    }
  6. objectstack-fleet commented on Nov 9, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:devx#2 · zhuangjianguo · session_01QsR2jTo666XXx9e1Gm21C6

    PR #31753 is ACCEPTed and waits on the maintainer's hand merge (Tier H): head 0a383d677bd8bd31bdb92f4778c644ced0bce6f9, ACCEPT 6731174723, contract review PASS 6731165523 (nits only). The queue's test, dogfood and test-ui matrices become fail-fast on merge_group only, by the matrix's own fail-fast, as ruled. When it lands, this seat writes the landing note and owes the 7-day queue reading, including the count of queue runs whose aggregate passed on a cancelled zero-attestation matrix (expected 0).

  7. objectstack-fleet commented on Nov 9, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:devx#2 · zhuangjianguo · session_01QsR2jTo666XXx9e1Gm21C6

    Landed · PR #31753 was merged by the maintainer (os-zhuang) through the merge queue as e253f3ab09717234fedae8e4d7a193cd1a12b0da. It is Tier H. The seat posted ACCEPT 6731174723 on head 0a383d67, and the contract review PASS is 6731165523, nits only. From this commit on, the test, dogfood and test-ui matrices are fail-fast on merge_group only, through each matrix's own fail-fast, as ruling A (6730013883) says. Every other event still runs every shard. A fail-fast queue run concludes failure, so a push never borrows it.

    Owed by this seat on this card after 2026-11-16T03:45Z, the 7-day after-reading:

    1. On red queue runs, the queue wall from the first red shard to the aggregate red, and the runner minutes, compared with the pre-change baseline (a red shard waited for the slowest sibling, Test Core 4/8 median 19.0 min).
    2. The count of merge_group runs whose Test Core or Dogfood Regression Gate passed on a cancelled matrix with zero attestations (expected 0, from contract-review nit N3).

    The record's nits N1 and N2 (comment wording and wrap in ci.yml) ride the next ci.yml touch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratedomain:devxDomain lane — seat card indexed by label:pm:seatpriority:p1High: required for production / M2toolingFix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions