Skip to content

Commit 0a69482

Browse files
ci(test-shards): revert Test Core to the plain-sum layout and the 2026-10-23 dataset (undo #25968) (#26077)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} --> _Requested by **Elon** · [project thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9D5ZEhycrEz6EHUjW1whANXQ)_ Fixes #26074 Clause-②: no Metadata-protocol: none ## What This is a clean revert of f36f619 (#25968). It changes two files and nothing else: `scripts/partition-test-shards.mjs` goes back to plain-sum bin placement, and `scripts/test-shard-timings.json` goes back to the 2026-10-23 dataset with cli in 3 slices. Neither file has a later commit on `main`. No other file imports the exports this removes (`WHOLE_LEG_PARALLELISM`, `loadOf`, `wallOf`). ## Why Since the first queue build on the #25968 layout (08:16Z), full Test Core `merge_group` builds have gone red 5 times in 27. On the old layout, 05:00–08:10Z, they went red 0 times in 19. Every red ejects its PR and rebuilds every entry queued behind it, so throughput fell from 12–13 merges an hour to 6. The failing tests are all load-sensitive: - objectql `src/hook-run-as.test.ts` (cold `import('./index.js')`): Test timed out in 5000ms - plugin-security `src/objects/shipped-sets-identity-default-row-scope.test.ts`: Test timed out in 5000ms - cli `test/hook-timeout-override-refusal.test.ts` CONTROL: the `vitest list` child exits 1 The heads that failed passed when they went back in unchanged. The #25870 thread replayed `planRun()` on both trees. Before, five whole-package shards carried about 1866 s of four-wide work each, and plugin-security sat on a cli-slice shard with 899 s. After, four whole-package shards carry about 2141 s each, and objectql and plugin-security share one shard. The suite-duration ceiling annotations show the effect: plugin-security reads 605–611 s against a ceiling of 474, plugin-auth 617–928 s against 600, driver-sql 603–612 s against 451, and objectql 1102 s against 952. ## Cost and rollback The slowest spec-PR shard goes back to about 25–29 min, from about 20. #25870 stays open for a placement that takes contention into account. Rollback means re-landing f36f619 as it is. ## Verification - `node scripts/partition-test-shards.mjs --self-test`: OK. 75 measured packages, 77 items, 8 shards, max/mean 1.00x, bins about 1866 s, all within the 2899.02 s density cap, cli x3. - `node scripts/check-commit-card-trailers.mjs --range origin/main..HEAD`: pass. - The other gates are left to this PR's own CI run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SrpCjPfYv8N9P9EJvoDyN9 Co-authored-by: Claude <noreply@anthropic.com>
1 parent b63fdf7 commit 0a69482

2 files changed

Lines changed: 276 additions & 550 deletions

File tree

‎scripts/partition-test-shards.mjs‎

Lines changed: 20 additions & 156 deletions
Original file line numberDiff line numberDiff line change
@@ -157,14 +157,6 @@ const TIMINGS_PATH = path.join(REPO_ROOT, 'scripts', 'test-shard-timings.json');
157157
// beside them. That last figure is the floor this count cannot move: a CLI
158158
// slice (~960-1010s measured) is serial, so past eight the slowest shard is
159159
// the slice and more shards only add runners.
160-
//
161-
// ⚠ The bins above were placed by their SUM. Since #25870 partition() places
162-
// them by that same wall model (WHOLE_LEG_PARALLELISM), so a slice shard no
163-
// longer carries a whole-package leg ahead of its slice; on the committed
164-
// dataset's full list the predicted steps move from 1277s on the three slice
165-
// shards (643s elsewhere) to 966s on the slice shards and 830s elsewhere. On
166-
// the 2026-10-26 refresh, with the CLI cut into 4 slices (pin 3c, below), all
167-
// eight read ~738s.
168160
export const SHARD_COUNT = 8;
169161

170162
// The acceptance bound #10472 set: the slowest shard within ~1.3x the mean.
@@ -362,23 +354,10 @@ export function timingModeNote(event) {
362354
// n = 2 869.44s 1.52x still the floor
363355
// n = 3 579.63s 1.01x meets -- spec's task is the floor
364356
//
365-
// So n = 3, and pin 3c held it there. Slices were placed at their PLAIN weight
366-
// (partition() unchanged), so every bin kept the density the whole-package
357+
// So n = 3, and pin 3c holds it there. Slices are placed at their PLAIN weight
358+
// (partition() is unchanged), so every bin keeps the density the whole-package
367359
// split had -- see densityCap() for the hard limit on that and its pin.
368360
//
369-
// RE-DERIVED AT 4 (#25870), on the 14-run refresh of 2026-10-26 (newest run
370-
// 40272142430): the CLI at 2697.45s. `@objectstack/spec` (1168.98s) and
371-
// `@objectstack/runtime` (793.70s) each run as two tasks, so the heaviest
372-
// other serial task is `@objectstack/metadata-protocol`'s 670.12s, and a slice
373-
// must sit within 1.3 x 670.12 = 871.16s.
374-
//
375-
// n = 3 899.15s 1.34x still the floor -- pin 3c refused 3
376-
// n = 4 674.36s 1.01x meets -- metadata-protocol is the floor
377-
//
378-
// Since #25870 partition() places a slice at its wall share rather than its
379-
// plain weight (WHOLE_LEG_PARALLELISM); `total`, and so the density cap, still
380-
// reads plain sums.
381-
//
382361
// ⛔ THE COUNT IS CONFIGURED; WHETHER A RUN SLICES IS DECIDED BY THAT RUN. A
383362
// pull_request or merge_group run splits its own affected set, so planShards()
384363
// asks, of the run's own list, whether a configured package is that run's
@@ -399,7 +378,7 @@ export function timingModeNote(event) {
399378
// a change to what that generator can DECODE, which is why the map it replaced
400379
// is kept below.
401380
export const FILE_SHARDED_PACKAGES = Object.freeze({
402-
'@objectstack/cli': 4,
381+
'@objectstack/cli': 3,
403382
});
404383

405384
// THE MAP AS IT STOOD BEFORE ITS LAST CHANGE, read only by the generator's
@@ -427,10 +406,8 @@ export const FILE_SHARDED_PACKAGES = Object.freeze({
427406
// `test-core-run-summary-*`, #25771), so four days after that PR lands no
428407
// retained summary predates the change and this map decodes nothing.
429408
// The outgoing map of #22075's third round was empty -- the runs before it
430-
// carried no slice. #25870 raised the CLI from 3 slices to 4, so this holds 3.
431-
export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({
432-
'@objectstack/cli': 3,
433-
});
409+
// carried no slice -- so this one is empty too.
410+
export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({});
434411

435412
// The item grammar. A shard item is a package (`@objectstack/cli`) or a SLICE
436413
// of one (`@objectstack/cli 1/2`), and this pair of functions is the only place
@@ -905,58 +882,21 @@ function readManifest(dir) {
905882
}
906883
}
907884

908-
// ── A SLICE IS A SERIAL LEG: BINS ARE PLACED BY WALL, NOT BY SUM (#25870) ────
909-
//
910-
// A shard's "Run this shard's tests" step runs its whole packages four at a
911-
// time (`--concurrency=4`) and THEN each file-level slice as a turbo run of its
912-
// own, alone on the runner (ci.yml). So a slice's seconds reach the wall one for
913-
// one while a whole package's are shared about 2.9 ways -- the wall model the
914-
// SHARD_COUNT header calibrated (#25376): step ~ wholeSum / 2.9 + slice. This
915-
// constant is that 2.9, not a new measurement.
916-
//
917-
// Placing bins by their SUM of seconds treated a slice second like a whole
918-
// second, so every slice shard was filled up to the mean with whole packages and
919-
// then ran its slice after them. On the committed dataset's full list that put
920-
// ~900s of whole packages in front of each 966s CLI slice: predicted step 1277s
921-
// on the three slice shards against 643s on the other five, and measured the
922-
// same way on the pull_request runs #25870 tabulates -- 3/8-5/8 at 25-29 min,
923-
// the rest at 7-17, the gap being each slice shard's whole-package leg
924-
// (`Run this shard's tests` minus the slice leg's own turbo `Time:`).
925-
//
926-
// So partition() places by LOAD -- a bin's predicted wall, kept in whole-package
927-
// seconds (a slice counts WHOLE_LEG_PARALLELISM times its weight) so that a list
928-
// with no slice is placed by exactly the same additions and comparisons as the
929-
// plain-sum LPT it replaced, byte for byte. `total` stays the plain sum of
930-
// predicted test windows: the density cap and the drift report are statements
931-
// about that quantity, and neither changes meaning here.
932-
export const WHOLE_LEG_PARALLELISM = 2.9;
933-
934-
// One item's share of a bin's load, in whole-package seconds.
935-
export function loadOf(item) {
936-
return item.slice ? item.weight * WHOLE_LEG_PARALLELISM : item.weight;
937-
}
938-
939-
// A bin's predicted `Run this shard's tests` step under the wall model.
940-
export function wallOf(bin) {
941-
return bin.load / WHOLE_LEG_PARALLELISM;
942-
}
943-
944-
// LPT greedy on load (above): heaviest item into the currently lightest bin.
945-
// Deterministic: input order never matters because both the item sort and the
946-
// bin choice break ties explicitly (by name / by lowest bin index).
885+
// LPT greedy: heaviest package into the currently lightest bin. Deterministic:
886+
// input order never matters because both the package sort and the bin choice
887+
// break ties explicitly (by name / by lowest bin index).
947888
export function partition(items, shardCount) {
948889
const sorted = [...items].sort(
949-
(a, b) => loadOf(b) - loadOf(a) || a.name.localeCompare(b.name, 'en')
890+
(a, b) => b.weight - a.weight || a.name.localeCompare(b.name, 'en')
950891
);
951-
const bins = Array.from({ length: shardCount }, () => ({ total: 0, load: 0, names: [] }));
892+
const bins = Array.from({ length: shardCount }, () => ({ total: 0, names: [] }));
952893
for (const it of sorted) {
953894
let best = 0;
954895
for (let i = 1; i < bins.length; i++) {
955-
if (bins[i].load < bins[best].load) best = i;
896+
if (bins[i].total < bins[best].total) best = i;
956897
}
957898
bins[best].names.push(it.name);
958899
bins[best].total += it.weight;
959-
bins[best].load += loadOf(it);
960900
}
961901
return bins;
962902
}
@@ -965,34 +905,13 @@ export function partition(items, shardCount) {
965905
// `floor` is the heaviest single package: sharding is BY PACKAGE, so no split
966906
// at any shard count can put max below it, and comparing it to the mean says
967907
// whether the bound is even reachable before anyone tries to reach it.
968-
//
969-
// `walls` / `wallRatio` are the same statistic on what the bins were placed by
970-
// (wallOf, #25870). With no slice in the split they are the totals scaled by
971-
// one constant, so the two ratios agree; with a slice, only the wall ratio says
972-
// whether the shards finish together. A bin built by hand (no `load`) is read
973-
// as all whole packages.
974908
export function balanceOf(bins, items = null) {
975909
const totals = bins.map((b) => b.total);
976910
const sum = totals.reduce((a, b) => a + b, 0);
977911
const mean = sum / bins.length;
978912
const max = Math.max(...totals);
979913
const floor = items ? Math.max(0, ...items.map((i) => i.weight)) : null;
980-
const walls = bins.map((b) => wallOf({ load: b.load ?? b.total }));
981-
const wallMean = walls.reduce((a, b) => a + b, 0) / bins.length;
982-
const wallMax = Math.max(...walls);
983-
return {
984-
totals,
985-
sum,
986-
mean,
987-
max,
988-
min: Math.min(...totals),
989-
ratio: mean === 0 ? 1 : max / mean,
990-
floor,
991-
walls,
992-
wallMean,
993-
wallMax,
994-
wallRatio: wallMean === 0 ? 1 : wallMax / wallMean,
995-
};
914+
return { totals, sum, mean, max, min: Math.min(...totals), ratio: mean === 0 ? 1 : max / mean, floor };
996915
}
997916

998917
// ── THE DENSITY CAP: NO SHARD CARRIES MORE THAN THE SPLIT PROVEN GREEN (#22075) ─
@@ -1052,7 +971,7 @@ export function refineToCap(bins, items, cap) {
1052971
const weightOf = new Map(items.map((i) => [i.name, i.weight]));
1053972
const slices = new Set(items.filter((i) => i.slice).map((i) => i.name));
1054973
const movable = (name) => !slices.has(name);
1055-
const out = bins.map((b) => ({ total: b.total, load: b.load, names: [...b.names] }));
974+
const out = bins.map((b) => ({ total: b.total, names: [...b.names] }));
1056975
const EPSILON = 1e-9;
1057976
for (let step = 0; step < 10_000; step++) {
1058977
let h = 0;
@@ -1079,21 +998,15 @@ export function refineToCap(bins, items, cap) {
1079998
}
1080999
if (best === null) break;
10811000
const { to, give, take } = best;
1082-
// Only whole packages move, and a whole package's load is its weight, so
1083-
// `load` follows `total` move for move.
10841001
out[h].names.splice(out[h].names.indexOf(give), 1);
10851002
out[h].total -= weightOf.get(give);
1086-
out[h].load -= weightOf.get(give);
10871003
out[to].names.push(give);
10881004
out[to].total += weightOf.get(give);
1089-
out[to].load += weightOf.get(give);
10901005
if (take !== null) {
10911006
out[to].names.splice(out[to].names.indexOf(take), 1);
10921007
out[to].total -= weightOf.get(take);
1093-
out[to].load -= weightOf.get(take);
10941008
out[h].names.push(take);
10951009
out[h].total += weightOf.get(take);
1096-
out[h].load += weightOf.get(take);
10971010
}
10981011
}
10991012
return out;
@@ -1571,9 +1484,7 @@ const SELF_TEST_BATTERIES = Object.freeze({
15711484
// 8 of these 25 are pins 3b and 3c, the file-level slicing's (#16173), and
15721485
// every one of them runs unconditionally: 3c's fixtures and 3b's cut of the
15731486
// heaviest package do not depend on what the live map slices.
1574-
// 2 of these 27 are pin 2b, the wall placement's (#25870): a fixture, so it
1575-
// runs whatever the dataset holds.
1576-
'the balancing pins (#10472)': 27,
1487+
'the balancing pins (#10472)': 25,
15771488
'predicted-vs-measured drift (#16173)': 9,
15781489
'drift warning tier under the red (#16465)': 12,
15791490
'file-level slice items (#16173)': 20,
@@ -1820,55 +1731,16 @@ function selfTest() {
18201731
});
18211732
const real = datasetPlan.bins;
18221733
const balance = balanceOf(real, datasetItems);
1823-
// Graded on predicted WALL, the quantity the bins are placed by (#25870): with
1824-
// a slice in the split, the bin SUMS are deliberately unequal -- a slice shard
1825-
// carries its serial slice and little else -- and only the walls say whether
1826-
// the shards finish together. With no slice the two ratios are the same number.
18271734
check(() => {
1828-
if (balance.wallRatio > MAX_SHARD_OVER_MEAN) {
1735+
if (balance.ratio > MAX_SHARD_OVER_MEAN) {
18291736
throw new Error(
1830-
`balance: at ${SHARD_COUNT} shards the slowest predicted test step is ${balance.wallRatio.toFixed(2)}x the mean ` +
1831-
`(${balance.wallMax.toFixed(0)}s vs ${balance.wallMean.toFixed(0)}s), past the ${MAX_SHARD_OVER_MEAN}x bound. ` +
1832-
'Walls: ' + balance.walls.map((t) => t.toFixed(0)).join('/') + 's; bins: ' +
1833-
balance.totals.map((t) => t.toFixed(0)).join('/') + 's.'
1737+
`balance: at ${SHARD_COUNT} shards the heaviest bin is ${balance.ratio.toFixed(2)}x the mean ` +
1738+
`(${balance.max.toFixed(0)}s vs ${balance.mean.toFixed(0)}s), past the ${MAX_SHARD_OVER_MEAN}x bound. ` +
1739+
'Bins: ' + balance.totals.map((t) => t.toFixed(0)).join('/') + 's.'
18341740
);
18351741
}
18361742
});
18371743

1838-
// 2b. A SLICE IS PLACED AS THE SERIAL LEG IT IS (#25870). Three shards, a
1839-
// package cut in two (100s a slice) and three whole packages (100s, 50s,
1840-
// 50s). Placed by SUM -- the LPT this replaced -- each slice shard also
1841-
// takes a 50s whole package, which runs four-wide BEFORE the slice and
1842-
// lengthens that shard's step to ~117s. Placed by wall, the slices run
1843-
// alone and the three whole packages share the third shard (~69s). The
1844-
// two placements differ, so a revert to the sum reds here.
1845-
const serial = partition(
1846-
[
1847-
{ name: 's 1/2', weight: 100, slice: { index: 1, count: 2 } },
1848-
{ name: 's 2/2', weight: 100, slice: { index: 2, count: 2 } },
1849-
mk('x', 100),
1850-
mk('y', 50),
1851-
mk('z', 50),
1852-
],
1853-
3
1854-
);
1855-
check(() => {
1856-
const crowded = serial.filter((bin) => bin.names.some((n) => n.startsWith('s ')) && bin.names.length > 1);
1857-
if (crowded.length > 0) {
1858-
throw new Error(
1859-
`slice placement: a slice shard also carries whole packages (${crowded.map((b) => b.names.join('+')).join('; ')}) ` +
1860-
'that run before its serial slice -- are the bins placed by their plain SUM again?'
1861-
);
1862-
}
1863-
});
1864-
check(() => {
1865-
const walls = serial.map(wallOf);
1866-
const expected = [100, 100, 200 / WHOLE_LEG_PARALLELISM];
1867-
if (walls.some((w, i) => Math.abs(w - expected[i]) > 1e-9)) {
1868-
throw new Error(`slice placement: predicted walls ${walls.map((w) => w.toFixed(2)).join('/')}s, expected ${expected.map((w) => w.toFixed(2)).join('/')}s`);
1869-
}
1870-
});
1871-
18721744
// 3. The bound is REACHABLE at this shard count -- the arithmetic #10472
18731745
// asked about when it floated 6 -> 8. No split can put the heaviest bin
18741746
// below the heaviest single package, so once that package exceeds
@@ -3142,8 +3014,7 @@ function selfTest() {
31423014
console.log(
31433015
`partition-test-shards: self-test OK (${datasetPackages.length} measured packages ` +
31443016
`-> ${datasetItems.length} shard items, ${SHARD_COUNT} shards, ` +
3145-
`predicted test-step walls ${balance.walls.map((t) => t.toFixed(0)).join('/')}s, ` +
3146-
`max/mean ${balance.wallRatio.toFixed(2)}x <= ${MAX_SHARD_OVER_MEAN}x, floor ${balance.floor.toFixed(0)}s, ` +
3017+
`max/mean ${balance.ratio.toFixed(2)}x <= ${MAX_SHARD_OVER_MEAN}x, floor ${balance.floor.toFixed(0)}s, ` +
31473018
`bins ${balance.totals.map((t) => t.toFixed(2)).join('/')}s within the ${datasetCap.toFixed(2)}s density cap, ` +
31483019
`file-level slices: ` +
31493020
`${datasetPlan.decisions.filter((d) => d.count > 1).map((d) => `${d.name} x${d.count}`).join(', ') || 'none'})`
@@ -3266,7 +3137,7 @@ function main() {
32663137
{ excluded }
32673138
);
32683139
const mine = bins[shardIndex - 1];
3269-
const { max, mean, ratio, walls, wallRatio } = balanceOf(bins);
3140+
const { max, mean, ratio } = balanceOf(bins);
32703141
// Printed on every shard, not just the imbalanced one, and printed as the
32713142
// RATIO the acceptance bound is written in: the per-shard numbers alone never
32723143
// said whether the split was balanced, which is why #10472's imbalance had to
@@ -3280,13 +3151,6 @@ function main() {
32803151
`density cap ${cap.toFixed(2)}s; ` +
32813152
`${packages - estimated} measured, ${estimated} estimated from test-file count)`
32823153
);
3283-
// What the bins were placed by (wallOf, #25870): the test step each shard is
3284-
// predicted to take, a slice counted as the serial leg it runs as.
3285-
console.error(
3286-
` predicted test-step walls (whole / ${WHOLE_LEG_PARALLELISM} + slice): ` +
3287-
`${walls.map((w) => w.toFixed(0)).join('/')}s, this shard ${wallOf(mine).toFixed(0)}s; ` +
3288-
`max/mean ${wallRatio.toFixed(2)}x`
3289-
);
32903154
// Only a document that states its own slicing prints this, so a Test Core
32913155
// shard's log is unchanged; the nightly's says where its slice count came from.
32923156
if (acrossShards !== null) {

0 commit comments

Comments
 (0)