Run test files in a seeded random order

[businessarshgoyal]

Summary

Implements #595: opt-in randomization of the test file order, seeded so a failing order can be replayed. Tests within a file are not reordered.

New lib/random-order.js holds a Mulberry32 PRNG plus a Fisher-Yates shuffle, so the order is a pure function of a 32-bit seed and stable across platforms and Node.js versions:

export const shuffleFiles = (files, seed) => /* Fisher-Yates, seeded */;

--randomize (or config randomize: true) turns it on; --seed=<integer> implies --randomize and reproduces a previous run. Api generates the seed once in its constructor, so watch mode keeps a stable order across reruns, and passes it to reporters via the run event (plan.randomSeed). The default reporter ends its summary with Test files ran in a random order. Reproduce with --seed=1234567890; the TAP reporter emits the same as a comment.

Ordering hooks in lib/api.js:

 selectedFiles = selectedFiles.toSorted(this.options.sortTestFiles ?? defaultComparator); // parallel runs
+if (this.randomSeed !== null) selectedFiles = shuffleFiles(selectedFiles, this.randomSeed);
 selectedFiles = chunkd(selectedFiles, currentIndex, totalRuns);
...
-selectedFiles = scheduler.failingTestsFirst(...);
+// Running failing test files first would defeat the randomized order.
+if (this.randomSeed === null) selectedFiles = scheduler.failingTestsFirst(...);

Deliberate constraints, all validated in lib/cli.js with tested error messages:

• sortTestFiles must not be configured while randomizing — the two orderings contradict each other.
• When parallel builds are utilized, --randomize requires an explicit --seed, otherwise each build would shuffle differently and files would be dropped or duplicated across chunks.
• --seed must be an integer in [0, 2**32 - 1] and must not be combined with --no-randomize.

Docs: a "Randomizing the test file order" section in docs/05-command-line.md, the two flags in the CLI help output, and randomize/seed in docs/06-configuration.md.

Validation: new test/random-order/ suite — unit tests for shuffle determinism and permutation, plus integration tests over a five-file fixture project asserting the same seed reproduces the order, different seeds diverge, the printed seed replays the run, TAP prints the seed, and source order is unchanged without the flag. npx xo, npx tsc --noEmit and npx test-ava pass; npx tap has one failure (test-tap/reporters/tap.js edge-case log) that also fails on unmodified main with the Node.js version available in my environment.

Coordination: #3458 and #3468 also target this issue. Both additionally randomize within files (touching lib/runner.js and lib/worker/base.js), which the issue discussion treats as a separate concern; this PR keeps the scope to file ordering and adds config support, documentation, the sortTestFiles/parallel-build guards, and a test suite. Happy to fold in anything preferred from those PRs, or close this one if a maintainer prefers a different approach — flagging it so the three aren't reviewed in isolation.

Fixes #595

Written by Devin

[businessarshgoyal]

The one red job here — Node.js (^26, ubuntu-latest) — is not caused by this PR. It is test-tap/reporters/tap.js (“edge cases”) failing because the checked-in tap.edgecases.v26.log records a Node-internal stack location that moved in Node 26.5:

actual:   at: 'compileSourceTextModule (node:internal/modules/esm/utils:355:16)'
expected: at: 'compileSourceTextModule (node:internal/modules/esm/utils:354:16)'

Evidence:

1. Reproduced on unmodified main (bbfd946, clean npm ci) with the same Node the job used (26.8.1 — Acquiring 26.8.1 in the job log):

   $ node -v && npx tap test-tap/reporters/tap.js
   v26.8.1
   not ok 5 - tap reporter - edge cases
     not ok 1 - Output did not match expectation: test-tap/reporters/tap.edgecases.v26.log
   

   Same diff, same subtest, no changes from this branch involved.

2. The line number is purely a Node patch-release artifact:

   $ printf 'do\n' > /tmp/bad.mjs
   $ node -e "import('/tmp/bad.mjs').catch(e => console.log(e.stack))"
   v26.0.0 -> at compileSourceTextModule (node:internal/modules/esm/utils:354:16)
   v26.4.0 -> at compileSourceTextModule (node:internal/modules/esm/utils:354:16)
   v26.5.1 -> at compileSourceTextModule (node:internal/modules/esm/utils:355:16)
   v26.6.0 -> at compileSourceTextModule (node:internal/modules/esm/utils:355:16)
   v26.8.1 -> at compileSourceTextModule (node:internal/modules/esm/utils:355:16)
   

   The snapshot was added in Add Node.js 26 support; drop 25 #3450 (2026-05-16), when 26.x still reported :354; main’s last CI run was 2026-05-17, before 26.5 shipped. Only the Linux jobs run the reporter tests (scripts/ci.sh), which is why just this one job is red.

Everything else on this branch is green under Node 26.8.1 — TEST_AVA_SKIP_WATCH_MODE=1 npx test-ava gives 194 passed / 6 skipped, and npx tap gives { total: 1625, pass: 1624, fail: 1 } with that single pre-existing failure.

I have deliberately not touched tap.edgecases.v26.log, since regenerating an unrelated snapshot just to green this PR would hide the real issue (and would re-break as soon as another Node patch moves the line). If you’d like, the durable fix is a sanitizer alongside the existing ones in test-tap/helper/report.js that strips line/column from node:internal/... frames — happy to send that as a separate PR.