You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a6bd648
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: README.md
+26-26Lines changed: 26 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
> The apps, generators, and bench runner in this repository were generated with [Claude Fable 5](https://www.anthropic.com/news/claude-fable-5-mythos-5) and reviewed by the Vitest team. The results below were measured by hand on the hardware stated in each section.
4
4
5
-
Generated reference apps for measuring Vitest performance. Each app models a real category of project — tiny utility packages, libraries, barrel-file graphs, DOM component suites, dependency-heavy services, a 1300-module monolith, a long-haul DOM suite that ages its workers — and the bench runner measures the options that move run time against each of them: `pool`, `environment` (jsdom, happy-dom and headless Chromium via browser mode), `isolate`, `fsModuleCache`, `maxWorkers`, cold vs warm caches.
5
+
Generated reference apps for measuring Vitest performance. Each app stands for one kind of project: a tiny utility package, a library, barrel-file graphs, DOM component suites, a dependency-heavy service, a 1300-module monolith, and a long-running DOM suite. The bench runner measures the options that change run time on each of them: `pool`, `environment` (jsdom, happy-dom, and headless Chromium via browser mode), `isolate`, `fsModuleCache`, `maxWorkers`, and cold vs warm caches.
Link the local build into the workspace instead of pointing `--vitest` at its binary. Dependencies of the apps such as `@testing-library/jest-dom/vitest` import `vitest` themselves, and Node resolves that import to the pinned install, so`--vitest` alone runs the tests against one `expect` and registers matchers on another (every jsdom cell fails with `Invalid Chai property`). The browser cells need the matching `@vitest/browser-playwright` as well.
27
+
Link the local build into the workspace instead of pointing `--vitest` at its binary. Dependencies such as `@testing-library/jest-dom/vitest` import `vitest` themselves, and Node resolves that import to the pinned install. With`--vitest` alone the tests use one `expect` and the matchers register on another, so every jsdom cell fails with `Invalid Chai property`. The browser cells also need the matching `@vitest/browser-playwright`.
28
28
29
29
Add overrides to `pnpm-workspace.yaml` (do not commit them):
pnpm bench --label branch # picks up the linked build, prints its version
45
45
```
46
46
47
-
Run `pnpm build` in the vitest repository before measuring: the link points at `dist/`. The linked build resolves `vite` from the vitest repository, not from the pin in this workspace, so check that both versions match when comparing Vite-sensitive cells. To go back to the pinned release, remove the overrides and the symlink and run `pnpm install` again.
47
+
Run `pnpm build` in the vitest repository before measuring, because the link points at `dist/`. The linked build resolves `vite` from the vitest repository, not from the pin in this workspace, so check that both versions match before comparing Vite-sensitive cells. To return to the pinned release, remove the overrides and the symlink and run `pnpm install` again.
48
48
49
49
### `bench` options
50
50
51
51
| option | values | default |
52
52
|---|---|---|
53
53
| `--apps` | comma-separated app names | all apps |
54
-
| `--matrix` | `quick` (1-2 cells per app), `default` (curated cells below), `full` (whole cross product — use with `--apps`) | `default` |
54
+
| `--matrix` | `quick` (1-2 cells per app), `default` (curated cells below), `full` (whole cross product, use with `--apps`) | `default` |
55
55
| `--runs` | timed reps per cell, median reported | `3` |
56
56
| `--label` | name of the result file, `results/<label>.json` | `local` |
57
57
| `--vitest` | path to a `vitest.mjs` binary (or `VITEST_BIN` env); only changes the binary, see [Benchmarking a local vitest build](#benchmarking-a-local-vitest-build) | the pinned install |
58
-
| `BENCH_FS_CACHE_MODE` | `stable` \| `experimental` — where the fs-cache option lives; auto-detected from the vitest version | auto |
58
+
| `BENCH_FS_CACHE_MODE` | `stable` \| `experimental`, where the fs-cache option lives; auto-detected from the vitest version | auto |
59
59
60
-
`cold`cells wipe every persistent cache (Vite deps/transform caches, vitest cache dirs, fs module cache) before each timed rep — what fresh CI pays. `warm` cells wipe once, prime with an untimed run, then measure — what repeated local runs pay. The host's `NODE_COMPILE_CACHE` is cleared either way; whatever a vitest version enables itself is part of its measurement.
60
+
`cold`cells wipe every persistent cache (Vite deps and transform caches, vitest cache dirs, fs module cache) before each timed rep. This is what a fresh CI run pays. `warm` cells wipe once, run once untimed to prime the caches, then measure. This is what repeated local runs pay. The host's `NODE_COMPILE_CACHE` is cleared in both cases; whatever a vitest version enables itself is part of its measurement.
61
61
62
62
### Running a single cell by hand
63
63
64
-
Every app is a normal standalone Vitest project. The committed configs read `BENCH_*` variables (see [tools/config/bench-config.js](tools/config/bench-config.js)), so any cell reproduces with plain `vitest run`:
64
+
Every app is a normal standalone Vitest project. The committed configs read `BENCH_*` variables (see [tools/config/bench-config.js](tools/config/bench-config.js)), so any cell can be reproduced with plain `vitest run`:
65
65
66
66
```sh
67
67
cd apps/design-system
@@ -78,15 +78,15 @@ BENCH_BROWSER=true pnpm test # headless Chromium
78
78
| `BENCH_MAX_WORKERS` | a number or a percentage like `50%` |
Apple M4 (10 cores), node v24.13.0. Whole-process wall clock of `vitest run`, median of 3 reps, both versions measured back to back on the same machine. 4.1.10 is the pinned install on vite 8.1.4; 5.0.0-rc.2 is a local build (with vitest-dev/vitest#11078) linked as described above, resolving vite 8.0.11 from the vitest repository. Regenerate with `pnpm bench --label vitest-4.1.10`, the linked build with `pnpm bench --label vitest-5.0`, then `node scripts/render-results.mjs results/vitest-4.1.10.json results/vitest-5.0.json`.
85
+
Apple M4 (10 cores), node v24.13.0. Whole-process wall clock of `vitest run`, median of 3 reps, both versions measured on the same machine in one session. 4.1.10 is the pinned install on vite 8.1.4. 5.0.0-rc.2 is a local build with vitest-dev/vitest#11078, linked as described above; it resolves vite 8.0.11 from the vitest repository. To regenerate: `pnpm bench --label vitest-4.1.10`, then `pnpm bench --label vitest-5.0` with the linked build, then `node scripts/render-results.mjs results/vitest-4.1.10.json results/vitest-5.0.json`.
86
86
87
87
### micro-utils
88
88
89
-
The median OSS package — surveys of Vitest usage put the median project at ~4 test files. 8 modules, 5 test files, no dependencies: startup overhead is everything, and the jsdom/happy-dom rows show what an inherited DOM environment costs a node-only suite.
89
+
The median open source package: 8 modules, 5 test files, no dependencies. Startup overhead is everything here. The jsdom and happy-dom rows show what a DOM environment costs a node-only suite.
@@ -103,7 +103,7 @@ The median OSS package — surveys of Vitest usage put the median project at ~4
103
103
104
104
### node-library
105
105
106
-
A mid-size published library: 127 modules in 3 layers, 40 test files that import the modules they test directly, so the per-file subgraphs are largely disjoint.
106
+
A mid-size library: 127 modules in 3 layers and 40 test files. Each test file imports the modules it tests directly, so the per-file graphs overlap little.
@@ -119,7 +119,7 @@ A mid-size published library: 127 modules in 3 layers, 40 test files that import
119
119
120
120
### node-backend
121
121
122
-
An express 5 + zod + pino + dayjs + lodash API service with 16 integration-style test files that do real work per test (hundreds of validations, CRUD flows over in-memory repos).
122
+
An API service on express 5, zod, pino, dayjs, and lodash. 16 integration-style test files with real work per test: hundreds of validations and CRUD flows over in-memory repositories.
@@ -134,7 +134,7 @@ An express 5 + zod + pino + dayjs + lodash API service with 16 integration-style
134
134
135
135
### deps-heavy
136
136
137
-
Thin glue over 10 real packages covering the shapes that matter for module handling: CJS monoliths (lodash, semver), many-file ESM graphs (lodash-es, date-fns, rxjs), single big ESM (zod), dual packages (yaml, uuid). Node pools pay externals once per worker; vm pools re-evaluate them per context.
137
+
Thin code over 10 real packages that cover the module shapes that matter: CJS monoliths (lodash, semver), many-file ESM graphs (lodash-es, date-fns, rxjs), one big ESM file (zod), and dual packages (yaml, uuid). Node pools load externals once per worker; vm pools evaluate them again in each context.
@@ -149,7 +149,7 @@ Thin glue over 10 real packages covering the shapes that matter for module handl
149
149
150
150
### react-spa
151
151
152
-
A product SPA tested with Testing Library: 92 ts/tsx modules across 6 features, CSS and CSS modules, hooks, a `vi.mock`ed api layer and a jest-dom setup file — run in jsdom, happy-dom and real Chromium.
152
+
A React SPA tested with Testing Library: 92 ts/tsx modules in 6 features, CSS and CSS modules, hooks, a mocked API layer, and a jest-dom setup file. Runs in jsdom, happy-dom, and real Chromium.
@@ -170,7 +170,7 @@ A product SPA tested with Testing Library: 92 ts/tsx modules across 6 features,
170
170
171
171
### vue-spa
172
172
173
-
37 single-file components plus composables, tested with @vue/test-utils. The SFC compilation through @vitejs/plugin-vue makes this the expensive-transform fixture.
173
+
37 single-file components plus composables, tested with @vue/test-utils. SFC compilation through @vitejs/plugin-vue makes this the expensive-transform app.
@@ -186,7 +186,7 @@ A product SPA tested with Testing Library: 92 ts/tsx modules across 6 features,
186
186
187
187
### design-system
188
188
189
-
80 components with per-component CSS, and every one of the 80 test files imports from the root barrel — each file pays the whole library plus a DOM environment. The classic component-library trap.
189
+
80 components with per-component CSS. Every one of the 80 test files imports from the root barrel, so each file pays for the whole library plus a DOM environment.
@@ -202,7 +202,7 @@ A product SPA tested with Testing Library: 92 ts/tsx modules across 6 features,
202
202
203
203
### barrel-hell
204
204
205
-
The same barrel pathology without DOM or JSX: 817 modules behind nested barrels, 20 test files using ~3 symbols each, so every file evaluates the full graph.
205
+
The same barrel problem without DOM or JSX: 817 modules behind nested barrels and 20 test files that use about 3 symbols each. Every file evaluates the full graph.
@@ -217,7 +217,7 @@ The same barrel pathology without DOM or JSX: 817 modules behind nested barrels,
217
217
218
218
### enterprise-monolith
219
219
220
-
Big-repo CI: ~1280 modules with 12-deep import chains, import cycles, path aliases, dynamic imports, JSON imports, and 15 jsdom-pragma files among the 150 test files (mixed environments fragment worker reuse).
220
+
A large monorepo: about 1280 modules with 12-deep import chains, import cycles, path aliases, dynamic imports, JSON imports, and 150 test files. 15 of them use a jsdom pragma, so mixed environments limit worker reuse.
@@ -233,7 +233,7 @@ Big-repo CI: ~1280 modules with 12-deep import chains, import cycles, path alias
233
233
234
234
### long-haul
235
235
236
-
The worker-lifetime endurance fixture: 80 jsdom test files through 2 workers, every file holding a ~15MB module-level dataset and rendering tables over it. Node pools rebuild the environment and re-import the externalized dependencies for each of a worker's 40 files; vm pool workers pay once, reuse compiled scripts across contexts, and get recycled several times per run by the pinned 512MB `vmMemoryLimit` — the recycle path no other app enters. Short fixtures understate the vm pools; this is the fixture where they win by a wide margin. (World *retention* is deliberately out of scope: workers report lazy heap numbers, so leak regressions are covered by Vitest's own reachability tests, not wall clock.)
236
+
A long-running DOM suite: 80 jsdom test files through 2 workers, each file holding a 15MB module-level dataset and rendering tables over it. Node pools rebuild the environment and import the external dependencies again for each of a worker's 40 files. vm pool workers pay once, reuse compiled scripts across contexts, and get recycled by the pinned 512MB `vmMemoryLimit` several times per run. This is the app where the vm poolswin by a wide margin. Memory retention across files is out of scope here.
@@ -246,7 +246,7 @@ The worker-lifetime endurance fixture: 80 jsdom test files through 2 workers, ev
246
246
247
247
### cpu-bound
248
248
249
-
30 test files that burn real CPU (hashing, sieving, matrix multiplication) on an 8-module graph. The tests themselves dominate, so only scheduling — `maxWorkers`, pool choice — changes anything.
249
+
30 test files that burn real CPU (hashing, sieving, matrix multiplication) on an 8-module graph. The tests dominate, so only scheduling (`maxWorkers`, pool choice) changes anything.
@@ -260,7 +260,7 @@ The worker-lifetime endurance fixture: 80 jsdom test files through 2 workers, ev
260
260
261
261
## Design
262
262
263
-
- Generators are deterministic — no randomness, structure comes from modular arithmetic — so `pnpm generate` always produces byte-identical sources and a generator diff is a reviewable change to a fixture's shape.
264
-
- Every dependency is pinned exactly and the lockfile is committed; the vitest/vite/jsdom/happy-dom/playwright pins are part of the measurement — bump them deliberately, in their own commit.
265
-
- Tests assert real behavior computed through the import graph, so a vitest correctness regression fails the bench instead of silently timing broken runs.
266
-
- CI (`smoke.yml`) only checks that every app generates and passes under the pinned vitest; shared runners are too noisy for timing — use quiet dedicated hardware and `compare.mjs` for A/B decisions.
263
+
- Generators are deterministic: no randomness, the structure comes from modular arithmetic. `pnpm generate` produces byte-identical sources, so a generator diff is a reviewable change to an app's shape.
264
+
- Every dependency is pinned exactly and the lockfile is committed. The vitest, vite, jsdom, happy-dom, and playwright pins are part of the measurement. Bump them on purpose, in their own commit.
265
+
- Tests assert real behavior computed through the import graph, so a vitest correctness regression fails the bench instead of timing broken runs.
266
+
- CI (`smoke.yml`) only checks that every app generates and passes under the pinned vitest. Shared runners are too noisy for timing; use quiet dedicated hardware and `compare.mjs` for A/B decisions.
0 commit comments