Skip to content

Commit 911336e

Browse files
sheremet-vasheremet-va
committed
docs: one coverage table with the run times only
1 parent 55c602c commit 911336e

2 files changed

Lines changed: 44 additions & 177 deletions

File tree

‎README.md‎

Lines changed: 14 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ pnpm bench:coverage --label vitest-5.0-coverage
9191
node scripts/render-coverage.mjs results/vitest-4.1.10-coverage.json results/vitest-5.0-coverage.json
9292
```
9393

94-
The reporter's `Duration` line covers the tests only. The coverage post-processing runs after it, so the runner sets `DEBUG=vitest:coverage` and records the provider's `Generate coverage total time` line next to the wall clock. `ast-v8-to-istanbul`, the converter under `@vitest/coverage-v8`, is part of the v8 measurement; the runner prints its version, and the result file records it in `meta.astV8ToIstanbul`.
94+
The reporter's `Duration` line covers the tests only, so the tables report the whole-process wall clock. The runner also sets `DEBUG=vitest:coverage` and stores the provider's `Generate coverage total time` in the result file (`generate`), together with the `ast-v8-to-istanbul` version under `@vitest/coverage-v8` (`meta.astV8ToIstanbul`).
9595

9696
## Results: vitest 4.1.10 vs 5.0.0
9797

@@ -273,119 +273,21 @@ A long-running DOM suite: 80 jsdom test files through 2 workers, each file holdi
273273

274274
## Coverage: vitest 4.1.10 vs 5.0.0
275275

276-
Same machine and protocol as above, `pnpm bench:coverage`, median of 3 reps (design-system and long-haul: 5 reps). 4.1.10 is the pinned install with `@vitest/coverage-v8` 4.1.10 on `ast-v8-to-istanbul` 1.0.4. 5.0.0 is the local build (5.0.0-rc.2 plus 41 commits, `130f79b8b`) with its own `@vitest/coverage-v8` on `ast-v8-to-istanbul` 1.0.5. Each app runs on its fastest warm row from the tables above, with the default coverage options: only files loaded by the tests are covered, and the `text`, `html`, `clover`, and `json` reports are written.
276+
Whole-process wall clock of `vitest run` on the bold row of each app above, median of 3 reps (design-system and long-haul: 5 reps), without coverage and with each provider. Same machine and protocol as above. 4.1.10 is the pinned install with `@vitest/coverage-v8` 4.1.10 on `ast-v8-to-istanbul` 1.0.4. 5.0.0 is the local build (5.0.0-rc.2 plus 41 commits, `130f79b8b`) with its own `@vitest/coverage-v8` on `ast-v8-to-istanbul` 1.0.5. Default coverage options: only files loaded by the tests are covered, and the `text`, `html`, `clover`, and `json` reports are written. To regenerate: `pnpm bench:coverage --label vitest-4.1.10-coverage`, then `pnpm bench:coverage --label vitest-5.0-coverage` with the linked build, then `node scripts/render-coverage.mjs results/vitest-4.1.10-coverage.json results/vitest-5.0-coverage.json`.
277277

278-
Columns: `wall` is the whole-process wall clock of `vitest run`. `overhead` is the wall minus the wall of the `none` row, so it includes collection, conversion, and the report files. `generate` is the provider's own `Generate coverage total time` from `DEBUG=vitest:coverage`: the conversion of the raw coverage into istanbul maps, without the report files. Values under about 50ms are inside the noise of one run.
279-
280-
### micro-utils
281-
282-
pool threads, env node, isolate false, fsModuleCache false, warm.
283-
284-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
285-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
286-
| none | 0.25s | 0.24s | −2% | — | — | — | — | — | — |
287-
| v8 | 0.29s | 0.29s | ~0 | 0.05s | 0.05s | +2% | 10ms | 15ms | +50% |
288-
| istanbul | 0.39s | 0.38s | −4% | 0.15s | 0.14s | −7% | 2ms | 2ms | ~0 |
289-
290-
### node-library
291-
292-
pool threads, env node, isolate false, fsModuleCache true, warm.
293-
294-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
295-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
296-
| none | 0.29s | 0.28s | −3% | — | — | — | — | — | — |
297-
| v8 | 0.53s | 0.51s | −4% | 0.25s | 0.23s | −5% | 93ms | 93ms | ~0 |
298-
| istanbul | 0.65s | 0.56s | −14% | 0.36s | 0.28s | −23% | 83ms | 93ms | +12% |
299-
300-
### node-backend
301-
302-
pool threads, env node, isolate false, fsModuleCache false, warm.
303-
304-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
305-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
306-
| none | 0.43s | 0.41s | −4% | — | — | — | — | — | — |
307-
| v8 | 0.55s | 0.54s | ~0 | 0.12s | 0.13s | +12% | 26ms | 32ms | +23% |
308-
| istanbul | 0.69s | 0.66s | −5% | 0.26s | 0.24s | −7% | 5ms | 5ms | ~0 |
309-
310-
### deps-heavy
311-
312-
pool vmThreads, env node, isolate false, fsModuleCache false, warm.
313-
314-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
315-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
316-
| none | 1.63s | 0.69s | −57% | — | — | — | — | — | — |
317-
| v8 | 1.86s | 0.92s | −51% | 0.23s | 0.23s | −3% | 22ms | 40ms | +82% |
318-
| istanbul | 1.82s | 0.86s | −53% | 0.19s | 0.16s | −16% | 4ms | 5ms | +25% |
319-
320-
### react-spa
321-
322-
pool threads, env happy-dom, isolate false, fsModuleCache false, warm.
323-
324-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
325-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
326-
| none | 0.70s | 0.69s | −2% | — | — | — | — | — | — |
327-
| v8 | 0.87s | 0.87s | ~0 | 0.16s | 0.18s | +10% | 49ms | 60ms | +22% |
328-
| istanbul | 0.98s | 0.95s | −3% | 0.27s | 0.25s | −7% | 8ms | 8ms | ~0 |
329-
330-
### vue-spa
331-
332-
pool threads, env happy-dom, isolate false, fsModuleCache false, warm.
333-
334-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
335-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
336-
| none | 0.68s | 0.70s | +4% | — | — | — | — | — | — |
337-
| v8 | 0.78s | 0.79s | ~0 | 0.11s | 0.09s | −17% | 30ms | 38ms | +27% |
338-
| istanbul | 0.99s | 0.93s | −7% | 0.32s | 0.22s | −31% | 5ms | 5ms | ~0 |
339-
340-
### design-system
341-
342-
pool forks, env happy-dom, isolate false, fsModuleCache false, warm.
343-
344-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
345-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
346-
| none | 0.93s | 0.92s | ~0 | — | — | — | — | — | — |
347-
| v8 | 1.20s | 1.22s | +2% | 0.27s | 0.29s | +9% | 83ms | 99ms | +19% |
348-
| istanbul | 1.45s | 1.37s | −5% | 0.52s | 0.45s | −13% | 70ms | 71ms | ~0 |
349-
350-
### barrel-hell
351-
352-
pool threads, env node, isolate false, fsModuleCache true, warm.
353-
354-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
355-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
356-
| none | 0.63s | 0.57s | −9% | — | — | — | — | — | — |
357-
| v8 | 1.51s | 1.29s | −15% | 0.88s | 0.71s | −19% | 458ms | 312ms | −32% |
358-
| istanbul | 2.40s | 1.69s | −30% | 1.77s | 1.12s | −37% | 504ms | 491ms | −3% |
359-
360-
### enterprise-monolith
361-
362-
pool threads, env node, isolate false, fsModuleCache true, warm.
363-
364-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
365-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
366-
| none | 1.96s | 1.96s | ~0 | — | — | — | — | — | — |
367-
| v8 | 4.97s | 4.07s | −18% | 3.01s | 2.11s | −30% | 1899ms | 1332ms | −30% |
368-
| istanbul | 10.84s | 6.88s | −36% | 8.87s | 4.92s | −45% | 3383ms | 3396ms | ~0 |
369-
370-
### long-haul
371-
372-
pool vmForks, env happy-dom, isolate true, fsModuleCache false, maxWorkers 2, warm.
373-
374-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
375-
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
376-
| none | 5.37s | 3.98s | −26% | — | — | — | — | — | — |
377-
| v8 | 8.16s | 7.48s | −8% | 2.79s | 3.50s | +26% | 125ms | 145ms | +16% |
378-
| istanbul | 6.45s | 4.75s | −26% | 1.08s | 0.78s | −28% | 18ms | 19ms | +6% |
379-
380-
### cpu-bound
381-
382-
pool forks, env node, isolate false, fsModuleCache false, maxWorkers 100%, warm.
383-
384-
| coverage | 4.1.10 wall | 5.0.0 wall | Δ | 4.1.10 overhead | 5.0.0 overhead | Δ | 4.1.10 generate | 5.0.0 generate | Δ |
278+
| app | 4.1.10 none | 5.0.0 none | Δ | 4.1.10 v8 | 5.0.0 v8 | Δ | 4.1.10 istanbul | 5.0.0 istanbul | Δ |
385279
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
386-
| none | 0.65s | 0.63s | −4% | — | — | — | — | — | — |
387-
| v8 | 1.00s | 1.00s | ~0 | 0.35s | 0.37s | +6% | 14ms | 29ms | +107% |
388-
| istanbul | 0.93s | 0.90s | −2% | 0.27s | 0.28s | +2% | 8ms | 6ms | −25% |
280+
| micro-utils | 0.25s | 0.24s | −2% | 0.29s | 0.29s | ~0 | 0.39s | 0.38s | −4% |
281+
| node-library | 0.29s | 0.28s | −3% | 0.53s | 0.51s | −4% | 0.65s | 0.56s | −14% |
282+
| node-backend | 0.43s | 0.41s | −4% | 0.55s | 0.54s | ~0 | 0.69s | 0.66s | −5% |
283+
| deps-heavy | 1.63s | 0.69s | −57% | 1.86s | 0.92s | −51% | 1.82s | 0.86s | −53% |
284+
| react-spa | 0.70s | 0.69s | −2% | 0.87s | 0.87s | ~0 | 0.98s | 0.95s | −3% |
285+
| vue-spa | 0.68s | 0.70s | +4% | 0.78s | 0.79s | ~0 | 0.99s | 0.93s | −7% |
286+
| design-system | 0.93s | 0.92s | ~0 | 1.20s | 1.22s | +2% | 1.45s | 1.37s | −5% |
287+
| barrel-hell | 0.63s | 0.57s | −9% | 1.51s | 1.29s | −15% | 2.40s | 1.69s | −30% |
288+
| enterprise-monolith | 1.96s | 1.96s | ~0 | 4.97s | 4.07s | −18% | 10.84s | 6.88s | −36% |
289+
| long-haul | 5.37s | 3.98s | −26% | 8.16s | 7.48s | −8% | 6.45s | 4.75s | −26% |
290+
| cpu-bound | 0.65s | 0.63s | −4% | 1.00s | 1.00s | ~0 | 0.93s | 0.90s | −2% |
389291

390292
## Design
391293

‎scripts/render-coverage.mjs‎

Lines changed: 30 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,11 @@
1-
// Renders coverage bench result files as per-app markdown tables: one row
2-
// per coverage provider on the app's fastest configuration. Several files
3-
// render side by side, one column group per vitest version, with the change
1+
// Renders coverage bench result files as one markdown table: one row per
2+
// app (its `best` configuration), the wall clock of `vitest run` without
3+
// coverage, with the v8 provider and with the istanbul provider. Several
4+
// files render side by side, one column per vitest version, with the change
45
// against the first file next to every later version.
56
// node scripts/render-coverage.mjs results/a.json [more.json ...]
6-
//
7-
// Columns per version:
8-
// wall whole-process wall clock of `vitest run` (median)
9-
// overhead wall minus the wall of the `none` row: what coverage adds
10-
// generate the provider's own `Generate coverage total time` (median),
11-
// the raw-coverage to istanbul-map conversion
127
import { readFileSync } from 'node:fs'
8+
import { COVERAGE_PROVIDERS } from './matrix.mjs'
139

1410
const files = process.argv.slice(2)
1511
if (files.length === 0) {
@@ -19,75 +15,44 @@ if (files.length === 0) {
1915

2016
const versions = files.map((file) => {
2117
const { meta, results } = JSON.parse(readFileSync(file, 'utf8'))
22-
return { meta, label: meta.vitest, results }
18+
return { label: meta.vitest, results }
2319
})
2420
const apps = [...new Set(versions.flatMap(version => version.results.map(row => row.app)))]
2521

2622
const fmt = value => (value == null ? '—' : `${(value / 1000).toFixed(2)}s`)
27-
const ms = value => (value == null ? '—' : `${Math.round(value)}ms`)
2823
const delta = (before, after) => {
2924
if (before == null || after == null)
3025
return '—'
31-
if (before === 0)
32-
return after === 0 ? '~0' : '—'
3326
const pct = ((after - before) / before) * 100
3427
return Math.abs(pct) < 1.5 ? '~0' : `${pct > 0 ? '+' : '−'}${Math.abs(pct).toFixed(0)}%`
3528
}
3629

37-
const describe = row => [
38-
`pool ${row.pool}`,
39-
`env ${row.env}`,
40-
`isolate ${row.isolate}`,
41-
`fsModuleCache ${row.fsCache}`,
42-
...(row.workers ? [`maxWorkers ${row.workers}`] : []),
43-
].join(', ')
30+
const many = versions.length > 1
31+
const head = ['app']
32+
const align = ['---']
33+
for (const provider of COVERAGE_PROVIDERS) {
34+
versions.forEach((version, index) => {
35+
head.push(many ? `${version.label} ${provider}` : provider)
36+
align.push('---:')
37+
if (index > 0) {
38+
head.push('Δ')
39+
align.push('---:')
40+
}
41+
})
42+
}
43+
console.log(`| ${head.join(' | ')} |`)
44+
console.log(`| ${align.join(' | ')} |`)
4445

4546
for (const app of apps) {
46-
const byVersion = versions.map(version => version.results.filter(row => row.app === app))
47-
const sample = byVersion.flat()[0]
48-
console.log(`### ${app}\n`)
49-
console.log(`${describe(sample)}, warm.\n`)
50-
51-
const many = versions.length > 1
52-
const head = ['coverage']
53-
const align = ['---']
54-
for (const metric of ['wall', 'overhead', 'generate']) {
55-
versions.forEach((version, index) => {
56-
head.push(many ? `${version.label} ${metric}` : metric)
57-
align.push('---:')
58-
if (index > 0) {
59-
head.push('Δ')
60-
align.push('---:')
61-
}
47+
const cells = [app]
48+
for (const provider of COVERAGE_PROVIDERS) {
49+
const rows = versions.map(version => version.results.find(row => row.app === app && row.coverage === provider))
50+
const value = row => (row && !row.error ? row.median : null)
51+
rows.forEach((row, index) => {
52+
cells.push(row?.error ? 'FAIL' : fmt(value(row)))
53+
if (index > 0)
54+
cells.push(delta(value(rows[0]), value(row)))
6255
})
6356
}
64-
console.log(`| ${head.join(' | ')} |`)
65-
console.log(`| ${align.join(' | ')} |`)
66-
67-
const providers = [...new Set(byVersion.flat().map(row => row.coverage))]
68-
for (const provider of providers) {
69-
const rows = byVersion.map(rows => rows.find(row => row.coverage === provider))
70-
const bases = byVersion.map(rows => rows.find(row => row.coverage === 'none'))
71-
const value = (row, base, metric) => {
72-
if (!row || row.error)
73-
return null
74-
if (metric === 'wall')
75-
return row.median
76-
if (metric === 'overhead')
77-
return provider === 'none' || !base || base.error ? null : row.median - base.median
78-
return row.generate
79-
}
80-
const cells = [provider]
81-
for (const metric of ['wall', 'overhead', 'generate']) {
82-
const format = metric === 'generate' ? ms : fmt
83-
rows.forEach((row, index) => {
84-
const current = value(row, bases[index], metric)
85-
cells.push(row?.error ? 'FAIL' : format(current))
86-
if (index > 0)
87-
cells.push(delta(value(rows[0], bases[0], metric), current))
88-
})
89-
}
90-
console.log(`| ${cells.join(' | ')} |`)
91-
}
92-
console.log('')
57+
console.log(`| ${cells.join(' | ')} |`)
9358
}

0 commit comments

Comments
 (0)