Skip to content

perf(verify): cache the one phase that is 88% of the gate - #98

Merged
wormeyman merged 1 commit into
mainfrom
perf/verify-test-cache
Aug 2, 2026
Merged

perf(verify): cache the one phase that is 88% of the gate#98
wormeyman merged 1 commit into
mainfrom
perf/verify-test-cache

Conversation

@wormeyman

Copy link
Copy Markdown
Collaborator

verify's test phase now runs through vp run --cache test instead of a bare vp test.

Why this phase and no other

phase time
vp check 2.0s
check:vue 3.0s
vp test 61.2s
preview:test 3.1s
serial total 69.3s

One phase is 88% of the gate, so it is the only one worth caching. Cold 64.9s -> warm 7.0s; the test phase itself goes 62.0s -> 0.6s, and the 7s remainder is the three phases that are not cached.

The cache was probed, not trusted

Established as content-keyed by trying to break it rather than by reading the flag's docs:

probe result
edit a source file miss, re-ran (61.4s)
plant a failing assertion miss, rc=1, still failed
touch with content unchanged hit - so it hashes content, not mtime
revert to a previously-seen tree miss - only the latest result is stored

A hit therefore cannot mask a regression, and the normal edit -> verify loop misses every time, by design.

Scope, stated plainly

  • No-op in CI. Every runner starts cold, so the required verify check runs in full regardless. This PR does not make CI faster.
  • Pays on deploy and little else. deploy:app runs verify immediately after you have probably just run one by hand.
  • A hit is a replay, not a run. Legitimate for file content per the probes above. Not established: whether the key covers inputs outside the tree - env vars, node version, or whether a Factorio install appeared or vanished (the oracle specs are it.skipIf(!oracleAvailable()), so their skip status can change with no file changing). CLAUDE.md says to use vp cache clean when chasing something environmental.

Rejected alternatives, recorded so they aren't retried

  • Parallel phases: 60.6s against 69.3s serial (13%), but it turns a 2s type error into a 61s one because vp check no longer runs first. It would also need a new script, since a dependsOn: ["check"] pulls in the check script - vp check --fix - which must never run in a deploy path.
  • More test workers: already at the optimum. 4 -> 74.7s, 8 -> 61.7s, 11 -> 61.8s against a 61.2s default; the spare cores on this machine are E-cores.

Incidentally, the gate proved non-vacuous during this change: the first run failed with rc=1 on a formatting issue in the CLAUDE.md I had just written.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LT6oy76sLRqoC1QzLocQ1h

`verify`'s test phase runs through `vp run --cache test` instead of a bare
`vp test`. Measured: `vp check` 2.0s, `check:vue` 3.0s, `vp test` 61.2s,
`preview:test` 3.1s - one phase is 88% of the gate, so it is the only one
worth caching. Cold 64.9s -> warm 7.0s (the test phase itself, 62.0s -> 0.6s).

The cache was established as content-keyed by trying to break it rather than
by reading the flag's docs:

| probe                              | result                        |
| ---------------------------------- | ----------------------------- |
| edit a source file                 | miss, re-ran (61.4s)          |
| plant a failing assertion          | miss, **rc=1, still failed**  |
| `touch`, content unchanged         | **hit** - hashes content      |
| revert to a previously-seen tree   | miss - only the latest stored |

So a hit cannot mask a regression, and the normal edit -> verify loop misses
every time by design. It is also a no-op in CI, where every runner starts
cold, so the required `verify` check is unaffected.

Two alternatives were measured and rejected, and CLAUDE.md records them so
they don't get retried: running the four phases in parallel is 60.6s against
69.3s serial (13%) but turns a 2s type error into a 61s one, and would need a
new script since `dependsOn: ["check"]` pulls in `vp check --fix`; and
`maxWorkers` is already optimal (4 -> 74.7s, 8 -> 61.7s, 11 -> 61.8s).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LT6oy76sLRqoC1QzLocQ1h
@wormeyman
wormeyman merged commit 340b70d into main Aug 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant