Skip to content

Save OPAM environments of dependency changes and bound the CI caches - #9294

Merged
cknitt merged 1 commit into
masterfrom
ci-cache-exact
Nov 13, 2026
Merged

cknitt merged 1 commit into
masterfrom
ci-cache-exact

Conversation

@cknitt

@cknitt cknitt commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Problems with the caching on master

  1. A PR that changes the opam files rebuilds every OPAM environment on every push. The environment key ends in a hash of *.opam, and only push runs save caches, so such a PR misses on every push: 531 s on Linux x64, 643 s on macOS, 953 s on Windows.
  2. The caches exceed GitHub's 10 GB limit (10.8 GB), so GitHub evicts live ones.
    • The coverage build state is keyed on a hash of compiler/**. Nearly every master push saves a new 266 MB entry that no later run matches: 11 entries, 2.9 GB.
    • Earlier generations of the OPAM environments and build states stay until GitHub's 7-day eviction.
    • A closed PR's caches (Node, apt) stay as well.
  3. The compiler build states grow with every master push. They include dune's shared cache (~/.cache/dune), which each master push restores, extends and saves again.
  4. The apt cache bypasses dpkg. setup-linux-dependencies unpacks a tarball of the installed files, so dpkg doesn't know those packages and opam takes them for missing. An opam install in a restored environment then reinstalls them, or rebuilds the whole switch. On a fresh environment the tarball is redundant anyway: ocaml/setup-ocaml installs the same packages with apt.

Changes

  • A PR that changes the opam files saves its OPAM environments. On a miss, setup-ocaml checks with git diff against the base whether the PR changes *.opam; if it does, it saves the environment it built. GitHub scopes caches saved by a PR to its merge ref, so the PR's later pushes hit exactly and no other branch sees them. Other PRs still don't save, since each would store a copy of master's environments. The first push per opam change, and the master push after the merge, still build from scratch.
  • Environments are only restored on an exact key, as now. An environment restored from earlier dependencies and then updated would depend on what it held before: pins, repository state, installed packages, global options. An exact key guarantees the environment is the one a fresh build of these opam files produced.
  • Each save deletes the earlier generations. After saving an environment, setup-ocaml deletes the caches with the same key prefix on the same ref, so each job keeps one environment per ref. A failure only warns.
  • Closed PRs' caches are deleted. The new cache_cleanup.yml runs on pull_request_target: closed, without checking out the PR, and can also be started by hand with a PR number.
  • The coverage build state is removed. Every run missed it, and the instrumented build takes about 30 s.
  • The compiler build states keep only _build, under a new v2 key prefix. _build is one tree that dune updates in place; the shared dune cache only accumulates.
  • System packages are installed with apt. setup-linux-dependencies runs apt-get install bubblewrap musl-tools rsync cmake, so dpkg knows them. The multilib packages are dropped: ocaml/setup-ocaml installs them when it builds an environment, and nothing else needs them.

Expected steady state: about 5 GB (OPAM environments 3.3 GB, build states about 0.9 GB, Node), plus one set of environments per open PR that changes the opam files.

After the merge, the old coverage-build-state-* and compiler-build-state-v1-* caches can be deleted by hand.

Validation

Tested on this PR with temporary commits, since squashed away: two edits to tools.opam (which change the opam hash) and an empty commit.

Push Expected Result
No opam change master's environments hit Setup OCaml 10–30 s on every OS
tools.opam edit every job builds from scratch and saves under refs/pull/9294/merge 6 environments saved; the 5 of an earlier hash were deleted
Empty commit the PR's own environments hit Setup OCaml 9–14 s (Windows 33 s), down from 531–953 s
Second tools.opam edit new environments saved, the previous ones deleted 6 saved, the previous 6 deleted
Edits reverted master's environments hit again Setup OCaml 9–14 s (Windows 39 s)

Still to check after the merge: the build time with _build-only build states, and that storage settles near 5 GB.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kbf3gGPsF3fHZn9rBxA7Hm

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Changelog

No changelog entry.

Semantic Diff

1 commit · Full page

✨ Save OPAM environments of dependency changes and bound the CI caches

81a2ded · cknitt + Claude · Nov 13, 2026, 11:47 UTC · new

Why

A pull request that changed the opam files rebuilt every OPAM environment on each push (531 s on Linux x64 to 953 s on Windows), since only pushes to master save caches. The caches exceeded GitHub's 10 GB, so live ones were evicted: the coverage build state, keyed on a hash of compiler/**, saved a new 266 MB entry on nearly every master push, earlier generations stayed until GitHub's 7-day eviction, and the compiler build states grew with dune's shared cache.

When an OPAM environment misses, a pull request that changes *.opam saves the one it built, in the cache scope of its merge ref, so its later pushes restore it. Each save deletes the earlier environments with the same key prefix on the same ref, and a new workflow deletes a closed pull request's caches. The coverage build state is gone, the compiler build states keep only _build, and the Linux system packages are installed with apt instead of unpacked from a cached tarball that dpkg didn't know about, so opam took them for missing.

Review

  • Why not restore the environment of earlier dependencies and update it? The result would depend on what that environment held: pins, repository state, installed packages, depext status, global options. Environments are still only restored on an exact key, so each is the one a fresh build of its opam files produced.
  • Why do only pull requests that change the opam files save? Any other pull request would store a copy of master's environments, 3.3 GB each.
  • Was it tested? On this pull request: a temporary opam change built and saved all six environments, the next push restored them in 9-39 s, a second change replaced them, and reverting it restored master's again.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ Compiler benchmarks

Allocated words

In total: 640,783,627 → 640,783,311 (±0.00%).

Generated projects: 920,886,686 → 920,886,786 (±0.00%).

None of the 265 benchmarks changed by 0.1% or more.

Instructions

In total: 9,209,672,036 → 9,209,672,036 (±0.00%).

None of the 33 benchmarks changed by 0.1% or more.

Output size

In total: 25,838,530 → 25,838,530 (±0.00%).

None of the 13 benchmarks changed by 0.1% or more.

Pull request 81a2ded compared with base 7f416af in this run.

@cknitt

cknitt commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T20:08:01.059354Z 07985f8 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 07985f8c1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pkg-pr-new

pkg-pr-new Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript@9294

@rescript/belt

npm i https://pkg.pr.new/@rescript/belt@9294

@rescript/darwin-arm64

npm i https://pkg.pr.new/@rescript/darwin-arm64@9294

@rescript/linux-arm64

npm i https://pkg.pr.new/@rescript/linux-arm64@9294

@rescript/linux-x64

npm i https://pkg.pr.new/@rescript/linux-x64@9294

@rescript/runtime

npm i https://pkg.pr.new/@rescript/runtime@9294

@rescript/win32-x64

npm i https://pkg.pr.new/@rescript/win32-x64@9294

commit: 81a2ded

@github-actions

Copy link
Copy Markdown

Why:
    A pull request that changed the opam files rebuilt every OPAM
    environment on each push (531 s on Linux x64 to 953 s on Windows),
    since only pushes to master save caches. The caches exceeded GitHub's
    10 GB, so live ones were evicted: the coverage build state, keyed on a
    hash of compiler/**, saved a new 266 MB entry on nearly every master
    push, earlier generations stayed until GitHub's 7-day eviction, and the
    compiler build states grew with dune's shared cache.

When an OPAM environment misses, a pull request that changes `*.opam` saves
the one it built, in the cache scope of its merge ref, so its later pushes
restore it. Each save deletes the earlier environments with the same key
prefix on the same ref, and a new workflow deletes a closed pull request's
caches. The coverage build state is gone, the compiler build states keep
only `_build`, and the Linux system packages are installed with apt instead
of unpacked from a cached tarball that dpkg didn't know about, so opam took
them for missing.

Review: Why not restore the environment of earlier dependencies and update
it? The result would depend on what that environment held: pins,
repository state, installed packages, depext status, global options.
Environments are still only restored on an exact key, so each is the one a
fresh build of its opam files produced.

Review: Why do only pull requests that change the opam files save? Any
other pull request would store a copy of master's environments, 3.3 GB
each.

Review: Was it tested? On this pull request: a temporary opam change built
and saved all six environments, the next push restored them in 9-39 s, a
second change replaced them, and reverting it restored master's again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kbf3gGPsF3fHZn9rBxA7Hm
Signed-off-by: Christoph Knittel <ck@cca.io>
@cknitt cknitt changed the title Restore OPAM environments only on exact keys and bound the CI caches Save OPAM environments of dependency changes and bound the CI caches Nov 13, 2026
@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.06%. Comparing base (7f416af) to head (81a2ded).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9294      +/-   ##
==========================================
- Coverage   82.07%   82.06%   -0.01%     
==========================================
  Files         455      455              
  Lines       65361    65361              
==========================================
- Hits        53642    53640       -2     
- Misses      11719    11721       +2     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cknitt
cknitt marked this pull request as ready for review November 13, 2026 12:11
@cknitt
cknitt enabled auto-merge (squash) November 13, 2026 12:14
@cknitt
cknitt merged commit ed12abd into master Nov 13, 2026
22 checks passed
@cknitt
cknitt deleted the ci-cache-exact branch November 13, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog:skip No release notes needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants