Record what an executable was built with, alongside the executable?

[jgabry]

Note: While working on #1235 I was reminded of several longstanding issues. I asked Claude to turn them into reports. This body has been rewritten to reflect the design settled in #1254 — the original version and the review that followed are in this issue's edit history.

---

CmdStanR cannot tell what produced an existing executable. The binary reports a handful of STAN_* flags via <exe> info and nothing else — not the user header, not the include paths, not the remaining cpp_options. Anything else lives only in the R object, so it is gone in the next session.

The fix is a record written beside the executable at build time. This issue is the record itself; #1255 is what acts on it.

What the record holds

• request — the build configuration, with stanc_options stored twice: what the caller supplied, and what cmdstanr injected. The two are disjoint, because cmdstanr injects only what the caller did not supply, and origin is what decides whether an option can force a rebuild. Everything else is stored once, in the form the build used: cpp_options, which cmdstanr adds nothing to once the user header reaches make as a flag rather than as a recorded option, and the effective include_paths, so a caller who passed none on a program with #include has dirname(stan_file) recorded. These fields explain a build rather than replay one — make/local's STANCFLAGS reach the same stanc invocation and appear in none of them, being covered by make/local's own hash. Feeding them back is not a supported operation
• reported_features — what the binary itself reports. Tri-state: known enabled, known disabled, or unknown, because <exe> info reports only what CmdStan chooses to. Absence must never read as disabled — that is STAN_THREADS in make/local not respected due to capitalisation conflict #765 waiting to happen again
• dependencies — content hashes of the Stan file, its resolved includes, the user header, and make/local, plus the ordered included_files vector from stanc --info (edits to included files are not detected #1237), which preserves source order and duplicates. Each also records the absolute path it had at build time as built_from, which is provenance: never rewritten, and not compared, so moving a project does not rebuild it. The user header's path is the one dependency path that is also compared (Rebuild when the recorded build no longer matches what was asked for #1255), as one instance of a general rule: directories that participate in C++ include resolution are compared as spellings, the -I flags in cpp_options being the other
• artifact — a hash of the executable this record describes
• builder — the CmdStan installation that produced it
• known_untracked_dependencies — gaps we can detect but not resolve (Document the dependencies cmdstanr does not track #1257). Named for what it is: an empty list means nothing was detected, never that the record is complete
• format_version — the version of how this information is interpreted, not of the JSON's shape. A change in option or build semantics obliges a bump even when the file's layout is unchanged — including a change to which options cmdstanr injects, since two versions injecting different sets would disagree about whether to rebuild the same executable

Recorded and compared are different questions, and the record has to answer both. Recording serves provenance and diagnosis; comparison serves the rebuild decision (#1255), and a field belongs there only if it determines the artifact's observable behaviour and is not already fixed by another compared field. Recorded-but-not-compared is the ordinary case rather than a list of exceptions, and the table's column says which — include_paths, built_from, reported_features and known_untracked_dependencies among them, and format_version, which is a readable-set test rather than a comparison, since a release that widens the set still reads records it no longer writes. Option origin is stored rather than inferred: cpp_options_supplied and stanc_options_supplied are compared, stanc_options_injected is recorded only, and supplied and injected are accumulated side by side during the build rather than one recovered from the other by subtraction. There is no cpp_options_injected, because cmdstanr injects no C++ option; if one ever does appear, the field arrives with the format_version bump a new injection already obliges. The effective --name stanc receives is the one injected value that also gets its own compared field, request.stanc_name: the injected list says who asked for the flag, stanc_name says what stanc got, and only the second can force a rebuild. The verdict compares only what the caller supplied, and a merged list cannot be split back apart without knowing this version's injection rules, which is the reconstruct-after-the-fact fragility this design removes everywhere else. That holds even if no option can arrive by both routes; --filename-in-msg makes the consequence visible but is not what the rule rests on, and it is the only one left that can, since name is rejected from stanc_options (#1254 §3, "A flag cmdstanr derives from another argument is not separately settable"). Because the injected set is recorded rather than described, no list of injected options has to be kept correct in prose. §4 of #1254 carries the per-field table.

Canonicalization is per-field, which is why #1250 is a prerequisite rather than adjacent work: named assignments sort and last-wins, opaque arguments preserve order, include paths preserve order because it controls shadowing. Stanc options are compared as the sorted argument vector they emit rather than as the R list, which collapses list("O1") and list("O1" = TRUE) without enumerating any option's semantics.

The record must be bound to its executable

Atomic replacement of the record alone does not bind the pair:

process A installs executable A
process B installs executable B
process B writes record B
process A writes record A
final state: executable B + record A

A crash between the two installations does the same, and the current code already acknowledges a crash window when replacing an executable — "A crash between renames may leave only the backup" (R/utils.R:267-272).

Hence the executable hash. Measured at 8.5 ms for a CmdStan binary, against 28.6 ms merely to spawn <exe> info, so cost is not an argument against it. It also makes a manually replaced or corrupt executable detectable, which is otherwise easy to assume impossible.

Installation is a transaction: stage both artifacts, install the executable, install the record, verify the pair, and roll back both on any failure — including a successful record install whose verification then fails. Record-write failure cannot be treated as non-fatal, since an executable without a valid record is immediately unusable under #1255.

Concurrency is out of scope for v1. The hash fixes crash-created and sequential mismatches. It does not fix TOCTOU — A can validate executable A, B replace it, and A then launch B. Concurrent compilation or use of one destination is unsupported; locking is tracked separately, and the project's existing answer to this shape of problem is workspace isolation (#1025).

A record in a format version this cmdstanr does not read rebuilds, and says so — newer or older, since a downgrade regenerates old-format records too. It is not refused and does not need force_recompile. The record is entirely derived data, so replacing one destroys nothing that a rebuild does not regenerate — the cost is a compile. The hazard worth reporting is the executable, which may have been built under option semantics this cmdstanr does not implement, so the rebuild states what happened rather than erroring.

The realistic case in the newer direction is one person downgrading cmdstanr on one machine, where the CmdStan installation, make/local and every path are unchanged and nothing else would trigger. Erroring there costs a user real friction for no protection.

"Unreadable" and "unsupported format" stay separate checks even though they share a behaviour, because the messages differ: an unreadable record could not be accepted as a record at all and is withheld whole, with no part of it reported, while an unsupported one reports the format version found against the ones understood. #1254 §6 defines the boundary and §4 fixes how much of a record the reader validates before accepting it.

The live bug this fixes

R/model.R:732-733 asks Make's question:

file.mtime(exe) < file.mtime(self$stan_file())

That is exe versus source, correct only if mtimes move monotonically forward. Extracting a project tarball over an existing build:

source content is now DIFFERENT
exe mtime : 2026-08-27 10:19:38.476
stan mtime: 2026-03-15 09:00:00.000
exe < stan (rebuild?): FALSE   <-- no rebuild

cmdstanr reports "up to date" and runs the old binary against the new source. tar -x, unzip, cp -p, rsync -a and backup restores all preserve archival mtimes and all reach this.

It is filed here rather than separately because it cannot be fixed before a record exists — the fix is comparing against the record instead of against the artifact.

Hashes rather than mtimes

Once the comparison is against the record, a stored mtime would also fix the case above. Hashing additionally removes spurious rebuilds: a branch round-trip leaves byte-identical content with a fresh mtime, which is ordinary when the Stan file is in git and the executable is a gitignored artifact beside it.

Measured on six realistic source files: 0.04 ms to stat, 0.31 ms to hash, against a rebuild measured at 6.7 s for bernoulli.stan with precompiled headers enabled and 13.8 s without — a floor rather than a typical figure, since a program with many user functions or ODE solves is substantially slower. Ninja uses mtime because it stats tens of thousands of files per invocation; we stat about six.

Note the "no false negatives" property is bounded by what is tracked — toolchain drift and #1257's untracked dependencies are false negatives by construction.

Adoption: the record replaces the subprocess

Part of the point of the record is that reading a file beats launching a binary.
Adoption via cmdstan_model(exe_file = ) therefore does not run the executable
when it does not have to:

1. resolve the path; reject missing files and directories
2. if a valid record matches the executable's hash, hydrate from it without
   launching — <exe> info is not called at all
3. otherwise fall back to <exe> info
4. convert launch failures or non-CmdStan results into the friendly cmdstan_model(exe_file = ) surfaces a raw processx error when the executable cannot be run #1246 error,
   naming the path
5. commit state only after validation succeeds — nothing mutates on failure

What adoption establishes is what the artifact is, not that it runs. A hash-matched record proves the binary is the one whose build was recorded, and nothing executes it, so an executable that cannot run on this machine adopts successfully and fails when something first runs it (#1254 §7, "Adoption establishes what the artifact is, not that it runs"). Launching stays the path for an artifact with no usable record, and it is the path that raises #1246. This is also what makes the two executable-only cases below distinguishable cheaply.

Also settled

• Absorbs $cpp_options() lists the same option under two spellings after merging executable metadata #1247. Canonical option spelling is part of the record format, not a separate display fix.
• Path normalisation: paths are stored normalised and absolute, for the record's own consistency rather than to decide anything — dependencies are identified by content, so relocating a project does not rebuild it (Rebuild when the recorded build no longer matches what was asked for #1255). Relocatable records stay rejected: storing paths relative to some root would need that root defined, plus symlink behaviour and out-of-project paths, and content identity already removes the reason anyone wanted them.
• Executable-only models (cmdstan_model(exe_file =)) split two ways: with a valid hash-bound record, provenance is known and should be reported; without one, explicitly unprovenanced. Never automatically rebuilt — there is no source.

Design note and rationale: #1254.