Phase 3 Rocq parity M2: LLM-replay closer + audit-H1 axiom gate - #38
Merged
Conversation
Rocq mirror of `replayLlmScriptOrFail` (Lean PR #35). When a Tier-3 cert arrives with reason `Tier3_replay_deferred` (the LLM-as-backend `lean-tactic-script` cert from `Adapter_llm` — SDK-shared, no Rocq-specific adapter work needed for M2), `close_or_fail` routes it to a new `Llm_replay.replay_script` closer instead of the fragment-keyed chain. Same audit-H1 contract as Lean: kernel replay + transitive-axiom subset check against the classical Rocq Stdlib allowlist (`ClassicalDedekindReals.sig_forall_dec`, `FunctionalExtensionality.functional_extensionality_dep`). A hallucinated `admit` / `Axiom Foo` / parse failure / non-closing script is a tactic failure with the goal left OPEN — never an admitted theorem. **New module** (`rocq-bridge/src/llm_replay.ml` + `.mli`): - Parses the candidate via `Procq.parse_string Pltac.tactic` + `Tacintern.intern_pure_tactic` — same idiom `invoke_named_tactic` already uses for stdlib tactics. - Captures the original goal's evar before invoking `Tacinterp.eval_tactic`. - After the script runs, queries the sigma for the evar's assignment via `Evd.find` + `Evd.evar_body`. Empty body => script didn't close => audit-fail. Defined body => proceed to axiom check. - Collects the proof term's transitive axioms via `Assumptions.assumptions` (the kernel internal API Rocq's `Print Assumptions` vernacular uses). `add_opaque:false` / `add_transparent:false` so opaques are treated as black boxes — audited at their own define site, not here. - Subset-checks the `Axiom (Constant kn, _)` entries against `axiom_allowlist`. Other variants (`Positive`, `Guarded`, `TypeInType`, `UIP`, `Variable`, `Opaque`, `Transparent`) are accepted as inherent Rocq machinery. - Failure modes all surface via a shared `audit_fail` helper with consistent audit-H1 messaging. The `indirect_accessor` is a dummy `{ access_proof = fun _ -> None }` since `add_opaque:false` means `access_proof` is never consulted. The `GlobRef.t` "owner" arg (used by the kernel for self-reference guard) is a sentinel `VarRef` that won't appear in real proof terms. **Wired into `close_or_fail`** (`pb_rocq_main.ml`): a new `Tier3_replay_deferred _` arm extracts the script from the cert payload (`Tier3_proof_trace { trace_data = String script }`) and dispatches to `Llm_replay.replay_script` — analogous to the early-return branch at the top of Lean's `closeOrFail`. **Test-only tactic** (`g_proof_broker.mlg`): `llm_replay_test "<script>"` — direct entry to `Llm_replay.replay_script`, bypassing the cert envelope. Lets CI exercise the audit-H1 contract without a live LLM endpoint. **Tests** (`rocq-bridge/theories/Test.v`): - Positive `pb_llm_replay_axiom_free`: a clean `intros; reflexivity` script closes a small Z-equality goal. Allowlisted with `[]` in `tools/axiom_allowlist.json` — same expected footprint as the existing axiom-free Rocq tests. - Negative non-closing: `assert_fails (llm_replay_test "idtac")`. - Negative parse-failure: `assert_fails (llm_replay_test "@@@ ...")`. Locally verified: SDK + Rocq plugin sources build clean; Python schema/gate suite + JSON allowlist all green. The Rocq theory build path (incl. the new test) needs CI — local opam env still hits the rocq-core 9.1.1 split-build-mode bug per the M1 PR notes; rocq-bridge CI job exercises everything end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
levineuwirth
added a commit
that referenced
this pull request
May 21, 2026
Documentation-only polish for the Phase-3 Rocq parity work (PRs #37/#38/#39). No code changes. * README §Status gets a new paragraph recording the Rocq Phase-3 surface: HO reifier + HOL closer + opt-in coq-hammer (M1), LLM- replay closer with kernel-Assumptions-API audit gate (M2), bridge-aware SDK + reconstruction fallback (M3). Updates the retrospective list to include phase-3-rocq-parity. * delta.md §2.4 extends the Phase-3-progress block with the M1- through-M4 narrative: SDK reuse pattern, plugin-side wiring effort, and confirmation that the §1.1 architectural payoff ("genuinely cross-system or accidentally Lean-shaped") landed empirically — only LLM prompt rendering needed bridge- awareness, and it consolidated cleanly into a single `dialect` record. "Phase 3 is now structurally complete on both bridges." * RETROSPECTIVES/phase-3-rocq-parity.md (new) captures the lessons in the established phase-{0,4,5} retro format: - Easier-than-expected: SDK reuse, kernel Assumptions API as a clean replacement for `Lean.collectAxioms`, `hauto` as the `aesop` analogue. - Harder-than-expected: cross-bridge drift surface (M1 needed three CI rounds to caught up three Lean-side milestones' worth of stale glue in load_default / adapter_registry / verifier-reason arm / render_path; captured as the `project-rocq-parity-sync-points` project memory), local rocq-core build bug forcing CI-only verification, and the realization that bridge-awareness is an SDK concern (the `dialect` lives in `Adapter_llm`, not in the plugin). - Carried forward: BV/UF reach on Rocq pending Stdlib BitVec library, cross-bridge IR round-trip validation, dune `(using rocq …)` flip when dune ships it, retros for Phases 1/2/3 Lean-side. The `fragment_of_logic` consolidation flagged in RETROSPECTIVES/phase-4.md as a Phase-4 carry had already shipped during an earlier audit pass; the file is now a single shared function in `sdk/lib/smtlib.ml`, with all three adapters (cvc4/cvc5/z3) calling into it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rocq mirror of `replayLlmScriptOrFail` (Lean PR #35). When a Tier-3 cert arrives with reason `Tier3_replay_deferred` (the LLM-as-backend `lean-tactic-script` cert from `Adapter_llm` — SDK-shared, no Rocq-specific adapter work needed for M2), `close_or_fail` routes it to a new `Llm_replay.replay_script` closer instead of the fragment-keyed chain.
Same audit-H1 contract as Lean: kernel replay + transitive-axiom subset check against the classical Rocq Stdlib allowlist (`ClassicalDedekindReals.sig_forall_dec`, `FunctionalExtensionality.functional_extensionality_dep`). A hallucinated `admit` / `Axiom Foo` / parse failure / non-closing script is a tactic failure with the goal left OPEN — never an admitted theorem.
How (audit H1)
New module `rocq-bridge/src/llm_replay.ml`:
Wired into `close_or_fail` (`pb_rocq_main.ml`): a new `Tier3_replay_deferred _` arm extracts the script from `cert.payload` (`Tier3_proof_trace { trace_data = String script }`) and dispatches.
Test-only tactic (`g_proof_broker.mlg`): `llm_replay_test "<script>"` — direct entry to `Llm_replay.replay_script`, bypassing the cert envelope. Lets CI exercise the audit-H1 contract without a live LLM endpoint.
Tests (CI-stable, no network)
Positive `pb_llm_replay_axiom_free`: clean `intros; reflexivity` script closes a small Z-equality goal. Allowlisted `[]`.
Negative `assert_fails (llm_replay_test "idtac")` — script runs but doesn't close.
Negative `assert_fails (llm_replay_test "@@@ ...")` — script doesn't parse.
The first `admit`/`Axiom Foo` rejection test is deferred — needs a local `Axiom` declaration in the test file to drive into the proof term, which adds setup complexity I'd rather defer until M2 is landing.
Verification
Locally: SDK + Rocq plugin sources build clean (`dune build sdk rocq-bridge/src` rc=0). Python schema/gate suite green. JSON allowlist valid. The Rocq theory build path (incl. `pb_llm_replay_axiom_free` + the negative `assert_fails` examples) needs CI — local opam env still hits the same rocq-core 9.1.1 split-build-mode bug from M1.
Scope
M2 is the home-side replay closer + test-only tactic only — the same scope as Lean PR #35. The `Adapter_llm` is currently Lean-flavored (renders Lean syntax, mints `lean-tactic-script` certs) — making it bridge-aware (so it produces Ltac for Rocq home systems) is its own slice, naturally part of M3 / the LLM-assisted-reconstruction work or its own preceding mini-slice. M2's test-only tactic exercises the closer in isolation.
Remaining M's
🤖 Generated with Claude Code