Skip to content

Phase 3 Rocq parity M2: LLM-replay closer + audit-H1 axiom gate - #38

Merged
levineuwirth merged 1 commit into
mainfrom
phase-3-rocq-parity-m2
May 21, 2026
Merged

Phase 3 Rocq parity M2: LLM-replay closer + audit-H1 axiom gate#38
levineuwirth merged 1 commit into
mainfrom
phase-3-rocq-parity-m2

Conversation

@levineuwirth

Copy link
Copy Markdown
Owner

What

Rocq mirror of `replayLlmScriptOrFail` (Lean PR #35). When a Tier-3 cert arrives with reason `Tier3_replay_deferred` (the LLM-as-backend `lean-tactic-script` cert from `Adapter_llm` — SDK-shared, no Rocq-specific adapter work needed for M2), `close_or_fail` routes it to a new `Llm_replay.replay_script` closer instead of the fragment-keyed chain.

Same audit-H1 contract as Lean: kernel replay + transitive-axiom subset check against the classical Rocq Stdlib allowlist (`ClassicalDedekindReals.sig_forall_dec`, `FunctionalExtensionality.functional_extensionality_dep`). A hallucinated `admit` / `Axiom Foo` / parse failure / non-closing script is a tactic failure with the goal left OPEN — never an admitted theorem.

How (audit H1)

New module `rocq-bridge/src/llm_replay.ml`:

  • Parse the candidate via `Procq.parse_string Pltac.tactic` + `Tacintern.intern_pure_tactic` — same idiom `invoke_named_tactic` already uses for stdlib tactics.
  • Capture the goal's evar; eval the tactic via `Tacinterp.eval_tactic`.
  • After the script runs, query the sigma for the evar's assignment (`Evd.find` + `Evd.evar_body`). Empty body ⇒ script didn't close ⇒ audit-fail.
  • Collect the proof term's transitive axioms via `Assumptions.assumptions` — the same kernel internal API Rocq's `Print Assumptions` vernacular uses. `add_opaque:false` so opaques are treated as black boxes (audited at their own define site). Dummy `indirect_accessor` is safe under that flag.
  • Subset-check the `Axiom (Constant kn, _)` entries against the allowlist. Other variants (`Positive`/`Guarded`/etc.) are accepted as inherent Rocq machinery.
  • Failure modes all surface via a shared `audit_fail` helper.

Wired into `close_or_fail` (`pb_rocq_main.ml`): a new `Tier3_replay_deferred _` arm extracts the script from `cert.payload` (`Tier3_proof_trace { trace_data = String script }`) and dispatches.

Test-only tactic (`g_proof_broker.mlg`): `llm_replay_test "<script>"` — direct entry to `Llm_replay.replay_script`, bypassing the cert envelope. Lets CI exercise the audit-H1 contract without a live LLM endpoint.

Tests (CI-stable, no network)

Positive `pb_llm_replay_axiom_free`: clean `intros; reflexivity` script closes a small Z-equality goal. Allowlisted `[]`.

Negative `assert_fails (llm_replay_test "idtac")` — script runs but doesn't close.

Negative `assert_fails (llm_replay_test "@@@ ...")` — script doesn't parse.

The first `admit`/`Axiom Foo` rejection test is deferred — needs a local `Axiom` declaration in the test file to drive into the proof term, which adds setup complexity I'd rather defer until M2 is landing.

Verification

Locally: SDK + Rocq plugin sources build clean (`dune build sdk rocq-bridge/src` rc=0). Python schema/gate suite green. JSON allowlist valid. The Rocq theory build path (incl. `pb_llm_replay_axiom_free` + the negative `assert_fails` examples) needs CI — local opam env still hits the same rocq-core 9.1.1 split-build-mode bug from M1.

Scope

M2 is the home-side replay closer + test-only tactic only — the same scope as Lean PR #35. The `Adapter_llm` is currently Lean-flavored (renders Lean syntax, mints `lean-tactic-script` certs) — making it bridge-aware (so it produces Ltac for Rocq home systems) is its own slice, naturally part of M3 / the LLM-assisted-reconstruction work or its own preceding mini-slice. M2's test-only tactic exercises the closer in isolation.

Remaining M's

  • M3: LLM-assisted reconstruction fallback (`tryLlmReconstruct?` mirror) — wires the SDK's `Llm_reconstruct.translate` into `close_or_fail`'s try/catch fallback for un-replayable certs. Likely also needs a bridge-aware prompt on the SDK side.
  • M4: Polish — `fragment_of_logic` consolidation (Phase-4 retro carry), docstring / docs / delta sync, retrospectives.

🤖 Generated with Claude Code

Rocq mirror of `replayLlmScriptOrFail` (Lean PR #35). When a
Tier-3 cert arrives with reason `Tier3_replay_deferred` (the
LLM-as-backend `lean-tactic-script` cert from `Adapter_llm` —
SDK-shared, no Rocq-specific adapter work needed for M2),
`close_or_fail` routes it to a new `Llm_replay.replay_script`
closer instead of the fragment-keyed chain. Same audit-H1
contract as Lean: kernel replay + transitive-axiom subset check
against the classical Rocq Stdlib allowlist
(`ClassicalDedekindReals.sig_forall_dec`,
`FunctionalExtensionality.functional_extensionality_dep`). A
hallucinated `admit` / `Axiom Foo` / parse failure / non-closing
script is a tactic failure with the goal left OPEN — never an
admitted theorem.

**New module** (`rocq-bridge/src/llm_replay.ml` + `.mli`):
- Parses the candidate via `Procq.parse_string Pltac.tactic` +
  `Tacintern.intern_pure_tactic` — same idiom
  `invoke_named_tactic` already uses for stdlib tactics.
- Captures the original goal's evar before invoking
  `Tacinterp.eval_tactic`.
- After the script runs, queries the sigma for the evar's
  assignment via `Evd.find` + `Evd.evar_body`. Empty body =>
  script didn't close => audit-fail. Defined body =>
  proceed to axiom check.
- Collects the proof term's transitive axioms via
  `Assumptions.assumptions` (the kernel internal API Rocq's
  `Print Assumptions` vernacular uses). `add_opaque:false` /
  `add_transparent:false` so opaques are treated as black
  boxes — audited at their own define site, not here.
- Subset-checks the `Axiom (Constant kn, _)` entries against
  `axiom_allowlist`. Other variants (`Positive`, `Guarded`,
  `TypeInType`, `UIP`, `Variable`, `Opaque`, `Transparent`)
  are accepted as inherent Rocq machinery.
- Failure modes all surface via a shared `audit_fail` helper
  with consistent audit-H1 messaging.

The `indirect_accessor` is a dummy `{ access_proof = fun _ -> None }`
since `add_opaque:false` means `access_proof` is never consulted.
The `GlobRef.t` "owner" arg (used by the kernel for
self-reference guard) is a sentinel `VarRef` that won't appear
in real proof terms.

**Wired into `close_or_fail`** (`pb_rocq_main.ml`): a new
`Tier3_replay_deferred _` arm extracts the script from the
cert payload (`Tier3_proof_trace { trace_data = String script }`)
and dispatches to `Llm_replay.replay_script` — analogous to the
early-return branch at the top of Lean's `closeOrFail`.

**Test-only tactic** (`g_proof_broker.mlg`):
`llm_replay_test "<script>"` — direct entry to
`Llm_replay.replay_script`, bypassing the cert envelope. Lets
CI exercise the audit-H1 contract without a live LLM endpoint.

**Tests** (`rocq-bridge/theories/Test.v`):
- Positive `pb_llm_replay_axiom_free`: a clean `intros; reflexivity`
  script closes a small Z-equality goal. Allowlisted with `[]`
  in `tools/axiom_allowlist.json` — same expected footprint as
  the existing axiom-free Rocq tests.
- Negative non-closing: `assert_fails (llm_replay_test "idtac")`.
- Negative parse-failure: `assert_fails (llm_replay_test "@@@ ...")`.

Locally verified: SDK + Rocq plugin sources build clean; Python
schema/gate suite + JSON allowlist all green. The Rocq theory
build path (incl. the new test) needs CI — local opam env still
hits the rocq-core 9.1.1 split-build-mode bug per the M1 PR
notes; rocq-bridge CI job exercises everything end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@levineuwirth levineuwirth self-assigned this May 21, 2026
@levineuwirth
levineuwirth merged commit 31cda1d into main May 21, 2026
7 checks passed
levineuwirth added a commit that referenced this pull request May 21, 2026
Documentation-only polish for the Phase-3 Rocq parity work
(PRs #37/#38/#39). No code changes.

* README §Status gets a new paragraph recording the Rocq Phase-3
  surface: HO reifier + HOL closer + opt-in coq-hammer (M1), LLM-
  replay closer with kernel-Assumptions-API audit gate (M2),
  bridge-aware SDK + reconstruction fallback (M3). Updates the
  retrospective list to include phase-3-rocq-parity.

* delta.md §2.4 extends the Phase-3-progress block with the M1-
  through-M4 narrative: SDK reuse pattern, plugin-side wiring
  effort, and confirmation that the §1.1 architectural payoff
  ("genuinely cross-system or accidentally Lean-shaped") landed
  empirically — only LLM prompt rendering needed bridge-
  awareness, and it consolidated cleanly into a single `dialect`
  record. "Phase 3 is now structurally complete on both bridges."

* RETROSPECTIVES/phase-3-rocq-parity.md (new) captures the
  lessons in the established phase-{0,4,5} retro format:
  - Easier-than-expected: SDK reuse, kernel Assumptions API as
    a clean replacement for `Lean.collectAxioms`, `hauto` as the
    `aesop` analogue.
  - Harder-than-expected: cross-bridge drift surface (M1 needed
    three CI rounds to caught up three Lean-side milestones'
    worth of stale glue in load_default / adapter_registry /
    verifier-reason arm / render_path; captured as the
    `project-rocq-parity-sync-points` project memory), local
    rocq-core build bug forcing CI-only verification, and the
    realization that bridge-awareness is an SDK concern (the
    `dialect` lives in `Adapter_llm`, not in the plugin).
  - Carried forward: BV/UF reach on Rocq pending Stdlib BitVec
    library, cross-bridge IR round-trip validation, dune
    `(using rocq …)` flip when dune ships it, retros for
    Phases 1/2/3 Lean-side.

The `fragment_of_logic` consolidation flagged in
RETROSPECTIVES/phase-4.md as a Phase-4 carry had already shipped
during an earlier audit pass; the file is now a single shared
function in `sdk/lib/smtlib.ml`, with all three adapters
(cvc4/cvc5/z3) calling into it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@levineuwirth
levineuwirth deleted the phase-3-rocq-parity-m2 branch August 31, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant