Skip to content

Commit 89e7124

Browse files
levineuwirthclaude
andcommitted
Phase 3 Rocq parity M3: bridge-aware LLM adapter + reconstruction fallback
Rocq mirror of Lean PR #36 (LLM-assisted Tier-3 reconstruction fallback), plus the load-bearing prerequisite that makes the fallback actually work cross-bridge: the SDK's `Adapter_llm` and `Llm_reconstruct` now produce bridge-appropriate prompts / trace_formats based on `ir.source_system.name`. **Bridge-aware SDK** (`sdk/lib/adapter_llm.ml`, `llm_reconstruct.ml`): - New `rocq_ty` / `rocq_term` / `rocq_paren` parallel to the existing Lean rendering helpers — translate IR's bridge-agnostic type-ref vocabulary (Int / Real / Prop + arrow chains) to Rocq Stdlib syntax (Z / R / Prop with ASCII `->`), and render shell_terms with Rocq's `/\` `\/` `~` connectives, `forall (x : T), body` quantifiers, ASCII Z-scope arithmetic. - New `dialect` record bundles per-bridge state (system_prompt, render_prompt, ty, term, trace_format, cert_format, annotation). `dialect_of_ir` dispatches on `ir.source_system.name`: "rocq" → `rocq_dialect`, everything else → `lean_dialect` (safer LLM-familiar default). - `Adapter_llm.dispatch` and `Llm_reconstruct.translate` both pick the dialect upfront — Rocq home systems get Rocq-flavored prompts asking for Ltac in a fenced ```coq block; minted certs carry `format = trace_format = "rocq-tactic-script"`. - `Verifier.verify` extends its Tier-3 dispatch to recognize both `lean-tactic-script` and `rocq-tactic-script` → both return `Tier3_replay_deferred { trace_format }` (envelope-only; soundness deferred to the home kernel's audit-H1 replay gate). **Closers tightened** (M3.b): Rocq's `close_or_fail` now matches `Tier3_replay_deferred { trace_format = "rocq-tactic-script" }` specifically. A `lean-tactic-script` cert reaching the Rocq closer (misconfigured manifest pointing this bridge at a Lean-flavored LLM endpoint, or future bug) is a clean user error rather than a silent parse failure when Ltac tries to chew on Lean syntax. **Reconstruction fallback** (M3.c, `rocq-bridge/src/pb_rocq_main.ml`): Mirror of Lean's `closeOrFailPrimary` + `closeOrFail` wrap. - The existing `close_or_fail` is renamed `close_or_fail_primary`. - New `replay_reconstructed_script trace_format script` runs the candidate through `Llm_replay.replay_script` (the same audit-H1 gate from M2) and on success emits a `Feedback.msg_info` line naming the source trace format — audit trail visible in build output. - New `try_llm_reconstruct ir cert?` gates on Tier-3 cert with `trace_format ≠ rocq-tactic-script`, calls `Proof_broker.Llm_reconstruct.translate` (direct-linked, no FFI), returns `Some fallback_tac` or `None`. - New `close_or_fail` wraps primary in `Proofview.tclORELSE`; on primary failure invokes `try_llm_reconstruct` lazily (HTTP only fires when primary actually fails) and routes to the fallback if a script came back, else re-raises the primary error. - Audit H1: reconstruction goes through the SAME audit gate as primary LLM-replay (kernel replay + axiom-footprint subset check), so the LLM never widens the trust base. **Test-only tactic** (`g_proof_broker.mlg`): new `llm_reconstruct_test "<format>" "<script>"` drives `replay_reconstructed_script` directly with a string literal — exercises the audit-H1 contract without a live LLM endpoint. Mirror of Lean's evalLlmReconstructTest. **Tests** (Rocq-side mirror of Lean's `llm_reconstruct_test` block): - Positive `pb_llm_reconstruct_axiom_free`: clean `intros; reflexivity` translation closes a Z-equality goal. Allowlisted `[]`. - Negative `assert_fails (llm_reconstruct_test ... "idtac")` — non-closing translation. - Negative `assert_fails (llm_reconstruct_test ... "@@@ ...")` — parse failure. **SDK tests** added bridge-aware coverage: Rocq-flavored `test_prompt_render_rocq` and `test_mock_endpoint_rocq` for `Adapter_llm`; `test_prompt_render_rocq` for `Llm_reconstruct`. Verifies Z translation, Theorem...Proof...Qed scaffold, ```coq fence, and that the Rocq-flavored cert's verify reason carries `trace_format = "rocq-tactic-script"`. Locally verified: SDK + Rocq plugin sources build clean (`dune build sdk rocq-bridge/src` rc=0). Full SDK suite passes including the new Rocq tests. Python schema/gate suite green. JSON allowlist valid. The Rocq theory build path (incl. the new fallback test and Print Assumptions on `pb_llm_reconstruct_axiom_free`) needs CI per the same broken-local-rocq-env caveat from M1/M2. With this, Phase 3 Rocq parity is structurally complete for deliverables #3 and #4. M4 (polish) carries the `fragment_of_logic` consolidation (Phase-4 retro carry) plus a README/delta sync to record the Rocq Phase-3 surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 31cda1d commit 89e7124

10 files changed

Lines changed: 573 additions & 56 deletions

File tree

rocq-bridge/src/g_proof_broker.mlg

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,23 @@ END
5858
TACTIC EXTEND ProofBrokerLlmReplayTest
5959
| [ "llm_replay_test" string(s) ] -> { Llm_replay.replay_script s }
6060
END
61+
62+
(* TEST-ONLY tactic for the LLM-assisted Tier-3 reconstruction
63+
fallback (roadmap §Phase 3 #4, Rocq parity M3). The live SDK
64+
path ([Llm_reconstruct.translate]) only runs when an endpoint
65+
is configured — for CI we skip the translation call and feed
66+
the candidate script directly into the {e same} closer the
67+
production path invokes after a successful translation
68+
([Pb_rocq_main.replay_reconstructed_script]). This pins the
69+
integration: a translated script flows through
70+
[Llm_replay.replay_script] (kernel replay + axiom-footprint
71+
gate), the same audit-H1 contract [llm_replay_test] exercises
72+
for primary LLM certs. First argument is the trace format of
73+
the (notionally un-replayable) source cert — surfaces in the
74+
audit feedback line. Mirror of [llm_reconstruct_test] in
75+
lean-bridge/Test/Tactic.lean. *)
76+
TACTIC EXTEND ProofBrokerLlmReconstructTest
77+
| [ "llm_reconstruct_test" string(fmt) string(s) ] -> {
78+
Pb_rocq_main.replay_reconstructed_script fmt s
79+
}
80+
END

rocq-bridge/src/pb_rocq_main.ml

Lines changed: 91 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,13 @@ let closer_for_fragment fragment : unit Proofview.tactic =
211211
cert-gated closer in the core plugin)"
212212
other))
213213

214-
let close_or_fail (p : path) : unit Proofview.tactic =
214+
(* Primary closer chain: fragment-keyed dispatch for verified
215+
reasons, plus the LLM-replay arm for [rocq-tactic-script]
216+
certs. Renamed from [close_or_fail] in M3.c — the public
217+
[close_or_fail] (below) is now a wrapper that catches a
218+
primary failure and tries LLM-assisted Tier-3 reconstruction
219+
before re-raising the error. *)
220+
let close_or_fail_primary (p : path) : unit Proofview.tactic =
215221
match p.cert, p.verify_reason with
216222
| None, _ ->
217223
CErrors.user_err Pp.(
@@ -229,14 +235,30 @@ let close_or_fail (p : path) : unit Proofview.tactic =
229235
Rocq Stdlib allowlist. Audit H1: a hallucinated [admit]
230236
or [Axiom Foo] is a tactic failure, never an admitted
231237
theorem. *)
232-
| Tier3_replay_deferred _ ->
238+
| Tier3_replay_deferred { trace_format = "rocq-tactic-script" } ->
239+
(* M3.b: tightened to only accept the Rocq-flavored cert.
240+
A [lean-tactic-script] cert reaching the Rocq closer is
241+
a configuration error (LLM adapter saw this bridge as
242+
Lean-flavored), not a tactic-soundness issue — surface
243+
it as a clear user error rather than silently trying to
244+
parse Lean tactics as Ltac. *)
233245
(match cert.payload with
234246
| Tier3_proof_trace { trace_data = `String script; _ } ->
235247
Llm_replay.replay_script script
236248
| _ ->
237249
CErrors.user_err Pp.(
238250
str "proof_broker: Tier-3 replay-deferred cert payload \
239251
is not a string trace; cannot replay"))
252+
| Tier3_replay_deferred { trace_format } ->
253+
CErrors.user_err Pp.(
254+
str (Printf.sprintf
255+
"proof_broker: LLM cert has trace_format=%S but the \
256+
Rocq home-system replayer only accepts \
257+
'rocq-tactic-script'. The LLM adapter likely saw \
258+
this IR as a Lean source_system; ensure the Rocq \
259+
reifier sets ir.source_system.name = \"rocq\" so \
260+
[Adapter_llm.dialect_of_ir] picks the Rocq dialect."
261+
trace_format))
240262
| Verified_envelope | Verified_farkas
241263
| Verified_case_split | Verified_tier3
242264
(* Verified_tier3_provenance gates the home-system closer
@@ -270,6 +292,73 @@ let close_or_fail (p : path) : unit Proofview.tactic =
270292
CErrors.user_err Pp.(
271293
str "proof_broker: internal — cert present but verify outcome missing")
272294

295+
(* --- M3.c: LLM-assisted Tier-3 reconstruction fallback ----------- *)
296+
297+
(* Run [Llm_replay.replay_script] on a candidate the LLM produced
298+
via reconstruction (rather than the primary LLM-as-backend
299+
path). On success, emit a [Feedback.msg_info] line naming the
300+
source trace format so the audit trail of how the goal closed
301+
is visible in the build output. Mirror of Lean's
302+
[replayReconstructedScript]. *)
303+
let replay_reconstructed_script (trace_format : string)
304+
(script : string) : unit Proofview.tactic =
305+
let announce =
306+
Proofview.tclLIFT (Proofview.NonLogical.make (fun () ->
307+
Feedback.msg_info (Pp.str (Printf.sprintf
308+
"proof_broker: closed via LLM Tier-3 reconstruction \
309+
(%s trace → Rocq Ltac script, kernel-checked, audit H1)."
310+
trace_format))))
311+
in
312+
Proofview.tclBIND (Llm_replay.replay_script script) (fun () -> announce)
313+
314+
(* Decide whether to invoke LLM-assisted reconstruction on the
315+
current cert + return a fallback tactic if so. Gates (no-op
316+
unless all hold):
317+
* a cert is in hand,
318+
* tier is 3 with a [Tier3_proof_trace] payload,
319+
* [trace_format] is not already [rocq-tactic-script] (which the
320+
primary LLM-replay arm at the top of [close_or_fail_primary]
321+
handles),
322+
* the SDK's [Llm_reconstruct.translate] returns a non-empty
323+
script (so this is silent when no endpoint is configured —
324+
the primary failure is re-raised).
325+
326+
The translate call runs immediately when this function is
327+
invoked; callers should only invoke it from the [tclORELSE]
328+
handler so the HTTP request is lazy (fires only on primary
329+
failure). *)
330+
let try_llm_reconstruct (ir : Ir.t) (cert_opt : Cert.t option)
331+
: unit Proofview.tactic option =
332+
match cert_opt with
333+
| None -> None
334+
| Some (cert : Cert.t) ->
335+
if cert.tier <> 3 then None
336+
else match cert.payload with
337+
| Tier3_proof_trace { trace_format = "rocq-tactic-script"; _ } ->
338+
None
339+
| Tier3_proof_trace { trace_format; _ } ->
340+
(match Proof_broker.Llm_reconstruct.translate ir cert with
341+
| Error _ -> None
342+
| Ok script ->
343+
Some (replay_reconstructed_script trace_format script))
344+
| _ -> None
345+
346+
(* Public closer: try the primary fragment-keyed chain; on any
347+
failure, try LLM-assisted Tier-3 reconstruction before
348+
re-raising the primary error. Mirror of Lean's [closeOrFail]
349+
try/catch wrap. Audit H1: the reconstruction path goes
350+
through the SAME audit gate [replay_reconstructed_script] →
351+
[Llm_replay.replay_script] uses for primary
352+
[rocq-tactic-script] certs — kernel replay + axiom-footprint
353+
subset check — so the LLM never widens the trust base. *)
354+
let close_or_fail (p : path) : unit Proofview.tactic =
355+
Proofview.tclORELSE
356+
(close_or_fail_primary p)
357+
(fun (primary_exn, primary_info) ->
358+
match try_llm_reconstruct p.ir p.cert with
359+
| None -> Proofview.tclZERO ~info:primary_info primary_exn
360+
| Some fallback -> fallback)
361+
273362
(* --- public entry points ------------------------------------------- *)
274363

275364
let run_close (names : Names.Id.t list option) : unit Proofview.tactic =

rocq-bridge/src/pb_rocq_main.mli

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,13 @@ val run_close_term : Names.Id.t list option -> unit Proofview.tactic
3838
arity > 2, non-Le hypotheses, etc.) — the user explicitly
3939
opted into term mode by typing [proof_broker_term] over
4040
plain [proof_broker]. *)
41+
42+
val replay_reconstructed_script :
43+
string -> string -> unit Proofview.tactic
44+
(** [replay_reconstructed_script trace_format script] feeds the
45+
LLM-translated script through [Llm_replay.replay_script]
46+
(the audit-H1 gate) and, on success, emits a
47+
[Feedback.msg_info] line naming the source trace format so
48+
the audit trail is visible in build output. Exported for the
49+
test-only [llm_reconstruct_test] tactic to drive the
50+
reconstruction-side closer without a live LLM endpoint. *)

rocq-bridge/theories/Test.v

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -703,3 +703,49 @@ Proof.
703703
assert_fails (llm_replay_test "@@@ not lean @@@").
704704
exact I.
705705
Qed.
706+
707+
(* ============================================================
708+
LLM-assisted Tier-3 reconstruction fallback
709+
(roadmap §Phase 3 deliverable 4, Rocq parity M3).
710+
711+
[llm_reconstruct_test "<traceFmt>" "<script>"] simulates a
712+
successful [Llm_reconstruct.translate] call (the LLM was
713+
asked to translate an un-replayable Tier-3 trace and
714+
returned the script) and routes the candidate through the
715+
SAME audit-H1 gate the production fallback uses. The
716+
translation step itself is a pure I/O wrapper around an
717+
untrusted oracle — no soundness contribution — so swapping
718+
the live SDK call for a literal script is faithful to what's
719+
being tested: that a candidate script makes it through
720+
[replay_reconstructed_script] → [Llm_replay.replay_script],
721+
with the audit-H1 gate firing identically. Mirror of the
722+
llm_reconstruct_test block in lean-bridge/Test/Tactic.lean. *)
723+
724+
(* Positive: a translated script the kernel independently
725+
accepts (axiom-free [intros; reflexivity]) closes the goal
726+
via the reconstruction fallback's replay closer. The build
727+
log carries the "closed via LLM Tier-3 reconstruction …"
728+
audit line. *)
729+
Theorem pb_llm_reconstruct_axiom_free : forall x : Z, x = x.
730+
Proof.
731+
llm_reconstruct_test "alethe-2024" "intros x; reflexivity".
732+
Qed.
733+
734+
Print pb_llm_reconstruct_axiom_free.
735+
Print Assumptions pb_llm_reconstruct_axiom_free.
736+
737+
(* Negative: a translation that runs without error but does not
738+
close the goal is a tactic failure, never a silent pass. *)
739+
Example pb_llm_reconstruct_rejects_non_closing : True.
740+
Proof.
741+
assert_fails (llm_reconstruct_test "alethe-2024" "idtac").
742+
exact I.
743+
Qed.
744+
745+
(* Negative: a translation that does not parse as Rocq Ltac is
746+
a tactic failure, not an admitted theorem. *)
747+
Example pb_llm_reconstruct_rejects_unparsable : True.
748+
Proof.
749+
assert_fails (llm_reconstruct_test "tstp-fof" "@@@ not lean @@@").
750+
exact I.
751+
Qed.

0 commit comments

Comments
 (0)