feat(dspy): support gepa 0.1.4 — parallel proposals, reflection-cost budget, batch evaluation, checkpoint-safe adapter state

[dbreunig]

📝 Changes Description

This PR bumps the pinned gepa dependency from 0.1.1 to 0.1.4 and surfaces the new upstream capabilities as first-class dspy.GEPA features:

• Pin bump to gepa[dspy]==0.1.4 — every API surface DSPy touches was verified unchanged upstream; the full GEPA test suite passes with no code changes required.
• New dspy.GEPA constructor args, forwarded verbatim to gepa.optimize: sampling_strategy / selection_strategy (gepa 0.1.4's multi-proposal iterations), acceptance_criterion ("strict_improvement", "improvement_or_equal", or a custom criterion), max_reflection_cost (USD budget for reflection spend), wandb_attach_existing / mlflow_attach_existing / tracking_key_prefix (embed GEPA in a training loop that owns the tracker run).
• TrackedReflectionLM — the reflection callable passed to gepa now exposes real cost/token totals derived from the reflection LM's history, so gepa's max_reflection_cost stopper actually works (a bare lambda would be wrapped in gepa's zero-cost TrackingLM and the stopper would never fire). The budget is scoped to the run via a baseline snapshot, and compile() raises if max_reflection_cost is set while LM history is disabled.
• DspyAdapter.batch_evaluate — gepa 0.1.4 discovers this seam automatically; multi-proposal iterations now evaluate up to 8 candidates concurrently (each still parallelized across examples by num_threads), with dspy's thread-local settings overrides propagated into the worker threads and per-worker usage-tracker isolation matching ParallelExecutor's pattern.
• get_adapter_state / set_adapter_state — persists the adapter's RNG state through gepa's checkpoint save/resume so log_dir resumes are deterministic.

✅ Contributor Checklist

• Pre-Commit checks are passing (locally and remotely)
• Title of your PR / MR corresponds to the required format
• Commit message follows required format {label}(dspy): {message}

⚠️ Warnings

• Inherited gepa 0.1.4 behavior change: upstream now routes all engine evaluations — including full valset evals — through the batch-evaluate path with trace capture enabled. Two consequences: (1) valset evaluations build per-example trajectories that the engine discards (memory/perf overhead proportional to valset size), and (2) a metric that conditions on its trace argument now receives a real trace during valset scoring where 0.1.1 passed trace=None, which can change valset scores for trace-sensitive metrics. This is upstream gepa's design (the batch_evaluate contract requires trajectories), not something this PR opts into.
• gepa 0.1.4's state schema is v5; run dirs written by 0.1.1 are migrated automatically on load, but a run started on 0.1.4 cannot be resumed on 0.1.1.
• Budget accounting upstream is more accurate (refiner calls now count toward max_metric_calls), so runs will not be iteration-for-iteration identical to 0.1.1 at the same budget.
• Users who previously passed any of the seven newly surfaced kwargs through gepa_kwargs would now get a duplicate-kwarg TypeError — impossible in practice, since DSPy was pinned to 0.1.1 where these parameters did not exist.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d08ca7d3fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Count typed LM history entries toward reflection cost

When reflection_lm uses DSPy's typed LM contract or runs with experimental=True, its history contains LMHistoryEntry objects, which implement Mapping but are not dict instances. This filter therefore ignores every reflection call and leaves total_cost at zero, so GEPA's max_reflection_cost stopper never fires and can allow unbounded spend. Accept mapping-like entries rather than only concrete dictionaries.

Useful? React with 👍 / 👎.

[dbreunig]

Fixed in 76216b2 — the filter is now isinstance(entry, Mapping), so typed LMHistoryEntry objects count toward the budget. Regression test: test_tracked_reflection_lm_counts_mapping_history_entries.

[chatgpt-codex-connector]

Preserve cumulative cost when LM history evicts entries

When a run makes more calls than dspy.settings.max_history_size, LM.update_history() evicts old entries, so this recomputed sum is only a rolling window rather than cumulative spend. For example, with a history size of 1 and $1 calls, total_cost remains $1 forever and a $2 budget never stops the run; a nonempty baseline can keep it at zero even longer. Since compile() explicitly accepts any nonzero history size, cost must be accumulated independently of the bounded history window.

Useful? React with 👍 / 👎.

[dbreunig]

Fixed in 76216b2 — TrackedReflectionLM no longer recomputes from the history window. It accumulates each entry once (keyed by uuid) into run-scoped running totals, synced on every wrapper call, every stripped_lm_call, and every property read, so instruction-proposal spend is tallied before eviction can occur even at max_history_size=1. Remaining caveat (documented): a custom proposer making more direct LM calls than the window holds within a single proposal step can still have the excess evicted before the next sync. Regression test: test_tracked_reflection_lm_survives_history_eviction reproduces the 3×$0.60 / window=1 scenario and asserts $1.80.

[greptile-apps]

Greptile Summary

GEPA now accumulates reflection spend independently of the bounded LM-history window, rejects reflection-cost limits when history cannot support accounting, and preserves callback parentage for concurrent batch evaluations. Executed checks disproved the two previously reported failures: three $0.60 reflection calls with a one-entry history window correctly total $1.80, and concurrent worker callbacks retain the enclosing trace as their parent.

Confidence Score: 5/5

No blocking failure remains.

No accepted P0 or P1 finding remains after exercising reflection-budget accounting, history validation, and concurrent callback propagation.

T-Rex Logs

What T-Rex did

• Ran `.venv/bin/python trex-artifacts/reflection-budget-00-repro-script.py --calls 3` with `max_history_size=1` (exit 0). Each retained history window contained `$0.60`, while the tracked total progressed from `$0.60` to `$1.20` to `$1.80`.
• Ran `.venv/bin/pytest -q tests/teleprompt/test_gepa.py -k 'tracked_reflection_lm'`; all four selected regression tests passed.
• This confirms reflection spend is accumulated before history entries are evicted and the configured budget observes the cumulative amount.
• Ran `uv run --frozen python trex-artifacts/gepa-callback-lineage-02-after.py` from `/home/user/repo` (exit 0) with two batch items inside an active enclosing callback trace.
• Both worker `Evaluate` callback events recorded the enclosing call ID as `parent_call_id`; they did not become root callbacks.

_{Ran code and verified through T-Rex}

_{Reviews (4): Last reviewed commit: "fix(gepa): preserve reflection cost vali..." | Re-trigger Greptile}

[dbreunig]

@greptile review

[BjarkeTornager]

Thanks for working on this. GEPA 0.1.4 support in DSPy would be very helpful for me as well. Would it be possible to rebase the PR and address the remaining Greptile issue?

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.