feat(gepa): parallelize candidate evaluation

[isaacbmiller]

Why this PR exists

Stacked on #10209, which upgrades DSPy to GEPA 0.1.4 while retaining GEPA's sequential candidate-evaluation fallback.

GEPA 0.1.4 can sample multiple proposals in one iteration, but leaves execution policy to adapter.batch_evaluate(). Without that method, DSPy evaluates candidates one after another. This PR implements that adapter capability while preserving DSPy's normal meaning of num_threads as the total evaluation budget.

Before and after

	#10209	This PR
Candidate evaluation	Ordered sequential fallback	Concurrent within num_threads
Examples within one candidate	Up to num_threads	A share of the same budget
Maximum controlled concurrency	num_threads	num_threads
New concurrency setting	None	None
max_reflection_cost	Explicitly unsupported	Still explicitly unsupported

Candidate and example concurrency do not multiply. Four candidates with num_threads=8 receive two example workers each; one candidate receives all eight. If candidates outnumber threads, at most eight run at once with one worker each.

With GEPA's default one-proposal strategy, execution remains equivalent to ordinary DSPy evaluation. Strategies such as IndependentSampling(4) or PxNSampling(2, 2) cause DSPy to partition the existing budget across candidates.

Design

DspyAdapter.batch_evaluate():

1. receives ordered (candidate, batch) items from GEPA;
2. resolves DSPy's existing num_threads setting;
3. runs at most min(number_of_candidates, num_threads) candidates concurrently;
4. divides the thread budget across candidate evaluations;
5. preserves DSPy ContextVar state and callback ancestry;
6. preserves result order;
7. forwards capture_traces; and
8. propagates exceptions without retry.

GEPA proposal batch
└── candidate workers × example workers ≤ num_threads

It uses a small ThreadPoolExecutor rather than DSPy's ParallelExecutor. ParallelExecutor catches failures and may resubmit stragglers; at this boundary, resubmission could duplicate paid LM calls.

Deliberately not included: reflection-cost accounting

GEPA's max_reflection_cost expects cumulative total_cost. DSPy's LM history records response cost, but history can be disabled, bounded/evicted, cleared, or replaced. Summing retained history would be unreliable, while adding a mutable _total_cost would create a second source of truth plus locking and serialization behavior.

This PR does neither. Non-None max_reflection_cost remains explicitly rejected. Cost budgeting should use a coherent DSPy-wide accounting design rather than GEPA-specific LM state.

User example

import dspy
from gepa.strategies.proposal_sampling import IndependentSampling
from gepa.strategies.proposal_selection import BestImprovement

optimizer = dspy.GEPA(
    metric=metric,
    max_metric_calls=2_000,
    reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32_000),
    reflection_minibatch_size=8,
    num_threads=8,  # Total across candidates and examples, as elsewhere in DSPy.
    log_dir="./gepa-run",
    track_stats=True,
    gepa_kwargs={
        "sampling_strategy": IndependentSampling(4),
        "selection_strategy": BestImprovement(),
        "acceptance_criterion": "strict_improvement",
    },
)

optimized = optimizer.compile(student, trainset=trainset, valset=valset)

This samples four independent proposals, evaluates them concurrently within eight total threads, retains the best improvement, and supports checkpoint resume through log_dir.

Measured behavior

A controlled end-to-end GEPA 0.1.4 benchmark compared this branch with #10209. Both used num_threads=4 for one complete IndependentSampling(4) iteration with four distinct improvements, AllImprovements, and an 80 ms simulated candidate evaluation:

	#10209 fallback	Shared-budget batching
Median of 3 runs	1.051 s	0.333 s
Peak candidates	1	4
Threads per candidate	N/A	1
Metric calls	13	13
Candidates including seed	5	5

That is a 3.16× speedup with identical work/results, consistent with GEPA's advertised “up to about 3 to 4×” result. It is a controlled latency benchmark, not a guarantee or a reproduction of GEPA's quality benchmarks.

A second probe used the real DSPy Evaluate path with four candidates × eight examples and num_threads=8. Peak simultaneous example evaluations remained exactly 8, proving this does not create 32 workers. All four ordered eight-example results were retained.

Risks and compatibility

Scheduling shape

More candidate program instances can be active concurrently, while each receives fewer example workers. Total DSPy-controlled concurrency remains bounded by num_threads. Uneven candidate durations may change utilization, but users do not need a second concurrency knob.

GEPA 0.1.4 → 0.1.5

GEPA 0.1.4 calls batch_evaluate(items) and always requests trajectories, including accepted-candidate full-validation evaluation. That is gepa-ai/gepa#428.

This adapter accepts the trace-aware contract expected from gepa-ai/gepa#430: batch_evaluate(items, *, capture_traces=...). Under that change, reflection requests traces while seed/full-validation evaluation do not. Against 0.1.4, the keyword is omitted and DSPy's default remains True, so the known limitation remains. Updating to 0.1.5 will activate correct routing without another adapter redesign; no temporary monkeypatch is included.

Review guide

The complete diff against #10209 is three files:

• dspy/teleprompt/gepa/gepa_utils.py: candidate batching and thread-budget partitioning
• dspy/teleprompt/gepa/gepa.py: documents shared num_threads semantics
• tests/teleprompt/test_gepa.py: overlap, ordering, context, trace forwarding, and 2+2 / 4 budget allocation

There are no new public concurrency parameters and no BaseLM, history, cost, lock, copy, or serialization changes.

Verification

• uv run pytest tests/teleprompt -q — 82 passed
• uv run pytest -q tests/clients/test_lm.py tests/teleprompt/test_gepa.py — 108 passed
• uv run pre-commit run --all-files — passed
• uv lock --check — passed
• git diff --check — passed

[greptile-apps]

Greptile Summary

GEPA now evaluates program variations concurrently and divides the configured thread budget across them, including any remainder. Execution checks confirmed that the complete budget is used, result order is preserved despite out-of-order completion, and caller context, tracing, and callback identity remain available to every worker.

The existing legacy language-model cost-accounting problem remains: bridged typed calls retain a per-response cost but do not maintain cumulative total_cost accounting. Budget-limited optimization can therefore fail to reflect actual aggregate provider spend.

Confidence Score: 4/5

The GEPA concurrency change behaved correctly in exercised allocation, ordering, context, tracing, and callback flows, but legacy typed-bridge calls still lack cumulative cost tracking.

One actionable non-security finding remains. isaacbmiller stated that the issue was fixed, but the current code has no BaseLM.total_cost accumulator and the executed synchronous and asynchronous explicit-request and experimental paths all observed total_cost as absent.

Files Needing Attention: dspy/clients/base_lm.py

T-Rex Logs

What T-Rex did

• Ran a GEPA concurrency harness against both revisions using eight threads and three work items; observed parent allocation [2,2,2] and PR allocation [3,3,2], with all workers preserving caller LM, trace setting, and callback call ID; focused batching tests and the complete GEPA test module passed.
• Disproved allocation-loss, ordering, and context/concurrency hypotheses by comparing harness results; the PR allocated more work items than the parent, forced out-of-order completion still produced input-ordered results, and workers retained context such as LM, trace, and ACTIVE_CALL_ID=parent-call.
• Posted a P1 finding proof and attached the legacy bridge cost reproduction scripts and outputs to support the finding details.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. dspy/clients/base_lm.py, line 196 (link)

   Executed a synchronous and asynchronous reproduction using a legacy-contract BaseLM thr...

   • Bug
     • Executed a synchronous and asynchronous reproduction using a legacy-contract BaseLM through explicit LMRequest and dspy.contextexperimental=True. Each response and history entry retained the expected 0.5 cost, but all four paths observed that BaseLM.totalcost is absent, so cumulative cost accounting is not implemented for these bridged calls.
   • Cause
     • T-Rex reproduced this while running the changed behavior, but it did not return a separate root-cause sentence.
   • Fix
     • Update the changed code so this failing path is handled, then rerun the same T-Rex check to confirm it passes.
   Artifacts

   
   

   Legacy bridge cost reproduction script source

   • Captured the exact executable Python reproduction that exercises legacy-contract BaseLM calls through explicit LMRequest and experimental synchronous and asynchronous paths, showing the test condition under review.

   Legacy bridge cost reproduction output

   • Executed the captured reproduction against the repository and observed response and history cost of 0.5 in all four flows while BaseLM total_cost was absent, proving the requested exactly-once aggregate cost contract is not implemented.

   

   _{Ran code and verified through T-Rex}

_{Reviews (12): Last reviewed commit: "fix(gepa): distribute candidate thread r..." | Re-trigger Greptile}