fix(gepa): align GEPA auto budget with evaluation costs

[j341nono]

Fixes #10245.

This PR updates GEPA.auto_budget() to reflect the current GEPA evaluation flow.

The bug

Before this change, the auto budget was calculated approximately as:

budget = V + 5C + 35T + FV

where:

F = floor((T + 1) / 5) + 1 + indicator(T < 5)

where V is the validation-set size, C is the candidate count, T is the estimated number of trials, and F is the assumed number of periodic full evaluations.

This calculation had three stale assumptions:

• the reflection minibatch size was always 35
• each candidate required five bootstrapping evaluations
• full validation occurred periodically every five trials

The current GEPA loop instead evaluates both the parent and child on the configured reflection minibatch and performs full validation only when a proposal passes minibatch acceptance.

Consequently, changing reflection_minibatch_size changed the actual metric-call cost but did not change the calculated auto budget.

The fix

The updated budget calculation is:

budget = V + 2RT + AV + K(5 + V)

where:

• V is the validation-set size
• R is the effective reflection minibatch size
• T is the estimated number of reflective trials
• A is the number of accepted-candidate validations reserved by auto mode
• K is the maximum number of merge attempts

The 2RT term accounts for evaluating both the parent and child on each reflection minibatch. AV reserves full-validation cost for accepted candidates. Each merge attempt reserves five metric calls for merge screening plus one full validation, represented by K(5 + V).

For custom batch samplers, the effective minibatch size is resolved from the sampler's minibatch_size attribute when available. Otherwise, the legacy value of 35 is used with a warning.

To preserve backward compatibility, auto_budget() continues to accept the legacy minibatch_size and full_eval_steps parameters. full_eval_steps is retained for compatibility but no longer affects the calculation.

compile() also calls auto_budget() using the original three-keyword form, so subclasses overriding the legacy signature continue to work.

The result remains a budget allocation rather than an exact prediction because rejected proposals skip full validation and custom GEPA policies may have different evaluation costs.

Verification

Added regression tests covering:

• minibatch-aware budget calculation
• merge-related evaluation costs
• propagation of reflection_minibatch_size through compile()
• default and custom batch samplers, including fallback behavior
• legacy three-argument auto_budget() calls
• legacy minibatch_size and full_eval_steps keywords
• subclasses overriding the previous auto_budget() signature

All focused GEPA tests and pre-commit checks pass.

[greptile-apps]

Greptile Summary

GEPA’s automatic metric-call budgeting now accounts for reflection minibatches, full validation of accepted programs, and configured merge capacity. The earlier custom-sampler issue is resolved: automatic mode accepts samplers with and without a minibatch_size attribute, forwards them unchanged to GEPA, and assigns a usable metric-call budget.

Confidence Score: 5/5

No blocking failure remains.

Runtime checks exercised automatic compilation with sized and opaque custom samplers, and focused GEPA budget tests passed.

T-Rex Logs

What T-Rex did

• I ran a deterministic GEPA runtime probe using GEPA.compile() with auto='light', reflection_minibatch_size=None, and both sized and opaque custom samplers; the compilation forwarded each original sampler, retained reflection_minibatch_size=None, and produced positive budgets (61 and 721).
• GEPA 0.1.4 runs were invoked with both sampler forms and completed optimization.
• The focused automatic-budget tests were executed and 15 relevant tests passed.
• The default-reflection budget probe consumed 67 evaluations and stopped at the configured budget.
• The merge probe began successfully and accepted its first reflective candidate, but did not produce a merge before the agent execution limit, so no merge could be established or rejected under the budget claim.

_{Ran code and verified through T-Rex}

_{Reviews (3): Last reviewed commit: "fix(dspy): preserve GEPA auto-budget API..." | Re-trigger Greptile}

[j341nono]

Updated the PR to preserve backward compatibility for auto_budget().

The legacy minibatch_size / full_eval_steps parameters are still accepted, and compile() now keeps the original three-keyword call form so subclasses overriding the previous signature continue to work. Added regression coverage for these cases as well.

[j341nono]

Hi! Just a quick follow-up on this PR.

It looks like the GitHub Actions workflows are currently awaiting maintainer approval. The PR is still up to date from my side, and I'm happy to make any additional changes if needed.

Thanks!