Add experimental ReAnchor calibration optimizer for Predict

[isaacbmiller]

Stacked on #10463.

dspy.experimental.ReAnchor calibrates decision outputs in Predict programs against a whole-program metric. It works with TypeSafe and generative LMs, without changing instructions, criteria, or demos.

Output	Fitted setting	Effect
Noul / bool	threshold	Converts P(True) to a Boolean
Score	cuts	Selects .level; leaves continuous .value unchanged
Choice / Literal	weights	Reweights option probabilities to select a value

from dspy.experimental import ReAnchor

optimizer = ReAnchor(metric=metric)
tuned = optimizer.compile(program, trainset=trainset, valset=valset)
print(optimizer.report)
tuned.save("tuned.json")

Search and acceptance

Candidates come from observed probabilities. Threshold search also tries zero when P(True)=0 is observed. A setting changes only when it improves the training score and passes a fold check: settings selected on the other folds must improve the combined held-out score. This discourages gains confined to small portions of the dataset, but can accept them when they recur across folds.

On generative LMs, ReAnchor can ask native bool and Literal outputs for probabilities while preserving their Python return types. It keeps this change only when the calibrated output beats the native output under the same fold check.

Execution and saved state

• compile returns a copy; the original program is unchanged.
• Numeric settings live in each predictor’s fields and use normal program save/load.
• Identical requests reuse cached answers. Changed upstream decisions can produce new downstream requests; native-output promotion also changes requests. Caching is required by default, but does not guarantee a fixed request count.
• Evidence collection is scoped to the current execution context and its DSPy workers.
• report includes training scores, fitted settings, observed evidence summaries, and fold-check results. Optional validation data is scored but never fitted. log_dir writes report.json.
• The API is experimental. RLM decision outputs are unsupported. Single-option Choice calibration currently raises an error.

Author-reported Jev results

Calibration-only results from the original PR evaluation; these benchmarks have not been rerun for the subsequent implementation changes.

Benchmark	Before	After
Email triage (1,000 test emails, unseen senders)	0.692	0.749
Trend-report judge (8 test files)	0.813	0.874

[greptile-apps]

Confidence Score: 5/5

Safe to merge; there are no outstanding blocking issues.

Summary

• Adds the experimental ReAnchor optimizer for fitting Boolean thresholds, score cuts, and choice weights against whole-program metrics.
• Preserves existing settings unless a fitted candidate produces a strictly better, fold-validated result.
• Handles adjacent floating-point Boolean probabilities by retaining the upper separating threshold.

Merge Safety

Safe to merge.

_{Reviews (16) · Last reviewed commit: "Retain Boolean thresholds between adjace..."}