Skip to content

Commit eedb9e7

Browse files
ampagentIsaac Miller
andcommitted
Unify ReAnchor field calibration and guard floating-point candidates
Try probability-based execution for every compatible output and restore its original configuration unless calibration is accepted. Keep client inspection confined to the optional cache check. Skip unrepresentable gap midpoints to prevent duplicate Score cuts found in the live trend benchmark. Amp-Thread-ID: https://ampcode.com/threads/T-01a0bf8d-a1dd-7603-906c-ea06c46ff7f7 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
1 parent c2ac1a7 commit eedb9e7

5 files changed

Lines changed: 119 additions & 70 deletions

File tree

‎docs/docs/api/experimental/ReAnchor.md‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,12 @@ print(optimizer.report)
4141
ReAnchor searches each output's numeric settings against the whole-program
4242
metric. A new setting must improve the training score and pass a fold check.
4343

44+
For every compatible output, ReAnchor saves the original configuration and score,
45+
enables probability-based execution, and fits the numeric settings. It then compares
46+
the fitted behavior against the original using a fold check. If rejected, it restores
47+
the exact original configuration, including removing an entry that was originally absent.
48+
The adapter handles the backend used on each call; calibration does not inspect it.
49+
4450
Before fitting each output, ReAnchor runs the program and records that output's
4551
probabilities on every call. A threshold anywhere between two neighboring P(True) values makes the
4652
same decisions, so ReAnchor tries the midpoint of each gap between them. For
@@ -78,7 +84,9 @@ additional backend calls. Native-output promotion also changes the request.
7884

7985
`compile` requires caching by default to avoid repeating identical backend
8086
calls; it does not guarantee a fixed request count. Pass `require_cache=False`
81-
to run without this check.
87+
to run without this check. The cache check inspects statically bound or globally
88+
configured clients. For programs selecting clients inside `forward()`, disable
89+
the check and manage caching on those clients.
8290

8391
## Native outputs on a generative LM
8492

@@ -88,19 +96,18 @@ has an entry in the predictor's `fields`, `Predict` asks the LM for
8896
probabilities instead. `Predict` then applies the threshold or weights to pick
8997
the value, and the output still returns a `bool` or a `Literal` member.
9098

91-
ReAnchor adds this entry for each native output on a generative LM. It keeps the
92-
entry only when the fitted setting beats the native value under the same fold
93-
check. Otherwise it removes the entry, and the output keeps its native behavior.
94-
The `report` row for a kept entry has `"promoted": True`. The first pass with
95-
probabilities sends new requests, because the request changes.
99+
The first pass with probabilities sends new requests, because the request changes.
100+
If the fitted behavior is rejected, restoring the original configuration returns
101+
an unconfigured native field to direct generation.
96102

97103
```python
98104
dspy.configure(lm=dspy.LM("your-provider/your-model")) # Use your model's identifier.
99105
tuned = ReAnchor(metric).compile(dspy.Predict(Match), trainset=trainset)
100106
```
101107

102-
A System One model such as Jev always returns probabilities, so this step does
103-
not apply there.
108+
A System One model such as Jev always returns probabilities. Enabling probability
109+
execution leaves its request unchanged; restoring a field restores its original
110+
decoding settings, not direct generation.
104111

105112
## Errors
106113

@@ -123,7 +130,10 @@ your backend's load.
123130
skipped it. Each fitted row has an `observed` entry with the number of calls
124131
and settings tried; threshold and cut reports also summarize observed values. Its
125132
`fold_check` entry counts the search steps whose better training score passed
126-
or failed the fold check.
133+
or failed the fold check. Per-field `train_score_original` is the score before
134+
enabling probabilities, `train_score_at_start` is the probability-based baseline,
135+
and `train_score` describes the retained behavior. Rejected configurations have
136+
a `skipped` reason instead of a fitted `value`.
127137

128138
Your metric may return a number or a `dspy.Prediction` with a `score`.
129139

‎dspy/teleprompt/reanchor/calibrate.py‎

Lines changed: 21 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,10 @@
2525
only when those held-out scores beat the current setting's. This discourages gains confined to
2626
small portions of the dataset, but can accept them when they recur across folds.
2727
28-
On a generative LM, a native `bool` or `Literal` output returns its value without probabilities
29-
unless it has an entry in `fields`. Calibration adds that entry, which asks the LM for
30-
probabilities, and keeps it only when the fitted setting beats the native output under the same
31-
fold check.
28+
Calibration enables probability-based execution for every compatible output by adding an entry
29+
in `fields`. After fitting, it compares against the original behavior under the same fold check
30+
and restores the original field configuration unless the fitted behavior wins. The adapter
31+
handles probability requests for the backend used on each call.
3232
3333
"""
3434

@@ -61,34 +61,23 @@ def predictors(program) -> list[tuple[str, Predict]]:
6161
return [(name, p) for name, p in program.named_parameters() if isinstance(p, Predict) and decision_outputs(p)]
6262

6363

64-
def resolved_lm(predict: Predict):
65-
"""The client the predictor calls: its own, or the configured one."""
64+
def caches(predict: Predict) -> bool:
65+
"""Check caching for a statically bound or configured client."""
6666
lm = predict.lm or dspy.settings.lm
6767
if lm is None:
68-
raise ValueError("ReAnchor needs an LM or decision client, bound to the predictor or configured globally.")
69-
return lm
70-
71-
72-
def caches(predict: Predict) -> bool:
73-
"""Whether repeated requests from the predictor are answered from the cache."""
74-
lm = resolved_lm(predict)
68+
raise ValueError(
69+
"ReAnchor cannot check caching without a bound or globally configured client. "
70+
"For runtime client selection, pass require_cache=False."
71+
)
7572
enabled = predict.config.get("cache", getattr(lm, "cache", False))
7673
return bool(enabled) and getattr(lm, "_cache_responses", True)
7774

7875

79-
def _state(predict: Predict, fields: dict) -> DecisionState:
80-
system_one = getattr(resolved_lm(predict), "supports_decision_requests", False) is True
81-
return DecisionState(predict.signature, fields, system_one=system_one)
82-
83-
84-
def evidenced(predict: Predict) -> set[str]:
85-
"""The outputs the predictor currently decodes from probabilities."""
86-
return set(_state(predict, predict.fields).types)
87-
88-
8976
def effective(predict: Predict, field: str) -> dict:
9077
"""The output's full configuration: its stored overrides on top of its type's defaults."""
91-
return _state(predict, {**predict.fields, field: predict.fields.get(field, {})}).fields[field]
78+
fields = dict(predict.fields)
79+
fields.setdefault(field, {})
80+
return DecisionState(predict.signature, fields).fields[field]
9281

9382

9483
def metric_value(metric: Callable, example, pred) -> float:
@@ -138,8 +127,7 @@ def score() -> list[float]:
138127
for name, predict in predictors(program):
139128
for field, kind in decision_outputs(predict).items():
140129
original = copy.deepcopy(predict.fields.get(field))
141-
promoted = field not in evidenced(predict)
142-
native = score() if promoted else None
130+
before = score()
143131
predict.fields[field] = effective(predict, field)
144132
evidence = _observe(program, predict, field, trainset, metric, num_threads)
145133
row = {"predictor": name, "field": field}
@@ -153,20 +141,11 @@ def score() -> list[float]:
153141
base = score()
154142
best, fitted, observed, check = fit(predict, field, kind, evidence, score, base)
155143
row.update(parameter=parameter, value=best, train_score=_mean(fitted), train_score_at_start=_mean(base))
156-
row.update(observed=observed, fold_check=check)
157-
if promoted:
158-
row["train_score_native"] = _mean(native)
159-
if _select(native, [(fitted, (), None)])[0] is None:
160-
_restore(predict, field, original)
161-
row = {
162-
"predictor": name,
163-
"field": field,
164-
"skipped": "probabilities did not beat the native output",
165-
"train_score_native": row["train_score_native"],
166-
"train_score": row["train_score"],
167-
}
168-
else:
169-
row["promoted"] = True
144+
row.update(observed=observed, fold_check=check, train_score_original=_mean(before))
145+
if _select(before, [(fitted, (), None)])[0] is None:
146+
_restore(predict, field, original)
147+
row.pop("value")
148+
row.update(skipped="fitted behavior did not beat the original", train_score=_mean(before))
170149
report.append(row)
171150
return report
172151

@@ -211,7 +190,8 @@ def _gaps(values: list[float], lo: float, hi: float, geometric: bool = False) ->
211190
gaps = []
212191
for a, b in itertools.pairwise(points):
213192
middle, width = (math.sqrt(a * b), math.log(b / a)) if geometric else ((a + b) / 2, b - a)
214-
gaps.append((_tidy(middle, a, b), width))
193+
if a < middle < b: # Adjacent floats can round the midpoint onto a boundary.
194+
gaps.append((_tidy(middle, a, b), width))
215195
return gaps
216196

217197

‎dspy/teleprompt/reanchor/reanchor.py‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,18 +21,17 @@ class ReAnchor(Teleprompter):
2121
backend's probabilities without adding request parameters. Identical requests reuse cached
2222
answers, but changed upstream decisions can produce new downstream requests.
2323
24-
On a generative LM, a native `bool` or `Literal` output is answered without probabilities.
25-
ReAnchor gives such an output an entry in the predictor's `fields`, which asks the LM for
26-
probabilities, and keeps the entry only when the fitted setting beats the native output on the
27-
training set.
24+
ReAnchor enables probability-based execution for every compatible output through the predictor's
25+
`fields`. It keeps each fitted configuration only when it beats the original behavior and passes
26+
a fold check; otherwise it restores the original configuration, including absent field entries.
2827
2928
Args:
3029
metric: Per-example metric to maximize, as in `dspy.Evaluate`. It may return a number, or a
3130
`dspy.Prediction` with a `score`.
3231
num_threads: Evaluation concurrency, as in `dspy.Evaluate`.
3332
log_dir: When set, report.json is written here.
34-
require_cache: When True, `compile` raises if a predictor's client does not cache responses.
35-
Set it to False to calibrate anyway; every candidate setting then sends new requests.
33+
require_cache: When True, check caching on statically bound or globally configured clients.
34+
Set it to False for runtime client selection or to allow uncached requests.
3635
3736
After `compile`, `report` holds the fitted parameters and the metric's mean before and after
3837
calibration, on the training set and on the validation set when one is given.
@@ -60,7 +59,7 @@ def compile(self, student, *, trainset, valset=None):
6059
found = predictors(program)
6160
if not found:
6261
raise ValueError("The student must contain at least one Predict with a decision output.")
63-
uncached = [name for name, predict in found if not caches(predict)]
62+
uncached = [name for name, predict in found if not caches(predict)] if self.require_cache else []
6463
if self.require_cache and uncached:
6564
raise ValueError(
6665
f"Predictors {uncached} do not cache responses, so every candidate setting would send new requests. "

‎tests/teleprompt/reanchor/test_reanchor.py‎

Lines changed: 54 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ def forward(self, pair):
9090
program = ReAnchor(metric, num_threads=2).compile(Wrapper(), trainset=examples())
9191
assert isinstance(program, Wrapper)
9292
assert program.judge.fields["match"]["threshold"] == 0.8
93-
assert "threshold" in program.judges[0].fields["match"]
93+
assert program.judges[0].fields == {} # No further gain after calibrating the first judge.
9494

9595

9696
class Items(dspy.Module):
@@ -193,19 +193,25 @@ def test_a_native_bool_on_a_generative_lm_is_promoted_when_probabilities_score_b
193193
assert student.fields == {}
194194
assert program.fields == {"match": {"threshold": 0.8}}
195195
row = optimizer.report["fitted"][0]
196-
assert row["promoted"] is True and row["train_score_native"] == 0.5556 and row["train_score"] == 0.7778
196+
assert "skipped" not in row and row["train_score_original"] == 0.5556 and row["train_score"] == 0.7778
197197

198198

199-
def test_a_native_bool_stays_native_when_probabilities_do_not_score_better():
199+
@pytest.mark.parametrize("accurate_probabilities", [True, False])
200+
def test_a_native_bool_stays_native_when_probabilities_do_not_score_better(accurate_probabilities):
200201
def right(inputs, evidence):
201202
answer = inputs["pair"].startswith("same")
202-
return {"match": noul(0.9 if answer else 0.1)} if evidence else {"match": answer}
203+
probability = (0.9 if answer else 0.1) if accurate_probabilities else 0.9
204+
return {"match": noul(probability)} if evidence else {"match": answer}
203205

204206
dspy.configure(lm=ComputedLM(right, adapter=dspy.JSONAdapter()))
205207
optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
206208
program = optimizer.compile(dspy.Predict(Sig), trainset=examples())
207209
assert program.fields == {}
208-
assert optimizer.report["fitted"][0]["skipped"] == "probabilities did not beat the native output"
210+
row = optimizer.report["fitted"][0]
211+
assert row["skipped"] == "fitted behavior did not beat the original"
212+
assert row["train_score"] == row["train_score_original"] == 1.0
213+
assert row["train_score_at_start"] == (1.0 if accurate_probabilities else 0.5556)
214+
assert "value" not in row
209215

210216

211217
def test_a_native_bool_stays_native_when_probabilities_win_only_one_example():
@@ -222,7 +228,49 @@ def one_miss(inputs, evidence):
222228
optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
223229
program = optimizer.compile(dspy.Predict(Sig), trainset=train)
224230
assert program.fields == {}
225-
assert optimizer.report["fitted"][0]["skipped"] == "probabilities did not beat the native output"
231+
assert optimizer.report["fitted"][0]["skipped"] == "fitted behavior did not beat the original"
232+
233+
234+
@pytest.mark.parametrize("binding", ["call", "context"])
235+
@pytest.mark.parametrize("backend", ["jev", "lm"])
236+
def test_runtime_clients_can_calibrate_without_static_cache_checks(jev, binding, backend):
237+
client = jev if backend == "jev" else ComputedLM(generative, adapter=dspy.JSONAdapter())
238+
239+
class Routed(dspy.Module):
240+
def __init__(self):
241+
super().__init__()
242+
self.judge = dspy.Predict(Sig)
243+
244+
def forward(self, pair):
245+
if binding == "call":
246+
return self.judge(pair=pair, lm=client)
247+
with dspy.context(lm=client):
248+
return self.judge(pair=pair)
249+
250+
with dspy.context(lm=None):
251+
with pytest.raises(ValueError, match="require_cache=False"):
252+
ReAnchor(metric).compile(Routed(), trainset=examples())
253+
optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
254+
program = optimizer.compile(Routed(), trainset=examples())
255+
assert program.judge.fields == {"match": {"threshold": 0.8}}
256+
assert program(pair="different").match is False
257+
assert optimizer.report["train_score"] == 0.7778
258+
259+
260+
@pytest.mark.parametrize("original", [None, {}, {"criteria": {"true": {"what": "same item"}}}])
261+
def test_rejected_jev_configuration_is_restored_exactly(original):
262+
student = dspy.Predict(Sig)
263+
if original is not None:
264+
student.fields["match"] = copy.deepcopy(original)
265+
# All True is already perfect; fitting must not leave materialized defaults behind.
266+
train = [dspy.Example(pair="same", match=True).with_inputs("pair") for _ in range(10)]
267+
optimizer = ReAnchor(metric, num_threads=2)
268+
program = optimizer.compile(student, trainset=train)
269+
assert program.fields == student.fields
270+
row = optimizer.report["fitted"][0]
271+
assert row["skipped"] == "fitted behavior did not beat the original"
272+
assert row["train_score_original"] == row["train_score"] == 1.0
273+
assert "value" not in row
226274

227275

228276
def test_a_partial_field_entry_is_filled_from_the_type_defaults(system_one):

‎tests/teleprompt/reanchor/test_reanchor_calibrate.py‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ def test_zero_probability_can_be_classified_on_either_side(system_one, target):
3232
program = dspy.Predict(Match)
3333
train = [dspy.Example(pair=str(i), match=target).with_inputs("pair") for i in range(10)]
3434
report = calibrate(program, train, lambda g, p: float(p.match == g.match), num_threads=2)
35-
assert program.fields["match"]["threshold"] == (0.0 if target else 0.5)
35+
assert program.fields == ({"match": {"threshold": 0.0}} if target else {})
3636
assert report[0]["train_score"] == 1.0
3737

3838

@@ -92,10 +92,22 @@ def test_score_cuts_stay_at_their_defaults_when_the_metric_ignores_the_level(sys
9292
program = dspy.Predict(RateLevel)
9393
train = [dspy.Example(item=k, rating={"lo": 0, "hi": 2}[k]).with_inputs("item") for k in ["lo", "hi"]]
9494
report = calibrate(program, train, lambda g, p, trace=None: 1 - abs(p.rating.value - g.rating) / 2, num_threads=2)
95-
assert program.fields["rating"]["cuts"] == [0.5, 1.5]
95+
assert program.fields == {}
9696
assert report[0]["train_score"] == report[0]["train_score_at_start"]
9797

9898

99+
def test_score_search_does_not_collapse_cuts_at_adjacent_floats(system_one):
100+
# The midpoint between 0.7 and this observed mean rounds back to 0.7.
101+
probability = 0.7000000000000001
102+
system_one(lambda state, name, q: score({0: 1 - probability, 1: probability, 2: 0.0}))
103+
program = dspy.Predict(RateLevel)
104+
program.fields["rating"] = {"cuts": [0.7, 1.5]}
105+
train = [dspy.Example(item=str(i), rating=1).with_inputs("item") for i in range(6)]
106+
report = calibrate(program, train, lambda g, p: float(p.rating.level == g.rating), num_threads=2)
107+
assert program.fields["rating"] == {"cuts": [0.7, 1.5]}
108+
assert report[0]["train_score"] == 1.0
109+
110+
99111
class Route(dspy.Signature):
100112
ticket: str = dspy.InputField()
101113
queue: Annotated[Literal["billing", "tech"], Choice[("billing", "Payments"), ("tech", "Bugs")]] = dspy.OutputField(
@@ -176,8 +188,8 @@ def test_outputs_the_metric_does_not_read_keep_their_numeric_settings(system_one
176188
assert rows["noise"]["train_score"] == rows["noise"]["train_score_at_start"]
177189
assert rows["kind"]["train_score"] == rows["kind"]["train_score_at_start"]
178190
assert "skipped" not in rows["match"] and program.fields["match"]["threshold"] == 0.8
179-
assert program.fields["noise"]["threshold"] == 0.5
180-
assert program.fields["kind"]["weights"] == {"x": 1.0, "y": 1.0}
191+
assert "noise" not in program.fields
192+
assert "kind" not in program.fields
181193
assert len({repr(c) for c in client.calls}) == 2
182194

183195

@@ -337,7 +349,7 @@ def test_a_threshold_stays_when_its_whole_gain_is_one_example(system_one):
337349
system_one(answer)
338350
program = dspy.Predict(Match)
339351
report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
340-
assert program.fields["match"]["threshold"] == 0.5
352+
assert program.fields == {}
341353
assert report[0]["fold_check"] == {"passed": 0, "failed": 1}
342354

343355

@@ -357,7 +369,7 @@ def test_score_cuts_stay_when_their_whole_gain_is_one_example(system_one):
357369
items = [f"lo{i}" for i in range(19)] + ["odd0"] + [f"hi{i}" for i in range(20)]
358370
train = [dspy.Example(item=k, rating=2 if k.startswith("hi") else 0).with_inputs("item") for k in items]
359371
report = calibrate(program, train, lambda g, p, trace=None: float(p.rating.level == g.rating), num_threads=2)
360-
assert program.fields["rating"]["cuts"] == [0.5, 1.5]
372+
assert program.fields == {}
361373
assert report[0]["fold_check"]["failed"] >= 1
362374

363375

@@ -368,5 +380,5 @@ def test_choice_weights_stay_when_their_whole_gain_is_one_example(system_one):
368380
items = [f"x{i}" for i in range(20)] + [f"y{i}" for i in range(19)] + ["odd0"]
369381
train = [dspy.Example(item=k, kind="x" if k.startswith("x") else "y").with_inputs("item") for k in items]
370382
report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)
371-
assert program.fields["kind"]["weights"] == {"x": 1.0, "y": 1.0}
383+
assert program.fields == {}
372384
assert report[0]["fold_check"]["failed"] >= 1

0 commit comments

Comments
 (0)