Skip to content

Commit 0f53eda

Browse files
ampagentIsaac Miller
andcommitted
Clarify ReAnchor user documentation and search guarantees
Amp-Thread-ID: https://ampcode.com/threads/T-01a0bf8d-a1dd-7603-906c-ea06c46ff7f7 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
1 parent fe9f147 commit 0f53eda

1 file changed

Lines changed: 14 additions & 15 deletions

File tree

‎docs/docs/api/experimental/ReAnchor.md‎

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -38,12 +38,11 @@ print(optimizer.report)
3838

3939
## What ReAnchor fits
4040

41-
ReAnchor tries settings for each output and keeps the one with the best mean
42-
metric score on the training examples.
41+
ReAnchor searches each output's numeric settings against the whole-program
42+
metric. A new setting must improve the training score and pass a fold check.
4343

44-
ReAnchor builds the settings to try from the training examples. First it runs
45-
the program once and records the probabilities behind each output on every
46-
call. A threshold anywhere between two neighboring P(True) values makes the
44+
Before fitting each output, ReAnchor runs the program and records that output's
45+
probabilities on every call. A threshold anywhere between two neighboring P(True) values makes the
4746
same decisions, so ReAnchor tries the midpoint of each gap between them. For
4847
example, when the model only returns P(True) of 0.98 and 1.0, ReAnchor tries
4948
0.5 and 0.99.
@@ -58,14 +57,14 @@ example, when the model only returns P(True) of 0.98 and 1.0, ReAnchor tries
5857
would flip on some call.
5958

6059
The current setting stays unless another setting scores strictly better and
61-
passes a fold check. The check splits the training examples into five parts.
62-
For each part, ReAnchor picks a setting using the other four parts and scores
63-
that pick on the held-out part. A new setting is kept only when those held-out
64-
scores beat the current setting. The check discourages gains confined to small
60+
passes a fold check. The check splits the training examples into up to five parts.
61+
For each part, ReAnchor picks a setting using the remaining parts and scores
62+
that pick on the held-out part. A new setting is kept only when the combined held-out
63+
score beats the current setting. The check discourages gains confined to small
6564
portions of the dataset, but can accept them when they recur across folds.
66-
When two settings score the same, ReAnchor picks the one in the
67-
widest gap, because it leaves the most room on either side. When an output returns many distinct
68-
values, ReAnchor thins the list to at most 40 settings, spaced evenly through
65+
Among improving candidates with equal scores, ReAnchor prefers the widest gap,
66+
which leaves the most room on either side. Each search step tries at most
67+
40 gap midpoints, thinning large lists to settings spaced evenly through
6968
the observed values. For Boolean outputs, it also tries threshold zero when
7069
P(True)=0 is observed, since that boundary is the only way to classify those
7170
answers as True.
@@ -96,7 +95,7 @@ The `report` row for a kept entry has `"promoted": True`. The first pass with
9695
probabilities sends new requests, because the request changes.
9796

9897
```python
99-
dspy.configure(lm=dspy.LM("openai/gpt-6-luna"))
98+
dspy.configure(lm=dspy.LM("your-provider/your-model")) # Use your model's identifier.
10099
tuned = ReAnchor(metric).compile(dspy.Predict(Match), trainset=trainset)
101100
```
102101

@@ -121,8 +120,8 @@ your backend's load.
121120
- `val_score_before` and `val_score`, the same scores on `valset` when you pass
122121
one. ReAnchor never fits settings on `valset`.
123122
- `fitted`, one row per output with the fitted value, or the reason ReAnchor
124-
skipped it. Each fitted row has an `observed` entry with the number of calls,
125-
the number of distinct values, and the number of settings tried. Its
123+
skipped it. Each fitted row has an `observed` entry with the number of calls
124+
and settings tried; threshold and cut reports also summarize observed values. Its
126125
`fold_check` entry counts the search steps whose better training score passed
127126
or failed the fold check.
128127

0 commit comments

Comments
 (0)