Skip to content

Commit e83509e

Browse files
ampagentIsaac Miller
andcommitted
Use ordered Score levels without numeric anchors
Amp-Thread-ID: https://ampcode.com/threads/T-01a0bf8d-a1dd-7603-906c-ea06c46ff7f7 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
1 parent b1ac14b commit e83509e

7 files changed

Lines changed: 93 additions & 79 deletions

File tree

‎docs/docs/api/experimental/Decide.md‎

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ from typing import Annotated, Literal
1515
import dspy
1616
from dspy.experimental import Choice, Decide, Noul, Score, TypeSafe
1717

18-
Severity = Score[(0, "Minor"), (2, "Disruptive"), (10, "Blocking")]
18+
Severity = Score["Minor", "Disruptive", "Blocking"]
1919
Category = Choice[("billing", "Payment issue"), ("technical", "Product malfunction")]
2020

2121
class Assess(dspy.Signature):
@@ -42,8 +42,9 @@ and Jev to obtain probability evidence, then decodes it locally.
4242
## Native and rich types
4343

4444
Native outputs return only the value. Bare `float` works with Predict but needs
45-
a Score rubric for Decide. Score anchors must be finite and strictly increasing;
46-
Decide supports 2–10 levels. Choice preserves string, integer, Boolean, and None
45+
a Score rubric for Decide. Declare Score descriptions in increasing level order;
46+
the value ranges from 0 to N−1, and Decide supports 2–10 levels.
47+
Choice preserves string, integer, Boolean, and None
4748
member types; colliding string labels such as `1` and `"1"` are rejected.
4849
Bracket configuration is a runtime convention, not a standard static generic.
4950

@@ -63,8 +64,8 @@ messages. Predict's adapter adds the field markers; Decide nests values under
6364
| --- | --- | --- |
6465
| `bool` | `True` | `true` |
6566
| `Noul` | `{"value": true, "confidence": 0.6}` | Same JSON object |
66-
| `Annotated[float, Severity]` | `6.6` | `6.6` |
67-
| `Severity` | `{"value": 6.6, "confidence": 0.61}` | Same JSON object |
67+
| `Annotated[float, Severity]` | `1.5` | `1.5` |
68+
| `Severity` | `{"value": 1.5, "confidence": 0.61}` | Same JSON object |
6869
| `Literal["billing", "technical"]` | `technical` | `"technical"` |
6970
| `Category` | `{"value": "technical", "confidence": 0.73}` | Same JSON object |
7071

@@ -75,13 +76,13 @@ present. Use `.value` explicitly when passing a rich result to a native input.
7576

7677
Predict uses the annotation's schema to parse the generated value. Decide maps
7778
native and rich forms to the same Jev primitive, then returns either the decoded
78-
native value or a rich object. Numeric anchors stay local; descriptions go to Jev.
79+
native value or a rich object. Ordered Score descriptions go to Jev as criteria.
7980

8081
| Annotation | Predict requests → returns | Decide question → returns |
8182
| --- | --- | --- |
8283
| `bool` | Boolean → `bool` | `type: "noul"` → thresholded `bool` |
8384
| `Noul` | JSON `{value: bool, confidence: number}` → `Noul` | Same noul question → `Noul` with derived value/confidence and raw probability |
84-
| `Annotated[float, Severity]` | Number in `[0, 10]`, with rubric → `float` | `type: "score"`, criteria `["Minor", "Disruptive", "Blocking"]` → expected anchor value as `float` |
85+
| `Annotated[float, Severity]` | Number in `[0, 2]`, with rubric → `float` | `type: "score"`, criteria `["Minor", "Disruptive", "Blocking"]` → expected level index as `float` |
8586
| `Severity` | JSON `{value: number, confidence: number}`, same range/rubric → `Severity` | Same score question → `Severity` with value, provider confidence, probabilities, and cut-selected level |
8687
| `Literal["billing", "technical"]` | One allowed member → native member | `type: "choice"`, criteria `{"billing": null, "technical": null}` → native member |
8788
| `Category` | JSON `{value: allowed member, confidence: number}`, with option descriptions → `Category` | Choice criteria `{"billing": "Payment issue", "technical": "Product malfunction"}` → `Category` with value, provider confidence, probabilities |
@@ -116,10 +117,11 @@ inference or save/load. No Jev-specific `OutputField` arguments are added.
116117
| Score `cuts` | `[0.5, 1.5, …]` | Select `.level` from mean level index; N−1 increasing boundaries inside `(0, N−1)` |
117118
| Choice `weights` | All `1.0` | Nonnegative, finite probability multipliers keyed by string labels; omitted options default to `1.0` |
118119

119-
**Score has no tunable weights.** Its continuous value is
120-
`sum(p[i] * anchors[i]) / sum(p)`. Its zero-based `.level` counts the cuts less than
121-
or equal to `sum(i * p[i]) / sum(p)`. Equality selects the higher level; cuts do
120+
**Score has no anchors or weights.** Its continuous `.value` is
121+
`sum(i * p[i]) / sum(p)`, the mean level index. Its zero-based `.level` counts the
122+
cuts less than or equal to that value. Equality selects the higher level; cuts do
122123
not change `.value`. Native float outputs expose only the continuous value.
124+
For probabilities `[0.1, 0.3, 0.6]`, `.value` is `1.5`; cuts `[0.5, 1.6]` select `.level = 1`.
123125

124126
Choice selects by `probability[label] * weight[label]`; raw probabilities are
125127
unchanged. All-unit weights retain the provider choice. Weighted ties prefer the
@@ -139,12 +141,12 @@ evidence without changing shared types or previous results.
139141
| Validated runtime inputs | JSON under `state.inputs`; explicit paths use `inputs.ticket` |
140142
| Each output name and type | `questions[name]` with `type: "noul"`, `"score"`, or `"choice"` |
141143
| Field description / module override | `questions[name].instructions` |
142-
| Type rubric/options / module override | `questions[name].criteria`; numeric anchors remain local |
144+
| Type rubric/options / module override | `questions[name].criteria` |
143145

144146
Overrides apply only to Decide; Predict continues to render the signature and type
145147
descriptions. Per-call `signature=` may change instructions/descriptions or switch
146148
between equivalent native/rich types, but must preserve output names and answer
147-
spaces. Use a new Decide for different options or anchors.
149+
spaces. Use a new Decide for different options or ordered levels.
148150

149151
## Confidence depends on its source
150152

@@ -179,7 +181,7 @@ restored.load("assess.json")
179181
| `client` | Explicit TypeSafe model, endpoint, cache setting, timeout; otherwise null |
180182
| `metadata` | DSPy's dependency versions |
181183

182-
State-only JSON excludes signature architecture/types/anchors, inputs, results,
184+
State-only JSON excludes signature architecture/types/declared levels, inputs, results,
183185
history, and API keys. Credentials come from the environment. Loading invalid
184186
configuration leaves the module unchanged; earlier unreleased PR formats are not
185187
migrated. Saved endpoints require `allow_unsafe_lm_state=True` for trusted files.

‎dspy/adapters/types/decision.py‎

Lines changed: 12 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
11
"""Decision values shared by System One predictors and generative LMs."""
22

3-
import itertools
43
import json
5-
import math
64
from functools import lru_cache
75
from typing import Annotated, ClassVar, Literal, get_args, get_origin
86

@@ -53,9 +51,9 @@ def __bool__(self):
5351
class Score(_Decision):
5452
"""A continuous score with a declared rubric and confidence.
5553
56-
Declare ``Score[(0, "poor"), (2, "fair"), (10, "excellent")]``. Anchors must
57-
be finite and strictly increasing. Predict generates value and confidence;
58-
Decide averages the fixed declared anchors using provider probabilities.
54+
Declare ``Score["poor", "fair", "excellent"]`` in increasing order.
55+
Predict generates a continuous value from 0 to N-1 and confidence;
56+
Decide averages the level indices using provider probabilities.
5957
``probabilities`` retains the raw distribution, keyed by rubric index.
6058
``level`` is the zero-based ordinal selected by Decide's cuts on the mean
6159
level index. It does not change the continuous value or provider confidence.
@@ -72,27 +70,24 @@ class Score(_Decision):
7270

7371
@classmethod
7472
def __class_getitem__(cls, options):
75-
options = _option_pairs(options)
76-
anchors = [value for value, _ in options]
77-
if (
78-
len(anchors) < 2
79-
or any(type(v) not in (int, float) or not math.isfinite(v) for v in anchors)
80-
or any(a >= b for a, b in itertools.pairwise(anchors))
81-
):
82-
raise ValueError("Score requires at least two finite, strictly increasing numeric anchors.")
73+
if not isinstance(options, tuple) or len(options) < 2 or any(not isinstance(v, str) for v in options):
74+
raise ValueError("Score requires at least two ordered level descriptions, e.g. Score['poor', 'excellent'].")
8375
return _score_type(options)
8476

8577
@classmethod
8678
def description(cls):
87-
return "Continuous score; intermediate values are allowed. Rubric: " + json.dumps(cls.options) + "."
79+
return (
80+
f"Continuous score from 0 to {len(cls.options) - 1}; intermediate values are allowed. "
81+
"Rubric (level index, description): " + json.dumps(list(enumerate(cls.options))) + "."
82+
)
8883

8984
@classmethod
9085
def __get_pydantic_core_schema__(cls, source, handler):
9186
if source is float:
9287
# Pydantic calls this hook when this configured type is Annotated metadata.
9388
if not cls.options:
9489
raise ValueError("A native Score requires a configured rubric.")
95-
return core_schema.float_schema(ge=cls.options[0][0], le=cls.options[-1][0], allow_inf_nan=False)
90+
return core_schema.float_schema(ge=0, le=len(cls.options) - 1, allow_inf_nan=False)
9691
return handler(source)
9792

9893
@classmethod
@@ -152,9 +147,9 @@ def _option_pairs(options):
152147
@lru_cache(maxsize=256)
153148
def _score_type(options):
154149
result = create_model(
155-
f"Score[{', '.join(repr(pair) for pair in options)}]",
150+
f"Score[{', '.join(repr(description) for description in options)}]",
156151
__base__=Score,
157-
value=(float, Field(ge=options[0][0], le=options[-1][0], allow_inf_nan=False)),
152+
value=(float, Field(ge=0, le=len(options) - 1, allow_inf_nan=False)),
158153
)
159154
result.options = options
160155
return result

‎dspy/predict/decide.py‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,14 @@
2424
class Decide(Module, Parameter):
2525
"""Answer a signature's closed-set outputs in one System One request.
2626
27-
Declare outputs using Noul, Score[(number, description), ...], and
27+
Declare outputs using Noul, Score["low", "medium", "high"], and
2828
Choice[(value, description), ...]. Native bool, Literal, and
2929
Annotated[float, Score[...]] return only the value. Bare float has no rubric.
3030
Unsupported outputs raise; Decide never falls back to a generative LM.
3131
3232
Each instance owns ``fields[name]`` configuration: a Boolean ``threshold``
3333
(initially 0.5), or ``cuts`` for Score level selection, halfway between level
34-
indices. Score values use the fixed declared anchors; cuts affect only .level.
34+
indices. Score values average those indices; cuts affect only .level.
3535
Choice weights are nonnegative probability multipliers keyed by string
3636
option labels, initially 1.0. Missing option weights also default to 1.0.
3737
Weighted Choice selection preserves raw probabilities and provider confidence;
@@ -165,7 +165,7 @@ def _questions(self, signature, types):
165165
if kind is Noul:
166166
question["type"] = "noul"
167167
elif issubclass(kind, Score):
168-
question.update(type="score", criteria=[desc for _, desc in kind.options])
168+
question.update(type="score", criteria=list(kind.options))
169169
else:
170170
question.update(type="choice", criteria={str(v): desc or None for v, desc in kind.options})
171171
if "criteria" in config:
@@ -228,16 +228,14 @@ def _decode(self, answers, signature, types, inputs, trace):
228228
)
229229
elif issubclass(kind, Score):
230230
probabilities = answer["probabilities"]
231-
anchors = [v for v, _ in kind.options]
232-
if set(probabilities) != set(range(len(anchors))) or sum(probabilities.values()) <= 0:
231+
if set(probabilities) != set(range(len(kind.options))) or sum(probabilities.values()) <= 0:
233232
raise ValueError(f"Invalid Score distribution for {name!r}.")
234-
value = sum(anchors[i] * p for i, p in probabilities.items()) / sum(probabilities.values())
235-
position = sum(i * p for i, p in probabilities.items()) / sum(probabilities.values())
233+
value = sum(i * p for i, p in probabilities.items()) / sum(probabilities.values())
236234
result = kind(
237235
value=value,
238236
confidence=answer["confidence"],
239237
probabilities=probabilities,
240-
level=sum(position >= cut for cut in self.fields[name]["cuts"]),
238+
level=sum(value >= cut for cut in self.fields[name]["cuts"]),
241239
)
242240
else:
243241
options = {str(v): v for v, _ in kind.options}

‎tests/adapters/test_decision_types.py‎

Lines changed: 31 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
from dspy.utils.dummies import DummyLM
1111
from tests.predict.test_decide import FakeClient
1212

13-
Severity = Score[(0, "Minor"), (2, "Disruptive"), (10, "Blocking")]
13+
Severity = Score["Minor", "Disruptive", "Blocking"]
1414
Category = Choice[("billing", "Payment issue"), ("technical", "Product malfunction")]
1515

1616

@@ -45,11 +45,11 @@ def decision_signature(rich, direction):
4545

4646
def decision_values(rich, evidence=False):
4747
if not rich:
48-
return {"urgent": True, "severity": 6.6, "category": "technical"}
48+
return {"urgent": True, "severity": 1.5, "category": "technical"}
4949
return {
5050
"urgent": Noul(value=True, confidence=0.6, **({"probability": 0.8} if evidence else {})),
5151
"severity": Severity(
52-
value=6.6, confidence=0.61, **({"probabilities": {0: 0.1, 1: 0.3, 2: 0.6}} if evidence else {})
52+
value=1.5, confidence=0.61, **({"probabilities": {0: 0.1, 1: 0.3, 2: 0.6}} if evidence else {})
5353
),
5454
"category": Category(
5555
value="technical",
@@ -83,7 +83,7 @@ def test_predict_outputs_and_generated_schema(adapter, rich):
8383
else:
8484
assert "confidence" not in json.dumps(schema)
8585
assert schema["properties"]["severity"]["minimum"] == 0
86-
assert schema["properties"]["severity"]["maximum"] == 10
86+
assert schema["properties"]["severity"]["maximum"] == 2
8787
assert schema["properties"]["category"]["enum"] == ["billing", "technical"]
8888

8989

@@ -102,7 +102,7 @@ def test_inputs_to_both_modules_preserve_values_and_context(rich, evidence):
102102
prompt = messages[-1]["content"]
103103
assert ('"confidence"' in prompt) is rich
104104
assert ('"probabilities"' in prompt) is (rich and evidence)
105-
assert "6.6" in prompt
105+
assert "1.5" in prompt
106106
assert "technical" in prompt
107107
client = FakeClient()
108108
module = Decide(signature, client=client)
@@ -117,10 +117,11 @@ def test_inputs_to_both_modules_preserve_values_and_context(rich, evidence):
117117

118118
@pytest.mark.parametrize("rich", [False, True])
119119
@pytest.mark.parametrize("adapter", [dspy.ChatAdapter(), dspy.JSONAdapter()])
120-
def test_score_range_is_enforced_for_native_and_rich(adapter, rich):
120+
@pytest.mark.parametrize("invalid", [-0.1, 2.1])
121+
def test_score_range_is_enforced_for_native_and_rich(adapter, rich, invalid):
121122
sig = decision_signature(rich, "output")
122123
values = decision_values(False)
123-
values["severity"] = 11
124+
values["severity"] = invalid
124125
if rich:
125126
values = {name: {"value": value, "confidence": 0.7} for name, value in values.items()}
126127
completion = adapter.format_assistant_message_content(sig, values)
@@ -131,10 +132,10 @@ def test_score_range_is_enforced_for_native_and_rich(adapter, rich):
131132
@pytest.mark.parametrize(
132133
"kind,options",
133134
[
134-
(Score, ((2, "a"), (1, "b"))),
135-
(Score, ((1, "a"), (1, "b"))),
136-
(Score, ((0, "a"), (float("nan"), "b"))),
137-
(Score, ((0, "a"),)),
135+
(Score, ((0, "a"), (1, "b"))), # Removed anchor syntax.
136+
(Score, ("a", 1)),
137+
(Score, ("a",)),
138+
(Score, "a"),
138139
(Choice, ((1, "a"), ("1", "b"))),
139140
(Choice, (([], "bad"),)),
140141
],
@@ -144,6 +145,24 @@ def test_invalid_options(kind, options):
144145
kind[options]
145146

146147

148+
@pytest.mark.parametrize("levels,maximum", [(("Low", "High"), 1), (("Last", "Second", "Third", "First"), 3)])
149+
def test_score_level_order_bounds_and_fractional_values(levels, maximum):
150+
score = Score[levels]
151+
assert score.options == levels
152+
assert json.dumps(list(enumerate(levels))) in score.description()
153+
for rich in (False, True):
154+
adapter = TypeAdapter(score if rich else Annotated[float, score])
155+
schema = adapter.json_schema()
156+
value_schema = schema["properties"]["value"] if rich else schema
157+
assert value_schema["minimum"] == 0
158+
assert value_schema["maximum"] == maximum
159+
for value in (0, 0.25, maximum):
160+
result = adapter.validate_python({"value": value, "confidence": 0.7} if rich else value)
161+
assert (result.value if rich else result) == value
162+
with pytest.raises(ValidationError):
163+
adapter.validate_python({"value": maximum + 0.1, "confidence": 0.7} if rich else maximum + 0.1)
164+
165+
147166
def test_choice_type_cache_preserves_bool_vs_int():
148167
boolean = Choice[(True, "yes"), (False, "no")]
149168
integer = Choice[(1, "yes"), (0, "no")]
@@ -152,7 +171,7 @@ def test_choice_type_cache_preserves_bool_vs_int():
152171
assert type(boolean(value=True, confidence=0.5).value) is bool
153172

154173

155-
@pytest.mark.parametrize("kind,value", [(Noul, True), (Severity, 4.2), (Category, "technical")])
174+
@pytest.mark.parametrize("kind,value", [(Noul, True), (Severity, 1.2), (Category, "technical")])
156175
def test_rich_confidence_required_and_serialization(kind, value):
157176
with pytest.raises(ValidationError):
158177
kind(value=value)

‎tests/clients/test_typesafe.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ def test_real_sdk_request_cache_usage_and_local_parameters(transport):
6262
assert second.flag.value is False
6363
assert first.flag.confidence == pytest.approx(0.6)
6464
assert second.flag.confidence == pytest.approx(1 / 9)
65-
assert first.rating.value == pytest.approx(6.7)
65+
assert first.rating.value == pytest.approx(1.5)
6666
assert second.rating.value == first.rating.value
6767
assert first.rating.level == 2
6868
assert second.rating.level == 1
@@ -99,7 +99,7 @@ async def test_async_sdk_and_shared_cache(transport):
9999
module.fields["label"]["weights"] = {"2": 0.1}
100100
assert (await module.acall(text="x")).label == "other"
101101
assert len(transport) == 1
102-
assert synchronous.rating == pytest.approx(6.7)
102+
assert synchronous.rating == pytest.approx(1.5)
103103

104104

105105
def test_cache_identity_and_controls(transport):

0 commit comments

Comments
 (0)