Repository navigation
fix(teleprompt): handle metric_threshold=0.0 in BootstrapFewShot - #10606
udbhav-shrinet wants to merge 2 commits into
Conversation
|
VANDRANKI
left a comment
There was a problem hiding this comment.
This is a community review. It does not clear the merge gate.
I read the diff and ran it. The fix is correct. I would approve, but GitHub does not let me approve without repository access, so this is a comment.
The change: in BootstrapFewShot._bootstrap_one_example (dspy/teleprompt/bootstrap.py), if self.metric_threshold: becomes if self.metric_threshold is not None:. With the old check, an explicit metric_threshold=0.0 was treated as unset, so the metric value was used as a bool. That rejects a score of 0.0 and accepts -0.1 (truthy).
What I ran (Windows, Python 3.13, f1260ff as main, PR head):
- I copied this PR's
test_bootstrap_zero_metric_thresholdonto main. It fails on main withassert 0 == 1on its first assertion (threshold 0.0 with score 0.0 should keep one demo). It passes on the PR head. I did not see main fail on the second assertion (threshold 0.0 with score -0.1), because the test stops at the first failure. tests/teleprompt/test_bootstrap.pyon the PR head, together with the tests from the two other open PRs, passed: 35 passed.
Other places: I searched dspy/ for metric_threshold. The other uses (random_search.py, mipro_optimizer_v2.py, teleprompt/utils.py) only pass the value through to BootstrapFewShot. I found no other truthiness check on it, so this one line is the whole fix.
Duplicates: #10604 and #10605 are open with the same one-line source change (the dspy/teleprompt/bootstrap.py hunks are identical in the three diffs). I ran their tests as well. All of them fail on main at the zero-threshold cases (10604: the 0.0, 0.0 and -0.1, 0.0 cases; 10605: its single zero-threshold test) and pass with the fix. #10604 has the broadest coverage: it also covers the None threshold and the nonzero thresholds. #10605 covers the same None and nonzero cases inside one test. Maintainers will probably want to pick one.
Nit on this PR's test: it calls dspy.settings.configure(...), while the other tests in the file use dspy.configure(...). I did not check whether settings.configure is deprecated.
…anded threshold coverage
|
Updated: switched to |
Summary
Fixes #10603
Description
In
BootstrapFewShot, settingmetric_threshold=0.0previously evaluated as falsy inif self.metric_threshold:, skipping numeric threshold comparison and treating the metric result as a boolean. This caused:0.0to be rejected (when they should be accepted under>= 0.0).-0.1to be accepted (because-0.1is truthy in Python).Changes
dspy/teleprompt/bootstrap.pyto checkif self.metric_threshold is not None:.tests/teleprompt/test_bootstrap.pyverifying behavior whenmetric_threshold=0.0.