Skip to content

fix(teleprompt): handle metric_threshold=0.0 in BootstrapFewShot - #10606

Open
udbhav-shrinet wants to merge 2 commits into
stanfordnlp:mainfrom
udbhav-shrinet:fix-bootstrap-metric-zero-threshold
Open

udbhav-shrinet wants to merge 2 commits into
stanfordnlp:mainfrom
udbhav-shrinet:fix-bootstrap-metric-zero-threshold

Conversation

@udbhav-shrinet

Copy link
Copy Markdown

Summary

Fixes #10603

Description

In BootstrapFewShot, setting metric_threshold=0.0 previously evaluated as falsy in if self.metric_threshold:, skipping numeric threshold comparison and treating the metric result as a boolean. This caused:

  1. Valid scores of 0.0 to be rejected (when they should be accepted under >= 0.0).
  2. Negative scores like -0.1 to be accepted (because -0.1 is truthy in Python).

Changes

  • Updated dspy/teleprompt/bootstrap.py to check if self.metric_threshold is not None:.
  • Added unit tests in tests/teleprompt/test_bootstrap.py verifying behavior when metric_threshold=0.0.

@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] No identified blocking issues. The change is safe to merge based on the reviewed code.

Summary

BootstrapFewShot now checks numeric thresholds whenever metric_threshold is not None, including 0.0.

  • Tests cover accepting zero scores and rejecting negative scores at a zero threshold.
  • Added checks cover None and positive thresholds.
  • No actionable issues found. Tests were not run.

Reviews (2) · Last reviewed commit: "test(teleprompt): address review feedbac..." · Reviewed by Greptile

@VANDRANKI VANDRANKI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a community review. It does not clear the merge gate.

I read the diff and ran it. The fix is correct. I would approve, but GitHub does not let me approve without repository access, so this is a comment.

The change: in BootstrapFewShot._bootstrap_one_example (dspy/teleprompt/bootstrap.py), if self.metric_threshold: becomes if self.metric_threshold is not None:. With the old check, an explicit metric_threshold=0.0 was treated as unset, so the metric value was used as a bool. That rejects a score of 0.0 and accepts -0.1 (truthy).

What I ran (Windows, Python 3.13, f1260ff as main, PR head):

  • I copied this PR's test_bootstrap_zero_metric_threshold onto main. It fails on main with assert 0 == 1 on its first assertion (threshold 0.0 with score 0.0 should keep one demo). It passes on the PR head. I did not see main fail on the second assertion (threshold 0.0 with score -0.1), because the test stops at the first failure.
  • tests/teleprompt/test_bootstrap.py on the PR head, together with the tests from the two other open PRs, passed: 35 passed.

Other places: I searched dspy/ for metric_threshold. The other uses (random_search.py, mipro_optimizer_v2.py, teleprompt/utils.py) only pass the value through to BootstrapFewShot. I found no other truthiness check on it, so this one line is the whole fix.

Duplicates: #10604 and #10605 are open with the same one-line source change (the dspy/teleprompt/bootstrap.py hunks are identical in the three diffs). I ran their tests as well. All of them fail on main at the zero-threshold cases (10604: the 0.0, 0.0 and -0.1, 0.0 cases; 10605: its single zero-threshold test) and pass with the fix. #10604 has the broadest coverage: it also covers the None threshold and the nonzero thresholds. #10605 covers the same None and nonzero cases inside one test. Maintainers will probably want to pick one.

Nit on this PR's test: it calls dspy.settings.configure(...), while the other tests in the file use dspy.configure(...). I did not check whether settings.configure is deprecated.

@udbhav-shrinet

Copy link
Copy Markdown
Author

Updated: switched to dspy.configure() to match file conventions and added test coverage for metric_threshold=None and positive thresholds.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] BootstrapFewShot treats metric_threshold=0.0 as unset

2 participants