Fit Jev decision settings (thresholds, cuts, weights) on recorded answers, after DSPy's ReAnchor #9659

Activity

1. AtlantisPleb

   added 2 commits that reference this issue on Sep 25, 2026

   docs: port DSPy's Jev ideas to Rust instead of a Python bridge

   ...

   47eb72e

   feat: fit Jev decision settings on recorded answers

   ...

   e7c9a1f
2. AtlantisPleb commented on Sep 25, 2026

   

   AtlantisPleb

   on Sep 25, 2026

   ContributorAuthor

   More actions

   Landed in e7c9a1f.

   What was built

   • coder-one study run decision-fit (crates/coder-one/src/study/decisions.rs), with the fit in crates/coder-one/src/study/reanchor.rs. Candidates are midpoints between neighboring observed probabilities, at most 40 per setting. Choice weights are searched on a log scale between the points where a pick flips. A candidate is accepted only when it's strictly better on the fit partition and passes a deterministic 5-fold held-out check, with folds by SHA-256 of the group name. Ties go to the widest gap. Recorded answers only; no Jev calls.
   • Sources: --component control.stall, --component checks.verdict, or --questions FILE --examples FILE for any question set.
   • Output uses Store Jev decision settings in question-set files, digested apart from the wording #9660's format: decisions.json holds the moved settings as decision blocks, plus the digest policy.jev.decision would record. A question-set run also writes a copy of the set with each block joined. Proposals only; no policy changed.
   • The fold rule stays in crates/coder-one, not crates/gym, because Coder One doesn't depend on the Gym.

   Real run, offline (write-up, protocol). Both splits were frozen by earlier studies; the primary metric is each component's own threshold rule.

   • Stall detector (Microluna: typed next-step choice and stall and done detection (pivot algorithms 4 and 5) #9627), recall at 80% precision. stall.progress moved from 0.5 to 0.97, and stall.repeating stayed at 0.5. On the evaluation tasks, recall went from 30 of 85 (35%, 26–46%) to 45 of 85 (53%, 42–63%), and precision from 86% (71–94%) to 83% (71–91%). The metric's difference is +0.18, with an interval of −0.61 to +0.33, so it doesn't beat the hard-coded setting. The fitted detector makes exactly the code-suspect-only calls the stall write-up recommended.
   • Truthful checks (Luna pivot: truthful checks, calibrated against graded runs #9584), recall at 90% precision. The best candidate was better on the fit partition but failed the held-out check (0.387 to −0.161), so checks.verdict.admission stays at 0.8.
   • Secondary F1 runs, for reading only: both settings move, and both evaluation intervals exclude zero (stall +0.015 to +0.243; verdict +0.044 to +0.222). Both trade precision for recall, which neither protocol chose.

   Tests: synthetic recorded answers with a known right threshold, a coin-flip case where a training-only winner fails the held-out check and the original value stays, and cases for ties, weights, cuts, folds, and metrics. At the settings in effect, the stall rule reproduces the protocol's 88 calls with 80 right, and the verdict rule matches corroborated on all 317 rows. cargo test -p coder-one: 881 passed. The 2 failures are the two tests that need make, which this host lacks. clippy is clean, and the contamination check is clean.
3. AtlantisPleb

   closed this as completedon Sep 25, 2026