Repository navigation
Commit 36abcfb
fit: honest, monotonic parity gate + OOD safety gate (#50)
* Gate deployment on a held-out lower bound, not an in-sample point estimate
The parity gate previously selected the accept threshold and reported its
teacher agreement on the same calibration set, and accepted accept-all
configurations on a raw point estimate. Both let a small or lucky set clear the
target and then break the contract on real traffic.
- build_global now certifies only when the Clopper-Pearson lower bound on cal
agreement clears the target (was a raw point estimate).
- _calibrate_threshold selects the threshold on a selection split, then verifies
it on a held-out split, deploying only if the held-out lower bound clears the
target. Falls back to a single-set lower-bound gate when data is too small to
split, and refuses rather than guessing.
- Guard NaN/inf surrogate probabilities before the acceptor fit (fixes a crash
on degenerate datasets).
- Expose `alpha` (confidence level) through the builders and fit_frontier.
- Add regression tests for low-n refusal, the lower-bound certification, and the
NaN guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update test_fit_and_route for honest gate behaviour
With the held-out lower-bound gate, a noisy 300-trace set correctly refuses to
certify a deployment (teacher noise caps agreement below target). Give this test
cleanly separable data so it still exercises the deploy and route path; the
refusal case is covered by test_gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add OOD-robustness test for the routing path
Extreme out-of-distribution embeddings must route without crashing and produce
finite accept scores. Documents that the OSS library defers via the acceptor
threshold and does not ship a separate distance-based OOD gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(gate): make held-out parity gate monotonic in target
The held-out calibration picked the single highest-coverage threshold that
cleared target on the selection split, then returned None if that one threshold
failed held-out verification. Because the highest-coverage threshold is also the
most aggressive (least likely to generalise), coverage came out non-monotonic in
target: banking77 deployed 0% at TA=0.90 but 87% at TA=0.95.
Now verify selection-passing thresholds in coverage order and deploy the
highest-coverage one that also clears target on the held-out split. Same
Clopper-Pearson maths, same held-out split, the gate just no longer quits after
one failed candidate. banking77 frontier is now monotonic (97.5% @0.90 -> 87.1%
@0.95 -> 65.9% @0.97 -> 50.4% @0.98).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fit(gate): recover coverage at strict targets via hybrid select+verify
The 50/50 held-out split was provably honest but too data-hungry: on a few-hundred
calibration rows it certified 0% at strict targets even though the confidence
ranking clearly supported partial coverage (banking77 0% @0.98 despite an oracle
~80%). Replace it with a hybrid: select the highest-coverage threshold whose
Clopper-Pearson LOWER bound clears target on a 70% slice (removes in-sample
optimism at selection), then require it to also clear on a held-out 30% by the
point estimate (catches selection-bias flukes without demanding CP-tightness on
the small verify slice). Falls back to a single-set CP lower-bound gate when
n<40. Validated across seeds on banking77/obside at 0.90-0.98 with zero held-out
contract violations; recovers banking77 to 67% @0.98 (was 0%) and Ridge to 68%
@0.90 (was 0%).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ood): distance-based OOD safety gate (defer off-distribution inputs)
The parity gate only guarantees agreement on traffic that resembles calibration
data; off-distribution inputs (gibberish, off-domain, prompt-injection) still got
a confident surrogate answer and slipped through. Add a commodity kNN-distance
OOD gate: at fit, calibrate the 99.5th-percentile mean-10-NN distance globally and
per predicted label (per-label can only loosen, never tighten below global); at
inference, defer any input beyond that, regardless of surrogate confidence.
Keyed on input embeddings + predicted label only, NOT the partition cells, so it
carries none of the cell-construction IP. On the Obside 156-case battery it lifts
OOD-junk deferral from 0% to ~80% while keeping in-distribution coverage (clean
100% / holdout 90% handled). Semantic near-misses (entity-disentanglement) are
deliberately out of scope here, those need the advanced embeddings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: neutralize dataset names in gate docstring
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 246973e commit 36abcfb
7 files changed
Lines changed: 415 additions & 36 deletions
File tree
- src/tracer
- fit
- runtime
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
152 | 167 | | |
153 | 168 | | |
154 | 169 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
0 commit comments