You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8c6f3d6
Browse filesBrowse the repository at this point in the historyBrowse files
The surrogate is **not another LLM** - it is a classical ML or shallow DL model (the model zoo includes logistic regression, SGD, gradient-boosted trees, random forests, and small feed-forward nets). This is what makes the cost reduction real: inference is CPU-bound, sub-millisecond, and free.
75
+
The surrogate is **not another LLM** - it is a classical ML or shallow DL model (the model zoo includes logistic regression, SGD, LightGBM, random forests, and small feed-forward nets). This is what makes the cost reduction real: inference is CPU-bound, sub-millisecond, and free.
75
76
76
77
1.**Fit** - train a suite of candidate surrogates on your LLM's classification traces; select the best via cross-validated teacher agreement
77
78
2.**Gate** - attach a learned acceptor that estimates, per-input, whether the surrogate will agree with the teacher
78
79
3.**Calibrate** - sweep the acceptor threshold to maximise coverage at your target parity (e.g. ≥ 95% teacher agreement)
79
80
4.**Guard** - block deployment if the best candidate cannot clear the parity bar on held-out data
80
81
81
-
## Explainability
82
-
83
-
Every fit produces a structured audit report alongside the routing policy -- not just aggregate numbers.
84
-
85
-
| Report component | What it answers |
86
-
|-----------------|-----------------|
87
-
|**Per-label slice stats**| Which intents are fully self-serve? Which always defer? |
88
-
|**Contrastive boundary pairs**| For each label: one handled example, one deferred -- what makes an input "easy" vs "hard"? |
89
-
|**Representative examples**| What does typical handled/deferred traffic look like? |
90
-
|**Temporal deltas**| Which intents improved between the last refit and this one? |
|[AGENTS.md](AGENTS.md)| Integration guide for AI coding assistants |
189
165
190
-
## Prior work
191
-
192
-
TRACER builds on and connects three research threads:
193
-
194
-
**Learn to Defer (L2D)** -- the algorithmic backbone. The surrogate + acceptor gate architecture is grounded in Mozannar & Sontag ([ICML 2020](https://proceedings.mlr.press/v119/mozannar20b.html)), Madras et al. ([NeurIPS 2018](https://papers.nips.cc/paper_files/paper/2018/hash/09d37c08f7b129e96277388757530c72-Abstract.html)), and Mao et al. ([NeurIPS 2023](https://proceedings.neurips.cc/paper_files/paper/2023/hash/7e1bf09d2e4e6e5c5a0b63b66b47ef42-Abstract.html)).
195
-
196
-
**Post-hoc calibration** -- the threshold sweep that enforces the parity guarantee follows the post-hoc deferral estimator framework of Narasimhan et al. ([ICML 2022](https://proceedings.mlr.press/v162/narasimhan22a.html)).
197
-
198
-
**LLM routing** -- the problem setting is shared with Hybrid LLM (Ding et al., [ICLR 2024](https://openreview.net/forum?id=02f3mUtqnM)) and RouteLLM (Ong et al., [ICLR 2025](https://openreview.net/forum?id=sSt9fROCn2)). TRACER's distinction: it targets classification specifically, uses the teacher's own trace distribution rather than preference data, and enforces a formal parity guarantee rather than a quality heuristic.
199
-
200
-
**Interpretability** -- the qualitative audit design is informed by Slice Finder (Chung et al., [ICDE 2019](https://ieeexplore.ieee.org/document/8731353)) and the contrastive explanation literature (Miller, [AIJ 2019](https://www.sciencedirect.com/science/article/pii/S0004370217301126)).
201
-
202
166
## Paper
203
167
204
168
A research paper detailing the approach, formal guarantees, ablation studies, limitations, and reproducible experiment tooling is in preparation. It will be linked here upon publication.
0 commit comments