You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1fe299a
Browse filesBrowse the repository at this point in the historyBrowse files
- Report rebrand to the Tracer light theme (Manrope, dots logo, brand-colour
donut, light Sankey); drop 'audit' wording across report + docs.
- Tree surrogates (dt/rf/et/gbt) are now OFF by default in 'tracer fit'; add
'--trees' to opt them in and '--skip' to drop named candidates. The lean
linear+MLP zoo keeps fits fast; trees help hard high-class tasks (Banking77).
- Trace loader accepts the same key aliases as the scanner (input/query/text/
prompt/question and teacher/teacher_output/label/intent/output/answer), so
middleware dumps load without reshaping.
- Docs: document --trees/--skip + aliases, note the Banking77 benchmark uses
--trees, fix a stale tracer.embed example.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The surrogate is **not another LLM** - it is a classical ML or shallow DL model (the model zoo includes logistic regression, SGD, LightGBM, random forests, and small feed-forward nets). This is what makes the cost reduction real: inference is CPU-bound, sub-millisecond, and free.
98
+
The surrogate is **not another LLM** - it is a classical ML or shallow DL model. By default the zoo is lightweight and fast (logistic regression, SGD, and small feed-forward nets); the tree-based models (decision tree, random forest, extra-trees, gradient boosting) are heavier and opt-in with `tracer fit --trees`. This is what makes the cost reduction real: inference is CPU-bound, sub-millisecond, and free.
99
99
100
100
1.**Fit** - train a suite of candidate surrogates on your LLM's classification traces; select the best via cross-validated teacher agreement
101
101
2.**Gate** - attach a learned acceptor that estimates, per-input, whether the surrogate will agree with the teacher
@@ -111,6 +111,8 @@ The surrogate is **not another LLM** - it is a classical ML or shallow DL model
_Banking77 is a 77-class task; the tree models help here, so these numbers are with `tracer fit --trees`. The lightweight default (linear + MLP) is faster and enough for most tasks._
115
+
114
116
## Continual learning flywheel
115
117
116
118
TRACER is not a one-shot fit. Every deferred input that reaches the LLM produces a new labeled trace, which feeds back into the next refit. As the surrogate sees more of the input distribution, its coverage grows - meaning fewer LLM calls, which in turn cost less, while the quality guarantee holds at every iteration.
|`--trees`| off | Add the tree-based surrogates (decision tree, random forest, extra-trees, gradient boosting). **Off by default** — they are slower and the linear + MLP heads are usually enough; turn them on for hard, high-class-count tasks. |
71
+
|`--skip`| none | Comma-separated surrogate models to drop from the zoo (e.g. `mlp_1h,mlp_2h`). |
72
+
73
+
The default zoo is lightweight (logistic regression, SGD, small MLPs) so a fit is fast. `--trees` adds the heavier tree models, which can lift coverage on difficult datasets (e.g. Banking77's 77 classes) at the cost of fit time.
74
+
75
+
Trace files accept common key aliases: the input can be any of `input, query, text, prompt, question` and the label any of `teacher, teacher_output, label, intent, output, answer`.
70
76
71
77
**Examples:**
72
78
73
79
```bash
74
-
# Basic fit with 90% parity target
80
+
# Basic fit with 90% parity target (lightweight zoo)
75
81
tracer fit traces.jsonl
76
82
83
+
# Add tree models for a hard, high-class-count task
84
+
tracer fit traces.jsonl --trees
85
+
77
86
# Stricter target, custom artifact dir
78
87
tracer fit traces.jsonl --target 0.95 --artifact-dir my-policy
Copy file name to clipboardExpand all lines: docs/concepts.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -134,7 +134,7 @@ Coverage grows monotonically as more traces accumulate. Easy intents (tight embe
134
134
135
135
---
136
136
137
-
## Qualitative audit
137
+
## Qualitative report
138
138
139
139
Every fit produces a structured explanation of the routing policy. The slice and boundary-pair design is informed by Slice Finder (Chung et al., [ICDE 2019](https://ieeexplore.ieee.org/document/8731353)) and the contrastive explanation literature (Miller, [AIJ 2019](https://www.sciencedirect.com/science/article/pii/S0004370217301126)).
0 commit comments