Skip to content

Commit 8c6f3d6

Browse files
committed
feat: Sankey routing diagram + embed in HTML report, tune demo to 92% TA
- Add tracer/analysis/sankey.py: generate_sankey() (standalone file, light theme), generate_sankey_div() (dark theme, embeds in HTML report), and shared _build_sankey_figure() builder - Embed Sankey as the first content section in the HTML audit report (right after traffic split stats); Plotly loaded via CDN; sankey.html also auto-generated alongside every report.html - Add tracer sankey CLI command (--format html/png/svg, --top-k, --no-open) - Add FitResult.get_sankey() method - Add tracer.generate_sankey() top-level API export - Add [viz] optional dependency (plotly>=5.0, kaleido>=0.2) - Tune Banking77 demo to target_teacher_agreement=0.92 -> ~91% coverage - Add docs/assets/sankey.png reference visual - Update README, docs/api.md, docs/cli.md with Sankey docs and new demo numbers Made-with: Cursor
1 parent 66f08ac commit 8c6f3d6

11 files changed

Lines changed: 537 additions & 159 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
**Trace-Based Adaptive Cost-Efficient Routing**
44

55
[![PyPI](https://img.shields.io/pypi/v/tracer-llm)](https://pypi.org/project/tracer-llm/)
6+
[![Downloads](https://static.pepy.tech/badge/tracer-llm/month)](https://pepy.tech/project/tracer-llm)
67
[![Python](https://img.shields.io/pypi/pyversions/tracer-llm)](https://pypi.org/project/tracer-llm/)
78
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
89
[![CI](https://img.shields.io/badge/CI-passing-brightgreen)](https://github.com/adrida/tracer/actions)
@@ -27,12 +28,12 @@ tracer demo
2728
2829
Routing Policy
2930
method l2d
30-
coverage 80.7% of traffic handled by surrogate
31-
teacher TA 0.951 surrogate matches teacher on handled traffic
31+
coverage 91.4% of traffic handled by surrogate
32+
teacher TA 0.920 surrogate matches teacher on handled traffic
3233
3334
Cost Projection (10k queries/day)
3435
Without TRACER 10,000 LLM calls/day $20.00/day
35-
With TRACER 1,926 LLM calls/day $ 3.85/day $5,894 saved/yr
36+
With TRACER 863 LLM calls/day $ 1.73/day $6,670 saved/yr
3637
```
3738

3839
## Quickstart
@@ -71,38 +72,13 @@ User query → [Embedder] → [ML Surrogate] → [Acceptor Gate]
7172
(traditional ML)
7273
```
7374

74-
The surrogate is **not another LLM** - it is a classical ML or shallow DL model (the model zoo includes logistic regression, SGD, gradient-boosted trees, random forests, and small feed-forward nets). This is what makes the cost reduction real: inference is CPU-bound, sub-millisecond, and free.
75+
The surrogate is **not another LLM** - it is a classical ML or shallow DL model (the model zoo includes logistic regression, SGD, LightGBM, random forests, and small feed-forward nets). This is what makes the cost reduction real: inference is CPU-bound, sub-millisecond, and free.
7576

7677
1. **Fit** - train a suite of candidate surrogates on your LLM's classification traces; select the best via cross-validated teacher agreement
7778
2. **Gate** - attach a learned acceptor that estimates, per-input, whether the surrogate will agree with the teacher
7879
3. **Calibrate** - sweep the acceptor threshold to maximise coverage at your target parity (e.g. ≥ 95% teacher agreement)
7980
4. **Guard** - block deployment if the best candidate cannot clear the parity bar on held-out data
8081

81-
## Explainability
82-
83-
Every fit produces a structured audit report alongside the routing policy -- not just aggregate numbers.
84-
85-
| Report component | What it answers |
86-
|-----------------|-----------------|
87-
| **Per-label slice stats** | Which intents are fully self-serve? Which always defer? |
88-
| **Contrastive boundary pairs** | For each label: one handled example, one deferred -- what makes an input "easy" vs "hard"? |
89-
| **Representative examples** | What does typical handled/deferred traffic look like? |
90-
| **Temporal deltas** | Which intents improved between the last refit and this one? |
91-
92-
```python
93-
result = tracer.fit("traces.jsonl", embeddings=X)
94-
qr = result.qualitative_report
95-
96-
print(qr.summary)
97-
# "Handled 9278/10003 (92.8%) by surrogate"
98-
99-
for bp in qr.boundary_pairs[:3]:
100-
print(f"{bp.teacher_label}: handled={bp.handled_preview!r} / deferred={bp.deferred_preview!r}")
101-
102-
# Generate self-contained HTML report
103-
tracer.generate_html_report(".tracer")
104-
```
105-
10682
## Benchmark results (Banking77 - 77-class intent classification)
10783

10884
| Metric | Value |
@@ -187,18 +163,6 @@ pip install tracer-llm[all] # everything
187163
| [Artifacts](docs/artifacts.md) | `.tracer/` directory schema |
188164
| [AGENTS.md](AGENTS.md) | Integration guide for AI coding assistants |
189165

190-
## Prior work
191-
192-
TRACER builds on and connects three research threads:
193-
194-
**Learn to Defer (L2D)** -- the algorithmic backbone. The surrogate + acceptor gate architecture is grounded in Mozannar & Sontag ([ICML 2020](https://proceedings.mlr.press/v119/mozannar20b.html)), Madras et al. ([NeurIPS 2018](https://papers.nips.cc/paper_files/paper/2018/hash/09d37c08f7b129e96277388757530c72-Abstract.html)), and Mao et al. ([NeurIPS 2023](https://proceedings.neurips.cc/paper_files/paper/2023/hash/7e1bf09d2e4e6e5c5a0b63b66b47ef42-Abstract.html)).
195-
196-
**Post-hoc calibration** -- the threshold sweep that enforces the parity guarantee follows the post-hoc deferral estimator framework of Narasimhan et al. ([ICML 2022](https://proceedings.mlr.press/v162/narasimhan22a.html)).
197-
198-
**LLM routing** -- the problem setting is shared with Hybrid LLM (Ding et al., [ICLR 2024](https://openreview.net/forum?id=02f3mUtqnM)) and RouteLLM (Ong et al., [ICLR 2025](https://openreview.net/forum?id=sSt9fROCn2)). TRACER's distinction: it targets classification specifically, uses the teacher's own trace distribution rather than preference data, and enforces a formal parity guarantee rather than a quality heuristic.
199-
200-
**Interpretability** -- the qualitative audit design is informed by Slice Finder (Chung et al., [ICDE 2019](https://ieeexplore.ieee.org/document/8731353)) and the contrastive explanation literature (Miller, [AIJ 2019](https://www.sciencedirect.com/science/article/pii/S0004370217301126)).
201-
202166
## Paper
203167

204168
A research paper detailing the approach, formal guarantees, ablation studies, limitations, and reproducible experiment tooling is in preparation. It will be linked here upon publication.

‎docs/api.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ result.manifest.embedding_dim # int
3535
result.qualitative_report # QualitativeReport | None
3636
result.notes # list[str], human-readable notes
3737
result.artifact_dir # str
38+
result.get_sankey() # generate Sankey diagram (requires tracer-llm[viz])
3839
```
3940

4041
**Example:**
@@ -378,6 +379,51 @@ webbrowser.open(f"file://{path}")
378379

379380
---
380381

382+
## `tracer.generate_sankey()`
383+
384+
Generate an interactive Sankey diagram of the routing flow.
385+
386+
```python
387+
tracer.generate_sankey(
388+
artifact_dir,
389+
output_path=None,
390+
fmt="html",
391+
top_k=15,
392+
title=None,
393+
) -> str
394+
```
395+
396+
**Parameters:**
397+
398+
| Name | Default | Description |
399+
|------|---------|-------------|
400+
| `artifact_dir` | required | Path to `.tracer/` directory |
401+
| `output_path` | `<artifact_dir>/sankey.<fmt>` | Where to write the output |
402+
| `fmt` | `"html"` | `"html"` for interactive, `"png"`, `"svg"`, `"pdf"`, or `"jpeg"` for static |
403+
| `top_k` | `15` | Number of top labels to show individually (rest grouped as "other") |
404+
| `title` | auto-generated | Custom diagram title |
405+
406+
**Returns:** `str` path to the generated file.
407+
408+
**Requires:** `pip install tracer-llm[viz]`
409+
410+
**Example:**
411+
412+
```python
413+
path = tracer.generate_sankey(".tracer")
414+
tracer.generate_sankey(".tracer", fmt="png", output_path="routing.png")
415+
```
416+
417+
Also available as a method on `FitResult`:
418+
419+
```python
420+
result = tracer.fit("traces.jsonl", embeddings=X)
421+
result.get_sankey() # interactive HTML
422+
result.get_sankey(fmt="png") # static image
423+
```
424+
425+
---
426+
381427
## `FitConfig`
382428

383429
Configuration for `fit()` and `update()`.

‎docs/assets/sankey.png‎

750 KB
Loading

‎docs/cli.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ tracer <command> [options]
1313
| `tracer update` | Refit with new traces (continual learning) |
1414
| `tracer report` | Print the policy manifest as JSON |
1515
| `tracer report-html` | Generate an HTML audit report and open it |
16+
| `tracer sankey` | Generate a Sankey routing flow diagram |
1617
| `tracer serve` | Start a prediction HTTP server |
1718

1819
---
@@ -208,6 +209,38 @@ The HTML is fully self-contained -- no external dependencies, works offline, can
208209

209210
---
210211

212+
## `tracer sankey`
213+
214+
Generate an interactive Sankey diagram showing how traffic flows from labels to the surrogate or teacher.
215+
216+
```bash
217+
tracer sankey [<artifact-dir>] [--output <path>] [--format <fmt>] [--top-k <n>] [--no-open]
218+
```
219+
220+
**Arguments:**
221+
222+
| Argument | Default | Description |
223+
|----------|---------|-------------|
224+
| `artifact-dir` | `.tracer` | Artifact directory |
225+
| `--output` | `<artifact-dir>/sankey.<format>` | Output path |
226+
| `--format` | `html` | `html` (interactive), `png`, or `svg` |
227+
| `--top-k` | `15` | Number of top labels to show individually |
228+
| `--no-open` | off | Don't open browser automatically (html only) |
229+
230+
**Examples:**
231+
232+
```bash
233+
tracer sankey
234+
235+
tracer sankey .tracer --format png --output routing-flow.png
236+
237+
tracer sankey --top-k 20 --no-open
238+
```
239+
240+
**Requires:** `pip install tracer-llm[viz]`
241+
242+
---
243+
211244
## `tracer serve`
212245

213246
Start a lightweight HTTP prediction server (stdlib, zero extra deps).

0 commit comments

Comments
 (0)