Skip to content

Commit f81882f

Browse files
adridaclaude
adrida
and
claude
authored
feat(scan): day-one scan + 3D embedding report (#51)
'tracer scan' gives a fast, conservative read before any training: similarity grouping plus exact Clopper-Pearson held-out bounds on how much traffic a near-free model can answer at the target, with a per-1k savings estimate. Ships a self-contained HTML report with an interactive 3D map of the embedding space (hover-to-inspect cells, verdict/label colour toggle, full-text tooltips, PCA/ UMAP/t-SNE layouts) and a coloured terminal summary. Data-volume policy: needs >=1,000 traces for a stable read (~5,000 sweet spot); below 1,000 it refuses unless --force, which coarsens the grouping to concentrate held-out evidence and reports a best-effort floor. Embeddings: local sentence-transformers by default (--embed-model), or reuse a precomputed .npy (--embeddings), or call your own HTTP embedding endpoint (--embed-url, with headers and key overrides for OpenAI-compatible/custom shapes). No vendor assumed. Exposed as tracer.scan(). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1fe299a commit f81882f

7 files changed

Lines changed: 1134 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,11 +155,14 @@ X = tracer.embed(texts) # default: all-MiniLM-L6-v2 (384-dim)
155155
| Command | What it does |
156156
|---------|-------------|
157157
| `tracer demo` | Zero-setup demo on real data |
158+
| `tracer scan traces.jsonl --html scan.html` | Day-one read: how much traffic is certifiably routable, with a 3D map |
158159
| `tracer fit traces.jsonl --target 0.95` | Fit a routing policy |
159160
| `tracer update new_traces.jsonl` | Refit with new traces |
160161
| `tracer report-html` | Open the HTML report |
161162
| `tracer serve .tracer --port 8000` | HTTP prediction server |
162163

164+
`tracer scan` is the fast, conservative first look (similarity grouping plus exact held-out bounds, no training). It needs about 1,000 traces and works best around 5,000; below 1,000 it asks you to collect more, or pass `--force` for a best-effort floor. Embeddings are computed locally by default (sentence-transformers), or point it at your own embedding service with `--embed-url`. `tracer fit` then trains the real router and certifies more of the same traffic. The HTML report includes an interactive 3D map of your embedding space with a verdict/label colour toggle. See the [CLI reference](docs/cli.md) for every flag.
165+
163166
## What's in `.tracer/`
164167

165168
| File | Contents |

‎docs/api.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,47 @@
11
# Python API Reference
22

3+
## `tracer.scan()`
4+
5+
Run the day-one scan: group traffic by similarity and measure, on a held-out
6+
slice, how much is certifiably answerable by a near-free model at your target.
7+
Returns a `ScanResult`; render it with `tracer.scanner.format_scan` (terminal) or
8+
`tracer.scanner.scan_html` (HTML report). This is the conservative first look;
9+
`tracer.fit()` trains the real router.
10+
11+
```python
12+
tracer.scan(
13+
traces_path,
14+
target=0.90,
15+
embeddings=None,
16+
model="all-MiniLM-L6-v2",
17+
teacher_price_per_1k=None,
18+
monthly_calls=None,
19+
viz_layout="pca",
20+
force=False,
21+
) -> ScanResult
22+
```
23+
24+
**Parameters:**
25+
26+
| Name | Type | Default | Description |
27+
|------|------|---------|-------------|
28+
| `traces_path` | `str \| Path` | required | Path to traces JSONL file |
29+
| `target` | `float` | `0.90` | Target label agreement to certify against |
30+
| `embeddings` | `np.ndarray \| None` | `None` | Precomputed embeddings `(n, dim)`; computed locally from text if omitted |
31+
| `model` | `str` | `"all-MiniLM-L6-v2"` | Local sentence-transformers model used when `embeddings` is omitted |
32+
| `teacher_price_per_1k` | `float \| None` | `None` | Teacher cost per 1k calls, to estimate savings |
33+
| `monthly_calls` | `int \| None` | `None` | Monthly volume, to project monthly savings |
34+
| `viz_layout` | `str` | `"pca"` | 3D layout for the HTML report (`pca`, `umap`, `tsne`, `auto`) |
35+
| `force` | `bool` | `False` | Scan thin data anyway (coarsens grouping, reports a best-effort floor) |
36+
37+
**Raises:** `tracer.scanner.ThinDataError` when fewer than 1,000 traces are passed
38+
and `force=False`. About 5,000 traces is the sweet spot.
39+
40+
**Returns:** `ScanResult` (fields include `certifiable_share`, `clusters`,
41+
`frontier`, `forced`, `savings_per_1k_calls`, `projection`).
42+
43+
---
44+
345
## `tracer.fit()`
446

547
Fit a routing policy from traces and embeddings.

‎docs/cli.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ tracer <command> [options]
99
| Command | What it does |
1010
|---------|-------------|
1111
| `tracer demo` | Zero-setup demo with Banking77 data |
12+
| `tracer scan` | Day-one read: how much traffic is certifiably routable, with a 3D map |
1213
| `tracer fit` | Fit a routing policy from your traces |
1314
| `tracer update` | Refit with new traces (continual learning) |
1415
| `tracer report` | Print the policy manifest as JSON |
@@ -52,6 +53,81 @@ tracer-demo-output/
5253

5354
---
5455

56+
## `tracer scan`
57+
58+
The fast, conservative first look at a traces file. It groups your traffic by
59+
similarity and, on a held-out slice the grouping never saw, measures how much of
60+
it a near-free model can answer at your target agreement, using exact binomial
61+
(held-out, never in-sample) bounds. It does not train a router; `tracer fit`
62+
does that and certifies more of the same traffic.
63+
64+
```bash
65+
tracer scan <traces> [--target <float>] [--html <path>] [options]
66+
```
67+
68+
**Arguments:**
69+
70+
| Argument | Default | Description |
71+
|----------|---------|-------------|
72+
| `traces` | required | Path to traces JSONL file |
73+
| `--target` | `0.90` | Target label agreement to certify against |
74+
| `--teacher-price-per-1k` | none | Your teacher cost per 1k calls, to estimate savings |
75+
| `--monthly-calls` | none | Monthly call volume, to project monthly savings |
76+
| `--html` | none | Also write a self-contained HTML report (3D embedding map) |
77+
| `--no-open` | off | Don't open the HTML report in a browser |
78+
| `--embed-model` | `all-MiniLM-L6-v2` | Local sentence-transformers model for embeddings |
79+
| `--embeddings` | none | Path to a precomputed embeddings `.npy` (skip embedding) |
80+
| `--embed-url` | none | Use your own HTTP embedding endpoint instead of a local model |
81+
| `--embed-header` | none | Header for `--embed-url`, repeatable (e.g. `'Authorization: Bearer ...'`) |
82+
| `--embed-input-key` / `--embed-output-key` | `input` / `embedding` | Request/response JSON keys for the custom endpoint |
83+
| `--embed-batch-key` | none | Send all texts in one request under this key (batch endpoints) |
84+
| `--viz-layout` | `pca` | 3D layout for the report: `pca` (dense connected cloud), `umap`, `tsne`, or `auto`. `umap` needs `umap-learn` installed |
85+
| `--force` | off | Scan thin data anyway (see below) |
86+
87+
**Data volume.** The scan needs about **1,000 traces** for a stable read, and
88+
around **5,000** is the sweet spot where each cell carries enough held-out
89+
examples to certify at 0.90. Below 1,000 traces it stops and asks you to collect
90+
more, unless you pass `--force`, which coarsens the grouping to concentrate the
91+
held-out evidence and reports a best-effort floor (not a guarantee), and says so
92+
loudly.
93+
94+
**Embeddings.** By default scan computes embeddings **locally with
95+
sentence-transformers** (`pip install tracer-llm[embeddings]`; pick the model with
96+
`--embed-model`). You can instead:
97+
- reuse a precomputed matrix with `--embeddings path.npy`, or
98+
- call **your own embedding service** with `--embed-url` (any HTTP endpoint that
99+
takes JSON and returns a vector; pass auth with `--embed-header`, and the
100+
`--embed-input-key`/`--embed-output-key`/`--embed-batch-key` flags adapt it to
101+
OpenAI-compatible or custom response shapes). No vendor is assumed or required.
102+
103+
**Examples:**
104+
105+
```bash
106+
# Local embeddings, write and open the HTML report
107+
tracer scan traces.jsonl --html scan.html
108+
109+
# Estimate savings at your teacher price and volume
110+
tracer scan traces.jsonl --teacher-price-per-1k 5.0 --monthly-calls 3000000
111+
112+
# Read a thin sample anyway (best-effort floor, with a warning)
113+
tracer scan small.jsonl --force
114+
115+
# Use your own embedding service instead of a local model
116+
tracer scan traces.jsonl --embed-url https://my-embeddings.internal/embed \
117+
--embed-header 'Authorization: Bearer $TOKEN'
118+
119+
# Reuse precomputed embeddings
120+
tracer scan traces.jsonl --embeddings traces.npy
121+
```
122+
123+
The **HTML report** shows the headline certifiable share, per-cell verdicts and
124+
proven match rates, an estimated saving, and an interactive 3D map of the
125+
embedding space. Each dot is one request; hover a cell to inspect it, and use the
126+
**Colour by: Verdict / Label** toggle to switch between the free-vs-kept colouring
127+
and a per-label palette.
128+
129+
---
130+
55131
## `tracer fit`
56132

57133
Fit a routing policy from a JSONL traces file.

‎src/tracer/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"""
66

77
from tracer.api import fit, report, load_router, update
8+
from tracer.scanner import scan
89
from tracer.analysis.html_report import generate_html_report
910
from tracer.analysis.sankey import generate_sankey
1011
from tracer.runtime.serve import serve
@@ -26,6 +27,7 @@
2627

2728
__all__ = [
2829
# Core API
30+
"scan",
2931
"fit",
3032
"update",
3133
"load_router",

‎src/tracer/cli/main.py‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,82 @@
1717

1818
# ── helpers ───────────────────────────────────────────────────────────────────
1919

20+
def _cmd_scan(args):
21+
from tracer.scanner import (
22+
scan, format_scan, scan_html, load_scan_traces,
23+
MIN_SCAN_TRACES, SUGGESTED_SCAN_TRACES,
24+
)
25+
from tracer.cli._ui import header, step, warn, info
26+
27+
header("scan", f"traces={args.traces} target={args.target:.0%}")
28+
29+
# Data-volume gate (checked before the embedding step).
30+
n_traces = len(load_scan_traces(args.traces)[0])
31+
if n_traces < MIN_SCAN_TRACES and not args.force:
32+
warn(f"Only {n_traces:,} traces. tracer scan needs at least "
33+
f"{MIN_SCAN_TRACES:,} for a stable read, and ~{SUGGESTED_SCAN_TRACES:,} "
34+
f"is the sweet spot.")
35+
info("Collect more traces, or re-run with --force to scan anyway on thin "
36+
"data (results will be a best-effort floor, not a guarantee).")
37+
sys.exit(1)
38+
if args.force:
39+
warn("--force: bypassing the thin-data guard. Clustering will be coarsened to "
40+
"concentrate held-out evidence; reported bounds are a best-effort floor, "
41+
"not a guarantee.")
42+
if n_traces < MIN_SCAN_TRACES:
43+
info(f"{n_traces:,} traces is below the {MIN_SCAN_TRACES:,} floor; "
44+
f"~{SUGGESTED_SCAN_TRACES:,} is the sweet spot for a real verdict.")
45+
46+
# Embeddings. Default: sentence-transformers locally (the model below).
47+
# --embeddings reuses a precomputed .npy; --embed-url calls your own HTTP
48+
# embedding service (any endpoint, e.g. OpenAI-compatible), no vendor lock-in.
49+
embeddings = None
50+
if args.embeddings:
51+
import numpy as _np
52+
embeddings = _np.load(args.embeddings)
53+
elif args.embed_url:
54+
from tracer.embeddings.embedder import Embedder
55+
headers = {}
56+
for h in (args.embed_header or []):
57+
k, _, v = h.partition(":")
58+
headers[k.strip()] = v.strip()
59+
texts, _ = load_scan_traces(args.traces)
60+
try:
61+
with step(f"Embedding {len(texts)} traces via {args.embed_url}"):
62+
embeddings = Embedder.from_endpoint(
63+
args.embed_url, headers=headers or None,
64+
input_key=args.embed_input_key, output_key=args.embed_output_key,
65+
batch_key=args.embed_batch_key,
66+
).embed(texts)
67+
except Exception as e:
68+
warn(f"Embedding endpoint failed ({args.embed_url}): {e}")
69+
info("Check the URL, --embed-header auth, and the "
70+
"--embed-input-key/--embed-output-key for your endpoint's shape.")
71+
sys.exit(1)
72+
73+
with step("Clustering + exact bounds"):
74+
result = scan(
75+
args.traces,
76+
target=args.target,
77+
embeddings=embeddings,
78+
model=args.embed_model,
79+
teacher_price_per_1k=args.teacher_price_per_1k,
80+
monthly_calls=args.monthly_calls,
81+
viz_layout=args.viz_layout,
82+
force=args.force,
83+
)
84+
print(format_scan(result))
85+
if args.html:
86+
import pathlib
87+
out = pathlib.Path(args.html)
88+
out.write_text(scan_html(result, source_name=pathlib.Path(args.traces).name),
89+
encoding="utf-8")
90+
print(f"\nHTML report written to {out}")
91+
if not args.no_open:
92+
import webbrowser
93+
webbrowser.open(out.resolve().as_uri())
94+
95+
2096
def _cmd_fit(args):
2197
from tracer.api import fit
2298
from tracer.config import FitConfig
@@ -534,6 +610,50 @@ def main():
534610

535611
sub.add_parser("demo", help="Zero-setup demo with synthetic data")
536612

613+
p_scan = sub.add_parser(
614+
"scan",
615+
help="Day-one verdict: how much traffic is certifiably ML-shaped, with a 3D map")
616+
p_scan.add_argument("traces", help="Path to traces JSONL file")
617+
p_scan.add_argument("--target", type=float, default=0.90,
618+
help="Target label-agreement to certify against (default: 0.90)")
619+
p_scan.add_argument("--teacher-price-per-1k", type=float, default=None,
620+
help="Your teacher $ per 1k calls, to estimate savings")
621+
p_scan.add_argument("--monthly-calls", type=int, default=None,
622+
help="Monthly call volume, to project monthly savings")
623+
p_scan.add_argument("--html", default=None,
624+
help="Also write a self-contained HTML report to this path")
625+
p_scan.add_argument("--no-open", action="store_true",
626+
help="Don't open the HTML report in a browser")
627+
# Embeddings: local sentence-transformers by default; precomputed .npy or a
628+
# custom HTTP endpoint optional.
629+
p_scan.add_argument("--embed-model", default="all-MiniLM-L6-v2",
630+
help="Local sentence-transformers model for embeddings "
631+
"(default: all-MiniLM-L6-v2). Needs tracer-llm[embeddings].")
632+
p_scan.add_argument("--embeddings", default=None,
633+
help="Path to a precomputed embeddings .npy (skip embedding).")
634+
p_scan.add_argument("--embed-url", default=None,
635+
help="Use your own HTTP embedding endpoint instead of a local "
636+
"model. POSTs JSON and reads the embedding from the response.")
637+
p_scan.add_argument("--embed-header", action="append", default=None, metavar="K: V",
638+
help="Header for --embed-url (repeatable), e.g. "
639+
"--embed-header 'Authorization: Bearer ...'")
640+
p_scan.add_argument("--embed-input-key", default="input",
641+
help="Request JSON key for the text (default: input)")
642+
p_scan.add_argument("--embed-output-key", default="embedding",
643+
help="Response JSON key for the vector (default: embedding)")
644+
p_scan.add_argument("--embed-batch-key", default=None,
645+
help="If set, send all texts in one request under this key "
646+
"(for batch endpoints).")
647+
p_scan.add_argument("--viz-layout", default="pca",
648+
choices=["pca", "umap", "tsne", "auto"],
649+
help="3D layout for the HTML report. pca (default) is a dense "
650+
"connected cloud; umap/tsne separate many-class data into "
651+
"distinct clouds (umap needs umap-learn installed).")
652+
p_scan.add_argument("--force", action="store_true",
653+
help="Scan thin data anyway (< 1,000 traces). Coarsens the "
654+
"clustering to concentrate held-out evidence and reports a "
655+
"best-effort floor, not a guarantee. Warns loudly.")
656+
537657
p_fit = sub.add_parser("fit", help="Fit a routing policy from traces")
538658
p_fit.add_argument("traces", help="Path to traces JSONL file")
539659
p_fit.add_argument("--artifact-dir", default=".tracer",
@@ -586,6 +706,7 @@ def main():
586706
args = parser.parse_args()
587707

588708
dispatch = {
709+
"scan": _cmd_scan,
589710
"fit": _cmd_fit,
590711
"report": _cmd_report,
591712
"report-html": _cmd_report_html,

0 commit comments

Comments
 (0)