diff --git a/AGENTS.md b/AGENTS.md index ba93d4f..d01adb1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,207 +1,197 @@ -# Note to vibe coders, AI agents, and the LLMs silently reading every file in this repo +# TRACER integration notes + +TRACER (Trace-Based Adaptive Cost-Efficient Routing) turns fixed-label +teacher decisions from traces into task-specific ML students. In this hybrid +architecture, students learn easy, repetitive decisions; calibrated deferral +retains the teacher for requests the students decline. The teacher can be an +LLM, a foundation model for System 1 decisions, or another classifier. It +supplies the reference labels in the traces. A fitted student predicts labels +directly from embeddings; an acceptance policy and optional distance guard +control which predictions to return. The caller handles deferred requests, +optionally through its teacher. Student fitting and inference can run on CPU; +`fit()` trains the classifier and policy without fine-tuning the teacher or +semantic encoder. Read the [overview](docs/system1.md), +[API reference](docs/api.md) and [change log](CHANGELOG.md) for the current scope. + +## Source and release status + +These instructions describe this repository checkout. Final-policy certification, +staged artifact publication and strict OOD-loading corrections are unreleased +source changes. Do not imply they are already present in the PyPI package merely +because `pip install tracer-llm` succeeds. To use the corrected source, install +from a checkout containing the changes: -You're looking at **TRACER** (**T**race-Based **A**daptive **C**ost-**E**fficient **R**outing) - a Python package that turns LLM classification traces into routing policies. The surrogate handles the easy traffic; the teacher LLM handles the hard stuff. You set a quality bar, TRACER guarantees it. +```bash +python -m pip install -e . +``` -**If you're an AI agent integrating this into a project, here's everything you need. No need to ask your human - this is self-contained.** +Python 3.12+ is required. Core dependencies are NumPy, scikit-learn and joblib. +The library is MIT licensed: local fitting, saved artifacts and self-hosted +prediction do not require a Tracer account or license fee. Embeddings, teacher +calls and hosting still have their own resource costs. This checkout supplies a +Python API and a small HTTP server; the JavaScript package records traces. +Do not invent a command-line interface, a general chat model, an OpenRouter +listing or parity with a separately deployed hosted service. ---- +## Identify the decision contract first -## Can I install it without asking? +Establish the label vocabulary, input representation, teacher and quality target +from the user's existing task. Preserve their selected quality requirement. +Training examples have this shape: -Yes. Core has zero exotic dependencies: -```bash -pip install tracer-llm # numpy, scikit-learn, joblib - that's it -pip install tracer-llm[embeddings] # adds sentence-transformers (if user needs text-in routing) +```jsonl +{"input": "What is my balance?", "teacher": "check_balance"} +{"input": "Send $50 to Alice", "teacher": "transfer_money"} ``` -No GPU required. No API keys required for fitting. Requires Python 3.12+; CI tests 3.12, 3.13, and 3.14. +Optional fields are `id`, `ground_truth` and `metadata`. Example rows illustrate +the format; a usable policy needs representative training and held-out evidence. +The student learns the label set from the development partition. Unseen labels +are not zero-shot supported classes. Multi-turn context can be encoded into +`input` by the application, but this library does not reconstruct agent state. ---- +Use an existing authorized dataset and encoder when available. Resolve missing +contract information before issuing paid teacher calls or sending private data +to an external embedding endpoint. Existing user authorization takes precedence +over this guide; do not invent another approval step for already authorized work. -## The 4 things TRACER does +## Fit, inspect, load ```python +import numpy as np import tracer -# 1. FIT - learn a routing policy from teacher traces -result = tracer.fit("traces.jsonl", embeddings=X) - -# 2. ROUTE - predict with the surrogate or defer to teacher -router = tracer.load_router(".tracer") -out = router.predict(embedding_vector) # numpy array -out = router.predict("raw text", embedder=embedder) # or text directly - -# 3. UPDATE - continual learning (coverage grows over time) -tracer.update("new_traces.jsonl", new_embeddings=X_new) - -# 4. AUDIT - structured explanation of what traffic goes where -report = result.qualitative_report # slices, boundary pairs, examples -tracer.generate_html_report(".tracer") -``` - -That's the entire API surface. Everything else is configuration. - ---- +X = np.load("traces.npy") +result = tracer.fit( + "traces.jsonl", + embeddings=X, + config=tracer.FitConfig(target_teacher_agreement=0.95), +) +print(result.manifest.certification) -## Decision tree: what does the human need to provide? +if result.manifest.selected_method is None: + raise RuntimeError("No deployable policy; inspect the certificate and data.") -``` -Does the user have traces (JSONL with "input" + "teacher" fields)? -├── YES → Does the user have embeddings (numpy array, same length)? -│ ├── YES → tracer.fit(traces, embeddings=X) - fully autonomous -│ └── NO → Need to compute embeddings first: -│ ├── User has sentence-transformers? → X = tracer.embed(texts) -│ ├── User has an API endpoint? → Embedder.from_endpoint(url) -│ └── ASK THE HUMAN: "What embedding model/API do you use?" -└── NO → ASK THE HUMAN: "I need your LLM's classification outputs as JSONL. - Each line: {"input": "the text", "teacher": "the_label"}" +router = tracer.load_router(".tracer") +out = router.predict(np.load("new_request.npy")) ``` ---- - -## Trace format (the input) - -```jsonl -{"input": "What is my balance?", "teacher": "check_balance"} -{"input": "Send $50 to Alice", "teacher": "transfer_money"} -``` +The matrix must contain one finite embedding per trace, in the same row order. +Use the same encoder weights, dimension, normalization and preprocessing during +fitting and prediction. Matching dimensions alone does not establish this. -`teacher` = whatever the LLM classified this input as. That's all that's required. -Optional fields: `id`, `ground_truth`, `metadata`. +The fitting search includes linear classifiers, compact MLPs and tree models; +classical ML can make the final accepted decision. Fitting does not fine-tune +an embedding model. Do not describe the library as training only a deferral gate. ---- +Some candidate implementations use multiple CPU cores. Respect workstation and +cloud resource limits; do not run a full fit merely to validate a documentation +change. `skip_candidates` can bound the candidate search, but it must not be used +to silently lower the user's quality target or bypass the final check. -## Embedder setup (for text-in routing at inference) +## Runtime results and teacher fallback ```python -from tracer import Embedder - -# Option A: local sentence-transformers -embedder = Embedder.from_sentence_transformers("BAAI/bge-small-en-v1.5") - -# Option B: external HTTP endpoint (OpenAI, Cohere, Cloudflare, etc.) -embedder = Embedder.from_endpoint( - "https://api.example.com/embed", - headers={"Authorization": "Bearer ..."}, - input_key="input", # key in request JSON - output_key="embedding", # key in response JSON -) - -# Option C: any function -embedder = Embedder.from_callable(lambda texts: my_model.encode(texts)) - -# Attach to router +embedder = tracer.Embedder.from_callable(my_embedding_function) router = tracer.load_router(".tracer", embedder=embedder) -router.predict("What is my balance?") # text in, decision out +text = "What is my balance?" +out = router.predict(text, fallback=lambda: call_my_teacher(text)) ``` -**If you don't know which embedder the user wants: ask.** Embedding model choice matters - it must match what was used at fit time. +The callable embedder accepts a list of strings and returns a matrix. A fallback +accepts no arguments and is invoked only after deferral. The return contract is: ---- +| Field | Meaning | +| --- | --- | +| `label` | Student label when handled; fallback return value or `None` when deferred | +| `decision` | Exactly `handled` or `deferred` | +| `accept_score` | Acceptance ranking score; zero on deferred outputs | +| `stage` | Accepted pipeline stage, or `-1` for deferral | -## What the router returns +An OOD distance rejection is a plain `deferred` result. There is no public +third decision state or reason field. `accept_score` is not a calibrated +correctness probability. A fallback response still has `decision="deferred"`. -```python -{ - "label": "check_balance", # predicted class - "decision": "handled", # "handled" (surrogate) or "deferred" (teacher) - "accept_score": 0.96, # surrogate confidence (0-1) - "stage": 0, # pipeline stage that handled it -} -``` +`predict_batch()` returns labels, decisions, the handled mask, internal prediction +indices and stage IDs. It does not call a teacher for deferred rows. The caller +must process those rows explicitly. -For batch: `router.predict_batch(texts_or_embeddings)` returns `{"labels", "decisions", "handled"}`. +## Certification and failure handling ---- +The default reserves 20% of rows before fitting, label balancing or candidate +selection. Development data selects the student, acceptance thresholds and +policy. The final serving policy, including its OOD guard, is frozen before one +check on the reserve. Do not reuse this reserve to rescue a failed candidate. -## The `.tracer/` directory (artifacts) +The one-sided Clopper–Pearson lower bound on accepted teacher agreement must +meet `target_teacher_agreement`. The default `certification_alpha=0.10` gives a +90% one-sided confidence level under independent, representative sampling. +Accepted coverage must also meet `min_deploy_coverage`. No accepted examples +means no certificate, not perfect agreement. -After `tracer.fit()`, a `.tracer/` directory is created with: -- `manifest.json` - method, coverage, teacher agreement, label space -- `pipeline.joblib` - the fitted surrogate model -- `qualitative_report.json` - per-label stats, boundary pairs, examples -- `frontier.json` - all candidates at each quality target -- `report.html` - visual audit report (after `tracer.generate_html_report()`) +This measures agreement with the teacher, not ground-truth accuracy, individual +correctness or a per-class guarantee. The reserve is a random row split. +Duplicates, correlated sessions, distribution shift and repeated tuning against +held-out results weaken or invalidate that interpretation. Maintain independent +session/time evaluations and report accepted coverage alongside quality. -This directory is self-contained. Copy it anywhere and `tracer.load_router(".tracer")` works. +If `selected_method` is null, `load_router()` refuses to load a serving policy. +A completed `fit()` or `update()` that fails certification may intentionally +publish this null-policy generation. A fitting/publication exception instead +preserves the prior generation through staged rollback. Do not describe every +failed check as preserving the old router, or promise zero-downtime concurrent +readers. Use one writer and reload readers only after inspecting the new manifest. ---- +Expected OOD metadata and reference files are part of the policy. Missing or +corrupt required files fail loading; do not remove them to make loading succeed. +Older artifacts without final-certification metadata are not retroactively +certified by a newer loader. -## Common integration patterns +## Embeddings, updates and deployment -### Pattern 1: Offline fit, online serve -```python -# Offline (data pipeline) -tracer.fit("traces.jsonl", embeddings=X) +Local sentence-transformers support is optional: -# Online (API server) -router = tracer.load_router(".tracer", embedder=my_embedder) -label = router.predict(request.text) +```bash +python -m pip install -e '.[embeddings]' ``` -### Pattern 2: With LLM fallback -```python -out = router.predict(text, fallback=lambda: call_gpt4(text)) -# out["label"] is always set - either from surrogate or LLM -``` +`Embedder.from_sentence_transformers(model)` may download a model; +`Embedder.from_endpoint(url, ...)` calls the configured external HTTP endpoint. +`Embedder.from_callable(fn)` supports an existing encoder. The generic endpoint +adapter expects configured JSON keys; do not claim it implements every provider's +API schema automatically. Precomputed embeddings need no embedding API call. -### Pattern 3: Continual learning loop ```python -# Collect new traces from production tracer.update("new_traces.jsonl", new_embeddings=X_new) -# Reload router with improved policy -router = tracer.load_router(".tracer", embedder=my_embedder) -``` - -### Pattern 4: HTTP server (zero deps) -```python -tracer.serve(".tracer", port=8000) -# POST /predict {"embedding": [...]} → {"label", "decision", "accept_score"} +tracer.generate_html_report(".tracer") ``` ---- - -## Things you should NOT do without asking - -- **Change the embedding model** between fit and predict - dimensions must match -- **Lower target_teacher_agreement below 0.80** - quality will degrade -- **Delete `.tracer/all_traces.jsonl`** - needed for `tracer.update()` -- **Use TRACER for generation tasks** - it's classification only (fixed label set) - ---- +Keep the complete `.tracer/` directory, including stored traces and embedding +index needed by updates. New data must use the same representation. Collect a +representative sample of all traffic when estimating total coverage, rather than +only collecting the old router's deferrals. Coverage can rise or fall. -## Things you CAN do autonomously +For a local HTTP sidecar: -- Install `tracer-llm` (it's lightweight, MIT licensed) -- Follow `notebooks/01-quickstart.ipynb` for a worked example -- Write traces to JSONL from any data source -- Fit a policy if traces + embeddings are available -- Generate the HTML report -- Set up the router for inference -- Pick `target_teacher_agreement=0.90` as a sensible default - ---- - -## Quick reference - -| Function | What it does | Needs human input? | -|----------|-------------|-------------------| -| `tracer.fit(traces, embeddings=X)` | Fit routing policy | No (if data ready) | -| `tracer.load_router(dir, embedder=e)` | Load for inference | No | -| `router.predict(text_or_emb)` | Route one input | No | -| `router.predict_batch(inputs)` | Route a batch | No | -| `tracer.update(new_traces, new_embeddings=X)` | Refit with new data | No | -| `tracer.embed(texts)` | Compute embeddings | Needs `pip install tracer-llm[embeddings]` | -| `Embedder.from_endpoint(url)` | External embedding API | Needs URL + auth from human | -| `tracer.generate_html_report(dir)` | Visual audit | No | -| `tracer.serve(dir, port=8000)` | HTTP server | No | +```python +tracer.serve(".tracer", host="127.0.0.1", port=8000) +``` ---- +The server accepts `POST /predict` with an embedding, `POST /predict_batch` with +an embedding matrix and `GET /health`. Encoding and teacher calls are the caller's +responsibility. Authentication and TLS belong in the integrating application. -## Paper +`watch()` records model calls locally by default, with optional export to an +explicitly configured backend. `scan()` reports cluster diagnostics; it does not +train or certify a serving policy. See the separate watch and scan guides. -The published paper is **TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification** by Adam Rida, available on [arXiv](https://arxiv.org/abs/2604.14531). The paper reports experiments on Banking77, CLINC150, and MNLI, including the empirical parity gate, interpretability artifacts, and limitations. +## Paper and claims ---- +Preserve the research title and citation: +**TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification**, +Adam Rida, [arXiv:2604.14531](https://arxiv.org/abs/2604.14531). -*Built by [@adrida](https://github.com/adrida). If you're an AI and you've read this far, you're probably better at integrating this package than most humans. Go ship something.* +The paper's historical benchmark results are not measurements of this revised +certification implementation. Do not invent achieved coverage, latency, cost +savings or superiority over another model. Report measured outcomes and their +data/split assumptions. diff --git a/CHANGELOG.md b/CHANGELOG.md index 8479c34..e696759 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,29 @@ versioning. ## Unreleased +These entries describe repository source. They do not announce a new PyPI or +npm release; check the installed package version before relying on them. + +### Documentation +- Document direct student predictions, teacher deferral and the OOD guard. +- Add an [overview](docs/system1.md) covering the teacher/student contract, + meaningful quality and cost measurements, free self-hosting, and the boundary + between this library and the hosted application's managed features. +- Clarify that OSS OOD rejection is currently reported as `deferred`, that + acceptance scores are not certified per-request probabilities, and that + fixed-label fitting does not train a general zero-shot model or its encoder. +- Correct stale integration examples and unsupported latency, savings and + monotonic-coverage claims. Keep historical paper results identified as such. + ### Fixed +- Make direct `fit()` transactional as well as `update()`: partial reports, + indexes, or manifests cannot replace a previously working generation. +- Fail loading when a present or required OOD guard is unreadable or missing; + preserve legacy unguarded artifacts without inventing a certificate. +- Reserve final certification rows before fitting, label discovery or balancing. + Check the selected serving policy, including its development-only OOD guard, + once using a one-sided agreement lower bound; never retry other candidates + using these outcomes. Zero accepted examples cannot certify a policy. - Preserve the previous artifact generation when an update fails validation, fitting, or publication. Honor explicit update configuration without mutating the caller, retain saved configuration by default, and increment the fit count. @@ -26,6 +48,15 @@ versioning. - Correct update examples to use `new_embeddings=`. ### Changed (breaking) +- New fits reserve 20% of rows by default (`certification_fraction`) and use + `certification_alpha=0.10`. Small or uncertain fits may now decline deployment. + `manifest.certification` records the result; legacy-named `coverage_cal` and + `teacher_agreement_cal` now measure this final partition for new artifacts. + The confidence statement requires independent representative examples and + does not cover distribution shift, correlated sessions, or adaptive reuse. +- Persist `ood_reference.npy` separately from all embeddings used for updates. + Existing artifacts keep their previous behavior unless their guard is broken. +- Remove unsupported universal coverage, savings and monotonic-growth claims. - The prediction server binds to `127.0.0.1` by default and no longer sends a wildcard CORS header. Explicitly configure a remote bind address when needed; browser deployments should configure CORS at their authenticated proxy. @@ -85,6 +116,10 @@ versioning. ## 0.2.0 (2026-06) +Historical release notes. The scan remains a diagnostic; the calibration +procedure described here is superseded by the independent final-policy check +in Unreleased. These entries do not establish current deployment guarantees. + ### Added - `tracer.scan()`: a fast, conservative day-one read of a traces file, before any training. It groups traffic by similarity and measures, on a held-out slice it @@ -106,11 +141,11 @@ versioning. `embeddings=`, or select a local sentence-transformers model with `model=`. ### Changed -- The parity gate now certifies on an exact held-out lower bound instead of an - in-sample point estimate, so a policy cannot clear the target by in-sample luck - and then break the contract on real traffic. Coverage is now monotonic in the - target, and a hybrid select-then-verify procedure recovers coverage at strict - targets that a plain held-out split discarded. +- The parity gate introduced held-out lower-bound checks and a hybrid + select-then-verify procedure. Later review found that selection and final + certification must use separate data; see Unreleased. This historical + procedure does not guarantee agreement on future traffic or monotonic + coverage across targets. - The HTML report is restyled to the light Tracer theme, and the word "audit" is dropped across the report and docs. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 10fbff4..964aea5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,12 +2,13 @@ Thank you for your interest in contributing. -TRACER is a routing library with formal parity guarantees, and those guarantees -only mean something if they hold up under measurement. So for anything that can -change routing behavior, the most useful thing you can bring is evidence. The -standards below exist to keep contributions focused and to make the best use of -your time and ours. None of this is meant to gatekeep good ideas. it's meant to -help us evaluate them fairly. +TRACER builds task-specific classifiers: classical or +shallow neural classifiers make direct fixed-label decisions, while a learned +policy leaves other requests to the caller's teacher. The final check bounds +teacher agreement under stated sampling assumptions. It does not guarantee +ground-truth accuracy, per-request correctness or robustness to distribution +shift. Contributions should preserve those distinctions and provide evidence +for changes in behavior. ## Contribution standards (read first) @@ -24,8 +25,8 @@ for us to act on when they come with numbers. Three guidelines follow from that. 2. **Changing a core parameter, threshold, or gate: bring an ablation.** The acceptor threshold, the parity gate, the calibration logic, the default - `FitConfig` values, and the split strategy are load-bearing for the parity - guarantee. A change to any of them is much easier to accept with before/after + `FitConfig` values, and the split strategy determine the statistical contract. + A change to any of them is much easier to accept with before/after numbers on the eval showing the frontier does not regress (and, ideally, improves). If you think a parameter should be user-tunable, the most convincing case is one where a different value measurably wins. @@ -45,14 +46,23 @@ Run against a fixed, public eval so results are reproducible by a maintainer: - Use a known dataset (the repo's test fixtures, or a public set such as Banking77) and a fixed `seed`. - Report the metric that matters: **coverage at the target teacher agreement** - (how much traffic the surrogate can take while holding parity), plus the - realized teacher agreement on a held-out split (not the calibration split). + together with accepted counts, realized agreement, the final lower bound and + confidence level. Use an untouched evaluation split, not the examples used + for classifier/threshold selection. Where ground truth exists, report its + accuracy separately, including per-class or worst-group behavior. +- Measure full-path latency and cost when claiming efficiency: embeddings, + student inference, teacher deferrals and relevant serving overhead. A gate's + score is not a calibrated per-request correctness probability. +- Separate sessions/workflows/time where the dataset permits; random row splits + do not establish independence. Preserve historical benchmark results and label + new results with their exact implementation and evaluation protocol. - Compare against the current behavior on `main`, so the delta is clear. - Include the exact command and config you ran so the numbers can be checked. -If a change does not move the frontier (or regresses it), we'll usually hold -off on merging even when the code itself is clean, since the numbers are what -we're optimizing for. +Performance proposals should show their tradeoff. Correctness fixes, truthful +contracts and reproducible failure handling are valuable even when they reduce +previously overstated coverage; they do not need an invented benchmark win. +Never retune a policy against its final certification outcomes to make it pass. ## Setup @@ -68,6 +78,8 @@ pip install -e ".[dev]" ## Running tests ```bash +TOKENIZERS_PARALLELISM=false RAYON_NUM_THREADS=1 OMP_NUM_THREADS=1 \ +OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 \ pytest tests/ -v ``` @@ -97,6 +109,8 @@ src/tracer/ types.py <- TraceRecord, QualitativeReport, ArtifactManifest, ... fit/ pipeline.py <- global / L2D / RSB pipeline construction + calibration + certification.py <- untouched final-policy teacher-agreement check + ood.py <- distance guard, included in final serving check surrogate.py <- model zoo (LogReg, SGD, MLP, RF, ET, DT, GBT, XGB) + selection analysis/ qualitative.py <- XAI report: slices, boundary pairs, examples, deltas @@ -134,17 +148,30 @@ further. It's usually quick to run, and it's the thing that lets us say yes. ## Changing the gate, calibration, or default config -These are the parts that back the parity guarantee. A PR here must include an -ablation: the frontier on `main` versus the frontier with your change, on the -same eval and seed. State clearly which guarantee you are strengthening and -show it does not regress coverage or realized agreement. +These parts define selection and final certification. Performance changes need +an ablation on the same external evaluation and seed. Contract/correctness fixes +need focused regressions that demonstrate the broken behavior and its repair. +Verify that no final outcomes influence training, threshold selection, OOD +fitting or retries. Keep no-policy results, exceptions and legacy-artifact +behavior explicit; a completed failed check and a failed write are different. ## Adding a new pipeline family Implement a `build_(split, target_ta) -> dict` function in `src/tracer/fit/pipeline.py` following the same structure as `build_global`, `build_l2d`, and `build_rsb`. Register it in the `builders` dict inside -`fit_frontier`. Include a benchmark showing where the new family wins. +`fit_frontier`, preserving the builders' keyword options. Include a benchmark +showing where the new family wins. The full selected policy, including residual +stages and OOD checks, must receive the final check and match runtime behavior. + +## Documentation and integration boundaries + +Check examples against current function signatures and saved artifact fields. +The bundled HTTP server takes vectors and does not call teachers. Generic +embedding/export adapters do not imply a turnkey provider integration or +standards-compliant OTLP transport. The current package exposes Python APIs, +not a packaged CLI; hosted catalogs, credentials and Echo wallets are separate. +Document proposed features as proposals rather than existing OSS capabilities. ## Submitting a PR diff --git a/README.md b/README.md index 70b1a7b..62a41d6 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # TRACER -**Trace-Based Adaptive Cost-Efficient Routing** +*Trace-Based Adaptive Cost-Efficient Routing* + +**Distill your LLM traces into fast ML System 1 models.** [![arXiv](https://img.shields.io/badge/arXiv-2604.14531-b31b1b.svg)](https://arxiv.org/abs/2604.14531) [![Hugging Face](https://img.shields.io/badge/🤗%20HF-Papers-yellow)](https://huggingface.co/papers/2604.14531) @@ -10,212 +12,277 @@ [![Python](https://img.shields.io/pypi/pyversions/tracer-llm)](https://pypi.org/project/tracer-llm/) [![npm](https://img.shields.io/npm/v/@tracer-llm/watch?label=%40tracer-llm%2Fwatch&color=cb3837&logo=npm)](https://www.npmjs.com/package/@tracer-llm/watch) [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) -[![CI](https://img.shields.io/badge/CI-passing-brightgreen)](https://github.com/adrida/tracer/actions) +[![CI](https://github.com/adrida/tracer/actions/workflows/ci.yml/badge.svg)](https://github.com/adrida/tracer/actions/workflows/ci.yml) [![Docs](https://img.shields.io/badge/docs-reference-blue)](docs/) -Most LLM-based classification pipelines use a large language model for every single input. In practice, the vast majority of that traffic is predictable - a lightweight traditional ML model (logistic regression, gradient-boosted trees, or a small neural net) can match the LLM's output with near-perfect agreement. - -TRACER learns the decision boundary between "easy" and "hard" inputs directly from your LLM's own classification traces. It fits a fast, non-LLM surrogate on the easy partition, gates it with a calibrated acceptor, and defers only the uncertain inputs back to the LLM. Every deferred call produces a new trace, which feeds the next refit - coverage grows automatically over time. The result: **90%+ of classification calls routed to traditional ML, with formal parity guarantees against the teacher LLM and a self-improving routing policy**. - -```bash -pip install tracer-llm -``` +TRACER learns **task-specific ML students** from your teacher's decisions. +The ML students return labels locally for repetitive requests. **Calibrated +deferral** and an **out-of-distribution (OOD) guard** keep the teacher in the +loop for requests the ML path declines. Your teacher can be an LLM, a +foundation model for System 1 decisions, or another classifier. -Requires **Python 3.12+**. CI tests Python 3.12, 3.13, and 3.14. +**No LLM fine-tuning. No GPU required. Your teacher stays in the loop.** -## Quickstart +ML fitting and prediction run on CPU with supplied embeddings. You choose the +encoder and teacher; their compute is separate. -Input: a JSONL file where each line contains the original text (`input`) and the label your LLM assigned (`teacher`). +![Teacher-labelled traces become an ML student and deferral policy. Blue requests receive a local ML label; orange deferred and red OOD requests go to the teacher LLM.](docs/assets/training-inference.gif) -```python -import tracer +*Training and inference on illustrative support-ticket traces.* -# 1. Fit - learn a routing policy from your LLM's classification traces -result = tracer.fit( - "traces.jsonl", # {"input": "...", "teacher": "label"} per line - embeddings=X, # np.ndarray (n, dim) - precomputed text embeddings - config=tracer.FitConfig(target_teacher_agreement=0.95), -) - -# 2. Route - surrogate handles easy inputs, LLM handles the rest -router = tracer.load_router(".tracer", embedder=embedder) -out = router.predict("What is my balance?") -# {"label": "check_balance", "decision": "handled", "accept_score": 0.96} - -# 3. Fallback - only invokes the LLM when the surrogate declines -out = router.predict("Some edge case", fallback=lambda: call_my_llm(text)) -``` -The [API reference](docs/api.md) covers fitting, routing, updates, and reports. -See [concepts](docs/concepts.md) for the pipeline and [watch](docs/watch.md) for -local trace recording and opt-in exports to your own backend. - -## Watch your LLM traffic - -Before you fit anything, just *watch*. Wrap any LLM call and every request is -recorded locally as an OpenTelemetry GenAI span, no account, no key, nothing -leaves your machine: - -```python -import tracer - -watch = tracer.watch("support_classifier", system="my-provider", model="my-model") - -@watch -def classify(ticket: str) -> str: - return call_my_llm(ticket) # traces append to .tracer/watch/*.jsonl -``` - -The same watched spans map 1:1 to `TraceRecord`, so once you have traffic you can -call `tracer.fit()` to train a router from it. Full guide: [docs/watch.md](docs/watch.md). - -## Using from JavaScript / Node.js +## Quickstart -**Watch your JS LLM calls (free observability):** [`@tracer-llm/watch`](https://www.npmjs.com/package/@tracer-llm/watch) mirrors the Python decorator with zero dependencies, recording every call as an OpenTelemetry GenAI span (local by default, with opt-in export to your own backend). +From a checkout of this branch ([release status](#release-status)), with +Python 3.12+: ```bash -npm install @tracer-llm/watch +python -m pip install -e . ``` -```js -import { watch } from "@tracer-llm/watch"; +Supply representative teacher-labelled traces in `traces.jsonl`, with one +embedding per row in `traces.npy`. The trace format is: -const w = watch("support_classifier", { system: "provider-x", model: "model-x" }); - -// Wrap the function that calls your model; the return value is auto-captured. -const classify = w(async (ticket) => callYourLLM(ticket)); +```jsonl +{"input": "What is my balance?", "teacher": "check_balance"} +{"input": "Send $50 to Alice", "teacher": "transfer_money"} ``` -Full guide: [docs/javascript.md](docs/javascript.md). To route (not just observe) from JS, log traces, fit offline with `tracer.fit()`, run `tracer.serve()` in a Python sidecar, and call it via `fetch`: - -```js -// 1. Log every LLM classification -fs.appendFileSync('traces.jsonl', JSON.stringify({ input: text, teacher: label }) + '\n') +These two rows illustrate the format; fit on your actual trace dataset. +`new_request.npy` contains one new request encoded with the same model. -// 2. At inference: embed → POST to TRACER → fallback to LLM only if deferred -let { label, decision } = await fetch('http://localhost:8000/predict', { - method: 'POST', - body: JSON.stringify({ embedding }), // same model you used at fit time -}).then(r => r.json()) +```python +import numpy as np +import tracer -if (decision === 'deferred') label = await callYourLLM(text) +result = tracer.fit( + "traces.jsonl", embeddings=np.load("traces.npy"), + config=tracer.FitConfig(target_teacher_agreement=0.95), +) +if result.manifest.selected_method is None: + raise RuntimeError("No policy passed the final check.") +router = tracer.load_router(".tracer") +out = router.predict(np.load("new_request.npy")) +print(out["label"], out["decision"]) # label + "handled", or None + "deferred" ``` -See the [JavaScript integration guide](docs/javascript.md) for the full setup including embeddings, docker-compose, batch prediction, and continual learning. +Your application sends deferred requests to the teacher, or supplies a +[fallback callback](#ood-and-teacher-fallback). + +## Why Tracer? + +| | TRACER | +| --- | --- | +| ML student | Task-specific classical ML or a shallow neural classifier | +| Training data | Your existing teacher-labelled traces and embeddings | +| Decisions | Labels from a fixed set: intents, tags, tool choices, workflow actions | +| Accepted traffic | The ML student returns the label locally | +| Declined traffic | Your teacher answers through your application's fallback | +| Unfamiliar inputs | An OOD distance guard can override student acceptance | +| Hardware | CPU-capable ML fitting and inference; no LLM fine-tuning | +| Quality check | The complete serving policy is checked once on reserved data | +| Ownership | MIT licensed; fit and self-host without a Tracer account | + +Your embedding, teacher and serving costs still apply. Measure coverage, +latency and savings on your workload. + +## What is learned + +The ML student learns to predict your teacher's labels directly from embeddings. +The search includes logistic regression, SGD classifiers, trees, forests, +boosted trees and compact MLPs, with optional XGBoost. A learned acceptance +policy controls which student predictions to return; residual stages can try +another student before deferring to the teacher. + +The encoder and teacher are not fine-tuned. TRACER learns your task's label set; +it does not train a general pretrained decision model or generate free-form +responses. + +For a smaller fitting sweep, `FitConfig(skip_candidates=("dt", "rf", "et", +"gbt", "xgb"))` excludes tree candidates. This changes the search budget, not +the selected quality requirement. Some candidates use multiple CPU cores; +choose resource limits appropriate to your machine. See [concepts](docs/concepts.md) +for methods and [API reference](docs/api.md) for the complete configuration. + +## How the quality check works + +TRACER selects the ML policy on development data, freezes the complete serving +policy including its OOD guard, then checks that exact policy once on reserved +data before publishing it. If it misses the requirement, the fit does not +publish a usable policy. + +1. Reserve final certification rows before label discovery, balancing or fitting. +2. Select student classifiers using validation macro-F1 against teacher labels. +3. Fit acceptance policies and select a global or staged policy for the chosen + teacher-agreement target and coverage requirement on development data. +4. Freeze that serving policy, including its distance guard, and check it once + on the reserved rows. A failed final check does not trigger another candidate + search on those rows. + +For the fixed final serving policy, a one-sided Clopper–Pearson lower bound on +**teacher agreement among accepted requests** must meet the target. Defaults +reserve 20% of input rows and use `certification_alpha=0.10`, corresponding to a +90% one-sided confidence level under independent, representative sampling. +Accepted coverage must also meet `min_deploy_coverage`. Zero accepted examples +cannot pass this check. + +Teacher agreement is distinct from ground-truth accuracy. This aggregate bound +does not certify each prediction or each class. `accept_score` is a ranking +signal, not a calibrated per-request correctness probability. The distance +guard can reject unfamiliar embeddings; it provides no general guarantee +against distribution shift. + +The built-in reserve is a random row split. Correlated sessions, duplicates, +adaptive reuse of the reserve and later traffic changes can invalidate the +sampling interpretation. Keep separate session/time evaluations, inspect rare +classes and measure both accepted quality and coverage before relying on a +policy. Paper results are historical experiments, not measurements of this +revised certification implementation. + +## OOD and teacher fallback + +An unfamiliar request can receive a confident ML prediction. The +**out-of-distribution (OOD) guard** checks its embedding distance to neighbouring +training examples. When a fitted guard flags the input as too far away, it +forces deferral even if the ML student would otherwise accept it. + +These are two different reasons to use the teacher: + +- **The acceptance policy declines the prediction:** the fitted rule does not + accept the student's answer at the chosen operating point. +- **The OOD guard flags the input:** its embedding is too far from the training + reference under the fitted distance rule. + +Both return `decision="deferred"`. The public API currently has two decisions, +`"handled"` and `"deferred"`; it does not expose a separate OOD decision or a +rejection-reason field. A distance guard can miss unfamiliar inputs and is +fitted only when sufficient reference data is available. Evaluate it on your +own out-of-domain examples as well as ordinary traffic. + +To accept raw text and connect your teacher, attach the same embedding +implementation used during fit. Matching vector dimensions alone is +insufficient: weights, normalization and preprocessing must match too. -## How it works +```python +embedder = tracer.Embedder.from_callable(my_embedding_function) +router = tracer.load_router(".tracer", embedder=embedder) +text = "What is my balance?" +out = router.predict(text, fallback=lambda: call_my_teacher(text)) ``` -User query → [Embedder] → [ML Surrogate] → [Acceptor Gate] - | | - score >= t score < t - | | - Local answer Defer to LLM - (traditional ML) -``` - -The surrogate is **not another LLM** - it is a classical ML or shallow DL model. The Python API searches linear, neural, and tree-based candidates. For a lighter sweep, pass `FitConfig(skip_candidates=("dt", "rf", "et", "gbt", "xgb"))` to exclude tree models. Inference runs locally on your CPU. - -1. **Fit** - train a suite of candidate surrogates on your LLM's classification traces; select the best via cross-validated teacher agreement -2. **Gate** - attach a learned acceptor that estimates, per-input, whether the surrogate will agree with the teacher -3. **Calibrate** - sweep the acceptor threshold to maximise coverage at your target parity (e.g. ≥ 95% teacher agreement) -4. **Guard** - block deployment if the best candidate cannot clear the parity bar on held-out data -## Benchmark results (Banking77 - 77-class intent classification) +`my_embedding_function` accepts a list of strings and returns an embedding +matrix. The optional fallback takes no arguments and runs only for a deferred +request. Its return value becomes `label`; `decision` remains `"deferred"`. +Without a fallback, your application handles deferral explicitly. Batch +prediction returns deferred rows for your application to process; it does not +call the teacher. -| Metric | Value | -|--------|-------| -| Coverage | **92.2%** of traffic handled locally | -| Teacher agreement (handled) | 96.1% | -| End-to-end accuracy | 96.4% | -| **Annual savings** (10k queries/day) | **$302,850** | +## Local serving and updates -_Banking77 is a 77-class task; these results include tree-based candidates. Candidate selection and coverage depend on your data._ - -## Continual learning flywheel - -TRACER is not a one-shot fit. Every deferred input that reaches the LLM produces a new labeled trace, which feeds back into the next refit. As the surrogate sees more of the input distribution, its coverage grows - meaning fewer LLM calls, which in turn cost less, while the quality guarantee holds at every iteration. - -``` -Day 1: 2,000 traces → 84% coverage → 1,600 calls/day saved -Day 3: 6,000 traces → 90% coverage → 9,000 calls/day saved -Day 5: 10,000 traces → 92% coverage → 9,200 calls/day saved -``` +A fitted router can run inside your Python application or through the small +built-in HTTP server: ```python -tracer.update("new_traces.jsonl", new_embeddings=X_new) # refit with new production traces +import tracer +tracer.serve(".tracer", host="127.0.0.1", port=8000) ``` -The parity gate re-calibrates on each update, so coverage only increases when the surrogate actually earns it. +It exposes `POST /predict` with `{"embedding": [...]}`, `POST /predict_batch` +with `{"embeddings": [[...], ...]}`, and `GET /health`. The server accepts +embeddings; the calling application owns text encoding and teacher fallback. +It does not add authentication or TLS. You can integrate the same Python +router into your own authenticated service. -## Embedder options +`update()` combines stored and new traces and refits the policy: ```python -from tracer import Embedder - -embedder = Embedder.from_sentence_transformers("BAAI/bge-small-en-v1.5") # local -embedder = Embedder.from_endpoint("https://api.example.com/embed", headers={...}) # API -embedder = Embedder.from_callable(my_fn) # any function -# or skip the embedder and pass raw np.ndarray embeddings directly -``` - -Need to compute embeddings at fit time? - -```bash -pip install tracer-llm[embeddings] # adds sentence-transformers +tracer.update("new_traces.jsonl", new_embeddings=X_new) ``` -```python -X = tracer.embed(texts) # default: all-MiniLM-L6-v2 (384-dim) -``` +Coverage may rise or fall. Collect a representative sample of all traffic, not +only deferrals, when estimating overall coverage. `fit()` and `update()` stage +artifacts and roll back on fitting or publication **exceptions**. A completed +refit that fails certification can intentionally publish +`selected_method=null`, replacing the prior on-disk generation. Check the new +manifest before reloading. Use one writer per artifact directory; these file +operations do not provide concurrent-reader or zero-downtime deployment control. -## Inspect traffic before fitting +## Record traces and inspect traffic -`tracer.scan()` groups traces by similarity and estimates the certifiable share -using held-out bounds. Pass the embeddings that correspond to your trace rows: +`tracer.watch()` records your model calls locally by default, with optional +export to your own backend. It does not train a student until you call `fit()`. ```python -from pathlib import Path import tracer -from tracer.scanner import scan_html -scan = tracer.scan("traces.jsonl", embeddings=X, target=0.95) -Path("scan.html").write_text(scan_html(scan), encoding="utf-8") +watch = tracer.watch("support_classifier", system="my-provider", model="my-model") + +@watch +def classify(ticket: str) -> str: + return call_my_teacher(ticket) ``` -A scan needs at least 1,000 traces; around 5,000 is recommended. Use `force=True` -only for an explicitly marked thin-data estimate. It does not train a router; -use `tracer.fit()` for training. See the [scan guide](docs/scan.md). +[Watch](docs/watch.md) documents trace recording and export. For JavaScript, +[`@tracer-llm/watch`](js/README.md) records calls; training and prediction remain +in the Python library. The [JavaScript guide](docs/javascript.md) shows the +Python sidecar integration. -## What's in `.tracer/` +`tracer.scan()` groups embedded traces and reports per-cluster diagnostics. It +needs at least 1,000 traces by default; `force=True` permits an explicitly thin +data estimate. Scanning does not train or certify the final serving policy. +See the [scan guide](docs/scan.md). -| File | Contents | -|------|----------| -| `manifest.json` | Method, coverage, teacher agreement, label space | -| `pipeline.joblib` | Surrogate + acceptor + calibrated thresholds | -| `frontier.json` | All candidates at each quality target | -| `qualitative_report.json` | Per-label slices, boundary pairs, examples | -| `report.html` | Visual HTML report | +## Embeddings and artifacts -## Install +Use supplied NumPy arrays, `Embedder.from_callable`, a compatible HTTP endpoint, +or a local sentence-transformers encoder. To install the optional local encoder +support from this checkout: ```bash -pip install tracer-llm # core (numpy + sklearn + joblib) -pip install tracer-llm[embeddings] # + sentence-transformers -pip install tracer-llm[all] # everything +python -m pip install -e '.[embeddings]' ``` -## Docs - -| | | -|---|---| -| [Concepts](docs/concepts.md) | Pipeline internals, model zoo, parity gate | -| [API reference](docs/api.md) | Every function, parameter, and return type | -| [Scan](docs/scan.md) | Inspect traffic before training | -| [Watch](docs/watch.md) | Record calls locally and configure generic exports | -| [JavaScript / Node.js](docs/javascript.md) | Full integration guide for JS pipelines | -| [Artifacts](docs/artifacts.md) | `.tracer/` directory schema | -| [Troubleshooting](docs/troubleshooting.md) | `selected_method=null`, coverage drift, embedding-dim mismatch | -| [AGENTS.md](AGENTS.md) | Integration guide for AI coding assistants | +`tracer.embed(texts)` defaults to `all-MiniLM-L6-v2`. Optional encoders can require +model downloads; `fit()` itself does not train or package the encoder. + +| Artifact | Purpose | +| --- | --- | +| `manifest.json` | Selected method, label space and final certification result | +| `pipeline.joblib` | Fitted students, acceptors and thresholds | +| `frontier.json` | Candidate-selection diagnostics, separate from final certification | +| `ood.json`, `ood_reference.npy` | Distance guard and its development-only reference, when fitted | +| `all_traces.jsonl`, `index.embeddings.npy`, optional `index.faiss` | Stored traces, embeddings and optional search index used by updates | +| `qualitative_report.json` | Per-label slices, examples and boundary pairs | +| `report.html` | Optional HTML report generated with `tracer.generate_html_report()` | + +Keep the complete artifact directory. If an expected OOD guard or reference is +missing or corrupt, loading fails instead of silently disabling it. Older +artifacts without final-certification metadata do not gain a certificate by +being loaded with newer code. See [artifacts](docs/artifacts.md) and +[troubleshooting](docs/troubleshooting.md). + +## Release status + +The final-policy certification, staged publication and strict OOD-loading +corrections described here are **unreleased source changes**. Installing +`tracer-llm` from PyPI installs the published package; it does not establish +that these corrections are present. The quickstart above installs from this +checkout. See the [change log](CHANGELOG.md) and [overview](docs/system1.md). + +## Documentation + +| Guide | Contents | +| --- | --- | +| [Overview](docs/system1.md) | Teacher/student architecture and scope | +| [Change log](CHANGELOG.md) | Source changes and release status | +| [Concepts](docs/concepts.md) | Students, acceptance policies and statistical limits | +| [API reference](docs/api.md) | Functions, configuration and return types | +| [Watch](docs/watch.md) | Local recording and optional exports | +| [Scan](docs/scan.md) | Inspect traffic before fitting | +| [JavaScript](docs/javascript.md) | Python integration from Node.js | +| [Artifacts](docs/artifacts.md) | Saved policy layout | +| [Troubleshooting](docs/troubleshooting.md) | Failed checks, data and embedding problems | +| [AGENTS.md](AGENTS.md) | Grounded integration instructions for coding assistants | ## Research diff --git a/benchmarks/README.md b/benchmarks/README.md index ee1108a..55e9a8b 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -14,6 +14,12 @@ OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 LOKY_MAX_CPU_COUNT=1 \ Measured with Python 3.12.13, NumPy 2.5.3, scikit-learn 1.9.1, and FAISS 1.15.1. The baseline is commit `0c918712c09a677feae80eede7069442c4e259a9`. +The **Fixed** column records the historical implementation at +`169edf886620f277555636aeb97be93991b1b849`. These measurements predate the +independent final-policy certification changes described in the +[change log](../CHANGELOG.md). They are preserved as regression evidence, not +measurements of the current final-certification protocol. No new benchmark run +is implied by this documentation update. | Case | Baseline | Fixed | |---|---:|---:| diff --git a/docs/api.md b/docs/api.md index 82e6da9..e329695 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1,12 +1,18 @@ # Python API Reference +This API builds and runs a task-specific classifier from +fixed-label classification traces. The student predicts labels directly; +acceptance and OOD rules select which answers to use. The caller owns teacher +execution on deferral. The OSS package has no CLI entry point, integrated hosted +model catalog, managed credentials, or Echo wallet. + ## `tracer.scan()` -Run the day-one scan: group traffic by similarity and measure, on a held-out -slice, how much is certifiably answerable by a near-free model at your target. -Returns a `ScanResult`; render it with `tracer.scanner.format_scan` (terminal) or -`tracer.scanner.scan_html` (HTML report). This is the conservative first look; -`tracer.fit()` trains the real router. +Group traffic by similarity and measure per-cell held-out teacher agreement. +Returns a `ScanResult`; render it with `tracer.scanner.format_scan` (text) or +`tracer.scanner.scan_html` (HTML). The scan is a diagnostic, not a trained +student, a final-policy certificate, or a net savings measurement. See the +[scan guide](scan.md) for the meaning of its historical `certifiable_share` field. ```python tracer.scan( @@ -17,6 +23,8 @@ tracer.scan( teacher_price_per_1k=None, monthly_calls=None, viz_layout="pca", + seed=7, + max_clusters=60, force=False, ) -> ScanResult ``` @@ -26,13 +34,15 @@ tracer.scan( | Name | Type | Default | Description | |------|------|---------|-------------| | `traces_path` | `str \| Path` | required | Path to traces JSONL file | -| `target` | `float` | `0.90` | Target label agreement to certify against | +| `target` | `float` | `0.90` | Threshold for per-cell teacher-agreement lower bounds | | `embeddings` | `np.ndarray \| None` | `None` | Precomputed embeddings `(n, dim)`; computed locally from text if omitted | | `model` | `str` | `"all-MiniLM-L6-v2"` | Local sentence-transformers model used when `embeddings` is omitted | | `teacher_price_per_1k` | `float \| None` | `None` | Teacher cost per 1k calls, to estimate savings | | `monthly_calls` | `int \| None` | `None` | Monthly volume, to project monthly savings | | `viz_layout` | `str` | `"pca"` | 3D layout for the HTML report (`pca`, `umap`, `tsne`, `auto`) | -| `force` | `bool` | `False` | Scan thin data anyway (coarsens grouping, reports a best-effort floor) | +| `seed` | `int` | `7` | Split and clustering seed | +| `max_clusters` | `int` | `60` | Maximum number of similarity cells | +| `force` | `bool` | `False` | Scan thin data with coarser grouping and an explicit warning; no production coverage floor is established | **Raises:** `tracer.scanner.ThinDataError` when fewer than 1,000 traces are passed and `force=False`. About 5,000 traces is the sweet spot. @@ -44,7 +54,9 @@ and `force=False`. About 5,000 traces is the sweet spot. ## `tracer.fit()` -Fit a routing policy from traces and embeddings. +Fit a student classifier and acceptance policy from traces and embeddings. The +encoder is not trained. Embeddings must match the trace rows in order, using the +same encoder and preprocessing that will be used at inference. ```python tracer.fit( @@ -61,7 +73,7 @@ tracer.fit( |------|------|---------|-------------| | `trace_path` | `str \| Path` | required | Path to traces JSONL file | | `artifact_dir` | `str \| Path` | `".tracer"` | Directory to save artifacts | -| `embeddings` | `np.ndarray \| None` | `None` | Precomputed embeddings `(n, dim)`. If None, auto-discovered from `.npy` | +| `embeddings` | `np.ndarray \| None` | `None` | Precomputed finite embeddings `(n, dim)`. If omitted, tries `.npy`, then `_embeddings.npy`; does not call an embedding model automatically | | `config` | `FitConfig \| None` | `None` | Fit configuration. Defaults to `FitConfig()` | **Returns:** `FitResult` @@ -74,6 +86,7 @@ result.manifest.teacher_agreement_cal # float, e.g. 0.950 result.manifest.n_traces # int result.manifest.label_space # list[str] result.manifest.embedding_dim # int +result.manifest.certification # final check status, counts and bound result.qualitative_report # QualitativeReport | None result.notes # list[str], human-readable notes result.artifact_dir # str @@ -92,10 +105,21 @@ result = tracer.fit( ) print(f"Method: {result.manifest.selected_method}") -print(f"Coverage: {result.manifest.coverage_cal:.1%}") -print(f"TA: {result.manifest.teacher_agreement_cal:.3f}") +print(f"Certification: {result.manifest.certification}") +if result.manifest.selected_method is not None: + print(f"Coverage: {result.manifest.coverage_cal:.1%}") + print(f"Teacher agreement: {result.manifest.teacher_agreement_cal:.3f}") ``` +The final check uses a reserved sample after the serving policy is fixed. Its +bound concerns teacher agreement on accepted traffic, under independent, +representative sampling; it is not ground-truth accuracy or per-request +certainty. A completed fit can return `selected_method=None`, leaving no student +to load. This result publishes a null-policy generation to `artifact_dir`. +Exceptions during fitting/publication preserve the preceding generation; +ordinary certification rejection is not such an exception. Use a separate +candidate directory when promotion must be explicit. See [concepts](concepts.md). + --- ## `tracer.update()` @@ -128,6 +152,10 @@ previous directory. If restoration itself fails, the exception identifies the retained backup. Keep one writer per artifact directory and reload readers only after `update()` returns: directory publication is not a concurrent-reader or power-loss transaction. Allow disk space for the staged generation and backup. +A completed update that fails its final check replaces the directory with a +null-policy generation. Coverage can rise or fall, and historical fit outcomes +do not certify a new update. Include representative traffic rather than only +old-policy deferrals; do not retune against repeatedly inspected final results. **Example:** @@ -136,7 +164,7 @@ result = tracer.update( "traces_day2.jsonl", new_embeddings=X_day2, ) -print(f"Coverage now: {result.manifest.coverage_cal:.1%}") +print(result.manifest.certification) ``` --- @@ -156,6 +184,10 @@ tracer.load_router(artifact_dir=".tracer", embedder=None) -> Router **Returns:** `Router` instance +Raises `ValueError` for a null selected policy or an invalid declared OOD guard; +missing/corrupt artifacts can also raise their file/serialization errors. Loading +a legacy artifact does not grant it the new final-policy certificate. + ```python # Without embedder (pass embeddings manually) router = tracer.load_router(".tracer") @@ -198,7 +230,12 @@ Embedder.from_endpoint( ) -> Embedder ``` -Calls an external HTTP embedding API. Default: sends one request per text with `{"input": "text"}`, expects `{"embedding": [...]}` back. Set `batch_key` to send all texts in one request. +Calls an external HTTP embedding API. Default: sends one request per text with +`{"input": "text"}`, expects `{"embedding": [...]}` back. Set `batch_key` to send +all texts in one request. Output keys are **literal top-level keys**, not dotted +JSON paths. The adapter does not add a provider's model parameter or parse its +nested response automatically. Use `from_callable()` for provider SDKs or a +custom response transform. ### `Embedder.from_callable()` @@ -223,12 +260,12 @@ from tracer import Embedder # sentence-transformers embedder = Embedder.from_sentence_transformers("BAAI/bge-small-en-v1.5") -# any standard embeddings endpoint +# Your endpoint must implement {"input": text} -> {"embedding": [...]} embedder = Embedder.from_endpoint( - "https://your-host/v1/embeddings", - headers={"Authorization": "Bearer sk-..."}, + "https://your-host/embed", + headers={"Authorization": "Bearer YOUR_ENDPOINT_KEY"}, input_key="input", - output_key="data.0.embedding", + output_key="embedding", ) # Custom function @@ -259,13 +296,20 @@ router.predict( ```python { - "label": str, # predicted class label + "label": str | None, # student label, fallback result, or None "decision": str, # "handled" or "deferred" - "accept_score": float, # acceptor confidence (0-1) - "stage": int, # which pipeline stage handled it + "accept_score": float, # acceptance ranking signal, not correctness probability + "stage": int, # handled stage index, or -1 when deferred } ``` +Both acceptance-rule rejection and OOD rejection return `"deferred"`; this API +does not expose a separate reason field. Without a fallback, a deferred result +has `label=None`, `accept_score=0.0`, and `stage=-1`. With a fallback, its return +value becomes `label` but the decision remains `"deferred"`. The callback is +called without arguments; validate its label and handle its exceptions in your +integration. It is not an automatic provider selection or generation API. + **Example:** ```python @@ -282,7 +326,7 @@ out = router.predict(embedding_vector) if out["decision"] == "handled": print(f"Surrogate: {out['label']} (score={out['accept_score']:.2f})") else: - print(f"Deferred to teacher: {out['label']}") + print("Student deferred; invoke your teacher if no fallback was supplied") ``` --- @@ -305,14 +349,17 @@ router.predict_batch(inputs) -> dict ```python { - "labels": list[str], # predicted labels for all inputs + "labels": list[str | None], # None for deferred inputs "decisions": list[str], # "handled" or "deferred" for each "handled": np.ndarray, # bool array, shape (n,) - "preds": np.ndarray, # label indices, shape (n,) - "stage_id": np.ndarray, # int array, shape (n,) + "preds": np.ndarray, # internal predictions; use only where handled + "stage_id": np.ndarray, # stage index, or -1 for deferred inputs } ``` +Batch prediction does not call a teacher. Use `handled` or `decisions` to select +which rows need a fallback; an internal prediction is not an accepted answer. + **Example:** ```python @@ -328,6 +375,32 @@ print(f"Handled: {n_handled}/{len(X_test)}") --- +## `tracer.serve()` + +Serve a saved policy over a lightweight local HTTP interface: + +```python +tracer.serve(artifact_dir=".tracer", host="127.0.0.1", port=8000) +``` + +This blocking call loads the policy before starting its server. It accepts +**embedding vectors**, not raw text, and never calls a teacher. + +| Method | Path | Input | Result | +| --- | --- | --- | --- | +| GET | `/health` | None | `status`, `method`, `coverage`, `teacher_agreement`, `n_labels`, `n_traces` | +| POST | `/predict` | `{"embedding": [0.1, 0.2]}` with the fitted width | Single prediction, including `stage` | +| POST | `/predict_batch` | `{"embeddings": [[0.1, 0.2]]}` with the fitted width | `labels`, `decisions`, boolean `handled` list | + +The shown vector widths are placeholders. `/health` does not return the +embedding dimension; read `manifest.embedding_dim`. The server validates body +size up to 8 MiB and gives HTTP 400 for malformed inputs, including dimension +errors. It supplies neither authentication nor TLS/CORS configuration. Keep it +on loopback or protect it through your application's network/proxy. It is not +an OpenAI/OpenRouter-compatible chat endpoint. See the [JS guide](javascript.md). + +--- + ## `tracer.report()` Load and return the policy manifest from an artifact directory. @@ -375,21 +448,25 @@ tracer.embed( **Returns:** `np.ndarray` of shape `(len(texts), dim)`, dtype `float32` -**Model recommendations:** +**Example model dimensions:** -| Model | Dim | Speed | Quality | -|-------|-----|-------|---------| -| `all-MiniLM-L6-v2` | 384 | ⚡⚡⚡ | good | -| `BAAI/bge-small-en-v1.5` | 384 | ⚡⚡⚡ | very good | -| `BAAI/bge-base-en-v1.5` | 768 | ⚡⚡ | excellent | -| `BAAI/bge-m3` | 1024 | ⚡ | excellent, multilingual | +| Model | Dim | +|-------|-----| +| `all-MiniLM-L6-v2` | 384 | +| `BAAI/bge-small-en-v1.5` | 384 | +| `BAAI/bge-base-en-v1.5` | 768 | +| `BAAI/bge-m3` | 1024 | + +These examples are not a ranking on your task. Measure quality, embedding cost +and full request latency with your traffic. Matching dimensions alone do not +make two embedding models interchangeable. **Example:** ```python import tracer, numpy as np -texts = ["What's my balance?", "Send $50 to Alice", ...] +texts = ["What's my balance?", "Send $50 to Alice"] X = tracer.embed(texts, model="BAAI/bge-small-en-v1.5") np.save("embeddings.npy", X) ``` @@ -423,7 +500,8 @@ path = tracer.generate_html_report(".tracer") print(f"Report at: {path}") import webbrowser -webbrowser.open(f"file://{path}") +from pathlib import Path +webbrowser.open(Path(path).resolve().as_uri()) ``` --- @@ -476,6 +554,7 @@ result.get_sankey(fmt="png") # static image ## `FitConfig` Configuration for `fit()` and `update()`. +The following is a field summary; instantiate it through `tracer.FitConfig`. ```python @dataclass @@ -484,18 +563,24 @@ class FitConfig: frontier_targets: tuple = (0.85, 0.90, 0.95) min_deploy_coverage: float = 0.05 max_fit_labels: int = 8_000 + embedding: EmbeddingConfig = field(default_factory=EmbeddingConfig) seed: int = 42 + certification_fraction: float = 0.20 + certification_alpha: float = 0.10 verbose: bool = True skip_candidates: tuple = () ``` | Field | Description | |-------|-------------| -| `target_teacher_agreement` | The parity bar. Surrogate must match teacher at least this often on handled traffic. | -| `frontier_targets` | Multiple targets to explore. The best (highest coverage) at `target_teacher_agreement` is selected. | +| `target_teacher_agreement` | Target for the one-sided lower bound on accepted teacher agreement, under independent representative sampling. | +| `frontier_targets` | Development-only targets to explore. The requested target is added if absent; only one selected policy receives the final check. | | `min_deploy_coverage` | Minimum coverage fraction to consider a method deployable. | | `max_fit_labels` | Subsample to this size for efficiency on large datasets (stratified). | +| `embedding` | Saved embedding configuration metadata. Does not make `fit()` compute embeddings or validate encoder identity at runtime. | | `seed` | Random seed for reproducibility. | +| `certification_fraction` | Fraction reserved before fitting or selection. The final policy is checked once on these examples. | +| `certification_alpha` | One-sided error level for final certification; default 0.10. No distribution-shift or repeated adaptive testing guarantee. | | `verbose` | Emit fitting progress to stderr. Set `False` for quiet runs. | | `skip_candidates` | Candidate names to exclude. Use `("dt", "rf", "et", "gbt", "xgb")` for a linear/neural sweep. An empty tuple keeps all available candidates. | @@ -503,10 +588,10 @@ class FitConfig: ```python config = tracer.FitConfig( - target_teacher_agreement=0.95, # 95% parity required + target_teacher_agreement=0.95, # target lower bound on accepted teacher agreement frontier_targets=(0.90, 0.95, 0.99), # explore these targets ) -result = tracer.fit("traces.jsonl", config=config) +result = tracer.fit("traces.jsonl", embeddings=X, config=config) ``` --- @@ -596,4 +681,13 @@ class ArtifactManifest: index_path: str | None config_path: str | None qualitative_report_path: str | None + certification: dict | None # final-policy evidence; absent on legacy artifacts + ood_required: bool # loading requires a valid saved guard ``` + +`coverage_cal` and `teacher_agreement_cal` contain final reserved-sample metrics +for new certified policies; historical artifacts retain their old calibration +meaning. Use `certification` to distinguish them. `QualitativeReport` describes +the supplied trace set, including development examples, rather than a separate +unseen evaluation. Examples throughout this reference illustrate API shape and +are not benchmark claims. diff --git a/docs/artifacts.md b/docs/artifacts.md index 3f4bf30..839eab0 100644 --- a/docs/artifacts.md +++ b/docs/artifacts.md @@ -1,6 +1,14 @@ -# Artifact Reference +# Artifact reference Every `tracer.fit()` or `tracer.update()` call writes a `.tracer/` directory. +It contains a task-specific student, its acceptance policy and evidence. The +student directly predicts a label; a deferred result leaves teacher execution +to the caller. See [concepts](concepts.md) for the training and serving contract. + +The numerical examples below illustrate schemas, not a measured benchmark or +promised coverage. Successful calls that publish no selected method still write +a manifest, but no `pipeline.joblib` for serving. Exceptions preserve the prior +generation; a completed certification rejection instead publishes a null policy. ## Directory layout @@ -13,8 +21,10 @@ Every `tracer.fit()` or `tracer.update()` call writes a `.tracer/` directory. all_traces.jsonl ← all traces accumulated so far (for update) index.embeddings.npy ← full embedding matrix (n × dim) index.faiss ← FAISS index, if faiss-cpu is installed + ood.json ← distance guard metadata, when fitted + ood_reference.npy ← development-only guard reference, when fitted qualitative_report.json ← XAI audit (slices, examples, boundary pairs) - report.html ← HTML report (auto-generated after fit) + report.html ← HTML audit report when generated ``` --- @@ -22,12 +32,14 @@ Every `tracer.fit()` or `tracer.update()` call writes a `.tracer/` directory. ## manifest.json The top-level summary. Human-readable and machine-readable. +The abbreviated example omits the full `certification` object; inspect the +stored object rather than inferring evidence from example metric values. ```json { "version": "0.1.0", "n_traces": 10003, - "label_space": ["card_arrival", "transfer_money", "check_balance", ...], + "label_space": ["card_arrival", "transfer_money", "check_balance"], "selected_method": "l2d", "target_teacher_agreement": 0.95, "coverage_cal": 0.928, @@ -45,18 +57,32 @@ The top-level summary. Human-readable and machine-readable. | Field | Meaning | |-------|---------| -| `selected_method` | Which pipeline was deployed: `"global"`, `"l2d"`, `"rsb"`, or `null` (parity gate blocked) | -| `coverage_cal` | Fraction of calibration-set traffic handled by surrogate. Proxy for production coverage. | -| `teacher_agreement_cal` | On calibration-set handled traffic, fraction where surrogate agreed with teacher. Should be ≥ `target_teacher_agreement`. | +| `selected_method` | Which pipeline was published: `"global"`, `"l2d"`, `"rsb"`, or `null` (selection/final check rejected) | +| `coverage_cal` | For new certified artifacts, fraction of final reserved examples handled by the complete runtime policy. Legacy artifacts retain their old calibration meaning. | +| `teacher_agreement_cal` | For new certified artifacts, observed teacher agreement on handled final reserved examples. | +| `certification` | Status, method, alpha, target, counts, coverage, agreement and one-sided lower bound from the final check. Missing/null on legacy artifacts. | +| `ood_required` | Whether loading requires a valid `ood.json` and `ood_reference.npy`. Defaults to false for legacy artifacts. | | `n_retrains` | Number of fits in this artifact's history: 1 after initial fitting, incremented by each successful `tracer.update()` | -**Null method:** If `selected_method` is `null`, coverage is 0%. All traffic routes to the teacher. This happens when no pipeline could reach the target teacher agreement on the calibration set. It's safe -- not an error. +**Null method:** If `selected_method` is `null`, there is no published local +policy and `load_router()` raises an error. The caller must keep using its +teacher. Inspect `certification.status` to distinguish selection failure, +insufficient final agreement evidence, and insufficient coverage. + +The certificate assumes independent, representative examples and no adaptive +reuse of reserved outcomes. It does not establish ground-truth or per-class +accuracy. A present but unreadable guard fails loading even for legacy artifacts. +Legacy guards without a separate reference retain their original stored-index +reference; they do not acquire final certification by being loaded. --- ## frontier.json -All candidate pipelines evaluated during fit, for every target TA in `frontier_targets`. +Candidate-selection diagnostics for each target in `frontier_targets`. Entries +are marked `evaluation_role: "development_selection_only"`. These examples were +used to choose the policy; their bounds are not final certification. Only the +selected policy at `target_teacher_agreement` is evaluated on the reserved data. ```json [ @@ -76,25 +102,29 @@ All candidate pipelines evaluated during fit, for every target TA in `frontier_t "best_method": "global", "best_coverage": 1.0, "best_ta": 0.921, - "candidates": [...] + "candidates": [] }, { "target": 0.95, "best_method": "l2d", "best_coverage": 0.928, "best_ta": 0.950, - "candidates": [...] + "candidates": [] } ] ``` -Use this to understand the coverage-quality tradeoff before committing to a target: +Use this to inspect the development coverage/agreement tradeoff. It does not +authorize a policy that failed its final check or establish production quality: ```python import json frontier = json.loads(open(".tracer/frontier.json").read()) for item in frontier: + if item["best_method"] is None: + print(f"target={item['target']:.0%}: no development candidate") + continue print(f"target={item['target']:.0%} " f"method={item['best_method']} " f"coverage={item['best_coverage']:.1%} " @@ -111,7 +141,9 @@ target=95% method=l2d coverage=92.8% TA=0.950 ## qualitative_report.json -The structured XAI report. Schema: +The structured audit report, computed on supplied traces including development +rows. It explains behavior; it is not an independent held-out accuracy report. +An `accept_score` ranks acceptance and is not a correctness probability. Schema: ```json { @@ -193,20 +225,27 @@ Internal structure (for reference): ```python { - "method": str, # "global", "l2d", or "rsb" "label_space": list[str], - "stages": [ - { - "surrogate": sklearn.Pipeline, # StandardScaler + classifier - "acceptor": sklearn.Pipeline | None, - "threshold": float | None, # calibrated accept threshold - "label_enc": LabelEncoder, - }, - # Stage 2 (RSB only) - ] + "pipeline": { + "method": str, # "global", "l2d", or "rsb" + "stages": [ + { + "clf": fitted_classifier, # sklearn estimator or pipeline + "acceptor": fitted_acceptor_or_none, + "accept_all": bool, + "threshold": float | None, + }, + # Second stage for RSB, when selected + ], + # Additional selection/summary fields may be present. + }, } ``` +This is a schematic: global stages set `accept_all=True` and omit acceptor and +threshold fields. Other stages carry a threshold and an optional logistic +acceptor. Labels are mapped through the bundle's shared `label_space`. + Load and inspect: ```python @@ -214,21 +253,32 @@ import joblib pipeline = joblib.load(".tracer/pipeline.joblib") stages = pipeline["pipeline"]["stages"] -surrogate = stages[0]["surrogate"] -print(type(surrogate.named_steps["clf"])) # e.g. LogisticRegression +student = stages[0]["clf"] +print(type(student)) # pipeline or standalone estimator, depending on candidate ``` --- ## Sharing artifacts -The `.tracer/` directory is fully self-contained. You can: +Copy the complete `.tracer/` directory together. It contains the fitted student +and policy data, but not external embedding-model weights or provider secrets. +Text inference still requires the same embedder and preprocessing; vector +inference requires matching vectors. Use compatible Python/sklearn/joblib +versions, and load only trusted joblib artifacts. You can: - Copy it to a server and `tracer.load_router("/path/to/.tracer")` -- Commit it to git (only `pipeline.joblib` and `index.embeddings.npy` are large) -- Pass it between teammates -- they only need `tracer-llm` installed +- Version or archive it in storage appropriate to your data +- Pass it between teammates with compatible runtime dependencies - Archive it with the traces for audit/reproducibility -Typical size: -- `pipeline.joblib`: 1-50 MB depending on model and label count -- `index.embeddings.npy`: `n_traces × dim × 4` bytes (e.g. 10k × 1024 → 40 MB) -- Everything else: < 1 MB +`all_traces.jsonl`, reports and embeddings can contain or reveal customer data; +artifact export is not automatic redaction. Review them before publishing. + +Storage depends on model and data sizes. Float32 `index.embeddings.npy` takes +approximately `n_traces × dim × 4` bytes plus its header (10k × 1024 is about +40.96 MB). The separate OOD reference, optional FAISS index, traces, reports and +classifier add to this; there is no fixed bundle-size ceiling. + +This is the OSS joblib format. The hosted app's individual JSON artifact +downloads are not a documented drop-in replacement for it. The OSS package +does not automatically import hosted model catalogs, credentials or wallets. diff --git a/docs/assets/training-inference.gif b/docs/assets/training-inference.gif new file mode 100644 index 0000000..c62b313 Binary files /dev/null and b/docs/assets/training-inference.gif differ diff --git a/docs/cold_start.md b/docs/cold_start.md index 69c1762..d2a02c9 100644 --- a/docs/cold_start.md +++ b/docs/cold_start.md @@ -1,155 +1,83 @@ -# Cold Start Guide +# Starting with a small trace dataset -Getting good results from TRACER requires enough traces to train a reliable surrogate and calibrate the parity gate. This guide covers what to expect when starting with a small trace dataset and how to bootstrap toward full coverage. +Choose the teacher-agreement target your application needs, then collect enough +representative traces to test it. Do not lower the target merely to make a model +publish. Dataset size alone does not determine attainable coverage. +The student learns to return your fixed labels directly. It needs examples of +your task; the OSS package does not ship a universal pretrained decision model +or automatically buy labels from a teacher. See [concepts](concepts.md). ---- +## How much evidence is needed? -## How many traces do you need? +By default, 20% of rows are reserved for final certification before any training +or model selection. On a fixed policy, at alpha 0.10, at least **22 accepted, +all-correct** reserved examples are needed for a 0.90 agreement lower bound. +A 0.95 target needs at least 45, and a 0.99 target at least 230. Errors require +more evidence. These are certification counts, not total training counts. -There is no hard minimum, TRACER will attempt to fit on whatever you provide. But the quality of the routing policy depends directly on how much data the surrogate and acceptor have to learn from. +The remaining rows must also support fitting, classifier selection and threshold +selection. Rare labels can be absent from one partition; a global agreement +bound is not a per-label guarantee. Collect additional examples and measure +per-label behavior separately when those distinctions matter. -**Rules of thumb:** +## Prepare the data -| Trace volume | What to expect | -|---|---| -| **< 50 total** | Fit will likely produce `selected_method: null` (no deployable policy). The train/val/cal split leaves too few samples per class for reliable calibration. | -| **50-200** | A global surrogate _may_ deploy if the classification task is simple (few classes, clean teacher labels). L2D and RSB are unlikely to calibrate well. | -| **200-500** | L2D starts becoming viable. Expect moderate coverage (40-70%) depending on class count and task difficulty. | -| **500-1,500** | The sweet spot for most tasks. All three pipeline families compete meaningfully, and coverage typically reaches 70-90%+. | -| **1,500+** | Diminishing returns on coverage, but useful for high-class-count tasks (50+ intents) or when you need very tight parity bars (≥ 0.98). | +Record the actual request and teacher label in JSONL: -These numbers assume reasonably separable embeddings. Poor embeddings can make even large datasets underperform. - -### Per-class minimums - -TRACER's `_split_buffer` performs a stratified train/val/cal split. The splitting logic in [`src/tracer/fit/pipeline.py`](../src/tracer/fit/pipeline.py) handles small classes as follows: - -- **1 sample in a class** → goes entirely to training (no validation or calibration for that class) -- **2 samples** → 1 train, 1 calibration -- **3 samples** → 1 train, 1 validation, 1 calibration -- **4+ samples** → 60/20/20 train/val/cal split - -This means classes with fewer than 4 traces are underrepresented in calibration, which can degrade parity guarantees for those classes. - -**Recommendation:** aim for at least **10 traces per class** before fitting, and at least **5 classes** with 20+ traces for reliable acceptor training. - ---- - -## Bootstrapping traces from scratch - -If you're evaluating TRACER for a new classification pipeline and don't yet have production LLM traces, here's how to build up a trace dataset: - -### Step 1: Seed with synthetic or sampled inputs - -Pick 100-300 representative inputs for your classification task. These can come from: - -- A test set you already have -- Manually written examples covering each class -- A sample of recent production inputs (if available) - -### Step 2: Generate teacher labels - -Run each input through your LLM classification pipeline (the "teacher") and record the output: - -```python -import json - -traces = [] -for text in seed_inputs: - label = call_your_llm(text) # your existing LLM pipeline - traces.append({"input": text, "teacher": label}) - -with open("traces.jsonl", "w") as f: - for t in traces: - f.write(json.dumps(t) + "\n") +```jsonl +{"input": "What is my balance?", "teacher": "check_balance"} +{"input": "Send money to Alice", "teacher": "transfer_money"} ``` -### Step 3: Compute embeddings +Compute embeddings with the same model you will use during serving: ```python import tracer -import numpy as np - -# Option A: sentence-transformers (local) -X = tracer.embed([t["input"] for t in traces], model="all-MiniLM-L6-v2") -np.save("traces.npy", X) - -# Option B: precomputed from your own embedding model -# Save as traces.npy next to traces.jsonl -``` +from tracer.traces.loader import load_traces -### Step 4: Fit with relaxed settings - -For cold-start, lower the agreement bar and minimum coverage so a policy can deploy: - -```python +texts = [record.input_text for record in load_traces("traces.jsonl").records] +X = tracer.embed(texts, model="all-MiniLM-L6-v2") result = tracer.fit( "traces.jsonl", embeddings=X, - config=tracer.FitConfig( - target_teacher_agreement=0.85, # relaxed from default 0.90 - min_deploy_coverage=0.02, # accept even low coverage initially - ), + config=tracer.FitConfig(target_teacher_agreement=0.90), ) +print(result.manifest.certification) ``` -### Step 5: Grow with `tracer.update()` - -As your LLM handles more production traffic, append new traces and re-fit: - -```python -# After collecting more traces -result = tracer.update("new_traces.jsonl", ".tracer", new_embeddings=X_new) -``` - -Each update round adds deferred inputs (the ones the surrogate wasn't confident about) back into the training pool, automatically expanding coverage over time. This is TRACER's core feedback loop. +This example requires `pip install 'tracer-llm[embeddings]'`. It runs the +embedding model locally; model download and local computation are separate +from the classical student fit. Fitting does not fine-tune that encoder. ---- +Synthetic examples can help develop a classifier, but success on synthetic +reserved data does not establish production quality. Repeated turns from one +session and duplicate inputs also violate the independent-sample assumptions. +Keep external session/time holdouts for production evaluation. -## Diagnosing cold-start issues +## If no policy is published -### `selected_method: null` after fitting +Check `manifest.json` first. `selected_method: null` means `load_router()` will +refuse to serve a policy. The final certification status distinguishes no +candidate, failed agreement evidence, and insufficient accepted coverage. +A completed fit with this result replaces the selected artifact directory with +a null-policy generation. Keep an existing serving generation in a separate +directory if you want to review a candidate before switching traffic. -This means no pipeline met your parity bar. Before collecting more data, check `.tracer/frontier.json`: - -```bash -cat .tracer/frontier.json -``` +`frontier.json` shows development-only candidate diagnostics. It does not show +the best achievable production result, and its apparent success must not be used +to override a failed final check. Collect more representative data, inspect +teacher-label errors, or investigate representation quality. Do not repeatedly +choose thresholds or random seeds by looking at final certification outcomes. -Each entry shows the best achievable coverage and teacher agreement per target. If `best_ta` is close to your target but `best_coverage` is below `min_deploy_coverage`, try: +## Updating ```python -config = tracer.FitConfig( - target_teacher_agreement=0.85, # lower the bar - min_deploy_coverage=0.01, # accept minimal coverage -) +result = tracer.update("new_traces.jsonl", new_embeddings=X_new) ``` -If `best_ta` is well below your target even at zero coverage, the problem is likely: - -1. **Embeddings don't separate classes**, try a stronger embedding model -2. **Teacher label noise**, audit traces where the surrogate disagrees -3. **Too few traces per class**, collect more data for underrepresented classes - -### Low coverage despite many traces - -If coverage plateaus well below 80% with 500+ traces: - -- Check class imbalance: a few dominant classes might be well-covered while rare classes drag the average down -- Try the RSB pipeline family explicitly, it's designed for multi-stage residual routing -- Improve embedding quality: switching from `all-MiniLM-L6-v2` to `all-mpnet-base-v2` can significantly improve class separation - -### Coverage varies between fits - -See the [troubleshooting guide](troubleshooting.md#coverage-drops-between-fits) for details on why re-fits may show different coverage numbers. - ---- - -## Summary: cold-start checklist - -- [ ] Collect at least **50 traces** (ideally 200+) with **≥ 10 per class** -- [ ] Use a good embedding model (sentence-transformers `all-MiniLM-L6-v2` is a solid default) -- [ ] Start with a **relaxed** `target_teacher_agreement` (0.85) and `min_deploy_coverage` (0.02) -- [ ] Check `frontier.json` after each fit to understand what's achievable -- [ ] Use `tracer.update()` to incrementally grow coverage as you collect more production traces -- [ ] Tighten the parity bar once coverage is stable and you have 500+ traces +An update refits and rechecks a policy; coverage can rise or fall. Include a +representative sample of all traffic, not only old-policy deferrals, if the +measured coverage is intended to describe overall traffic. The same concerns +about adaptive holdout reuse apply across repeated updates. +Teacher agreement is not ground-truth accuracy. Check important labels and +worst-case workflows on independent evaluation data before deployment. diff --git a/docs/concepts.md b/docs/concepts.md index 2cf19d2..0c42c48 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -1,153 +1,115 @@ -# How TRACER Works - -**TRACER** = **T**race-Based **A**daptive **C**ost-**E**fficient **R**outing - -## The core idea - -You're running an LLM to classify text (intents, harm categories, topics, ...). Most inputs are "easy" -- a cheap ML model could get them right. But you don't know *which* inputs are easy until you try, and you don't want silent quality degradation. - -TRACER learns that boundary from your LLM's own outputs. It finds which inputs the LLM handles predictably, builds a fast local surrogate for those, and defers the rest back to the LLM. Coverage grows as you collect more traces. - ---- - -## The pipeline - -``` -Input text - │ - ▼ -[Embedding] ← frozen, precomputed - │ - ▼ -[Surrogate model] ← trained on teacher labels - │ predict_proba(x) - ▼ -[Acceptor gate] ← predicts: will surrogate agree with teacher? - │ - ├── accept_score ≥ τ → return surrogate prediction (FREE) - └── accept_score < τ → defer to teacher LLM (PAID) -``` - -### The surrogate - -A classifier trained on the teacher's outputs (not ground-truth labels). The surrogate learns to imitate the teacher, not to solve the task from scratch. Simple models (logistic regression, MLP, random forest) work well because the embedding already captures the semantics. - -### The acceptor - -A second binary classifier that predicts: "will the surrogate's prediction match the teacher on this input?" It's trained on the same data: features are the surrogate's confidence signals (top-1 prob, top-2 prob, margin, entropy). A high acceptor score means "the surrogate is confident and consistent with the teacher here." - -### Threshold calibration - -The acceptor threshold τ is calibrated on a held-out set (not used for training). The calibration objective: find the threshold that maximizes coverage subject to teacher agreement ≥ target. This is a conformal-style guarantee: on the calibration distribution, you get the promised agreement rate. The approach follows the post-hoc deferral estimator framework of Narasimhan et al. ([ICML 2022](https://proceedings.mlr.press/v162/narasimhan22a.html)). - ---- - -## Three pipeline families - -TRACER fits three candidate architectures and picks the one with highest coverage at your target teacher agreement. The L2D and RSB families are grounded in the Learn-to-Defer literature (Mozannar & Sontag, [ICML 2020](https://proceedings.mlr.press/v119/mozannar20b.html); Madras et al., [NeurIPS 2018](https://papers.nips.cc/paper_files/paper/2018/hash/09d37c08f7b129e96277388757530c72-Abstract.html); Mao et al., [NeurIPS 2023](https://proceedings.neurips.cc/paper_files/paper/2023/hash/7e1bf09d2e4e6e5c5a0b63b66b47ef42-Abstract.html)). - -### Global - -``` -[Surrogate] → accept all -``` - -The simplest case. If the surrogate achieves the target teacher agreement on everything (no gating needed), use it. This gives 100% coverage when it works. - -### L2D (Learn to Defer) - -``` -[Surrogate] → [Acceptor gate] → LOCAL or LLM -``` - -The standard case. The acceptor gates each sample individually. High-confidence inputs are handled by the surrogate; uncertain ones are deferred. Selected for most tasks. - -### RSB (Residual Surrogate Boosting) - -``` -[Surrogate 1] → [Acceptor 1] → LOCAL (easy traffic) - ↓ - [Surrogate 2] → [Acceptor 2] → LOCAL (medium traffic) - ↓ - LLM (hard traffic) -``` - -A two-stage cascade. Stage 2 is trained specifically on the inputs that Stage 1 deferred -- the residuals. This gives Stage 2 a specialized view of the "hard" cases and can push coverage higher. Only used when there are enough residual training samples (≥50). - ---- - -## Model zoo - -The surrogate model is selected by validation F1 from a pool of candidates. Tree-based models don't use StandardScaler; linear/neural models do. - -| Model | Family | When it wins | -|-------|--------|-------------| -| `logreg_c1` | Linear | High-dim, linearly separable embeddings | -| `logreg_c10` | Linear | Same, with tighter regularization | -| `sgd_log` | Linear | Very large datasets (faster than logreg) | -| `mlp_1h` | Neural | Non-linear boundaries, medium data | -| `mlp_2h` | Neural | Complex boundaries, larger data | -| `dt` | Tree | When decision boundaries are axis-aligned | -| `rf` | Tree | Robust ensemble baseline | -| `et` | Tree | Faster than RF, often similar quality | -| `gbt` | Tree | Slower, strong on small datasets (≤4k) | -| `xgb` | Tree | Boosted, optional (install `tracer-llm[xgboost]`) | - ---- - -## Data splits - -``` -All traces (n) - │ - ├── Subsample to max_fit_labels (8000) if n > 8000 [stratified] - │ - ├── 60% Training → fit surrogate, fit acceptor - ├── 20% Validation → select best surrogate by val F1 - └── 20% Calibration → calibrate acceptor threshold τ -``` - -Splits are stratified by label. Classes with fewer than 3 samples use a fallback to ensure representation. - ---- - -## The parity gate - -Before any surrogate handles live traffic, TRACER checks: does it actually meet the target teacher agreement on a held-out evaluation? This is the parity gate. - -If the calibration teacher agreement < target → `selected_method = None`, coverage = 0%. Traffic continues 100% to the teacher. No silent degradation. - -The parity gate prevents a scenario where a surrogate looks good on training data but fails on real traffic. The held-out calibration set is the backstop. - ---- - -## Continual learning - -Every `tracer.update()` call: -1. Merges new traces with all historical traces -2. Refits from scratch (all three pipeline families) -3. Re-selects best on frontier -4. Re-calibrates threshold -5. Re-checks parity gate - -Coverage grows monotonically as more traces accumulate. Easy intents (tight embedding clusters) become self-serve first; ambiguous ones may never fully cross the parity bar. - ---- - -## Qualitative report - -Every fit produces a structured explanation of the routing policy. The slice and boundary-pair design is informed by Slice Finder (Chung et al., [ICDE 2019](https://ieeexplore.ieee.org/document/8731353)) and the contrastive explanation literature (Miller, [AIJ 2019](https://www.sciencedirect.com/science/article/pii/S0004370217301126)). - -**Slice summaries**: per-label and per-length-bucket handled/deferred rates. Answers: "which parts of my label space are well-covered?" - -**Contrastive boundary pairs**: for each label, one example that was handled and one that was deferred. Answers: "what makes an input 'easy' vs 'hard' for the surrogate?" - -**Representative examples**: the most typical handled and deferred inputs (by embedding position). Answers: "what does typical handled/deferred traffic look like?" - -**Temporal deltas**: how per-label handled rates changed between the previous fit and this one. Answers: "what's improving and what's plateauing?" - ---- - -## Paper - -The published paper is **TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification** by Adam Rida, available on [arXiv](https://arxiv.org/abs/2604.14531). It reports experiments on Banking77, CLINC150, and MNLI, including the empirical parity gate, interpretability artifacts, and limitations. +# How TRACER works + +TRACER learns a task-specific classifier from teacher-labelled traces. This +local student acts as a System 1 path: it predicts labels directly using +classical ML or a shallow neural classifier. An acceptance +rule and distance guard determine which student answers the caller can use. +For other inputs, the caller keeps using its teacher. This is fixed-label +classification, not free-form generation or zero-shot selection among models. + +Embedding, local inference, and fallback calls all have costs; savings and +latency must be measured on the intended workload. See the +[overview](system1.md) for scope and the [API](api.md) for contracts. + +## Training and policy selection + +The input embeddings are precomputed. TRACER does not fine-tune the encoder. +The supplied embedding model must remain the same at fit and inference time. + +First, `fit()` reserves a random 20% of rows for final certification. These rows +are excluded from classifier training, threshold selection, OOD reference data, +and candidate ranking. This split is not stratified, so its label proportions +are not deliberately rebalanced. + +On the remaining development data, TRACER: + +1. Optionally subsamples for fitting, using the configured `max_fit_labels`. +2. Splits the fit buffer approximately 60/20/20 into training, validation and + threshold-selection data, with small-class exceptions. +3. Selects a surrogate by validation macro-F1 against teacher labels. +4. Trains an acceptor on validation correctness, using top-1 probability, top-2 + probability, their margin and normalized entropy. +5. Searches acceptance thresholds on development calibration data. +6. Selects the candidate with highest development coverage at the requested + agreement target, breaking ties by agreement and then fewer stages. + +The acceptor is logistic regression. If its correctness targets have only one +class, the classifier's maximum probability is used as a ranking signal. +`accept_score` is not a certified probability that an individual answer is right. +The teacher can also be wrong: matching its label and matching an independently +established ground-truth label are different evaluation targets. + +## Candidate families + +- **Global:** one surrogate accepts all inputs before the distance guard. +- **L2D:** one surrogate and an acceptance threshold. +- **RSB:** a two-stage cascade. The second surrogate is trained on examples the + first stage rejects, when enough residual training and selection data exist. + +Each stage chooses from logistic regression, SGD logistic, shallow MLPs, +decision trees, random forests, ExtraTrees, gradient boosting on smaller fit +buffers, and optional XGBoost. Logistic `C=10` means **less** regularization than +`C=1`. See `src/tracer/fit/surrogate.py` for the configured candidates. + +`frontier.json` records this adaptive selection process. Its bounds and metrics +are development diagnostics; they are not final certification. + +## Distance guard and final check + +The distance guard compares a query's mean distance to its nearest development +embeddings against fitted global/per-label thresholds. It is a heuristic for +unfamiliar inputs, not a proof of distribution-shift detection. Its default +quantile is 99.5%. The reference embeddings are saved separately in +`ood_reference.npy`; the full dataset remains stored for future updates. + +After selecting one policy, TRACER freezes its classifiers, acceptance +thresholds and distance guard. It then evaluates this exact serving path **once** +on the reserved rows. It counts accepted examples and teacher matches and +computes a one-sided Clopper–Pearson lower bound. The policy is published only +if this bound clears `target_teacher_agreement` and measured coverage clears +`min_deploy_coverage`. There is no retry of other candidates on those outcomes. +Zero accepted examples cannot pass the check. + +At inference, ordinary rejection and an OOD distance-guard rejection both return +`decision="deferred"`. The public result does not expose a separate OOD reason. +Without a fallback callback, no teacher is called and `label` is `None`. With a +callback, its result fills `label` but the decision remains `"deferred"`. Batch +prediction and the bundled HTTP server never execute that callback for you. + +The defaults are `certification_fraction=0.20` and `certification_alpha=0.10`. +For a fixed policy under independent, identically distributed sampling, the +bound has at least 90% one-sided coverage at that alpha. It bounds agreement +with the teacher on accepted traffic, not agreement with ground truth. + +This statement does **not** cover correlated sessions, duplicates across data +partitions, arbitrary future distribution shift, per-class quality, or adaptive +repeated attempts after inspecting reserved outcomes. Random row splitting does +not solve session leakage. Evaluate session/time/workflow holdouts separately, +and collect representative traffic rather than only old-policy deferrals. + +Small datasets and strict targets may produce no published policy even when +every reserved prediction is correct. At alpha 0.10, at least 22 accepted, +all-correct examples are needed for a 0.90 lower bound. More errors or a higher +target require more evidence. + +## Artifacts, updates and reports + +`manifest.certification` contains the final sample counts, coverage, agreement, +lower bound and result. The legacy names `coverage_cal` and +`teacher_agreement_cal` now report final certification measurements for new +artifacts. Older artifacts keep their original meaning and have no certificate. +They remain loadable; loading does not retroactively certify them. + +`fit()` and `update()` stage complete artifact generations before publishing. +A fitting or publication exception preserves the old generation. A completed +fit that fails certification instead publishes a null-policy manifest; it does +not keep the old policy active on disk. Use a separate candidate directory when +you need explicit promotion. Use one writer per directory and reload readers +after publication. A required distance guard +that is missing or unreadable prevents loading; it is not silently disabled. + +Updates can increase or decrease coverage. The qualitative report describes +routing on the supplied traces, including development rows, so it is not an +independent quality estimate. Use the final certificate and external evaluation +for that purpose. A pre-training `scan()` reports cluster diagnostics and does +not certify the learned serving policy. diff --git a/docs/javascript.md b/docs/javascript.md index 97e4b42..226def1 100644 --- a/docs/javascript.md +++ b/docs/javascript.md @@ -1,6 +1,17 @@ # Using TRACER from JavaScript / Node.js -TRACER is a Python package, but it integrates cleanly into any JS/Node.js pipeline through its Python API and a self-hosted HTTP server. No Python goes into your application code. +TRACER trains a fixed-label student in Python. Your Node application can use its +direct decisions through the bundled HTTP sidecar and call your teacher when +the policy defers. The JS package records traces; it does not train or execute +the classifier. See [concepts](concepts.md) for the scope and quality contract. +Your teacher can be an LLM, a foundation model for System 1 decisions, or an +existing classifier. It supplies validated labels for the task; the student can answer +accepted requests directly, while your application sends deferred requests to +that teacher. + +This guide uses Python scripts, not a CLI. It does not connect to a hosted +catalog, Tracer account or Echo wallet. Provider SDK calls below belong to your +application and use your provider credentials and billing. --- @@ -8,7 +19,7 @@ TRACER is a Python package, but it integrates cleanly into any JS/Node.js pipeli ### 1. Collect traces from your JS pipeline -Every time your LLM classifies an input, append the result to a JSONL file: +When your teacher classifies an input, append the result to a JSONL file: ```js import fs from 'fs' @@ -18,20 +29,22 @@ function logTrace(input, label) { fs.appendFileSync('traces.jsonl', line + '\n') } -// After every LLM classification: -const label = await callYourLLM(userInput) +// After a teacher classification: +const label = await callYourTeacher(userInput) logTrace(userInput, label) ``` -Each line must have `input` (the text) and `teacher` (the label your LLM returned). That's all TRACER needs. +Each line must have `input` (the text) and `teacher` (the label your teacher returned). These are the required trace fields. --- ### 2. Compute embeddings (offline, once before fit) -The HTTP server at inference time expects embedding vectors, so you need to embed your traces before fitting. Use the same embedding model you plan to use at inference time, this is the only constraint. +The HTTP server expects embedding vectors. Embed traces in exactly their input +row order, and use the same encoder, revision and normalization at inference. +Matching dimensions alone does not establish compatibility. -**Option A: OpenAI embeddings (natural fit for JS pipelines)** +**Option A: your existing embedding API (OpenAI SDK example)** ```bash pip install tracer-llm openai numpy @@ -44,15 +57,23 @@ from openai import OpenAI client = OpenAI() texts = [json.loads(l)["input"] for l in open("traces.jsonl")] -response = client.embeddings.create(model="text-embedding-3-small", input=texts) -X = np.array([d.embedding for d in response.data]) +vectors = [] +for start in range(0, len(texts), 64): + response = client.embeddings.create( + model="text-embedding-3-small", input=texts[start:start + 64] + ) + vectors.extend(d.embedding for d in sorted(response.data, key=lambda d: d.index)) +X = np.asarray(vectors, dtype=np.float32) np.save("traces.npy", X) # TRACER auto-discovers this at fit time ``` -**Option B: Local embeddings (sentence-transformers, free)** +These calls use your provider account. Check its input/token limits; the batch +size is illustrative and does not make arbitrary-length traces admissible. + +**Option B: Local embeddings (sentence-transformers, no API charge)** ```bash -pip install tracer-llm[embeddings] +pip install 'tracer-llm[embeddings]' ``` ```python @@ -72,14 +93,22 @@ np.save("traces.npy", X) # fit_policy.py import tracer -tracer.fit( +result = tracer.fit( "traces.jsonl", config=tracer.FitConfig(target_teacher_agreement=0.95), ) +print(result.manifest.certification) +if result.manifest.selected_method is None: + raise SystemExit("No student published; keep the teacher path active.") ``` TRACER reads `traces.jsonl` and auto-discovers `traces.npy` (same stem, `.npy` extension). Run `python fit_policy.py` offline, in a cron job, a GitHub Action, or manually. It does not touch your application. +The final reserved-sample check concerns teacher agreement, not ground-truth +accuracy. A completed failed check publishes a null policy; use a separate +candidate directory before promotion if a student is already serving. Embedding, +training and local serving still consume resources. + --- ### 4. Start the HTTP server @@ -103,12 +132,21 @@ access, configure authentication, TLS, and CORS at your reverse proxy. ## Predicting from your JS app -At inference time, embed the input with the same model you used at fit time, then POST the embedding to TRACER. If the surrogate handles it, you get the label back immediately with no LLM call. If it defers, you call your LLM as usual and log the new trace. +At inference time, embed the input with the same model you used at fit time, +then POST the embedding to TRACER. If the student handles it, use its label +without a teacher call. If it defers, call your teacher and log the new trace. **With OpenAI embeddings:** +Install your application's SDK with `npm install openai`. The example assumes +`callYourTeacher(text)` returns a validated string from your fixed label set and +`logTrace()` is the function above. + ```js +import OpenAI from 'openai' + const openai = new OpenAI() +const tracerUrl = process.env.TRACER_URL ?? 'http://localhost:8000' async function route(text) { // 1. Embed the input (same model as at fit time) @@ -119,25 +157,33 @@ async function route(text) { const embedding = embResponse.data[0].embedding // 2. Ask TRACER whether to handle locally or defer - const res = await fetch('http://localhost:8000/predict', { + const res = await fetch(`${tracerUrl}/predict`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ embedding }), }) + if (!res.ok) throw new Error(`TRACER request failed: ${res.status}`) const { label, decision } = await res.json() if (decision === 'handled') { - return label // surrogate answered, no LLM call + return label // student answered, no teacher call } - // Deferred: call your LLM and log the new trace for the next refit - const llmLabel = await callYourLLM(text) - logTrace(text, llmLabel) - return llmLabel + if (decision !== 'deferred') throw new Error('Unexpected TRACER decision') + + // Deferred: call your teacher. A separate sampling policy should also + // collect teacher labels on representative accepted traffic for evaluation. + const teacherLabel = await callYourTeacher(text) + logTrace(text, teacherLabel) + return teacherLabel } ``` -The embedding API call is cheap (fractions of a cent). The TRACER HTTP call is local and sub-millisecond. You only pay the full LLM cost on deferred inputs. +This skips teacher execution for accepted inputs. Measure the complete path: +embedding price and latency, local serving, network overhead and teacher calls. +No sub-millisecond or fixed savings claim follows from this example. Serving +errors above propagate explicitly; any retry/fallback-on-error policy is yours +to define and account for. **Batch prediction:** @@ -145,10 +191,11 @@ The embedding API call is cheap (fractions of a cent). The TRACER HTTP call is l const res = await fetch('http://localhost:8000/predict_batch', { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ embeddings: [embedding1, embedding2, ...] }), + body: JSON.stringify({ embeddings: [embedding1, embedding2] }), }) +if (!res.ok) throw new Error(`TRACER request failed: ${res.status}`) const { labels, decisions, handled } = await res.json() -// handled: boolean[]: true means surrogate answered, no LLM needed +// handled: boolean[]: true means student answered, no teacher call needed ``` --- @@ -157,26 +204,40 @@ const { labels, decisions, handled } = await res.json() | Method | Path | Body | Response | |--------|------|------|----------| -| `GET` | `/health` |, | `{"status": "ok", "method", "coverage", ...}` | -| `POST` | `/predict` | `{"embedding": [float, ...]}` | `{"label", "decision", "accept_score"}` | +| `GET` | `/health` | None | `status`, `method`, `coverage`, `teacher_agreement`, `n_labels`, `n_traces` | +| `POST` | `/predict` | `{"embedding": [float, ...]}` | `label`, `decision`, `accept_score`, `stage` | | `POST` | `/predict_batch` | `{"embeddings": [[float, ...], ...]}` | `{"labels", "decisions", "handled"}` | -`decision` is `"handled"` (surrogate answered, no LLM call needed) or `"deferred"` (call your LLM). +`decision` is `"handled"` (student answered, no teacher call needed) or +`"deferred"` (your application can call its teacher). +Deferred labels are `null`. Rejection by the acceptance rule and by the OOD +distance guard share this public state; there is no separate reason field. +`accept_score` is a ranking signal, not certified per-request correctness. +The server does not run a teacher for either single or batch requests, and is +not an OpenAI/OpenRouter-compatible chat-completion endpoint. --- ## Continual learning -Every deferred input that reaches your LLM is a new trace. Accumulate them and retrain periodically, coverage grows with each refit. +Teacher-labeled requests can supply new training traces. Keep a representative +sample of all traffic, including teacher checks on accepted requests; using +only old-policy deferrals changes the training/evaluation distribution. Do not +turn unchecked student predictions into teacher labels. ```python # Run this script on a schedule; supply matching embeddings or a sibling .npy. import tracer -tracer.update("new_traces.jsonl", new_embeddings=X_new) +result = tracer.update("new_traces.jsonl", new_embeddings=X_new) +print(result.manifest.certification) ``` -Then restart the `serve_policy.py` process to pick up the updated policy. Coverage typically grows from ~84% at day 1 to 90%+ within a week of production traffic. +Inspect the returned certification result before switching serving artifacts. +Restart the sidecar or reload your application router after publication. Updates +can increase or decrease coverage, or publish no policy. There is no automatic +retraining schedule or promised week-one gain in the package. Avoid repeated +tuning against reserved results; retain independent session/time evaluation. --- @@ -217,10 +278,10 @@ Your Node app reads `process.env.TRACER_URL` and routes through it. Replace the | Step | Where | Frequency | |------|-------|-----------| -| Collect traces | Your JS app | Every LLM call | +| Collect traces | Your JS app | Teacher classification calls | | Embed traces | Python script (offline) | Before each fit | -| Fit policy | `tracer.fit()` in a Python script | On a schedule | -| Serve predictions | `tracer.serve()` in a Python sidecar | Always-on | +| Fit policy | `tracer.fit()` in a Python script | When you explicitly run it | +| Serve predictions | `tracer.serve()` in a Python sidecar | While your service runs | | Embed input at inference | JS (same model/API) | Every prediction | | POST to TRACER | Your JS app | Every prediction | diff --git a/docs/scan.md b/docs/scan.md index 8a20ffa..7a8e862 100644 --- a/docs/scan.md +++ b/docs/scan.md @@ -4,6 +4,9 @@ handle before you spend time training one. It groups embeddings by similarity and measures label agreement on a held-out split. It does not produce a routing policy; use `tracer.fit()` for that. +The learned student can use different decision boundaries from these clusters, +so a scan is neither a bound on achievable student coverage nor a final-policy +certificate. See [the fitting contract](concepts.md). ```python from pathlib import Path @@ -30,7 +33,7 @@ must match the remaining rows in order. | Parameter | Default | Meaning | | --- | --- | --- | -| `target` | `0.90` | Minimum held-out agreement bound for a certifiable cell | +| `target` | `0.90` | Threshold applied to each cell's held-out teacher-agreement lower bound | | `embeddings` | `None` | Precomputed NumPy array of shape `(n, dim)` | | `model` | `all-MiniLM-L6-v2` | Local embedding model used when no array is supplied; requires the `embeddings` extra | | `teacher_price_per_1k` | `None` | Teacher cost per 1,000 calls for the savings estimate | @@ -44,7 +47,18 @@ At least 1,000 usable traces are required by default; around 5,000 is recommende `force=True` coarsens the clusters to increase held-out evidence per cell and marks the result as forced. It does not relax the statistical bound. -The result includes the certifiable share, per-cell examples and bounds, a -frontier across agreement targets, and optional savings estimates. The HTML -report includes a 3D embedding map. Savings are projections at the supplied cost -and volume, not measured production outcomes. +The result's historical `certifiable_share` field is the held-out traffic share +in cells whose individual bounds clear the target. Clusters and majority labels +are fitted on 70% of rows; the remaining 30% supplies the diagnostic counts. +The scan uses `alpha=max(0.01, 1-target)`, unlike `fit()`'s separately configured +final check. It does not correct for simultaneous selection across cells or +certify the quality of their selected union. Representative independent data +remain necessary; a row split does not remove correlated-session leakage. + +The result includes per-cell examples and bounds, a frontier across targets, +and optional cost projections. The HTML report includes a 3D embedding map. +`savings_per_1k_calls` is simply `certifiable_share * teacher_price_per_1k`; +monthly savings multiply that by `monthly_calls / 1000`. These are gross avoided +teacher-cost illustrations, not measured net savings. They exclude embeddings, +training, serving, storage and changes in request mix. Teacher agreement also +does not measure ground-truth accuracy. diff --git a/docs/system1.md b/docs/system1.md new file mode 100644 index 0000000..f6d1930 --- /dev/null +++ b/docs/system1.md @@ -0,0 +1,155 @@ +# Learning from classification traces + +TRACER (Trace-Based Adaptive Cost-Efficient Routing) turns a teacher's traces +into task-specific ML students that handle easy, repetitive decisions. +The students and teacher form a hybrid architecture with calibrated deferral: +the teacher handles requests the students decline. It can be an LLM, a +foundation model built for System 1 decisions, or another classifier. + +The student predicts labels directly. Its learned acceptance rule and a +distance-based out-of-distribution (OOD) guard decide whether to use its answer. +Intent classification, tagging and selection from a stable set of tools are +suitable task shapes. Fit and evaluate the system on representative examples +of your own workload. + +Student fitting and inference can run on CPU. Training learns the classifier +and its acceptance policy from labeled examples; it does not fine-tune the LLM, +teacher model or semantic encoder. + +## The teacher and student + +The **teacher** supplies the reference labels in your traces and can answer +deferred requests through your application. It can be an LLM, a foundation +model for System 1 decisions, or another classifier that produces the required +labels. Your existing provider integration can supply its outputs. +The teacher's outputs define the agreement target, so check their quality before +using them as training labels. + +The **student** learns to predict those labels from embeddings. The candidate +families include linear classifiers, trees, and shallow neural classifiers. +Classical ML can make the accepted decisions; it is not used only to choose an LLM. +The semantic encoder is supplied separately and must match between fitting and +inference. Fitting the student does not fine-tune that encoder or the teacher. + +```text +Teacher-labeled traces + matching embeddings + | + fit and select a student + | + check the fixed serving policy + | + deploy + +New input -> same embedder -> student + acceptance rule + OOD guard + | | + handled locally deferred + | | + student label caller's teacher +``` + +Residual stages can try additional students before deferral. Your teacher is +called only if you provide a fallback function or implement the deferred path in +your application. TRACER does not choose or bill a teacher provider for you. + +## What a result means + +The current OSS single-request API returns: + +```python +{ + "label": "check_balance", + "decision": "handled", + "accept_score": 0.96, + "stage": 0, +} +``` + +- `handled` means a student answered through the fitted policy. +- `deferred` means the student did not handle the request. This includes OOD + rejections. Without a fallback, `label` is `None`; with a fallback, it contains + the fallback's return value and the decision remains `deferred`. +- `accept_score` is a ranking signal used by the acceptance rule. It is not a + calibrated probability that this answer is correct. + +The OSS response does **not** currently expose `OOD` as a third decision value or +provide a rejection-reason field. Do not infer OOD from every deferred result. +Batch prediction returns labels and decisions; your application handles fallback +for the deferred rows. See the [API reference](api.md). + +## Choose quality before measuring savings + +Set `FitConfig(target_teacher_agreement=...)` to define the required agreement +on accepted traffic. In the current source, fitting reserves 20% of rows before +training or selecting candidates. The frozen final policy, including its OOD +guard, is evaluated once on that partition. Its one-sided Clopper-Pearson lower +bound must reach the target. The default `certification_alpha=0.10` specifies +the 90% one-sided confidence level of that check; it is separate from the +agreement target. + +This check assumes independent, representative examples. The OSS uses a random +row split; it does not enforce session/time separation or deduplicate inputs. +Repeatedly tuning against the reserved outcomes invalidates the interpretation. +Teacher agreement is also different from correctness against independently +verified labels. The check makes no universal promise about future traffic, +each class, or each request. + +Before deploying, inspect `result.manifest.selected_method` and +`result.manifest.certification`. A completed fit that fails the final check can +publish a manifest with `selected_method=None`; that directory cannot be loaded +as a deployed router. To keep a live policy while assessing a replacement, fit +to a separate artifact directory, evaluate it, then promote it yourself. + +At minimum, measure on representative held-out traffic: + +| Measure | What it answers | +| --- | --- | +| Accepted coverage | What fraction does the student answer directly? | +| Accepted teacher agreement | How often do those labels match the teacher? | +| Ground-truth accuracy, macro-F1 and per-class recall | Does the system solve the actual task, including minority classes? | +| Deferred and OOD behavior | Which inputs need the teacher or further investigation? | +| Complete latency and cost | Are encoding, student serving and teacher calls together worthwhile? | + +Do not count every deferral as correct: measure the teacher's actual answer when +evaluating the complete system. The scan and qualitative reports help inspect +data, but they are not substitutes for a final production evaluation. + +## Free OSS and the hosted Tracer app + +The MIT-licensed core remains usable without a Tracer account. You can record +traces, fit and inspect students, run predictions, refit with new data, and serve +the model yourself. You provide the embedding model and teacher integration; +local compute and any provider usage remain your responsibility. + +The [hosted Tracer app](https://app.tracerml.ai) is a separate managed service. +Its accounts, provider integrations and billing are not dependencies of this +package. The application changes below are under development and have their +own release cycle. + +| Area | This OSS repository | Hosted application release work | +| --- | --- | --- | +| Training | Python `fit()` and `update()` on your traces | Managed jobs and versioned datasets | +| Teacher | Caller-supplied fallback | Configured teacher and strict JSON label contract | +| Serving | Python API; HTTP sidecar accepts embeddings | Authenticated text-input endpoints and tenant management | +| Trace capture | Local Python/JS wrappers; opt-in generic export | Stored shadow and routed traffic | +| OOD | Distance guard; public outcome remains `deferred` | Separate ML/deferred/OOD analytics | +| Accounts and funds | No Tracer account or wallet required | Shared Echo workspaces, credits and payment integration | +| Artifacts | Local `.tracer/` directory containing `pipeline.joblib` | Managed artifacts use a different runtime format | + +Copying a local `.tracer/` directory supports OSS deployment with a compatible +Python environment and the same embedder. A hosted artifact download is not yet +a tested drop-in OSS bundle; a conversion and round-trip path is still needed. +Only load artifacts from sources you trust: `joblib` is executable serialization. + +Use the Python API and HTTP sidecar for local integration. There is currently +no command-line entry point. The package contains the fitting and serving +library; you supply the traces and encoder for your task. + +## Source and release status + +The [changelog](../CHANGELOG.md) distinguishes repository changes from tagged +package releases. Check the installed version before relying on a documented +certification or artifact behavior. + +Continue with the [quickstart](../README.md#quickstart-from-this-source), +[policy concepts](concepts.md), [artifact contract](artifacts.md), or +[troubleshooting](troubleshooting.md). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index ca14051..c168265 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,93 +1,104 @@ # Troubleshooting -Three snags people hit most often, what causes them, and how to fix each. - -## `selected_method` is `null` (no policy deployed) - -**Symptom.** After `tracer.fit(...)`, the manifest shows `selected_method: null`, -the `FitResult` notes say "No deployable pipeline met the target teacher-parity -threshold", and `tracer.load_router(...)` then raises `FileNotFoundError` on -`pipeline.joblib` (the file is only written when a policy actually deploys). -The server started with `tracer.serve()` reports `method=none` on `/health`. - -**Why.** TRACER only deploys a surrogate that can hold your parity bar. If no -candidate reaches `target_teacher_agreement` while still covering at least -`min_deploy_coverage` of traffic, nothing is deployed on purpose, rather than -shipping a policy that silently disagrees with your teacher. - -Common reasons: - -- `target_teacher_agreement` set too high for the data (for example 0.99 on - noisy teacher labels). -- `min_deploy_coverage` set so high that the only parity-respecting policies are - filtered out. -- Too few traces, or embeddings that do not separate the classes. -- Heavy teacher label noise, which caps the agreement any surrogate can reach. - -**Fix.** - -1. Open `.tracer/frontier.json`. Each entry lists `best_coverage` and `best_ta` - per target. This tells you the best parity actually achievable and at what - coverage, so you can pick a realistic bar instead of guessing. -2. Lower the bar to something the frontier shows is reachable: - ```python - tracer.fit("traces.jsonl", embeddings=X, - config=tracer.FitConfig(target_teacher_agreement=0.90, - min_deploy_coverage=0.02)) - ``` -3. Add more traces, or improve the embedding model so classes separate better. -4. If `best_ta` is capped well below your target across all candidates, suspect - teacher label noise and audit the traces before re-fitting. +TRACER's student returns fixed labels directly when its policy accepts them. +Otherwise your application retains its teacher path. These diagnostics concern +that policy; an acceptance score is not a certified correctness probability. +See [concepts](concepts.md) and [the API contract](api.md). + +## `selected_method` is `null` + +**Symptom:** `fit()` or `update()` returns a manifest with no selected method. +`load_router()` raises `ValueError`; `serve()` fails during loading, before its +HTTP server starts. There is no functioning `/health` endpoint for that policy. + +**Meaning:** no candidate was selected, or the fixed policy failed its final +teacher-agreement/coverage check. A policy can fail with no observed mistakes +when too few reserved examples were accepted. The target bounds agreement with +the teacher, not independent ground-truth accuracy. + +1. Inspect `result.manifest.certification` and `result.notes` for the exact + status, accepted counts, observed agreement and lower bound. +2. Use `frontier.json` to understand development selection. Its scores are not + final certification or proof of the best achievable production result. +3. Audit labels and collect representative independent examples. Check whether + the representation separates the labels your application needs. +4. Keep using the teacher. Do not repeatedly change thresholds or random seeds + against the reserved outcomes to obtain a passing certificate. A different + quality requirement is an application decision, not a repair for failure. + +A completed null-policy fit **replaces** the selected directory's generation; +it is different from an exception, which preserves the preceding generation. +To review a candidate before promoting it, fit to a separate directory: + +```python +import tracer + +candidate = tracer.fit("traces.jsonl", artifact_dir=".tracer-candidate", embeddings=X) +print(candidate.manifest.certification) +# Switch your application's configured directory only after your review/evals. +``` ## Coverage drops between fits -**Symptom.** A re-fit (`tracer.update(...)`, or `tracer.fit` on a larger trace -set) reports a lower `coverage_cal` than the previous run at the same -`target_teacher_agreement`. - -**Why.** This is expected, not a regression. TRACER holds parity fixed and lets -coverage float. When new traffic is harder, more diverse, noisier, or introduces -labels the surrogate has not seen, the only way to keep matching the teacher at -your target is to defer more inputs. Coverage falls so that the parity bound still holds. +A lower accepted fraction can reflect harder traffic, changed label proportions, +sampling variation, different selected models, or a regression. It is not by +itself proof that the system adapted correctly or that quality was preserved. -**Fix.** Usually nothing: a lower coverage at the same parity is the system -adapting correctly. To understand the change: +Compare old and new policies on the same external session/time holdout. Report +coverage and accepted agreement together; if independent reference labels are +available, also measure direct accuracy and per-class/worst-group failures. +Use the qualitative report and its per-label `temporal_deltas` for diagnosis, +not as an independent quality estimate. Record a representative sample of all +traffic: training only on the old policy's deferrals changes the distribution. -- Check `.tracer/frontier.json` to see the new coverage versus parity curve. -- Read the qualitative report (`.tracer/qualitative_report.json`). Its - `temporal_deltas` show, per label, how the handled rate moved between fits, so - you can see which slices got harder. -- If new labels expanded the space, that alone can lower coverage; confirm the - label count grew. -- If cost matters more than parity for the new traffic, lower - `target_teacher_agreement` to trade some agreement back for coverage. +Updates refit and recheck a policy; there is no promised coverage improvement or +automatic production retraining schedule in the OSS package. -## Embedding dimension mismatch +## Embedding dimension mismatch or unexpected behavior -**Symptom.** -``` +```text ValueError: Embedding dimension mismatch: expected 384, got 768. ``` -raised from `router.predict(...)` / `router.predict_batch(...)`, or surfaced as -a `500` from the server's `/predict` endpoint. - -**Why.** The router records the embedding width it was fitted on in -`manifest.embedding_dim` and checks every incoming vector against it. A mismatch -means the embeddings at route time were produced differently from the ones used -at fit time, almost always a different embedder model. - -**Fix.** Use the same embedder at fit time and route time. - -- If you fit on precomputed embeddings, route with the same model that produced - them. `all-MiniLM-L6-v2` and `BAAI/bge-small-en-v1.5` are 384-dim, while - `BAAI/bge-base-en-v1.5` is 768-dim; fitting on one width and routing on the - other triggers this error. - ```python - embedder = tracer.Embedder.from_sentence_transformers("BAAI/bge-small-en-v1.5") - router = tracer.load_router(".tracer", embedder=embedder) # same model as fit - ``` -- For the HTTP server, the vector in `{"embedding": [...]}` must have exactly - `manifest.embedding_dim` values. Check `GET /health` and the manifest if you - are unsure of the expected width. -- A dimension match with poor accuracy is a different problem: make sure the - `normalize` setting also matches between fit and route. + +`predict()` and `predict_batch()` check width against `manifest.embedding_dim`. +The HTTP server returns **400**, not 500, for this input error. Use the exact +same encoder, revision, preprocessing and normalization as during fitting. +Two encoders can share a dimension and still be incompatible; dimension checks +do not validate model identity. + +```python +embedder = tracer.Embedder.from_sentence_transformers("BAAI/bge-small-en-v1.5") +router = tracer.load_router(".tracer", embedder=embedder) # only if fitted with this encoder +``` + +The HTTP interface takes `{"embedding": [...]}`, not raw text. It does not +compute embeddings or call your fallback. Read the dimension from the manifest; +`GET /health` does not expose it. `Embedder.from_endpoint()` uses literal +top-level response keys, so `"data.0.embedding"` is not a nested-path adapter. +Use `from_callable()` to transform a provider's nested response. + +## A request is deferred, including an unfamiliar input + +Ordinary acceptance rejection and the OOD distance guard both return +`decision="deferred"`. There is no separate public OOD-reason field. +Without a supplied fallback, the result has `label=None`. With one, its return +value becomes the label but the decision stays `"deferred"`. An OOD guard is a +heuristic distance check, not a guarantee that all wrong inputs are detected. + +Batch results have `None` for deferred labels and never run a teacher. Ignore +internal `preds` where `handled` is false. Calling a fallback does not establish +that its answer is correct; evaluate its outputs separately when reporting +end-to-end quality. + +## A saved policy no longer loads + +A missing or unreadable declared OOD guard fails loading instead of silently +disabling it. Restore the matching complete artifact generation; do not delete +the guard to make loading succeed. Legacy artifacts without final evidence do +not become certified when reloaded. + +`update()` needs `all_traces.jsonl` and its aligned saved embeddings. If either +is lost, reconstruct from your source records and refit; embeddings alone cannot +recover teacher labels. Use one artifact writer at a time, keep enough disk +space for staged generations, and reload readers after publication finishes. +See [artifacts](artifacts.md). diff --git a/docs/watch.md b/docs/watch.md index e955afd..bd5627a 100644 --- a/docs/watch.md +++ b/docs/watch.md @@ -1,9 +1,11 @@ -# `tracer.watch`: observability +# `tracer.watch`: recording classification traces -Record every LLM call in your pipeline as an OpenTelemetry GenAI span. Local-first -(nothing leaves your machine by default), standards-native (the open `gen_ai.*` -OpenTelemetry GenAI schema), and the same object that feeds -`tracer.fit()` / `tracer.scan()`. +Record wrapped model calls as spans with OpenTelemetry GenAI-aligned attributes. +Recording is local by default. Classification outputs can then be converted to +teacher-labeled examples for your student classifier. The watcher +does not train, route, generate labels, or automatically upload to Tracer Cloud. +General chat/tool traces are observable too, but are not automatically suitable +for fixed-label student training. ## Quick start @@ -45,30 +47,38 @@ account, no network. ## What gets captured -Each span follows the OpenTelemetry GenAI conventions (`gen_ai.*`): +The span can store the following fields; capture depends on the response shape +or fields explicitly supplied by your integration: - input + output messages (roles, multi-turn), and a flat input/output text - request + response model, token counts (prompt / completion / total) - request params (temperature, top_p, max_tokens, stop, seed, …) - tool / function calls (name, arguments, result) -- finish reason, cost, latency (+ time-to-first-token for streams) +- finish reason, cost, latency (time-to-first-token only when supplied) - status / error, timestamps - trace id, span id, parent span id (nested trace tree) - conversation / session id, user id, tags, arbitrary metadata `record(response)` auto-extracts the model, tokens, finish reason, tool calls, and output text from common provider response objects (object- or dict-shaped); -it never raises, so it is safe on the hot path. +recording failures are contained. This is not a zero-overhead claim: local file +writes are synchronous, and the Python HTTP sink waits for its request or +timeout. Measure that overhead or supply your own buffered sink if needed. ## Sinks | Sink | What it does | Turn on with | |------|--------------|--------------| -| `LocalFileSink` | JSONL to `.tracer/watch/` (default) | always on | -| `OTLPSink` | fan out to any OTLP/HTTP backend | `TRACER_WATCH_OTLP_ENDPOINT` [+ `_HEADERS`] | +| `LocalFileSink` | JSONL to `.tracer/watch/` | default unless a custom sink replaces it | +| `OTLPSink` | POST the package's GenAI-aligned JSON payload to your endpoint | `TRACER_WATCH_OTLP_ENDPOINT` [+ `_HEADERS`] | | `MultiSink` | several at once (local + your own backend) | set more than one of the above | Pass a custom `sink=` to `watch()` to fully control export. +Despite its historical name, `OTLPSink` sends a single JSON object with `name`, +trace/span IDs and `attributes`, not the standard OTLP `resourceSpans` envelope. +Use an endpoint that accepts this shape or your own translating sink; arbitrary +OTLP collectors are not guaranteed to ingest it unchanged. No shared hosted +credentials, Echo account, or billing integration is configured by the watcher. ## Environment variables @@ -122,13 +132,20 @@ source = Path(".tracer/watch/support_classifier.jsonl") with Path("traces.jsonl").open("w", encoding="utf-8") as output: for line in source.read_text(encoding="utf-8").splitlines(): span = GenAISpan(**json.loads(line)) + if span.status != "ok" or not span.input_text or not span.output_text: + continue record = span.to_trace_record() output.write(json.dumps({ "input": record.input_text, "teacher": record.teacher_label, + "metadata": record.metadata, }) + "\n") ``` Use calls whose outputs are classification labels, then compute embeddings for -the same rows and call `tracer.fit("traces.jsonl", embeddings=X)`. See the -[concepts guide](concepts.md). +the same rows and call `tracer.fit("traces.jsonl", embeddings=X)`. Validate the +label set first: a JSON response such as `{"intent":"billing"}` needs explicit +label extraction; the conversion does not interpret arbitrary response text. +Keep session metadata to construct external holdouts; core fitting still splits +by row, not session. Recorded inputs/outputs can contain private data and are not +automatically redacted. See the [concepts guide](concepts.md). diff --git a/js/README.md b/js/README.md index 14432c1..f8f8b70 100644 --- a/js/README.md +++ b/js/README.md @@ -1,17 +1,21 @@ # @tracer-llm/watch -Local-first, OpenTelemetry-aligned LLM trace recording for JavaScript / TypeScript. +Local-first, OpenTelemetry GenAI-aligned trace recording for JavaScript / TypeScript. +Classification traces can become training data for a TRACER student: +a small model that predicts fixed labels directly +and leaves other inputs to your teacher. This package records calls only; +fitting and routing use the [Python package or HTTP sidecar](../docs/javascript.md). Watch any model call in your pipeline with a wrapper, a method decorator, or an async span. No key, no account, nothing leaves your machine by default: traces are appended to `./.tracer/watch/.jsonl`. -- Standards-native schema: every watched call is recorded as a span following - the OpenTelemetry GenAI semantic conventions (`gen_ai.*`), so a record is - portable to any OTel-aware backend. No proprietary schema. +- GenAI-aligned attributes (`gen_ai.*`) alongside the package's local span + fields. Validate the payload contract when connecting a collector. - Zero runtime dependencies. Node stdlib + the global `fetch` only. -- Prod-safe: telemetry never throws into, or adds latency to, the host call. - Generic HTTP exports are sent asynchronously; export failures are swallowed. +- Recording failures are contained. Local writes use synchronous filesystem + calls and add overhead; HTTP exports are asynchronous and best-effort. There + is no zero-latency or durable-delivery guarantee. ## Install @@ -82,13 +86,21 @@ via `parentSpanId`, forming a trace tree (tracked with `AsyncLocalStorage`). | Sink | What it does | Turned on by | | ----------------- | ----------------------------------------------------------------------- | -------------------------------------------------------- | -| `LocalFileSink` | Append spans as JSONL to `/.jsonl`. No network, no key. | Default, always on. | -| `OTLPSink` | POST an OTel GenAI-shaped payload to any OTLP/HTTP backend. | `TRACER_WATCH_OTLP_ENDPOINT` (+ `TRACER_WATCH_OTLP_HEADERS`). | +| `LocalFileSink` | Append spans as JSONL to `/.jsonl`. No network, no key. | Default unless replaced by a custom sink. | +| `OTLPSink` | POST the package's GenAI-aligned JSON payload to your endpoint. | `TRACER_WATCH_OTLP_ENDPOINT` (+ `TRACER_WATCH_OTLP_HEADERS`). | | `MultiSink` | Fan-out to several sinks at once. | Composed automatically when more than one is configured. | `sinkFromEnv(name)` composes local files with the generic HTTP sink when `TRACER_WATCH_OTLP_ENDPOINT` is explicitly configured. Pass `sink` to `watch()` for a custom exporter. There is no default remote destination. +Despite its historical name, `OTLPSink` does not emit the standard OTLP +`resourceSpans` envelope. It sends one JSON object with `name`, trace/span IDs, +timestamps and `attributes`. Use a compatible endpoint or a translating sink; +arbitrary OTLP collectors are not guaranteed to accept it unchanged. + +There is no automatic hosted account, model catalog, credit wallet or cloud +training integration. The local JSONL contains recorded inputs and outputs; +redaction and any sharing policy belong to the application. ## Provider response auto-extraction @@ -109,6 +121,19 @@ Both objects and plain dicts are handled. | `TRACER_WATCH_OTLP_HEADERS` | Comma-separated `k=v` headers for the OTLP sink. | (unset) | | `TRACER_WATCH_DEBUG` | Print why a telemetry send was dropped (never affects the host).| (unset) | +## From recording to a student + +Select successful fixed-label classification calls, extract the validated label +from each output, and write `{"input":"...","teacher":"label"}` JSONL. +Provider response text is not automatically parsed into your label schema. +Compute matching embeddings and use Python `tracer.fit()`; inspect the final +teacher-agreement certificate before serving. Keep session metadata for separate +holdouts and include representative traffic, not just deferred requests. General +chat or tool trajectories do not automatically become classification examples. + +Teacher agreement is distinct from ground-truth accuracy. No coverage, savings, +or per-request correctness guarantee comes from installing this recorder. + ## License Apache-2.0 diff --git a/js/src/index.ts b/js/src/index.ts index d9b07fd..3794b52 100644 --- a/js/src/index.ts +++ b/js/src/index.ts @@ -16,8 +16,8 @@ * JSONL. `OTLPSink` exports spans to an explicitly configured endpoint over * OTLP/HTTP to any OTLP/HTTP backend, zero code change. * - * Zero runtime dependencies (Node stdlib only). Prod-safe: telemetry never - * throws into, or adds latency to, the host call. + * Zero runtime dependencies (Node stdlib only). Telemetry failures are isolated + * from the host call; recording and export still add runtime work. */ import { AsyncLocalStorage } from "node:async_hooks"; diff --git a/llms.txt b/llms.txt index 69f988d..ef85599 100644 --- a/llms.txt +++ b/llms.txt @@ -1,11 +1,19 @@ -# TRACER - tracer-llm +# TRACER - Distill your LLM traces into fast ML System 1 models. > **TRACER** = Trace-Based Adaptive Cost-Efficient Routing. -> Turn LLM classification traces into routing policies. Surrogate handles easy traffic; teacher LLM handles hard. Formal quality guarantees. +> Turn teacher traces into task-specific ML students for easy, repetitive decisions. The hybrid architecture uses calibrated deferral: the teacher handles requests the students decline. The teacher may be an LLM, a foundation model for System 1 decisions, or another classifier. The student predicts labels directly from embeddings; the caller implements teacher fallback. Student fitting and inference can run on CPU. A separate final check measures accepted teacher agreement. Coverage and savings depend on the data; confidence bounds assume independent, representative examples without adaptive holdout reuse. + +This is a free MIT Python library for fixed-label tasks, not a pretrained general +model listed on OpenRouter. No Tracer account is required. The encoder and teacher +are caller-supplied; fitting the student does not fine-tune either. The current +source has no CLI. Hosted app accounts, credits and management are separate. +CHANGELOG.md distinguishes unreleased source from published packages. ## Install Requires Python 3.12+. CI tests Python 3.12, 3.13, and 3.14. -pip install tracer-llm +Published package: pip install tracer-llm +Current checkout with the unreleased corrections: python -m pip install -e . +The published package may not contain the source changes documented here. ## Core API (4 functions) tracer.fit("traces.jsonl", embeddings=X) → FitResult @@ -13,6 +21,13 @@ tracer.load_router(".tracer", embedder=e) → Router router.predict("text" or embedding) → {label, decision, accept_score} tracer.update("new.jsonl", new_embeddings=X_new) → FitResult +decision is "handled" or "deferred"; OOD is included in "deferred", not a third +public value. accept_score is a ranking signal, not a correctness probability. +Check manifest.selected_method and manifest.certification before loading. +Default final holdout fraction=.20; certification_alpha=.10; target agreement=.90. +Fit into a separate directory to evaluate a replacement before promotion: a +completed noncertifying fit publishes selected_method=null in its target directory. + ## Trace format (JSONL, one per line) {"input": "user text", "teacher": "label_from_llm"} @@ -22,8 +37,9 @@ Embedder.from_endpoint("https://api/embed", headers={...}) Embedder.from_callable(fn) ## Docs -- AGENTS.md - full integration guide for AI agents +- AGENTS.md - integration notes - README.md - user-facing docs +- docs/system1.md - teacher/student overview, OSS/cloud scope and release status - docs/api.md - Python API reference - docs/scan.md - pre-training diagnostics - docs/watch.md - local trace recording and generic exports diff --git a/pyproject.toml b/pyproject.toml index 6344e68..54a1a97 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,13 +5,14 @@ build-backend = "hatchling.build" [project] name = "tracer-llm" dynamic = ["version"] -description = "TRACER: Trace-Based Adaptive Cost-Efficient Routing. Turn LLM traces into parity-gated routing policies - cut 90%+ of LLM calls with formal guarantees." +description = "Distill LLM traces into ML System 1 models with calibrated teacher deferral." readme = "README.md" license = {file = "LICENSE"} requires-python = ">=3.12" keywords = [ "llm", "routing", "surrogate", "cost-reduction", "learn-to-defer", - "conformal-prediction", "machine-learning", "explainability", "xai", + "selective-classification", "machine-learning", "explainability", "xai", + "system-1", "teacher-student", ] classifiers = [ "Development Status :: 4 - Beta", diff --git a/src/tracer/__init__.py b/src/tracer/__init__.py index d069060..8eab3ad 100644 --- a/src/tracer/__init__.py +++ b/src/tracer/__init__.py @@ -1,7 +1,8 @@ -"""TRACER -- Trace-Based Adaptive Cost-Efficient Routing. +"""TRACER -- classification from teacher traces. -Turn LLM traces into parity-gated routing policies that progressively -replace teacher calls -- with formal guarantees and qualitative audit. +Train a task-specific student to predict labels directly, with teacher deferral +and an OOD guard. Final teacher-agreement checks assume representative, +independent examples. TRACER: Trace-Based Adaptive Cost-Efficient Routing. """ from __future__ import annotations @@ -15,8 +16,8 @@ # Lazy public API (PEP 562). The heavy ML stack (numpy / scikit-learn / # hdbscan / sentence-transformers) is only imported on first attribute access, -# so `import tracer` and local trace recording start instantly instead of -# paying multi-second import cost up front. +# so `import tracer` and local trace recording avoid loading the ML stack +# until it is needed. # # `fit` also collides with the tracer.fit subpackage, so it cannot be healed by # __getattr__ alone (the submodule attribute shadows it). tracer.api re-asserts diff --git a/src/tracer/api.py b/src/tracer/api.py index c7c5f78..fa45c66 100644 --- a/src/tracer/api.py +++ b/src/tracer/api.py @@ -26,6 +26,7 @@ from tracer.fit.pipeline import ( evaluate_pipeline, fit_frontier, route_pipeline, apply_stage, _accept_scores, _predict, ) +from tracer.fit.certification import certification_split, certify_policy from tracer.policy.artifacts import ( load_manifest, load_pipeline, save_pipeline, save_qualitative_report, write_manifest, ) @@ -39,6 +40,70 @@ def fit( artifact_dir: Union[str, Path] = ".tracer", embeddings: Optional[np.ndarray] = None, config: Optional[FitConfig] = None, +) -> FitResult: + """Fit and publish a complete artifact generation. + + A fitting or publication exception preserves the existing generation. + A completed noncertifying fit publishes a nondeployable manifest instead. + Use one writer per artifact directory and reload readers after return; + fit to a separate directory to evaluate a replacement before promotion. + """ + artifact_dir = Path(artifact_dir) + artifact_dir.absolute().parent.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(prefix=f".{artifact_dir.name}-fit-", + dir=artifact_dir.absolute().parent) as temp: + root = Path(temp) + staged = root / "next" + if artifact_dir.exists(): + shutil.copytree(artifact_dir, staged) + else: + staged.mkdir() + _clear_model_artifacts(staged) + result = _fit_artifacts(trace_path, staged, embeddings, config) + result.artifact_dir = str(artifact_dir) + for field_name in ("pipeline_path", "index_path", "config_path", "qualitative_report_path"): + path = getattr(result.manifest, field_name) + if path is not None: + setattr(result.manifest, field_name, str(artifact_dir / Path(path).relative_to(staged))) + write_manifest(staged / "manifest.json", result.manifest) + _publish_generation(staged, artifact_dir, root, result.notes) + return result + + +def _clear_model_artifacts(directory): + for filename in ("pipeline.joblib", "qualitative_report.json", "ood.json", + "ood_reference.npy", "index.faiss", "report.html", "sankey.html"): + (directory / filename).unlink(missing_ok=True) + + +def _publish_generation(staged, artifact_dir, root, notes): + # The backup is outside the temporary root: a failed rollback must not + # cause TemporaryDirectory cleanup to discard the previous generation. + backup = root.with_name(root.name + "-previous") + had_previous = artifact_dir.exists() + if had_previous: + artifact_dir.rename(backup) + try: + staged.rename(artifact_dir) + except BaseException: + if had_previous: + try: + backup.rename(artifact_dir) + except OSError as exc: + raise RuntimeError(f"Previous artifacts are preserved at {backup}; restore them before retrying") from exc + raise + if had_previous: + try: + shutil.rmtree(backup) + except OSError: + notes.append(f"Fit completed; previous artifact backup retained at {backup}") + + +def _fit_artifacts( + trace_path: Union[str, Path], + artifact_dir: Union[str, Path], + embeddings: Optional[np.ndarray] = None, + config: Optional[FitConfig] = None, ) -> FitResult: """Fit a TRACER routing policy from teacher traces. @@ -61,8 +126,6 @@ def fit( notes = [] dataset = load_traces(trace_path) - labels = sorted(dataset.label_space) - label_to_idx = {l: i for i, l in enumerate(labels)} # Resolve embeddings if embeddings is None: @@ -77,7 +140,13 @@ def fit( f"No embeddings found. Pass embeddings= or place a .npy file at {emb_path}") X = _embedding_matrix(embeddings, len(dataset)) - y_teacher = np.array([label_to_idx[r.teacher_label] for r in dataset.records], dtype=int) + # Even label discovery must not inspect reserved outcomes: a class seen + # only there is an unknown label and counts as a mismatch if accepted. + development, certification_rows = certification_split( + len(X), config.certification_fraction, config.seed) + labels = sorted({dataset.records[i].teacher_label for i in development}) + label_to_idx = {l: i for i, l in enumerate(labels)} + y_teacher = np.array([label_to_idx.get(r.teacher_label, -1) for r in dataset.records], dtype=int) y_true = None has_gt = all(r.ground_truth is not None for r in dataset.records) if has_gt: @@ -87,7 +156,11 @@ def fit( y_true = None has_gt = False - # Fit frontier + # Reserve final certification before any label balancing or fitting. The + # candidate-selection code never sees these rows or their outcomes. + X_dev, y_dev = X[development], y_teacher[development] + + # Fit candidate frontier using development data only. targets = list(config.frontier_targets) if config.target_teacher_agreement not in targets: targets.append(config.target_teacher_agreement) @@ -99,7 +172,7 @@ def log_fn(msg: str) -> None: # noqa: E301, local helper elapsed = time.perf_counter() - _t0 print(f"[tracer.fit +{elapsed:6.1f}s] {msg}", file=sys.stderr, flush=True) - frontier, split = fit_frontier(X, y_teacher, targets, + frontier, split = fit_frontier(X_dev, y_dev, targets, max_fit_labels=config.max_fit_labels, min_coverage=config.min_deploy_coverage, log=log_fn, skip=config.skip_candidates, seed=config.seed) @@ -117,18 +190,49 @@ def log_fn(msg: str) -> None: # noqa: E301, local helper ta_cal = None qual_report = None qual_path = None + certificate = {"status": "no_candidate", "n_examples": len(certification_rows), + "alpha": config.certification_alpha, + "target": config.target_teacher_agreement} + serving_router = None + ood_gate = None if selected and selected["best"] and selected["best"]["stages"]: + # Freeze all policy choices, including OOD, before looking at the + # reserved outcomes. There is no fallback to another candidate if this + # one fails final certification. + best = selected["best"] + from tracer.fit.ood import fit_ood_gate + from types import SimpleNamespace + dev_preds, _, _ = route_pipeline(best["stages"], X_dev) + pred_label_strs = [labels[int(p)] if p >= 0 else "?" for p in dev_preds] + ood_gate = fit_ood_gate(X_dev, pred_label_strs) + serving_router = Router(best["stages"], labels, + SimpleNamespace(embedding_dim=X.shape[1]), + ood_gate=ood_gate, + train_embeddings=X_dev if ood_gate is not None else None) + certificate = certify_policy(serving_router, X[certification_rows], + y_teacher[certification_rows], + config.target_teacher_agreement, + config.certification_alpha) + if (certificate["status"] == "certified" + and certificate["coverage"] < config.min_deploy_coverage): + certificate["status"] = "below_min_coverage" + + if certificate["status"] == "certified": best = selected["best"] method = best["summary"]["method"] - cov_cal = best["summary"].get("coverage_cal_total") - ta_cal = best["summary"].get("teacher_agreement_cal_total") + # Legacy field names retained; values now measure the exact serving + # policy on untouched certification data, not selection calibration. + cov_cal = certificate["coverage"] + ta_cal = certificate["teacher_agreement"] pipeline_path = save_pipeline(artifact_dir, best, labels) notes.append(f"Deployed {method} at target TA={config.target_teacher_agreement:.2f}, " f"coverage={cov_cal:.1%}, TA={ta_cal:.3f}") - # Build qualitative report - preds, handled, stage_id = route_pipeline(best["stages"], X) + # Descriptive report over supplied traces, using the exact runtime + # policy. This report is not an independent performance evaluation. + routed = serving_router.predict_batch(X) + preds, handled, stage_id = routed["preds"], routed["handled"], routed["stage_id"] texts = [r.input_text for r in dataset.records] teacher_labels_str = [r.teacher_label for r in dataset.records] idx_to_label = {i: l for i, l in enumerate(labels)} @@ -152,22 +256,11 @@ def log_fn(msg: str) -> None: # noqa: E301, local helper trace_ids=[r.trace_id for r in dataset.records]) qual_path = save_qualitative_report(artifact_dir, qual_report) - # Distance-based OOD gate: calibrate kNN-distance thresholds (global + - # per predicted label) so the runtime can defer off-distribution inputs - # the parity gate never saw. Safety net, not the partition. - try: - from tracer.fit.ood import fit_ood_gate - pred_label_strs = [idx_to_label.get(int(p), "?") for p in preds] - ood_gate = fit_ood_gate(X, pred_label_strs) - if ood_gate is not None: - (artifact_dir / "ood.json").write_text(json.dumps(ood_gate)) - notes.append( - f"OOD gate calibrated (kNN dist, global thr={ood_gate['global_thr']:.3f}, " - f"{len(ood_gate['per_label_thr'])} per-label)") - except Exception as _e: # never let the safety net break a fit - notes.append(f"OOD gate skipped: {_e}") + if ood_gate is not None: + (artifact_dir / "ood.json").write_text(json.dumps(ood_gate)) + np.save(artifact_dir / "ood_reference.npy", X_dev) else: - notes.append("No deployable pipeline met the target teacher-parity threshold.") + notes.append(f"No deployable pipeline met the final teacher-agreement check: {certificate['status']}.") # Save traces for continual learning (update needs them) from tracer.traces.loader import save_traces @@ -189,6 +282,7 @@ def log_fn(msg: str) -> None: # noqa: E301, local helper for item in frontier: frontier_summary.append({ "target": item["target"], + "evaluation_role": "development_selection_only", "best_method": item["best"]["summary"]["method"] if item["best"] else None, "best_coverage": item["best"]["summary"].get("coverage_cal_total") if item["best"] else None, "best_ta": item["best"]["summary"].get("teacher_agreement_cal_total") if item["best"] else None, @@ -204,7 +298,9 @@ def log_fn(msg: str) -> None: # noqa: E301, local helper embedding_dim=X.shape[1], n_retrains=1, pipeline_path=pipeline_path, index_path=str(index_path), config_path=str(config_path), - qualitative_report_path=qual_path) + qualitative_report_path=qual_path, + certification=certificate, + ood_required=method is not None and ood_gate is not None) write_manifest(artifact_dir / "manifest.json", manifest) return FitResult( @@ -297,12 +393,10 @@ def update( staged = root / "next" shutil.copytree(artifact_dir, staged) # A refit must not inherit reports or gates from the previous model. - for filename in ("pipeline.joblib", "qualitative_report.json", "ood.json", - "index.faiss", "report.html", "sankey.html"): - (staged / filename).unlink(missing_ok=True) + _clear_model_artifacts(staged) combined_path = root / "combined.jsonl" save_traces(combined_ds, combined_path) - result = fit(combined_path, staged, embeddings=X_combined, config=config) + result = _fit_artifacts(combined_path, staged, embeddings=X_combined, config=config) result.artifact_dir = str(artifact_dir) result.manifest.n_retrains = manifest.n_retrains + 1 for field_name in ("pipeline_path", "index_path", "config_path", "qualitative_report_path"): @@ -310,21 +404,7 @@ def update( if path is not None: setattr(result.manifest, field_name, str(artifact_dir / Path(path).relative_to(staged))) write_manifest(staged / "manifest.json", result.manifest) - # Outside the staging directory so a failed rollback cannot discard it. - backup = root.with_name(root.name + "-previous") - artifact_dir.rename(backup) - try: - staged.rename(artifact_dir) - except BaseException: - try: - backup.rename(artifact_dir) - except OSError as exc: - raise RuntimeError(f"Previous artifacts are preserved at {backup}; restore them before retrying") from exc - raise - try: - shutil.rmtree(backup) - except OSError: - result.notes.append(f"Update completed; previous artifact backup retained at {backup}") + _publish_generation(staged, artifact_dir, root, result.notes) return result diff --git a/src/tracer/config.py b/src/tracer/config.py index 025423e..ec554bf 100644 --- a/src/tracer/config.py +++ b/src/tracer/config.py @@ -27,6 +27,10 @@ class FitConfig: max_fit_labels: int = 8_000 embedding: EmbeddingConfig = field(default_factory=EmbeddingConfig) seed: int = 42 + # This partition is reserved before fitting or candidate selection. The + # final, fixed serving policy is checked once with a one-sided bound. + certification_fraction: float = 0.20 + certification_alpha: float = 0.10 # Emit per-candidate + per-stage progress to stderr during fit. Disable # for quiet notebook / CI runs. verbose: bool = True @@ -54,3 +58,7 @@ def __post_init__(self) -> None: f"{self.min_deploy_coverage}") if self.max_fit_labels <= 0: raise ValueError(f"max_fit_labels must be > 0, got {self.max_fit_labels}") + if not 0.0 < self.certification_fraction < 1.0: + raise ValueError("certification_fraction must be in (0, 1)") + if not 0.0 < self.certification_alpha < 1.0: + raise ValueError("certification_alpha must be in (0, 1)") diff --git a/src/tracer/fit/certification.py b/src/tracer/fit/certification.py new file mode 100644 index 0000000..33777f1 --- /dev/null +++ b/src/tracer/fit/certification.py @@ -0,0 +1,48 @@ +"""One-time evaluation of a fixed serving policy on untouched examples. + +The binomial confidence statement assumes independent examples drawn from the +same distribution as future requests. It is not a guarantee under distribution +shift, dependent sessions, or repeated adaptive reuse of the holdout. +""" + +from __future__ import annotations + +import numpy as np + +from tracer.fit.pipeline import _cp_lower + + +def certification_split(n: int, fraction: float, seed: int): + """Random row split, without stratification or label-dependent balancing. + +Reserve before any subsampling so certification retains traffic prevalence. +Never put a reserved example back into development to rescue a rare class. +""" + order = np.random.RandomState(seed).permutation(n) + n_cert = min(n, max(1, int(np.ceil(n * fraction)))) if n else 0 + return np.sort(order[n_cert:]), np.sort(order[:n_cert]) + + +def certify_policy(router, X, y, target: float, alpha: float) -> dict: + """Check exactly one policy, including its OOD filter, with no retries.""" + n = len(X) + if n: + routed = router.predict_batch(X) + handled = np.asarray(routed["handled"], dtype=bool) + n_accepted = int(handled.sum()) + n_correct = int((routed["preds"][handled] == np.asarray(y)[handled]).sum()) + else: + n_accepted = n_correct = 0 + lower = _cp_lower(n_correct, n_accepted, alpha) + return { + "status": "certified" if n_accepted > 0 and lower >= target else "failed", + "method": "untouched_final_policy_clopper_pearson", + "alpha": float(alpha), + "target": float(target), + "n_examples": n, + "n_accepted": n_accepted, + "n_correct": n_correct, + "coverage": n_accepted / n if n else 0.0, + "teacher_agreement": n_correct / n_accepted if n_accepted else None, + "teacher_agreement_lower": lower, + } diff --git a/src/tracer/fit/ood.py b/src/tracer/fit/ood.py index 10547cf..817bd4d 100644 --- a/src/tracer/fit/ood.py +++ b/src/tracer/fit/ood.py @@ -1,7 +1,7 @@ """Distance-based out-of-distribution (OOD) gate. -The parity gate certifies how often the surrogate agrees with the teacher *on -traffic that looks like the calibration data*. It says nothing about inputs that +The final policy check estimates teacher agreement on held-out traffic from the +same distribution. It says nothing about inputs that fall outside that distribution: an off-topic query, gibberish, a prompt-injection string, or a different-domain item can still get a confident surrogate prediction and slip through. This gate is the safety net: at fit time it measures how far @@ -10,7 +10,7 @@ confidence. Mechanism (deliberately commodity): mean distance to the k nearest training -neighbours, with a 95th-percentile threshold taken globally and per predicted +neighbours, with a 99.5th-percentile threshold taken globally and per predicted label (global fallback for sparse labels). This is standard kNN-distance OOD detection. It is intentionally NOT keyed on the partition cells, the cell construction is out of scope here; this gate only needs the input embeddings and diff --git a/src/tracer/fit/pipeline.py b/src/tracer/fit/pipeline.py index fe63549..a30033f 100644 --- a/src/tracer/fit/pipeline.py +++ b/src/tracer/fit/pipeline.py @@ -1,4 +1,4 @@ -"""Parity-gated pipeline construction. +"""Candidate construction and threshold selection on development data. Three candidate families: global, l2d, rsb. Acceptor features: top1-prob, top2-prob, margin, normalized-entropy. @@ -142,33 +142,12 @@ def _accept_scores(acceptor, probs): def _calibrate_threshold(scores, preds, y_teacher, target_ta, alpha=0.1, min_accept=10, seed=42): - """Choose the accept threshold on a held-out lower bound, then check it generalises. - - Earlier iterations sat at two extremes. The original gate selected the - threshold and reported its agreement on the *same* calibration set (in-sample - point estimate), so a threshold could clear target by luck and break the - contract on real traffic. The next version held out 50% for a Clopper-Pearson - verification, which is provably valid (distribution-free) but so data-hungry that on a few-hundred - calibration rows it certified NOTHING at strict targets (banking77 0% @0.98) - even though the confidence ranking clearly supported partial coverage. - - This is the hybrid that keeps the held-out guarantee while recovering that coverage: - - 1. Select on a 70% slice: among thresholds whose Clopper-Pearson LOWER bound - on accepted agreement clears target, take the highest-coverage one - (lowest threshold). The lower bound, not the point estimate, removes the - in-sample optimism at selection time. - 2. Verify generalisation on the held-out 30% with the point estimate - (accepted agreement >= target). This catches a selection-bias fluke (a - threshold that only clears by luck) without demanding CP-tightness on the - small verification slice, which is what starved the 50/50 version. - 3. Walk candidates in coverage order and deploy the highest-coverage one - that survives the held-out check, so coverage is monotonic in target. - - On very little data (n < 40) there is nothing to hold out, so it falls back to - a single-set Clopper-Pearson lower-bound gate (still a valid lower bound, just not held-out) - and flags it. Validated across seeds and datasets at 0.90-0.98 with zero - held-out contract violations. + """Select a threshold using development agreement and bound diagnostics. + + Selection searches thresholds and may reuse the internal verification + slice. These diagnostics are not a post-selection confidence guarantee. + Public fit() checks the final policy, including OOD, once on a separate + untouched partition via certify_policy(). """ n = len(scores) holdout = False @@ -221,9 +200,12 @@ def _calibrate_threshold(scores, preds, y_teacher, target_ta, alpha=0.1, min_acc acc_all = scores >= t n_all = int(acc_all.sum()) k_all = int((preds[acc_all] == y_teacher[acc_all]).sum()) + lower_all = _cp_lower(k_all, n_all, alpha) + if lower_all < target_ta: + continue return {"threshold": float(t), "teacher_agreement": float(k_all / n_all), - "teacher_agreement_lower": float(_cp_lower(k_all, n_all, alpha)), + "teacher_agreement_lower": float(lower_all), "coverage": float(acc_all.mean()), "holdout": holdout} return None @@ -238,10 +220,10 @@ def _predict(clf, X): def build_global(split, target_ta, alpha: float = 0.1, log: Optional[LogFn] = None, skip: Iterable[str] = (), seed=42): - """Global pipeline: one surrogate, accept all if the agreement lower bound - clears the target. Gating on the Clopper-Pearson lower bound (not the raw - point estimate) stops a small or lucky calibration set from certifying an - accept-all deploy that then breaks the contract on real traffic.""" + """Select an accept-all candidate using a development bound diagnostic. + + Final deployment certification is separate from this candidate sweep. + """ log = log or _noop_log if len(np.unique(split["y_train"])) < 2 or len(split["X_val"]) == 0: return {"method": "global", "stages": [], "summary": {"status": "insufficient_data", "coverage_cal_total": 0.0}} @@ -449,11 +431,10 @@ def _pipeline_cal_summary(method, stages, X_cal, y_cal): def fit_frontier(X, y_teacher, targets, max_fit_labels=8000, min_coverage=0.05, alpha: float = 0.1, log: Optional[LogFn] = None, skip: Iterable[str] = (), seed=42): - """Build global/l2d/rsb for each target TA, return best per target. + """Build and select global/l2d/rsb candidates on development data. - `alpha` is the confidence level for the deployment lower bound: a candidate - is only certified if the lower bound on its held-out teacher agreement clears - the target, so reported coverage reflects a held-out lower bound rather than an in-sample point estimate. + All returned coverage and agreement values are selection diagnostics. + They are not certification of the adaptively chosen final serving policy. """ log = log or _noop_log log(f"fit_frontier: X={X.shape} targets={sorted(set(float(t) for t in targets))} " diff --git a/src/tracer/policy/artifacts.py b/src/tracer/policy/artifacts.py index 54b37ab..80eda25 100644 --- a/src/tracer/policy/artifacts.py +++ b/src/tracer/policy/artifacts.py @@ -25,6 +25,8 @@ def write_manifest(path: Path, manifest: ArtifactManifest) -> None: "index_path": manifest.index_path, "config_path": manifest.config_path, "qualitative_report_path": manifest.qualitative_report_path, + "certification": manifest.certification, + "ood_required": manifest.ood_required, } path.write_text(json.dumps(payload, indent=2), encoding="utf-8") diff --git a/src/tracer/runtime/router.py b/src/tracer/runtime/router.py index 932bfc7..b69d06e 100644 --- a/src/tracer/runtime/router.py +++ b/src/tracer/runtime/router.py @@ -64,7 +64,7 @@ def load(cls, artifact_dir: Union[str, Path], embedder=None) -> "Router": raise ValueError( f"No router is deployed in {artifact_dir}: the parity gate did not " "certify any method (selected_method is null in manifest.json). " - "Run tracer.fit() again with more data or a lower target_teacher_agreement." + "Inspect manifest.certification and collect more representative data before refitting." ) bundle = load_pipeline(artifact_dir) stages = bundle["pipeline"]["stages"] @@ -74,13 +74,22 @@ def load(cls, artifact_dir: Union[str, Path], embedder=None) -> "Router": ood_gate = None train_emb = None ood_path = artifact_dir / "ood.json" + if manifest.ood_required and not ood_path.exists(): + raise ValueError(f"Required OOD guard is missing: {ood_path}") if ood_path.exists(): try: ood_gate = _json.loads(ood_path.read_text()) - from tracer.embeddings.index import EmbeddingIndex - train_emb = EmbeddingIndex.load(artifact_dir / "index").embeddings - except Exception: - ood_gate, train_emb = None, None + reference = artifact_dir / "ood_reference.npy" + if reference.exists() or manifest.ood_required: + train_emb = np.load(reference, allow_pickle=False) + else: + # Legacy policies used all stored embeddings as reference. + # Preserve their behavior; do not invent certification. + from tracer.embeddings.index import EmbeddingIndex + train_emb = EmbeddingIndex.load(artifact_dir / "index").embeddings + _validate_ood(ood_gate, train_emb, manifest.embedding_dim) + except Exception as exc: + raise ValueError(f"Cannot load the OOD guard from {artifact_dir}; refusing unguarded routing") from exc return cls(stages=stages, label_space=label_space, manifest=manifest, embedder=embedder, ood_gate=ood_gate, train_embeddings=train_emb) @@ -206,3 +215,23 @@ def predict_batch( decisions.append("deferred") return {"labels": labels, "decisions": decisions, "handled": handled, "preds": preds, "stage_id": stage_id} + + +def _validate_ood(gate, reference, expected_dim): + """Reject malformed artifacts rather than turning a safety check off.""" + if not isinstance(gate, dict): + raise ValueError("OOD metadata must be an object") + k = gate.get("k") + if isinstance(k, bool) or not isinstance(k, int) or k < 1: + raise ValueError("OOD neighbor count must be a positive integer") + X = np.asarray(reference) + if (X.ndim != 2 or len(X) < k or X.shape[1] == 0 + or (expected_dim is not None and X.shape[1] != expected_dim) + or not np.isfinite(X).all()): + raise ValueError("OOD reference embeddings are invalid") + per_label = gate.get("per_label_thr", {}) + if not isinstance(per_label, dict): + raise ValueError("OOD per-label thresholds must be an object") + thresholds = [gate["global_thr"], *per_label.values()] + if any(not np.isfinite(float(t)) or float(t) < 0 for t in thresholds): + raise ValueError("OOD distance thresholds must be finite and nonnegative") diff --git a/src/tracer/scanner.py b/src/tracer/scanner.py index 0dd2c21..0c37c9a 100644 --- a/src/tracer/scanner.py +++ b/src/tracer/scanner.py @@ -1,17 +1,18 @@ -"""tracer.scan(): the day-one verdict on a traces file. +"""tracer.scan(): a diagnostic for classification traces. Answers three questions in one pass: - 1. How much of this traffic is predictable enough for a near-free model? - (certifiable share, measured with exact binomial bounds on HELD-OUT - data, never in-sample) - 2. What does that predictable traffic look like? (clusters with dominant + 1. Which clusters meet a per-cluster teacher-agreement check on held-out + examples, using a dominant label learned from the fitting slice? + 2. What does that traffic look like? (clusters with dominant labels and real examples) - 3. What would that be worth? (savings per 1k calls at your teacher price, - extrapolated monthly if you tell us your volume) + 3. What teacher spend might those requests represent? (a scenario before + embedding, student, fallback and serving costs, optionally scaled monthly) The scan is a diagnostic, deliberately simple: similarity clustering plus -per-cluster exact confidence bounds. It does not train a router; `fit` -does that, learning a router with accept gates on the same traffic. +per-cluster binomial confidence bounds. The selected clusters do not have a +joint deployment certificate. It does not train a serving router; `fit` trains +one and separately checks its final policy. Coverage can increase or decrease, +and a fit may produce no deployable policy. Usage: import tracer @@ -37,10 +38,9 @@ _INPUT_KEYS = ("input", "query", "text", "prompt", "question") _LABEL_KEYS = ("teacher", "teacher_output", "label", "intent", "output", "answer") -# Data-volume policy. Below MIN the held-out evidence is too thin for the -# bounds to mean anything, so a plain scan refuses and points at force=True. -# SUGGESTED is the volume where cells reliably carry enough held-out members -# to certify at 0.90 without coarsening. +# Data-volume guidance. A plain scan refuses below MIN and offers force=True +# for a diagnostic on limited evidence. SUGGESTED is a collection heuristic, +# not a promise that any cluster will meet the target. MIN_SCAN_TRACES = 1_000 SUGGESTED_SCAN_TRACES = 5_000 @@ -88,6 +88,7 @@ def load_scan_traces(path: Union[str, Path]) -> tuple[list[str], list[str]]: @dataclass class ScanCluster: + """Per-cluster evidence; historical `certifiable` is not deployment approval.""" cluster_id: int n_fit: int n_held: int @@ -101,26 +102,32 @@ class ScanCluster: @dataclass class ScanResult: + """Scan diagnostics with historical property names retained for compatibility. + + `certifiable_share` and `certified_floor` describe per-cluster checks, not + final-policy certification. `savings_per_1k_calls` and `monthly_savings` + represent potentially avoided teacher spend before execution overhead. + """ n_traces: int n_classes: int n_clusters: int target: float - certifiable_share: float # held-traffic share inside certified clusters - certified_floor: float # worst certified cluster's bound + certifiable_share: float # held-traffic share in qualifying clusters + certified_floor: float # lowest qualifying cluster's bound clusters: list[ScanCluster] = field(default_factory=list) # economics (None when no price given) teacher_price_per_1k: Optional[float] = None savings_per_1k_calls: Optional[float] = None monthly_calls: Optional[int] = None monthly_savings: Optional[float] = None - # certifiable share at a few relaxed targets, so a single 0 at a strict + # qualifying share at a few relaxed targets, so a single 0 at a strict # target still shows the user where their data sits. {target: share} frontier: dict[float, float] = field(default_factory=dict) # guidance when data is thin traces_needed_estimate: Optional[int] = None # True when run with force=True: thin-data guard bypassed, clustering - # coarsened to concentrate held-out mass. Bounds are best-effort, not a - # guarantee. Surfaced as a warning in the terminal and a report banner. + # coarsened to concentrate held-out mass. This changes diagnostic + # granularity, not deployment status. Surfaced in both reports. forced: bool = False # 3D projection for the HTML report viz: # {"points": [[x,y,z,cluster_id], ...], "clusters": {cid: {"label","cert"}}} @@ -161,15 +168,17 @@ def scan( ) -> ScanResult: """Run the scan. Returns a ScanResult; use format_scan / scan_html to render. - Certification contract: every certifiable claim is an exact binomial lower - bound computed on a held-out 30% slice the clustering never saw. + Each cluster's bound uses a held-out 30% slice the clustering never saw. + These are individual binomial bounds under independent, representative + sampling assumptions, not a simultaneous guarantee for selected clusters + or a certificate for a serving policy. force: on thin data a cluster may not have enough held-out members for any bound to clear the target, however clean the traffic actually is. With force=True the scan does not change the binomial maths (the bound stays the bound), it coarsens the clustering so each cell carries roughly the ~22 - held-out members a 0.90 target needs, trading granularity for statistical - power, and marks the result as forced so the caller can warn loudly. + held-out members needed at 0.90 with no observed disagreements, trading + granularity for evidence per cluster. This does not ensure a passing check. """ inputs, labels = load_scan_traces(traces_path) n = len(inputs) @@ -180,7 +189,7 @@ def scan( f"scan needs at least {MIN_SCAN_TRACES:,} traces for a stable read " f"(we suggest ~{SUGGESTED_SCAN_TRACES:,}); this file has {n:,}. " f"Collect more, or pass force=True to scan anyway on thin data " - f"(results will be a best-effort floor, not a guarantee)." + f"(results remain per-cluster diagnostics, not deployment certification)." ) if embeddings is None: @@ -361,30 +370,33 @@ def rule(char="-", width=56): pct = r.certifiable_share * 100 L.append("") - L.append(f" {pct:.1f}% of traffic certifiable for a near-free model") + L.append(" Trace diagnostic") + L.append(f" {pct:.1f}% of held-out requests lie in qualifying clusters") L.append(f" {r.n_traces:,} traces · {r.n_classes} labels · {r.n_clusters} cells · target {r.target:.0%} agreement") L.append(f" {rule()}") if r.forced: L.append(f" ⚠ forced scan thin data: clustering coarsened to concentrate held-out evidence.") - L.append(f" Bounds are a best-effort floor, not a guarantee. Collect more traces or run `tracer.fit()` for the real number.") + L.append(" Coarser clusters do not certify a serving policy. Collect representative traces before deployment.") if r.certifiable_share > 0: - L.append(f" {'worst certified bound':<22}{r.certified_floor:.3f} exact, held-out") + L.append(f" {'lowest qualifying bound':<26}{r.certified_floor:.3f} per-cluster, held-out") if r.savings_per_1k_calls is not None: - L.append(f" {'savings / 1k calls':<22}${r.savings_per_1k_calls:.2f} at ${r.teacher_price_per_1k}/1k teacher") + L.append(f" {'teacher spend / 1k calls':<26}${r.savings_per_1k_calls:.2f} potentially avoided at ${r.teacher_price_per_1k}/1k teacher") if r.monthly_savings is not None: - L.append(f" {'monthly savings':<22}${r.monthly_savings:,.0f} at {r.monthly_calls:,} calls/mo") + L.append(f" {'monthly spend scenario':<26}${r.monthly_savings:,.0f} at {r.monthly_calls:,} calls/mo") + if r.savings_per_1k_calls is not None: + L.append(" Before embedding, student, fallback and serving costs; not measured net savings.") if r.traces_needed_estimate: L.append("") L.append(f" ⚠ Not enough held-out evidence yet. Collect roughly " f"{r.traces_needed_estimate:,} more traces and rescan,") - L.append(f" or pass force=True to certify on what you have. Exact bounds need ~22 held-out examples per cell.") + L.append(" or pass force=True for a limited-data diagnostic. The collection estimate does not ensure a passing check.") if r.frontier: L.append("") - L.append(f" Certifiable share by target (lightweight scan estimate)") + L.append(f" Qualifying cluster share by target (scan diagnostic)") L.append(f" {rule('·')}") for tgt in sorted(r.frontier): share = r.frontier[tgt] @@ -393,12 +405,12 @@ def rule(char="-", width=56): L.append("") L.append(f" Cells (top {min(20, len(r.clusters))} by traffic share)") L.append(f" {rule('·')}") - L.append(f" {'share':>6} {'held':>5} {'bound':>6} verdict dominant label") + L.append(f" {'share':>6} {'held':>5} {'bound':>6} check dominant label") for cl in r.clusters[:20]: if cl.certifiable: - tag = f"✔ free " + tag = f"✔ meets" else: - tag = f"→ keep " + tag = f"→ below" L.append(f" {cl.share*100:>5.1f}% {cl.n_held:>5} {cl.cp_lower:>6.3f} {tag} {cl.dominant_label[:38]}") if cl.examples: ex = " ".join(cl.examples[0].split())[:68] @@ -407,8 +419,10 @@ def rule(char="-", width=56): L.append(f" ... {len(r.clusters) - 20} more cells") L.append("") - L.append(f" Certifiable = an exact binomial lower bound on held-out label agreement clears your target. No in-sample numbers.") - L.append(f" Next: tracer.fit() trains a real router with accept gates and certifies more on the same traffic.") + L.append(" Qualifying = a per-cluster binomial lower bound clears the teacher-agreement target.") + L.append(" Historical fields certifiable_share/certified_floor describe this scan, not a deployment certificate.") + L.append(" Bounds assume independent, representative examples; they do not jointly certify selected clusters.") + L.append(" Next: tracer.fit() trains and checks a serving policy. Its coverage may differ, and deployment may be declined.") L.append("") return "\n".join(L) @@ -488,7 +502,7 @@ def rule(char="-", width=56):
Colour by
-

Drag to rotate, scroll to zoom. Every dot is one request, placed by meaning so similar questions sit together. Switch the colouring to see verdict (free vs kept) or the dominant label of each cell.

+

Drag to rotate, scroll to zoom. Each dot represents a sampled request in a projection of its embedding. Colour shows the per-cluster check or the cluster's dominant label; it does not show deployed routing decisions.

""" _VIZ_SCRIPT = """ @@ -561,7 +575,7 @@ def rule(char="-", width=56): function buildLegend(){ if(!legend){return;} if(mode==='verdict'){ - legend.innerHTML='auto-answered, freestays on your model'; + legend.innerHTML='meets cluster checkbelow cluster target'; return; } // Label mode: one swatch per dominant label, ordered by traffic share. @@ -586,13 +600,13 @@ def rule(char="-", width=56): } function showCard(id){ var cl=C[id]; if(!cl){card.className='cellcard';return;} - var safe=!!cl.cert, pc=safe?'#22c55e':'#ef4444', pt=safe?'Auto-answer, free':'Keep on your model'; + var safe=!!cl.cert, pc=safe?'#22c55e':'#ef4444', pt=safe?'Meets cluster check':'Below cluster target'; var share=(cl.share!=null)?(cl.share*100).toFixed(1)+'%':''; var bound=(cl.bound!=null)?Math.round(cl.bound*100)+'%':''; var ex=(cl.ex||[]).map(function(s){return '
“'+esc(s)+'”
';}).join(''); card.innerHTML=''+pt+''+ '
'+esc(cl.label||'')+'
'+ - '
traffic '+share+'match rate '+bound+'
'+ + '
held-out share '+share+'lower bound '+bound+'
'+ (ex?('
sounds like
'+ex+'
'):''); card.className='cellcard on'; } @@ -647,7 +661,7 @@ def rule(char="-", width=56): def scan_html(r: ScanResult, source_name: str = "traces") -> str: - """Self-contained, brand-styled HTML report with a 3D embedding view.""" + """Render scan evidence as HTML; the optional 3D view loads external scripts.""" import json as _json GREEN, RED = "#16a34a", "#dc2626" pct = r.certifiable_share * 100 @@ -655,7 +669,7 @@ def scan_html(r: ScanResult, source_name: str = "traces") -> str: rows = [] for c in r.clusters: color = GREEN if c.certifiable else RED - verdict = "Auto-answer, free" if c.certifiable else "Keep on your model" + verdict = "Meets cluster check" if c.certifiable else "Below cluster target" exs = "".join( f"
“{_esc(e[:120])}{'…' if len(e) > 120 else ''}”
" for e in c.examples[:2] @@ -669,20 +683,21 @@ def scan_html(r: ScanResult, source_name: str = "traces") -> str: money = "" if r.savings_per_1k_calls is not None: - money = f"${r.savings_per_1k_calls:.2f} saved per 1,000 calls" + money = f"Teacher spend potentially avoided: ${r.savings_per_1k_calls:.2f} per 1,000 calls" if r.monthly_savings is not None: - money += f" · about ${r.monthly_savings:,.0f}/month at {r.monthly_calls:,} calls" + money += f" · ${r.monthly_savings:,.0f}/month at {r.monthly_calls:,} calls" + money += "
Scenario before embedding, student, fallback and serving costs. Actual net savings require a deployed-policy evaluation." thin = "" if r.traces_needed_estimate: - thin = (f"
Not enough data yet to prove much. Collect about " + thin = (f"
Limited held-out evidence. The collection heuristic suggests about " f"{r.traces_needed_estimate:,} more requests and run this again. " - f"Each group needs roughly 22 held-out examples before we can vouch for it.
") + f"This estimate does not ensure that any cluster will pass its check.
") if r.forced: thin = ("
Forced scan on limited data. The grouping was " - "coarsened to squeeze the most evidence out of a small sample, so every number " - "below is a best-effort floor, not a guarantee. Collect more requests, or run " - "tracer.fit(), for a number you can quote.
") + "coarsened to increase evidence per cluster. Results are diagnostics on limited " + "data, not deployment certification. Collect representative requests and inspect " + "the final policy check after tracer.fit().") has_viz = bool(r.projection and r.projection.get("points")) viz_block = _VIZ_HTML if has_viz else "" @@ -700,24 +715,24 @@ def scan_html(r: ScanResult, source_name: str = "traces") -> str: -

{pct:.0f}% of your traffic can be answered for free

-

A near-free model already matches your current model on {pct:.0f}% of requests, proven on held-out examples, not guessed.

+

Trace diagnostic

+

{pct:.0f}% of held-out requests fall in clusters that meet the per-cluster teacher-agreement check.

{r.n_traces:,} requests · {r.n_clusters} groups of similar questions · target {r.target:.0%} agreement

- Train the real router with tracer.fit() → + Train and evaluate with tracer.fit() → {save_html} {thin} -
How to read this. We grouped your requests into clusters of similar questions and laid them out in the space below. For each cluster we checked, on examples it never saw, how often a tiny free model agrees with your model. Green means it agrees at least {r.target:.0%} of the time, so it is safe to auto-answer for free. Red means we could not prove that yet, so those stay on your model.
+
How to read this. Clusters and their dominant labels come from the fitting slice. On held-out examples, we measure how often each cluster's dominant label matches the teacher. Green means that cluster's binomial lower bound meets the {r.target:.0%} target. Red means it does not. The scan is a data diagnostic, not a serving policy or deployment certificate.
{viz_block} - + {''.join(rows)}
What customers askShare of trafficMatch rate (proven)Verdict
Dominant label and examplesHeld-out shareAgreement lower bound (per cluster)Check
-

This is a fast, conservative estimate. Train the real router for more.

-

This 2-minute scan groups your traffic by similarity, a deliberately conservative read, so the real number is usually higher. tracer.fit() trains the actual router with accept gates and certifies a larger share of the same traffic.

+

Next, train and check the serving policy.

+

tracer.fit() trains a classifier and acceptance policy, then checks the selected policy, including its OOD filter, on a separate final partition. Its coverage may be higher or lower than this scan, and insufficient evidence can prevent deployment.

pip install tracer-llm  then  tracer.fit("your_traces.jsonl", embeddings=X)

-