Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,15 @@ on:

jobs:
test:
name: Python ${{ matrix.python-version }} (${{ matrix.dependencies }})
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
python-version: ["3.12", "3.13", "3.14"]
dependencies: [latest]
include:
- python-version: "3.12"
dependencies: minimum

steps:
- uses: actions/checkout@v4
Expand All @@ -24,7 +29,28 @@ jobs:
- name: Install package and dev dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"
if [ "${{ matrix.dependencies }}" = "minimum" ]; then
python -m pip install -e ".[dev]" "numpy==1.26.4" "scikit-learn==1.4.2" "joblib==1.3.2"
else
python -m pip install -e ".[dev]"
fi
python -m pip check

- name: Run tests
run: pytest tests/ -v --tb=short

watch-js:
runs-on: ubuntu-latest
defaults:
run:
working-directory: js
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: npm
cache-dependency-path: js/package-lock.json
- run: npm ci
- run: npm test -- --maxWorkers=1 --minWorkers=1
- run: npm run build
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.14"
- name: Verify pushed tag matches package version
run: |
tag="${GITHUB_REF_NAME#v}"
Expand Down
9 changes: 4 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ pip install tracer-llm # numpy, scikit-learn, joblib - that's it
pip install tracer-llm[embeddings] # adds sentence-transformers (if user needs text-in routing)
```

No GPU required. No API keys required for fitting. Works on Python 3.9+.
No GPU required. No API keys required for fitting. Requires Python 3.12+; CI tests 3.12, 3.13, and 3.14.

---

Expand Down Expand Up @@ -154,8 +154,8 @@ router = tracer.load_router(".tracer", embedder=my_embedder)
```

### Pattern 4: HTTP server (zero deps)
```bash
tracer serve .tracer --port 8000
```python
tracer.serve(".tracer", port=8000)
# POST /predict {"embedding": [...]} → {"label", "decision", "accept_score"}
```

Expand All @@ -173,7 +173,7 @@ tracer serve .tracer --port 8000
## Things you CAN do autonomously

- Install `tracer-llm` (it's lightweight, MIT licensed)
- Run `tracer demo` to show the user what it does
- Follow `notebooks/01-quickstart.ipynb` for a worked example
- Write traces to JSONL from any data source
- Fit a policy if traces + embeddings are available
- Generate the HTML report
Expand All @@ -195,7 +195,6 @@ tracer serve .tracer --port 8000
| `Embedder.from_endpoint(url)` | External embedding API | Needs URL + auth from human |
| `tracer.generate_html_report(dir)` | Visual audit | No |
| `tracer.serve(dir, port=8000)` | HTTP server | No |
| `tracer demo` | Interactive demo | No |

---

Expand Down
63 changes: 29 additions & 34 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,26 @@
All notable changes to TRACER are recorded here. This project follows semantic
versioning.

## Unreleased

### Changed (breaking)
- Require Python 3.12 or newer; drop support for Python 3.9–3.11. CI tests
Python 3.12, 3.13, and 3.14, plus the minimum core dependencies on 3.12.
- Allow NumPy 1.26.4 through 2.x so Python 3.14 can install compatible wheels.
Require scikit-learn 1.4.2+ and joblib 1.3.2+ in core, and PyTorch 2.4.1+
for the optional embedding extras. Remove the obsolete NumPy downgrade advice.
- Build releases and run the documented Python sidecar on Python 3.14.

### Removed (breaking)
- The `tracer` command-line executable and command implementations. Use the
Python APIs for fitting, scanning, updating, reports, and serving predictions.
- The Tracer app client, authentication, and automatic app telemetry exporters
in both Python and JavaScript. Existing app credentials no longer trigger
uploads. Local recording, custom sinks, and explicitly configured generic
HTTP exports remain available.
- App connection instructions and command-line documentation. All current
examples use the library APIs.

## 0.3.3 (2026-06)

### Fixed
Expand All @@ -20,48 +40,27 @@ versioning.
`ValueError` messages. Previously the count excluded blank lines, so
the reported line was lower than the actual position of the bad record.

## 0.3.1 (2026-06)

### Changed
- `tracer cloud` commands that take a tracer (`get`, `training`, `analytics`,
`label-space`, `traces`, ...) now accept either the tracer slug or its id.
- `tracer cloud scan` works without logging in, matching the public web scan.

### Fixed
- `tracer cloud keys create` and `tracer cloud ingest-keys create` now honor
`--json` and emit the created record (including the one-time secret) as JSON.

## 0.3.0 (2026-06)

### Added
- `tracer.watch`: a decorator (and HTTP fallback) that observes the LLM calls
your pipeline already makes and ships them, free, to Tracer Cloud, where they
accumulate and auto-optimize once there are enough. Captures the full request
and response on the OpenTelemetry GenAI (`gen_ai.*`) schema, with pluggable
sinks (local file, Tracer Cloud, OTLP, or several at once).
- `tracer.watch`: a decorator that records LLM calls on the OpenTelemetry
GenAI (`gen_ai.*`) schema, with pluggable local and generic export sinks.
- `@tracer-llm/watch`: a zero-dependency JavaScript/TypeScript mirror of the watch
decorator (async-context aware), so JS pipelines get the same one-line
observability as Python.
- `tracer cloud`: a full command-line interface for Tracer Cloud, at parity with
the dashboard. Browser or password login, then manage tracers (create, quick
and bulk uploads, agentic `onboard`, rename, delete), training (retrain,
auto-retrain, promote/rollback, live status), routing (`route`), test
batteries, the model library, API keys, observability ingest keys, billing,
analytics, trace selection (`traces`), and a public `scan`.
- Lazy package initialization so `import tracer` stays fast and only pulls in the
heavy pieces when you actually use them.

### Changed
- `tracer scan` is more robust on messy uploads: tolerant input/label aliasing
and a clarification path so ambiguous files still produce a result.
- Trace ingestion accepts additional input and label aliases.

### Docs
- New guides for the watch decorator and the `tracer cloud` CLI.
- New guide for the watch decorator.

## 0.2.0 (2026-06)

### Added
- `tracer scan`: a fast, conservative day-one read of a traces file, before any
- `tracer.scan()`: a fast, conservative day-one read of a traces file, before any
training. It groups traffic by similarity and measures, on a held-out slice it
never saw, how much a near-free model can answer at your target agreement,
using exact Clopper-Pearson bounds, with an optional per-1k and monthly savings
Expand All @@ -72,24 +71,20 @@ versioning.
far from the training distribution (kNN distance, global and per-predicted-label
thresholds) regardless of surrogate confidence, so off-distribution traffic goes
to the teacher instead of getting a confident guess.
- `tracer fit --trees` to opt in the tree surrogates, and `--skip` to drop named
candidates from the zoo.
- `FitConfig.skip_candidates` to drop named candidates from the zoo, including
tree surrogates when a lighter sweep is desired.
- Trace loaders accept common key aliases for both input and label
(`input/query/text/prompt/question` and
`teacher/teacher_output/label/intent/output/answer`).
- Bring-your-own embeddings for `tracer scan`: local sentence-transformers by
default (`--embed-model`), a precomputed `.npy` (`--embeddings`), or your own
HTTP embedding endpoint (`--embed-url`, with header and response-key options).
- Bring-your-own embeddings for `tracer.scan()`: pass a precomputed array with
`embeddings=`, or select a local sentence-transformers model with `model=`.

### Changed
- The parity gate now certifies on an exact held-out lower bound instead of an
in-sample point estimate, so a policy cannot clear the target by in-sample luck
and then break the contract on real traffic. Coverage is now monotonic in the
target, and a hybrid select-then-verify procedure recovers coverage at strict
targets that a plain held-out split discarded.
- Tree surrogates (decision tree, random forest, extra-trees, gradient boosting)
are now off by default in `tracer fit`; the default zoo is the fast linear and
MLP heads. Use `--trees` for hard, high-class-count tasks.
- The HTML report is restyled to the light Tracer theme, and the word "audit" is
dropped across the report and docs.

Expand Down
21 changes: 15 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ we're optimizing for.

## Setup

Use Python 3.12 or newer. CI tests 3.12, 3.13, and 3.14 with current
dependencies, plus 3.12 with the minimum supported core dependencies.

```bash
git clone https://github.com/adrida/tracer
cd tracer
Expand All @@ -68,13 +71,20 @@ pip install -e ".[dev]"
pytest tests/ -v
```

Tests use synthetic data and run in temporary directories. No external
dependencies or API keys required.
Tests use synthetic data and run in temporary directories. No model downloads
or API keys are required. To check the JavaScript recorder:

```bash
cd js
npm ci
npm test -- --maxWorkers=1 --minWorkers=1
npm run build
```

## Quick sanity check

```bash
tracer demo
pytest tests/test_router.py tests/test_watch.py -q
```

## Project structure
Expand All @@ -101,9 +111,8 @@ src/tracer/
runtime/
router.py <- production Router class
serve.py <- lightweight HTTP prediction server (stdlib only)
cli/
main.py <- tracer CLI entry point (fit, report, update, demo, serve)
_ui.py <- terminal formatting and progress display
scanner.py <- pre-training traffic diagnostics and reports
watch.py <- local trace recording and generic export sinks
```

## Adding a new surrogate model
Expand Down
Loading
Loading