Fix routing correctness and harden updates, tracing, and reports (#87)

Author: adrida

.github/workflows/ci.yml

@@ -17,6 +17,8 @@ jobs:
         include:
           - python-version: "3.12"
             dependencies: minimum
+          - python-version: "3.12"
+            dependencies: faiss
 
     steps:
       - uses: actions/checkout@v4
@@ -31,6 +33,8 @@ jobs:
           python -m pip install --upgrade pip
           if [ "${{ matrix.dependencies }}" = "minimum" ]; then
             python -m pip install -e ".[dev]" "numpy==1.26.4" "scikit-learn==1.4.2" "joblib==1.3.2"
+          elif [ "${{ matrix.dependencies }}" = "faiss" ]; then
+            python -m pip install -e ".[dev,faiss]" plotly
           else
             python -m pip install -e ".[dev]"
           fi

AGENTS.md

@@ -32,7 +32,7 @@ out = router.predict(embedding_vector)              # numpy array
 out = router.predict("raw text", embedder=embedder) # or text directly
 
 # 3. UPDATE - continual learning (coverage grows over time)
-tracer.update("new_traces.jsonl", embeddings=X_new)
+tracer.update("new_traces.jsonl", new_embeddings=X_new)
 
 # 4. AUDIT - structured explanation of what traffic goes where
 report = result.qualitative_report  # slices, boundary pairs, examples
@@ -148,7 +148,7 @@ out = router.predict(text, fallback=lambda: call_gpt4(text))
 ### Pattern 3: Continual learning loop
 ```python
 # Collect new traces from production
-tracer.update("new_traces.jsonl", embeddings=X_new)
+tracer.update("new_traces.jsonl", new_embeddings=X_new)
 # Reload router with improved policy
 router = tracer.load_router(".tracer", embedder=my_embedder)
 ```
@@ -190,7 +190,7 @@ tracer.serve(".tracer", port=8000)
 | `tracer.load_router(dir, embedder=e)` | Load for inference | No |
 | `router.predict(text_or_emb)` | Route one input | No |
 | `router.predict_batch(inputs)` | Route a batch | No |
-| `tracer.update(new_traces, embeddings=X)` | Refit with new data | No |
+| `tracer.update(new_traces, new_embeddings=X)` | Refit with new data | No |
 | `tracer.embed(texts)` | Compute embeddings | Needs `pip install tracer-llm[embeddings]` |
 | `Embedder.from_endpoint(url)` | External embedding API | Needs URL + auth from human |
 | `tracer.generate_html_report(dir)` | Visual audit | No |

CHANGELOG.md

@@ -5,7 +5,33 @@ versioning.
 
 ## Unreleased
 
+### Fixed
+- Preserve the previous artifact generation when an update fails validation,
+  fitting, or publication. Honor explicit update configuration without mutating
+  the caller, retain saved configuration by default, and increment the fit count.
+- Preserve classifier label IDs in residual stages and select the largest
+  eligible acceptance set during small-data calibration.
+- Keep original embedding scales when building FAISS indexes and reuse the OOD
+  nearest-neighbor index across predictions. Existing FAISS artifacts trained
+  with unnormalized inputs need refitting to recover the original scale.
+- Apply configured seeds to subsampling, splits, calibration, and model training.
+- Record async outputs, errors, cancellation, timing, and parent spans correctly;
+  isolate tracing storage/export failures from application calls.
+- Escape trace and artifact text in reports and reject unsafe watcher filenames
+  in Python and JavaScript.
+- Preserve deferred confidence scores in reports; validate prediction shapes and
+  finite values; report actual file lines for null teacher labels.
+- Handle concurrent HTTP requests and client disconnects, reject malformed or
+  oversized request bodies, and close the server cleanly on interruption.
+- Correct update examples to use `new_embeddings=`.
+
 ### Changed (breaking)
+- The prediction server binds to `127.0.0.1` by default and no longer sends a
+  wildcard CORS header. Explicitly configure a remote bind address when needed;
+  browser deployments should configure CORS at their authenticated proxy.
+- Watcher filenames must start with a letter or digit and use only ASCII
+  letters, digits, dots, underscores, or hyphens, up to 128 characters.
+- Invalid prediction inputs return HTTP 400 rather than 500.
 - Require Python 3.12 or newer; drop support for Python 3.9–3.11. CI tests
   Python 3.12, 3.13, and 3.14, plus the minimum core dependencies on 3.12.
 - Allow NumPy 1.26.4 through 2.x so Python 3.14 can install compatible wheels.

README.md

@@ -142,7 +142,7 @@ Day 5: 10,000 traces → 92% coverage → 9,200 calls/day saved
 ```
 
 ```python
-tracer.update("new_traces.jsonl", embeddings=X_new)  # refit with new production traces
+tracer.update("new_traces.jsonl", new_embeddings=X_new)  # refit with new production traces
 ```
 
 The parity gate re-calibrates on each update, so coverage only increases when the surrogate actually earns it.

benchmarks/README.md

@@ -0,0 +1,38 @@
+# Routing reliability ablation
+
+`routing_reliability.py` exercises three deterministic failure cases without
+external data, model downloads, or API calls. The inputs and seeds are in the
+script. These are targeted regression measurements, not general workload claims.
+
+Run the same script against each checkout with identical dependencies:
+
+```bash
+pip install -e '.[faiss]'
+OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 LOKY_MAX_CPU_COUNT=1 \
+  python benchmarks/routing_reliability.py
+```
+
+Measured with Python 3.12.13, NumPy 2.5.3, scikit-learn 1.9.1, and FAISS 1.15.1.
+The baseline is commit `0c918712c09a677feae80eede7069442c4e259a9`.
+
+| Case | Baseline | Fixed |
+|---|---:|---:|
+| Small calibration: accepted fraction | 30.56% | 91.67% |
+| Small calibration: agreement on accepted rows | 100% | 90.91% |
+| Small calibration: Clopper–Pearson lower bound | 81.11% | 80.86% |
+| Model trained on a label subset: label agreement | 0% | 100% |
+| FAISS fit/reload: held-out coverage | 0% | 99% |
+| FAISS fit/reload: agreement on handled held-out rows | Not applicable (none handled) | 100% |
+| FAISS fit: caller input unchanged | No | Yes |
+
+The small-calibration case has 36 rows, a target of 80%, and alpha 0.1.
+Both selections satisfy the same lower-bound test; the correction chooses the
+larger eligible set. The label-subset case compares six predictions against
+known labels `{1, 3}`. The FAISS case uses 240 training rows and **600 separate,
+newly generated held-out rows**, target agreement 90%, and the same linear model
+candidate in both versions. OOD detection remains Euclidean and retains its
+existing threshold rule.
+
+The regression suite also covers staged-update rollback, retained recovery
+backups, explicit configuration, reproducible seeds, async tracing and
+cancellation, report escaping, path validation, and HTTP error/concurrency paths.

benchmarks/routing_reliability.py

@@ -0,0 +1,70 @@
+"""Small deterministic routing ablation; no model downloads or API calls.
+
+Run with `python benchmarks/routing_reliability.py`. Install the optional faiss
+extra to include the fit/reload case. Output is JSON suitable for comparing two
+checkouts with the same Python and dependencies.
+"""
+import json
+import tempfile
+from pathlib import Path
+
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+
+from tracer.api import fit, load_router
+from tracer.config import FitConfig
+from tracer.fit.pipeline import _calibrate_threshold, _predict
+
+
+def run():
+    scores = np.linspace(.50, .99, 36)
+    teacher = np.zeros(36, dtype=int)
+    predictions = teacher.copy()
+    predictions[:6] = 1
+    chosen = _calibrate_threshold(scores, predictions, teacher, .8)
+    X = np.array([[-3.], [-2.], [-1.], [1.], [2.], [3.]])
+    y = np.array([1, 1, 1, 3, 3, 3])
+    model = LogisticRegression(random_state=42).fit(X, y)
+    results = {
+        'small_calibration': {key: chosen[key] for key in
+                              ('coverage', 'teacher_agreement', 'teacher_agreement_lower')},
+        'subset_label_agreement': float((_predict(model, X)[0] == y).mean()),
+    }
+    try:
+        import faiss
+    except ImportError:
+        results['fit_reload'] = 'skipped: install tracer-llm[faiss]'
+        return results
+
+    rng = np.random.RandomState(24)
+    centers = rng.normal(size=(3, 8)) * 10
+    y_train = np.repeat(np.arange(3), 80)
+    y_test = np.repeat(np.arange(3), 200)
+    train = (centers[y_train] + rng.normal(size=(240, 8)) * .2).astype(np.float32)
+    heldout = (centers[y_test] + rng.normal(size=(600, 8)) * .2).astype(np.float32)
+    original = train.copy()
+    skip = ('logreg_c10', 'sgd_log', 'mlp_1h', 'mlp_2h', 'dt', 'rf', 'et', 'gbt', 'xgb')
+    with tempfile.TemporaryDirectory() as directory:
+        root = Path(directory)
+        traces = root / 'traces.jsonl'
+        traces.write_text(''.join(json.dumps({'input': str(i), 'teacher': str(label)}) + '\n'
+                                 for i, label in enumerate(y_train)))
+        fitted = fit(traces, root / 'model', train,
+                     FitConfig(frontier_targets=(.9,), skip_candidates=skip, verbose=False))
+        router = load_router(root / 'model')
+        output = router.predict_batch(heldout)
+        accepted = output['handled']
+        labels = np.asarray(output['labels'])
+        results['fit_reload'] = {
+            'faiss_version': faiss.__version__,
+            'training_input_unchanged': bool(np.array_equal(train, original)),
+            'calibration_coverage': fitted.manifest.coverage_cal,
+            'heldout_coverage': float(accepted.mean()),
+            'heldout_teacher_agreement': float((labels[accepted] == y_test[accepted].astype(str)).mean()) if accepted.any() else None,
+            'heldout_rows': len(y_test),
+        }
+    return results
+
+
+if __name__ == '__main__':
+    print(json.dumps(run(), indent=2))

docs/api.md

@@ -118,10 +118,17 @@ tracer.update(
 | `new_trace_path` | `str \| Path` | required | Path to NEW traces JSONL file |
 | `artifact_dir` | `str \| Path` | `".tracer"` | Existing artifact directory to update |
 | `new_embeddings` | `np.ndarray \| None` | `None` | Embeddings for the NEW traces only `(n_new, dim)` |
-| `config` | `FitConfig \| None` | `None` | If None, re-uses `target_teacher_agreement` from the existing manifest |
+| `config` | `FitConfig \| None` | `None` | If None, reuses the saved fit configuration. An explicit configuration takes precedence and is not mutated. |
 
 **Returns:** `FitResult` (same as `fit()`)
 
+The replacement model is fitted in a staging directory. Validation or fitting
+failures leave existing artifacts intact; a failed publication restores the
+previous directory. If restoration itself fails, the exception identifies the
+retained backup. Keep one writer per artifact directory and reload readers only
+after `update()` returns: directory publication is not a concurrent-reader or
+power-loss transaction. Allow disk space for the staged generation and backup.
+
 **Example:**
 
 ```python

docs/artifacts.md

@@ -48,7 +48,7 @@ The top-level summary. Human-readable and machine-readable.
 | `selected_method` | Which pipeline was deployed: `"global"`, `"l2d"`, `"rsb"`, or `null` (parity gate blocked) |
 | `coverage_cal` | Fraction of calibration-set traffic handled by surrogate. Proxy for production coverage. |
 | `teacher_agreement_cal` | On calibration-set handled traffic, fraction where surrogate agreed with teacher. Should be ≥ `target_teacher_agreement`. |
-| `n_retrains` | How many times this artifact has been updated via `tracer.update()` |
+| `n_retrains` | Number of fits in this artifact's history: 1 after initial fitting, incremented by each successful `tracer.update()` |
 
 **Null method:** If `selected_method` is `null`, coverage is 0%. All traffic routes to the teacher. This happens when no pipeline could reach the target teacher agreement on the calibration set. It's safe -- not an error.
 

docs/cold_start.md

@@ -99,7 +99,7 @@ As your LLM handles more production traffic, append new traces and re-fit:
 
 ```python
 # After collecting more traces
-result = tracer.update("new_traces.jsonl", ".tracer", embeddings=X_new)
+result = tracer.update("new_traces.jsonl", ".tracer", new_embeddings=X_new)
 ```
 
 Each update round adds deferred inputs (the ones the surrogate wasn't confident about) back into the training pool, automatically expanding coverage over time. This is TRACER's core feedback loop.

docs/javascript.md

@@ -94,6 +94,11 @@ tracer.serve(".tracer", host="127.0.0.1", port=8000)
 
 Run this as a sidecar next to your Node app, same server, same docker-compose, same machine. This example binds to `127.0.0.1:8000`. For a container sidecar, bind to `0.0.0.0` inside the container and limit network access to your application.
 
+The server defaults to loopback, accepts JSON bodies up to 8 MiB, and uses a
+10-second socket timeout. It handles requests concurrently. Invalid input returns
+HTTP 400. It provides no authentication or CORS headers; for browser or remote
+access, configure authentication, TLS, and CORS at your reverse proxy.
+
 ---
 
 ## Predicting from your JS app
@@ -168,7 +173,7 @@ Every deferred input that reaches your LLM is a new trace. Accumulate them and r
 # Run this script on a schedule; supply matching embeddings or a sibling .npy.
 import tracer
 
-tracer.update("new_traces.jsonl", embeddings=X_new)
+tracer.update("new_traces.jsonl", new_embeddings=X_new)
 ```
 
 Then restart the `serve_policy.py` process to pick up the updated policy. Coverage typically grows from ~84% at day 1 to 90%+ within a week of production traffic.