|
| 1 | +# Troubleshooting |
| 2 | + |
| 3 | +Three snags people hit most often, what causes them, and how to fix each. |
| 4 | + |
| 5 | +## `selected_method` is `null` (no policy deployed) |
| 6 | + |
| 7 | +**Symptom.** After `tracer.fit(...)`, the manifest shows `selected_method: null`, |
| 8 | +the `FitResult` notes say "No deployable pipeline met the target teacher-parity |
| 9 | +threshold", and `tracer.load_router(...)` then raises `FileNotFoundError` on |
| 10 | +`pipeline.joblib` (the file is only written when a policy actually deploys). |
| 11 | +`tracer serve` reports `method=none` on `/health`. |
| 12 | + |
| 13 | +**Why.** TRACER only deploys a surrogate that can hold your parity bar. If no |
| 14 | +candidate reaches `target_teacher_agreement` while still covering at least |
| 15 | +`min_deploy_coverage` of traffic, nothing is deployed on purpose, rather than |
| 16 | +shipping a policy that silently disagrees with your teacher. |
| 17 | + |
| 18 | +Common reasons: |
| 19 | + |
| 20 | +- `target_teacher_agreement` set too high for the data (for example 0.99 on |
| 21 | + noisy teacher labels). |
| 22 | +- `min_deploy_coverage` set so high that the only parity-respecting policies are |
| 23 | + filtered out. |
| 24 | +- Too few traces, or embeddings that do not separate the classes. |
| 25 | +- Heavy teacher label noise, which caps the agreement any surrogate can reach. |
| 26 | + |
| 27 | +**Fix.** |
| 28 | + |
| 29 | +1. Open `.tracer/frontier.json`. Each entry lists `best_coverage` and `best_ta` |
| 30 | + per target. This tells you the best parity actually achievable and at what |
| 31 | + coverage, so you can pick a realistic bar instead of guessing. |
| 32 | +2. Lower the bar to something the frontier shows is reachable: |
| 33 | + ```python |
| 34 | + tracer.fit("traces.jsonl", embeddings=X, |
| 35 | + config=tracer.FitConfig(target_teacher_agreement=0.90, |
| 36 | + min_deploy_coverage=0.02)) |
| 37 | + ``` |
| 38 | +3. Add more traces, or improve the embedding model so classes separate better. |
| 39 | +4. If `best_ta` is capped well below your target across all candidates, suspect |
| 40 | + teacher label noise and audit the traces before re-fitting. |
| 41 | + |
| 42 | +## Coverage drops between fits |
| 43 | + |
| 44 | +**Symptom.** A re-fit (`tracer.update(...)`, or `tracer.fit` on a larger trace |
| 45 | +set) reports a lower `coverage_cal` than the previous run at the same |
| 46 | +`target_teacher_agreement`. |
| 47 | + |
| 48 | +**Why.** This is expected, not a regression. TRACER holds parity fixed and lets |
| 49 | +coverage float. When new traffic is harder, more diverse, noisier, or introduces |
| 50 | +labels the surrogate has not seen, the only way to keep matching the teacher at |
| 51 | +your target is to defer more inputs. Coverage falls so that parity stays honest. |
| 52 | + |
| 53 | +**Fix.** Usually nothing: a lower coverage at the same parity is the system |
| 54 | +adapting correctly. To understand the change: |
| 55 | + |
| 56 | +- Check `.tracer/frontier.json` to see the new coverage versus parity curve. |
| 57 | +- Read the qualitative report (`.tracer/qualitative_report.json`). Its |
| 58 | + `temporal_deltas` show, per label, how the handled rate moved between fits, so |
| 59 | + you can see which slices got harder. |
| 60 | +- If new labels expanded the space, that alone can lower coverage; confirm the |
| 61 | + label count grew. |
| 62 | +- If cost matters more than parity for the new traffic, lower |
| 63 | + `target_teacher_agreement` to trade some agreement back for coverage. |
| 64 | + |
| 65 | +## Embedding dimension mismatch |
| 66 | + |
| 67 | +**Symptom.** |
| 68 | +``` |
| 69 | +ValueError: Embedding dimension mismatch: expected 384, got 768. |
| 70 | +``` |
| 71 | +raised from `router.predict(...)` / `router.predict_batch(...)`, or surfaced as |
| 72 | +a `500` from the server's `/predict` endpoint. |
| 73 | + |
| 74 | +**Why.** The router records the embedding width it was fitted on in |
| 75 | +`manifest.embedding_dim` and checks every incoming vector against it. A mismatch |
| 76 | +means the embeddings at route time were produced differently from the ones used |
| 77 | +at fit time, almost always a different embedder model. |
| 78 | + |
| 79 | +**Fix.** Use the same embedder at fit time and route time. |
| 80 | + |
| 81 | +- If you fit on precomputed embeddings, route with the same model that produced |
| 82 | + them. `all-MiniLM-L6-v2` and `BAAI/bge-small-en-v1.5` are 384-dim, while |
| 83 | + `BAAI/bge-base-en-v1.5` is 768-dim; fitting on one width and routing on the |
| 84 | + other triggers this error. |
| 85 | + ```python |
| 86 | + embedder = tracer.Embedder.from_sentence_transformers("BAAI/bge-small-en-v1.5") |
| 87 | + router = tracer.load_router(".tracer", embedder=embedder) # same model as fit |
| 88 | + ``` |
| 89 | +- For the HTTP server, the vector in `{"embedding": [...]}` must have exactly |
| 90 | + `manifest.embedding_dim` values. Check `GET /health` and the manifest if you |
| 91 | + are unsure of the expected width. |
| 92 | +- A dimension match with poor accuracy is a different problem: make sure the |
| 93 | + `normalize` setting also matches between fit and route. |
0 commit comments