Add the Tacit evaluation harness and results with escalation

- scripts/eval_tacit.py: JevBench public set and bev-decision test through Tacit against vllm serve,
  one forward or adaptive; every README Tacit number now comes from it (bench/results_tacit/2026-10-06/)
- README (EN/zh): adaptive results for Tacit-9B and Tacit-4B
- Tacit: first_level=0 for score levels numbered from 0

Author: MorrisZJ

AGENTS.md

@@ -2,7 +2,7 @@
 
 ## Ground rules
 
-1. **No fabricated data, ever.** Published numbers come from committed result JSON under `bench/results_*/<date>/` (the runs behind the shipped tables are `bench/results_v01/2026-09-22/`, `bench/results_layout/2026-09-27/` and `bench/results_tacit/2026-10-05/`). If a run did not happen, the cell is empty. Every number in prose names its JSON, and `tests/test_readme_numbers.py` checks the README tables against theirs.
+1. **No fabricated data, ever.** Published numbers come from committed result JSON under `bench/results_*/<date>/` (the runs behind the shipped tables are `bench/results_v01/2026-09-22/`, `bench/results_layout/2026-09-27/` and `bench/results_tacit/2026-10-06/`). If a run did not happen, the cell is empty. Every number in prose names its JSON, and `tests/test_readme_numbers.py` checks the README tables against theirs.
 2. **No hidden generation.** `Decider` never samples tokens. `Tacit` generates only for an escalated decision (`adaptive=True`), and every escalation is reported (`route="cot"`) and counted against the cap. Any other code path that samples tokens is a bug.
 3. **Level and route are mandatory.** Every `Decision` carries `level` (`raw` / `L0`); every Tacit decision carries `route` (`one_forward` / `cot`). Tests assert both.
 4. **Backends are thin.** A backend implements `next_token_logprobs(prompts, token_ids)`; the transformers backend adds the optional methods the Decider probes for (`score_shared` for shared-prefix scoring, `hidden_states` / `hidden_states_to` for reading an intermediate block). Nothing else goes in `anyjev/backends/`: debiasing and calibration live above the backend and are tested once. Tacit's engines live in `anyjev/tacit.py` and implement `read` and `reason_read`.

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## Unreleased
+
+- **New: `scripts/eval_tacit.py`, the evaluation harness behind the README's Tacit numbers.** It decides
+  every item of the JevBench public set (from a checkout of github.com/fstandhartinger/jevbench) or the
+  bev-decision test split through `Tacit` against one or more `vllm serve` replicas, one forward or
+  adaptive, in blocks of 1,000 decisions (one window of the escalation cap each), and writes the summary
+  JSON the README tables are read from. The item mapping is the one the published numbers used (checked
+  item by item on all 46,551 items). `pip install "anyjev[eval]"` adds what it needs.
+- **README: results with escalation.** Tacit-9B and Tacit-4B at the default `adaptive=True` settings;
+  every Tacit number now comes from `bench/results_tacit/2026-10-06/` (`scripts/eval_tacit.py`), which
+  replaces `bench/results_tacit/2026-10-05/one_forward.json` (the maintainers' transformers evaluation).
+  Through `vllm serve` one forward stays within bf16 noise of that evaluation on bev-decision (−31 to
+  +80 decisions of 46,320) and differs by at most three items of 231 on JevBench (Tacit-8B 0.749 → 0.736,
+  Tacit-1.7B 0.623 → 0.619; the other three unchanged). With escalation: Tacit-9B JevBench 0.887, bev-decision
+  0.755; Tacit-4B 0.758 and 0.704, with 14–18% of decisions sent to reasoning.
+- `Tacit`: a `score` decision can number its levels from 0 (`first_level=0`, also in the gateway's
+  JSON), for rubrics that count from 0; the prompt shows and reads `0..K-1`. Default unchanged (1).
+
 ## 0.3.0 (2026-10-06)
 
 AnyJev now has two directions: **training-free** (`Decider`, levels `raw` and `L0`, any open LLM) and

README.md

@@ -51,7 +51,7 @@ d["answer"], d["probs"], d["route"]    # an option, {option: probability}, "one_
 ```
 
 `kind` is `"choice"` (default), `"yes_no"` or `"score"` (`options` are ordered levels, lowest
-first). `decide_batch([...])` takes a list of such dicts.
+first, numbered from 1; `first_level=0` numbers them from 0). `decide_batch([...])` takes a list of such dicts.
 
 **Send the hardest decisions to reasoning, with a cap.**
 
@@ -161,22 +161,24 @@ automate at ≤5% error goes from **7.7% to 46.3%**, with no labels at all.
 
 <div align="center">
 
-| model | base | JevBench public (231) | bev-decision test (46,320) |
-|:--|:--|:--:|:--:|
-| [Tacit-9B](https://huggingface.co/morriszjm/Tacit-9B) | Qwen3.5-9B | **0.823** | **0.725** |
-| [Tacit-8B](https://huggingface.co/morriszjm/Tacit-8B) | Qwen3-8B | 0.749 | 0.662 |
-| [Tacit-4B](https://huggingface.co/morriszjm/Tacit-4B) | Qwen3-4B | 0.723 | 0.663 |
-| [Tacit-2B](https://huggingface.co/morriszjm/Tacit-2B) | Qwen3.5-2B | 0.671 | 0.626 |
-| [Tacit-1.7B](https://huggingface.co/morriszjm/Tacit-1.7B) | Qwen3-1.7B | 0.623 | 0.598 |
+| model | base | JevBench public (231) | adaptive | bev-decision test (46,320) | adaptive |
+|:--|:--|:--:|:--:|:--:|:--:|
+| [Tacit-9B](https://huggingface.co/morriszjm/Tacit-9B) | Qwen3.5-9B | 0.823 | **0.887** <sub>15.2%</sub> | 0.727 | **0.755** <sub>18.1%</sub> |
+| [Tacit-8B](https://huggingface.co/morriszjm/Tacit-8B) | Qwen3-8B | 0.736 | — | 0.663 | — |
+| [Tacit-4B](https://huggingface.co/morriszjm/Tacit-4B) | Qwen3-4B | 0.723 | **0.758** <sub>14.3%</sub> | 0.663 | **0.704** <sub>17.8%</sub> |
+| [Tacit-2B](https://huggingface.co/morriszjm/Tacit-2B) | Qwen3.5-2B | 0.671 | — | 0.626 | — |
+| [Tacit-1.7B](https://huggingface.co/morriszjm/Tacit-1.7B) | Qwen3-1.7B | 0.619 | — | 0.598 | — |
 
-<sub>Accuracy with one forward pass per decision (`adaptive=False`), the options in the benchmark's
-order, on every item of both test sets ([JevBench](https://github.com/fstandhartinger/jevbench) public
-set; [bev-decision](https://huggingface.co/datasets/avbiswas/bev-decision) test split) ·
-`bench/results_tacit/2026-10-05/one_forward.json`</sub>
+<sub>Accuracy on every item of both sets ([JevBench](https://github.com/fstandhartinger/jevbench) public
+set; [bev-decision](https://huggingface.co/datasets/avbiswas/bev-decision) test split), in the order they
+are stored. First column: one forward pass per decision (`adaptive=False`). Adaptive: `adaptive=True` at
+the defaults (`tau=0.5`, at most 20% of every 1,000 decisions), with the share of decisions sent to
+reasoning in small print. Served by `vllm serve` (vLLM 0.17.1), run with `scripts/eval_tacit.py` · `bench/results_tacit/2026-10-06/`</sub>
 
 </div>
 
-Results with escalation (`adaptive=True`) will be published together with the evaluation harness.
+On average, escalation adds 567 (JevBench) and 458 (bev-decision) generated tokens per decision for Tacit-9B,
+and 305 and 187 for Tacit-4B. Results with escalation for Tacit-8B, Tacit-2B and Tacit-1.7B come next.
 
 ## 🧠 How it works
 
@@ -202,7 +204,8 @@ Every Tacit decision carries its `route`.
 - [x] `choice`, `noul` and `score` from one prefill; L0 with zero labels
 - [x] The rotation budget: L0 at about 7 of 18 rotations, with a stopping threshold certified without labels
 - [x] **Tacit-1.7B, 2B, 4B, 8B and 9B** on Hugging Face; `Tacit` in the library on transformers, vLLM and `vllm serve`, with capped escalation to reasoning and an HTTP gateway
-- [ ] 🚧 Results with escalation on JevBench and bev-decision, released with the evaluation harness
+- [x] The evaluation harness (`scripts/eval_tacit.py`); results with escalation for Tacit-9B and Tacit-4B
+- [ ] 🚧 Results with escalation for Tacit-8B, Tacit-2B and Tacit-1.7B
 - [ ] 🚧 Label-free early exit: read a decision from part of the model's depth, choosing the depth by agreement with the full model
 - [ ] **Agent-loop evaluation**: the same decisions inside a real agent
 - [ ] More log-prob backends (SGLang, llama.cpp, MLX, Ollama), span readout beyond 26 options

README.zh-CN.md

@@ -50,7 +50,7 @@ d = tacit.decide(state="Customer: my package was due last Monday and it still ha
 d["answer"], d["probs"], d["route"]    # 一个选项、{选项: 概率}、"one_forward" 或 "cot"
 ```
 
-`kind` 可以是 `"choice"`（默认）、`"yes_no"` 或 `"score"`（这时 `options` 是从低到高排好的档位）。
+`kind` 可以是 `"choice"`（默认）、`"yes_no"` 或 `"score"`（这时 `options` 是从低到高排好的档位，从 1 开始编号；`first_level=0` 则从 0 开始）。
 `decide_batch([...])` 接收由这样的 dict 组成的列表。
 
 **把最难的决策交给推理，并设上限。**
@@ -152,21 +152,22 @@ vLLM 上每秒决策数是原来的 2.2×，transformers 上是 2.3×–2.7×，
 
 <div align="center">
 
-| 模型 | 基座 | JevBench 公开集（231） | bev-decision 测试集（46,320） |
-|:--|:--|:--:|:--:|
-| [Tacit-9B](https://huggingface.co/morriszjm/Tacit-9B) | Qwen3.5-9B | **0.823** | **0.725** |
-| [Tacit-8B](https://huggingface.co/morriszjm/Tacit-8B) | Qwen3-8B | 0.749 | 0.662 |
-| [Tacit-4B](https://huggingface.co/morriszjm/Tacit-4B) | Qwen3-4B | 0.723 | 0.663 |
-| [Tacit-2B](https://huggingface.co/morriszjm/Tacit-2B) | Qwen3.5-2B | 0.671 | 0.626 |
-| [Tacit-1.7B](https://huggingface.co/morriszjm/Tacit-1.7B) | Qwen3-1.7B | 0.623 | 0.598 |
+| 模型 | 基座 | JevBench 公开集（231） | 分流 | bev-decision 测试集（46,320） | 分流 |
+|:--|:--|:--:|:--:|:--:|:--:|
+| [Tacit-9B](https://huggingface.co/morriszjm/Tacit-9B) | Qwen3.5-9B | 0.823 | **0.887** <sub>15.2%</sub> | 0.727 | **0.755** <sub>18.1%</sub> |
+| [Tacit-8B](https://huggingface.co/morriszjm/Tacit-8B) | Qwen3-8B | 0.736 | — | 0.663 | — |
+| [Tacit-4B](https://huggingface.co/morriszjm/Tacit-4B) | Qwen3-4B | 0.723 | **0.758** <sub>14.3%</sub> | 0.663 | **0.704** <sub>17.8%</sub> |
+| [Tacit-2B](https://huggingface.co/morriszjm/Tacit-2B) | Qwen3.5-2B | 0.671 | — | 0.626 | — |
+| [Tacit-1.7B](https://huggingface.co/morriszjm/Tacit-1.7B) | Qwen3-1.7B | 0.619 | — | 0.598 | — |
 
-<sub>每个决策一次前向（`adaptive=False`），选项按基准里给的顺序，两个测试集全部样本上的准确率
-（[JevBench](https://github.com/fstandhartinger/jevbench) 公开集；[bev-decision](https://huggingface.co/datasets/avbiswas/bev-decision) 测试集）·
-`bench/results_tacit/2026-10-05/one_forward.json`</sub>
+<sub>两个测试集全部样本上的准确率（[JevBench](https://github.com/fstandhartinger/jevbench) 公开集；
+[bev-decision](https://huggingface.co/datasets/avbiswas/bev-decision) 测试集），按数据集存储顺序。第一列：每个决策一次前向
+（`adaptive=False`）。分流：`adaptive=True`，默认参数（`tau=0.5`，每 1,000 个决策里最多 20% 转去推理），小字是转去推理的比例。
+用 `vllm serve`（vLLM 0.17.1）服务，`scripts/eval_tacit.py` 运行 · `bench/results_tacit/2026-10-06/`</sub>
 
 </div>
 
-打开转推理（`adaptive=True`）的结果，会和评测代码一起发布。
+打开转推理后，平均每个决策多生成的 token：Tacit-9B 在 JevBench 上 567 个、bev-decision 上 458 个；Tacit-4B 分别是 305 和 187 个。Tacit-8B、2B、1.7B 打开转推理的结果随后补上。
 
 ## 🧠 怎么读出一个决策
 
@@ -191,7 +192,8 @@ vLLM 上每秒决策数是原来的 2.2×，transformers 上是 2.3×–2.7×，
 - [x] `choice`、`noul`、`score` 一次 prefill 读出；零标签的 L0
 - [x] 旋转预算：L0 平均读 18 次轮换里的 7 次左右，停止门槛不用标签就能证明
 - [x] **Tacit-1.7B、2B、4B、8B、9B** 发布在 Hugging Face；库里的 `Tacit` 支持 transformers、vLLM 和 `vllm serve`，带有上限的转推理和 HTTP 网关
-- [ ] 🚧 JevBench 和 bev-decision 上打开转推理的结果，和评测代码一起发布
+- [x] 评测代码（`scripts/eval_tacit.py`）；Tacit-9B 和 Tacit-4B 打开转推理的结果
+- [ ] 🚧 Tacit-8B、2B、1.7B 打开转推理的结果
 - [ ] 🚧 无标签的提前退出：只用模型的一部分层读出决策，层数按与完整模型的一致率来选
 - [ ] **Agent 循环里的评测**：把同样的决策放进真实的 agent 里
 - [ ] 更多 log-prob 后端（SGLang、llama.cpp、MLX、Ollama），以及超过 26 个选项的 span 读法

ROADMAP.md

@@ -30,13 +30,16 @@ The version in brackets is the one an item first shipped in.
   Opt-in until the L0 tables are regenerated at the new defaults.
 - [x] **Tacit models on Hugging Face** (0.3). Tacit-1.7B, 2B, 4B, 8B and 9B, one forward per decision,
   in the [Tacit collection](https://huggingface.co/collections/morriszjm/tacit-6ac41d0b50af9e5417c5c234);
-  accuracy on JevBench (public, 231) and bev-decision (test, 46,320) in
-  `bench/results_tacit/2026-10-05/one_forward.json`.
+  accuracy on JevBench (public, 231) and bev-decision (test, 46,320) in `bench/results_tacit/2026-10-06/`.
 - [x] **`Tacit` in the library** (0.3). `anyjev.Tacit` on transformers, in-process vLLM and a running
   `vllm serve`; `adaptive=True` sends decisions whose top-two margin is under `tau` to the model's own
   reasoning and reads the answer as a label distribution afterwards; at most `max_cot_share` of the
   last `cot_window` decisions escalate. `python -m anyjev.serve` puts an HTTP gateway with one shared
   cap in front of a vLLM server. Tested on CPU with a fake engine (`tests/test_tacit.py`).
+- [x] **The evaluation harness, and results with escalation** (unreleased). `scripts/eval_tacit.py` decides
+  every item of the JevBench public set and the bev-decision test split through `Tacit` against `vllm serve`,
+  one forward or adaptive, and writes the JSON the README tables are read from (`bench/results_tacit/2026-10-06/`):
+  one forward for all five Tacit models, adaptive for Tacit-9B and Tacit-4B.
 - [x] **Releases on PyPI** (0.1.0, 0.2.0). Each was installed from the index into a fresh venv, with
   `__version__` matching `pyproject.toml`.
 
@@ -46,11 +49,9 @@ test-time adaptation, `observe`), the shipped heads, the demos and the benchmark
 
 ## Now
 
-- [ ] **Results with escalation, and the evaluation harness.** Why: the published Tacit numbers are one
-  forward; `adaptive=True` is the setting most deployments will use, and every number must be
-  reproducible from shipped code. *Done:* the harness runs JevBench and bev-decision from one command
-  per model, its JSON is committed with the share of decisions escalated and the generated tokens per
-  decision, and both READMEs quote it.
+- [ ] **Results with escalation for Tacit-8B, Tacit-2B and Tacit-1.7B.** Why: the READMEs show escalation
+  for Tacit-9B and Tacit-4B only. *Done:* `scripts/eval_tacit.py --adaptive` on both sets for each, the JSON
+  in `bench/results_tacit/`, and the empty cells filled.
 - [ ] **Label-free early exit** (0.3.1). Why: a decision often does not need the model's full depth. A
   map into the final layer's basis, fitted without labels, reads the decision from a truncated
   forward, and the depth is chosen by agreement with the full model's answer. The hidden-state

THIRD_PARTY.md

@@ -2,8 +2,8 @@
 
 | What | Where | License | Used for |
 |---|---|---|---|
-| JevBench, public set | https://github.com/fstandhartinger/jevbench (`datasets/public`: easy, original, hard; 231 items) | MIT | Tacit results, `bench/results_tacit/` |
-| bev-decision | https://huggingface.co/datasets/avbiswas/bev-decision (default config, test split) | not stated on the dataset card (`license: unknown`) | Tacit results, `bench/results_tacit/`; evaluation only |
+| JevBench, public set | https://github.com/fstandhartinger/jevbench (`datasets/public`: easy, original, hard; 231 items) | MIT | Tacit results, `scripts/eval_tacit.py` (from a local checkout), `bench/results_tacit/` |
+| bev-decision | https://huggingface.co/datasets/avbiswas/bev-decision (default config, test split) | not stated on the dataset card (`license: unknown`) | Tacit results, `scripts/eval_tacit.py`, `bench/results_tacit/`; evaluation only |
 | banking77 (mteb parquet mirror) | https://huggingface.co/datasets/mteb/banking77 | CC-BY-4.0 | the raw / L0 table (`banking20`, `bench/results_v01/`) and the README GIF |
 | 20 Newsgroups (SetFit mirror) | https://huggingface.co/datasets/SetFit/20_newsgroups | see dataset card | `newsgroups` in `bench/results_v01/` and `bench/results_layout/` |
 | deepset/prompt-injections | https://huggingface.co/datasets/deepset/prompt-injections | Apache-2.0 | `injection` in `bench/results_v01/` |

anyjev/tacit.py

@@ -43,7 +43,14 @@
 import numpy as np
 
 from anyjev.question import Question
-from anyjev.readout import DEFAULT_SYSTEM, build_prompt, label_ids_for_perm, render_chat, resolve_labels
+from anyjev.readout import (
+    DEFAULT_SYSTEM,
+    build_prompt,
+    label_ids_for_perm,
+    map_label_tokens,
+    render_chat,
+    resolve_labels,
+)
 
 COT_SYSTEM = "You are a careful assistant that answers questions about a given context."
 KINDS = {"choice": "choice", "yes_no": "noul", "noul": "noul", "score": "score"}
@@ -349,10 +356,12 @@ def from_pretrained(cls, repo: str, *, engine: str = "transformers", dtype: str
 
     # ------------------------------------------------------------------ public API
     def decide(self, state: str, question: str, options: Optional[Sequence[str]] = None,
-               kind: str = "choice") -> Dict[str, Any]:
+               kind: str = "choice", first_level: int = 1) -> Dict[str, Any]:
         """One decision. kind: "choice" (pick one of `options`), "yes_no", or "score" (`options` are
-        ordered levels, lowest first)."""
-        return self.decide_batch([dict(state=state, question=question, options=options, kind=kind)])[0]
+        ordered levels, lowest first, numbered from `first_level`: 1 by default, 0 for a rubric that
+        counts from 0)."""
+        return self.decide_batch([dict(state=state, question=question, options=options, kind=kind,
+                                       first_level=first_level)])[0]
 
     def decide_batch(self, items: Sequence[Dict[str, Any]]) -> List[Dict[str, Any]]:
         """Each item: dict(state=..., question=..., options=[...], kind=...). Within a batch the least
@@ -414,7 +423,15 @@ def _normalise(self, it: Dict[str, Any]) -> Dict[str, Any]:
         if "question" not in it:
             raise ValueError("a decision needs a question")
         q = make_question(it["question"], it.get("options"), it.get("kind", "choice"))
-        labels, ids = resolve_labels(self.tok, q)
+        first = it.get("first_level", 1)
+        if first not in (0, 1):
+            raise ValueError("first_level must be 0 or 1")
+        if q.kind == "score" and first == 0:
+            # levels shown and read as 0..K-1, for rubrics that number them that way
+            labels = [str(j) for j in range(q.k)]
+            ids = map_label_tokens(self.tok, labels)
+        else:
+            labels, ids = resolve_labels(self.tok, q)
         return dict(state=str(it.get("state", "")), q=q, labels=labels,
                     ids=label_ids_for_perm(q, ids, list(range(q.k))))