Copy-Inflation in Search Agents — diagnosis, and a voting weight that survives it

Code and data for "Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents via Retrieval Grounding", Findings of the ACL: EMNLP 2026.

Paper: https://arxiv.org/abs/2608.24024

The finding

When an LLM acts as a search agent, 78–92% of the tokens it generates are copied from documents sitting in its own context — and those copied tokens carry systematically inflated log-probabilities. We call this copy inflation, and it is measurable in every model family we tested (GPT-OSS, Tongyi, GLM, Kimi, MiniMax, DeepSeek) across four corpora.

The consequence is that a search agent's confidence stops discriminating. Its scores flatten within a question, which is exactly the spread that weighted voting consumes, so confidence-weighted voting collapses toward plain majority voting. Two interventions pin down the direction of causation:

Intervention	Result
Recompute confidence on non-copy tokens only	within-question spread returns in 8/10 cells (up to 2.4×), but discrimination does not — AUC falls in 9/10
Remove the documents and re-score the same tokens	copied tokens lose ~2× more log-probability than non-copy ones (−0.221 vs −0.106; larger in 82/100 rollouts)

The contamination resists internal repair: the discriminative information lives in precisely the tokens a repair would remove. That matters beyond voting — early stopping, routing, abstention, confidence-shaped rewards and log-probability-based hallucination detection all read the same broken signal.

diagnostics/ is a standalone toolkit for measuring this on your own agent. It needs no model, no API key, and no retraining — point it at your rollouts and it reports the copy fraction, the inflation gap, and the within-question variance share of any candidate weight, which together predict in advance whether confidence-based methods will work on your setup.

The method that follows from it

If the signal inside the context is contaminated, read one from outside it. Retrieval-Grounded Voting (RGV) weights each rollout by the lexical overlap between its answer prose and the documents that rollout actually retrieved:

w_i = max_{d in D_i}  |T(P_i) ∩ T(d)| / |T(P_i)|

No log-probabilities, no extra LLM call, ~0.3 ms per rollout on one CPU thread.

What is in this release

Voting + scoring	rgv/ — the RGV scorer and weighted-vote protocol, the frozen extractors, the archive builder, the redaction tool
Agents	agents/, search/, launchers/ — the ReAct agent, search services, and one launcher per cell
Grading	grading/ — the Qwen3-32B judge
Diagnostics	diagnostics/ — standalone copy-inflation toolkit: per-token copy classifier, within-question variance-share meter, docs-removed rescoring
Experiments	experiments/ — every response-period experiment (weak retrieval, distractor injection, length controls, Best-of-N, embedding baseline, prompt arms)
Figures	figures/ — regeneration scripts for the paper's figures
Data	34 cells on HuggingFace: full trajectories, per-token log-probabilities, judge outputs (22 GB)

Data: https://huggingface.co/datasets/kookhh0827/copy-inflation-search-agents (34 cells · 125k rollout records · 245 GB raw, 23 GB compressed)

Quick start

Download one cell and score it. The scorer needs nothing but NumPy.

pip install numpy
huggingface-cli download kookhh0827/copy-inflation-search-agents \
  --repo-type dataset --include "data/frames/gpt-oss-120b/*" --local-dir cell
cd cell/data/frames/gpt-oss-120b
zstd -dc runs.tar.zst  | tar -x     # trajectories -> r0/ .. r7/
zstd -dc evals.tar.zst | tar -x -C ../..   # judge verdicts

RGV itself is a few lines: for each rollout, take the answer prose and the documents that rollout retrieved, and measure how much of the prose is anchored in the best-matching document.

w_i = max(len(T(prose) & T(d)) / len(T(prose)) for d in docs)

rgv/vote.py implements that plus the weighted-vote protocol, and reads the directory layout the archives expand to:

python rgv/vote.py --runs-dir runs --evals-dir evals --stats-dir stats

To score your own agent's rollouts you only need two things per rollout: the final answer prose, and the text of the documents that rollout retrieved. See rgv/README.md for the record format.

A note on the numbers

This release is built for analysis of what happens inside these agents — per-token log-probabilities, tool outputs, and judge verdicts. rgv/vote.py ships one fixed scorer configuration (prose-recall, max over documents, strict answer clustering); the paper's Appendix E reports how the overlap function is varied. Numbers you compute from these archives depend on the extraction and clustering choices you make, so treat them as your own measurement rather than as a reproduction of a specific table.

Reproducing from scratch

pip install -r requirements-rollout.txt          # agent + search service
cp .env.example .env                             # add Fireworks + Serper keys

bash scripts/start_web_search.sh &               # BrowseComp / GAIA / FRAMES
bash scripts/start_corpus_search.sh &            # BrowseComp-Plus (FAISS)
bash launchers/frames/launch_oss-120b.sh         # 8 rollouts x 150 questions

pip install -r requirements-grading.txt          # vLLM + Qwen3-32B judge
bash scripts/grade_all.sh frames oss-120b

python rgv/vote.py --runs-dir runs/frames/oss-120b \
                   --evals-dir evals/frames/oss-120b \
                   --stats-dir stats/frames/oss-120b

Rollouts are resumable: re-running a launcher continues from the last checkpoint. Hardware we used: 8× NVIDIA RTX PRO 6000 for local Tongyi and judge serving; every other model via the Fireworks API.

Copy-inflation diagnostic

The diagnostic is standalone — point it at any agent's rollouts to see whether its confidence signal is contaminated.

python diagnostics/copy_mask.py   --runs-dir runs/... --stats-dir stats/...
python diagnostics/copy_report.py --cell frames/gpt-oss-120b

It reports the copy fraction, the copy-vs-non-copy log-probability gap, and the within-question variance share of each candidate weight — the three quantities that predict, in advance, whether confidence voting will work on a given setup.

Benchmarks we do not redistribute

GAIA is access-gated and its terms forbid resharing the validation split; HLE's authors ask that the benchmark not be re-uploaded. For those two we ship question identifiers only (data/gaia/README.md), and the released rollouts have their question and gold-answer fields redacted (rgv/redact.py). Everything the agent itself produced is intact. BrowseComp-Plus (MIT), BrowseComp (MIT), FRAMES (Apache-2.0) and HotpotQA (CC BY-SA 4.0) are shipped in full. Appendix O of the paper lists every artifact and its licence.

Licence

Code: Apache-2.0 (LICENSE). Released data: CC BY 4.0 for our own contributions — trajectories, scores, judge labels, annotations. Text returned by the environment (corpus passages, fetched web pages) remains under its original terms and is redistributed with its source identifiers retained.

Citation

@inproceedings{kook2026rgv,
  title     = {Beyond Confidence: Test-Time Scaling for Multi-Turn Search Agents
               via Retrieval Grounding},
  author    = {Kook, Hyunho and So, Junhyuk and Fu, Tianyu and
               Zheng, Haizhong and Chen, Beidi},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  eprint    = {2608.24024},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

Acknowledgments

The rollouts in this release were generated with inference credits provided by Fireworks AI. Eight parallel rollouts per question, across four benchmarks and five models, each a full multi-turn search trajectory with per-token log-probabilities retained — 128,800 rollouts in total — needed serving at a scale we could not have reached otherwise.