Skip to content

REPRODUCE.md

Latest commit

 

History

History
123 lines (92 loc) · 4.44 KB

REPRODUCE.md

File metadata and controls

123 lines (92 loc) · 4.44 KB
Raw

Reproducing from the release

Three entry points, in increasing order of cost. Every command below was run end to end before this file was written.

1. Score one cell from the released data (~2 minutes)

Nothing to run, nothing to serve: download one cell, expand it, score it.

pip install numpy huggingface_hub zstandard
git clone https://github.com/kookhh0827/copy-inflation-search-agents.git
cd copy-inflation-search-agents

huggingface-cli download kookhh0827/copy-inflation-search-agents \
  --repo-type dataset --include "data/frames/gpt-oss-120b/*" --local-dir cell

cd cell/data/frames/gpt-oss-120b
for s in runs evals stats; do
  mkdir -p $s && zstd -dc $s.tar.zst | tar -x -C $s
done

Each archive expands to r0/ … r7/, one directory per rollout index:

runs/r0/run_<timestamp>.json          trajectory: tool calls + returned text,
                                      reasoning spans, final answer, usage
evals/r0/run_<timestamp>_eval.json    judge verdict, extracted answer, gold
stats/r0/run_<timestamp>.prob.json    per-token top-5 log-probabilities

That is exactly the layout the voter expects:

python ../../../../rgv/vote.py \
  --runs-dir runs --evals-dir evals --stats-dir stats
Loaded 150 questions

Method                    Accuracy
-----------------------------------
Single (per-rollout)         79.4%
Simple Majority              78.7%
RGV (prose-recall)           79.3%
Jaccard                      80.0%
DeepConf                     82.7%
Verb                         80.7%
Oracle (pass@8)              92.7%

n_questions=150, n_rollouts=1200

--stats-dir is what makes the DeepConf row possible; drop it and that row falls back to uniform weights. Numbers depend on the extraction and clustering choices the scorer makes — see rgv/vote.py, which is about 200 lines and has no dependency beyond NumPy.

2. Run the copy-inflation diagnostic on the same cell

This is the paper's central measurement, and it needs no model and no API key.

python diagnostics/copy_mask.py --runs-dir runs --stats-dir stats
python diagnostics/copy_report.py --cell frames/gpt-oss-120b

It reports, per cell:

  • copy fraction — how much of what the agent generated is a substring of its own retrieved documents (78–92% across the cells in the paper),
  • copy-inflation gap — mean log-probability of copied vs non-copy tokens,
  • within-question variance share — how much of a weight's spread lives inside a question rather than between questions. This is the quantity weighted voting consumes; when it collapses, confidence voting degenerates into majority voting.

Point it at your own agent's rollouts to find out whether its confidence signal is contaminated before you rely on it.

3. Generate rollouts from scratch (hours, needs keys and GPUs)

pip install -r requirements-rollout.txt
cp .env.example .env          # FIREWORKS_API_KEY, SERPER_KEY_ID

bash scripts/start_web_search.sh &        # BrowseComp / GAIA / FRAMES
bash scripts/start_corpus_search.sh &     # BrowseComp-Plus (FAISS index)
bash launchers/frames/launch_oss-120b.sh  # 8 rollouts x 150 questions

pip install -r requirements-grading.txt   # vLLM + Qwen3-32B judge
bash scripts/grade_all.sh frames oss-120b

python rgv/vote.py --runs-dir runs/frames/oss-120b \
                   --evals-dir evals/frames/oss-120b \
                   --stats-dir stats/frames/oss-120b

Launchers are resumable: re-running one continues from the last checkpoint rather than repeating completed questions. Tongyi-DeepResearch is served locally (scripts/start_tongyi_vllm.sh, 8× RTX PRO 6000 in our setup); the other four models go through the Fireworks API.

Sampling parameters per model are in configs/models.yaml, and each launcher records what it actually used in the metadata field of every run file — so the released archives carry their own provenance.

Which cell to pick

rgv/manifest.py lists all 34 cells with their question counts. Two notes:

  • browsecomp-plus/tongyi-deepresearch is the 830-question pool used for the paper's deep-dive analyses, but it has no per-token log-probabilities on disk. For token-level work use browsecomp-plus/gpt-oss-120b or any FRAMES cell instead.
  • gaia/* and hle/* ship with question and gold-answer text redacted, since those benchmarks do not permit redistribution. Everything the agent produced is intact; join on qid against the official releases to recover the questions.