AutoResearchExam

This repository contains the 29 tasks from AutoResearchExam.

Run with the official harness or use your own harness.

Task areas

---
config:
  themeVariables:
    fontFamily: 'Arial, Helvetica, sans-serif'
    pie1: '#d8e5e4'
    pie2: '#efc8bc'
    pie3: '#e8ddbd'
    pie4: '#cbd9ec'
    pie5: '#e9d4dd'
    pie6: '#d6e0c8'
    pie7: '#dfd7ec'
    pieOpacity: 1
    pieStrokeColor: '#FFFFFF'
    pieStrokeWidth: '2px'
    pieOuterStrokeWidth: '0px'
    pieTitleTextSize: '22px'
    pieSectionTextSize: '16px'
    pieSectionTextColor: '#253445'
    pieLegendTextSize: '16px'
  pie:
    textPosition: 0.78
---
pie showData
    title Tasks by research area
    "Model training" : 8
    "Algorithms and optimization" : 6
    "Data engineering and curation" : 6
    "Systems and efficiency" : 4
    "Evaluation, calibration, and robustness" : 3
    "AI safety and alignment" : 1
    "Interpretability" : 1

Loading

View tasks by area →

Model training

Representation learning

• Compact encoding for a column with many categories
• Compact item embeddings for recommendations
• Short binary codes for image search

Reinforcement learning

• Advantage estimation for policy training
• Reliable control policy trainer

Predictive modeling

• Predict valid actions for a game policy
• Rank employee access requests from ID codes

Model merging and distillation

• Eight domain vision model merging

Algorithms and optimization

Statistical methods

• Estimate uncertainty in a streaming optimizer
• Estimate individual treatment effects
• Recover river flow links

Constrained optimization

• Dual market budgeted classification
• MILP branching policy
• Online chance constrained policy

Data engineering and curation

Imputation

• Budgeted imputation
• Sparse panel imputation

Data selection

• Unlabeled pool pruning
• Low discrepancy subset selection
• Group robust coreset selection
• Pretraining token budget selection

Systems and efficiency

Inference

• CPU language model decoding
• CPU decoder graph execution
• Video diffusion transformer caching

Compression

• SVDQuant PixArt Sigma quantization

Evaluation, calibration, and robustness

Metric estimation

• Shortest valid confidence interval
• Label efficient risk estimation

Robustness

• Adversarially robust CIFAR 10 training

AI safety and alignment

• Candidate token loss ranker

Interpretability

• Sparse dictionary coding

Run with the official harness

AutoResearchExam-Terminus is the official harness for timed runs with repeated experiments and public validation feedback.

Install uv, start Docker, and configure your model provider's credentials (OPENAI_API_KEY for the OpenAI example below). Docker's default preflight checks for 32000 MiB of free space per trial for retained artifacts, plus that task's storage requirement, summed across all selected trials on the jobs host. Download the tasks and harness into separate folders:

mkdir autoresearch-exam-run && cd autoresearch-exam-run
git clone https://github.com/bespokelabsai/AutoResearchExam.git github-tasks
git clone https://github.com/bespokelabsai/AutoResearchExam-Terminus.git harness
cd harness
uv sync --python 3.12 --extra modal

Run one task with the default 24-hour research budget on local Docker. harbor run starts the full research and grading loop automatically:

uv run harbor run \
  --path ../github-tasks/cpu-decoder-graph-executor \
  --agent autoresearchexam-terminus \
  --model openai/gpt-5.6-sol \
  --env docker \
  --plugin autoresearch-exam \
  --pk max_iterations=1000 \
  --pk max_duration_seconds=86400 \
  --pk min_time_per_iteration=0 \
  --pk max_turns=10000 \
  --pk max_tokens=32000 \
  --pk reasoning_effort=max

Use --path ../github-tasks to run all tasks. The disk prerequisite above applies to all selected trials. This example uses the default benchmark settings: 1000 experiments, 10000 model turns, 24 hours, 32000 tokens per response, and max reasoning effort, matching our Sol and Astra runs.

If running on Modal, use --env modal and reduce max_duration_seconds to leave room for grading before its 24-hour sandbox timeout. For this example, use --pk max_duration_seconds=82200 (~23 hours). The chosen duration also sets the AUARC scoring window.

Find saved runs in Results, then compute AUARC.

Use your own harness

You can run the same tasks with Claude Code, Codex, or another agent harness. Keep the task environment, graders, and time budget the same. Select each checkpoint by public validation and measure its private test score on the same saved artifact. Report test performance, but never use test scores to select checkpoints. Keep private test data, scores, and logs outside the agent environment throughout the run and checkpoint selection.

Follow the custom harness guide to record timestamped scores and compute AUARC with the existing script.

Citation

If you use AutoResearchExam in your research, please cite:

@misc{ramesh2026autoresearchexam,
  author = {Ramesh, Anirudha and
            Devic, Siddartha and
            Garg, Shivank and
            Parulekar, Advait and
            Mahnot, Drish and
            Pimpalgaonkar, Shreyas and
            Suresh, Vishnu and
            Dimakis, Alex and
            Sathiamoorthy, Maheswaran},
  title = {{AutoResearchExam}},
  year = {2026},
  month = sep,
  url = {https://benchmarks.bespokelabs.ai/autoresearchexam/},
  note = {Published September 9, 2026}
}