Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Turn underspecified scientific research instructions into executable rubrics for guided execution, verification, and targeted revision.
AutoSciRub is a plugin bundle for AI research agents. It packages the workflow from the paper Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents as installable skills that work across Codex, Claude Code, and OpenClaw.
The repository provides a general-purpose research workflow and optional benchmark-specific adaptations. The general workflow turns instructions, literature, and available data into an executable rubric. The ResearchClawBench adaptation specializes that workflow for the benchmark's task structure, public-source protocol recovery, and report/figure requirements, with configurable environments and execution budgets.
The trajectories, reports, and evaluation records from our main ResearchClawBench experiments are available on Hugging Face.
2026-10-17 — Added an optional ResearchClawBench skill bundle, with environment setup guidance, configurable budgets, and structural evidence checks. See the adaptation guide for installation, provenance, and evaluation scope; this release has not been benchmarked.
2026-09-03 — AutoSciRub reached 33.2 Pass@1 with Codex + GPT-5.6-Sol in the September 2026 ResearchClawBench leaderboard snapshot.
ResearchClawBench leaderboard snapshot recorded on 2026-09-03 04:05 UTC.
2026-08-13 — AutoSciRub achieved 2nd place on the ResearchClawBench leaderboard, with a 25.1 Pass@1 score using Codex + GPT-5.6-Terra.
- Rubric induction at inference time — builds a goal skeleton, grounds it in literature, and adapts to the data actually available in the task.
- Executable rubric contract — each criterion names the required analysis, evidence artifacts, and satisfaction condition.
- Criterion-level verification — verification reports which specific criteria are unmet and why, instead of a single score.
- Targeted revision loop — revisions are driven by failed criteria, not vague feedback.
- Configurable retrieval backends — built-in support for arXiv, OpenAlex, Semantic Scholar, and Tavily through a single script and a shared YAML config.
- General-purpose default workflow — adapts to your research project's paths, data, and prepared environments; optional bundles make benchmark-specific conventions explicit.
| Workflow | Intended use | What it provides |
|---|---|---|
| General AutoSciRub (default) | Research projects with their own instructions, data, and output conventions | Eight skills for rubric induction, literature grounding, data exploration, execution guidance, verification, and targeted revision |
| ResearchClawBench adaptation | ResearchClawBench task workspaces | Six specialized skills for instruction/source evidence mapping, complete experimental comparisons, report and figure verification, and bounded repair |
Both use the same rubric-guided research approach. Adaptations make a benchmark's input/output conventions and evidence requirements explicit. Research environments are prepared by the user; the ResearchClawBench guide explains how to configure Python interpreter paths and task budgets. Public task inputs and permitted literature provide the scientific context; hidden evaluation materials are excluded.
The bundles can coexist. Select autoscirub for the general workflow or
rcbench-workflow-controller for the adaptation, and follow that controller's
configuration and artifact contract for the task.
The Quick Start below uses the general workflow. For benchmark tasks, follow the ResearchClawBench setup guide.
Requires Python 3.10+ and a research agent (Codex, Claude Code, or OpenClaw). AutoSciRub's scripts prepare and inspect a project; your agent performs the research.
Linux / macOS:
git clone https://github.com/zjunlp/AutoSciRub.git
cd AutoSciRub
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python scripts/install.py codex # or: claude, openclawWindows PowerShell:
git clone https://github.com/zjunlp/AutoSciRub.git
cd AutoSciRub
py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe scripts/install.py codexUse this Python environment when running the helpers. In PowerShell, replace
python below with .\.venv\Scripts\python.exe if you have not activated it.
python scripts/autoscirub.py demo --project ./autoscirub-demoNo model, API key, PDF upload, or external query is used. The demo copies a tiny synthetic task into a new directory and checks that exactly one allowed local document is indexed while the excluded folder is omitted. It is a setup check, not a claim that an agent has completed the scientific task.
To try the complete workflow, open your agent in autoscirub-demo/
and ask: “Use AutoSciRub to complete INSTRUCTIONS.md using local files only.”
The sample explains its expected numerical results in
examples/minimal/README.md.
Run these commands from the AutoSciRub checkout:
python scripts/autoscirub.py init --project /path/to/my-research
python scripts/autoscirub.py doctor --project /path/to/my-researchOn Windows use a quoted Windows project path. init creates a config without
overwriting an existing one; it does not copy or alter your research data.
Open a new agent session in your research project, using the Python environment above if you want the bundled helpers, and give it a concrete instruction:
Use AutoSciRub to compare treatment and control in data/measurements.csv. Save the analysis code, a results table, and a short report. Use local sources from papers/ and perform at most one revision round.
Other supported scopes:
- Rubric only: “Use AutoSciRub to make an executable rubric for this task. Stop before experiments.”
- Verification only: “Verify report.md against the existing AutoSciRub rubric. Do not revise it.”
Choose a bundle when installing; generic is the default:
python scripts/install.py codex # General workflow: eight skills
python scripts/install.py codex --bundle researchclawbench # Benchmark adaptation: six skillsReplace codex with claude or openclaw for that host. The installer copies
the selected skills and their bundled resources into the platform's personal
skill directory. It does not register a marketplace or run the agent. These
directories follow the platform documentation:
| Platform | Personal directory | Documentation |
|---|---|---|
| Codex | .agents/skills under your user directory |
Codex skills |
| Claude Code | .claude/skills under your user directory |
Claude skills |
| OpenClaw | .openclaw/skills under your user directory |
OpenClaw skills |
Preview or check an installation:
python scripts/install.py codex --dry-run
python scripts/install.py codex --check
python scripts/install.py codex --bundle researchclawbench --check--check verifies the installed files against this checkout. To verify
actual discovery, open a new agent session and ask it to list the AutoSciRub
skills. Agent policy or workspace-level overrides can still affect availability.
After updating your checkout, rerun the installer. Identical files are left
alone. Previously installed, unmodified skills are updated; locally edited or
unmanaged same-name skills cause an actionable error before any skill is changed.
For a custom location, use --skills-dir and configure your agent to load it.
The Bash and PowerShell wrappers call the same installer:
./scripts/install.sh codex or ./scripts/install.ps1 codex.
Select the adaptation with Bash's --bundle researchclawbench or PowerShell's
-Bundle researchclawbench. PowerShell also accepts -Check, -DryRun, and
-SkillsDir.
Existing marketplace users may continue using the plugin package. Avoid enabling
both that plugin and the personal copies simultaneously. The manifests remain in
plugins/autoscirub/; the personal-skill installer does not update an existing
plugin cache.
This section describes the general workflow. The ResearchClawBench adaptation
uses a separate .rcbench/profile.yaml; see its
configuration and environment guide.
init creates PROJECT/.autoscirub/config.yaml. YAML is parsed with PyYAML;
the same JSON Schema validates both partial overrides and the resolved settings.
Comments, quoted strings, lists, and anchors are supported. Duplicate keys,
unknown fields, misspelled provider names, and invalid types fail before retrieval.
YAML merge keys that create duplicate keys are rejected; use explicit sections.
For local-only work, a complete minimal config is:
retrieval:
enabled_sources: [local_literature]
local_literature:
paths: [papers]
exclusions:
excluded_paths: [papers/private]Local inputs and exclusions are exact files or directories, not glob patterns. Directories are searched recursively; excluded subtrees and links to excluded files are omitted. Local indexing provides filenames and paths, not full-text search or scientific relevance ranking. It does not upload documents.
Path rules:
--projectselects the research root; otherwise the current directory is used.- With no
--config, readPROJECT/.autoscirub/config.yamlif it exists. - An explicit
--configpath is relative to the shell's current directory. - All paths inside the config are relative to the research root, not the config file.
- State directory precedence:
--state-dir→AUTOSCIRUB_STATE_DIR→ configstate_dir→.autoscirub. - Retrieval output defaults to
STATE/literature_sources.json;--outoverrides it. - MinerU output defaults to
STATE/cache/mineru/; its--output-diroverrides it. - AutoSciRub skills use the same resolved state directory. Reports, code, and scientific results stay in the locations agreed with the user.
python scripts/autoscirub.py config validate --project /path/to/my-research
python scripts/autoscirub.py retrieve --project /path/to/my-research --query "research question"Use repeatable --query arguments for separate queries, or --goals for a
rubric skeleton. Run retrieve --help for the complete options.
The original skill script remains directly executable.
Full defaults: config/sources.example.yaml. The editor-facing config/sources.schema.json points to the canonical schema bundled with the retrieval skill.
The default config enables arXiv, OpenAlex, Semantic Scholar, and local indexing.
Tavily is opt-in: add it to enabled_sources and set its provider enabled: true.
Credentials are read only from environment variables:
| Variable | Purpose |
|---|---|
OPENALEX_EMAIL |
OpenAlex contact email |
OPENALEX_API_KEY |
OpenAlex access, when required by the service |
SEMANTIC_SCHOLAR_API_KEY |
Semantic Scholar authentication/rate allowance |
TAVILY_API_KEY |
Required for Tavily |
MINERU_API_TOKEN |
MinerU standard API |
Remote retrieval sends query text to the selected providers. MinerU is a separate,
optional converter that uploads input documents; leave it disabled for local-only
work. API access, rates, and costs are controlled by those services. web
means the agent's native web search, not an implemented provider in this script.
The retriever prints each source's progress and saves structured provider_results.
| Situation | Behavior / next step |
|---|---|
| Invalid configuration | Exit 2 with the field to correct; no requests made |
| Missing Python dependency | Install requirements.txt using the same Python executable |
| Missing local document | Check project-relative local_literature.paths; shown as a failed source |
| Missing required key | Source skipped with the variable name to set |
| Successful query, no usable results | Status empty, exit 0; try broader terms or local papers |
| Some sources fail | Status partial, exit 0; available sources retained with failure details |
| Every source fails or none can run | Exit 3; write literature_sources.failure.json, preserving the previous index |
| HTTP 401 / 403 / 429 | Check provider access or rate allowance; no automatic unbounded retries |
| Skill not available in agent | Run install --check, restart the agent, inspect its local skill settings |
| Installer detects local edits | Preserve your changes or choose a different configured skill directory |
A default request can wait up to 30 seconds before a network timeout. This version
uses sequential requests and reports progress; it does not automatically retry or
resume requests. doctor checks local configuration, dependencies, paths,
and environment-variable presence without making paid requests or testing keys.
The state files below belong to the general workflow. The ResearchClawBench bundle has its own artifact contract.
The controller performs seven steps: instruction goals, literature grounding, data exploration, criterion synthesis, research execution, criterion verification, and bounded revision. The agent runs the scientific experiments; the CLI does not replace it. Simple tasks need only a few goals and criteria, not a fixed quota.
The default revision limit is one round (workflow.max_revision_rounds).
Every full-workflow revision is followed by verification. Missing inputs remain
explicit unmet requirements; all_satisfied cannot be true while any
in-scope criterion is unmet. Verification-only requests never authorize edits.
Framework state includes:
run_manifest.json: controller-maintained task, stage, budget, and output locations.rubric_skeleton.json,literature_grounding.json,task_data_profile.json, andexecutable_rubric.json: planning and evidence contracts.literature_sources.json: optional helper-generated retrieval index.verification_report.jsonandrevisions/round-NNN/: checks and revisions.cache/mineru/: optional converted literature.
python -m pip install -r requirements.txt
python scripts/validate.py
python -m unittest discover -s tests -vTests use local fixtures and mocked network responses. GitHub Actions runs them on Linux, macOS, and Windows with Python 3.10 and 3.12, plus Bash and PowerShell installer smoke checks. They verify helper behavior, not model reasoning quality or live provider availability.
If you use AutoSciRub in your research, please cite our paper:
@article{autoscirub,
title={Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents},
author={Wang, Xuehai and Qin, Haowei and Liu, Tongxin and Li, Junkai and Xu, Buqiang and Zhang, Jintian and Chen, Yijun and Xue, Zirui and Deng, Shumin},
journal={arXiv preprint arXiv:2608.31076},
year={2026},
doi={10.48550/arXiv.2608.31076},
url={https://arxiv.org/abs/2608.31076}
}MIT. See LICENSE.