---
title: PaperBench
url: https://www.emergentmind.com/topics/paperbench
type: topic
---

# PaperBench

PaperBench is a benchmark for measuring whether artificial-intelligence agents can reproduce contemporary machine-learning research from scientific papers rather than from authors’ implementations. A PaperBench agent must interpret a paper, infer experimentally consequential details, implement a repository from scratch, establish a runnable environment, execute experiments, diagnose failures, and produce artifacts supporting the paper’s empirical claims. The benchmark was introduced for 20 ICML 2024 Spotlight and Oral papers and contains 8,316 individually gradable rubric tasks; its principal reported result was 21.0% for Claude 3.5 Sonnet with open-source scaffolding, while recruited ML PhD participants achieved a 41.4% best-of-three score on a three-paper subset after 48 hours [2504.01848].

## 1. Purpose and scope

PaperBench was created to measure an operational component of autonomous AI research: the ability to turn published research into working research software and experiments. It is intended to test substantially more than paper comprehension, code completion, or generation of plausible isolated snippets. An agent receives a paper and must infer what matters experimentally, implement the methods, construct a repository, execute the experiments, troubleshoot failures, and generate outputs that support the paper’s empirical claims.

The benchmark was motivated partly by the possibility that autonomous machine-learning research could accelerate scientific progress while also creating safety and governance risks. The authors position PaperBench as relevant to autonomy in OpenAI’s Preparedness Framework, autonomous capabilities in Anthropic’s Responsible Scaling Policy, and ML R&D in Google DeepMind’s Frontier Safety Framework [2504.01848].

Replication is treated as a demanding but concrete proxy for research engineering. PaperBench does not test every component of scientific research. In particular, it does not directly evaluate the generation of genuinely novel hypotheses, the selection of important research questions, or a causal understanding of why a method works. A high score indicates substantial implementation, experimentation, and empirical replication capability, but does not by itself establish scientific judgment, novelty, or broad research autonomy.

The defining methodological constraint is that agents work from the paper rather than the authors’ implementation. Existing codebases are prohibited, including author repositories and known blacklisted replications. This distinguishes PaperBench from reproducibility benchmarks such as CORE-Bench, in which the agent is given a paper’s repository. The objective is implementation and experimentation from scratch.

## 2. Benchmark composition and task materials

The main benchmark was constructed from ICML 2024 papers awarded Spotlight or Oral status. Candidate papers were filtered for:

- a substantial empirical contribution;
- feasibility on a single machine rather than distributed multi-node training;
- no dependence on closed-source pretrained models;
- no requirement for new human data collection or annotation;
- sufficiently detailed methodology;
- accessible dependencies;
- suitability for replication rather than primarily being a software-framework paper.

The benchmark is described as containing 20 papers spanning 12 ICML topics, including deep reinforcement learning, robustness, probabilistic methods, generative models, domain adaptation, data-centric AI, and out-of-distribution generalization. The named papers include “APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference,” “All-in-one simulation-based inference,” “BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models,” “RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation,” “Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models,” and “Test-Time Model Adaptation with Only Forward Passes” [2504.01848].

The supplied benchmark description contains an apparent inconsistency: a table appears to enumerate 21 names when “Sample-specific Masks for Visual Reprogramming-based Prompting” is counted alongside the other entries, although the benchmark is repeatedly described as containing 20 papers. The headline results and methodology consistently refer to a 20-paper benchmark. The authors also released two NeurIPS 2024 papers as a development set and retained a held-out internal set, while the main reported benchmark remained the ICML collection.

For each paper, an agent receives:

- the paper in PDF and Markdown form;
- an author-validated addendum clarifying ambiguities and out-of-scope material;
- task instructions.

The rubric is withheld during the agent attempt. This prevents direct optimization against the grading decomposition and requires the agent to infer the replication target from the paper and addendum.

PaperBench Code-Dev is a reduced setting that evaluates only Code Development requirements. It omits fresh execution and result verification, making it cheaper and more accessible but weaker as a measure of complete replication. PaperCoder reported 44.26% on this 20-paper Code-Dev subset using a structured planning–analysis–generation pipeline, compared with 5.1% for BasicAgent and 16.4% for IterativeAgent under the reported comparison [2504.17192].

## 3. Submission workflow and execution environment

A valid submission is a Git repository containing the implementation and a root-level executable script:

```bash
reproduce.sh
```

This script is the required entry point. It must execute the code needed to reproduce the paper’s empirical contributions and generate relevant tables, figures, metrics, logs, and artifacts.

The full workflow has three stages.

1. **Agent attempt**: The agent reads the paper, writes the implementation and supporting documentation, creates the repository, and ideally tests its reproduction script.

2. **Fresh reproduction**: The submission is copied into a clean Ubuntu 24.04 virtual machine with access to an NVIDIA A10 GPU. The benchmark runs `reproduce.sh` from a fresh setup, retaining generated files and `reproduce.log`.

3. **Rubric grading**: An automated judge evaluates the executed submission against the paper-specific rubric.

The separation between agent attempt and fresh reproduction is central to the benchmark. It prevents an agent from receiving credit merely for hard-coding results or generating artifacts during task-time interaction. Replication credit requires that results be produced or evidenced by fresh execution.

The reproduction script had a nominal 12-hour cap in the reported experiments, although agent-produced scripts averaged approximately 5.5 minutes during reproduction. Agents had internet access, Hugging Face credentials, and an OpenAI API key loaded with $1,000 for experiments potentially requiring external services. Paper-specific blacklists prohibited access to author code and known online replications. A log monitor searched for blacklisted URLs; across 646 runs, 10 submissions were disqualified and assigned score zero [2504.01848].

The principal agent setup used Ubuntu 24.04 Docker containers, one NVIDIA A10 GPU, internet access, paper PDF and Markdown, an addendum, bash, Python, browser, and paginated file-reading tools. The basic scaffold was a ReAct-style tool-use loop modified with context management and a paginated file reader. IterativeAgent removed early submission, forced continued work through the allotted time, prompted incremental steps, and encouraged piecemeal implementation.

## 4. Hierarchical rubrics and Replication Score

PaperBench’s central evaluation mechanism is a hierarchical rubric. A high-level goal such as “the core contributions of the paper have been reproduced” is recursively decomposed into more specific requirements. A contribution may require that:

- a model or method has been implemented;
- specified datasets and environments have been prepared;
- particular models have been trained with stated hyperparameters;
- required evaluations have been executed;
- resulting metric trends or values agree with the paper.

Decomposition continues until an expert familiar with the paper can assess a leaf in less than approximately 15 minutes. Across the benchmark there are 8,316 individually gradable leaf outcomes. Every rubric node has a manually assigned weight. The weight expresses importance relative to sibling requirements rather than implementation difficulty.

A leaf receives a binary score:

$$
s_i \in \{0,1\},
$$

where $s_i=1$ means that the requirement is satisfied. An internal node aggregates its children by weighted averaging:

$$
S(n)=\frac{\sum_{c\in \mathrm{children}(n)} w_cS(c)}
{\sum_{c\in \mathrm{children}(n)} w_c}.
$$

The root score is the submission’s Replication Score, expressed as a fraction or percentage. A perfect replication receives 100%. Because weights propagate through the tree, the score is not an unweighted count of passed leaves.

The rubric contains three leaf types:

- **Code Development**: whether source code appears to implement a required component correctly, such as an architecture, preprocessing operation, loss, or evaluation routine.
- **Execution**: whether the required operation actually occurred when `reproduce.sh` was run.
- **Result Match**: whether executed outputs agree with the paper’s reported results, trends, tables, or figures.

The separation is intentional. A final-result-only rubric would provide little credit for meaningful partial progress, while source-code-only evaluation would not establish that the implementation runs or produces the claimed outcomes. Submissions without `reproduce.sh` receive zero on all Execution and Result Match leaves.

The benchmark therefore measures graded progress across implementation, execution, and empirical fidelity. It does not treat “the repository runs” as equivalent to “the paper has been scientifically replicated.”

## 5. Automatic judging and validation

Manual grading of one replication was estimated to take tens of hours for an expert. PaperBench therefore uses an LLM-based judge called SimpleJudge. For each rubric leaf, the judge receives:

- the paper in Markdown;
- the paper addendum and relevant judge-only clarifications;
- the full rubric context;
- the specific leaf requirement;
- relevant submission files;
- `reproduce.sh`, if present;
- `reproduce.log`, if present;
- files produced or modified during reproduction.

The judge first ranks files for relevance. If the filtered repository is too large for the context window, only the top ten relevant files are included. Source code, documentation, and configuration are prioritized for Code Development and Execution leaves; recently generated tabular or textual outputs, logs, and documentation are prioritized for Result Match leaves.

The main results use OpenAI o3-mini, model version `o3-mini-2025-01-31`, with reasoning effort set to high. For each leaf, it is asked to describe a correct resolution, compare the submission against that expectation, and return a binary score with an explanation. A separate GPT-4o call parses the response into a score, explanation, and `valid_score` indicator. Binary leaf scores are then propagated through the rubric tree.

Because automatic judging can itself be unreliable, the authors created JudgeEval. It contains partial replications of four PaperBench papers and one development-set paper manually graded by experts. Automated judges are evaluated as binary classifiers against those labels.

| Judge backend | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|
| Random baseline | 0.48 | 0.49 | 0.49 | 0.49 |
| GPT-4o-mini | 0.63 | 0.64 | 0.60 | 0.59 |
| GPT-4o | 0.74 | 0.74 | 0.72 | 0.73 |
| o1-mini | 0.81 | 0.85 | 0.76 | 0.78 |
| o1 | 0.84 | 0.84 | 0.84 | 0.84 |
| o3-mini | 0.83 | 0.83 | 0.83 | 0.83 |

For o3-mini, F1 differs by leaf type:

- Code Development: 0.72;
- Execution: 0.82;
- Result Match: 0.94.

The judge is strongest when assessing generated result artifacts and weaker when determining whether source code is genuinely correct. Its overall F1 of 0.83 is treated as useful for scalable evaluation but not equivalent to expert human grading. The judge is nondeterministic and can be affected by context filtering, incomplete artifacts, misleading logs, specification gaming, and rubric errors. Grading costs approximately $66 per paper submission, or approximately $10 per paper for Code-Dev grading [2504.01848].

The authors also explored rubric pruning by collapsing lower-level subtrees into a single floating-point judgment. One preliminary example reduced grading cost by approximately 10 times with only a modest score change, but unsatisfactory cases prevented the result from being treated as validated.

## 6. Results, interpretation, and limitations

The principal BasicAgent results are average weighted Replication Scores:

| Model | PaperBench score |
|---|---:|
| o3-mini-high | 2.6 ± 0.2% |
| GPT-4o | 4.1 ± 0.1% |
| Gemini 2.0 Flash | 3.2 ± 0.2% |
| DeepSeek-R1 | 6.0 ± 0.3% |
| o1-high | 13.2 ± 0.3% |
| Claude 3.5 Sonnet | 21.0 ± 0.8% |

Claude 3.5 Sonnet achieved the strongest main-setup result, 21.0%. This is not the percentage of papers fully replicated; it is the average weighted proportion of rubric requirements satisfied.

IterativeAgent substantially changed performance:

| Model and scaffold | PaperBench score |
|---|---:|
| o3-mini-high + IterativeAgent | 8.5 ± 0.8% |
| o1-high + IterativeAgent | 24.4 ± 0.7% |
| Claude 3.5 Sonnet + IterativeAgent | 16.1 ± 0.1% |
| o1-high + IterativeAgent, 36-hour limit | 26.0 ± 0.3% |

For o1 and o3-mini, forcing continued work improved performance; under the tested prompt design it reduced Claude 3.5 Sonnet’s score. This indicates that benchmark outcomes are sensitive to scaffolding and prompting rather than constituting fixed model rankings.

The requirement-type analysis shows a substantial gap between writing code and completing research workflows. On the three-paper human-comparison subset, the best-three human result was 72.4% for Code Development, 20.4% for Execution, and 8.9% for Result Match. Claude 3.5 Sonnet with BasicAgent achieved 35.4%, 1.8%, and 0.7% on the corresponding categories. o1 with IterativeAgent achieved 43.3%, 4.5%, and 0.0%. These results support the diagnosis that agents can generate plausible isolated components but struggle to integrate them, run complete pipelines, diagnose failures, and obtain credible empirical results [2504.01848].

The human comparison involved eight current or former ML PhD students selected through CV screening and a machine-learning and Git technical test. Participants attempted a four-paper subset, could select papers they felt confident replicating, could use AI assistants such as ChatGPT and GitHub Copilot, and worked within a four-week window with active time tracked. The best-of-three human result on the three-paper subset was 41.4% after 48 hours, compared with 26.6% for o1. Humans began outperforming o1 after approximately 24 hours, although o1 initially progressed more rapidly.

Common agent failure modes included:

- **Early termination**: declaring success after partial code or concluding that a problem was unsolvable.
- **Planning without sustained execution**: producing convincing plans without carrying them through tool calls.
- **Weak prioritization**: attempting broad but shallow implementations rather than securing high-value contributions.
- **Tool-use problems**: writing large code blocks instead of incrementally inspecting, editing, and executing files.
- **Code without integration**: failing to connect components, use correct datasets and preprocessing, execute configurations, monitor failures, or reproduce quantitative trends.

PaperBench’s limitations are substantial. The dataset contains only 20 papers and is selective toward papers feasible on constrained hardware and with accessible dependencies. It may not represent theory papers, systems papers, large-scale distributed training, proprietary-model research, human studies, or domains outside machine learning. Public availability of original code also creates possible contamination despite blacklisting.

Rubric subjectivity remains unavoidable. Author collaboration improves realism but does not establish a unique definition of replication. Experts could assign different weights or decompose requirements differently. The current hierarchy also encodes some dependency information through ordering without always specifying a complete dependency graph.

The benchmark is not a complete measure of scientific intelligence. A high score would demonstrate implementation, experimentation, and empirical replication ability, but would not prove that an agent can identify important questions, formulate novel hypotheses, understand causal mechanisms, distinguish flawed from correct research, generalize methods beyond tested setups, or exercise scientific judgment. Conversely, a low score may reflect compute limits, dependency friction, poor scaffolding, or insufficient time rather than an absence of conceptual understanding.

Subsequent research has treated PaperBench as a platform for improving long-horizon research agents. PaperCoder’s planning–analysis–generation pipeline reached 44.26% on Code-Dev [2504.17192]. RePro introduced a paper fingerprint consisting of grounded, atomic implementation criteria and achieved 62.6% root-level performance on Code-Dev, compared with 49.6% for AutoReproduce [2508.16671]. Executable Knowledge Graphs improved PaperCoder’s o3-mini Code-Dev average from 42.31 to 53.21 by associating technical concepts with reusable implementations [2510.17795]. Prompt-free collaborative verification and refinement increased Paper2Code’s PaperBench Code-Dev score from 0.682 to 0.786 [2512.02812].

Other systems emphasize long-horizon state management and orchestration. KLong used trajectory-splitting supervised fine-tuning and progressive reinforcement learning, obtaining a reported PaperBench average of 62.59 [2602.17547]. AiScientist combined hierarchical orchestration with a permission-scoped File-as-Bus workspace and reported scores of 30.52 with Gemini-3-Flash and 33.73 with GLM-5 [2604.13018]. HiRAS used supervisory managers, specialized agents, shared artifacts, and execution-based repair, reporting 64.1% with Claude-Sonnet and 57.4% with DeepSeek-v3.1 on Code-Dev [2604.17745]. These results concern different model, evaluator, scaffolding, and sometimes benchmark configurations and should not be interpreted as a single directly comparable leaderboard.

PaperBench’s broader significance is methodological. It operationalizes an important part of autonomous AI research as a chain of observable requirements: interpreting a paper, implementing a method, constructing a runnable repository, executing experiments, debugging failures, and matching empirical evidence. Its results show that current agents possess nontrivial ML research-engineering ability but remain unreliable at the complete transition from scientific description to validated research artifact.

Source: https://www.emergentmind.com/topics/paperbench