PaperBench Code-Dev Benchmark

• The paper presents a standardized benchmark with hierarchical rubrics to evaluate AI agents' ability to convert natural language research into functional, modular codebases.
• PaperBench Code-Dev focuses on reproducing code from 20 ICML 2024 papers using detailed criteria and an automated LLM judging system to measure modular implementation accuracy.
• The benchmark highlights significant performance gaps between AI agents and human developers, with scores ranging from 5% to 43.4% on code replication tasks.

PaperBench Code-Dev Benchmark

PaperBench Code-Dev Benchmark is a standardized, rubric-driven evaluation suite designed to assess the code development capabilities of AI agents in replicating current machine learning research papers, specifically focusing on their ability to convert natural language descriptions of technical contributions into correct, modular codebases. It forms the lightweight variant of the broader PaperBench project, with emphasis on end-to-end codebase construction rather than execution or results replication. The benchmark was introduced and validated on 20 ICML 2024 Spotlight and Oral papers, integrating hierarchical verification rubrics co-developed with original authors, and an automated LLM-assisted judging system to enable fine-grained, scalable, and consistent performance measurement across models and agent scaffolds (Starace et al., 2 Apr 2025).

1. Benchmark Scope and Data Selection

PaperBench Code-Dev focuses on research code reproduction at a granularity matching real ML publications.

Paper Selection and Scope:

• 20 ICML 2024 Spotlight and Oral papers were selected after a staged filter: removal of papers requiring proprietary compute, closed-source models, inaccessible dependencies, pure theory, or framework-level contributions.
• Each paper’s authors contributed clarifications and scope limitations as addenda, ensuring the Code-Dev variant is tractable for LLM agents under standardized computing restrictions (Ubuntu 24.04, Python 3.10+, standard ML libraries).
• For each paper, agents are only provided:
  • paper.md (PDF-to-Markdown main text, including all equations and technical appendices)
  • addendum.md (clarifications, constraints, and data location notes)
  • blacklist.txt (URLs not allowed for consultation)
  • Agents must deliver a structured repository at /home/submission/ including all implementation files, a modular README.md, and a reproduce.sh entry point (which is not required to execute experiments for Code-Dev) (Starace et al., 2 Apr 2025).

2. Rubric Design and Grading Hierarchy

Hierarchical Rubric Structure:

• Rubrics are co-developed between benchmark organizers and paper authors. Each rubric is a tree:
  • Root node represents “Full paper replicated”
  • First level: one node per principal contribution (e.g. “Transformer backbone”, “Novel optimizer module”)
  • Deeper levels: recursively refined sub-tasks down to atomic “leaf” nodes, each denoting an unambiguous, unit-verifiable code-development requirement (e.g. “Forward pass computes positional embedding”, “Implements AdamW update rule”)
• For the Code-Dev variant, only Code-Dev leaves are scored, all execution- and result-matching leaves are ignored.
• Each leaf node carries a manually assigned weight; normalized per-paper Code-Dev score is $$R_{\text{code}} = \frac{ \sum_i w_i s_i }{ \sum_i w_i }\times 100\%$$ where $s_i\in\{0,1\}$ denotes pass/fail for leaf $i$, and $w_i$ its weight.
• For dataset-level reporting, mean Code-Dev score is averaged across all 20 papers (Starace et al., 2 Apr 2025).

3. Agent Development Environment and Scaffold

Compute and Environment Constraints:

• Submissions are evaluated in a standardized Ubuntu 24.04 Docker image containing common ML libraries (PyTorch, JAX, scikit-learn, pandas, numpy).
• LLM agents (e.g. GPT-4o, Claude 3.5 Sonnet, DeepSeek-R1, Gemini 2.0 Flash) interact with the environment through open-source scaffolds following a lightweight ReAct protocol.
• Scaffold exposes:
  • Shell and Python command execution
  • File read/write and context navigation tools
  • Automated context trimming to respect LLM token limitations

Best Practice Enforcements:

• Modularization: code must live under /src or /experiments
• Environment reproducibility: dependencies pinned in requirements.txt or conda environment.yml
• Version control: agents commit frequently with meaningful git messages
• Documentation: structured README.md and reproduce.sh scripts mandated for every submission (Starace et al., 2 Apr 2025).

4. Automated Judging: LLM-Based Code-Dev Grader

Judging Architecture:

• Automated “SimpleJudge” LLM (typically o3-mini-high) is used for leaf-node assessment:
  • “# Expectations”
  • “# Reality” (interpretation of codebase implementation)
  • “# Score” (0 or 1)
  • 3. Scores are aggregated bottom-up for normalized per-paper grading.
• LLM judge is validated against a human-annotated subset (5 papers, 4–6 submissions each), with macro F1 overall 0.83 (Code-Dev leaves: 0.72, Execution: 0.82, Result-Match: 0.94). Unit cost is ~$66/paper versus$1200/paper human baseline (Starace et al., 2 Apr 2025).

5. Empirical Results, Diagnostics, and Comparative Baselines

Experimental Protocol:

• Each (model, paper) pair is run three times per scaffold (BasicAgent, IterativeAgent).
• Each agent-run is limited to 6 hours for Code-Dev (versus 12 for full PaperBench).
• Metrics: Code-Dev score $R_{\text{code}}$ per paper, and overall mean $\pm$ SEM.
• In primary experiments:
  • o1-IterativeAgent achieves mean $R_{\text{code}} = 43.4\% \pm 0.8\%$
  • o3-mini-IterativeAgent: $R_{\text{code}} = 16.4\% \pm 1.4\%$
  • Gemini 2.0 Flash: $\approx 5\%$
  • Claude 3.5 Sonnet (full replication, BasicAgent): $R_{\text{full}} = 21.0\% \pm 0.8\%$
• Human baseline (best-of-3 on 3-paper subset, 48h): $R_{\text{full}} = 41.4\%$ (humans start slower but overtake LLMs after 24 hours; LLM plateau on code synthesis after 1 hour)
• High per-paper variance observed, weak but present correlation $r=0.48$ between Code-Dev and full-replication scores (Starace et al., 2 Apr 2025).

6. Open-Source Resources and Reproducibility

Artifact Availability and Pipeline:

• Full benchmark suite (rubrics, papers, agent scaffolds, judge, scripts):
  • https://github.com/openai/paperbench
  • Includes: agent/basic_agent.py, agent/iterative_agent.py, judge/simple_judge.py, judge/judge_eval.py, Dockerfile, example papers, and configuration templates.
• Reproduction steps:
  • Clone repository, build Docker image, export LLM API keys, run scripts for generation and grading, add new papers via prescribed YAML/JSON format, support for adding human-graded examples.
• Designed for rapid extensibility to new papers, judges, and scaffolds (Starace et al., 2 Apr 2025).

7. Insights, Limitations, and Future Directions

PaperBench Code-Dev demonstrates that current LLM agents remain substantially limited in accurately converting real ML research into functional, well-structured codebases even when execution and result-matching requirements are lifted. Achieved scores trail human PhD baselines, with error analysis indicating failures in modular implementation, correct parameterization, and matching rubric-prescribed technical details.

Proposed advances include tighter integration of semantic, code-structure, and performance-signal reasoning in agents, richer context management to overcome sliding-window LLM limitations, and improvements in rubric granularity and LLM judge calibration. Extensions to support more papers from different venues, multi-language pipelines, and human-in-the-loop agent collaboration are anticipated directions for rigorous, scalable benchmarking of next-generation research-code development agents (Starace et al., 2 Apr 2025).