Paper2Code Benchmark: Reproducible Code from Papers

• The paper introduces a benchmark that systematically evaluates LLMs' capability to generate complete code repositories from ML research papers.
• Paper2Code integrates hierarchical planning and dependency-aware generation to achieve robust replication accuracy (RepAcc) for multi-module systems.
• The benchmark demonstrates significant improvements using advanced multi-agent systems like PaperCoder over traditional iterative code synthesis methods.

Paper2Code Benchmark

LLM-based code generation has accelerated the need for systematic evaluation benchmarks that measure model fidelity in reproducing the algorithms and software artifacts described in scientific literature. The Paper2Code Benchmark, as instantiated in the context of ML research, specifically addresses the automated transformation of scientific papers into functional, modular, and architecturally consistent code repositories. It captures a substance and granularity far beyond statement- or function-level code synthesis, focusing on the reproducibility of entire research systems and their building blocks in the absence of reference implementations. Paper2Code is centered around the Code-Dev track of PaperBench, targeting hierarchical implementation requirements as annotated by original paper authors, and is evaluated via the PaperCoder multi-agent LLM system, providing a controlled, quantitatively robust foundation for code reproduction from scientific discourse (Seo et al., 24 Apr 2025).

1. Scope and Dataset Composition

Paper2Code, through the PaperBench Code-Dev benchmark, evaluates LLMs' ability to generate code repositories that reproduce implementation requirements derived from recent ML papers.

• Benchmark Tasks: 20 ML papers from ICML 2024, covering time-series forecasting, graph learning, computer vision, and diffusion models.
• Evaluation Unit: Each paper defines a tree of implementation requirements (e.g., data loader, model architecture, training loop, evaluation script). The benchmark asks whether a candidate repository contains correct functional implementations for these requirement nodes.
• Input: Only the PDF of the target paper is provided; no code, documentation, or supplementary material.
• Ground Truth: Author-released repositories serve as optional verification, but most evaluation relies on fulfillment of explicit requirements as specified by paper authors.

The design sharply contrasts with standard benchmarks by focusing on the higher-level structural, modular, and architectural aspects of code, rather than single-method or microbenchmark granularity (Seo et al., 24 Apr 2025).

2. Evaluation Metrics and Formal Definitions

Paper2Code employs a requirement-aligned metric, Replication Accuracy (RepAcc), computed as:

$$\textrm{RepAcc} = 100 \cdot \frac{1}{M} \sum_{r=1}^M I_r$$

where $M$ is the total number of code-development requirements across all papers and $I_r$ is an indicator (1 if requirement $r$ is correctly implemented, 0 otherwise).

For broader context, standard metrics commonly used in code-generation literature, such as exact match (EM) and execution accuracy (ExecAcc), are provided:

• $$\textrm{EM} = (1/N) \sum_{i=1}^N \mathbb{1}[pred_i = gt_i]$$
• $$\textrm{ExecAcc} = (1/N) \sum_{i=1}^N \mathbb{1}[run(pred_i) = expected_i]$$

However, Paper2Code's primary reporting is via RepAcc, emphasizing fidelity to paper-defined functional nodes rather than character-level equivalence or mere test passing. Automated end-to-end execution accuracy is not reported, reflecting the complexity and diversity of ML system implementations assessed (Seo et al., 24 Apr 2025).

3. Baseline Agents, Systems, and Results

PaperBench Code-Dev evaluates both single-agent and multi-agent LLM-based systems for paper-to-code conversion.

• Baselines:
  • BasicAgent: A single-pass agent using a ReAct-style prompt sequence.
  • IterativeAgent: A decomposition-driven agent that incrementally breaks down tasks via tool-augmented subgoal prompts.
• PaperCoder: A three-stage, multi-agent LLM framework specifically architected for the paper-to-code setting, leveraging planning (roadmap, UML/source diagrams, file dependency graphs), analysis (module-level plans), and dependency-aware generation.

Replication Scores (mean over three runs, ICML 2024 Code-Dev set):

Model	Replication Score (%)
BasicAgent	5.1 ± 0.8
IterativeAgent	16.4 ± 1.4
PaperCoder	44.26

The results highlight a large (>2.7×) advantage for PaperCoder over iterative prompting, supporting the use of architectural reasoning and modular scaffolding in complex code synthesis (Seo et al., 24 Apr 2025).

4. Pipeline and Methodology: The PaperCoder Approach

The PaperCoder system embodies a hierarchical, pipeline-structured approach:

1. Planning:
   • Global roadmap construction.
   • Generation of architectural diagrams (class/sequence/UML).
   • Automated extraction of file dependencies and config files (e.g., YAML for hyperparameters).
2. Analysis:
   • File-level implementation plans, detailing I/O contracts, dependencies, and integration hooks.
3. Generation:
   • Generation of code per module/file in a dependency-aware order.
   • Each file is produced with explicit reference to UML/dataflow, ensuring alignment across the repository.

Key innovations shown to underpin PaperCoder's success include:

• Explicit config.yaml creation to ground hyperparameters and data paths.
• Sequenced module implementation avoiding missing dependencies and hallucinated imports.
• Structural diagramming (call graphs, class relationships) ensuring interface consistency and functional integration.

These stages are executed by specialized LLM agents with prompt engineering tuned to code synthesis from textually-described academic mechanisms (Seo et al., 24 Apr 2025).

5. Limitations and Outstanding Challenges

The Paper2Code paradigm, while making substantial progress, faces notable methodological and practical limitations:

• Domain-specificity: Current evaluation is restricted to ML papers; extrapolating to systems, biology, or domain sciences may require new planning agents and datasets.
• Automated End-to-End Testing: Current scoring checks for module implementation rather than full system executability. Fully automated, standardized integration testing remains unsolved.
• Scalability: Large or highly technical papers may outrun LLM context windows, impacting planning and coherence.
• Manual Debugging: Even with high RepAcc, manual correction is sometimes necessary for empirical reproducibility, especially where papers omit configuration or rely on domain knowledge.
• Evaluation Granularity: RepAcc reports requirement-level correctness, which may not reflect functional parity at finer (e.g., experiment, metric) levels (Seo et al., 24 Apr 2025).

A plausible implication is that moving toward robust, automated evaluation of paper-to-code artifacts will require advances in hierarchical code analysis, test-synthesis at the experiment level, and cross-domain architectural extraction.

6. Relationship to Broader Code-Generation Benchmarks

Paper2Code occupies an orthogonal position with respect to traditional code generation benchmarks such as HumanEval (Python-only, function-level), JavaBench (project-level Java with prescribed skeletons), or CoderEval (context-rich standalone/non-standalone function generation).

Distinctive characteristics include:

• The centrality of natural-language research documents, with no reliance on pre-existing code skeletons or APIs.
• Emphasis on modular, multi-file, and dependency-aware code repositories.
• Evaluation against author-defined requirements, as opposed to reference-based unit testing or metric-driven microbenchmarks.

This approach aligns with ongoing efforts to evaluate LLMs' practical value in scientific reproducibility, codebase regeneration, and large-scale automation of research pipelines (Seo et al., 24 Apr 2025).

7. Future Directions

Immediate research directions highlighted in the Paper2Code literature include:

• Extension to Non-ML Domains: Adapting the agent and benchmark methodology for domains with different conventions (e.g., systems, experimental sciences).
• Automated Executability Metrics: Developing scalable, platform-generic harnesses to run end-to-end generated repositories and compare outputs to published results.
• Scaling to Long, Multi-Section Papers: Efficient strategies for context management (e.g., memory networks, retrieval augmentation) when source PDFs exceed LLM context limits.
• Fault Localization and Repair Agents: Integrating diagnostic sub-agents that localize, classify, and self-repair failed code modules to further close the loop toward fully autonomous research code generation.

As LLMs approach parity with human programmers for small-scale algorithm synthesis, the Paper2Code paradigm sets a trajectory for the next frontier: faithfully and scalably reifying full scientific workflows from natural language specification at the research level (Seo et al., 24 Apr 2025).