Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning (2504.17192v3)

Abstract: Despite the rapid growth of machine learning research, corresponding code implementations are often unavailable, making it slow and labor-intensive for researchers to reproduce results and build upon prior work. In the meantime, recent LLMs excel at understanding scientific documents and generating high-quality code. Inspired by this, we introduce PaperCoder, a multi-agent LLM framework that transforms machine learning papers into functional code repositories. PaperCoder operates in three stages: planning, where it constructs a high-level roadmap, designs the system architecture with diagrams, identifies file dependencies, and generates configuration files; analysis, which focuses on interpreting implementation-specific details; and generation, where modular, dependency-aware code is produced. Moreover, each phase is instantiated through a set of specialized agents designed to collaborate effectively across the pipeline. We then evaluate PaperCoder on generating code implementations from machine learning papers based on both model-based and human evaluations, particularly from the authors of those papers, with author-released repositories as ground truth if available. Our results demonstrate the effectiveness of PaperCoder in creating high-quality, faithful implementations. Furthermore, it consistently shows strengths in the recently released PaperBench benchmark, surpassing strong baselines by substantial margins. Code is available at: https://github.com/going-doer/Paper2Code.

• The paper introduces PaperCoder, a multi-agent LLM framework that translates machine learning research papers into executable code repositories.
• It employs a structured three-stage workflow—planning, analysis, and coding—to decompose and implement code generation akin to human software development.
• Evaluations on new benchmarks and human ratings highlight improved reproducibility, high code completeness, and minimal debugging effort.

The paper "Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning" (2504.17192) addresses the challenge of low code availability for machine learning research papers, which hinders reproducibility and slows down scientific progress. Manually re-implementing methods from papers is a time-consuming and labor-intensive process. To tackle this, the authors introduce PaperCoder, a multi-agent LLM framework designed to automatically generate executable code repositories directly from machine learning papers.

PaperCoder emulates the typical human software development workflow by decomposing the code generation task into three structured stages:

1. Planning: In this initial stage, PaperCoder creates a high-level roadmap for implementation. This involves several steps performed by a specialized "plan agent":
   • Overall Plan: Summarizing the core elements of the paper relevant to implementation.
   • Architecture Design: Designing the system architecture, including identifying necessary files (file list), defining data structures and interfaces using class diagrams (UML notation), and illustrating dynamic interactions with sequence diagrams (UML notation).
   • Logic Design: Analyzing file dependencies and determining the optimal order for implementing files to ensure a correct build and execution flow. This produces an ordered file list along with detailed logic descriptions for each file.
   • Configuration File Generation: Generating a configuration file (e.g., config.yaml) containing hyperparameters and settings required for experiments, which can be reviewed and modified by users to reduce potential hallucinations during coding.
2. Analysis: Following the planning phase, the analysis stage focuses on interpreting implementation-specific details for each individual file identified in the plan. A specialized "analysis agent" takes the paper and the planning artifacts as input to produce detailed, file-level specifications. These specifications outline the purpose of each file, required inputs/outputs, interactions with other modules, and any constraints derived from the paper.
3. Coding: The final stage involves the actual code generation. A "coder agent" synthesizes the entire codebase based on the outputs from the planning and analysis stages, including the paper content, overall plan, architecture design, logic design (especially the ordered file list), configuration file, and the detailed file-specific analyses. Code is generated sequentially according to the dependency order determined during logic design, with previously generated code provided as context for subsequent files. The authors note that generated code may require debugging, although detailed debugging strategies are outside the paper's scope.

The authors evaluate PaperCoder extensively using two benchmarks:

• Paper2Code Benchmark: A new dataset created by the authors comprising 90 papers from ICML, NeurIPS, and ICLR 2024 with publicly available code repositories under 70,000 tokens. Evaluation is performed using model-based metrics in two settings: reference-based (comparing generated code against the paper and author-released code) and reference-free (comparing generated code against the paper only).
• PaperBench Code-Dev Benchmark: A pre-existing benchmark consisting of 20 ICML 2024 papers, used to evaluate replication accuracy based on hierarchical implementation requirements defined by the original authors.

Results show that PaperCoder consistently outperforms strong baselines like ChatDev and MetaGPT, as well as naive approaches (using only the abstract or full paper directly), across all evaluation metrics on the Paper2Code benchmark. PaperCoder achieves the highest average correctness scores in both reference-based and reference-free settings. It also produces code repositories with a significantly higher number of functions compared to baselines, indicating greater granularity and completeness. On the PaperBench Code-Dev benchmark, PaperCoder achieves a substantially higher replication score than the baseline agents (BasicAgent and IterativeAgent).

Crucially, human evaluations conducted with 13 authors of the original papers demonstrate the practical utility of PaperCoder. 77% of authors rated PaperCoder's output as the best among alternatives, and 85% found the generated repositories helpful for reproducing their work compared to starting from scratch. The main reasons cited for preferring PaperCoder were completeness, clean structure, and faithfulness to the original paper. Section-level analysis showed high coverage (85%) for implementing the core "Method" section. An executability analysis on five representative papers indicated that the generated code was largely functional, requiring only minor modifications (average 0.48% of total lines) for successful execution, often involving routine fixes like updating API calls.

Ablation studies confirm the importance of PaperCoder's multi-stage pipeline, showing performance improvements as planning, analysis, and configuration components are added incrementally. Experiments with different LLM backbones highlight the significant impact of the underlying model capabilities, with o3-mini-high performing best across all evaluation settings. The authors also found a strong positive correlation (Pearson $r=0.79$) between reference-based and reference-free model evaluations, suggesting the latter is a reliable proxy when ground truth code is unavailable.

The primary limitations identified are the current scope being limited to machine learning papers and the reliance on model-based evaluation, although human evaluations and manual debugging case studies provide supplementary validation. Future work includes expanding to other scientific domains and developing more scalable automated execution-based evaluation and debugging capabilities.

Overall, PaperCoder demonstrates a promising approach to automating the translation of machine learning research papers into functional code repositories, potentially accelerating scientific discovery and improving reproducibility in the field.