---
title: 'RExBench: Benchmark for LLM Code Extensions'
url: https://www.emergentmind.com/topics/rexbench
type: topic
---

# RExBench: Benchmark for LLM Code Extensions

RExBench is a benchmark suite designed to measure the capability of large language model (LLM)-based coding agents to autonomously implement realistic research extensions. Its purpose is to systematically evaluate progress toward LLM agents that can perform nontrivial extensions to existing research codebases, a critical function for automating the research pipeline in machine learning and related scientific domains. Unlike prior benchmarks focused on code repair or isolated algorithmic challenges, RExBench centers on the end-to-end process of understanding a research paper, interpreting expert instructions for novel hypothesis-driven extensions, and executing complex codebase modifications that result in fully validated new experiments. 

## 1. Definition and Construction

RExBench comprises 12 extension tasks derived from published AI/ML research papers, each accompanied by a codebase and a domain expert-written instruction describing a novel, never-before-implemented experiment or research hypothesis. The benchmark is explicitly robust to data contamination: all extension tasks are unpublished and unavailable to public large-scale models, ensuring that LLM outputs cannot simply reflect pre-existing Internet data.

For each task, the agent receives the original paper (typically in PDF or Markdown), the associated codebase, and the extension instruction. The agent then outputs a patch file—a set of code edits—to implement the extension. An automatic evaluation infrastructure applies this patch in a clean environment and runs the resulting pipeline, comparing outputs against expert reference solutions to assess success. The evaluation criteria require high fidelity: for deterministic experiments, output must match exactly; for stochastic pipelines, qualitative and quantitative agreement is enforced by running experiments with seed-controlled, expert-generated references.

## 2. Design of Tasks and Evaluation Protocol

The suite’s 12 tasks span a representative set of extension scenarios across contemporary AI/ML research, including but not limited to:

- Model and algorithmic modifications (e.g., changing architectures, adding new training objectives),
- Data and preprocessing changes (e.g., altering datasets in linguistically or empirically controlled ways),
- Experimental setup and evaluation procedure adjustments (e.g., implementing alternative metrics or ablations).

An exemplary task requires the agent to modify a linguistics dataset so that novel word replacements correspond to actual English words matched for part-of-speech and frequency, regenerate altered data, rerun experiments, and wrap the whole process in a single script—all adhering to strict output conventions dictated by the instruction.

Evaluation involves a multi-pronged set of outcomes:

- **Final Success Rate**: The proportion of tasks for which the agent’s patch results in output that matches (exact or within seed-bound) the reference.
- **Execution Success Rate**: Whether the edited code runs end-to-end without errors.
- **File Recall**: The fraction of gold-standard modified files that are also edited by the agent, defined by:
  \[
  \text{File Recall} = \frac{|\text{Files}_{\text{agent}} \cap \text{Files}_{\text{gold}}|}{|\text{Files}_{\text{gold}}|}
  \]
All evaluations are performed in a containerized environment, mapping tasks to resources such as A100 GPUs, and driven by automated scripts (e.g., `run_apptainer.sh`). Reference solutions are never public, ensuring evaluation integrity.

## 3. Agent Benchmarks and Outcomes

Nine LLM agent configurations, spanning three frameworks (aider, Claude Code, OpenHands) and multiple model backbones (Claude 3.7 Sonnet, OpenAI o1, o4-mini, DeepSeek-R1), were systematically assessed on RExBench. Each agent was required to autonomously produce patches implementing each of the 12 tasks. 

Observed performances were as follows:

- The highest final success rate for any agent/task combination was 25% (OpenHands + Claude 3.7 Sonnet and Claude Code with Claude 3.7 Sonnet).
- Most agent/model pairs, particularly those based on OpenAI o1 and DeepSeek-R1, yielded close to 0% success.
- Execution success (patch runs without error) exceeded final success, indicating that code which runs is frequently semantically incorrect.
- File recall was generally high, suggesting that the agents could often localize the correct files even when their edits failed at the logic level.

The inclusion of human-written hints (both at the information-localization and detailed step-by-step levels) improved performance for all agents, with the best configuration reaching 39% success. However, this improvement saturated quickly, and overly detailed hints sometimes led to performance declines, suggesting bottlenecks in agent comprehension or adaptation to guidance.

| Agent       | Best Success (%) | Backbone           | With Hints   |
|-------------|------------------|--------------------|--------------|
| aider       | 14               | Claude 3.7 Sonnet  | Up to 17     |
| Claude Code | 25               | Claude 3.7 Sonnet  | Up to 28     |
| OpenHands   | 25               | Claude 3.7 Sonnet  | Up to 39     |
| ...         | ...              | o1, o4-mini, DeepSeekR1 | ~0–8   |

## 4. Sources of Difficulty and Error Analysis

A salient finding of RExBench is the diverse range of failure modes exhibited by current coding agents, classified as follows:

- **Complex Codebase Understanding**: Tasks frequently require nuanced global reasoning about codebase structure, data flow, and integration, beyond the scope of local modifications.
- **Planning and Decomposition**: Agents often attempt one-shot fixes instead of modular, staged changes, leading to incomplete or non-functional solutions.
- **Execution Errors**: Common Python and programming errors (attribute errors, import problems, file-not-found, syntax errors) and non-operative (empty) code edits were prevalent.
- **Implicit (Logic) Errors**: The most difficult category consists of patches that run successfully but yield the wrong scientific output due to subtle bugs, incorrect hyperparameters, or misaligned logic.
- **Hint Utilization Difficulties**: Hints did not linearly increase success; some agents failed to use granular advice effectively, possibly due to conflicting planning routines between frameworks and LLM self-reasoning.

The primary statistical predictor of agent failure was the number of necessary line changes in the gold-standard reference, rather than codebase size, popularity, or field of the underlying research.

## 5. Broader Implications and Future Development

RExBench provides strong evidence that, as of 2025, LLM-based coding agents are markedly limited in their capacity to autonomously implement nontrivial research extensions, even with moderate human scaffolding. Explicit agent success is rare, with implicit incorrectness presenting a significant undetected risk for scientific automation.

A plausible implication is the need for advances in agent planning and decomposition, improved comprehension of experimental context, and better integration of guidance and tool-use capabilities. The high prevalence of plausible but incorrect solutions highlights risks to the reproducibility and validity of AI-accelerated research pipelines.

Recognized directions for development include:

- Expansion to broader research domains and more complex multi-stage extensions.
- Introduction of intermediate, process-level assessments and external checking tools for error localization and diagnosis.
- Community curation of new benchmarks and systematic evaluation of both agent and human-augmented workflows.
- Continued balancing of benchmark realism (complexity and openness of tasks) with automatable and reproducible assessment methods.
- Consideration of societal impacts, particularly in error detection and mitigation, prior to possible deployment in production research environments.

## 6. Summary

RExBench systematically measures the ability of LLM-driven coding agents to autonomously implement research experiment extensions by requiring integration of novel, unpublished experimental logic into real codebases under rigorous, contamination-robust, and fully automated evaluation. Results to date indicate that while current agents possess limited code search and file-localization ability, they are unable to reliably execute multi-step, logic-rich modifications required for authentic research progress, even with moderate human guidance. RExBench thus establishes a baseline for progress toward autonomous research agents, and serves as a controlled platform for tracking community advancement in this critical area.

Source: https://www.emergentmind.com/topics/rexbench