---
title: 'RECODE-H: Research Code Benchmark'
url: https://www.emergentmind.com/topics/recode-h
type: topic
---

# RECODE-H: Research Code Benchmark

Searching arXiv for recent papers on “RECODE-H” and closely related entries to ground the article.
RECODE-H is a benchmark for evaluating large language model agents on research code development through iterative, feedback-driven interaction rather than one-shot code completion. It is defined as “Research COde DEvelopment – with Hierarchical feedback” and comprises 102 repository-level tasks derived from real research papers and their official open-source repositories. Each task requires implementing or revising research-method code inside an existing codebase, with performance measured over multi-turn interactions in which an agent writes code, runs unit tests, receives LLM-simulated human feedback at one of five feedback levels, and iteratively refines its solution [2510.06186].

## 1. Definition and scope

RECODE-H was introduced to address a specific gap in the evaluation of LLM-based coding systems: existing benchmarks for research code generation largely assume a one-shot setting, whereas realistic scientific programming is iterative and feedback-driven. In RECODE-H, an agent is evaluated not only on whether it can produce correct code, but on whether it can use diagnostic feedback to improve code across multiple turns. This benchmark therefore targets research implementation workflows rather than generic software engineering or isolated function synthesis [2510.06186].

The benchmark contains 102 tasks drawn from real papers and repositories. Each task focuses on implementing parts of an algorithm or method, including functions, classes, modules, loss functions, or training steps inside a real repository. The task distribution spans machine learning and deep learning, NLP, vision, and other computational science areas, with an emphasis on translating paper mathematics and method descriptions into executable repository-compatible code rather than solving toy programming exercises [2510.06186].

A central property of RECODE-H is that it operationalizes researcher–agent collaboration. The agent is placed in an environment that includes paper excerpts, structured implementation instructions, annotated canonical code, unit tests, repository browsing tools, and simulated human feedback. This setup makes the benchmark a study of adaptive coding under guidance rather than static code generation alone [2510.06186].

## 2. Task construction and benchmark components

Each RECODE-H task is built from a real research paper and its official open-source repository. The benchmark materials are curated by PhD-level researchers familiar with the underlying methods. Tasks are repository-level and often require implementing multiple functions or full classes, rather than filling in a single short routine [2510.06186].

For each task, the benchmark provides four components. First, it includes paper-derived context consisting of LaTeX excerpts from the relevant method sections, equations, and descriptions. Second, it provides a structured instruction written and refined by human annotators, specifying the target function or class name, a natural-language description of its role, explicit input and output signatures with names, shapes, dtypes, and semantics, and clarifications where the paper is ambiguous or incomplete. Third, it includes explanatory comments on the canonical repository code, clarifying how the implementation corresponds to the paper, where implementation tricks appear, and where discrepancies exist between paper and code. Fourth, it includes unit tests, with at least one test for each interface and an annotator target of at least 80% coverage of the canonical code [2510.06186].

This design makes RECODE-H neither a pure paper-to-code benchmark nor a pure repository-repair benchmark. It requires alignment across three objects: the paper specification, the repository conventions, and the unit-test-defined functional behavior. A plausible implication is that the benchmark is particularly sensitive to failures of method interpretation and repository integration, rather than only to syntax errors.

| Component | Content | Role |
|---|---|---|
| Paper-derived context | LaTeX excerpts, equations, descriptions | Grounds implementation in the research method |
| Structured instruction | Function/class target, purpose, I/O signatures, clarifications | Defines the implementation task precisely |
| Canonical-code comments | Annotated reference implementation | Explains paper-to-code correspondence |
| Unit tests | Interface-specific tests, target coverage ≥80% | Measures functional correctness |

The benchmark therefore models research coding as specification alignment under partial observability. The paper text and structured instruction may underdetermine implementation details, while repository context and feedback supply the additional constraints needed to converge on the canonical behavior [2510.06186].

## 3. Interactive protocol and hierarchical feedback

RECODE-H converts each task into a multi-turn dialogue between an LLM agent and an LLM-simulated researcher. On turn \(t\), the agent observes the paper excerpt \(P\), instruction \(I\), current code \(C_t\), prior interaction history \(H_{<t}\), and previous test results and feedback. It can then take repository-aware actions such as READ, BROWSE, RETRIEVE, REPLACE, and SUBMIT. A SUBMIT action executes unit tests, collects logs, and triggers feedback generation [2510.06186].

The paper formalizes the turn state as
\[
s_t = (P, I, C_t, H_{<t}, \text{logs}_{\le t}),
\]
with a feedback function
\[
F_{\ell}(C_t, \text{logs}_t, C^\star, \text{comments}) \rightarrow \text{textual feedback},
\]
where \(\ell \in \{0,1,2,3,4\}\) is the feedback level, \(C^\star\) is the canonical code, and the output is feedback conditioned on test results, annotated reference code, and task specification [2510.06186].

The distinctive contribution is the five-level feedback hierarchy. Level 0 gives only failure status and raw logs. Level 1 adds a high-level error description. Level 2 adds an explanation of why the error occurred. Level 3 adds actionable correction guidance in natural language. Level 4 adds the correct code snippet for the faulty part, drawn from canonical code. The hierarchy thus forms a controlled ladder of supervision from uninterpreted execution traces to direct code-level correction [2510.06186].

| Feedback level | Content |
|---|---|
| Level 0 | Raw execution/test logs only |
| Level 1 | Level 0 plus high-level error description |
| Level 2 | Level 1 plus causal diagnostic explanation |
| Level 3 | Level 2 plus actionable correction guidance |
| Level 4 | Level 3 plus correct code snippet |

This hierarchy is not merely an annotation convenience. It is the organizing principle of the benchmark’s experimental design, enabling controlled study of how much semantic and corrective signal different models can absorb from feedback. The paper’s results suggest that even modest semantic enrichment over raw logs materially changes agent behavior [2510.06186].

## 4. ReCodeAgent and the execution framework

Alongside the benchmark, the paper introduces ReCodeAgent, a generic tool-using LLM agent for interacting with RECODE-H. ReCodeAgent follows a ReAct-style loop organized into observation, reflection, planning, and action. On each turn it gathers the current repository state, code, logs, and feedback; analyzes failure causes and deviations from the paper and instructions; plans concrete edits or retrieval steps; and issues one of the tool actions READ, BROWSE, RETRIEVE, REPLACE, or SUBMIT [2510.06186].

The system prompt enforces a structured output format:
```text
reflect: [reasoning and plan]
action: [one of READ / RETRIEVE / BROWSE / REPLACE / SUBMIT]
```
This is intended to encourage explicit reasoning and repository-aware tool use rather than direct unguided rewriting [2510.06186].

To manage long interaction histories, ReCodeAgent uses Reflexion-style memory. It stores a finite memory of recent observations and actions, and when capacity exceeds a threshold of 5, older entries are summarized into a meta-memory that records unresolved failures, key design decisions, and the evolution of the code. This mechanism is intended to mitigate context-window growth while preserving salient debugging information [2510.06186].

ReCodeAgent differs from naive iterative sampling in three respects. It is explicitly multi-turn, it conditions revisions on structured feedback rather than only on stochastic retries, and it is repository-aware through browsing and retrieval. This suggests that RECODE-H is also an evaluation of agent scaffolding quality, not just base-model coding skill.

## 5. Evaluation protocol, metrics, and empirical findings

RECODE-H evaluates both functional correctness and code similarity. If a task is first solved at turn \(k\), its Mean Reciprocal Rank is
\[
\text{MRR} = \frac{1}{k},
\]
and \(0\) if never solved. Recall@\(\,n\) measures the fraction of tasks solved within \(n\) turns. The benchmark also reports average test-case pass rate for partial correctness, and code similarity metrics using canonical code \(C^\star\) and model code \(\hat{C}\): CodeBLEU and CodeBERTScore [2510.06186].

The experimental setup runs each task for up to 10 interaction turns, with up to 3 actions per turn before a forced SUBMIT. Models evaluated include GPT-5-nano, GPT-5-mini, GPT-5, DeepSeek-V3.1, Claude-Sonnet-4, Gemini-2.5-pro, and Gemini-2.5-flash. Decoding is deterministic with temperature \(0\) and top-\(p=1\). The feedback model is GPT-o4-mini for all tasks and levels [2510.06186].

The main empirical finding is that richer feedback consistently improves performance, with the largest jump typically occurring between Level 0 and Level 1. For Recall after 10 turns, GPT-5 improves from 0.294 at Level 0 to 0.716 at Level 4; DeepSeek-V3.1 improves from 0.108 to 0.706; GPT-5-mini from 0.196 to 0.667; and Gemini-2.5-flash from 0.088 to 0.588 [2510.06186]. The paper characterizes these gains as substantial, while also noting diminishing returns at higher feedback levels.

Model families differ in feedback adoption efficiency. The GPT-5 family shows clear scaling behavior, with GPT-5-nano \(<\) GPT-5-mini \(<\) GPT-5 across all feedback levels. DeepSeek-V3.1 begins weakly at Level 0 but improves sharply with richer feedback, reaching Recall \(= 0.706\) and MRR \(= 0.210\) at Level 4. Claude-Sonnet-4 improves less dramatically and appears to plateau earlier. Gemini-2.5-flash benefits more reliably than Gemini-2.5-pro, whose gains fluctuate and are smaller [2510.06186].

The turn-by-turn dynamics reinforce this interpretation. With Levels 3 and 4 feedback, strong models such as GPT-5 and DeepSeek-V3.1 achieve most gains within 3–4 turns, whereas Level 0 improvements are slower and plateau at lower pass rates. This suggests that semantic compression of error state into concise feedback is a major factor in sample-efficient interactive correction [2510.06186].

## 6. Error taxonomy, interpretation, and position in the literature

The paper classifies agent failures into four types: Type 1 syntax and runtime errors, Type 2 paper or instruction misunderstanding, Type 3 missing knowledge and context, and Type 4 repository integration errors. Across models, Type 2 and Type 3 dominate by a large margin. For GPT-5, Type 2 accounts for 34.0% and Type 3 for 50.3%, while Type 1 is only about 11% and Type 4 about 4% [2510.06186]. This indicates that the principal challenge in research code development is not low-level syntax generation but faithful interpretation of research methods and implicit implementation context.

The paper also analyzes feedback adoption and resolution. Successful fixes almost always coincide with explicit adoption of feedback, while blind successes are rare. Strong improvers such as GPT-5, GPT-5-mini, DeepSeek-V3.1, and Gemini-2.5-flash show adoption rates increasing with feedback level, reaching roughly 90% or higher at Level 4. Weaker improvers such as Claude-Sonnet-4 and Gemini-2.5-pro exhibit lower or even declining adoption at higher levels, corresponding to flatter performance improvements [2510.06186].

A case study on implementing a min-\(p\) sampling strategy illustrates the benchmark’s emphasis on algorithmic alignment rather than superficial correctness. The model’s initial solution used a different logic for `min_tokens_to_keep`, producing subtle discrepancies despite plausible code structure. Feedback at roughly Levels 3–4 identified the logical mismatch, explained the canonical use of `torch.argsort`, `gather`, and `scatter`, and supplied a corrective snippet; the revised implementation then matched canonical behavior and passed tests [2510.06186]. This example shows that RECODE-H is designed to expose semantic deviations that are invisible to shallow code inspection.

Within the broader benchmark landscape, RECODE-H occupies the intersection of research-code benchmarks and interactive coding benchmarks. The paper relates it to function-level code benchmarks such as HumanEval and MBPP, repository-level benchmarks such as SWE-bench, DevEval, and CrossCodeEval, research-code benchmarks such as MLE-Bench, PaperBench, SciReplicate, and ResearchCodeBench, and interactive coding benchmarks such as ConvCodeWorld, MINT, and InterCode [2510.06186]. Its novelty lies in combining repository-level research implementation tasks with explicit multi-turn interaction and a five-level feedback hierarchy.

The benchmark has several stated limitations. Its scale is 102 tasks, which is high quality but limited in coverage; it is biased toward CS and ML-style research; feedback is simulated by GPT-o4-mini rather than produced by actual researchers; unit tests may miss latent bugs or overfit to canonical implementations; and the use of canonical code in feedback raises leakage concerns, although the paper reports leakage as negligible for Levels 1–3, below 2%, with Level 4 intentionally including code segments [2510.06186]. These constraints delimit what the benchmark measures: a reproducible approximation to researcher–agent collaboration rather than human-in-the-loop field deployment.

A common ambiguity arises from the name. “RECODE-H” in this context refers specifically to the benchmark “Research COde DEvelopment – with Hierarchical feedback” and should be distinguished from unrelated “ReCode” or “RECODE” systems in agent planning, code robustness, code-update reinforcement learning, visual relation detection, or reactor neutrino experiments [2510.06186]. This suggests that the benchmark’s most stable identifier in scholarly use is its task definition and feedback hierarchy, not the broader ReCode naming family.

Source: https://www.emergentmind.com/topics/recode-h