---
title: 'RACodeBench: Real-World Code Repair Benchmark'
url: https://www.emergentmind.com/topics/racodebench
type: topic
---

# RACodeBench: Real-World Code Repair Benchmark

RACodeBench is a benchmark dataset and evaluation protocol constructed to measure the effectiveness and efficiency of automated code repair techniques on realistic, real-world buggy code. Designed to overcome the limitations of prior synthetic benchmarks, RACodeBench focuses on real user-submitted defects, rich algorithmic diversity, and fine-grained error annotation, facilitating rigorous comparative evaluation and diagnostic analysis in the domain of program repair, especially for large language model (LLM)-based and retrieval-augmented code repair methods [2509.02330].

## 1. Motivation and Design Rationale

Synthetic code-repair benchmarks have historically failed to capture the complexity and distributional characteristics of real programming errors encountered by users. RACodeBench was created to bridge this gap by curating a large-scale dataset from Codeforces, a competitive programming platform with a diverse, publicly available submission history. Its design objectives include:

- Representing the actual distribution of bug types and algorithmic intricacies observed in user code.
- Structuring data to allow efficient, realistic assessment of retrieval-based and LLM-driven repair methods.
- Enforcing strict dataset splits to prevent information leakage and enable trustworthy generalization measurements.

A plausible implication is that RACodeBench provides a more robust substrate for research in code repair than synthetic datasets, especially for benchmarking retrieval-augmented and prompt-driven repair agents [2509.02330].

## 2. Dataset Construction and Characteristics

RACodeBench comprises 4,200 buggy/fixed code pairs selected from 1,200 Codeforces problems, each chosen for having at least three distinct user-submitted incorrect solutions later fixed by their original authors. The processing pipeline includes:

- **Parsing and AST Validation**: Ensures that only syntactically valid fixed versions are included.
- **Normalization**: Removes comments and standardizes identifier names, increasing the signal-to-noise ratio for pattern-based retrieval and LLM models.
- **Tree-based Diffing**: Extracts minimal edit scripts between buggy and fixed versions to support fine-grained diagnostic tasks.

Bug types are annotated via a semi-automated process and manual curation, with the following empirical distribution:

| Bug Type          | Percent (%) |
|-------------------|------------|
| Off-by-one        | 25         |
| Logic errors      | 22         |
| Incorrect loops   | 18         |
| Edge-case omission| 15         |
| Null/uninit var   | 10         |
| Other             | 10         |

Algorithmic domains covered include dynamic programming (25%), greedy algorithms (20%), graph search (20%), sorting/searching (15%), number theory/combinatorics (10%), and miscellaneous paradigms (10%).

Dataset splits:
- **Train**: 840 problems (≈2,940 pairs)
- **Validation**: 120 problems (≈420 pairs)
- **Test**: 240 problems (≈840 pairs)

All splits are stratified by domain and difficulty, and no problem appears simultaneously in retrieval KB and benchmark splits.

## 3. Evaluation Protocol and Metrics

RACodeBench measures both the correctness of repaired code and the resource cost of achieving such repairs. The evaluation protocol includes two principal accuracy metrics:

- **Strict Accuracy** (`StrictAcc`): The proportion of test problems for which, among the $N$ generated repairs, at least one exactly matches all reference outputs across all $K$ test cases:
  $$
  \text{StrictAcc} = \frac{1}{P}\sum_{p=1}^P \max_{1\le j\le N}\prod_{k=1}^K \mathbf{1} \{\mathrm{exec}(\text{code}_p^j, x_{p,k}) = y_{p,k}\}
  $$
- **Test Pass Rate** (`PassRate`): The average maximum per-problem fraction of passed test cases across all repairs:
  $$
  \text{PassRate} = \frac{1}{P}\sum_{p=1}^P \max_{1\le j\le N}\frac{1}{K} \sum_{k=1}^K \mathbf{1} \{\mathrm{exec}(\text{code}_p^j, x_{p,k}) = y_{p,k}\}
  $$

**Inference cost** is reported as the total number of LLM calls (spanning both retrieval and generation steps), making it possible to assess repair accuracy as a function of computational expenditure.

Baselines include:
- **Best-of-N**: $N$ independent generations, select best by test pass.
- **Self-Repair**: Feedback-driven sequence (generation, feedback, second attempt), total $N$ calls per problem.
- **Retrieval-augmented methods**: Such as ReCode, with explicit allocation of LLM calls to both problem classification and repair.

## 4. Empirical Performance and Failure Modes

Experiments on RACodeBench with GPT-4o-mini ($N = 8$ LLM calls) show that retrieval-augmented in-context methods like ReCode significantly outperform standard baselines at equal inference cost:

| Method        | Test Pass Rate (%) | Strict Accuracy (%) | Inference Calls |
|---------------|-------------------|--------------------|-----------------|
| Best-of-N     | 31.09             | 21.25              | 8               |
| Self-Repair   | 34.79             | 24.58              | 8               |
| ReCode        | 41.06             | 30.41              | 8 (2+6)         |

ReCode exhibits:
- Higher PassRate (+6.27 points) and StrictAcc (+5.83 points) versus strongest baseline.
- Faster convergence: surpasses Best-of-N by call 4 out of 8.
- Superior out-of-distribution (OOD) generalization; for instance, on AtCoder data, ReCode attains 37% PassRate with only five LLM calls, while baselines require approximately 18.

Failure analyses reveal that retrieval quality is sometimes degraded by multi-paradigm problems (e.g., graph and dynamic programming combinations), rare language constructs, and logic errors requiring mathematical insight beyond exemplar patterns.

## 5. Diagnostic and Practical Implications

RACodeBench provides fine-grained error annotations and a hierarchical structure, supporting diagnostic analyses such as bug type- or domain-specific breakdowns. Recommendations for usage include:

- Reporting both PassRate and StrictAcc to distinguish full from partial correctness.
- Always disclosing inference cost (number of LLM calls or latency).
- Preventing overlap between retrieval KB and evaluation splits to simulate genuine out-of-distribution repair scenarios.
- Leveraging detailed error annotation for post hoc analysis of systemic strengths and weaknesses in repair strategies.

A plausible implication is that RACodeBench enables research not only into accuracy but also into computational efficiency and robustness, given its precisely defined resource accounting and split discipline [2509.02330].

## 6. Relationship to Related Benchmarks

RACodeBench departs from previous approaches by grounding its instances in real user behavior, curating a rich and error-diverse library surpassing the synthetic, single-function, or codelet-focused datasets common in earlier work. While HumanEval, SWE-bench, and similar benchmarks provide valuable generalization targets, they lack the algorithmic diversity, fine-grained error labeling, and realistic repair traces that RACodeBench supplies.

RACodeBench is not directly concerned with root-cause analysis, such as that benchmarked by RCABench [2303.05029]; its focus is specifically on functional repair verified by I/O test suites. Nevertheless, the use of community-curated errors and the emphasis on fine-grained, interpretable evaluation metrics show an analogous response to similar dataset and evaluation deficiencies.

## 7. Limitations and Prospective Extensions

RACodeBench, while substantially advancing the realism and diagnostic value of code repair benchmarks, has certain constraints:
- Coverage is limited to C++, Python, and Java, reflecting the Codeforces population.
- Its design does not directly address programming tasks outside competitive programming (e.g., large multi-module systems).
- The dataset is static and may require expansion as algorithms or languages evolve.

Future improvements could include augmentations for language or domain breadth, task complexity, or annotating multi-turn repair conversations for feedback-based repair strategies.

---

In summary, RACodeBench stands as a rigorously constructed, real-world benchmark for code repair, enabling accurate, efficient, and interpretable evaluation of LLM-based and retrieval-augmented approaches in automated program repair [2509.02330].

Source: https://www.emergentmind.com/topics/racodebench