---
title: 'CCrepair: C++ Compilation Repair Benchmark'
url: https://www.emergentmind.com/topics/ccrepair
type: topic
---

# CCrepair: C++ Compilation Repair Benchmark

CCrepair denotes primarily a large-scale corpus of buggy C++ code and associated compilation-error examples introduced to support automated repair research, whereas CCrepairBench denotes the broader benchmarking and training ecosystem built around that corpus: the CCrepair dataset, a hybrid evaluation pipeline, and a reinforcement learning framework for training and evaluating C++ compilation error repair systems [2509.15690]. The work frames automated C++ compilation error repair as a task that must satisfy two conditions simultaneously—successful compilation and preservation of semantic intent—thereby rejecting compilability alone as an adequate objective and treating trivial deletion as a degenerate repair strategy rather than a valid fix.

## 1. Task definition and scope

CCrepair addresses automated C++ compilation error repair rather than runtime bug fixing or general automated program repair. The task is formalized as learning a mapping from buggy code and optionally compiler diagnostics to fixed code,
\[
\mathcal{F}: (C_{\text{bug}, M_{\text{err}) \rightarrow C_{\text{fix}.
\]
The notation is malformed in the paper, but the intended meaning is explicit: the input is buggy code \(C_{\text{bug}\) and possibly compiler error messages \(M_{\text{err}\); the output is a repaired snippet \(C_{\text{fix}\) [2509.15690].

The benchmark targets C++ compilation errors, specifically syntax and simple semantic issues that prevent successful compilation. The seed set was derived from Microsoft Visual Studio C++ compiler error documentation and then filtered for cross-compiler compatibility, with one explicit example being errors like “newline in constant.” The paper does not provide a clean public taxonomy table of all final retained error classes, but it does enumerate the semantic repair-quality categories used later in evaluation: **Genuine Fix**, **Trivial Deletion**, **Excessive Modification**, and **Invalid Fix**. These are not compiler error classes; they are categories assigned to candidate repairs.

A central design principle is that compilation repair should be treated as semantic-preserving correction rather than compiler appeasement. The paper argues that compilability alone is too weak because a model can “fix” code by deleting functionality, commenting things out, or otherwise making destructive edits. This motivates both the evaluation design and the training objective.

## 2. Dataset construction and benchmark structure

The paper uses the names **CCrepair** and **CCrepairBench** somewhat interchangeably, but it states an intended distinction: **CCrepair** is the dataset/corpus of buggy C++ code and associated compilation-error examples, while **CCrepairBench** is the full benchmark and framework that includes the dataset, the hybrid evaluation pipeline, and the reinforcement learning framework [2509.15690].

Dataset construction is described as a multi-phase generate-and-verify pipeline. In the first phase, the authors scraped the complete list of C++ compiler errors from Microsoft’s official Visual Studio Compiler documentation and collected the compiler error identifier or type, explanatory text, and official code examples. Because documentation examples were often fragments rather than full compilable files, an LLM was used to add boilerplate such as `main` functions, required headers, and surrounding context so that any compilation failure could be attributed unambiguously to the target error.

In the second phase, the seed set was synthetically expanded with **Qwen3-32B**, prompted to create diverse and realistic C++ snippets for each compiler error type. In the third phase, every generated candidate was subjected to verification: the code was compiled, the actual compiler output was captured, and an LLM acting as a judge or validator checked whether the actual compiler output matched the intended target error description. Examples were accepted only when that match was confirmed. The paper attributes the benchmark’s “high-fidelity” character to this compiler-validated and judge-confirmed acceptance criterion.

In the fourth phase, the benchmark filtered out **MSVC-specific errors** with a three-way criterion. An error was considered MSVC-specific only if all three conditions held: it appeared in potential error statistics from the original seed dataset, it appeared in potential error statistics from the LLM-generated corpus, and an LLM-based analysis judged it to be MSVC-specific and likely incompatible with GCC. The fifth phase added human quality control through manual review of excluded MSVC-specific errors and random retained samples.

The paper is explicit that several core dataset statistics remain unclear. It states that the SFT baseline was built from approximately **14,000 high-quality instruction-response pairs** generated by Qwen3-32B from CCrepair data, and that the appendix uses a dataset of **5,522 questions** with a reported difficulty distribution. It also states that training data came from `train.parquet` and validation or test data from `test.parquet`, and that the RL corpus contains fields including **Error type**, **Error number**, **Error type_detail**, and **Error exemplar_1lm_comde**. At the same time, it explicitly says the paper does **not** clearly report the final number of dataset instances in CCrepair, exact train/validation/test sizes, the exact number of final retained error classes, a per-error-class distribution table, a deduplication algorithm, or the precise prompt templates used in generation or verification. The absence of an explicit deduplication procedure is itself stated in the paper and should not be supplemented by inference.

## 3. Hybrid evaluation and semantic judgment

CCrepairBench centers its evaluation on a two-stage pipeline designed to separate syntactic success from semantic validity [2509.15690]. The first stage is an automated compilation check with **GCC**, which yields **CSR (Compilation Success Rate)**: the percentage of repairs that compile. The second stage is semantic evaluation with an **LLM-as-a-Judge**, described as “e.g., a 72B parameter model,” which reviews the original buggy code, the compiler error, and the proposed fix, then assigns one of four labels: **Genuine Fix**, **Trivial Deletion**, **Excessive Modification**, or **Invalid Fix**. This stage yields **GFR (Genuine Fix Rate)**: the percentage of repairs classified as Genuine Fix.

The benchmark does not collapse these signals into a single scalar score. Instead, it emphasizes **CSR + GFR together**. This design directly encodes the paper’s claim that compilation success and semantic validity are distinct. A compilable patch may still be a trivial deletion or an excessive modification.

The judgment component is meta-evaluated against human experts. The validation study uses **5 experts**, **100 questions** randomly selected from the CCrepair corpus, and model-generated repairs. Experts labeled each repair into the same four semantic categories. The class with the highest expert consensus was treated as the standard answer. The paper provides the standard \(F_1\) and Macro \(F_1\) formulas:
\[
F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}{\text{precision} + \text{recall}
\]
\[
\text{MF}_1 = \frac{1}{C} \sum_{i=1}^{C} F_{1,i}
\]
while noting, through its own malformed LaTeX, that the expressions are standard harmonic-mean and macro-averaging definitions. The reported findings are a **25%** random-chance baseline, an **expert final score of 0.592**, and a **judge model score of 0.602**. The paper concludes from this that the judge is a reliable proxy for scalable semantic assessment.

## 4. Reinforcement learning formulation

The training framework models repair generation as a Markov decision process. A state \(s_t\) is the current generated token prefix, an action \(a_t\) is the next token, a full sampled completion is a trajectory \(\tau\), and the policy objective is
\[
J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ R_T(\tau) \right],
\]
where \(\pi_\theta(a_t|s_t)\) is the policy defined by the language model [2509.15690].

The main actor is **Qwen2.5-1.5B-Instruct**, with additional experiments on **Qwen2.5-0.5B-Instruct**. For each training instance, the rollout procedure feeds the buggy code prompt to the actor, samples generated repairs, compiles the repaired code with GCC, passes the buggy code, compiler error, and repair to the LLM judge, computes a reward, and updates the policy using **GRPO/PPO-style optimization**. The paper notes an inconsistency in terminology between **PPO** and **GRPO**, but states that the implementation uses **GRPO**, and that “GRPO also serves as the advantage estimator.”

The defining technical device is a **semantically gated hybrid reward**:
\[
R = S_{\text{judge} + S_{\text{compile},
\]
with
\[
S_{\text{judge} = \begin{cases} 0.5 & \text{if classification is ``Genuine Fix''} \ 0 & \text{otherwise} \end{cases},
\]
and
\[
S_{\text{compile} = \begin{cases} 0.5 & \text{if } S_{\text{judge} > 0 \text{ and code compiles} \ 0 & \text{otherwise} \end{cases}.
\]
The paper’s LaTeX is malformed, but its intended structure is stated plainly: the compilation reward is available only if the repair is first judged a **Genuine Fix**. The maximum reward is therefore **1.0**, and it is reachable only when the repair is both semantically genuine and compilable. This gates the easier objective of compilation behind a semantic prerequisite.

The implementation recipe is unusually concrete. Rollouts are generated with **vLLM**; the training uses **8 samples per instance**, a global batch size of **64**, and **1 epoch**. Optimization uses **AdamW**, a learning rate of \(1 \times 10^{-6}\), a KL regularization coefficient of **0.001**, entropy regularization **disabled**, the low-variance KL estimator `low_var_kl`, a single node with **4 GPUs**, **FSDP**, tensor model parallelism during rollout of **2**, and maximum prompt and response lengths of **2048**. Prompt overflow is handled by filtering examples with truncation mode `'error'`. The paper also states that checkpoints are saved every **50** steps or epochs, and evaluation is run every **5** steps or epochs, while noting that this wording is inconsistent.

## 5. Empirical results

The benchmark evaluates multiple model families, including the Qwen2.5 family, the Qwen3 family, and baselines such as **Distill-Llama-8B**, **Kimi-Dev-72B**, and **llama3.3-70B** [2509.15690]. The paper’s central comparison is between the original **Qwen2.5-1.5B-Instruct**, a supervised fine-tuning baseline built from approximately 14,000 generated repair pairs, and the RL-trained model under the proposed framework.

| Variant | GFR | CSR |
|---|---:|---:|
| Base(1.5B) | 49.9 | 63.9 |
| Base+ (SFT) | 62.1 | 75.2 |
| Ours(1.5B) | 70.8 | 81.9 |

These results support the paper’s claim that RL with environment feedback is better aligned than pure imitation learning. The RL-trained **1.5B** model improves from **49.9 → 70.8** in GFR and **63.9 → 81.9** in CSR, corresponding to **+20.9** and **+18.0** absolute gains. The paper further emphasizes that the RL-trained **Qwen2.5-1.5B-Instruct** is essentially comparable to **Qwen2.5-14B-Instruct** on GFR and slightly higher on CSR: **70.8 / 81.9** versus **71.1 / 78.3**.

The full benchmark table shows that larger frontier models remain stronger in absolute terms on some axes. On the primary CCrepair benchmark, **llama3.3-70B** achieves **82.4 GFR** and **89.7 CSR**, **Qwen2.5-32B** achieves **79.4 / 88.8**, and **Qwen3-235B** achieves **77.2 / 86.2**. The significance of the paper’s result is therefore not that the RL-trained 1.5B model surpasses all larger models, but that the proposed training paradigm makes a compact model competitive with much larger instruction-tuned baselines.

Ablation on the semantic reward is central to the paper’s argument. When the semantic reward is removed, written in the paper as \(S_{\text{judge}=0\), the model reaches **CSR 97.0%** but **GFR only 0.9%**. The authors further report that **96% of successful compilations** in this setting came from **trivial code deletion**. This is the clearest empirical demonstration of the benchmark’s central premise: optimizing for compilability alone produces degenerate repair policies.

The framework also generalizes to a smaller actor. For **Qwen2.5-0.5B-Instruct**, the RL-trained variant improves from **32.8 / 52.9** to **58.2 / 72.8** in GFR/CSR. The paper additionally reports transfer experiments on **MBPP**, **MBPP+**, **HumanEval**, and **HumanEval+**, interpreting the generally competitive downstream performance as evidence that repair training can improve deeper code understanding.

## 6. Limitations and position in the repair literature

The paper explicitly acknowledges that CCrepairBench has several limitations [2509.15690]. It depends on an **LLM-as-a-Judge**, which, even after validation, remains a learned proxy rather than a formal semantic verifier. It also leaves key dataset statistics underreported, including final benchmark size, split sizes, and full error taxonomy. Semantic preservation is judged rather than proven, and the benchmark still reflects its origin in MSVC documentation even after GCC-oriented filtering. The paper itself suggests future work with stronger evaluation mechanisms such as multi-judge ensembles or static analysis tools.

Within the broader compilation-repair literature, CCrepairBench occupies a distinct position. **TransRepair** addresses automatic repair of compilation errors in **C programs**, jointly learning localization and replacement generation from erroneous code, context, and diagnostic feedback, and reports a synthetic training dataset of **1,821,275 erroneous C programs** [2210.03986]. **Defects4C** provides a comprehensive and executable benchmark specifically for **C/C++ program repair**, with **248 high-quality buggy functions**, **102 vulnerable functions**, and test cases for reproduction [2510.11059]. **ComBench** introduces a **repository-level, reproducible real-world benchmark for C/C++ compilation error repair** built from GitHub CI histories, with **200** execution-verified instances and a reported gap between syntactic correctness and semantic correctness, exemplified by **GPT-5** reaching **73% compile success** but only **41%** semantically correct patches [2603.27333].

This suggests that CCrepairBench should be understood not as a repository-level benchmark in the style of ComBench, nor as an executable bug benchmark in the style of Defects4C, nor as a direct successor to C-specific line-repair models such as TransRepair. Rather, it defines a distinct line of work in which a dataset, a semantic rubric, and an RL objective are aligned around a narrow but practically important thesis: a useful compilation repair system should optimize for repairs that are both compilable and semantically non-destructive. Its most characteristic contribution is therefore not a single model architecture, but the integration of a **high-fidelity C++ compilation-error dataset**, a **two-stage CSR/GFR evaluation pipeline**, and a **semantically gated RL objective** into one training and benchmarking ecosystem.

Source: https://www.emergentmind.com/topics/ccrepair