CCrepair: C++ Compilation Repair Benchmark

• CCrepair is a large-scale dataset of buggy C++ code paired with compiler errors to benchmark semantically sound repair systems.
• It employs a multi-phase generate-and-verify pipeline, using synthetic expansion and compiler validation to curate high-fidelity examples.
• The reinforcement learning framework uses a semantically gated reward to promote genuine fixes over trivial deletions, enhancing both CSR and GFR.

CCrepair denotes primarily a large-scale corpus of buggy C++ code and associated compilation-error examples introduced to support automated repair research, whereas CCrepairBench denotes the broader benchmarking and training ecosystem built around that corpus: the CCrepair dataset, a hybrid evaluation pipeline, and a reinforcement learning framework for training and evaluating C++ compilation error repair systems (Sun et al., 19 Sep 2025). The work frames automated C++ compilation error repair as a task that must satisfy two conditions simultaneously—successful compilation and preservation of semantic intent—thereby rejecting compilability alone as an adequate objective and treating trivial deletion as a degenerate repair strategy rather than a valid fix.

1. Task definition and scope

CCrepair addresses automated C++ compilation error repair rather than runtime bug fixing or general automated program repair. The task is formalized as learning a mapping from buggy code and optionally compiler diagnostics to fixed code,

$\mathcal{F}: (C_{\text{bug}, M_{\text{err}) \rightarrow C_{\text{fix}.$

The notation is malformed in the paper, but the intended meaning is explicit: the input is buggy code $C_{\text{bug}$ and possibly compiler error messages $M_{\text{err}$; the output is a repaired snippet $C_{\text{fix}$ (Sun et al., 19 Sep 2025).

The benchmark targets C++ compilation errors, specifically syntax and simple semantic issues that prevent successful compilation. The seed set was derived from Microsoft Visual Studio C++ compiler error documentation and then filtered for cross-compiler compatibility, with one explicit example being errors like “newline in constant.” The paper does not provide a clean public taxonomy table of all final retained error classes, but it does enumerate the semantic repair-quality categories used later in evaluation: Genuine Fix, Trivial Deletion, Excessive Modification, and Invalid Fix. These are not compiler error classes; they are categories assigned to candidate repairs.

A central design principle is that compilation repair should be treated as semantic-preserving correction rather than compiler appeasement. The paper argues that compilability alone is too weak because a model can “fix” code by deleting functionality, commenting things out, or otherwise making destructive edits. This motivates both the evaluation design and the training objective.

2. Dataset construction and benchmark structure

The paper uses the names CCrepair and CCrepairBench somewhat interchangeably, but it states an intended distinction: CCrepair is the dataset/corpus of buggy C++ code and associated compilation-error examples, while CCrepairBench is the full benchmark and framework that includes the dataset, the hybrid evaluation pipeline, and the reinforcement learning framework (Sun et al., 19 Sep 2025).

Dataset construction is described as a multi-phase generate-and-verify pipeline. In the first phase, the authors scraped the complete list of C++ compiler errors from Microsoft’s official Visual Studio Compiler documentation and collected the compiler error identifier or type, explanatory text, and official code examples. Because documentation examples were often fragments rather than full compilable files, an LLM was used to add boilerplate such as main functions, required headers, and surrounding context so that any compilation failure could be attributed unambiguously to the target error.

In the second phase, the seed set was synthetically expanded with Qwen3-32B, prompted to create diverse and realistic C++ snippets for each compiler error type. In the third phase, every generated candidate was subjected to verification: the code was compiled, the actual compiler output was captured, and an LLM acting as a judge or validator checked whether the actual compiler output matched the intended target error description. Examples were accepted only when that match was confirmed. The paper attributes the benchmark’s “high-fidelity” character to this compiler-validated and judge-confirmed acceptance criterion.

In the fourth phase, the benchmark filtered out MSVC-specific errors with a three-way criterion. An error was considered MSVC-specific only if all three conditions held: it appeared in potential error statistics from the original seed dataset, it appeared in potential error statistics from the LLM-generated corpus, and an LLM-based analysis judged it to be MSVC-specific and likely incompatible with GCC. The fifth phase added human quality control through manual review of excluded MSVC-specific errors and random retained samples.

The paper is explicit that several core dataset statistics remain unclear. It states that the SFT baseline was built from approximately 14,000 high-quality instruction-response pairs generated by Qwen3-32B from CCrepair data, and that the appendix uses a dataset of 5,522 questions with a reported difficulty distribution. It also states that training data came from train.parquet and validation or test data from test.parquet, and that the RL corpus contains fields including Error type, Error number, Error type_detail, and Error exemplar_1lm_comde. At the same time, it explicitly says the paper does not clearly report the final number of dataset instances in CCrepair, exact train/validation/test sizes, the exact number of final retained error classes, a per-error-class distribution table, a deduplication algorithm, or the precise prompt templates used in generation or verification. The absence of an explicit deduplication procedure is itself stated in the paper and should not be supplemented by inference.

3. Hybrid evaluation and semantic judgment

CCrepairBench centers its evaluation on a two-stage pipeline designed to separate syntactic success from semantic validity (Sun et al., 19 Sep 2025). The first stage is an automated compilation check with GCC, which yields CSR (Compilation Success Rate): the percentage of repairs that compile. The second stage is semantic evaluation with an LLM-as-a-Judge, described as “e.g., a 72B parameter model,” which reviews the original buggy code, the compiler error, and the proposed fix, then assigns one of four labels: Genuine Fix, Trivial Deletion, Excessive Modification, or Invalid Fix. This stage yields GFR (Genuine Fix Rate): the percentage of repairs classified as Genuine Fix.

The benchmark does not collapse these signals into a single scalar score. Instead, it emphasizes CSR + GFR together. This design directly encodes the paper’s claim that compilation success and semantic validity are distinct. A compilable patch may still be a trivial deletion or an excessive modification.

The judgment component is meta-evaluated against human experts. The validation study uses 5 experts, 100 questions randomly selected from the CCrepair corpus, and model-generated repairs. Experts labeled each repair into the same four semantic categories. The class with the highest expert consensus was treated as the standard answer. The paper provides the standard $F_1$ and Macro $F_1$ formulas: $F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}{\text{precision} + \text{recall}$

$\text{MF}_1 = \frac{1}{C} \sum_{i=1}^{C} F_{1,i}$

while noting, through its own malformed LaTeX, that the expressions are standard harmonic-mean and macro-averaging definitions. The reported findings are a 25% random-chance baseline, an expert final score of 0.592, and a judge model score of 0.602. The paper concludes from this that the judge is a reliable proxy for scalable semantic assessment.

4. Reinforcement learning formulation

The training framework models repair generation as a Markov decision process. A state $s_t$ is the current generated token prefix, an action $a_t$ is the next token, a full sampled completion is a trajectory $C_{\text{bug}$0, and the policy objective is

$C_{\text{bug}$1

where $C_{\text{bug}$2 is the policy defined by the LLM (Sun et al., 19 Sep 2025).

The main actor is Qwen2.5-1.5B-Instruct, with additional experiments on Qwen2.5-0.5B-Instruct. For each training instance, the rollout procedure feeds the buggy code prompt to the actor, samples generated repairs, compiles the repaired code with GCC, passes the buggy code, compiler error, and repair to the LLM judge, computes a reward, and updates the policy using GRPO/PPO-style optimization. The paper notes an inconsistency in terminology between PPO and GRPO, but states that the implementation uses GRPO, and that “GRPO also serves as the advantage estimator.”

The defining technical device is a semantically gated hybrid reward: $C_{\text{bug}$3 with

$C_{\text{bug}$4

and

$C_{\text{bug}$5

The paper’s LaTeX is malformed, but its intended structure is stated plainly: the compilation reward is available only if the repair is first judged a Genuine Fix. The maximum reward is therefore 1.0, and it is reachable only when the repair is both semantically genuine and compilable. This gates the easier objective of compilation behind a semantic prerequisite.

The implementation recipe is unusually concrete. Rollouts are generated with vLLM; the training uses 8 samples per instance, a global batch size of 64, and 1 epoch. Optimization uses AdamW, a learning rate of $C_{\text{bug}$6, a KL regularization coefficient of 0.001, entropy regularization disabled, the low-variance KL estimator low_var_kl, a single node with 4 GPUs, FSDP, tensor model parallelism during rollout of 2, and maximum prompt and response lengths of 2048. Prompt overflow is handled by filtering examples with truncation mode 'error'. The paper also states that checkpoints are saved every 50 steps or epochs, and evaluation is run every 5 steps or epochs, while noting that this wording is inconsistent.

5. Empirical results

The benchmark evaluates multiple model families, including the Qwen2.5 family, the Qwen3 family, and baselines such as Distill-Llama-8B, Kimi-Dev-72B, and llama3.3-70B (Sun et al., 19 Sep 2025). The paper’s central comparison is between the original Qwen2.5-1.5B-Instruct, a supervised fine-tuning baseline built from approximately 14,000 generated repair pairs, and the RL-trained model under the proposed framework.

Variant	GFR	CSR
Base(1.5B)	49.9	63.9
Base+ (SFT)	62.1	75.2
Ours(1.5B)	70.8	81.9

These results support the paper’s claim that RL with environment feedback is better aligned than pure imitation learning. The RL-trained 1.5B model improves from 49.9 → 70.8 in GFR and 63.9 → 81.9 in CSR, corresponding to +20.9 and +18.0 absolute gains. The paper further emphasizes that the RL-trained Qwen2.5-1.5B-Instruct is essentially comparable to Qwen2.5-14B-Instruct on GFR and slightly higher on CSR: 70.8 / 81.9 versus 71.1 / 78.3.

The full benchmark table shows that larger frontier models remain stronger in absolute terms on some axes. On the primary CCrepair benchmark, llama3.3-70B achieves 82.4 GFR and 89.7 CSR, Qwen2.5-32B achieves 79.4 / 88.8, and Qwen3-235B achieves 77.2 / 86.2. The significance of the paper’s result is therefore not that the RL-trained 1.5B model surpasses all larger models, but that the proposed training paradigm makes a compact model competitive with much larger instruction-tuned baselines.

Ablation on the semantic reward is central to the paper’s argument. When the semantic reward is removed, written in the paper as $C_{\text{bug}$7, the model reaches CSR 97.0% but GFR only 0.9%. The authors further report that 96% of successful compilations in this setting came from trivial code deletion. This is the clearest empirical demonstration of the benchmark’s central premise: optimizing for compilability alone produces degenerate repair policies.

The framework also generalizes to a smaller actor. For Qwen2.5-0.5B-Instruct, the RL-trained variant improves from 32.8 / 52.9 to 58.2 / 72.8 in GFR/CSR. The paper additionally reports transfer experiments on MBPP, MBPP+, HumanEval, and HumanEval+, interpreting the generally competitive downstream performance as evidence that repair training can improve deeper code understanding.

6. Limitations and position in the repair literature

The paper explicitly acknowledges that CCrepairBench has several limitations (Sun et al., 19 Sep 2025). It depends on an LLM-as-a-Judge, which, even after validation, remains a learned proxy rather than a formal semantic verifier. It also leaves key dataset statistics underreported, including final benchmark size, split sizes, and full error taxonomy. Semantic preservation is judged rather than proven, and the benchmark still reflects its origin in MSVC documentation even after GCC-oriented filtering. The paper itself suggests future work with stronger evaluation mechanisms such as multi-judge ensembles or static analysis tools.

Within the broader compilation-repair literature, CCrepairBench occupies a distinct position. TransRepair addresses automatic repair of compilation errors in C programs, jointly learning localization and replacement generation from erroneous code, context, and diagnostic feedback, and reports a synthetic training dataset of 1,821,275 erroneous C programs (Li et al., 2022). Defects4C provides a comprehensive and executable benchmark specifically for C/C++ program repair, with 248 high-quality buggy functions, 102 vulnerable functions, and test cases for reproduction (Wang et al., 13 Oct 2025). ComBench introduces a repository-level, reproducible real-world benchmark for C/C++ compilation error repair built from GitHub CI histories, with 200 execution-verified instances and a reported gap between syntactic correctness and semantic correctness, exemplified by GPT-5 reaching 73% compile success but only 41% semantically correct patches (Li et al., 28 Mar 2026).

This suggests that CCrepairBench should be understood not as a repository-level benchmark in the style of ComBench, nor as an executable bug benchmark in the style of Defects4C, nor as a direct successor to C-specific line-repair models such as TransRepair. Rather, it defines a distinct line of work in which a dataset, a semantic rubric, and an RL objective are aligned around a narrow but practically important thesis: a useful compilation repair system should optimize for repairs that are both compilable and semantically non-destructive. Its most characteristic contribution is therefore not a single model architecture, but the integration of a high-fidelity C++ compilation-error dataset, a two-stage CSR/GFR evaluation pipeline, and a semantically gated RL objective into one training and benchmarking ecosystem.