Defects4C: C/C++ Repair Benchmark

• Defects4C is a benchmark for automated program repair in C/C++ featuring confirmed bug functions, paired tests, and executable evaluation environments.
• It supports both performance evaluation and retraining of repair techniques using a Docker-based reproduction setup and CI integration.
• Empirical studies with LLMs reveal a significant performance gap compared to Java benchmarks, emphasizing challenges in handling memory safety and logic errors.

Searching arXiv for the specified paper and closely related benchmark/APR work to ground the article. Defects4C is a benchmark for automated program repair (APR) in C/C++ built from real-world repositories and designed to be executable, reproducible, and suitable for both evaluation and retraining of learning-based repair systems. It was introduced to address the asymmetry between relatively mature Java-oriented APR benchmarking and the lack of a comparably rigorous C/C++ benchmark, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. The benchmark contains bug-relevant commits at repository scale, a curated set of human-confirmed faulty functions, vulnerable functions, and paired test cases for reproduction, and it is accompanied by an empirical study of 24 LLMs for C/C++ repair (Wang et al., 13 Oct 2025).

1. Origin, scope, and intended role

Defects4C was introduced in the context of a persistent gap in APR research: substantial progress had been made in Java-based APR, largely facilitated by benchmarks like Defects4J, whereas C/C++ repair lacked a high-quality, open-source benchmark tailored to the language family (Wang et al., 13 Oct 2025). The benchmark is therefore positioned as both a dataset and an execution framework.

Its scope is explicitly twofold. First, it supports rigorous evaluation of repair techniques through paired buggy and patched program states with test cases. Second, it supports retraining of learning-based approaches through a much larger upstream collection of bug-relevant commits. The benchmark distinguishes between general bugs and vulnerabilities, with the latter collected from CVE-linked patches.

A central design choice is the focus on single-function commits for the confirmed benchmark instances. This does not imply that C/C++ defects are intrinsically single-function phenomena; rather, it reflects a tractability criterion for executable benchmarking. A plausible implication is that Defects4C prioritizes controlled evaluation over full ecological coverage of multi-function and cross-file failures.

2. Dataset construction and curation pipeline

The construction process begins with GitHub’s public archives and applies explicit repository-level selection criteria: non-fork, redistributable license, at least 200 stars, and activity between January 2015 and December 2023. From 110,441 candidates ranked by star count, the top 500 projects were selected (Wang et al., 13 Oct 2025).

Commit harvesting proceeds along two channels. Bug-relevant commits were filtered by keyword heuristics in commit messages, including terms such as fix, bug, error, and vulnerability, yielding 38 million potentially bug-related commits. Vulnerability commits were extracted from the official CVEProject list, retaining only CVEs with a single patched commit ID, which produced 14,488 commits.

The raw commit pool then underwent repository-level filtering, diff-duplication removal, and language filtering, leaving 9 million valid bug-relevant commits. From these, only commits whose patch affected exactly one function and for which at least one test suite existed were retained, resulting in 76K commits, including 249 vulnerability commits. Test matching was then performed by compiling and executing each test case $t_i$ on the pre-commit and post-commit versions, retaining only tests satisfying

$\text{Fail}(V_0,t_i)\;▸\;\text{Pass}(V_1,t_i)\,.$

This stage yielded 3,785 general-bug commits and 102 vulnerability commits. Human confirmation followed in three annotation rounds by security experts, with Cohen’s $\kappa$ reaching 0.88. The final high-confidence benchmark contains 248 real-bug functions and 102 vulnerable functions, each paired with at least one bug-revealing unit test (Wang et al., 13 Oct 2025).

Component	Raw commits	Confirmed
Bug-relevant commits	38 M	248
CVE-related commits	14.5 K	102
Total confirmed bugs	–	350

This pipeline is notable for separating large-scale weakly filtered data from the final human-confirmed benchmark. That separation matters methodologically because it allows both benchmark evaluation and downstream retraining without conflating noisy mining artifacts with validated repair targets.

3. Fault taxonomy and benchmark organization

The confirmed faults are categorized by error type and subtype. The benchmark spans signature errors, sanitizer-related faults, memory errors, and logic organization defects, with distinct distributions across general bugs and vulnerabilities (Wang et al., 13 Oct 2025).

Category	Error sub-type	Bugs / Vulns
Signature	Incorrect Function Usage	19 / 3
Signature	Faulty Input Type	12 / 2
Signature	Incorrect Return Value	19 / 3
Signature	Variable Misuse	25 / 3
Sanitizer	Control Expression Error	66 / 6
Memory Error	Null Pointer Deref.	6 / 6
Memory Error	Resource Consumption	9 / 5
Memory Error	Memory Overflow	5 / 61
Logic Organization	Improper Condition	67 / 11
Logic Organization	Wrong Call Sequence	20 / 2

Two aspects of this distribution are especially important. First, general bugs are concentrated in sanitizer and logic-organization subtypes, especially Control Expression Error and Improper Condition. Second, vulnerabilities are dominated by Memory Overflow, with 61 confirmed vulnerable functions in that subtype. This asymmetry indicates that the vulnerability subset is not merely a relabeling of ordinary bugs; it has a distinctly memory-safety-heavy profile.

Each benchmark instance is identified by a fault ID of the form \<proj\>-\<bugID\> and is laid out as a self-contained directory:

• src/: project sources at pre-commit
• patched/: project sources at post-commit
• tests/: minimal test harness
• metadata.json: patch location, fault type, test commands

This organization makes the benchmark executable at the granularity of individual faults while preserving both buggy and fixed program states.

4. Reproduction environment and execution model

Defects4C provides a Docker-based reproduction environment based on Ubuntu 22.04 with GCC 12 and Clang 15 toolchains installed, together with make, cmake, autotools, libgtest-dev, and common C/C++ libraries (Wang et al., 13 Oct 2025). Dependency management is localized: each fault directory contains a small build script, either a Makefile or CMakeLists.txt, that pulls in only the files under src/ and tests/.

The benchmark also includes containerization and CI support. The root-level Dockerfile copies the benchmark into /bench and sets WORKDIR /bench, while the GitHub Actions verification script builds the Docker image and then iterates over fault directories with make test. The paper states that all patch applications and tests are performed inside isolated Docker containers via a CLI endpoint, /fix_with_patch.

This emphasis on containerized execution has a methodological consequence. It reduces environmental nondeterminism and makes APR evaluation less sensitive to local toolchain drift, missing dependencies, or repository evolution. A plausible implication is that Defects4C is designed not merely as a static corpus but as an executable benchmark infrastructure.

5. Evaluation protocol and repair metrics

Defects4C distinguishes sharply between plausible and correct patches. A candidate patch is plausible if it compiles and passes the entire provided test suite. A patch is correct if it also addresses the underlying bug semantically, with the confirmed patches in Defects4C serving as ground truth (Wang et al., 13 Oct 2025). This distinction is central to APR evaluation, particularly in settings where test suites may be incomplete.

The benchmark reports several metrics. It uses Accuracy (Pass@k), unbiased pass@k following EvalPlus, and Precision / Recall for multi-patch settings. The unbiased pass@k estimator is given as

$$\mathrm{pass}@k \;=\; 1 - \prod_{i=0}^{k-1}\tfrac{N-1-i}{N}\,,$$

where $N$ is the total number of unique fix attempts generated and $s$ is the number of successful repairs within $k$ tries.

Two repair protocols are defined. In single-round repair, one prompt is supplied, the model returns one patch, and the patch is compiled and tested. In conversation-based repair, following Xia et al., the model is allowed up to $M$ attempts and up to $L$ turns per attempt; the reported configuration uses $M=10$ and $\text{Fail}(V_0,t_i)\;▸\;\text{Pass}(V_1,t_i)\,.$0, for a total budget of 30 tries. On each failure, the compiler error is appended to the next prompt. Each compile/test run is bounded at 30 seconds, and each LLM call is bounded at 60 seconds.

One common misconception in APR benchmarking is that passing tests is equivalent to semantic correctness. Defects4C explicitly rejects that equivalence by separating plausibility from correctness and by grounding correctness in confirmed fixes.

6. Empirical results on 24 LLMs

The empirical study evaluates 24 pre-trained LLMs, including GPT-4, GPT-3.5-Turbo, several CodeLlama variants, Deepseek-Coder variants, WizardCoder variants, Gemma, phi-2, Magicoder-S-DS, Mixtral-8x7B, and Phind-CodeLlama (Wang et al., 13 Oct 2025). Prompting varies by fix granularity: single-line or single-hunk repair uses a masked buggy region marked as >>>[INFILL]<<<, whereas single-function repair provides the full function and asks the model to fix it so that the tests pass. Hyperparameters include temperature settings of 0.2 and 0.8 for single-round repair and 1.0 for conversations, together with greedy decoding versus sampling.

For single-round repair on the 248 general bugs, the paper reports the following selected results:

Model	k*=1 (greedy)	pass@10	pass@100 (T=0.8)
GPT-4	9.0%	–	–
GPT-3.5-Turbo	8.5%	20.0%	38.9%
CodeLlama-Instruct-7B	4.8%	20.5%	45.7%
WizardCoder-Python-34B	5.5%	23.0%	45.1%
Phind-CodeLlama-34B	6.1%	20.6%	38.4%

In the conversation-based setting, the best-performing model, GPT-3.5-Turbo at $\text{Fail}(V_0,t_i)\;▸\;\text{Pass}(V_1,t_i)\,.$1, repairs 27 of 248 general bugs, or 10.9%, and only 1 of 102 vulnerabilities, or 1.0%, within 30 tries.

Fault-type analysis further differentiates model behavior. Signature and Sanitizer errors are reported as moderately easier; for example, GPT-3.5 fixes 13 of 66 sanitizer defects. Logic-organization fixes, especially multi-branch conditions, are challenging, with LLMs typically below 5% success. Vulnerabilities, particularly buffer-overflow-related cases, remain markedly difficult, with repair rates below 7%. Memory-overflow vulnerabilities are identified as the hardest group.

These results support the paper’s larger comparison: state-of-the-art LLMs that achieve greater than 70% pass@1 on Java benchmarks such as Defects4J fall below 10% on C/C++ in Defects4C. This suggests that current LLM-APR systems do not transfer smoothly from managed-language repair to low-level systems-language repair.

7. Research significance, limitations, and projected directions

Defects4C is described as the first large-scale, reproducible C/C++ repair benchmark with 350 human-confirmed faults and paired tests (Wang et al., 13 Oct 2025). Its significance lies not only in providing confirmed bugs and vulnerabilities, but also in exposing a measurable performance gap for LLM-based repair in C/C++.

The paper reports that fine-tuning on 20K function-level commits yields modest gains, with an average +84.9% relative improvement in pass@k, while absolute performance remains below industrial needs. This combination of relative improvement and low absolute success is important: it indicates that data adaptation helps, but does not remove the underlying difficulty of C/C++ repair.

The recommendations presented are correspondingly hybrid and systems-oriented. They include data augmentation through negative tests or input mutations, hybrid pipelines combining LLM generation with static analysis such as Clang Tidy, context expansion through global declarations and cross-file snippets, and specialized fine-tuning via LoRA adapters targeted at memory-safety patterns such as buffer bounds and null checks.

The open challenges identified in the paper are also structurally informative. They include extending beyond single-function commits to multi-function and cross-file bugs, incorporating dynamic symbolic execution feedback to expose low-level memory states, enriching the benchmark with negative patches to evaluate patch correctness more robustly, and investigating continual fine-tuning methods that adapt to per-project coding styles and API usages.

Taken together, these directions indicate that Defects4C is not only a benchmark of model capability but also a diagnostic instrument for the current limits of LLM-APR in C/C++. Its CLI/HTTP interface, Docker reproduction environment, and experimental artifacts are publicly available at https://sites.google.com/view/defects4c (Wang et al., 13 Oct 2025).