QuixBugs Benchmark: Multi-Language APR Suite
- QuixBugs Benchmark is a multi-language repository of compact algorithm implementations with localized defects, designed to assess automated program repair and related debugging techniques.
- It supports diverse evaluation scenarios—from LLM-based repair and differential testing to interactive debugging—by providing paired buggy and correct implementations.
- Empirical results highlight strong performance improvements in APR with modern LLMs, while also revealing challenges such as overfitting, oracle weaknesses, and data leakage.
QuixBugs is a multi-language benchmark for automated program repair (APR) built around buggy implementations of classic algorithms, their corresponding correct versions, and associated tests. Introduced from the Quixey Challenge, it was intended to diversify APR evaluation beyond repeatedly used corpora such as Defects4J, ManyBugs, and IntroClass by focusing on small, algorithmically rich programs rather than large, project-integrated defects (Ye et al., 2018). In subsequent work, QuixBugs has become a standard testbed not only for APR, but also for LLM-based repair, differential testing, unit-test generation, interactive debugging, and derived logical-error benchmarks (Etemadi et al., 2024, Li et al., 2023, Xu et al., 2024).
1. Origin, scope, and benchmark identity
QuixBugs is commonly described as a benchmark of 40 buggy programs implemented in both Python and Java, with each subject representing a classic algorithmic task such as sorting, graph traversal, dynamic programming, recursion, number theory, or string processing (Ye et al., 2018, Hu et al., 2024). Multiple studies characterize the benchmark as originating from the Quixey Challenge, where participants were given short programs containing a single one-line bug and one minute to fix it (Kusama et al., 22 Aug 2025). That provenance explains the benchmark’s emphasis on compact functions, localized defects, and clearly defined functional intent.
The benchmark’s central design objective is diversity. The original large empirical study on QuixBugs explicitly motivates it as a counterweight to the external-validity threat created when APR research repeatedly evaluates on the same datasets (Ye et al., 2018). Because QuixBugs is multi-language and algorithmic, it supports cross-language comparisons and isolates function-level repair from build-system, dependency, and project-integration issues that dominate larger corpora (Kusama et al., 22 Aug 2025).
A notable reporting discrepancy exists in the later literature. Canonical APR studies and most LLM repair evaluations use 40 programs per language (Ye et al., 2018, Yin et al., 2024, Kusama et al., 22 Aug 2025), whereas the paper introducing QuixBugs-LE states that QuixBugs contains 50 Python and 40 Java programs and uses the correct implementations as the seed corpus for logical-error injection (Xu et al., 2024). This suggests that “QuixBugs” can refer either to the canonical 40-program APR suite or to a broader source corpus used in derivative datasets, and papers therefore need to specify the exact subset and language configuration they employ.
2. Program characteristics and oracle structure
QuixBugs programs are small but non-trivial. In the comprehensive Java study, program length ranges from 9 to 67 LOC, 14 programs involve recursion, and 13 include nested loops (Ye et al., 2018). Although the benchmark is compact, its behavior is algorithmically varied: frequent bug types include missing function calls, incorrect comparison operators, and missing lines of code, while observed failure symptoms include incorrect outputs in 26 programs, stack overflows in 5, array index errors in 5, timeouts or infinite loops in 3, null pointer exceptions in 2, and concurrent modification in 1 (Ye et al., 2018).
The test suites are a distinctive part of the benchmark. One study emphasizes that QuixBugs has unusual oracle characteristics relative to other APR corpora: 15 of 40 subjects initially have only failing tests and no passing tests, and three timeout-related subjects use tests with a 60-second limit (Ye et al., 2018). Another evaluation describes the bundled tests as lightweight, deterministic, and input–output based, with no flakiness or environment-dependent behavior, and states that no programs were filtered, no tests were removed, and no timeouts were modified (Kusama et al., 22 Aug 2025). These two descriptions are consistent in portraying QuixBugs as controlled and execution-oriented, but also as vulnerable to incomplete functional specification.
The Java side of the benchmark has itself been curated for APR use. The 2018 study prepared a “usable” Java version by fixing compilation issues, correcting incorrect test data, generating or curating 259 JUnit test methods across the 40 programs, and establishing ground-truth Java patches ported from Python, with build automation integrated through Travis and Gradle (Ye et al., 2018). Later studies continue to use language-specific structures such as QuixBugs-Java and QuixBugs-Python, and one LLM repair paper reports the dataset statistics in terms of single-function, single-hunk, and single-line defects: QuixBugs-Java has 40 total bugs, all 40 single-function, 37 single-hunk, and 37 single-line; QuixBugs-Python has 40 total bugs, and all 40 are single-function, single-hunk, and single-line (Yin et al., 2024).
These properties make QuixBugs unusually well suited to function-level repair research. The benchmark is large enough to expose variation across algorithms and bug types, yet small enough that the surrounding engineering context does not dominate the experiment.
3. Evaluation methodology and correctness criteria
QuixBugs has been evaluated under several distinct methodological regimes, but most studies share a common execution-based core: a patch or generated function is validated against the provided tests, and success is typically defined by passing the full test suite for that program (Yin et al., 2024, Kusama et al., 22 Aug 2025, Hu et al., 2024). In the 2018 APR study, a plausible patch or test-suite adequate patch is a patch that makes all provided human-written JUnit tests pass, a correct patch is one that is identical or semantically equivalent to the human fix, and an overfitting patch is one that passes the available tests but does not truly fix the bug (Ye et al., 2018).
For QuixBugs, repairability is often reported as a count or rate over the 40 programs. One standard formulation is
LLM papers use the same general pattern with different notation, for example
or, in multi-sample settings,
These formulations appear in QuixBugs evaluations of ThinkRepair and quantized small LLMs, respectively (Yin et al., 2024, Kusama et al., 22 Aug 2025).
The more difficult issue is semantic correctness beyond the provided tests. The 2018 study evaluates 338 unique plausible patches produced by ten Java APR tools and finds that 53.3% are overfitting according to manual assessment (Ye et al., 2018). To analyze this, it applies three automated patch-correctness assessment techniques—RGT_Evosuite, RGT_InputSampling, and GT_Invariants—with reported overfitting-detection accuracies of 98.2%, 80.8%, and 58.3%, respectively (Ye et al., 2018). Later LLM-based studies frequently retain manual semantic validation as part of the protocol. ThinkRepair, for example, defines a patch as “fixed” when it passes the full QuixBugs suite, but also manually checks plausible patches for semantic equivalence (Yin et al., 2024). ContrastRepair likewise distinguishes plausible from correct patches by requiring manual inspection after test passing (Kong et al., 2024).
Evaluation workflows vary substantially across papers. Codex is evaluated with a single completion per bug and prompt configuration (Prenner et al., 2021). DeepDebug uses up to 100 sampled candidate patches per Python bug within a one-minute budget (Drain et al., 2021). ThinkRepair allows 25 independent sessions per bug, up to 5 interactions per session, and a 5-hour end-to-end timeout, although it reports that the average total time is lower than 20 minutes per bug (Yin et al., 2024). A later study on small LLMs treats each task as complete-function generation, samples 200 candidate patches per bug with nucleus sampling, and counts a bug as fixed if any candidate passes all tests (Kusama et al., 22 Aug 2025). ROSE, by contrast, evaluates QuixBugs without relying on the benchmark’s test suite as the online oracle, instead using debugger stopping points and comparative execution during interactive repair (Reiss et al., 2022).
This methodological heterogeneity is central to interpreting QuixBugs results. A plausible implication is that raw “bugs fixed” counts are highly informative within a paper’s protocol, but are not automatically commensurate across papers that differ in language, fault-localization assumptions, number of attempts, interaction loops, and semantic-validation policy.
4. Reported empirical performance on QuixBugs
The benchmark has served as a stable reference point across several generations of APR systems, from search-based and synthesis-based tools to LLM-centric and interaction-heavy workflows. Representative results are summarized below.
| Study | Setting | Reported QuixBugs result |
|---|---|---|
| Comprehensive APR study (Ye et al., 2018) | 10 Java APR tools | 16/40 with at least one plausible patch; 7/40 correct repairs |
| DeepDebug (Drain et al., 2021) | Python, no line-level hints, 1-minute budget | 21/40 fixed; 1 false positive |
| Codex evaluation (Prenner et al., 2021) | Single completion per bug | Python 23/40 best prompt; Java 14/40 best prompt |
| ChatGPT analysis (Sobania et al., 2023) | Python, four baseline runs; one follow-up hint | 19/40 baseline; 31/40 with one hint |
| ThinkRepair (Yin et al., 2024) | ChatGPT backend, two manual few-shot examples | Java 39/40; Python 40/40 |
| ContrastRepair (Kong et al., 2024) | ChatGPT with contrastive test pairs | 40/40 Python; 40/40 Java |
| GPT-family evaluation (Hu et al., 2024) | Python, initial plus secondary error-feedback prompt | O1-preview 40/40; O1-mini 40/40; GPT-4o 38/40 |
| Quantized SLM study (Kusama et al., 22 Aug 2025) | 200 candidates per bug | Codex 39/40 Python, 30/40 Java; top SLMs 38/40 Python and 29–30/40 Java |
The earliest large study establishes the benchmark’s baseline difficulty. Among ten Java APR tools run on all 40 programs, only 16 subjects receive at least one plausible patch, and only 7 subjects have correct repairs after manual assessment (Ye et al., 2018). DeepDebug improves the Python side to 21/40 under a one-minute-per-bug constraint and lowers the false positive rate to 5% (Drain et al., 2021). Codex later reaches 23/40 on Python with code-and-docstring prompts, 21/40 on Python without docstrings, and 14/40 on Java (Prenner et al., 2021).
LLM dialogue and feedback then materially change the landscape. ChatGPT fixes 19/40 Python bugs in the basic setting and 31/40 with a single conversational hint derived from a failing test case (Sobania et al., 2023). ThinkRepair reports 39/40 on QuixBugs-Java and 40/40 on QuixBugs-Python, with identical counts under perfect fault information and method-level fault information (Yin et al., 2024). ContrastRepair reports 40/40 correct fixes on both Python and Java at the aggregate per-language level and attributes the gain to contrastive failing–passing test pairs (Kong et al., 2024). GPT-o1 models also reach 40/40 on the Python suite after a second prompt containing raw failure output, whereas GPT-4o reaches 38/40 (Hu et al., 2024).
A further development is that small models no longer obviously trail larger ones on this benchmark. In the 2025 study of quantized small LLMs, Codex attains 39/40 on Python and 30/40 on Java, while Phi-3-3.8B and Qwen2.5-Coder-3B-Instruct both reach 38/40 on Python and 29/40 on Java; Code Llama-7B reaches 37/40 on Python and 30/40 on Java (Kusama et al., 22 Aug 2025). That paper explicitly reports that several modern SLMs match or exceed GPT-NeoX-20B on QuixBugs, indicating that scale alone is not determinative for this benchmark (Kusama et al., 22 Aug 2025).
Taken together, these results show a sharp historical shift. QuixBugs was initially difficult enough that correct repairs were rare and overfitting common, but the strongest contemporary LLM-based systems now approach or achieve complete coverage under their respective protocols.
5. Methodological roles beyond standard APR
QuixBugs has evolved from an APR benchmark into a broader experimental substrate for software-engineering research. One reason is that its functions are small, self-contained, and paired across languages, which permits reuse in settings that do not require project build systems or long execution traces.
In differential testing, Mokav uses QuixBugs as a source of buggy–fixed Python program pairs. It selects a 32-pair subset where an example test yields identical output on both versions, then searches for a difference exposing test (DET), formally an input such that . Mokav reports 30/32 successes after one iteration and 32/32 by iteration 4, outperforming both Pynguin and Differential Prompting, each of which reaches 16/32 (Etemadi et al., 2024). Here QuixBugs functions not as a repair benchmark but as a controlled corpus of semantically close program variants.
In interactive debugging, ROSE uses all 40 Java QuixBugs programs but deliberately avoids the full test infrastructure. Instead, it constructs minimal drivers, invokes repair from debugger stopping points, and validates candidate repairs by comparative execution. Under this protocol, ROSE suggests correct repairs for 17/40 programs, with a median total time of about 5 seconds and a median time of 0.8 seconds to show the correct repair (Reiss et al., 2022). This demonstrates that QuixBugs can also serve as a benchmark for low-latency, developer-in-the-loop repair rather than only offline test-suite-based APR.
In unit-test generation, a GPT-4-based Code Interpreter study selects 20 Python QuixBugs functions amenable to input/output-only testing and evaluates test quality by comparing outputs of the correct and incorrect implementations. Across prompts and resamples, it reports a mean of 7.78 test cases per response, a mean fraction of correct test cases of 0.79, a mean fraction of failing test cases of 0.57, and a mean fraction of test cases that are both correct and failing of 0.40 (Li et al., 2023). The benchmark’s paired correct and buggy implementations are essential here because they permit direct evaluation of whether generated tests actually expose the bug.
QuixBugs has also spawned derivative datasets. LecPrompt introduces QuixBugs-LE by injecting logical errors into correct QuixBugs programs through iterative masking and filling, producing 550 Python and 662 Java programs with an average of 2.6 logical errors per program (Xu et al., 2024). On this derived benchmark, LecPrompt reports Python top-1 token-level repair accuracy of 74.58% and full program-level repair accuracy of 27.4%, with Java scores of 69.23% and 24.7%, respectively (Xu et al., 2024). This suggests that QuixBugs now underpins not only patch-generation studies but also token-level localization and logical-error correction research.
6. Limitations, controversies, and current status
The benchmark’s most persistent technical limitation is oracle weakness. The 2018 large-scale study shows that among 338 unique plausible patches, 180 are overfitting, yielding an overfitting rate of 53.3% (Ye et al., 2018). That paper also documents a concrete failure mode in which automatically generated or sampled tests can still miss semantically wrong patches, such as a NEXT_PERMUTATION patch that only fails on repeated values (Ye et al., 2018). Later work continues to distinguish passing the QuixBugs tests from true semantic correctness, indicating that test-suite adequacy alone remains insufficient.
A second limitation is benchmark scope. QuixBugs is intentionally small and algorithmic, and several papers treat this as both strength and weakness. The GPT-family evaluation argues that with O1 models reaching 40/40 after secondary error feedback and GPT-4o reaching 38/40, QuixBugs is increasingly becoming a “solved” benchmark for state-of-the-art LLMs (Hu et al., 2024). The small-language-model study makes a related point differently: QuixBugs remains useful for rapid, controlled evaluation of function-level repair and efficiency trade-offs, but its single-line bugs and short deterministic tests limit generalizability (Kusama et al., 22 Aug 2025). These statements are compatible: QuixBugs remains experimentally useful, but its discriminative power for frontier models is narrowing.
A third and more recent concern is data leakage. LessLeak-Bench reports that QuixBugs-Python has 40 samples, 84 automatically identified candidate duplicate pairs, 84 manually confirmed duplicate pairs, and a leaked count of 40, producing a leakage ratio
The paper attributes this to direct inclusion of a QuixBugs repository, specifically “PatrickShaw/QuixBugs,” in StarCoder’s pre-training corpus (Zhou et al., 10 Feb 2025). For The-Stack-derived models, the study therefore recommends not using QuixBugs-Python for fair evaluation. This does not invalidate all historical QuixBugs results, but it does materially change how the benchmark should be interpreted in the LLM era.
Despite these limitations, QuixBugs retains a clear methodological niche. It offers multilingual, function-level, deterministic evaluation with rich historical baselines; it is easy to repurpose for repair, testing, and localization tasks; and it exposes interaction effects among prompts, feedback, sampling, and model specialization with unusual clarity (Yin et al., 2024, Kusama et al., 22 Aug 2025). A plausible implication is that QuixBugs is now best viewed less as a standalone final benchmark and more as a controlled diagnostic instrument: valuable for rapid, comparative, and mechanistic studies, but increasingly inadequate as the sole basis for claims about real-world program-repair capability.
