BugsInPy: Python Bug Benchmark

• BugsInPy is a curated benchmark and framework for Python software testing and debugging, modeled after Defects4J for real-world bug analysis.
• It comprises 493–501 reproducible, isolated bugs across 17 projects, supporting research in fault localization, type-error detection, and program repair.
• The framework abstracts project-specific details and uses rigorous curation, providing actionable insights for testing, debugging, and automated repair studies.

Searching arXiv for recent and foundational papers on BugsInPy and closely related benchmark usage. BugsInPy is a curated benchmark and supporting framework for Python software testing and debugging, designed to play for Python the same role that Defects4J played for Java. It is presented as both a database of real bugs in Python projects and an extensible framework for accessing buggy and fixed revisions, running the tests that expose bugs, and integrating testing and debugging tools. The original database paper reports 493 real, reproducible, isolated bugs from 17 real-world Python programs; later studies frequently treat BugsInPy as a 501-bug, 17-project corpus or as filtered subsets of that corpus, depending on reproducibility and task-specific preprocessing (Widyasari et al., 2024).

1. Origin, purpose, and relation to other bug benchmarks

BugsInPy was created in response to a specific imbalance in the software engineering research ecosystem: Python had become one of the most popular programming languages, yet testing and debugging research for Python lacked a Defects4J-like benchmark. The benchmark is explicitly motivated by the view that a high-quality curated dataset lowers the barrier to entry for reproducible experimentation, enables fair comparisons across tools, and helps catalyze progress in testing, debugging, fault localization, automatic program repair, test generation, mutation analysis, and code coverage measurement (Widyasari et al., 2024).

The benchmark is explicitly inspired by Defects4J and adopts the same core design principles: real-world bugs, reproducibility via failing tests that pass after the fix, isolated patches, and easy retrieval of buggy versions and corresponding tests. The original paper positions BugsInPy as a Python analogue of Defects4J and contrasts it with collections such as SIR, Siemens, Bugs.Jar, Bugswarm, Defexts, BugsJS, and QuixBugs. In that framing, BugsInPy is intended as a curated benchmark of authentic historical defects rather than a synthetic or weakly filtered corpus (Widyasari et al., 2024).

Subsequent work treats BugsInPy as a canonical benchmark for Python research in much the same way. It is repeatedly used as the main Python benchmark for automated program repair, fault localization, type-error detection, unit-test generation, and local-LLM bug detection, and it is often described as a database of existing bugs in Python programs that enables controlled testing and debugging studies (Parasaram et al., 2024).

2. Curation methodology and inclusion criteria

The benchmark construction process is deliberately conservative. A candidate bug must satisfy three requirements: the bug is in source code, the bug is reproducible, and the bug is isolated. Accordingly, the creators exclude commits that only change configuration files, build scripts, documentation, or test cases. Reproducibility requires that at least one test from the fixed version fail on the faulty version and pass on the fixed version. Isolation requires that the faulty and fixed versions differ only by the bug fix rather than by unrelated refactoring or feature additions (Widyasari et al., 2024).

The mining workflow begins from version-control history, with particular attention to commits that modify or add test files, since those are plausible bug-fixing commits with reproducible failing tests. Test files are heuristically identified by names containing test and imports of testing libraries such as unittest or pytest. For each candidate commit, the authors manually inspect the commit message, the source code, and linked issue information if present. The paper states that at least two authors independently inspect the commits, and only commits with agreement are retained (Widyasari et al., 2024).

The paper also reports intermediate filtering counts. It initially identified 796 commits; 66 commits were omitted because the authors did not agree that they qualified as bug-fixing commits. After that step, 730 commits remained. A further 235 commits were omitted because the authors did not agree that they were isolated. The maintainers explicitly chose not to “clean” tangled commits by manually removing unrelated changes, because they wanted the buggy and fixed versions to remain authentic versions from real project histories (Widyasari et al., 2024).

This curation strategy gives BugsInPy a specific methodological profile: it is not merely a patch collection, but a benchmark in which each retained defect is tied to an executable exposure condition. In later empirical work, this property is formalized by representing a bug as $b=\langle p_b^-, p_b^+, F_b, P_b\rangle$, where $p_b^-$ is the faulty version, $p_b^+$ is the fixed version, $F_b$ are tests that fail on the faulty version and pass on the fixed version, and $P_b$ are tests that pass on both versions (Rezaalipour et al., 2023).

3. Benchmark contents, project composition, and execution framework

For each bug, BugsInPy provides three primary artifacts: a revision mapping from bug identifier to Git revision in the original repository, the patch of the isolated bug, and the tests that expose the bug. This is the core substrate that makes the benchmark useful for repair, localization, and regression-oriented studies. The benchmark also includes a database abstraction layer that hides repository-specific checkout details and build/test idiosyncrasies, so users can request a faulty or fixed version without manually reconstructing the original repository state (Widyasari et al., 2024).

The initial release comprises the following 17 projects and bug counts (Widyasari et al., 2024):

Project	Bugs
ansible/ansible	18
cookiecutter/cookiecutter	4
cool-RR/PySnooper	3
explosion/spaCy	10
huge-success/sanic	5
jakubroztocil/httpie	5
keras-team/keras	45
matplotlib/matplotlib	30
nvbn/thefuck	32
pandas-dev/pandas	169
psf/black	15
scrapy/scrapy	40
spotify/luigi	33
tiangolo/fastapi	16
tornadoweb/tornado	16
tqdm/tqdm	9
ytdl-org/youtube-dl	43

In aggregate, the paper reports 1,253.5 KLoC of code, 486 KLoC of test code, 112,602 tests, and 470.2K stars across the included projects. The benchmark is therefore explicitly not a toy collection; it spans data science, machine learning, web frameworks, developer tools, and command-line utilities, with substantial variation in project size and testing scale (Widyasari et al., 2024).

The execution framework abstracts over project-specific setup. The maintainers manually studied project documentation, README files, websites, and sometimes source code to record the commands needed to install dependencies, run setup.py when needed, build the project, and execute the test suite. The initial release abstracts over the two test frameworks used by the projects, namely unittest and pytest. On top of this abstraction layer, the framework supports test set selection, test input generation via PythonFuzz, mutation analysis using MutPy, and code coverage computation using coverage.py (Widyasari et al., 2024).

4. Role as an empirical benchmark in Python software engineering research

BugsInPy has become a central benchmark for research on Python bug repair and diagnosis. In LLM-based automated program repair, it is used to study prompt construction, fact selection, and context injection at scale. One study evaluates 314 bugs from open-source Python projects in BugsInPy using over 19K prompts composed from seven bug-related facts, including code context, stack traces, issue descriptions, test information, and angelic values. That study concludes that each fact is beneficial for some bugs, repair effectiveness is non-monotonic in the number of facts, there is no globally optimal fact set, and the proposed bug-specific selector Maniple repairs 88 bugs on a 157-bug testing dataset, 17% above the best configuration (Parasaram et al., 2024).

A later APR study uses 314 bug–patch pairs from BugsInPy to evaluate a layered knowledge injection framework. It reports that progressive expansion from bug-level context to repository-level and project-level knowledge improves repair rates for both evaluated models. With Llama 3.3, the study reports 65% fixed (207/314) at the Bug Knowledge Layer, 74% fixed (235/314) at the Repository Knowledge Layer, and 79% fixed (250/314) at the Project Knowledge Layer. The same work compares against prior results attributed to Maniple at 56% (177/314) and describes the best layered result as a 23% improvement over prior work (Ehsani et al., 30 Jun 2025).

BugsInPy is equally central in empirical fault localization. Two closely related studies evaluate 135 real-world bugs from 13 open-source projects using seven fault localization techniques in four families: SBFL with DStar, Ochiai, and Tarantula; MBFL with Metallaxis and Muse; predicate switching; and stack trace fault localization. Across the benchmark, the top-5 counts reported for FauxPy are 40 for Metallaxis, 34 for Muse, 9 for predicate switching, 57 for DStar, 58 for Ochiai, 58 for Tarantula, and 8 for stack traces. These results are used to argue that SBFL is the strongest general-purpose choice on this Python benchmark, MBFL is slower and generally weaker here, and predicate switching and stack traces are specialized techniques with narrow applicability (Rezaalipour et al., 2024).

The benchmark is also widely used for type-error research. In TypeFix, BugsInPy is preprocessed into 54 type errors after splitting multi-bug commits and removing duplicates; on that benchmark, TypeFix reports 26 / 41 correct / plausible repairs and 74.1% template coverage of developer patches (Peng et al., 2023). In RTED, BugsInPy is paired with TypeBugs for LLM-based unit-test generation aimed at detecting Python type errors; on BugsInPy specifically, the paper reports Precision = 0.78, Recall = 0.50, F1 = 0.61, Accuracy = 0.70, 34 detected bugs, and 13 false positives for the full system (Yang et al., 3 Jul 2025).

The benchmark is similarly important in test-generation studies that target fault detection rather than coverage. A comparative study of LLM-generated versus human-written unit tests filters BugsInPy down to 29 bugs and reports that retrieval-augmented LLM-generated tests detect faults in 20 of 29 bugs (69.0%), compared with 5 of 29 bugs (17.2%) for general-purpose human-written tests, with Fisher’s exact test $p < 0.001$ and Cohen’s $h = 1.10$ (Vathana et al., 7 Jun 2026).

BugsInPy also supports studies of offline, locally executed LLMs. One such evaluation attempts to process the full 501-bug dataset, filters it to 349 bugs across 17 projects, and uses a zero-shot, function-level setup with LLaMA 3.2 and Mistral deployed locally via Ollama. It reports 43.3% accuracy for LLaMA 3.2 and 44.4% accuracy for Mistral, together with a large number of “partial” responses, suggesting that local models can often identify suspicious regions without pinpointing the exact repair (Vulićević, 25 Apr 2026).

5. Extensions, derivatives, and benchmark generalizations

BugsInPy has been extended rather than merely reused. One major derivative is Tests4Py, which is explicitly described as being derived from BugsInPy and as taking BugsInPy as both “the source of inspiration and foundation” and the “database of existing bugs in Python programs.” Tests4Py retains real bugs and fixes but adds per-subject functional oracles, system-test harnesses, input grammars, and support for generated unit tests and system tests. It includes 73 bugs from 7 real-world Python applications and 6 bugs from 4 example programs, uses the same bug identifiers as BugsInPy, and reports twenty unique unit tests per fault (Smytzek et al., 2023).

Another major extension is BugsInPy-mf, which addresses the limitation that benchmark entries are usually treated as single-fault versions even though real project versions often contain multiple concurrent faults. BugsInPy-mf does not inject synthetic bugs into the program source. Instead, it identifies additional already-existing faults in earlier versions using test case transplantation and fault location translation. Across the extension, the paper reports on average 18.6 faults per version in 501 versions across 17 Python projects, with 6.3 test cases transplanted per version and an average drop rate of 14.3% when transplantation succeeds but location translation fails (Callaghan et al., 2024).

These derivatives make explicit two distinct limitations of the original benchmark. First, BugsInPy’s original oracle model is centered on regression-style failing unit tests, which is highly effective for repair and localization but less suitable for system-level test generation without additional harnesses and oracles. Second, its standard usage largely assumes one identified fault per entry, whereas real software versions can contain several faults whose interactions matter for localization and repair. Tests4Py and BugsInPy-mf address these limitations in different directions: one by broadening the oracle and execution model, the other by broadening the fault model (Smytzek et al., 2023).

6. Reproducibility, limitations, and long-term significance

Although BugsInPy is curated, later studies document reproducibility and processing limitations. A cross-language APR study notes that BugsInPy has 493 bugs in 17 Python projects, but cites prior work finding that only 67% of the expected results could be reproduced in the unmodified dataset, leaving 337 reproducible bugs; that study uses 82 reproducible single-hunk bugs in one experiment and a stratified sample of 100 bugs in a later experiment (Campos et al., 3 Jun 2025). A fault-localization study reports that from an original 501 bugs from 17 projects, 120 bugs failed to reproduce, 39 bugs missed development dependencies, and 2 bugs had empty ground truth, leaving 340 reproducible bugs in 13 projects before later sampling 135 faults for the main experiments (Rezaalipour et al., 2023).

Task-specific preprocessing can further reduce the usable subset. The local-LLM bug-detection study excludes bugs if the source file cannot be recovered, the buggy line cannot be found, or no enclosing def or class can be identified, reducing 501 bugs to 349, or 69.7% of the full dataset (Vulićević, 25 Apr 2026). These observations do not negate the value of the benchmark, but they show that benchmark usage often depends on environment recreation, project dependencies, and task-specific assumptions such as function-level extraction or fault-localization granularity.

The original database paper also discusses broader threats to validity. The benchmark is limited to Python 3 projects, to popular GitHub repositories, and to projects with more than 10,000 stars, so it is biased toward large and well-maintained open-source systems. The authors acknowledge possible mislabeling despite manual curation, the possibility of bugs in the benchmark infrastructure itself, and the fact that benchmark construction requires subjective judgment about whether a commit is a true bug fix and whether it is sufficiently isolated (Widyasari et al., 2024).

Even with those limitations, BugsInPy occupies a foundational position in Python software engineering research. It provides real bugs, fixed versions, reproducible failing tests, isolated patches, and an execution framework that abstracts away substantial engineering overhead. Later work repeatedly uses it as the main real-world Python benchmark for automated program repair, fault localization, type-error detection, unit-test generation, and LLM-based debugging. A plausible implication is that BugsInPy’s enduring importance lies not only in its size, but in the combination of historical authenticity, reproducible exposure, and tool-oriented accessibility that makes controlled Python debugging studies possible at scale (Widyasari et al., 2024).