BugForge: Constructing and Utilizing DBMS Bug Repository to Enhance DBMS Testing

Abstract: DBMSs are complex systems prone to bugs that may lead to system failures or compromise data integrity. Establishing unified DBMS bug repositories is crucial for systematically organizing bug-related data, enabling code improvement, and supporting automated testing. In particular, bug reports often contain valuable test inputs and bug-triggering clues that help explore rare execution paths and expose critical buggy behavior, thereby guiding automated DBMS testing. However, the heterogeneity of bug reports, along with their incomplete or inaccurate content, makes it challenging to build unified repositories and convert them into high-quality test cases. In this paper, we propose BugForge, a framework that constructs standardized DBMS bug repositories and leverages them to generate high-quality test cases to enhance DBMS testing. Specifically, BugForge progressively collects bug reports, then employs syntax-aware processing and input-adaptive raw PoC extraction to construct a DBMS bug repository. The repository stores structured bug-related data, including bug metadata and raw PoCs that entail potential bug-triggering semantics. These data are further refined into high-quality test cases through semantic-guided adaptation, thereby enabling enhanced DBMS testing methods, including DBMS fuzzing, regression testing, and cross-DBMS bug discovery. We implemented BugForge for PostgreSQL, MySQL, MariaDB, and MonetDB, totally integrated 37,632 bug reports spanning up to 28 years. Based on the repository, BugForge uncovered 35 previously unknown bugs with 22 confirmed by developers, demonstrating the value of constructing and utilizing bug repositories for DBMS testing.

• The paper introduces BugForge, an automated pipeline that constructs and extracts actionable PoCs from heterogeneous DBMS bug reports using a hybrid LLM-RAG approach.
• It presents a multi-stage process for repairing and adapting SQL PoCs, achieving up to 30% higher code coverage and a 130% increase in bug discoveries.
• Empirical validation on over 37K bug reports demonstrates its efficacy in enhancing fuzzing, regression testing, and cross-DBMS validation with an 89% adaptation success rate.

BugForge: Automated Construction and Utilization of DBMS Bug Repositories for Enhanced Database Testing

Motivation and Challenges

The exponential growth in the complexity and adoption of modern Database Management Systems (DBMSs) has significantly heightened the need for systematic, large-scale bug discovery, regression assurance, and robust cross-system validation. Despite the proliferation of bug repositories in other software domains, DBMSs have historically lacked comprehensive, standardized repositories of actionable defects. Consumer bug reports are heterogeneous, largely unstructured, and typically encode critical context, environment dependencies, and potential proof-of-concept (PoC) triggers in ambiguous natural language, which substantially impedes automated extraction and test generation pipelines.

A central technical bottleneck is highlighted by cases where bug-triggering sequences are embedded in report prose rather than clearly delimited code blocks.

Figure 2: MySQL Bug Report #110804 exemplifies the challenge of extracting PoCs from prose, where the sequence to trigger the bug is ambiguous and must be carefully reconstructed from the report context.

Such complexity introduces high rates of false negatives (missed PoCs) and false positives (malformed or irreproducible tests) in current DBMS testing and fuzzing methodologies.

BugForge Framework: Automated Repository Construction

BugForge introduces a comprehensive, automated, end-to-end pipeline for constructing actionable DBMS bug repositories and generating high-quality test cases from real-world bug reports. The system architecture consists of phased modules for incremental report ingestion, syntax-aware report preprocessing, adaptive contextual extraction using LLMs with retrieval-augmented generation (RAG), and multi-stage, semantics-preserving adaptation of raw PoCs.

Figure 4: Schematic of BugForge: incremental bug report collection, syntax-aware and LLM-augmented extraction of raw PoCs, followed by semantic-guided adaptation and stable environment orchestration for high-assurance DBMS testing.

Key innovations in the repository construction process include:

• Incremental, Source-Aware Ingestion: Bug reports are continuously harvested from official trackers and community repositories, with adaptive synchronizations to minimize redundant I/O and ensure long-term consistency.
• Syntax-Aware Fragment Identification: Heuristic analysis localizes natural-language segments and their embedded SQL, using block delimiters, SQL syntax patterns, and score-based SQL feature weighting to prioritize promising PoC candidates.
• Hybrid LLM-RAG Extraction Pipeline: Contextual prompts and reference exemplars guide LLMs to reconstruct usable raw PoCs from fragmented and ambiguous report content, automatically tracking quality and updating the reference base for continuous improvement.
• Structured, Versioned Case Construction: Extracted PoCs are persistently linked with metadata (CVEs, affected versions, developer status, and report evolution history) to allow for downstream analytics and systematic regression monitoring.

Semantic-Guided Test Case Adaptation

Post-extraction, BugForge addresses the challenge of restoring executability and semantic fidelity for high-quality test generation:

• Feedback-Driven Iterative Repair: Initially malformed or incomplete PoCs are subjected to staged execution in target DBMSs, with diagnostic classification (e.g., syntax, configuration, semantic errors) and targeted LLM-driven repair until an execution-valid path is restored.
• Semantic-Constrained Adaptation: Adaptation is constrained by fixed semantic anchors—critical schema objects, clause structure, and operator presence—extracted from the raw PoC, ensuring that critical bug-triggering logic is not lost in the pursuit of executability.
• Risk-Stratified Environment Stabilization: A hierarchical impact assessment parses test-level and global-environment-altering statements, dynamically orchestrating container or service restarts only where necessary to maintain case isolation, execution independence, and high throughput.

Repository Utilization in Advanced DBMS Testing

BugForge’s resulting repositories—spanning over 37,000 historical reports and 35,000+ extracted PoCs for MySQL, MariaDB, PostgreSQL, and MonetDB—establish a new foundation for advanced DBMS testing workflows:

• Fuzzing Campaigns: Historical bug-triggering structures, when used as mutation seeds, substantially increase code branch coverage (e.g., a 22% to 30% improvement versus standard test cases), enabling deeper and more diverse code path exploration in state-of-the-art fuzzers such as Griffin and Squirrel.
• Automated Regression Testing: High-quality, semantic-rich test cases generated from fixed bugs are replayed across release sequences to discover regression faults, effectively catching both known and previously undisclosed remanent bugs.
• Cross-DBMS Transfer Testing: Repositories enable systematic transfer of PoCs across related DBMS dialects (e.g., MySQL→MariaDB), uncovering latent execution and compatibility issues that are invisible to single-system techniques.

Empirical Results and Notable Claims

• Repository Scope: BugForge produced a repository featuring 37,632 bug reports and 35,530 raw PoCs, with coverage of 2,597 CVEs across four major DBMSs.
• Testing Efficacy: Out of 35 previously unknown bugs found via repository-enhanced testing, 22 received prompt developer confirmation spanning critical components like optimizers and parsers.
• Coverage and Quality: Test seeds derived from BugForge consistently enabled up to 30% greater code coverage and 130% more bug discoveries compared to DBMS built-in/fuzzer-generated test cases.
• Adaptation Effectiveness: The system’s hybrid adaptation pipeline delivered an adaptation success rate above 89%, balancing high executability with semantic retention.

Implications, Limitations, and Outlook

BugForge establishes a scalable, extensible baseline for leveraging real-world DBMS bug data to catalyze improved code quality, comprehensive vulnerability identification, and robust regression assurance. Its design—decoupling system-independent adaptation logic from backend-specific execution environments—suggests robust applicability to other DBMSs and potentially to adjacent domains with semi-structured test artifacts.

Practical implications include: enabling more targeted and efficient fuzzing, reducing manual triage workload, and facilitating data-driven analytics of defect patterns for both academic research and industrial QA pipelines. The system’s demonstrated extensibility provides a blueprint for automating test generation even as SQL dialects and DBMS architectures evolve.

Future research directions include broader dialect coverage, integration of deeper semantic reasoning for heterogeneous environments, and the development of analytic tools leveraging the curated repository for root cause analysis, repair suggestion synthesis, and meta-testing across system versions.

Conclusion

BugForge advances the state of the art in automated DBMS bug repository construction and utilization, setting a new standard for fuzzing, regression, and cross-system SQL engine validation. By integrating modern LLMs with domain-specific extraction heuristics and semantically-aware adaptation strategies, it dynamically bridges the gap between noisy, unstructured bug reports and actionable, high-impact test cases—yielding measurable gains in branch coverage and defect discovery for contemporary DBMS stacks (2604.03024).