Dinkel: Testing Graph Database Engines via State-Aware Query Generation (2408.07525v2)

Abstract: Graph database management systems (GDBMSs) store and manipulate graph data and form a core part of many data-driven applications. To ensure their reliability, several approaches have been proposed to test GDBMSs by generating queries in Cypher, the most popular graph query language. However, Cypher allows queries with complicated state changes and data dependencies, which existing approaches do not support and thus fail to generate valid, complex queries, thereby missing many bugs in GDBMSs. In this paper, we propose a novel state-aware testing approach to generate complex Cypher queries for GDBMSs. Our approach models two kinds of graph state, query context and graph schema. Query context describes the available Cypher variables and their corresponding scopes, whereas graph schema summarizes the manipulated graph labels and properties. While generating Cypher queries, we modify the graph states on the fly to ensure each clause within the query can reference the correct state information. In this way, our approach can generate Cypher queries with multiple state changes and complicated data dependencies while retaining high query validity. We implemented this approach as a fully automatic GDBMS testing framework, Dinkel, and evaluated it on three popular open-source GDBMSs, namely Neo4j, RedisGraph, and Apache AGE. In total, Dinkel found 60 bugs, among which 58 were confirmed and 51 fixed. Our evaluation results show that Dinkel can effectively generate complex queries with high validity (93.43%). Compared to existing approaches, Dinkel can cover over 60% more code and find more bugs within the 48-hour testing campaign. We expect Dinkel's powerful test-case generation to benefit GDBMS testing and help strengthen the reliability of GDBMSs.

• The paper introduces a novel state-aware approach that incrementally generates valid, complex Cypher queries to uncover bugs in graph database engines.
• It evaluates Dinkel on Neo4j, RedisGraph, and Apache AGE, revealing 60 unique bugs with a 93.43% query validity rate and significantly improved code coverage.
• The framework effectively handles advanced query features like FOREACH, CALL, and UNION, setting a new standard for testing the robustness of GDBMSs.

Overview of "Dinkel: Testing Graph Database Engines via State-Aware Query Generation"

The paper "Dinkel: Testing Graph Database Engines via State-Aware Query Generation" introduces a novel approach for testing Graph Database Management Systems (GDBMSs) through the generation of complex Cypher queries. The authors aim to address limitations in existing approaches that fail to model intricate state changes and data dependencies inherent to Cypher queries.

Problem Statement

GDBMSs, such as Neo4j, RedisGraph, and Apache AGE, are essential for efficiently managing interconnected data via graph structures. Ensuring their reliability is crucial, as bugs in these systems can lead to severe data corruption and security issues. Existing methods for testing GDBMSs generate relatively simple Cypher queries, missing critical bugs due to the inability to handle complex state modifications and dependencies.

Proposed Solution

This paper proposes a state-aware approach that systematically models two kinds of graph state: query context and graph schema.

• Query Context: Pertains to variables declared in a query, including their types and scopes.
• Graph Schema: Relates to graph structures such as labels and properties.

Using these abstractions, the authors propose an incremental query generation method. They generate Cypher queries clause by clause, updating the graph state information at each step to ensure the generation of valid and complex queries.

Implementation

The authors implemented their approach into a fully automatic framework named Dinkel. This framework not only supports query generation but also integrates mechanisms for query reduction and efficient bug deduplication.

Dinkel's complexity is apparent in its diverse support for Cypher clauses and its ability to efficiently update and manage the graph state model. Additionally, it extends its utility by handling complex subqueries and clauses such as FOREACH, CALL, UNION, EXISTS, and COUNT.

Evaluation

Experiments: The authors evaluated Dinkel on the latest versions of three prominent GDBMSs: Neo4j, RedisGraph, and Apache AGE.

Results:

• Dinkel identified 60 unique bugs, with 58 confirmed and 51 fixed.
• The validity rate of generated queries was 93.43%.
• Dinkel's generated queries averaged 21.72 clauses and contained multiple complex data dependencies.
• Code coverage was significantly higher than existing tools, showing improvements of 67% and 85% in Neo4j and RedisGraph, respectively.

Notable Findings

• Bug Types: The authors categorized the bugs into internal errors and crashes, providing specific examples of high-impact bugs and their triggering queries.
• Clause Effectiveness: The comprehensive analysis indicated that the approach could discover deep-rooted issues by modeling and leveraging complex state changes that other tools overlooked.
• High Complexity Handling: Dinkel efficiently handled advanced Cypher features and clauses, which were fundamental in discovering 27 unique bugs that existing tools missed.

Practical Implications

Dinkel's state-aware approach enhances the reliability and robustness of GDBMSs by emphasizing thorough and complex query generation. This focus not only improves bug detection capability but also facilitates deeper testing of GDBMS functionalities by covering broader code paths.

Future Directions

The research opens up several avenues for further exploration, such as integrating Dinkel with various test-oracle constructions for logic bug detection or extending the approach to other query languages and database paradigms. By providing a robust foundation for generating complex and valid queries, Dinkel has the potential to inspire subsequent innovations in GDBMS testing methodologies.

In conclusion, this paper successfully addresses a critical gap in GDBMS testing by proposing and validating a state-aware query generation approach that effectively discovers significant and previously undetected bugs in widely-used GDBMSs.