QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs (1912.09686v1)

Abstract: RESTful APIs are an increasingly common way to expose software systems functionality and it is therefore of high interest to find methods to automatically test and verify such APIs. To lower the barrier for industry adoption, such methods needs to be straightforward to use with a low effort. This paper introduces a method to explore the behaviour of a RESTful API. This is done by using automatic property-based tests produced from OpenAPI documents that describe the REST API under test. We describe how this method creates artifacts that can be leveraged both as property-based test generators and as a source of validation for results (i.e., as test oracles). Experimental results, on both industrial and open source services, indicate how this approach is a low effort way of finding real faults. Furthermore, it supports building additional knowledge about the system under test by automatically exposing misalignment of specification and implementation. Since the tests are generated from the OpenAPI document this method automatically evolves test cases as the REST API evolves.

• The paper introduces QuickREST as an automated method that uses property-based testing to generate tests for OpenAPI-described RESTful APIs.
• It distinguishes between stateless and stateful test strategies to improve test coverage and detect discrepancies in API behavior.
• Experimental results demonstrate QuickREST’s ability to uncover real faults and simplify debugging through effective failure shrinking.

QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs

The paper in question discusses the introduction of QuickREST, a method for automatically generating property-based tests for RESTful APIs described by OpenAPI documents. This method is designed to provide an accessible, low-effort solution that is intended to be easy to adopt in industrial settings where efficient testing of software systems is crucial.

Overview of the Method

QuickREST leverages OpenAPI, a popular specification for describing RESTful APIs, to automatically create test cases and oracles. The method utilizes property-based testing (PBT), a technique known for its ability to generate random test scenarios and verify predefined properties. In this instance, PBT helps to ensure that the API behaves as specified. The method's black-box approach allows it to be applicable across various platforms and systems without requiring detailed knowledge of the underlying implementation. This is beneficial for developers and testers looking to validate APIs with minimal configuration or setup effort.

Experimental Results

The methodology was applied to both industrial and open-source services, including GitLab, demonstrating its practicality and utility in varied environments. The experimental results highlighted the capability of QuickREST to uncover real faults in APIs, such as input validation bugs and specification deviations, with minimal human intervention. By automatically generating tests directly from the OpenAPI documents, the tests evolve alongside changes in the API, ensuring sustained relevance through API version updates.

One key finding was the distinction between stateless and stateful test generations. Stateless tests randomly generate input values without consideration of the API's current state, whereas stateful tests use previously observed data to inform input generation, allowing for intelligent testing sequences that mirror realistic API usage scenarios. This distinction aids in achieving comprehensive coverage of API behavior, particularly in complex systems where interdependent API calls occur.

Insights and Implications

QuickREST provides numerous insights beyond simple fault detection. By comparing returned API call data against the OpenAPI specifications, it identifies discrepancies that suggest under-specification. Understanding such misalignments prompts refinement of API documentation and supports better alignment between intended and actual API behavior.

Furthermore, the shrinking feature—where simpler reproducing cases are derived from initial failures—enhances debugging efficacy, enabling developers to pinpoint root causes with clarity. Overall, the methodology fosters a productive interaction between exploratory testing and automated verification, empowering QA teams to focus on higher-level analysis while offloading repetitive testing tasks.

Future Developments

Potential enhancements of the QuickREST tool include incorporating machine learning algorithms to optimize generator configurations based on historical test outcomes, thereby increasing fault detection potential and response code coverage dynamically. Moreover, integrating formal models or abstractions of expected API behavior could further refine test case generation and validation approaches, enabling even more nuanced exploration of complex API ecosystems.

In summary, QuickREST exemplifies a pragmatic approach to automated API testing that provides both efficient fault detection and knowledge generation about the system under test, proving valuable for continued API development and maintenance. Its advancements contribute to the broader field of software testing, offering insights for evolving methods to manage increasingly complex web services.