Combining Static and Dynamic Approaches for Mining and Testing Constraints for RESTful API Testing (2504.17287v1)

Abstract: In API testing, deriving logical constraints on API response bodies is crucial in generating the test cases to cover various aspects of RESTful APIs. However, existing approaches are limited to dynamic analysis in which constraints are extracted from the execution of APIs as part of the system under test. The key limitation of such a dynamic approach is its under-estimation in which inputs in API executions are not sufficiently diverse to uncover actual constraints on API response bodies. In this paper, we propose to combine a novel static analysis approach (in which the constraints for API response bodies are mined from API specifications), with the dynamic approach (which relies on API execution data). We leverage LLMs to comprehend the API specifications, mine constraints for response bodies, and generate test cases. To reduce LLMs' hallucination, we apply an Observation-Confirmation (OC) scheme which uses initial prompts to contextualize constraints. %, allowing subsequent prompts to more accurately confirm their presence. Our empirical results show that~LLMs with OC prompting achieve high precision in constraint mining with the average of 91.2%. When combining static and dynamic analysis, our tool, RBCTest , achieves a precision of 78.5%. RBCTest detects 107 constraints that the dynamic approach misses and 46 more precise constraints. We also use its generated test cases to detect 21 mismatches between the API specification and actual response data for 8 real-world APIs. Four of the mismatches were, in fact, reported in developers' forums.

Combining Static and Dynamic Approaches for RESTful API Testing

The paper "Combining Static and Dynamic Approaches for Mining and Testing Constraints for RESTful API Testing" introduces a novel method for enhancing the test generation and oracle validation process of RESTful APIs. The authors propose an integrated approach that leverages both static analysis using LLMs and dynamic analysis through execution data to more precisely identify constraints on API response bodies.

Overview of the Approach

The paper highlights the limitation of existing API testing methods that predominantly rely on dynamic analysis. These methods often fall short because they depend solely on execution data, which might not cover all possible scenarios due to insufficiently diverse input data. Emphasizing the importance of logical constraints on API responses, the authors propose combining static analysis with LLMs and dynamic analysis to overcome these limitations.

1. Static Analysis Using LLMs: The paper introduces a technique for mining constraints from API specifications using LLMs. This approach uses sophisticated LLMs to extract logical constraints from textual descriptions within API specifications, such as the OpenAPI Specification (OAS). The authors implement an Observation-Confirmation scheme to enhance the precision of the constraints mined by the LLMs.
2. Dynamic Analysis: The dynamic component extends the AGORA tool, which is based on Daikon, to infer invariants from the execution data. AGORA functions by analyzing previous API requests and responses to learn expected behavior through detected invariants.
3. Combining Results: The integration of static and dynamic findings yields a more comprehensive constraint set, improving the identification and validation of constraints. The approach enhances test oracles by combining constraints gleaned from both API documentation and runtime data.

Empirical Results

The paper discusses empirical evaluations using two datasets: the AGORA dataset and a curated dataset comprising real-world APIs such as Stripe and GitLab. Notable results include:

• Precision: LLM-based static analysis achieves a precision of 91.2% in constraint mining, while the combined approach of RBCTest yields a precision of 78.5%, underscoring the utility of integrating static and dynamic analyses.
• Increased Coverage: The combined approach identified 107 constraints unique to static analysis and 527 unique to dynamic analysis. Without the combination, several constraints might remain undetected.
• Detection of Bugs: The tests generated from the mined constraints revealed inconsistencies between API specifications and actual executions, identifying 21 mismatches in real-world APIs, four of which were confirmed by users and reported on developer forums.

Implications and Future Directions

The success of combining static and dynamic approaches opens new avenues for RESTful API testing frameworks. The constraints mined from specifications ensure coverage of logical correctness, which dynamic methods might miss. Moreover, the method provides a means to effectively test APIs in various development stages and allows for identifying mismatches that may indicate either specification errors or application bugs.

Future research could explore enhancing semantic verifiers to further improve precision and recall rates. Additionally, expanding the scope of LLM capabilities in understanding complex inter-parameter dependencies might lead to even more refined API testing strategies.

In conclusion, the integrated approach presented in the paper provides a robust framework for RESTful API testing, offering valuable insights into constraint mining and test oracle generation, thus promising enhancements in both theoretical understanding and practical application in API development and testing.