BOSQTGEN: Breaking the Sound Barrier in Test Generation

Abstract: Modern software is increasingly built by composing APIs, elevating the API contract to a critical role. Inadequate contracts, however, lead to mismatched expectations and failures, creating a pressing need for robust conformance testing. Current test generation techniques are hindered by key challenges: polyglot systems, source code inaccessibility, a cost-reliability trade-off, and, most critically, the difficulty of generating structured inputs. We introduce BOSQTGEN, a novel black-box methodology and tool for API test generation. BOSQTGEN utilizes a novel approach for decomposing API specifications into primitives, using LLMs to suggest coherent strata for them, and employing combinatorial testing to efficiently sample over these values. This approach ensures coverage of critical interactions while avoiding the redundancy of random sampling. The resulting BOSQTGEN system achieves an average of 82% code coverage on RESTful benchmarks, often a 20% or more increase over prior state-of-the-art systems and nearing parity with hand-written test suites. Providing a fully API-driven approach to test generation, enables developers to automatically create high-quality test cases for validation or test-driven development.