Protocol Testing with I/O Grammars (2509.20308v1)

Abstract: Generating software tests faces two fundamental problems. First, one needs to generate inputs that are syntactically and semantically correct, yet sufficiently diverse to cover behavior. Second, one needs an oracle to check outputs whether a test case is correct or not. Both problems become apparent in protocol testing, where inputs are messages exchanged between parties, and outputs are the responses of these parties. In this paper, we propose a novel approach to protocol testing that combines input generation and output checking in a single framework. We introduce I/O grammars as the first means to completely specify the syntax and semantics of protocols, including messages, states, and interactions. Our implementation, based on the FANDANGO framework, takes a single I/O grammar, and can act as a test generator, as a mock object, and as an oracle for a client, a server, or both (or actually any number of parties), a versatility not found in any existing tool or formalism. User-defined _constraints}_can have the generator focus on arbitrary protocol features; $k$-path guidance systematically covers states, messages, responses, and value alternatives in a unified fashion. We evaluate the effectiveness of our approach by applying it to several protocols, including DNS, FTP, and SMTP. We demonstrate that I/O grammars can specify advanced protocol features correctly and completely, while also enabling output validation of the programs under test. In its evaluation, we find that systematic coverage of the I/O grammar results in much quicker coverage of the input and response spaces (and thus functionality) compared to the random-based state-of-the-art approaches.