FlyCatcher: Neural Inference of Runtime Checkers from Tests

Abstract: Complex software systems often suffer from silent failures, i.e., violations of the intended semantics that do not cause explicit errors. A promising approach to detect such errors is to use system-specific runtime checkers that monitor the execution of a system and check for violations of the intended semantics. However, writing such checkers for a given software system is challenging and time-consuming, and hence, rarely done in practice. This work presents FlyCatcher, an automated approach to derive runtime checkers from existing tests, i.e., from a resource available for most software systems. The critical challenge of such an approach is to generalize the behavioral properties encoded in a test case to arbitrary executions of a system. FlyCatcher addresses this challenge through a combination of LLM-based synthesis, static analysis, and dynamic validation, which infers a checker that monitors specific method calls and asserts properties that should hold when they are called. The inferred checkers are stateful, i.e., they reason about the system's behavior by maintaining a shadow state that abstracts the actual system state as needed by the checker. Our evaluation applies FlyCatcher to 400 tests from four widely used, complex software systems. The approach infers 334 checkers, out of which 300 are found to be correct via cross-validation. Compared with a state-of-the-art approach, our approach infers 2.6x more correct checkers, which enables it to detect 5.2x more errors. By contributing to the automated inference of runtime checkers from tests, this work enables the broader adoption of runtime checking as a practical approach to detect silent failures in complex software systems.

• The paper introduces FlyCatcher, which employs LLM-guided synthesis to automatically generate runtime semantic checkers from test suites.
• It combines lightweight static analysis with dynamic validation and a unique shadow state mechanism to robustly track complex system invariants.
• Evaluated on major Java platforms, FlyCatcher infers 2.6x more correct checkers and detects 5.2x more semantic bugs than existing approaches.

Neural Inference of Semantic Runtime Checkers from Tests: The FlyCatcher Approach

Introduction and Problem Setting

FlyCatcher, presented in "FlyCatcher: Neural Inference of Runtime Checkers from Tests" (2604.22028), addresses the critical gap in scalable, system-specific runtime semantic checking for large, production software. Silent failures—semantic violations that avoid explicit program errors—persist as a significant class of bugs in complex infrastructure software, corrupting data and undermining reliability while eluding conventional testing and formal verification. Manual construction of effective runtime checkers remains rare due to the non-trivial effort and requisite domain expertise. Existing automated approaches, including invariant mining and static test generalization, fall short in generalizing semantic intent and handling system statefulness robustly.

The core problem targeted by FlyCatcher is: Given a program test suite, can one infer robust, generalized runtime checkers that detect semantic deviations at execution time, capturing the high-level properties encoded in tests and enabling continual silent failure detection?

Methodology: LLM-Guided Synthesis and Validation Loop

FlyCatcher approaches this problem by combining LLM-based code synthesis, static code analysis, and staged dynamic validation. Given a target test $t$, context tests $T_{context}$, and validation tests $T_{val}$, the pipeline proceeds as follows:

• State-changing Method Identification: Lightweight static analysis and CodeQL extraction enumerate methods with side effects within $t$ and $T_{context}$, augmented by LLM-based textual annotation to identify non-trivial state-updating operations.
• LLM-based Checker Synthesis: Prompted with test code, method descriptions, example checkers, and import context, an LLM generates candidate runtime checkers. Each checker is constructed as a parameterized, static method that updates a "shadow state"—an auxiliary abstraction that tracks the relevant logical system state—on every state-changing method invocation and performs assertions modeling the semantic properties from the test(s).
• Shadow State Mechanism: The shadow state is a fine-grained mapping from target objects to tracked properties ($S: O \rightarrow (P \rightarrow V)$). It enables the generated checkers to handle complex, stateful invariants (e.g., list size congruence, object membership, etc.), updating as operations occur.
• Validation and Feedback Loop: Candidate checkers undergo static validation (syntax, assertion presence, absence of hallucinated calls, explicit method signatures) and dynamic validation (checker-instrumented workload re-execution) against unseen validation tests. Failure triggers iterative reframing through structured, LLM-fed feedback until a correct checker emerges or a retry budget is exhausted.

Instrumented target systems invoke the generated checkers on all identified state-changing operations at runtime, enabling continuous monitoring both during regression testing and in production contexts.

Evaluation: Effectiveness, Coverage, and Overhead

FlyCatcher's evaluation targets four complex, widely-used Java platforms: Apache Zookeeper, Cassandra, HDFS, and HBase, offering a comprehensive and realistic test bed. Across 400 randomly sampled target tests (100 per system), FlyCatcher’s pipeline achieves:

• 334 validated checkers from 400 target tests (with Claude Sonnet 4 LLM), of which
• 300 are cross-validated, i.e., correct under broader test suite workloads beyond their synthesis context.

Relative to the strongest baseline (T2C [lou2025deriving]), FlyCatcher infers 2.6x more correct checkers. In mutation analysis, FlyCatcher-generated checkers detect 5.2x more semantic bugs (26/5 mutants) than T2C, even when the underlying test workload misses them.

False positive rates are low (<0.00003 in Zookeeper, zero in Cassandra across millions of checker calls in Jepsen-induced fault injection workloads), demonstrating practical deployability.

FlyCatcher's synthesis overhead remains moderate: the median time for checker generation is 15 seconds/checker, with monetary LLM query cost at $0.60$/checker (Claude Sonnet 4). At runtime, checker instrumentation incurs a typical 2.7–40.3% overhead (upper bound in tested scenarios), with the expectation of lower overhead for full system runs.

Figure 2: Distribution of time (seconds per checker) to generate runtime checkers with FlyCatcher, reflecting scalable synthesis behavior.

Technical Contributions and Contrasts

• LLM Generalization: FlyCatcher leverages LLMs to infer abstract, parameterized program properties from concrete test code, successfully decoupling semantic property learning from fixed test constants and workloads.
• Shadow State Abstractions: Introduction of explicit, checkable shadow state enables robust tracking of system invariants that depend on complex multi-method interactions and temporal object properties, a substantial improvement over approaches lacking state modeling.
• Iterative Feedback-driven Synthesis: The static and dynamic validation loop, with explicit LLM-guided code repair and explicit diagnostics, systematically filters invalid/hallucinated outputs—an essential component for reliable automation.
• Demonstrated Practicality: The pipeline scales to large-scale, real-world platforms with high test and codebase complexity, with minimal manual configuration.

Implications, Limitations, and Prospects

Practical Impact: FlyCatcher demonstrates that LLMs—when suitably integrated with program analysis and validation techniques—enable practical, semi-automatic derivation of complex, domain-specific runtime semantic checkers from test code. This can lower the barrier to continuous semantic monitoring in production, significantly increasing silent failure detection rates.

Limitations: The approach is constrained by the representativeness and expressiveness of test code; properties not manifested or indirectly tested cannot be captured. Mutants missed by FlyCatcher often reflect insufficient test coverage or semantic equivalence. While the feedback loop mitigates LLM hallucinations, reliance on non-deterministic LLM outputs introduces some variability.

Theory and Future AI Directions: The pipeline validates the hypothesis that program understanding tasks involving behavioral semantic reasoning are tractable for current LLMs when augmented with static contextualization and robust validation. Future extension could explore even tighter coupling with symbolic execution, grafting of program-specific type-state or protocol mining (using model-based formalism), and application beyond Java to other ecosystems and languages. Further, continuous learning via online revalidation in production traces could yield self-improving semantic checker systems.

Conclusion

FlyCatcher (2604.22028) advances fully-automated, LLM-guided inference of runtime semantic checkers, offering scalable, robust monitoring for silent failures in large software systems. By synthesizing parameterized, stateful checkers grounded in existing tests, and leveraging iterative validation to ensure soundness, it enables broader adoption of runtime semantics checking without deep manual specification effort. Practical results on four complex systems show superior correctness, coverage, and usability over prior art. The approach exposes new avenues for AI-assisted software reliability—suggesting that generative models, when substantively constrained and validated, are viable partners in automated program analysis and runtime assurance.