Agentic Model Checking

Abstract: Verifying LLM-generated systems code is hard: bugs are prevalent, formal specifications are missing, and safety contracts are encoded implicitly at call sites rather than enforced at function boundaries. We propose agentic model checking, a paradigm that couples LLM agents with a bounded model checking backend under the principle agents propose, solvers verify: agents handle tasks requiring semantic judgment (spec inference, check selection, counterexample classification, refinement proposal) while BMC discharges every soundness-relevant decision. The paradigm rests on three commitments. Specifications are inferred top-down from caller context in a restricted DSL that translates deterministically into the backend's assume/assert primitives, with optional functional-correctness clauses lifting verification from panic-freeness to behavioural faithfulness. Verification is compositional: each function is checked in isolation against its spec with callees replaced by postcondition-constrained stubs, so per-query cost scales with a single function's state space and refinements propagate automatically to callers. Counterexamples are not bug reports: they pass through a validation pipeline (reachability, callee feasibility, dynamic replay, realism audit) that distinguishes active in-tree crashes from latent public-API failures, while modelling artifacts drive a refinement loop rather than being suppressed. We instantiate the approach in BMC-Agent and evaluate it on LLM-generated kernel and compiler code in C and Rust alongside mature OSS-Fuzz-hardened libraries, confirming real defects, producing bounded clean verifications on heavily-fuzzed surfaces, and establishing functional equivalence on selected algorithmic functions.

• The paper proposes an agentic model checking framework that leverages LLM agents to infer specifications and delegate verification to deterministic BMC backends.
• It employs top-down specification inference, compositional verification, and multi-stage counterexample validation to enforce memory and arithmetic safety.
• Empirical evaluations on C and Rust code demonstrate the method's ability to surface realistic bugs missed by traditional LLM-only or static analysis approaches.

Agentic Model Checking: An Authoritative Summary

Problem Context and Motivation

LLM-generated systems code increasingly permeates critical software domains such as OS kernels, device drivers, and compilers, in both C and Rust. Verification of such code is particularly challenging due to the lack of explicit formal specifications, prevalence of low-level bugs (e.g., integer overflows, OOB accesses), and a common anti-pattern: invariants and safety contracts are maintained at call sites rather than enforced at function boundaries. Traditional bounded model checking (BMC) and compositional verification techniques are powerful, but they rely on explicit specifications, which are largely absent in AI-generated code. Prior LLM-only or hybrid LLM-static analysis approaches scale but fail for arithmetic and memory safety, while classical deductive verification techniques are annotation-heavy and don’t scale to large, loop-heavy systems code.

Agentic Model Checking Paradigm

Agentic model checking is introduced as a paradigm coupling LLM agents with deterministic BMC backends (CBMC for C, Kani for Rust), under the principle: "agents propose, solvers verify." LLM agents handle tasks requiring semantic judgment—including per-function specification inference, check selection, counterexample classification, and refinement proposal. The BMC backend acts as a soundness anchor, discharging all verification decisions and producing concrete counterexamples.

The architectural commitments are:

• Top-down Specification Inference: Specifications (pre/postconditions) are inferred from caller context in a restricted DSL that deterministically translates to BMC assume/assert primitives. Functional-correctness clauses are generated that lift specs from panic-freeness to behavioral faithfulness.
• Compositional Verification: Functions are verified in isolation, with callees replaced by stubs constrained by their postconditions. This leverages assume-guarantee discipline for scalability and parallelism, with per-query cost scaling only with the function’s state space.
• Validated Bug Reporting: Counterexamples from BMC are not treated as bug reports directly but undergo multi-stage validation (reachability, callee feasibility, dynamic replay, realism audit) to distinguish active, realistic bugs from latent or infeasible errors. Unrealistic witnesses trigger an adaptive refinement loop, propagating spec improvements through the call graph.

  Figure 1: BMC-Agent architecture, coupling agentic specification inference, flag selection, BMC execution, multi-stage validation, and adaptive refinement with persistent knowledge-base storage.

Architecture Details

Specification Generation

LLM agents traverse the call graph top-down, inferring per-function pre/postconditions in a restricted DSL. Specifications are validated by deterministic parsing and are generated twice for robustness (caller intent vs implementation behavior). The DSL is tailored for OS- and driver-style bugs and directly translatable to CBMC/Kani primitives.

Optional functional-correctness specs are generated for algorithmic helpers (e.g., reference equivalence, algebraic identities, round-trip properties) and AND-merged into postconditions, elevating verification from syntactic safety to semantic correctness.

Per-Function Check Selection

An agentic flag selector enables only those arithmetic or memory safety checks semantically warranted by the function’s context (e.g., unsigned/signed overflow, conversion, pointer checks) to avoid verification noise and improve precision, a critical mechanism for surfacing subtle bugs.

BMC and Deduplication

The BMC engine synthesizes harnesses per function, initializing nondeterministic inputs, injecting environmental assumptions, and translating specifications to assume/assert code. Outcomes are classified as verified clean, counterexample, or inconclusive. Counterexample deduplication collapses redundant witnesses to representative classes before validation.

Validation Pipeline

Four validation stages ensure only realistic, reachable bugs are reported:

1. Input Reachability: Propagation of witness inputs up the call graph, determining system-entry reachability.
2. Callee Feasibility: Substitution of real callee bodies for stubbed returns, ensuring witness is reachable from real call sites.
3. Dynamic Replay: Host-side GCC harness execution of counterexample, with signal capture (e.g., SIGSEGV) indicating concrete defects.
4. Realism Audit: Deterministic witness-pattern detectors filter modeling artifacts; LLM audits classify realism and latent status.

Unrealistic witnesses are not suppressed but initiate adaptive refinement, tightening specs or remediating artifacts at verifier, caller, or callee-contract level.

Adaptive Refinement and Persistent Knowledge

The pipeline incorporates a CEGAR-derived multi-level refinement loop at the specification level: spurious counterexamples drive iterative spec tightening, which propagates compositional updates to callers. Refinements and witness patterns are persisted in a knowledge base and pattern library, amortizing future verification costs and improving precision over successive runs.

Scalability

The architecture achieves scalability via compositional decomposition (per-function isolation), parallelism within call-graph layers, selective flag enabling, and persistent knowledge-carry-forward. Per-query cost scales with function complexity, not codebase size.

Empirical Evaluation and Results

BMC-Agent was evaluated on four corpora:

• VibeOS kernel (C): 34 confirmed realistic findings across 12 modules (out of 675 functions), with 16 reproduced as runtime faults.
• OSS-Fuzz-hardened libraries: jq, OpenSSL, libcurl, libxml2, protobuf upb; two undefined-behavior defects in jq, bounded clean verifications on heavily-fuzzed surfaces, and verified functional equivalence in wire-format parsers.
• Realtek r8125 Linux driver (C): CAP_NET_ADMIN-gated MMIO bounds-check bypass found.
• claudes-c-compiler (Rust): 25 real bugs, nearly all panic-class defects on public-API byte helpers; bounded functional-equivalence proofs for ELF hash and header-write helpers.

Strong numerical results: Across the VibeOS kernel and CCC, agentic model checking surfacing dozens of realistic, system-entry point bugs that are missed by the type system or are not immediately visible to LLM-only reviewers. The realism audit is empirically effective at triaging modeling artifacts and focusing developer attention.

Practical and Theoretical Implications

Practically, agentic model checking provides a scalable, automated pipeline for verifying LLM-generated systems code, producing actionable bug reports with concrete evidence, and positive verification of hardened library surfaces. Its compositional refinement loop amortizes token cost and verification effort across sweeps and scales to large codebases without annotation overhead.

Theoretically, the paradigm demonstrates how coupling semantic judgment (from LLMs) with deterministic verification (BMC) can bridge the specification gap for AI-generated code, propagating bug-finding and hardening tasks up the call graph and embedding CEGAR at the specification level. The ability to produce functional-equivalence witnesses in addition to panic-free proofs pushes verified software assurance beyond defensive programming.

Limitations and Future Directions

Known limitations include dependence on LLM spec correctness (over-weak specs may miss bugs), unwinding bound restricting loop-depth exploration, and dynamic harnesses relying on host-side execution for signal reporting. Spec fabrication remains nontrivial for domain-heavy encodings, remedied by realism checks and pattern detectors.

Future work is outlined on quantitative evaluation at scale (precision/recall, staged ablation, BMC-alone baselines), multi-source spec triangulation, evaluation on historical buggy corpora, and further amortization of token cost via persistent knowledge-base and stage distillation.

Conclusion

Agentic model checking, instantiated in BMC-Agent, provides a robust paradigm for automated verification of LLM-generated systems code, coupling semantic inference and deterministic model checking to produce validated bug reports, positive verification evidence, and bounded functional-correctness results. Its compositional, agent-driven architecture scales to whole codebases, adapts specifications over time, and accommodates language diversity (C, Rust). The implications for automated assurance of AI-generated software are substantial, laying foundations for scalable verification pipelines that blend linguistic semantic power with formal reasoning rigor.