Agentic Model Checking

• Agentic model checking is a verification approach focusing on agents’ local states, observations, and interactions rather than traditional global transition systems.
• It combines formal tools from epistemic, temporal, deontic, and strategic logics to analyze agent knowledge, intentions, and coalition capabilities.
• Extensions include runtime, probabilistic, and solver-backed techniques for verifying both decision logic and generated code in complex agentic systems.

Agentic model checking is a family of verification approaches in which the primary object of analysis is an agent, a coalition of agents, or an agentic workflow rather than an undifferentiated transition system. In the literature, the term covers several related practices: verifying what agents know in interpreted systems, what BDI or organization-aware agents believe, intend, and are obliged to do, what coalitions can achieve under strategic semantics, what tool-using or LLM-based agents do over execution traces, and what LLM agents infer and verify about generated code. Across these strands, the shared idea is that the model internalizes agent-local state, observations, permissions, messages, tools, or traces, while the specification language ranges over temporal, epistemic, strategic, probabilistic, or deontic properties of that agentic structure (Koleini et al., 2014, Dennis et al., 2013, Koohestani et al., 30 Jan 2026, Sun et al., 20 May 2026, Sheffler, 19 Aug 2025).

1. Scope and major research strands

The literature uses the expression in several distinct but connected senses. Earlier work treats agentic model checking as model checking for agent-oriented formalisms: interpreted systems, BDI programs, organization-aware agents, and multi-agent strategic models. More recent work extends the label to runtime verification and probabilistic assurance for tool-using agents, and to solver-backed verification pipelines in which LLM agents orchestrate specification inference, counterexample analysis, and refinement while a deterministic backend remains the source of truth (Koleini et al., 2014, Dennis et al., 2013, Alrahman et al., 2022, Koohestani et al., 30 Jan 2026, Sun et al., 20 May 2026).

Strand	Formal substrate	Representative focus
Epistemic-temporal verification	Interpreted systems, ACTLK, POL	Agent knowledge, information flow, observation updates
Agent-program verification	BDI programs, PSL, AIL/AJPF, CTL/PCTL	Beliefs, goals, intentions, organizational constraints
Strategic and interaction-centric MAS	ATL/ATLE, symbolic transition systems	Coalitions, reconfiguration, protocols, coalition abilities
Runtime and probabilistic assurance	Event traces, MDPs, PCTL, monitors	Online bounds, anomaly detection, workflow conformance
Solver-backed code verification	BMC with agent-guided orchestration	Spec inference, bounded proof, counterexample triage

A plausible implication is that “agentic model checking” is best understood as an umbrella term rather than a single formalism. Some works retain the classical model-checking emphasis on exhaustive exploration over finite-state abstractions, while others move toward runtime monitoring, probabilistic estimation, or bounded verification over actual executions and generated harnesses (Ferrando et al., 2022, Sheffler, 19 Aug 2025, Sun et al., 20 May 2026).

2. Epistemic and observation-centered foundations

A foundational strand models agents explicitly through local states and epistemic indistinguishability. In interpreted systems, a global state is a tuple of environment and agent-local states, and agent knowledge is induced by equality of local state: $s \sim_i s'$ iff $l_i(s)=l_i(s')$. The ACTLK fragment then combines universal CTL operators with knowledge operators, enabling specifications such as $AG \neg K_i(secret)$ and $AG(request_i \rightarrow K_i(authorized_i))$ (Koleini et al., 2014).

This framework is used to verify dynamic access-control policies in which agents gain knowledge both by direct observation and by logical inference. The encoding introduces, for each proposition $p$ and agent $i$, a local copy $p_i^{loc}$ and a read-enabled flag $p_i^r$. When read access is granted, the next state sets $p_i^r := true$ and refreshes $p_i^{loc}$ to the current value of $l_i(s)=l_i(s')$0; when access is absent, $l_i(s)=l_i(s')$1 and the local copy persists. The key lemma states that if $l_i(s)=l_i(s')$2 is true and the local copy is synchronized with $l_i(s)=l_i(s')$3, then the agent knows $l_i(s)=l_i(s')$4 or knows $l_i(s)=l_i(s')$5. This lets model checking detect information-flow vulnerabilities caused not only by reading but also by reasoning (Koleini et al., 2014).

The conference review system case study makes this point concrete. The confidentiality requirement

$l_i(s)=l_i(s')$6

is violated, while the corresponding positive-knowledge property with $l_i(s)=l_i(s')$7 holds after all papers are assigned. The counterexample is “pure reasoning”: knowledge arises without any direct read of the reviewer relation. This distinguishes epistemic model checking from readability-only approximations such as RW and PoliVer (Koleini et al., 2014).

To control state explosion, this strand uses variable-hiding abstraction and refinement. Concrete states are quotiented by agreement on visible propositions, yielding an abstract interpreted system $l_i(s)=l_i(s')$8 with preserved labeling, temporal steps, and epistemic steps. For ACTLK safety properties with finite, tree-like counterexamples, a temporal-epistemic CEGAR loop checks spuriousness by forward feasibility, epistemic reachability, and backward consistency, then reveals hidden propositions from conflict clauses. The preservation theorem is one-sided: if the abstract model satisfies an ACTLK safety property, then the concrete model satisfies it as well (Koleini et al., 2014).

A related epistemic line replaces read-permission dynamics with expectation-driven public observations. Public Observation Logic models each world with a regular-expression expectation $l_i(s)=l_i(s')$9 and updates the model after an observation $AG \neg K_i(secret)$0 by eliminating worlds whose residues satisfy $AG \neg K_i(secret)$1. The model-checking problem for POL is PSPACE-complete; the Word fragment is in PTIME, the Star-Existential fragment is NP-complete, and the Existential and Star fragments are PSPACE-hard. This yields another form of agentic model checking in which agents revise knowledge by matching observed action strings against expected protocols (Chakraborty et al., 2022).

3. Agent programs, organizations, and strategic multi-agent systems

A second major strand verifies the decision-making implementations of autonomous agents. In BDI-style systems programmed in Gwendolen and executed through AIL/AJPF, specifications are written in a Property Specification Language extending linear-time temporal logic with modalities for beliefs, goals, intentions, actions, and percepts:

$AG \neg K_i(secret)$2

This enables properties such as “the robot only believes it can leave if it believes it has found the human or believes the area is empty,” and “the agent only performs lane changes when it believes the corresponding lane is safe.” The method verifies the actual decision program and composes the result with separate continuous analysis through an assume–guarantee rule: if $AG \neg K_i(secret)$3, $AG \neg K_i(secret)$4, and $AG \neg K_i(secret)$5, then $AG \neg K_i(secret)$6 (Dennis et al., 2013).

Organization-aware verification extends this setup with explicitly organizational state. AORTA adds roles, objectives, dependencies, activated obligations, and violations through predicates such as $AG \neg K_i(secret)$7, $AG \neg K_i(secret)$8, and $AG \neg K_i(secret)$9. AJPF is extended with organizational modalities $AG(request_i \rightarrow K_i(authorized_i))$0 and $AG(request_i \rightarrow K_i(authorized_i))$1, allowing verification not only of what agents believe or intend, but also of whether they enact roles, satisfy obligations, and use dependency-based coordination correctly (Jensen, 2015).

Another line compiles agent specifications into standard symbolic model checkers while preserving semantics. vGOAL defines a specification

$AG(request_i \rightarrow K_i(authorized_i))$2

constructs a transition system for CTL checking and a DTMC for PCTL checking, and proves bisimulation and probabilistic bisimulation between the operational semantics and the generated models. The resulting pipeline targets NuSMV and Storm, and includes a quick error detection algorithm that reduces repeated goals and checks single-goal combinations before full multi-agent verification (Yang et al., 2024).

Strategic and interaction-centric variants move closer to ATL-like reasoning. R-CHECK targets reconfigurable multi-agent systems in which communication structure, channel connectivity, and coalitions are state-dependent. Its high-level agent language compiles to a symbolic global transition relation $AG(request_i \rightarrow K_i(authorized_i))$3, and command labels become logical atoms so that properties can refer directly to sends, receives, synchronizations, and joint missions, rather than only to low-level state variables (Alrahman et al., 2022). EPC and ATLE push this further by combining process-calculus operational semantics with epistemic operators $AG(request_i \rightarrow K_i(authorized_i))$4, $AG(request_i \rightarrow K_i(authorized_i))$5, $AG(request_i \rightarrow K_i(authorized_i))$6, and $AG(request_i \rightarrow K_i(authorized_i))$7 and strategic modalities such as $AG(request_i \rightarrow K_i(authorized_i))$8, $AG(request_i \rightarrow K_i(authorized_i))$9, and $p$0 (Yu et al., 30 Jan 2025).

A notable complication is undecidability. For ATL on imperfect-information concurrent game structures with perfect recall, model checking is not decidable in general. One response is to combine model checking and runtime verification: verify perfect-information sub-models statically and runtime-monitor the residual temporal behavior, with sound one-sided preservation back to the strategic setting (Ferrando et al., 2022). Another response, from a deontic angle, is to move to tractable action-permission logics in which modalities such as weak permission to admit, weak permission to ensure, strong permission to ensure, and strong permission to admit all remain polynomial-time model checking problems (Shi, 2024).

4. Runtime and probabilistic verification for agentic AI

Recent work applies agentic model checking to tool-using and LLM-based agents whose behavior unfolds as long stochastic traces rather than finite design-time models. Dynamic Probabilistic Assurance learns an MDP online from typed lifecycle events; TriCEGAR automates the key missing step by learning a predicate tree from traces, using information gain over next-action distributions to induce abstract states, refining the abstraction when a witness is unsupported by the observed trace trie, and then model checking quantitative PCTL properties such as $p$1 and $p$2 (Koohestani et al., 30 Jan 2026).

In this setting, actions are tool invocations, states are abstract leaves of the learned predicate tree, and transition probabilities are estimated empirically:

$p$3

The same learned MDP supports run-likelihood anomaly detection, with likelihood

$p$4

and log-likelihood computed as a sum. This gives agentic model checking a runtime, probabilistic interpretation: it becomes online assurance and guardrailing rather than purely offline proof (Koohestani et al., 30 Jan 2026).

A more lightweight runtime-monitoring approach appears in Oroboro. Here, the system is instrumented at a chosen sampling event such as before_tool_call, and assertions are written in a regular-expression-like temporal language over predicates like xferToGreeting, sayHello, and xferToWeather. The assertion

$p$5

corresponds, in the instantaneous-event view, to

$p$6

The monitor reports structured MATCH and FAILURE traces, and the paper shows that assertions hold across runs with a stronger model but fail under weaker models because of improper tool sequencing and failed coordination handoffs (Sheffler, 19 Aug 2025).

A broader orchestration-oriented formalization models a host agent together with a task lifecycle. The host agent receives a user request, resolves intent, discovers external entities, builds a task DAG, dispatches subtasks through a protocol-agnostic communication layer, and aggregates results. The task lifecycle includes states such as CREATED, READY, DISPATCHING, IN_PROGRESS, COMPLETED, FAILED, RETRY_SCHEDULED, FALLBACK_SELECTED, CANCELED, and ERROR. Over this model, the literature defines 17 host-agent properties and 14 task-lifecycle properties, including

$p$7

and

$p$8

thereby making MCP and A2A interactions first-class verification objects (Allegrini et al., 15 Oct 2025).

The term is also used in a looser, application-specific sense for multi-step evidence-based checking of model outputs. UrduFactCheck describes “agentic model checking” as a modular fact-checking workflow with specialized components such as ClaimProcessor, QueryGenerator, Retriever, Translators, and Verifier, together with a thresholded retrieval controller that escalates from monolingual to translation-based evidence when $p$9. This use is verification-oriented, but it is closer to tool-augmented fact checking than to classical temporal-logic model checking (Ahmad et al., 21 May 2025).

5. Solver-backed verification of generated code

A particularly narrow and recent use of the term appears in bounded verification of LLM-generated systems code. “Agentic Model Checking” is defined there as a paradigm coupling LLM agents to a bounded model checking backend under the principle “agents propose, solvers verify.” LLM agents infer per-function specifications from caller context, select relevant checks, classify counterexamples, and propose refinements, while CBMC or Kani discharges all soundness-relevant decisions (Sun et al., 20 May 2026).

This formulation rests on three commitments. First, specifications are inferred top-down from caller context in a restricted DSL that maps deterministically to backend assume/assert primitives. Second, verification is compositional: each function is checked in isolation against its inferred preconditions, postconditions, and optional functional_spec, with callees replaced by postcondition-constrained stubs. Third, counterexamples are not reported directly; they pass through a validation pipeline consisting of input reachability, callee feasibility, dynamic replay, and realism audit, with spurious or artifact-driven failures routed into a refinement loop guarded by deterministic soundness checks (Sun et al., 20 May 2026).

The approach is instantiated in BMC-Agent, with CBMC for C and Kani for Rust. The harness synthesizer emits backend assumptions for preconditions, calls the target function, and asserts postconditions and functional specifications; old(EXPR) is mapped to backend-specific pre-state snapshots. The optional functional layer lifts checking from panic-freeness to behavioral faithfulness, catching bugs that defensive specifications would miss, such as the align_up functional-correctness violation reported in the Rust compiler case study (Sun et al., 20 May 2026).

Empirically, this strand reports 62 confirmed real bugs across LLM-generated kernel and compiler code and mature OSS-Fuzz-hardened libraries: 34 in VibeOS, 2 in jq, 1 in the out-of-tree r8125 driver, and 25 in the Rust compiler project CCC. It also reports bounded clean verifications for several hardened parser surfaces, such as 15 of 24 OpenSSL ASN.1 leaf functions and all 54 of 54 functions in libxml2’s pattern.c (Sun et al., 20 May 2026). This usage of agentic model checking is therefore agentic in orchestration, but solver-centric in soundness.

6. Limits, controversies, and open directions

The strongest cautionary line in the literature argues that general verification of agentic behavior is fundamentally limited. In a very general agent model with histories $i$0 and policies $i$1, and for any non-trivial viable deontology $i$2, the decision problem “is $i$3?” is not computable. The argument proceeds via Rice’s theorem: the property of always preserving $i$4 is a non-trivial semantic property of computable partial functions. The same work argues that validation against physical-world outcomes is futile under general conditions, and that layered architectures which verify “ethical” or intentional subcomponents do not thereby verify actual actions or outcomes (Jilk, 2016).

Even within restricted settings, guarantees are often conditional. Epistemic CEGAR for dynamic access control is sound for ACTLK safety properties with finite, tree-like counterexamples, but liveness, fairness, common knowledge, and distributed knowledge remain outside the fully automated refinement result (Koleini et al., 2014). Runtime verification over agent traces is observational rather than exhaustive: absence of violations over sampled runs does not constitute a proof of correctness (Sheffler, 19 Aug 2025). Probabilistic runtime assurance depends on trace quality, abstraction quality, and recalibration under distribution shift (Koohestani et al., 30 Jan 2026). Solver-backed bounded verification is explicitly bounded and spec-relative; “verified clean” means no violation within unwind bound $i$5 under the inferred specification, not whole-program or unbounded completeness (Sun et al., 20 May 2026).

There is also a recurring semantic gap between internal correctness and external correctness. BDI and organization-aware verification can show that an agent never deliberately chooses what it believes unsafe, that it forms the right intentions, or that it satisfies encoded obligations (Dennis et al., 2013, Jensen, 2015). This suggests strong assurance about decision-making logic, but not by itself about physical outcomes, social consequences, or the adequacy of the underlying abstractions. The same issue appears in recent host-agent models, where liveness properties such as eventual response rely on fairness assumptions over RPC and JSON-RPC channels (Allegrini et al., 15 Oct 2025).

Open problems follow these fault lines. The literature explicitly identifies extensions to richer epistemics, liveness and fairness under abstraction-refinement, private observations, dynamic protocols, strategy logics for reconfigurable systems, probabilistic CEGAR, timing operators for runtime monitors, automated property mining, stronger treatments of specification correctness, and broader evaluation on historical bug corpora (Koleini et al., 2014, Chakraborty et al., 2022, Alrahman et al., 2022, Koohestani et al., 30 Jan 2026, Sheffler, 19 Aug 2025, Sun et al., 20 May 2026). Taken together, these directions indicate that agentic model checking is not converging toward a single canonical formalism; instead, it is developing as a verification perspective centered on agents, their information, their interactions, and the computational artifacts that surround them.