TriCEGAR: Agentic AI Runtime Verification

• TriCEGAR is an automated, trace-driven mechanism that builds behavioral models from execution traces to verify agentic AI systems.
• It integrates online Markov Decision Process construction with probabilistic model checking and likelihood-based anomaly detection for real-time assurance.
• Its dynamic counterexample-guided refinement eliminates manual state abstraction, supporting scalable and precise verification in complex environments.

TriCEGAR is an automated, trace-driven, counterexample-guided abstraction refinement mechanism for runtime verification (RV) of agentic AI systems operating in stochastic, high-dimensional environments. Developed to address the challenge of scalable, application-agnostic assurance for agents—such as LLM-based tool-using workflows—TriCEGAR eliminates the need for manually engineered state abstractions by constructing behavioral models from execution traces and dynamically refining them using decision trees and formal counterexamples. The system supports online Markov Decision Process (MDP) construction, probabilistic model checking (PMC), and likelihood-based anomaly detection, all integrated within standard agent frameworks (Koohestani et al., 30 Jan 2026).

1. Motivation and Context

Agentic AI systems, typified by LLM-based agents that invoke external tools, manage evolving representations, and adapt prompts or workflows over time, exhibit highly stochastic behavior due to nondeterministic tool outputs, environmental variation, and agent evolution. Traditional offline or deterministic verification is typically inadequate, as transient and probabilistic failures (e.g., alignment slips, stochastic loop exits) often manifest only at runtime. RV approaches provide empirical assurance over live traces, but depend critically on state abstraction: the reduction of complex, high-dimensional execution states into finite monitorable states suitable for model checking.

Earlier runtime verification frameworks, such as AgentGuard/Dynamic Probabilistic Assurance (DPA), required developers to explicitly define these abstractions, tying verification to brittle, application-specific heuristics and impeding adoption. TriCEGAR addresses this by inferring abstractions directly from traces, using a counterexample-guided approach to balance succinctness of the model with monitoring precision.

2. Architecture and Data Structures

TriCEGAR maintains and synchronizes three principal online data structures:

• Typed execution traces: Each agent tool call is intercepted, logging a concrete snapshot

$$s_c = \langle V_{\mathrm{goal}}, V_{\mathrm{check}}, V_{\mathrm{state}} \rangle,$$

where $V_{\mathrm{goal}}$ encodes immutable task context, $V_{\mathrm{check}}$ captures workflow flags, and $V_{\mathrm{state}}$ stores dynamic counters, statistics, and optionally text embeddings.

• Predicate-tree abstraction $T$: A binary decision tree in which internal nodes are predicates over trace variables (e.g., threshold tests, boolean flags), and leaves correspond to finite abstract states $S$. A mapping function $\alpha_T$ is used to assign each concrete state $s_c$ to its abstract state by traversing the predicates from root to leaf.
• Constructed MDP $\mathcal{M} = (S, A, P)$: The abstract states $S$ are the leaves of $T$, $A$ is the finite set of observed tool actions, and $P(s, a, s') = \frac{C(s, a, s')}{C(s, a)}$ gives empirically observed, unsmoothed transition probabilities. Transition counts $C(\cdot)$ are maintained online.

The system executes in iterative cycles: mapping new logged concrete states through $\alpha_T$, updating the transition count matrix, invoking model checking queries, and—if property violations are detected—triggering counterexample-guided abstraction refinement.

3. Predicate-Tree Learning and Counterexample-Guided Refinement

The predicate-tree abstraction is constructed using an ID3-style, top-down recursive splitting process:

1. Initial Tree Construction: For a batch of concrete states/actions $(B, A)$, candidate predicates $\pi$ are drawn from a domain-specific grammar, including numeric thresholds, boolean variables, collection-emptiness, and text-embedding hyperplanes. At each node:
   • Compute conditional entropy $H(A|B)$ on the empirical next-action distribution.
   • For each $\pi$, calculate the information gain (IG):

   $$\mathrm{IG}(B, \pi) = H(A | B) - \frac{|B_{\mathrm{true}}|}{|B|} H(A | B_{\mathrm{true}}) - \frac{|B_{\mathrm{false}}|}{|B|} H(A | B_{\mathrm{false}})$$

- Select the predicate $\pi^*$ with the highest IG, provided $\max \mathrm{IG} \geq \gamma$ (minimum gain threshold). Otherwise, terminate recursion with a leaf.

1. Refinement via Counterexamples: Following model checking, if the property (e.g., upper bound on failure probability) is violated, an abstract counterexample path $\tau$ is extracted. The system attempts to concretize $\tau$ in the trace trie. If $\tau$ is spurious (no match exists), the abstract state $s_i$ at the point of failure is refined: the set of concrete states mapping there are re-split using the IG criterion, and the refinements are propagated through the predicate tree, trace trie, and MDP structure. PMC is rerun after each refinement, continuing until either the property is satisfied or no further gainful splits are possible.

4. MDP Induction and Probabilistic Model Checking

The induced abstract MDP $\mathcal{M}$ is defined by:

• State space $S$: leaves of the current predicate tree.
• Action set $A$: observed tool types in the trace.
• Transition function: $P(s,a,s') = \frac{C(s,a,s')}{C(s,a)}$, $C(s,a)$ computed as a sum over outgoing transitions. No smoothing is applied; absent transitions are assigned zero probability.

Reachability queries are performed via external PMC engines (e.g., Storm). Key probability bounds include:

• $P_{\max}(\Diamond\,\mathsf{success})$: maximum probability of reaching success.
• $P_{\min}(\Diamond\,\mathsf{failure})$: minimum probability of reaching failure.

These extrema are computed over the full space of nondeterministic strategies $\sigma$ via standard Bellman-type value iteration. The probability bounds define risk envelopes for runtime guardrails; for instance, an alarm is raised if $P_{\min}(\Diamond\,\mathsf{failure})$ exceeds a policy-defined threshold $\tau_{\mathrm{fail}}$.

5. Likelihood-Based Anomaly Detection

TriCEGAR supports anomaly detection through model-induced run likelihoods. For an abstract run

$\rho = (s_0, a_0, s_1, \dots, a_{n-1}, s_n),$

the log-likelihood is computed as

$$\ell(\rho) = \sum_{i=0}^{n-1} \log P(s_i, a_i, s_{i+1}).$$

Offline, a reference distribution (mean $\mu$, standard deviation $\sigma$) is computed from historical run log-likelihoods. A run $\rho$ is flagged as anomalous if $\ell(\rho) < \mu - z_{1-\alpha} \sigma$. Online anomaly detection utilizes length-$k$ prefixes and checkpointed conditional means/standard deviations to account for drift with run length, raising alerts when

$\ell_k(\rho) < \mu_k - z_{1-\alpha} \sigma_k.$

6. Implementation and Empirical Results

TriCEGAR's implementation features:

• Trace storage: Typed records with primitive fields and optional embeddings are stored in both an append-only log and a prefix trie, with trie nodes maintaining pointers to concrete records and abstract state handles.
• Three-dimensional linkage: Each abstract state links to its predicate tree node, MDP vertex, and set of trie node endpoints, ensuring local consistency during predicate splits and abstraction refinement.
• Performance: Tree-splitting and MDP updates over several thousand transitions complete in under 100 ms, with PMC queries for small MDPs (≤50 states) completing in ≤200 ms. Trace-trie updates are $O(1)$ per transition.
• Case study: For a file-I/O agent, 1,000 baseline traces and 1,000 anomalous traces were generated. The initial abstraction (depth-4 tree on iteration count, filesWritten$>$0, lastFileRead path) robustly detected length-based anomalies. Overly general ratio-based anomalies were mitigated by a refinement split on filesWrittenCount$>$threshold. PMC bounds tracked long-term probabilities of success (tested$\,\rightarrow\,$committed) and failure (error or budget overrun), while low-likelihood runs were consistently flagged before commitment.

A summary of TriCEGAR's components and functions is presented below:

Component	Core Function	Update Frequency
Typed Trace Log/Trie	Capture agent lifecycle events and support concretization	Per tool invocation
Predicate Tree ($T$)	State abstraction, counterexample-driven refinement	Periodic/incremental
Abstracted MDP ($\mathcal{M}$)	Enables PMC and anomaly detection	After batch update

TriCEGAR is implemented natively in agent frameworks (e.g., Java/Kotlin with Koog-style interceptor), with model export in Storm's JSON and asynchronous PMC query execution.

7. Significance and Impact

TriCEGAR advances runtime assurance for agentic AI by decoupling verification from static, application-specific abstractions and enabling online, data-driven model construction directly from execution traces. This supports quantitative risk assessment, rapid turn-around of verification properties, and outlier detection within operational workflows. A plausible implication is that automated state abstraction may substantively lower barriers to deploying formal guardrails for adaptive, tool-using agents, especially in dynamic or evolving task environments (Koohestani et al., 30 Jan 2026).