- The paper introduces Dynamic Probabilistic Assurance (DPA) to continuously verify AI agents through runtime monitoring and probabilistic model checking.
- It presents a middleware framework that abstracts observed agent actions into formal events and employs online learning to update a probabilistic model represented as an MDP.
- Validated on a RepairAgent use case, AgentGuard enhances reliability by quantifying success probabilities and expected execution times in dynamic environments.
AgentGuard: Runtime Verification of AI Agents
Introduction
"AgentGuard: Runtime Verification of AI Agents" (2509.23864) addresses the critical challenges posed by the unpredictability and emergent behaviors of agentic AI systems. Traditional verification methodologies are inadequate for these systems due to their stochastic nature; instead, a probabilistic assurance approach is necessary. The paper introduces AgentGuard, a framework that implements a novel paradigm called Dynamic Probabilistic Assurance (DPA). This framework provides continuous, real-time verification of AI agents through runtime monitoring, online learning, and probabilistic model checking.
Problem Statement
The transition from generative AI to agentic AI represents a significant paradigm shift. These AI agents operate autonomously by perceiving, reasoning, planning, and acting in their environment. Such autonomy introduces inherent unpredictability, rendering deterministic verification methods ineffective. The paper identifies key challenges faced by agentic systems: stochasticity, hallucinations, emergent behaviors, and new vulnerabilities. Traditional software verification fails to address these challenges because it assumes deterministic logic, highlighting the need for a probabilistic approach that provides assurances on the likelihood of success or failure.
Background
The paper situates AgentGuard within the broader research landscape of trustworthy AI, defining a "verification stack" that ranges from formal verification of neural networks to multi-agent collaboration frameworks. Formal verification at the level of neural networks is computationally infeasible for large models, and black-box testing or LLM-assisted formalization offers only limited assurance. The paper critiques current verification strategies that focus on process conformance but fail to analyze emergent probabilistic behavior, resulting in a gap that AgentGuard aims to fill.
The AgentGuard Framework
AgentGuard embodies the DPA paradigm, shifting verification to a continuous and adaptive process. The framework adopts runtime verification (RV) to evaluate execution traces and constructs a probabilistic model, in the form of a Markov Decision Process (MDP), to represent the agent's behavior. The architecture consists of components for trace monitoring, model learning, and probabilistic model checking. It operates as a middleware layer, abstracting observed agent actions into formal events and updating the MDP with online learning techniques. Probabilistic model checking is then employed to verify agent behavior against predefined properties.
Practical Implementation
The framework's architecture is described through its main components: the Trace Monitor and Event Abstraction, Online Model Learner, Probabilistic Model Checker, and Assurance Dashboard. Real-time interaction with agentic systems is managed through these components, with PVC providing quantitative verification and the assurance dashboard communicating feedback or triggering automated responses.
Validation: RepairAgent Use Case
AgentGuard is validated through its application to RepairAgent, an autonomous program repair agent. The framework captures and models RepairAgent's decision-making process, verifying quantitative properties like probability of success and expected completion time. This enhances RepairAgent's reliability, managing repair cycles and identifying failure probabilities. These insights offer practical tool integration and real-time performance feedback, demonstrating AgentGuard's efficacy.
Discussion
The paper discusses the DPA paradigm's viability through the POC framework and presents potential enhancements. Future research is encouraged to automate state abstraction and incorporate incremental verification, adapting to increasingly complex agent behaviors. The consideration of stochastic games and multi-agent scenarios is suggested for further development.
Conclusion
AgentGuard represents a substantial advancement in AI safety assurance, shifting verification from static analysis to continuous, dynamic assurance. Through integrating runtime verification and probabilistic model checking, the framework delivers live and adaptive guarantees for AI agents. This approach enhances agent reliability and transparency, paving the way for robust AI systems capable of operating safely in dynamic environments.
The findings suggest a significant potential for further research into automated state abstractions and more efficient verification algorithms to extend AgentGuard's applicability across various agentic AI implementations.