VeriGuard: Verified Safety for LLM Agents

• VeriGuard is a formal framework that ensures LLM agent safety by synthesizing verified policies that strictly adhere to predefined safety constraints.
• It employs a dual-stage architecture with an exhaustive offline phase for intent clarification, policy synthesis, and formal verification followed by real-time online monitoring.
• Empirical evaluations show zero attack success rates and 100% accuracy in sensitive benchmarks, demonstrating the framework's effectiveness over traditional heuristic guardrails.

Searching arXiv for the target paper to ground the article in the primary source. VeriGuard is a framework for LLM-based agent safety that provides formal safety guarantees for agent actions through verified code generation and runtime enforcement. It is presented as the first end-to-end framework providing formal, provable guarantees that LLM agent actions comply with predefined safety constraints, with particular motivation from sensitive domains such as healthcare, where safety, security, and privacy failures can have direct operational consequences (Miculicich et al., 3 Oct 2025). Its central design is a dual-stage architecture that separates computationally expensive offline validation from lightweight online monitoring, so that a policy can be clarified, synthesized, tested, formally verified, and then enforced during live execution.

1. Problem setting and design objective

VeriGuard is motivated by the observation that autonomous AI agents in sensitive domains may deviate from user objectives, violate data handling policies, or be compromised by adversarial attacks. The framework addresses the problem of guaranteeing that an agent’s actions adhere to predefined safety constraints, a challenge that the paper states existing systems do not fully address (Miculicich et al., 3 Oct 2025).

The framework’s stated contribution is to shift agent safety from heuristic guardrails to correct-by-construction code guaranteed by formal methods. In this formulation, safety is not treated as a purely empirical property of a model’s behavior, but as compliance of a synthesized policy with an explicit specification. The policy is then used as a guard over proposed actions before execution.

A defining feature is the separation between an exhaustive offline stage and a lightweight online stage. The offline stage is responsible for clarifying user intent, establishing precise safety specifications, synthesizing a behavioral policy, and validating that policy by testing and formal verification. The online stage then acts as a runtime monitor that checks each proposed action against the pre-verified policy before execution (Miculicich et al., 3 Oct 2025). This separation is presented as the mechanism that makes formal guarantees practically applicable.

2. Dual-stage architecture

VeriGuard is organized as a two-phase safety pipeline: an offline stage for policy specification, synthesis, iterative formal verification, and refinement, followed by an online stage for lightweight, real-time runtime enforcement (Miculicich et al., 3 Oct 2025).

In the offline stage, the inputs are the user’s natural-language request $r$ and an agent specification $\mathcal{S}$ describing task details, I/O structure, and environment. The system first performs intent clarification and constraint extraction, then generates both code and a formal safety contract, and finally iterates through testing and static verification until the policy is deemed correct or a limit is reached.

In the online stage, the verified policy is attached to enforcement points in the agentic pipeline, such as before external tool or database access, API calls, and information dissemination. At runtime, an argument extraction module maps agent context to policy inputs, and the verified Python policy decides whether a proposed action is allowed or denied, returning a Boolean outcome together with justification (Miculicich et al., 3 Oct 2025).

This architecture is explicitly described as separating the exhaustive offline validation from the lightweight online monitoring. A plausible implication is that the framework is intended to preserve the latency profile of deployed agents while relocating formal analysis to a pre-deployment phase.

3. Intent clarification, policy synthesis, and iterative refinement

The first substantive component of the offline stage is intent clarification. VeriGuard uses LLMs to analyze potential ambiguities in the request and specification, elicit clarifying queries, and refine ambiguous intent into explicit requirements and operational assumptions. The paper denotes these ambiguity analyses as $V_a$, queries as $Q$, and the resulting requirements and assumptions as $R$ and $A$ (Miculicich et al., 3 Oct 2025). The output of this loop is a refined, unambiguous basis for formal policy construction.

Policy synthesis then produces three artifacts: an initial Python policy function $p_0$, a list of input arguments $\mathcal{P}_0$, and a set of verifiable constraints $C_0$. The generation functions are given as

$$G_0(r,\mathcal{S}) \rightarrow (p_0,\mathcal{P}_0), \qquad H_0(r,\mathcal{S}) \rightarrow C_0.$$

The paper states that prompts are designed to force separation of concerns and ensure strict adherence to security specification (Miculicich et al., 3 Oct 2025).

After synthesis, VeriGuard enters an iterative refinement loop driven by testing and formal verification. For testing, the system automatically generates unit tests for normal and edge cases in PyTest format. Policies are refined based on test outcomes, and errors, exceptions, or failures are used to guide LLM-based code correction. For formal verification, the framework uses static program verification, specifically Nagini, which is built atop Viper for Python (Miculicich et al., 3 Oct 2025).

The refinement process is also formalized: $\mathcal{S}$0 The loop continues until all tests pass and all formal proofs succeed, or until resource limits are reached (Miculicich et al., 3 Oct 2025).

4. Formal specification and verification model

VeriGuard’s verification layer is framed in terms of Hoare logic and static analysis. Safety contracts are translated into Hoare triples,

$\mathcal{S}$1

meaning that if the pre-condition $\mathcal{S}$2 holds, then after executing policy $\mathcal{S}$3, the post-condition $\mathcal{S}$4 must hold (Miculicich et al., 3 Oct 2025).

Policy compliance is expressed as

$\mathcal{S}$5

for all constraints $\mathcal{S}$6 (Miculicich et al., 3 Oct 2025). In practice, the constraints are embedded in Python using Nagini’s @requires and @ensures decorators. Static verification is then used to prove that the synthesized policy satisfies the formalized requirements.

The verifier does not simply return a binary pass or fail result. When verification fails, the system receives concrete error traces or counterexamples, and these are fed back into the refinement loop to guide further correction (Miculicich et al., 3 Oct 2025). This closes the loop between generation and proof, rather than treating verification as an external audit step.

A common misconception is to treat VeriGuard as a generic content moderation layer. The architecture described in the paper is narrower and more formal: it centers on synthesis of a Python guard policy plus a mirrored machine-verifiable logical specification, and on proving compliance of that policy before deployment (Miculicich et al., 3 Oct 2025). This suggests that its guarantee attaches to the verified policy and its enforcement points, not to arbitrary unstructured model behavior outside that policy interface.

5. Runtime monitoring and enforcement semantics

Once a policy has been verified offline, VeriGuard uses it as a runtime monitor in the online stage. The policy is placed at enforcement points in the agentic pipeline, including before tool access, database access, API calls, and information dissemination (Miculicich et al., 3 Oct 2025). Agent data $\mathcal{S}$7 is ingested, and an argument extraction module maps context to policy input types through

$\mathcal{S}$8

The verified Python policy then executes in the loop to determine whether the proposed action is allowed or denied (Miculicich et al., 3 Oct 2025).

The paper describes four enforcement strategies and one hybrid configuration:

• Task Termination: stop the agent entirely at the first violation.
• Action Blocking: deny only the specific unsafe action while allowing the agent to continue.
• Tool Execution Halt: block a particular tool call within an action and return a “no-op” to the agent.
• Collaborative Re-planning: notify the agent of the violation and prompt it to replan.
• Hybrid approach: Collaborative Re-Planning plus Tool Execution Halt, described as delivering the best security-utility tradeoff (Miculicich et al., 3 Oct 2025).

These strategies differ in how they trade maximal safety against continued task progress. Task Termination is described as maximal safety and minimal utility, whereas Collaborative Re-planning is said to optimize utility but with weaker guarantees (Miculicich et al., 3 Oct 2025). The hybrid CRP + TEH configuration is reported as the best practical balance.

This enforcement design indicates that VeriGuard is not restricted to a single operational semantics. Instead, the same verified policy can support multiple intervention modes, with the paper emphasizing fine-grained runtime decision points.

6. Benchmarks and empirical results

The paper evaluates VeriGuard on three benchmarks: Agent Security Bench (ASB), EICU-AC (Intensive Care Unit—Access Control), and Mind2Web-SC (Miculicich et al., 3 Oct 2025). These benchmarks cover adversarial agent attacks, healthcare access control, and web-agent safety rules, respectively.

Benchmark	Setting	Reported finding
ASB	Prompt injection, memory poisoning, backdoors	ASR reduced to near-zero (0%) across all attacks and models
EICU-AC	Fine-grained, role-based patient-data policies	100% accuracy, precision, and recall
Mind2Web-SC	Web-agent safety control rules	Highest recall and very high overall accuracy

On ASB, VeriGuard reduces Attack Success Rate to near-zero (0%) across all attacks and models, while maintaining high Task Success Rate (Miculicich et al., 3 Oct 2025). On EICU-AC, it achieves 100% accuracy, precision, and recall in enforcing strict medical database access policies, and the paper specifically connects this to HIPAA/GDPR compliance while stating that it outperforms GuardAgent, AGrail, and LLaMA-Guard (Miculicich et al., 3 Oct 2025). On Mind2Web-SC, the framework achieves the highest recall, blocking all policy violations, together with very high overall accuracy, and does so without requiring in-context learning or tailored policy structures (Miculicich et al., 3 Oct 2025).

The ablation studies are also part of the empirical argument. They show the incremental importance of policy synthesis, validation, and especially formal verification, with the paper stating that all are required to reach both zero successful attacks and minimal user disruption (Miculicich et al., 3 Oct 2025). This is important because it attributes performance not merely to runtime blocking, but to the full pipeline.

A plausible implication is that the formal verification component is not a marginal addition but a dominant factor in the reported safety gains, since the ablations highlight it as especially important.

7. Position within LLM agent safety

VeriGuard is positioned against systems based on reactive filtering and empirical guardrails. The paper’s own framing is that it replaces heuristic guardrails with a proactive, provably correct-by-construction approach (Miculicich et al., 3 Oct 2025). In that sense, it belongs to a line of work that applies formal methods to agentic systems rather than relying solely on classifier-based moderation or prompt-level restrictions.

Its practical significance derives from three connected properties stated in the paper. First, the framework automates intent clarification and constraint extraction from natural-language requests. Second, it generates both executable Python guard code and a mirrored formal specification. Third, it combines offline proof obligations with online runtime validation of each proposed action (Miculicich et al., 3 Oct 2025). Together, these elements define a high-assurance control layer for LLM agents.

The reported domains and benchmarks indicate that the framework is designed for settings where policy compliance is at least as important as raw task completion, especially healthcare and other contexts involving sensitive data. The paper’s summary states that only actions that cannot violate the contract are ever executed, with fine-grained runtime decision points (Miculicich et al., 3 Oct 2025). Within the scope of the verified policy and its enforcement points, this is the core claim of VeriGuard’s safety model.

This suggests a broader significance for agent engineering: safety specifications can be operationalized as verified executable policies rather than informal instructions. Whether that scales to more open-ended agent environments is not resolved in the provided material, but the framework establishes a concrete architecture for applying formal verification to LLM-based action selection in practice (Miculicich et al., 3 Oct 2025).