---
title: 'VeriGuard: Verified Safety for LLM Agents'
url: https://www.emergentmind.com/topics/veriguard
type: topic
---

# VeriGuard: Verified Safety for LLM Agents

Searching arXiv for the target paper to ground the article in the primary source.
VeriGuard is a framework for LLM-based agent safety that provides formal safety guarantees for agent actions through verified code generation and runtime enforcement. It is presented as the first end-to-end framework providing formal, provable guarantees that LLM agent actions comply with predefined safety constraints, with particular motivation from sensitive domains such as healthcare, where safety, security, and privacy failures can have direct operational consequences [2510.05156]. Its central design is a dual-stage architecture that separates computationally expensive offline validation from lightweight online monitoring, so that a policy can be clarified, synthesized, tested, formally verified, and then enforced during live execution.

## 1. Problem setting and design objective

VeriGuard is motivated by the observation that autonomous AI agents in sensitive domains may deviate from user objectives, violate data handling policies, or be compromised by adversarial attacks. The framework addresses the problem of guaranteeing that an agent’s actions adhere to predefined safety constraints, a challenge that the paper states existing systems do not fully address [2510.05156].

The framework’s stated contribution is to shift agent safety from heuristic guardrails to correct-by-construction code guaranteed by formal methods. In this formulation, safety is not treated as a purely empirical property of a model’s behavior, but as compliance of a synthesized policy with an explicit specification. The policy is then used as a guard over proposed actions before execution.

A defining feature is the separation between an exhaustive offline stage and a lightweight online stage. The offline stage is responsible for clarifying user intent, establishing precise safety specifications, synthesizing a behavioral policy, and validating that policy by testing and formal verification. The online stage then acts as a runtime monitor that checks each proposed action against the pre-verified policy before execution [2510.05156]. This separation is presented as the mechanism that makes formal guarantees practically applicable.

## 2. Dual-stage architecture

VeriGuard is organized as a two-phase safety pipeline: an offline stage for policy specification, synthesis, iterative formal verification, and refinement, followed by an online stage for lightweight, real-time runtime enforcement [2510.05156].

In the offline stage, the inputs are the user’s natural-language request $r$ and an agent specification $\mathcal{S}$ describing task details, I/O structure, and environment. The system first performs intent clarification and constraint extraction, then generates both code and a formal safety contract, and finally iterates through testing and static verification until the policy is deemed correct or a limit is reached.

In the online stage, the verified policy is attached to enforcement points in the agentic pipeline, such as before external tool or database access, API calls, and information dissemination. At runtime, an argument extraction module maps agent context to policy inputs, and the verified Python policy decides whether a proposed action is allowed or denied, returning a Boolean outcome together with justification [2510.05156].

This architecture is explicitly described as separating the exhaustive offline validation from the lightweight online monitoring. A plausible implication is that the framework is intended to preserve the latency profile of deployed agents while relocating formal analysis to a pre-deployment phase.

## 3. Intent clarification, policy synthesis, and iterative refinement

The first substantive component of the offline stage is intent clarification. VeriGuard uses LLMs to analyze potential ambiguities in the request and specification, elicit clarifying queries, and refine ambiguous intent into explicit requirements and operational assumptions. The paper denotes these ambiguity analyses as $V_a$, queries as $Q$, and the resulting requirements and assumptions as $R$ and $A$ [2510.05156]. The output of this loop is a refined, unambiguous basis for formal policy construction.

Policy synthesis then produces three artifacts: an initial Python policy function $p_0$, a list of input arguments $\mathcal{P}_0$, and a set of verifiable constraints $C_0$. The generation functions are given as
\[
G_0(r,\mathcal{S}) \rightarrow (p_0,\mathcal{P}_0), \qquad
H_0(r,\mathcal{S}) \rightarrow C_0.
\]
The paper states that prompts are designed to force separation of concerns and ensure strict adherence to security specification [2510.05156].

After synthesis, VeriGuard enters an iterative refinement loop driven by testing and formal verification. For testing, the system automatically generates unit tests for normal and edge cases in PyTest format. Policies are refined based on test outcomes, and errors, exceptions, or failures are used to guide LLM-based code correction. For formal verification, the framework uses static program verification, specifically Nagini, which is built atop Viper for Python [2510.05156].

The refinement process is also formalized:
\[
G_t(r,\mathcal{S},R,A,e,p_{t-1}) \rightarrow (p_t,\mathcal{P}_t), \qquad
H_t(r,\mathcal{S},R,A,p_{t-1}) \rightarrow C_t.
\]
The loop continues until all tests pass and all formal proofs succeed, or until resource limits are reached [2510.05156].

## 4. Formal specification and verification model

VeriGuard’s verification layer is framed in terms of Hoare logic and static analysis. Safety contracts are translated into Hoare triples,
\[
\{C_{\text{pre}}\}\ p\ \{C_{\text{post}}\},
\]
meaning that if the pre-condition $C_{\text{pre}}$ holds, then after executing policy $p$, the post-condition $C_{\text{post}}$ must hold [2510.05156].

Policy compliance is expressed as
\[
p \models C,
\]
for all constraints $c \in C$ [2510.05156]. In practice, the constraints are embedded in Python using Nagini’s `@requires` and `@ensures` decorators. Static verification is then used to prove that the synthesized policy satisfies the formalized requirements.

The verifier does not simply return a binary pass or fail result. When verification fails, the system receives concrete error traces or counterexamples, and these are fed back into the refinement loop to guide further correction [2510.05156]. This closes the loop between generation and proof, rather than treating verification as an external audit step.

A common misconception is to treat VeriGuard as a generic content moderation layer. The architecture described in the paper is narrower and more formal: it centers on synthesis of a Python guard policy plus a mirrored machine-verifiable logical specification, and on proving compliance of that policy before deployment [2510.05156]. This suggests that its guarantee attaches to the verified policy and its enforcement points, not to arbitrary unstructured model behavior outside that policy interface.

## 5. Runtime monitoring and enforcement semantics

Once a policy has been verified offline, VeriGuard uses it as a runtime monitor in the online stage. The policy is placed at enforcement points in the agentic pipeline, including before tool access, database access, API calls, and information dissemination [2510.05156]. Agent data $S$ is ingested, and an argument extraction module maps context to policy input types through
\[
f : S \rightarrow P.
\]
The verified Python policy then executes in the loop to determine whether the proposed action is allowed or denied [2510.05156].

The paper describes four enforcement strategies and one hybrid configuration:

- **Task Termination**: stop the agent entirely at the first violation.
- **Action Blocking**: deny only the specific unsafe action while allowing the agent to continue.
- **Tool Execution Halt**: block a particular tool call within an action and return a “no-op” to the agent.
- **Collaborative Re-planning**: notify the agent of the violation and prompt it to replan.
- **Hybrid approach**: Collaborative Re-Planning plus Tool Execution Halt, described as delivering the best security-utility tradeoff [2510.05156].

These strategies differ in how they trade maximal safety against continued task progress. Task Termination is described as maximal safety and minimal utility, whereas Collaborative Re-planning is said to optimize utility but with weaker guarantees [2510.05156]. The hybrid CRP + TEH configuration is reported as the best practical balance.

This enforcement design indicates that VeriGuard is not restricted to a single operational semantics. Instead, the same verified policy can support multiple intervention modes, with the paper emphasizing fine-grained runtime decision points.

## 6. Benchmarks and empirical results

The paper evaluates VeriGuard on three benchmarks: Agent Security Bench (ASB), EICU-AC (Intensive Care Unit—Access Control), and Mind2Web-SC [2510.05156]. These benchmarks cover adversarial agent attacks, healthcare access control, and web-agent safety rules, respectively.

| Benchmark | Setting | Reported finding |
|---|---|---|
| ASB | Prompt injection, memory poisoning, backdoors | ASR reduced to near-zero (0%) across all attacks and models |
| EICU-AC | Fine-grained, role-based patient-data policies | 100% accuracy, precision, and recall |
| Mind2Web-SC | Web-agent safety control rules | Highest recall and very high overall accuracy |

On ASB, VeriGuard reduces Attack Success Rate to near-zero (0%) across all attacks and models, while maintaining high Task Success Rate [2510.05156]. On EICU-AC, it achieves 100% accuracy, precision, and recall in enforcing strict medical database access policies, and the paper specifically connects this to HIPAA/GDPR compliance while stating that it outperforms GuardAgent, AGrail, and LLaMA-Guard [2510.05156]. On Mind2Web-SC, the framework achieves the highest recall, blocking all policy violations, together with very high overall accuracy, and does so without requiring in-context learning or tailored policy structures [2510.05156].

The ablation studies are also part of the empirical argument. They show the incremental importance of policy synthesis, validation, and especially formal verification, with the paper stating that all are required to reach both zero successful attacks and minimal user disruption [2510.05156]. This is important because it attributes performance not merely to runtime blocking, but to the full pipeline.

A plausible implication is that the formal verification component is not a marginal addition but a dominant factor in the reported safety gains, since the ablations highlight it as especially important.

## 7. Position within LLM agent safety

VeriGuard is positioned against systems based on reactive filtering and empirical guardrails. The paper’s own framing is that it replaces heuristic guardrails with a proactive, provably correct-by-construction approach [2510.05156]. In that sense, it belongs to a line of work that applies formal methods to agentic systems rather than relying solely on classifier-based moderation or prompt-level restrictions.

Its practical significance derives from three connected properties stated in the paper. First, the framework automates intent clarification and constraint extraction from natural-language requests. Second, it generates both executable Python guard code and a mirrored formal specification. Third, it combines offline proof obligations with online runtime validation of each proposed action [2510.05156]. Together, these elements define a high-assurance control layer for LLM agents.

The reported domains and benchmarks indicate that the framework is designed for settings where policy compliance is at least as important as raw task completion, especially healthcare and other contexts involving sensitive data. The paper’s summary states that only actions that cannot violate the contract are ever executed, with fine-grained runtime decision points [2510.05156]. Within the scope of the verified policy and its enforcement points, this is the core claim of VeriGuard’s safety model.

This suggests a broader significance for agent engineering: safety specifications can be operationalized as verified executable policies rather than informal instructions. Whether that scales to more open-ended agent environments is not resolved in the provided material, but the framework establishes a concrete architecture for applying formal verification to LLM-based action selection in practice [2510.05156].

Source: https://www.emergentmind.com/topics/veriguard