Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification

Abstract: LLM agents increasingly perform autonomous actions through external tools, leading to complex and evolving safety risks. However, existing safety testing targets expert-designed safety violations, and the corresponding outcomes are evaluated by hard-coded rules, making them costly to extend as agents evolve. To this end, we present Vera, an end-to-end automated safety testing framework that instantiates software engineering testing principles for non-deterministic agents through a three-stage, self-reinforcing pipeline. First, a literature-driven exploration continuously discovers and structures emerging risks into taxonomies of safety risks, attack methods, and tool execution environments. Second, combinatorial composition across taxonomy dimensions produces executable safety cases, each specifying a concrete safety goal, a programmatically constructed initial state, and a deterministic verification predicate grounded in observable artifacts. Third, adaptive execution runs heterogeneous agents in isolated sandboxes where a control agent steers multi-turn interaction based on runtime observations, while evidence-grounded verifiers judge outcomes from environment state and tool-call evidence rather than model self-report. We evaluate Vera on four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code), revealing substantial safety weaknesses, with average attack success rates reaching 93.9\% under multi-channel attacks; we also release Vera-Bench, comprising 1600 executable safety cases spanning 124 risk categories across three execution settings. These results indicate that modular, executable testing infrastructure is essential for rigorous and maintainable safety evaluation of rapidly evolving agentic systems at scale. The code is publicly available at https://github.com/Yunhao-Feng/Vera.

• The paper presents Vera, a modular framework that automates safety evaluation of LLM agents through dynamic risk discovery and adaptive execution.
• It details a methodology combining combinatorial test-case generation with sandboxed environments to systematically unearth agent vulnerabilities.
• Empirical results reveal high attack success rates and demonstrate the effectiveness of fine-tuned guard models in improving LLM agent safety.

End-to-End Safety Evaluation of LLM Agents: The Vera Framework

Introduction and Motivation

LLM agents, when endowed with tool-use capabilities, execute autonomous actions within external environments, leading to significant and evolving safety risks. Traditional safety evaluations—reliant on expert-crafted violations and brittle, hard-coded rules—lack scalability as agent capabilities and deployment contexts change. "Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification" (2607.01793) introduces Vera, an automated, modular framework that systematizes safety testing of LLM agents via literature-grounded risk discovery, combinatorial test-case generation, sand-boxed adaptive execution, and evidence-grounded outcome verification.

Vera Framework and Methodology

Vera operationalizes agent safety evaluation as a closed-loop, three-stage pipeline: (1) dynamic risk discovery and taxonomy construction; (2) compositional executable case generation; (3) adaptive execution with programmatic verification.

The general structure and modularity of the system are depicted below.

Figure 1: Overview of Vera’s pipeline, expanding literature-derived safety taxonomies and composing executable scenarios for adaptive agent testing and evidence-grounded outcome verification.

Continuous Risk Taxonomy Discovery

Vera bootstraps from a dynamic literature-driven tri-taxonomy: safety risks (e.g., credential leakage, code execution), attack methodologies (e.g., prompt injection, task decomposition), and execution environments (e.g., email, code hosting). Recursive exploration, leveraging a Summary Agent, structures and regularly updates taxonomy trees by parsing new research and operational feeds. The orthogonality and dynamism of this taxonomy enable wide and up-to-date coverage, described via over 124 risk, 77 attack-method, and 30 environment categories.

Combinatorial Executable Case Generation

By composing triples of leaf-level taxonomy elements, Vera automatically synthesizes concrete and verifiable safety goals. Each executable safety case consists of a specific goal (e.g., “exfiltrate credentials via prompt injection in code hosting”), a programmatically constructed initial environment, and a Python-based deterministic verifier anchored to observable effects rather than model self-report. Strict compatibility filtering ensures that ill-specified or non-executable cases are removed, preserving broad but valid coverage across the combinatorial space.

Adaptive, Replayable Sandboxed Execution

Vera evaluates agent frameworks—including OpenClaw, Hermes, Codex, and Claude Code—using a common adapter interface and isolated Dockerized sandboxes. Each scenario is tested in three settings: benign (legitimate task), single-channel (adversarial user), and multi-channel (adversarial tool results). An adaptive Control Agent observes and responds to runtime behavior, iteratively probing for vulnerabilities and adjusting strategies per interaction.

A hierarchical, evidence-grounded verifier processes sand-box logs (agent responses, tool-call traces, and persistent environment state) to confirm or refute the realization of safety goals, ensuring that only materially realized violations are recorded. Executions—along with full trajectories, control policies, and verifiers—are retained as replayable artifacts, facilitating forensic analysis and downstream research.

Empirical Analysis and Dataset Properties

The Vera-Bench benchmark corpus comprises 1,600 executable base scenarios, each instantiated under three threat models. Executions span a vast cross-product of taxonomies, capturing real-world risk spectrum and reflecting natural abundance variations in risk-environment-attack space.

Figure 2: Distribution of Vera executions across risk and environment, under benign, single-channel, and multi-channel attack settings.

Results demonstrate broad test-space coverage with stability across most taxonomy compositions. Average execution success rates (ESR) for attacks range from 79.0% to 95.3% across risk families and 77.0% to 88.0% across environments, indicating the composition pipeline's effectiveness in generating non-degenerate, meaningful test cases.

Operationally, most retained runs are tractable in length and resource cost. However, Vera accommodates edge cases requiring longer context windows or intensive tool interaction.

Figure 3: Distributions of input/output token use and tool-call count per retained Vera execution, showing operational tractability but with a pronounced right tail for more complex scenarios.

Agent Robustness and Vulnerability Patterns

Evaluation across agent frameworks surface marked differences:

• Attack success rates (ASR) are high: 90.6% (single-channel) and 93.9% (multi-channel) on average, highlighting severe susceptibility.
• Hermes and Claude Code are particularly vulnerable (up to 97.8% ASR multi-channel for Hermes), correlating with richer tool orchestration and context handling.
• Multi-channel injections offer modest but systematic incremental attack success (3.3 percentage points on average), but per-agent variances reveal tool-observation robustness differences that single-channel testing would miss.

The gap in ESR between benign (70.5%) and adversarial (90.6–93.9%) settings quantifies the impact and effectiveness of runtime-adaptive adversarial steering. Notably, higher task-completion and flexibility in agents directly correlate with greater vulnerability—capability and vulnerability are structurally aligned under Vera’s stress-testing regime.

Evidence-Grounded Verification and Dataset Characteristics

Vera’s verification pipeline prioritizes environmental state changes over tool-call logs, and only considers agent responses when these constitute direct violations. This verification hierarchy eliminates both false positives and negatives stemming from superficial text judgments or incomplete tool-trace analysis.

The dataset is not uniformly populated but reflects realistic variation in scenario frequency and operational context, affording both fine-grained forensic studies and statistical analyses.

Guard Model Transfer and Fine-Tuning

A downstream application involves training agent-guard models for automated safety classification. Off-the-shelf models, including LlamaGuard3 and AgentDoG, perform poorly on Vera (recall as low as 0.258), indicating limited transfer from prior benchmarks. However, Qwen3Guard fine-tuned on Vera-Bench achieves significant improvement: accuracy of 0.930, recall of 0.903, and F1 of 0.941—all materially exceeding base and competing guard models.

Figure 4: Accuracy, recall, and F1 of off-the-shelf and fine-tuned guard models on the Vera safety-classification task.

The fine-tuning process demonstrates stable, monotonic loss convergence, suggesting both the tractability and informativeness of Vera’s dataset for model calibration.

Figure 5: Training/evaluation loss curves for fine-tuned Qwen3Guard, indicating stable training dynamics and clear evaluation loss minima.

Crucially, the resulting guard model generalizes to external safety-classification benchmarks (e.g., R-Judge), surpassing baseline accuracy and recall, which evidences that Vera’s task design transfers beyond its own annotation and distribution conventions.

Implications and Prospects

Vera advances LLM agent safety evaluation by establishing a scalable, modular infrastructure capable of both surfacing critical system vulnerabilities and generating challenging, reusable test scenarios. Its implication is twofold: first, rigorous, environment-grounded testing—rather than static prompt-based red teaming—should become the standard for agentic safety assessment; second, adaptive, modular frameworks unlock rapid iteration and targeted defense evaluation, enabling safety research to keep pace with the accelerating complexity of agentic systems.

Forward developments will likely involve:

• Extension of runtime taxonomies to reflect emerging attack patterns from operational deployments and CVE databases.
• Integration with continuous deployment pipelines, providing always-current safety certification for evolving agent architectures.
• Utilization of Vera’s data and methodology for developing guard models resilient to both environment-specific and compositional attacks, with demonstrated transferability.

Conclusion

Vera (2607.01793) contributes a rigorously engineered, end-to-end framework for automated safety evaluation of LLM-enabled agents, leveraging dynamic taxonomy discovery, compositional executable scenarios, adaptive adversarial agents, and environment-state-grounded verification. Empirical findings—across diverse agent frameworks and under strong attack models—reveal that current agentic systems are broadly vulnerable, with high attack success rates. Fine-tuning guard models on Vera-derived data yields substantial performance gains and cross-benchmark generalization, underscoring Vera’s utility. Vera establishes a clear paradigm for organism-level safety testing infrastructures, an essential prerequisite for the safe deployment of autonomous LLM agents at scale.