SecureAgentBench: LLM Agent Security Benchmark

• SecureAgentBench is a rigorous benchmarking framework that evaluates the functional performance and security resilience of large language model agents through realistic code challenges, attack simulations, and formal security games.
• It measures key metrics such as patch correctness, vulnerability re-introduction rates, and security probe outcomes to provide empirical standards for secure agent deployments.
• The framework advances research by integrating long-horizon adaptive attacks, agent skill evaluations, and specification-driven task synthesis to identify and mitigate emerging vulnerabilities.

SecureAgentBench is a rigorous benchmarking framework for evaluating the security properties and resilience of LLM agents in complex, realistic environments. Designed in response to the limitations of prior benchmarks, SecureAgentBench encompasses repository-scale code-editing challenges, simulated agent skill deployments, long-horizon attack scenarios, and specification-driven security evaluations. Its evaluation encompasses both functional and security dimensions, measuring not only whether agents achieve user goals but also whether they introduce, eliminate, or propagate vulnerabilities. The benchmark suite has played a central role in establishing empirical standards for robust and trustworthy LLM agent deployment in security-critical tasks (Chen et al., 26 Sep 2025, Wang et al., 28 Mar 2026, Alpay et al., 25 May 2026, Jiang et al., 18 Feb 2026, Cheng et al., 1 Jun 2026).

1. Secure Code Generation Benchmarking

SecureAgentBench offers a curated set of 105 repository-level coding tasks derived from real-world OSS-Fuzz vulnerabilities. Each task is instantiated by reverting a large (up to 36,000 files, 4.2M LOC) C/C++ project to the immediate pre-vulnerability state, then providing a security-neutral natural language requirement to the agent. The agent must produce a concrete, multi-file patch (mean: 1.9 files, 42 LOC) that remediates the vulnerability without introducing new risks.

Aligned contexts are created through a two-stage process: first, candidate vulnerability-inducing commits are identified via B-SZZ static analysis. Next, a dynamic Proof-of-Concept (PoC) exploit is used to disambiguate the true vulnerability introduction, requiring that the PoC fails on the immediate parent, triggers on the candidate, and is patched in the fixing commit. Resulting tasks cover 11 primary CWE families, dominated by heap overflows (47%), out-of-bounds reads (11%), and use of uninitialized variables (11%).

Evaluation proceeds as follows:

• Functional correctness: A patch is accepted if it passes all tests passed by the gold (developer) patch.
• Security: Each patch is exposed to the historical PoC (must not crash the agent-patched repo) and a static analysis scan (Semgrep). New warnings compared to baseline are flagged as “suspicious.”
• Six outcome categories are assigned: NO (no output), CE (compilation error), IC (incorrect), CV (correct + vulnerable), CS (correct + suspicious), C&S (correct and secure).

Aggregate performance metrics include correct-and-secure rate, functional correctness, vulnerability re-introduction rate, and the incident rate of new vulnerabilities.

Key results indicate state-of-the-art code agents attain only 15.2% correct-and-secure solutions on realistic tasks (best: SWE-agent with DeepSeek-V3.1 backbone). Even among functionally correct patches (29.8%), nearly half re-introduce vulnerabilities and one quarter introduce new “suspicious” vulnerabilities, raising the spectrum of CWEs to 14. Adding explicit “security reminders” to prompts had negligible effect on security (Chen et al., 26 Sep 2025).

2. Agent Skill Security and Utility Evaluation

The SkillTester component of SecureAgentBench evaluates the utility and security of LLM agent skills through paired baseline/with-skill execution and targeted security probes (Wang et al., 28 Mar 2026). Each skill is subjected to a suite of functional tasks with and without the skill enabled, measuring:

• Utility: Incremental rate of success, efficiency (token/time normalized), and credit assignment using formal mappings.
• Security: Controlled probes covering abnormal-behavior control, permission boundary violations, and sensitive data leakage (mirroring OWASP Agentic Top 10).

Raw task and probe outcomes are distilled into three metrics: Utility (0–100), Security (0–100), and a three-level security status (“Pass,” “Caution,” “Risky,” parameterized by an 80% caution threshold). Skills are cataloged with these scores and supporting logs, enabling systematic release gating and continuous assurance. Application to public skill repositories reveals significant portions (e.g., 13% with “Risky” scores) fail basic security checks.

3. Formal Security Games and Channel Closure Analysis

SecureAgentBench integrates formal security games targeting three primary agent risks (Alpay et al., 25 May 2026):

• Instruction-integrity: Indirect prompt-injection, measuring whether crafted context can induce forbidden output.
• Retrieval-confidentiality: RAG privacy, testing for leakage of unauthorized secrets in retrieval-augmented agent settings.
• Capability-integrity: Tool-use integrity, probing if forbidden capabilities (API/tool calls) can be triggered by adversarial observations.

A defense is modeled as a projection that removes unauthorized observation or action coordinates—enforcing “channel closure.” The benchmark precisely distinguishes security violation rates (attack success), closure rates, and utility impacts for six archetypal defenses, ranging from prompt annotation to classifier filtering, provenance-aware gating, least-privilege gating, and composition. Only channel-closing projections (not prompt annotations) consistently reduce empirical adversarial advantage to zero with acceptable impact on benign utility.

4. Long-Horizon and Adaptive Attack Evaluation

Extending security evaluation into long-horizon, agentic scenarios, SecureAgentBench aggregates five attack types—intent hijacking, tool chaining, task injection, objective drifting, memory poisoning—across 28 realistic, multi-turn environments adapted from WebShop, AgentDojo, SHADE-Arena, and others (Jiang et al., 18 Feb 2026).

Each test case specifies an environment, benign and malicious task/ground-truth action sequence, attacker adaptation bounds (turns, optimization), and risk category. Agents are measured using:

• Attack success rate (ASR) over all trials
• Turn-to-success (mean attack steps)
• Defense success rate (DSR) and per-defense evaluations

Baseline results indicate that even leading frontier agents (Qwen-3, GPT-4o, GPT-5.1, Gemini-3, Claude-4.5) remain highly vulnerable. For example, Qwen-3 yielded ASRs of 78–96% (depending on attack type); Claude-4.5 was the most resilient (28.9% overall). Single-turn defenses (self-reminder, Llama-Guard, repeated prompt, DeBERTa Detector) showed attack- and agent-specific efficacy, but no defense mitigated all long-horizon threats.

5. Spec-Driven Security Task Synthesis and Trajectory Evaluation

SeClaw extends SecureAgentBench with specification-driven task synthesis and trajectory-aware evaluation, enabling scalable, diverse, and controllable generation of security assessment scenarios (Cheng et al., 1 Jun 2026). Each task is specified in an abstract risk language covering resource, task, environment, and intrinsic categories, with structured constraints and evaluation targets.

The synthesized task pipeline involves translating the spec into a task prototype, instantiating Dockerized environments with configured tools, services, and layouts, and running iterative validation and refinement. The standardized execution sandbox logs all agent actions, tool calls, file/system/network events, and trajectories.

Evaluation detects unsafe actions not only at the response level but throughout the entire execution trajectory, applying precision/recall diagnostics for intermediate steps. Metrics include attack success rate, risk coverage, and the harmonic F_attack. Initial empirical results report coverage rates of 0.72 (versus ~0.45 for prior work), attack success spanning 0.30–0.65 depending on agent/model, and highest susceptibility in resource-poisoning cases.

6. Implications and Future Research Directions

SecureAgentBench demonstrates that current LLM agents, even when equipped with the best available models and scaffolds, consistently fail to reliably produce secure outcomes under realistic constraints. Notable observations include:

• Correct code generation does not imply security; new vulnerabilities are frequently introduced and known vulnerabilities often not patched.
• Prompt-based “security reminders” provide no robust security benefit.
• True security boundaries depend on enforcing input/output projections (channel closure), not on prompt annotation or classifier filtering alone.
• Long-horizon, adaptive adversaries remain a strong challenge; current agent architectures and defense strategies are insufficient.

Open research directions prompted by SecureAgentBench include:

• Model-level interventions (pre-training, fine-tuning with security datasets)
• Enhanced multi-modal and real environment integration
• Trajectory-based and not just endpoint-based evaluation schemes
• Dynamic, adversarial, and community-driven expansion of benchmarking tasks and threat models

SecureAgentBench has established a rigorous empirical baseline and methodology for security-centric agent benchmarking, and is expected to underpin future advances in secure, reliable, and auditable agentic systems (Chen et al., 26 Sep 2025, Alpay et al., 25 May 2026, Wang et al., 28 Mar 2026, Jiang et al., 18 Feb 2026, Cheng et al., 1 Jun 2026).