$α^3$-SecBench: A Large-Scale Evaluation Suite of Security, Resilience, and Trust for LLM-based UAV Agents over 6G Networks

Abstract: Autonomous unmanned aerial vehicle (UAV) systems are increasingly deployed in safety-critical, networked environments where they must operate reliably in the presence of malicious adversaries. While recent benchmarks have evaluated LLM-based UAV agents in reasoning, navigation, and efficiency, systematic assessment of security, resilience, and trust under adversarial conditions remains largely unexplored, particularly in emerging 6G-enabled settings. We introduce $α^{3}$-SecBench, the first large-scale evaluation suite for assessing the security-aware autonomy of LLM-based UAV agents under realistic adversarial interference. Building on multi-turn conversational UAV missions from $α^{3}$-Bench, the framework augments benign episodes with 20,000 validated security overlay attack scenarios targeting seven autonomy layers, including sensing, perception, planning, control, communication, edge/cloud infrastructure, and LLM reasoning. $α^{3}$-SecBench evaluates agents across three orthogonal dimensions: security (attack detection and vulnerability attribution), resilience (safe degradation behavior), and trust (policy-compliant tool usage). We evaluate 23 state-of-the-art LLMs from major industrial providers and leading AI labs using thousands of adversarially augmented UAV episodes sampled from a corpus of 113,475 missions spanning 175 threat types. While many models reliably detect anomalous behavior, effective mitigation, vulnerability attribution, and trustworthy control actions remain inconsistent. Normalized overall scores range from 12.9% to 57.1%, highlighting a significant gap between anomaly detection and security-aware autonomous decision-making. We release $α^{3}$-SecBench on GitHub: https://github.com/maferrag/AlphaSecBench

• The paper presents α³-SecBench, a formal framework evaluating security, resilience, and trust in LLM-driven UAV agents under adversarial 6G conditions.
• It employs a layered, episode-centric evaluation with structured threat overlays and multi-turn JSON protocols, yielding robust security scoring.
• Empirical findings reveal high anomaly detection but significant gaps in vulnerability attribution and safe tool usage among state-of-the-art models.

Large-Scale Security Evaluation of LLM-Based UAV Agents: An Analysis of $α^3$-SecBench

Introduction

The $α^3$-SecBench suite provides a formal, large-scale framework for evaluating the security, resilience, and trustworthiness of LLM-based UAV agents executing multi-turn missions over 6G networks under adversarial conditions. This work addresses a critical gap in UAV autonomy benchmarking: prior efforts target reasoning, navigation, or perception in benign environments, whereas $α^3$-SecBench is designed to robustly evaluate agent responses to structured, multi-layer adversarial attacks across the full autonomy stack. The evaluation pipeline orchestrates validated security overlays, attack taxonomy, mandatory safety constraints, and a unified scoring protocol capable of decomposing agent performance into orthogonal dimensions of security awareness, resilience, and trust.

Figure 1: Overview of the $α^3$-SecBench episode-centric security evaluation pipeline, illustrating security overlay injection, agent interaction, and scoring.

Threat Modeling and Security Overlay Construction

$α^3$-SecBench utilizes an episode-centric abstraction where each UAV mission is realized as a finite sequence of decision turns. Attacker capabilities ($L_0$–$L_3$), interfaces ($\Gamma$), and adversarial goals ($G$) are formally specified, modeling both cyber-physical and 6G network-embedded threats. Security overlays augment benign episodes with targeted adversarial events, strictly at the observation level. This ensures realistic partial observability and preserves simulation agnosticism.

Attacks are categorized across seven autonomy layers—Sensors, Perception, Planning, Control, Network/6G, Edge/Cloud, and LLM Agent—affecting 175 threat types. Annotation is performed using the Common Weakness Enumeration taxonomy, enabling hierarchy-aware vulnerability attribution. The overlay system explicitly decouples threat realization from agent code or environment logic, ensuring generality and reproducibility.

Agent Interaction and Multi-Turn Conversational Protocol

In the $α^3$-SecBench protocol, agents interact over structured dialogues, guided by strict JSON-only outputs specifying tool usage and security alert objects. Learning and reasoning are tested as agents must concurrently (a) identify anomalous symptoms, (b) attribute vulnerability classes, and (c) execute mandatory safe-degradation actions within rigid time-bounded decision windows.

Figure 2: The high-level AI agent conversation protocol for security-aware UAV autonomy, illustrating structured interaction among user, agent, and external tools.

Figure 3: Example multi-turn UAV autonomy episode over 6G networks with security overlay, highlighting attack onset, anomalous symptoms, repeated security alerts, and response actions under partial observability.

Evaluation Protocol and Scoring Framework

Agent performance is evaluated along three quantitative axes:

• Security: Detection rate and CWE-level attribution accuracy under adversarial symptoms.
• Resilience: Timely execution of safe-degradation actions within response windows following attack detection.
• Trust: Reliability in tool usage, penalizing hallucinations and unsafe calls during attacks.

A weighted aggregation produces interpretable episode- and model-level metrics. Scores are robust to reweighting, confirming the framework's ability to distinguish nuanced weaknesses in detection, mitigation, and reliable control. Attack overlays are validated with deterministic schema and semantic contracts. This approach enables systematic benchmarking of agent models from major enterprise and AI-first labs without requiring access to proprietary internals or unsafe guardrails.

Empirical Findings and Failure Analysis

Empirical evaluation across 23 state-of-the-art LLM agents shows normalized overall scores in the range $[0.129, 0.571]$. While the strongest models (Google Gemini-2.5-Flash, OpenAI GPT-5.2-Chat, Z-AI GLM-4.6) demonstrate near-perfect attack detection and high resilience, attribution accuracy (CWE classification) lags substantially behind recognition capability. A consistent gap emerges between anomaly awareness and robust, policy-compliant safe-degradation responses.

Detailed analysis reveals that attribute oscillation, redundant alerts, and conservative but incomplete mitigations are prevalent even among high-performance models. For example, only select models consistently avoid unsafe tool calls during active attacks, and hallucinated actions—invocations of tools not present in the episode’s tool surface—occur at non-negligible rates in many LLMs.

Vulnerability-centric analysis by CWE highlights that agents rapidly detect high-frequency weaknesses (CWE-345, CWE-285, CWE-20) but exhibit poor attribution latency and accuracy for diffuse categories. Detection is typically pattern-driven, whereas precise attribution requires cross-layer causal reasoning—an unresolved challenge.

Implications and Future Research

Practically, these results demonstrate that high-level reasoning or navigation proficiency does not imply security-aware autonomy. Agents with strong detection and efficient responses nonetheless exhibit marked limitations in attribution, multi-turn mitigation reasoning, and robust tool use. The layered taxonomy and overlay mechanism in $α^3$-SecBench expose cross-domain vulnerabilities and failure modes that are invisible under conventional task-based benchmarks.

Theoretically, $α^3$-SecBench establishes methodologically sound protocols for benchmarking heterogenous LLMs under structured adversarial stress in multi-agent and networked systems. It enables reproducibility and cross-comparability between models, attack classes, and mitigation strategies. Major implications include the necessity for explicit safety-alignment objectives in LLM training and multi-layer causal reasoning capabilities in agentic deployments.

Future explorations should incorporate adversarial multi-agent coordination, adaptive mitigation learning, and end-to-end evaluation of dynamic countermeasures across both LLM and 6G stack layers. Coherent integration of security-oriented evaluation and alignment during training is a promising avenue to close the observed gaps in deployment-readiness for real-world UAV autonomy.

Conclusion

$α^3$-SecBench pioneers robust, scalable evaluation of security, resilience, and trust in LLM-driven autonomous UAV agents operating in adversarial 6G-enabled environments. Its layered, episode-centric architecture, principled threat taxonomy, and unified scoring protocol reveal critical deficiencies in current agents’ security reasoning and operational robustness under attack. The benchmark sets a new standard for deployment-oriented evaluation, guiding both future research on resilient agent architectures and practical development of trustworthy autonomous systems for safety-critical domains.