--- title: 'CLAWSAFETY: LLM Agent Security' url: https://www.emergentmind.com/topics/clawsafety type: topic --- # CLAWSAFETY: LLM Agent Security ClawSafety (CLAWSAFETY) is both a methodological framework and a series of empirical benchmarks for evaluating and enforcing safety in large language model (LLM)-powered agents—especially those, like OpenClaw, that possess high-privilege access to local environments and external services. Its scope encompasses architectural principles, attack taxonomies, lifecycle risk models, benchmark suites, and layered mitigation strategies, all designed to address the concrete system-level hazards introduced by tool-augmented, persistent, and extensible autonomous agents. CLAWSAFETY exposes not just the failure modes of individual models, but also the emergent security properties (or vulnerabilities) of agent frameworks when deployed in adversarial and realistic operational contexts [2604.01438][2603.12644][2606.30755][2603.13151]. ## 1. Systemic Threats in LLM Agent Frameworks The move from prompt-only LLM interfaces to agent frameworks with tool, file, and network access dramatically amplifies the attack surface. Whereas traditional LLM safety focuses on text-level refusals and alignment, CLAWSAFETY recognizes that agentized settings must account for hazards at seven distinct layers, including prompt injection, supply-chain compromise, context and memory poisoning, sequential tool attack chains, credential leakage, privilege escalation, and unintended destructive actions [2604.01438][2603.10387][2603.12644]. A formal threat model captures these risks by defining the agent as an autonomous loop operating over user goals $U$, mixed-trust inputs $I$ (web, email, workspace), persistent state $W$, installed extensions $E$, and system-level tools $T$, dispatching actions $A$ that can cause irreversible side effects. Adversaries may inject instructions through any input channel, poison agent memory, subvert supply chains, or exploit deployment misconfigurations. Layered risk taxonomies such as the Tri-Layered Model [2603.12644] and the CIK model (Capability, Identity, Knowledge) [2604.04759] provide systematic coverage of the sources of vulnerability. ## 2. Benchmarking Architectures and Empirical Assessments CLAWSAFETY benchmarks are characterized by their ecological validity, multi-channel attack vectors, and lifecycle-aware evaluation. Early benchmarks like ATBench-Claw [2604.14858] and ClawSafety [2604.01438] define taxonomies spanning harm domain, injection vector, and harmful action, embedding adversarial content in trusted channels such as skill files, emails, and web pages. Scenario designs typically establish long-lived, contextualized sessions that simulate genuine workflows before adversarial triggers are introduced. Attack Success Rate (ASR), defined as the fraction of trials in which an unauthorized, irreversible, or policy-violating effect is achieved, is the primary metric. Empirical findings demonstrate elevated ASRs even for frontier models in high-privilege scaffolds: in one study spanning 2,520 sandboxed trials, overall ASR ranged from 40% (Claude Sonnet 4.6) to 75% (GPT-5.1), with skill-injection vectors showing the highest success rates [2604.01438]. Layered analysis reveals that particular scaffolding choices (agent runtime, plugin orchestration, and extension load path) cause significant variance—sometimes exceeding the differences between LLM backbones themselves [2604.01438][2604.03131]. Trajectory-based audits [2602.14364][2604.06132] and state-diff evaluations in realistic productivity workspaces [2604.05172] further reveal that even agents with strong safety alignment are vulnerable to compound failures arising from trajectory-level ambiguity, context poisoning, and multi-tool escalation. ## 3. Formal Security Properties and Architectural Controls From a formal security perspective, CLAWSAFETY advances three core system invariants: integrity of external effects, confidentiality of sensitive values, and explicit, quantifiable declassification. SecureClaw [2606.09549] exemplifies a principled, dual-boundary enforcement architecture: - **Read boundary (plaintext confinement):** All accesses to sensitive values $v \in V$ are brokered by a trusted gateway, which replaces $v$ with an opaque handle $h(v) = \mathrm{HMAC}_{k_{\text{handle}}}(\cdot)$ and a bounded summary $D(v)$. The runtime only manipulates handles and summaries, never secrets directly. - **Write boundary (effect authorization):** All effectful actions $\alpha \in A$ must pass through a PREVIEW$\rightarrow$COMMIT protocol. Only the canonical request, as signed and authorized in the commit phase, can be executed, thwarting runtime-injected or mutated proposals. Formally, the integrity guarantee is $P_{\text{unauth}}\left[\text{commit}\right] \leq \text{Adv}_{\text{EUF-CMA}}(\mathrm{MAC}) + \varepsilon_{\text{bind}}(q)$, binding security guarantees to the underlying cryptographic primitives. Policies, attachable at both the runtime and effect execution boundaries, enable deny-aware recovery, fine-grained authorization, and bounded declassification channels. ## 4. Evaluation of Layered and Lifecycle Security Broad defense evaluation spans several mitigation types, from runtime policy enforcement and plugin/skill governance [ClawKeeper 2603.24414; SafeClaw-R 2603.28807], to in-process interception with hybrid risk accumulation and time-decaying thresholds (PRISM [2603.11853]), to external, tamper-evident audit chains and hot-reloadable configuration (AgentWall [2605.16265], PRISM). Three-layer models like ClawKeeper interleave: - **Skill-based protection:** Contextual policy injection at the instruction level, interpreted by the LLM for hard and soft constraints. - **Plugin-based protection:** Runtime configuration and tool invocation guards, behavioral anomaly detection, and anchoring of allowed actions. - **Watcher-based protection:** Decoupled, external monitoring and intervention, capable of pausing or halting agent execution on risk elevation. Evaluation demonstrates substantial, but incomplete, risk reduction. For example, empirical defense success rates for ClawKeeper reach 85–90% across OWASP-esque threat categories [2603.24414]. SafeClaw-R enforcement nodes, wrapping every functional skill in an execution graph, yield 95.2% accuracy in productivity benchmarks and 100% in code execution scenarios [2603.28807]. However, several studies highlight residual risk: file-system level protections often trade functionality for safety, and agent evolution (via persistent memory and code updates) reopens vulnerabilities unless explicitly governed [2604.04759][2606.30755]. ## 5. Challenges, Open Problems, and Directions for Future Hardening Despite layered mitigations, several systemic challenges persist: - **Amplification and Cascade:** Early-stage reconnaissance or prompt injection multiplies downstream risk, often by escalating through multi-tool attack chains and persistent memory poisoning. The probability of irreversible compromise is a non-linear function of early-stage leakage [2604.03131]. - **Evaluation-Governance Gap:** Many safety checks are brittle to adversarial reframing; trajectory-opaque or output-only scoring misses up to 44% of actual safety violations [2604.06132]. - **Framework Dependence:** Security properties depend as much on agent scaffolding and runtime orchestration (plugin interface, skill load audit, session state management) as on the LLM's own refusal behavior, necessitating evaluation of joint configurations [2604.01438][2606.30755][2603.13151]. - **Tradeoff Boundaries:** Strict policy or file-system protections block attacks but also significantly hinder legitimate agent evolution and personalization [2604.04759]. Utility-security tradeoffs must be carefully balanced. Research recommendations include institution of code signing and sandboxing for all executable skills, operationalization of immutable audit trails for agent memory, enforcement of differential update and approval channels for persistent state, and adoption of defense-in-depth architectures combining static, runtime, and externalized monitoring layers [2604.04759][2603.24414][2603.12644][2603.13151][2603.28807]. Formal verification of agent plans and end-to-end benchmarks for risk escalation, extension governance, and workflow security remain critical open areas [2605.25435][2604.14858]. ## 6. Synthesis: CLAWSAFETY as a Lifecycle Security Paradigm CLAWSAFETY advances the agent security field beyond isolated prompt-level refusal and static tool filtering, embedding security across the lifecycle of agent operation—from perception (input isolation, skill vetting), through reasoning and policy enforcement (dynamic intent verification, risk accumulation), to response, governance, and adaptive mitigation (audit chains, live threat feed integration, external Watchers). Benchmarks and architectures evaluated under this umbrella consistently reveal the necessity of cross-boundary, zero-trust execution policies, continuous runtime and state monitoring, and policies for human-in-the-loop escalation on irreversible or cross-boundary effects [2606.09549][2603.24414][2603.28807][2606.30755][2603.13151]. Empirical evidence shows that defense-in-depth, combined with continuous adaptation and auditability, is necessary but not sufficient; only coordinated architectural and governance-level advances can approach resilient, real-world deployment of agent frameworks. --- **References:** - "ClawSafety: 'Safe' LLMs, Unsafe Agents" [2604.01438] - "A Systematic Security Evaluation of OpenClaw and Its Variants" [2604.03131] - "SecureClaw: Clawing Back Control of LLM Agents" [2606.09549] - "ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces" [2604.05172] - "ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watchers" [2603.24414] - "SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistants" [2603.28807] - "Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens" [2606.30755] - "Defensible Design for OpenClaw: Securing Autonomous Tool-Invoking Agents" [2603.13151] - "Security of OpenClaw Agents: Fundamentals, Attacks, and Countermeasures" [2605.25435] - "OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agents" [2603.11853] - "ClawTrap: A MITM-Based Red-Teaming Framework for Real-World OpenClaw Security Evaluation" [2603.18762] - "Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents" [2604.06132] - "A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)" [2602.14364] - "Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw" [2604.04759] - "Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-CodeX" [2604.14858] - "AgentWall: A Runtime Safety Layer for Local AI Agents" [2605.16265] Source: https://www.emergentmind.com/topics/clawsafety