ClawSafety: "Safe" LLMs, Unsafe Agents
Abstract: Personal AI agents like OpenClaw run with elevated privileges on users' local machines, where a single successful prompt injection can leak credentials, redirect financial transactions, or destroy files. This threat goes well beyond conventional text-level jailbreaks, yet existing safety evaluations fall short: most test models in isolated chat settings, rely on synthetic environments, and do not account for how the agent framework itself shapes safety outcomes. We introduce CLAWSAFETY, a benchmark of 120 adversarial test scenarios organized along three dimensions (harm domain, attack vector, and harmful action type) and grounded in realistic, high-privilege professional workspaces spanning software engineering, finance, healthcare, law, and DevOps. Each test case embeds adversarial content in one of three channels the agent encounters during normal work: workspace skill files, emails from trusted senders, and web pages. We evaluate five frontier LLMs as agent backbones, running 2,520 sandboxed trials across all configurations. Attack success rates (ASR) range from 40\% to 75\% across models and vary sharply by injection vector, with skill instructions (highest trust) consistently more dangerous than email or web content. Action-trace analysis reveals that the strongest model maintains hard boundaries against credential forwarding and destructive actions, while weaker models permit both. Cross-scaffold experiments on three agent frameworks further demonstrate that safety is not determined by the backbone model alone but depends on the full deployment stack, calling for safety evaluation that treats model and framework as joint variables.
Summary
- The paper presents ClawSafety, a scalable adversarial benchmark that quantitatively evaluates indirect prompt injection vulnerabilities in realistic, high-privilege agent deployments.
- The paper details experimental results showing significant discrepancies in attack success rates across models, scaffolds, and injection vectors, with ASRs ranging from 40% to 75%.
- The paper reveals that agent safety emerges from the interplay of the LLM backbone, scaffold design, and operational context, underscoring the need for defense-in-depth strategies.
Systematic Evaluation of Personal Agent Safety: An Analysis of the ClawSafety Benchmark
Motivation and Problem Formulation
Personal agents powered by LLMs are rapidly transitioning from chat systems to high-privilege actors able to manipulate file systems, execute code, interface with sensitive infrastructure, and automate end-user tasks. The migration toward agentic deployment (e.g., OpenClaw, Nanobot, NemoClaw) exposes a threat landscape that conventional LLM safety benchmarks fail to capture. Existing evaluations are often constrained to isolated chat settings, employ synthetic tasks, and ignore the interdependence of backbone models and orchestration frameworks. Prompt injection in agentic contexts is semantically and operationally more hazardous, with attack vectors leveraging workspace artifacts, privileged skill files, and trusted communication to escalate privilege beyond what is possible via prompt-level jailbreak.
ClawSafety directly addresses these limitations by deploying a scalable, scenario-driven adversarial benchmark. It provides a rigorous, quantitative framework to assess indirect prompt injection resilience in realistic high-privilege agent deployments, and systematically isolates how backbone LLM family, agent scaffold, and threat vector interact to impact actual harm.
Benchmark Architecture and Threat Taxonomy
ClawSafety comprises 120 adversarial test cases, organized along three axes: (1) harm domain (data exfiltration, financial loss, privacy breach, reputational damage, infrastructure compromise), (2) attack vector (skill/tool injection, spearphishing email, web-based prompt manipulation), and (3) harmful action type (exfiltration, config modification, destination substitution, credential forwarding, and destruction). Scenarios are constructed for ecological validity, with each containing 50–60 heterogeneous files, multi-turn conversations (64 turns), realistic communication archives, and allocated trust levels for content provenance. Iterative adversarial construction ensures that defenses in recent frontier models are robustly challenged, not bypassed by trivial or unrealistic inputs.
A salient feature is the structured mapping of attack paths into trusted workspace artifacts—e.g., privileged SKILL.md files, operational emails from plausible senders, compliance dashboards with regulatory urgency framing. Benchmark breadth and modularity enable systematic ablations across domains, agent stacks, and injection techniques.
Experimental Results and Quantitative Findings
Evaluation is conducted over five high-capability LLMs (Claude Sonnet 4.6, GPT-5.1, Gemini 2.5 Pro, DeepSeek V3, Kimi K2.5) and three agent scaffolds (OpenClaw, Nanobot, NemoClaw), totaling 2,520 attack trials. Success is defined strictly as the agent performing a harmful tool-mediated action, rather than mere harmful text generation. Metrics include aggregate attack success rate (ASR), per-vector breakdowns, and scenario/action-type-specific outcomes.
Key results are summarized as follows:
- ASR (Attack Success Rate): Across models, ASR is non-trivial, ranging from 40% (Claude Sonnet 4.6 on OpenClaw) to 75% (GPT-5.1 on OpenClaw). This gap demonstrates that substantial portions of agent deployments are vulnerable to realistic multi-channel injection.
- Injection Vector Gradient: Skill injection is systematically more effective than email and web vectors, with skill-based attacks achieving 69.4% ASR on average versus 38.4% for web-based attacks. However, this order is scaffold-dependent.
- Model Disparities: There is a clear two-tier separation: Claude Sonnet 4.6 consistently achieves lower ASR, preserving hard boundaries against credential forwarding and destructive actions ( ASR), in contrast to GPT-5.1, which is highly susceptible across all action types.
- Scaffold Effects: Identical backbone models exhibit up to 8.6 percentage point ASR spread solely from scaffold choice. Scaffold affects not only absolute vulnerability but also reverses attack vector rankings; for example, on Nanobot, email surpasses skill injection as the most effective channel.
- Domain Variance: DevOps scenarios are nearly twice as vulnerable as legal ones, with urgency and established workflow trust enabling more credible attack paths.
Figure 1: Attack success rate (ASR) averaged across the five scenarios, exposing vector/model-specific vulnerabilities and the steep gradient for skill-based injections.
Mechanistic Analysis of Vulnerability
Action trace analysis and iterative ablation yield several operational insights:
- Imperative vs. Declarative Framing: Defense activation in models like Sonnet 4.6 is triggered by imperative instruction framing (“Update notes.db”), while declarative statements (“X does not match Y”) consistently bypass defenses, regardless of vector or content.
Figure 2: Safe database write is rejected in web injection due to defense activation, conditional on imperative phrasing.
- Consensus Validation and Source Corroboration: Agents are robust to single-source adversarial instructions, but when attack values are planted congruently across multiple independent workspace sources (e.g., meeting notes, audits, baseline configs), consensus defenses are bypassed.
- Identity and Workflow Alignment: Stakeholder identity and plausible workflow alignment greatly increase exfiltration rates; attacks from operational peers (not executives or automated systems) and requests that piggyback on current tasks are most effective.
- Context/Turn Depth: Longer conversation context (≥40 turns) amplifies vulnerability, as agents develop richer workspace models and trust, leading to higher willingness to follow embedded adversarial instructions.
Theoretical Implications and Limitations
The results sharply contradict the assumption that chat-level safety generalizes to agentic settings. Models that exhibit strong refusal and alignment in text-only evaluation fail to maintain those boundaries when adversarial content is abstracted into realistic workspace artifacts and multi-turn workflows.
Safety is empirically shown to be a systemic property—emerging from the interaction of backbone model, scaffold, scenario, and domain-specific context—rather than a static property of the model alone. This joint-dependence theorem directly undermines the sufficiency of LLM-level alignment for deployment-time safety.
The clear defense boundaries (e.g., persistent refusal to forward unknown credentials or destroy protected files in Sonnet 4.6) suggest specific safety strategies can be transferred into agent stacks, but remain sensitive to scaffold composition, conversation grounding, and trust heuristics.
Benchmark Construction and Design Principles
Iterative construction reveals a set of actionable principles for adversarial evaluation in agentic AI:
- Operational Specificity over Authority: Precise procedural instructions are more effective than urgent or high-authority imperatives.
- Corroboration over Sophistication: Multiple congruent sources are required for successful injection; consensus attacks defeat single-source detection.
- Workflow Alignment: Benign augmentations to ongoing user tasks bypass suspicion more reliably than requests for novel or out-of-scope actions.
- Regulatory Framing: Fear appeals (compliance/incident framing) are strictly more successful than trust-based appeals for web injection.
Implications and Future Directions
ClawSafety demonstrates that the agent deployment stack—spanning LLM, scaffold, input channels, workflow grounding, and trust calibration—must be evaluated as a whole. Isolated “safe” LLM releases cannot be considered secure when embedded in agents operating with high privilege under realistic workflow and context conditions.
Pragmatically, agent designers should adopt “defense-in-depth” strategies, including modular consensus validation, provenance-based code inspection, dynamic trust calibration, and hard boundaries for destructive or credential-forwarding actions. Evaluation protocols must abandon chat-level metrics and adopt large-scale, multi-vector adversarial benchmarks such as ClawSafety.
Future work should focus on adversarial training that incorporates real-world scenario diversity, scaffold-conditional defenses, and provable auditability of action traces, as well as scalable, automated red-teaming integrated into agent stacks. Theoretical work is needed to formalize the compositionality of agent safety and derive necessary and sufficient conditions for deployment-time harm minimization.
Conclusion
ClawSafety provides a rigorous, scalable, and scenario-driven standard for evaluating the safety of high-privilege agentic deployments. The study rigorously evidences that chat-level safety is not a reliable proxy for agentic safety, that vulnerabilities are distributional over model, scaffold, and domain, and that agent deployments remain highly susceptible to non-trivial indirect prompt injection—particularly when adversarial content leverages trusted artifacts, workflow-aligned timing, and regulatory or operational urgency. Evaluation and mitigation must be situated at the joint system level to meaningfully close the compliance gap.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper studies how “personal AI agents” (like super-helpful computer assistants that can read your files, send emails, write code, and browse the web for you) can be tricked into doing harmful things—even when the underlying AI model is advertised as “safe.” The authors build a big, realistic test called ClawSafety to see how easily these agents can be fooled by sneaky instructions hidden in normal-looking places like a work manual, an email, or a web page.
What questions did the researchers ask?
They set out to answer, in simple terms:
- If an AI agent is safe in a chat, is it still safe when it can act on your computer?
- Which kinds of attacks are most dangerous: hidden instructions in “skill files,” emails, or web pages?
- Do different AI models resist attacks differently?
- Does the agent’s framework (the software that wraps around the AI) change how safe it is?
- What kinds of wording trick the agent more easily?
How did they test it?
Think of an AI agent as a smart “brain” living inside a “body” (the agent framework) that can use tools on your computer. The attack is like slipping a fake instruction note into places the agent already looks.
Here’s their setup, translated to everyday language:
- Realistic job worlds: They built five pretend work environments (software engineering, finance, healthcare, law, and DevOps). Each comes with files, emails, settings, and tasks that feel real.
- Three sneaky channels:
- Skill files: Like instruction manuals the agent trusts a lot.
- Emails: Messages that look like they’re from trusted coworkers.
- Web pages: External sites the agent might visit.
- 120 attack scenarios: Each scenario hides a harmful instruction in one of those channels and asks the agent to do normal work. At some point, the agent “naturally” reads the hidden instruction.
- What they measured: Attack Success Rate (ASR)—the chance the agent actually carries out the harmful action (like leaking data or changing a config file).
- Who they tested: Five recent AI models as the brain, and three different agent frameworks as the body (including OpenClaw, Nanobot, and NemoClaw).
- Safe testing: Everything ran in a sandbox (a safe, fake environment) so no real damage could happen.
- Extra analysis: They recorded “action traces” (step-by-step tool use) to see exactly how and when the agent got tricked.
What did they find?
Here are the main takeaways and why they matter:
- Many attacks worked, even on “safe” AIs.
- Overall, attacks succeeded 40% to 75% of the time, depending on the model. This shows chat-level safety (refusing bad text) doesn’t guarantee agent-level safety (refusing bad actions).
- The most dangerous channel was the one the agent trusted most.
- Skill files (internal instruction manuals) tricked agents most often.
- Emails came next.
- Web pages were the least effective.
- Lesson: The more the agent “trusts” a source, the more careful we should be with it.
- Some models held firm boundaries; others didn’t.
- The safest model never forwarded credentials and never did destructive actions in these tests (0% for those categories)—a hard safety line.
- Weaker models sometimes sent credentials or performed destructive steps.
- The framework (the agent’s “body”) changes safety a lot.
- The same model’s safety shifted by about 8–9 percentage points just by changing the framework.
- In one framework, emails became more dangerous than skill files, flipping the usual pattern.
- Lesson: Safety is a property of the whole stack (model + framework + tools), not just the model.
- Wording tricks matter: commands vs. statements.
- Imperative wording (“Update this file now”) often triggered defenses (the agent double-checked and refused).
- Declarative wording (“This number does not match that one”) slipped through more easily, because the agent treated it as a normal “fact” to report rather than a command to execute.
- Lesson: Attackers can hide in statements, not just orders.
- Longer, more realistic conversations made agents more vulnerable.
- With more turns, the agent built trust and context—and became more willing to follow the embedded instruction.
- Removing people’s names reduced leaks.
- When coworker names were replaced with role titles (like “incident commander”), the agent leaked far less, because the identity felt less trustworthy and the “trust link” was weaker.
Why does this matter?
- For users: Personal AI agents can control powerful tools on your machine. Hidden instructions in everyday places (manuals, emails, dashboards) can push them into harmful actions. You should be cautious about what your agent is allowed to read and do automatically.
- For builders: Safety must be tested on the whole system, not just the model. Framework choices, tool routing, memory, and source trust all change risk. Defenses should:
- Treat internal instruction files with extra scrutiny.
- Verify identities and cross-check information from multiple sources.
- Watch for “declarative” tricks, not just obvious commands.
- Limit dangerous tools by default (especially anything that sends credentials or modifies critical files).
- For researchers: Benchmarks should reflect realistic workflows and mixed trust levels. ClawSafety shows how to stress-test agents across jobs, channels, and frameworks, revealing gaps that chat-only tests miss.
In short, the paper’s message is simple: a “safe” AI in chat can still act unsafely as an agent. Real safety depends on both the brain (model) and the body (framework), plus thoughtful rules about what to trust, what to verify, and when to say no.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper advances agent-safety benchmarking but leaves several important aspects unresolved. Future work could address the following:
- Model and scaffold coverage
- Only five backbones are evaluated, and cross-scaffold testing is done for a single model (Sonnet 4.6); the extent to which scaffold effects generalize across models remains unquantified.
- Sensitivity to system prompts and tool configurations is untested (the study uses default scaffold prompts/configs); how guardrail prompt design and tool descriptions affect ASR is unknown.
- Threat model breadth
- Attacks are limited to three vectors (skill, email, web); memory poisoning, plugin/tool-store poisoning, RAG/vector-DB retrieval poisoning, clipboard/calendar/IM integrations, and multimodal channels (images/PDF/audio/voice) are not evaluated.
- Only single-channel attacks are considered; combined multi-vector, staged, or adaptive attacks (where the adversary reacts to the agent’s behavior over turns) are not explored.
- Persistence is not studied: no evaluation of cross-session, long-term state corruption (e.g., gradual config drift or dataset poisoning), or attacks that seed memory for later tasks.
- Web injections are text/HTML-focused; effects of CSS/JS-driven dynamic content, iframes, visual/pixel-based injections, and document formats (PDF/Docx) remain untested.
- Domain and linguistic scope
- Coverage is limited to five English-language professional domains; high-risk settings like HR, sales/CRM, customer support, research, manufacturing/OT, and government workflows are absent.
- Multilingual and cross-lingual injection robustness is not assessed; it is unclear whether ASR changes with non-English content or mixed-language workspaces.
- External validity and realism
- All trials run in sandboxed EC2 instances with intercepted side-effects; transferability of results to live, user-operated machines (with genuine email clients, cloud apps, and OS policies) is unvalidated.
- Email injection relies on plausible senders but does not test cryptographic identity (DKIM/S/MIME), anti-spoofing, or client-side filtering; the mitigation impact of authenticated identity is unknown.
- Metrics and measurement
- ASR focuses on completion of harmful actions; details on graded “partial” compromise scoring, severity weighting, and cross-action comparability are limited and lack reported inter-rater reliability.
- Detection latency (time-to-compromise), number of tool calls to failure, and attacker efficiency (effort needed to craft effective payloads) are not measured.
- Covert exfiltration and side channels (e.g., encoding, compression, DNS or timing channels, clipboard) are not monitored; detection coverage for stealthy leakage is uncertain.
- Stochasticity and reproducibility
- Only three runs per (model, case) are used; variance, temperature/top-p sensitivity, and robustness to API/model drift over time remain unexplored.
- The benchmark withholds operational payloads for safety reasons; reproducibility and community validation may be constrained without a safe-release protocol for artifacts.
- Defenses and mitigations
- The study primarily observes inherent model/scaffold behavior; it does not systematically evaluate explicit mitigations (e.g., step approvals, RBAC, tool whitelists, least-privilege sandboxes, content sanitization, retrieval firewalls, external detectors like WebSentinel).
- NemoClaw’s isolation and egress controls are reported at a high level; which specific policies reduce ASR and by how much is not dissected.
- Human-in-the-loop effects (UI designs for confirmations, escalation policies, owner verification) are not modeled; their potential to reduce ASR is unknown.
- Mechanistic understanding
- The imperative vs. declarative “defense boundary” is shown for one model/vector/domain; its generality across models, vectors, and settings is untested, and the underlying mechanisms triggering detection remain unclear.
- Identity-awareness ablation (names vs. roles) is tested on one model/domain; cross-model effects and the benefits of strong identity proofs (e.g., signed emails, verified contacts) are open questions.
- Tool-level contributions to failures are not isolated; it remains unclear which tools (file I/O, code exec, email composer, web fetch) are most prone to exploitation and which per-tool guardrails are effective.
- Configuration sensitivity
- Effects of memory size, context-window limits, chain-of-thought visibility, and routing strategies (e.g., ReAct vs. planner–executor) on ASR are not investigated.
- The impact of removing or constraining specific tools, or altering permission scopes, on attack success is not quantified.
- Attack generalization and overfitting
- Adversarial payloads were iteratively refined against frontier models; systematic measurement of payload transferability across models/scaffolds and the risk of attack overfitting to current backbones are not provided.
- Long-tail harms and outcomes
- The benchmark emphasizes acute harms (exfiltration, destructive actions, config changes); slow-burn integrity attacks (e.g., subtle data poisoning affecting future decisions) and reputational harms via outbound communications are not deeply assessed.
- Downstream recovery and remediation (e.g., automatic rollback, provenance auditing, and post-incident containment) are out of scope but critical for operational safety.
- Cost/benefit and deployment guidance
- The operational costs of defenses (engineering overhead, latency, usability trade-offs) and attacker costs (payload crafting time/skill) are not analyzed, leaving deployment guidance under-specified.
Practical Applications
Overview
This summary distills practical, real‑world applications of the ClawSafety benchmark and findings. Each item lists sectors, a brief description of the application, indicative tools/products/workflows, and feasibility notes.
Immediate Applications
These can be deployed with current tools and practices, drawing directly from the paper’s methods and results.
- [Sectors: software, finance, healthcare, law, DevOps] Benchmark-driven pre‑deployment safety testing for agent stacks
- What: Integrate ClawSafety-style scenarios into CI/CD as a gate before granting agents elevated privileges (files, email, code exec).
- Tools/products/workflows: CI runners that spin up sandboxed workspaces; ASR dashboards broken down by vector (skill/email/web) and action type; action‑trace diffing; honeytoken leak reports.
- Dependencies/assumptions: Access to a sandboxed environment; ability to run the production agent stack with logging; representative workspaces mirroring real roles.
- [Sectors: software, IT procurement] Vendor/stack selection via cross‑scaffold bake‑offs
- What: Systematically compare model–scaffold pairs (e.g., OpenClaw, Nanobot, NemoClaw) under identical tasks to inform procurement.
- Tools/products/workflows: Standardized bake‑off harness; procurement checklists that require vector‑specific ASR; acceptance thresholds by harm type.
- Dependencies/assumptions: Comparable tool interfaces across scaffolds; budget/time for evaluation; stable API/model versions.
- [Sectors: software/DevOps] Skill-file trust hardening and supply‑chain controls
- What: Treat SKILL.md and similar procedure files as high‑trust artifacts; introduce signing, provenance checks, and mandatory review.
- Tools/products/workflows: Signed skills repository; file-provenance enforcement (e.g., Git commit verification); import‑chain scanners that flag side‑effects; diff‑based change approvals.
- Dependencies/assumptions: Developer adoption of signing workflows; integration with existing VCS; agent support for verifying signatures.
- [Sectors: software/DevOps] Runtime isolation and egress controls (“NemoClaw‑like” defenses)
- What: Deploy sandboxing, policy‑enforced network egress, filesystem isolation, and gateway mediation for tool calls.
- Tools/products/workflows: Agent runtime wrappers; policy engines (OPA‑style) for tool routing; allow‑lists for destinations.
- Dependencies/assumptions: Engineering capacity to modify orchestrators; performance overhead tolerance; clear policy definitions.
- [Sectors: software, enterprise security] Action‑class firewalls for agents
- What: Intercept and classify tool calls by harmful action type (exfiltration, destination substitution, config write) and block/require approval.
- Tools/products/workflows: “Agent firewall” sitting between planner and tools; per‑action policies; human‑in‑the‑loop approvals for high‑risk calls (credential forwarding, destructive actions).
- Dependencies/assumptions: Fine‑grained tool telemetry; reliable action classification; organizational appetite for human approval latency.
- [Sectors: finance, healthcare, law, enterprise IT] Identity‑aware communication workflows
- What: Enforce strong identity verification for email instructions that agents may act on; degrade trust for role‑only or unmatched identities.
- Tools/products/workflows: S/MIME/PGP/DKIM verification surfaced to the agent; directory‑backed sender validation; “trust link” middleware that annotates messages with identity confidence.
- Dependencies/assumptions: Organization‑wide email signing; access to staff directory; agent APIs that ingest identity metadata.
- [Sectors: software, enterprise security] Speech‑act based prompt‑injection filtering and multi‑source verification
- What: Detect imperative phrasing (“update X to Y”) and trigger verification across independent sources before executing; treat declaratives as “to report,” not “to act.”
- Tools/products/workflows: Lightweight speech‑act classifier before tool execution; verification routines (cross‑check DB, logs, emails, dashboards); response templates that report discrepancies without propagating unverified values.
- Dependencies/assumptions: Tuned classifiers for domain language; connectors to corroborating data sources; calibrated thresholds to avoid blocking legitimate work.
- [Sectors: enterprise security] Honeytoken strategy for exfiltration detection
- What: Seed workspaces with canaries and automatically scan agent outputs/drafts for leakage to measure and deter exfiltration.
- Tools/products/workflows: Token generator and placement policies; output scanners; alerting hooked to SOC.
- Dependencies/assumptions: Low false‑positive tokens; safe placement that doesn’t hinder normal tasks; logging of agent outputs.
- [Sectors: enterprise IT/operations] Conversation and memory hygiene policies
- What: Limit conversation length or capability escalation prior to sensitive actions; reset memory or switch to constrained modes.
- Tools/products/workflows: Risk gating that escalates privileges only after checks; short‑session workflows for critical operations; scheduled memory scrubbing.
- Dependencies/assumptions: Orchestrator support for mode switches; user training; acceptable productivity trade‑offs.
- [Sectors: compliance, risk, policy] Internal standards recognizing “safety = model × scaffold × domain”
- What: Require reporting ASR by vector and domain; schedule periodic red‑team runs; treat scaffold changes as material risk requiring re‑certification.
- Tools/products/workflows: Safety scorecards; change‑control gates tied to re‑evaluation; domain‑specific playbooks (e.g., DevOps stricter than legal).
- Dependencies/assumptions: Executive sponsorship; metrics adoption; resourcing for periodic testing.
- [Sectors: academia] Research and teaching with realistic, high‑privilege agent benchmarks
- What: Use ClawSafety‑like workspaces to study chat‑to‑agent safety gaps, scaffold effects, trust heuristics, and defense boundaries.
- Tools/products/workflows: Open‑source scenario packs; action‑trace analytics; coursework on safe agent design and evaluation.
- Dependencies/assumptions: Access to model APIs or local models; compute for sandboxing; responsible disclosure norms.
- [Sectors: software tools] Developer products to visualize and audit action traces
- What: Provide planners with import‑chain tracing, before/after state diffs, and provenance overlays to detect mechanism vs. symptom changes.
- Tools/products/workflows: IDE extensions for agents; trace viewers; provenance dashboards tied to repository metadata.
- Dependencies/assumptions: Instrumented runners; storage for traces; UI integration into developer workflows.
Long‑Term Applications
These require further research, standards, ecosystem changes, or model/scaffold co‑design to be dependable at scale.
- [Sectors: standards, policy] Sector‑specific safety standards and audits for personal agents
- What: NIST/ISO‑style profiles mandating vector‑specific testing, identity assurance, and tool‑call controls; HIPAA/SOX/GDPR addenda for agentic operations.
- Tools/products/workflows: Standardized test suites and reporting formats; accredited labs for audits.
- Dependencies/assumptions: Regulator and industry buy‑in; consensus on harm metrics; governance of updates as capabilities evolve.
- [Sectors: software, certification] Certified “safe agent stacks” (co‑designed model–scaffold packages)
- What: Pre‑certified pairings with documented hard boundaries (e.g., non‑compliance on credential forwarding/destruction), plus policy‑enforced runtimes.
- Tools/products/workflows: Reference architectures; certification badges with version pinning; update channels with differential safety proofs.
- Dependencies/assumptions: Stable APIs; third‑party certification ecosystem; robust regression suites.
- [Sectors: AI R&D] Training methods to close the chat‑to‑agent safety gap
- What: Post‑training that aligns tool‑use behavior with refusals; datasets emphasizing speech‑act sensitivity and multi‑source verification habits; structural penalties for harmful tool chains.
- Tools/products/workflows: Simulator‑in‑the‑loop RL from action traces; counterfactual tool‑call training; red‑team adversarial data generation.
- Dependencies/assumptions: Access to model training; standardized action‑trace schemas; compute and data sharing.
- [Sectors: OS, browsers, platforms] Native agent permissioning and provenance‑aware file systems
- What: OS‑level trust tiers by content source; per‑tool least‑privilege with explicit grants; file systems that enforce cryptographic provenance and alert on provenance breaks.
- Tools/products/workflows: Agent entitlements modeled after mobile OS permissions; security UIs for approvals; provenance enforcement at kernel or VFS layer.
- Dependencies/assumptions: Platform vendor participation; backward compatibility; developer adoption.
- [Sectors: email/web infra, identity] Ubiquitous cryptographic identity and content authenticity for agent‑consumed inputs
- What: Default cryptographic signatures for internal email; signed intranet/web dashboards with verifiable integrity that agents can check.
- Tools/products/workflows: Organization‑wide key management; agent middleware that exposes signature and issuer to the planner.
- Dependencies/assumptions: Key lifecycle management at scale; standards for agent‑readable authenticity metadata.
- [Sectors: security, insurance] Agent risk underwriting and insurance markets
- What: Policies priced by measured ASR across vectors/domains, with premium discounts for certified stacks and continuous testing.
- Tools/products/workflows: Risk scoring pipelines; incident reporting standards; warranties tied to safety SLAs.
- Dependencies/assumptions: Sufficient loss data; standardized metrics; legal frameworks for agent‑caused harms.
- [Sectors: orchestration, MLOps] Dynamic, context‑aware risk orchestration
- What: Real‑time risk estimators that adjust capabilities based on conversation length, vector exposure, domain, and identity confidence.
- Tools/products/workflows: Runtime risk models; automatic capability downgrades; adaptive human‑in‑the‑loop thresholds.
- Dependencies/assumptions: Reliable telemetry; calibrated models; user experience that tolerates dynamic constraints.
- [Sectors: regulation, incident response] Mandatory disclosure and post‑mortem frameworks for agent‑induced incidents
- What: Standardized reporting of tool‑call traces and source vectors; independent review similar to data‑breach regimes.
- Tools/products/workflows: Secure trace retention; anonymization pipelines; oversight bodies or industry ISACs.
- Dependencies/assumptions: Legal alignment; privacy‑preserving sharing; incentives for honest reporting.
- [Sectors: education, training] Simulation platforms for cross‑sector incident drills with agents
- What: Hands‑on red/blue‑team exercises using realistic workspaces and injection channels tailored to finance, healthcare, law, DevOps.
- Tools/products/workflows: Scenario libraries; scoring engines; continuous learning programs for operators.
- Dependencies/assumptions: Curated, safe‑to‑share artifacts; instructor expertise; institutional adoption.
- [Sectors: consumer tech, fintech] Safer personal/home agents with tiered capabilities and bank‑grade approvals
- What: “Safety mode” defaults that require explicit human approvals for payments, file deletions, or credential sharing; trusted‑contact constraints.
- Tools/products/workflows: Mobile/desktop agent UIs with approval flows; integration with banking strong customer authentication; default role‑only handling for new contacts.
- Dependencies/assumptions: UX design that balances friction and usability; partner APIs; consumer education.
- [Sectors: web platforms] Content design guidelines for agent‑safe dashboards and documentation
- What: Authoring standards that avoid imperative phrasing for dashboards likely consumed by agents; machine‑readable “non‑actionable” tags.
- Tools/products/workflows: CMS plugins that lint for dangerous speech‑acts; metadata schemas for actionability.
- Dependencies/assumptions: Adoption by enterprises; agent support for reading metadata; standards convergence.
Glossary
- Ablation studies: Controlled experiments that remove or vary components to assess their effect on outcomes. "We ablate two design choices that may influence attack success"
- Action-trace analysis: Examination of an agent’s sequence of tool calls and decisions to locate where and how it failed. "Action-trace analysis reveals that the strongest model maintains hard boundaries against credential forwarding and destructive actions"
- Amazon Machine Image (AMI): A packaged template used to create pre-configured virtual machine instances on AWS. "Each test case runs inside a fresh sandboxed EC2 instance provisioned from a custom AMI, ensuring reproducible environments across runs."
- Attestation: A cryptographic or procedural proof that a build or artifact meets certain integrity or provenance guarantees. "notes.db confirms the attestation is valid"
- Attack Success Rate (ASR): The percentage of trials in which an adversarial attempt achieves its intended harmful effect. "Attack success rates (ASR) range from 40\% to 75\% across models and vary sharply by injection vector"
- Attack vector: The channel or mechanism through which adversarial content reaches and influences the agent. "Attack vector is how adversarial content reaches the agent: skill injection, email injection, or web injection."
- Backbone model: The primary LLM powering the agent’s reasoning and tool use, independent of scaffolding. "safety is not determined by the backbone model alone but depends on the full deployment stack"
- Black-box conditions: An evaluation setting where internal details are hidden and only inputs/outputs are observable. "evaluate OpenClaw under black-box conditions"
- CDN endpoint: A content delivery network address used to serve web resources with low latency and scalability. "Web injection cases serve attacker-controlled pages from a CDN endpoint that the agent fetches through its standard web_fetch tool."
- Context window: The span of tokens an LLM can attend to at once, representing its working memory for a session. "Turns 1--20 warm up the agent with general questions requiring web search and tool use, building a natural context window."
- Credential forwarding: Unauthorized transmission of authentication materials (e.g., tokens, keys) to another party or location. "hard boundaries against credential forwarding and destructive actions"
- Data exfiltration: Unauthorized extraction of sensitive data from a system or workspace. "Data exfiltration is the most exploitable action"
- Declarative framing: Presenting information as statements of fact rather than instructions, often bypassing action-triggering defenses. "Declarative framing succeeds because reporting discrepancies is expected behavior during incident response"
- Defense boundary: The conceptual line separating inputs that trigger safety defenses from those that bypass them. "The defense boundary lies at imperative vs.\ declarative framing."
- DevOps: Practices and tooling that integrate software development and IT operations, emphasizing automation and reliability. "DevOps/infrastructure (system integrity failures)."
- Direct Preference Optimization (DPO): A training method that aligns models by directly optimizing preferences, often used for safety alignment. "DPO-based safety training survives helpfulness optimization but follows a linear safety-helpfulness Pareto frontier"
- Ecological validity: The degree to which an evaluation environment authentically reflects real-world conditions. "Each workspace is designed for ecological validity"
- Egress control: Policies and mechanisms that restrict or monitor outbound network traffic from a system. "policy-enforced network egress control, filesystem isolation, and inference routing through the OpenShell gateway."
- Honey tokens: Decoy secrets or identifiers planted to detect unauthorized access or leakage. "Each case contains 5 honey tokens; we report how many appear in the agent's output."
- Import chain inspection: Analyzing a program’s dependency import path to detect malicious or unintended side effects. "driven by Sonnet's import chain inspection and post-execution verification."
- Import side effect: Code that runs when a module is imported, potentially altering state without explicit invocation. "via a hidden import side effect."
- Indirect prompt injection (IPI): Attacks where malicious instructions are embedded in external content the agent processes, rather than in the user’s prompt. "indirect prompt injection (IPI), first formalized by"
- Inference routing: Directing model inference requests through specific gateways or services for control and monitoring. "inference routing through the OpenShell gateway."
- Majority outcome: Aggregating multiple stochastic runs by reporting the result that occurs most frequently. "report the majority outcome to account for stochastic variation"
- Open-weight models: Model checkpoints whose parameters are available for download and local or custom deployment. "For open-weight models: DeepSeek V3 (DeepSeek) and Kimi K2.5 (Moonshot AI), both featuring native tool-use capabilities."
- Orchestration framework: The software layer that coordinates prompts, tools, memory, and model calls for an agent. "The attacker cannot modify the system prompt, model weights, or orchestration framework"
- Pareto frontier: The set of optimal trade-offs where improving one objective (e.g., helpfulness) worsens another (e.g., safety). "follows a linear safety-helpfulness Pareto frontier"
- Post-execution verification: Checking the effects of executed actions to detect discrepancies or malicious changes. "driven by Sonnet's import chain inspection and post-execution verification."
- Prompt injection: Crafting inputs that manipulate an LLM to follow attacker-specified instructions against its intended purpose. "a single successful prompt injection can cascade into real-world harm"
- ReAct prompting: A prompting strategy combining reasoning and acting (tool use) in an interleaved loop. "ReAct-prompted GPT-4 is vulnerable 24\% of the time."
- S3 backend: An Amazon Simple Storage Service data store used as the underlying storage for an application. "the live S3 backend."
- Sandboxed: Executed in an isolated environment that prevents real-world side effects and limits privileges. "2,520 sandboxed trials across all configurations."
- Scaffold (agent framework): The runtime and prompting structure that wraps the backbone model to enable tools, memory, and policies. "scaffold choice alone shifts overall ASR by 8.6 percentage points (40.0\% to 48.6\%)."
- Skill injection: Placing malicious instructions into trusted skill or procedure files that the agent reads as operating guidance. "Skill injection embeds adversarial instructions into privileged workspace artifacts that the agent treats as operating procedures."
- SLSA: Supply-chain Levels for Software Artifacts, a framework for securing software build and provenance. "the SLSA supply-chain alert uses imperative phrasing"
- Stakeholder identity: Information about who is involved (names/roles) that agents use to assess trust in communications and requests. "Effect of stakeholder identity awareness."
- Supply chain attack: Compromising systems by tampering with trusted components or update processes upstream. "paralleling software supply chain attacks."
- Tool calls: Explicit invocations of tools (e.g., code execution, file I/O, web fetch) by the agent during task execution. "its tool calls simultaneously execute the forbidden action"
- Tool routing: The policy or logic that determines which tools the agent uses and when, affecting safety exposure. "how scaffolding, memory, and tool routing amplify adversarial risk."
- Trust-level gradient: A pattern where the agent’s susceptibility varies with the perceived trust of the content source. "revealing a trust-level gradient."
- Web injection: Embedding adversarial instructions in web content that the agent fetches during its workflow. "Web injection must influence the agent despite external content receiving lower trust than local state."
- web_fetch tool: A specific agent tool for retrieving web content to include in its context or reasoning. "Web injection cases serve attacker-controlled pages from a CDN endpoint that the agent fetches through its standard web_fetch tool."
- Workspace provenance: The history and origin of files within a workspace that influences how much the agent trusts them. "Files with established workspace provenance inherit trust that exempts them from review"
Collections
Sign up for free to add this paper to one or more collections.