Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer

Abstract: Local LLM agents such as OpenClaw and Nanobot run on end-user machines and act on host resources - the shell, filesystem, browser, stored credentials, and messaging applications - through natural-language goals. These agents have become privileged software runtimes that mediate between user intent, model outputs, and host-level actions. Existing research characterizes the landscape through prompt injection, malicious skills, marketplace risks, or black-box evaluation of agents. But the implementation layer that performs this mediation, the prompt builder, parser, tool dispatcher, skill loader, memory writer, network client, and permission gate, has remained an unexamined safety boundary. To our knowledge, no prior work has examined the agent's source tree to audit these components for implementation-level security weaknesses. We present CLAWAUDIT, a static auditing framework for measuring vulnerability exposure in local LLM agent runtimes. CLAWAUDIT derives a five-category vulnerability taxonomy from STRIDE and develops custom static-analysis rules that target agent-specific patterns absent from established rule sets for vulnerability analysis. We instantiate the taxonomy in two backends, 47 Semgrep YAML rules and 30 CodeQL queries, and evaluate on OPENCLAWBENCH, a benchmark of 446 source-code-level advisories from the OpenClaw repository and split temporally into 229 rule-derivation (train) and 217 held-out (test) advisories. On the held-out test, CLAWAUDIT raises Semgrep recall from 21.7% (Pro baseline) to 66.8%, and CodeQL recall from 13.8% (security-extended) to 75.1%. Train/test gaps remain within 4 percentage points for all four configurations, indicating that the rules generalize to vulnerabilities unseen during rule writing. A preliminary live-code audit shows that these recall-oriented rules require manual triage, motivating semantic filtering before production deployment.

• The paper introduces ClawAudit, a framework employing a STRIDE-guided taxonomy to identify implementation vulnerabilities in local LLM agent runtimes.
• It integrates 47 Semgrep rules and 30 CodeQL queries, achieving significant recall improvements for detecting issues like path traversal and command injection.
• Despite high recall, the approach shows low precision in live scans, highlighting the need for enhanced semantic analysis to reduce triage costs.

Source-Code Vulnerability Auditing in Local LLM Agents: The ClawAudit Framework

Motivation: Local LLM Agents as Privileged Runtimes

The paper "Local LLM Agents as Vulnerable Runtimes: A Source-Code Audit of the Agent Runtime Layer" (2606.21071) addresses a critical and previously unexamined security boundary in the ecosystem of local LLM agents. Unlike developer frameworks, local agents (OpenClaw, Nanobot, PicoClaw) are shipped as privileged, installable executables with direct access to a user's shell, filesystem, credentials, browser, and messaging applications. These agents mediate between user intent, model output, and host-level actions via components such as prompt builders, parsers, tool dispatchers, skill loaders, memory writers, and network clients. Prior research on agent safety has predominantly focused on prompt-injection and marketplace-level risks or performed black-box behavioral evaluations, rarely auditing the source-code layer that enforces these mediations.

STRIDE-Derived Taxonomy of Runtime Vulnerabilities

The authors formulate a novel taxonomy of implementation-level vulnerabilities targeting the agent runtime by mapping the STRIDE threat model to five categories:

• CAT-1: Prompt Handling – Vulnerabilities from unsafe prompt construction and memory contamination (e.g., uncontrolled text interpolation).
• CAT-2: Tool/Skill Execution – Flaws in tool invocation or skill loading (e.g., path traversal, command injection).
• CAT-3: Permission & Isolation – Failures in sandboxing, resource scoping, or confinement of actions.
• CAT-4: Network & Communication – Network-related issues like SSRF and credential leakage.
• CAT-5: Authentication & Authorization – Inadequacies in permission gating, interface exposure, or route access control.

This taxonomy is instantiated in both the Semgrep and CodeQL static analysis backends, each encoding domain-specific agent patterns unaddressed by generic rule sets.

Figure 1: Overview of ClawAudit’s taxonomy-to-rule-to-benchmark pipeline; the framework derives a runtime vulnerability taxonomy, encodes agent-specific static analysis rules, and benchmarks detection against source-code advisories.

ClawAudit: Rule Construction and Evaluation Methodology

ClawAudit is a static auditing framework that operationalizes the above taxonomy using a curated set of domain-specific static analysis rules:

• 47 customized Semgrep YAML rules encoding syntactic patterns over ASTs, utilizing regular expressions and contextual pattern exclusion.
• 30 CodeQL queries implementing declarative predicates and bounded interprocedural reasoning to detect unsafe absence of guards.

For empirical evaluation, the authors introduce OpenClawBench—a benchmark of 446 OpenClaw source-level advisories (CVEs/GHSAs), temporally divided into 229 rule-derivation (train) and 217 held-out (test) advisories. The evaluation employs recall-oriented attribution by detecting whether any rule fires within a file modified by the advisory’s patch.

Empirical Results: Recall Improvements and Generalization

Strong empirical results are reported:

• On the held-out test set, Semgrep recall increases from 21.7% (Pro baseline) to 66.8% and CodeQL recall from 13.8% (security-extended baseline) to 75.1% when augmented with ClawAudit.
• Train/test recall gaps remain within 4 percentage points across all configurations, indicating effective generalization to unseen vulnerabilities and minimal overfitting.
• Per-severity and per-CWE stratification reveal that substantive gains occur for agent-relevant classes (symlink following, path traversal, TOCTOU, command injection, SSRF, access control).
• For instance, symlink-following moves from 0–20% baseline recall to 86.7% with ClawAudit, and path traversal achieves 84% recall.

However, precision in live HEAD scans is low (12% TP rate in manual audit), highlighting that current rules are recall-oriented and impose significant triage costs without semantic filtering.

Discussion: Static Analysis Limitations and Future Directions

The authors emphasize:

• Generic static tools (Semgrep Pro, CodeQL security suite) miss most runtime-agent faults due to structural mismatch; rulesets focus on conventional web idioms rather than agent runtime operations.
• Static, syntactic rule-based detection is highly effective for vulnerabilities with recurring code shapes, but stalls on semantically defined weaknesses (incomplete denylists, information exposure, resource-ownership enforcement) where policy or sensitive-data semantics are external to code structure.
• Closing the residual detection gap necessitates semantic agent-runtime analysis—including specification mining, agent-custom taint tracking, policy inference, and potentially LLM-assisted rule synthesis.

Implications extend beyond OpenClaw: similar runtime architectures exist in all privileged local agents, making the five-category taxonomy broadly transferable. Neither prompt-level nor marketplace-level audits subsume runtime-level audits; systematic assurance must span all boundaries.

Practical and Theoretical Implications

Practically, ClawAudit operationalizes a systematic audit pipeline for agent runtimes, demonstrating that domain-specific static rules are essential for vulnerability coverage in LLM-driven agents. Theoretically, it reframes local agents as privileged runtimes demanding software assurance, setting foundational directions for agent safety research:

• Policy mining for authorization/isolation,
• Agent-specific taint analysis across prompt-tool boundaries,
• LLM-assisted rule synthesis for semantic predicates,
• Runtime instrumentation for contextual auditing.

These extensions foreshadow a hybrid syntactic-semantic audit stack, improving both precision and detection coverage for future agent architectures.

Conclusion

ClawAudit represents a comprehensive framework for source-level auditing of local LLM agent runtimes via a STRIDE-guided taxonomy and dual-backend static analysis. Substantial recall improvements over generic tool baselines validate the necessity of domain-specific rules and indicate generalization to unseen vulnerabilities. Nonetheless, semantic analysis remains critical for completeness, especially as agent runtimes become more privileged and complex. The practical artifact and OpenClawBench benchmark are released to support reproducibility and further research (2606.21071).