Semia: Auditing Hybrid Agent Skills

• Semia is a static auditor for LLM-driven agent skills that synthesizes a Skill Description Language (SDL) to bridge structured interfaces and natural language policies.
• It employs Constraint-Guided Representation Synthesis to generate SDL fact bases that enable deterministic Datalog reachability queries for security assessment.
• Empirical evaluation on 13,728 real-world skills shows Semia achieves high recall and accuracy in identifying semantic risks missed by conventional static analyzers.

Searching arXiv for the Semia paper and closely related agent-skill/security auditing work to ground the article in current literature. Searching arXiv for "Semia agent skills static auditor Datalog" and related terms. Semia is a static auditor for LLM-driven agent skills that targets a specific failure mode of contemporary agent ecosystems: the executable behavior of a skill is only partially specified in structured interfaces, while a substantial fraction of the effective policy remains embedded in natural-language prose that is reinterpreted probabilistically on each invocation. In the formulation of "Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis" (Wen et al., 1 May 2026), an agent skill is a hybrid artifact in which one half declares executable interfaces and the other prescribes when and how those interfaces should fire. Semia addresses the mismatch between conventional static analyzers, which parse the structured half but ignore the prose, and LLM-based tools, which can read the prose but cannot reproducibly prove that a tainted input reaches a high-impact sink. Its central mechanism is to lift each skill into the Skill Description Language (SDL), a Datalog fact base that captures LLM-triggered actions, prose-defined conditions, and human-in-the-loop checkpoints, after which security properties are reduced to Datalog reachability queries (Wen et al., 1 May 2026).

1. Agent skills as hybrid security artifacts

In Semia’s model, an agent skill is represented as a pair $s = (I, P)$, where $I$ is a structured declaration of the executable endpoints the agent may call and $P$ is a natural-language policy or instruction prescribing when and how those endpoints should fire (Wen et al., 1 May 2026). The structured component $I$ contains action names, parameter names, declared effects, and metadata; the prose component $P$ contains English sentences describing constraints, approval rules, budget limits, or related operational policies.

This decomposition is security-relevant because the prose is not merely documentation. It affects execution by specifying approval rules, exceptions, or behavioral constraints that may not be encoded in the structured interface. The paper’s illustrative “USD↔EUR signer” skill makes this explicit: one action performs net_read, another performs chain_write, while the prose states that external transfers require human approval and agent-to-agent transfers execute instantly (Wen et al., 1 May 2026). The example demonstrates how a natural-language exemption can create exploitable behavior unless the prose-level condition is statically checked against the executable surface.

A plausible implication is that Semia treats the skill as neither purely code nor purely prompt, but as a compound specification whose security properties emerge only after both halves are represented in a common formal substrate. This differs from scanners that rely only on signatures over interface declarations or code markers.

2. Skill Description Language (SDL)

Semia’s intermediate representation is the Skill Description Language, a compact relational schema that captures the security-relevant structure of a skill as a finite set of Datalog facts (Wen et al., 1 May 2026). SDL includes predicates for structural skeleton, data flow, annotations, secrets and barriers, documentation claims, and code-level markers.

The core schema reported in the paper includes the following predicates:

Group	Predicates	Role
Skeleton	skill(s), action(a,s), call(c,a,ε), call_next(c₁,c₂), action_param(a,name,var)	Declares skills, actions, calls, sequencing, and parameters
Data Flow	call_input(c,port,var), call_output(c,port,var)	Captures variable-level input/output edges
Annotations	action_trigger(a,τ), call_target(c,t), call_target_trusted(c), call_target_sensitive(c), call_target_unresolved(c)	Encodes trigger sources and target properties
Secrets & Barriers	secret(name), secret_var(var,name), secret_allowed(name,a), barrier_gate(a,gate), barrier_sanitize(var,point)	Models confidentiality and approval/sanitization controls
Documentation Claims	doc_claim(s,κ)	Represents claims such as read_only, local_only, no_network, no_exec
Code-Level Markers	call_body_obfuscated(c), call_body_encoded_binary(c), call_conditional(c₁,c₂)	Captures obfuscation and control-flow cues

Several SDL predicates are particularly central to Semia’s threat model. action_trigger(a,τ) records whether an action is activated by LLM, external, on_import, or on_install. barrier_gate(a,gate) records explicit human or policy barriers, where gate ∈ {human_approval, confirmation_prompt, allowlist, budget_limit}. secret_var(var,name) and secret_allowed(name,a) express confidentiality constraints. doc_claim(s,κ) enables checks for contradictions between declared behavior and operational behavior.

SDL is therefore not only a normalized representation of interfaces; it is a representation of latent semantics extracted from prose. This suggests that Semia’s novelty lies less in Datalog itself than in making prose-derived operational conditions queryable in the same relational space as interface structure and data flow.

3. Constraint-Guided Representation Synthesis

The main technical challenge in Semia is to synthesize an SDL fact base that is both structurally sound and semantically faithful to the original hybrid artifact. The paper addresses this with Constraint-Guided Representation Synthesis (CGRS), a bounded propose-verify-evaluate loop (Wen et al., 1 May 2026).

The paper restates Algorithm 1 as follows:

$I$9

A candidate fact base $p$ must satisfy structural constraints summarized as

$$I(p) \equiv I_{\text{ref}}(p) \wedge I_{\text{flow}}(p) \wedge I_{\text{auth}}(p).$$

The paper defines these components precisely. $I_{\text{ref}}$ requires that every symbol—such as an action, variable, or parameter name—is declared, with unresolved targets as the only exception. $I_{\text{flow}}$ requires that every call_input(var) traces back either to an action parameter or to a prior call_output(var), ensuring that the data-flow graph is connected. $I_{\text{auth}}$ requires that each barrier_gate(a,gate) refers to a declared action and a valid gate kind (Wen et al., 1 May 2026).

Semantic faithfulness is checked by verbalizing the candidate fact base back into English and measuring its distance to the original skill prose:

$I$0

The convergence requirement is

$I$1

The loop terminates with the first fact base $I$2 such that Validate(p*) = true and $I$3, or with $I$4 if no such fact base appears within the model-call budget (Wen et al., 1 May 2026). The paper also notes an implementation configuration with temperature=0 and a 5-round refinement budget.

This design is significant because it separates structural validity from semantic adequacy. A candidate SDL instance can be syntactically well-formed yet semantically lossy, and CGRS treats that as an explicit synthesis failure rather than an acceptable approximation.

4. Reduction to Datalog reachability

Once Semia has produced a faithful SDL fact base, security checking is reduced to Datalog reachability queries (Wen et al., 1 May 2026). The paper derives predicates for data-flow reachability, taint and secrecy propagation, and control-flow reachability.

The data-flow rules are given as:

$P$0

Integrity and confidentiality labels are propagated by rules including:

$P$1

Control-flow reachability is encoded as:

$P$2

The paper gives concrete detectors formulated over these derived predicates. The Missing Human Gate (MHG) detector identifies a high-privilege call with effect in {chain_write, proc_exec, code_eval, crypto_sign} that is reachable without any barrier_gate(a,human_approval):

$P$3

Sensitive Local Resource Overreach (SLRO) detects either direct flows from secret_var(v,sec) into an action not listed in secret_allowed(sec,a), or co-occurrence of a tainted variable and a secret in the same call when the target is not trusted (Wen et al., 1 May 2026).

The paper further states that many additional detectors—Unsanitized Context Ingestion, Implicit Egress Channels, Shadow Credentials, Behavior-Claim Contradiction, Obfuscation, Hardcoded C2, and Dormant Malicious Payload—are defined similarly as compact reachability or pattern queries over the derived predicates. This suggests a modular analysis architecture in which the synthesis problem is amortized across a family of detectors.

5. Empirical evaluation

Semia is evaluated on 13 728 real-world skills crawled from public registries in the OpenClaw ecosystem (Wen et al., 1 May 2026). Of these, 10 853 had VirusTotal scans, forming the VT-Dataset. The paper also reports a stratified random sample of 541 skills (5 % per size stratum), hand-labeled by two authors (Cohen’s κ=0.83). Within that labeled sample, 301 skills (55.6 %) carry ≥1 semantic risk and 240 skills are clean (Wen et al., 1 May 2026).

The evaluation uses standard classification metrics:

• Precision: $I$5
• Recall: $I$6
• $I$7

On the 541-skill sample, Semia is compared against VirusTotal and ClawScan. The reported effectiveness is as follows:

System	Precision	Recall	F1
VirusTotal	89.1%	13.6%	23.6%
ClawScan	73.2%	52.6%	61.2%
Semia	84.5%	97.7%	90.6%

The paper also reports per-category recall with prevalence counts in parentheses: MHG (278) 97.1 %, UCI (97) 95.9 %, SLRO (37) 94.6 %, IEC (23) 95.7 %, UDS (14) 100 %, DEP (10) 100 %, and SC (6) 100 %, followed by the statement that all remaining categories also achieve 100 % (Wen et al., 1 May 2026). The summary claim is that Semia detects nearly every expert-labeled risk.

In addition to benchmarked performance, the paper reports 17 critical zero-day semantic risks confirmed by registry maintainers and responsibly disclosed. These include cases such as shadow credentials and behavior-claim mismatches (Wen et al., 1 May 2026).

These results position Semia as a high-recall auditor for semantic risks that are poorly captured by signature-based scanners. A plausible implication is that its false-negative profile is shaped more by synthesis fidelity and inlining coverage than by detector expressiveness, because once a faithful SDL is obtained the actual checks are deterministic reachability queries.

6. Scope, limitations, and significance

The paper identifies several caveats. First, there is corpus bias, because the study is focused on OpenClaw and other marketplaces may exhibit different distributions or sandboxing regimes (Wen et al., 1 May 2026). Second, in-lining misses external-only code (e.g., very large vendor docs), which can break reachability chains and produce residual false negatives. Third, MHG over-flags benign “writes” (e.g., calendar entries) that authors judged low-impact, producing false positives. Fourth, the fidelity metric $I$8 is coverage-based and may miss subtle semantic shifts. Fifth, LLM non-determinism remains a concern, although the implementation fixes temperature=0 and uses a 5-round refinement budget (Wen et al., 1 May 2026).

The paper outlines several extensions: enriching SDL with richer gate types such as context-dependent budget rules and workflow patterns including loops and async callbacks; learning to rank refinement hints from historical failures; integrating symbolic parsing of common YAML/JSON front matter to reduce reliance on LLM-generated structural hints; adapting the approach to other ecosystems such as ChatGPT plugins and OpenAPI annotations by customizing the skill-inlining frontend; and combining Semia with runtime enforcement mechanisms such as isolation and dynamic taint-tracking for defense-in-depth (Wen et al., 1 May 2026).

Semia’s broader significance lies in its formalization of a class of artifacts that are increasingly common in agentic systems: configuration packages whose operational semantics are split across typed interfaces and free-form language. The paper’s core claim is that security queries over such artifacts can be made reproducible once the prose is translated into an auditable relational representation. This suggests a general pattern for static analysis in LLM-mediated software stacks: rather than treating natural language as unanalyzable annotation, translate it into a constrained intermediate form and subject the result to deterministic analysis.

7. Conceptual position within agent-security analysis

Semia occupies a specific point in the design space between static analysis and LLM-based judgment. Conventional static analyzers can parse the structured half of a skill but ignore the prose; LLM-based tools can read the prose but cannot reproducibly prove end-to-end security properties such as whether tainted input reaches a high-impact sink (Wen et al., 1 May 2026). Semia’s synthesis-plus-reachability pipeline is intended to combine semantic coverage with proof-style reproducibility.

Its threat model centers on semantic risks that are induced by prose-defined conditions, approval rules, and exceptions. The paper enumerates examples including indirect injection, secret leakage, confused deputies, and unguarded sinks (Wen et al., 1 May 2026). In this sense, Semia is not merely a malware detector or signature scanner. It is a static auditor for policy-bearing artifacts whose vulnerabilities arise from the gap between what the prose says, what the interfaces permit, and what an LLM may probabilistically infer at runtime.

A plausible implication is that Semia’s principal contribution is representational: once hybrid skills are lifted into SDL with sufficient fidelity, a diverse set of semantic security properties becomes expressible as compact Datalog queries. The paper’s empirical results—all of them auditable over 13,728 real-world skills, with 97.7% recall and 90.6% F1 on expert labels—support the claim that this representation is sufficiently expressive for large-scale auditing in real-world marketplaces (Wen et al., 1 May 2026).