--- title: 'Semia: Auditing Hybrid Agent Skills' url: https://www.emergentmind.com/topics/semia type: topic --- # Semia: Auditing Hybrid Agent Skills Searching arXiv for the Semia paper and closely related agent-skill/security auditing work to ground the article in current literature. Searching arXiv for "Semia agent skills static auditor Datalog" and related terms. Semia is a static auditor for LLM-driven agent skills that targets a specific failure mode of contemporary agent ecosystems: the executable behavior of a skill is only partially specified in structured interfaces, while a substantial fraction of the effective policy remains embedded in natural-language prose that is reinterpreted probabilistically on each invocation. In the formulation of "Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis" [2605.00314], an agent skill is a hybrid artifact in which one half declares executable interfaces and the other prescribes when and how those interfaces should fire. Semia addresses the mismatch between conventional static analyzers, which parse the structured half but ignore the prose, and LLM-based tools, which can read the prose but cannot reproducibly prove that a tainted input reaches a high-impact sink. Its central mechanism is to lift each skill into the Skill Description Language (SDL), a Datalog fact base that captures LLM-triggered actions, prose-defined conditions, and human-in-the-loop checkpoints, after which security properties are reduced to Datalog reachability queries [2605.00314]. ## 1. Agent skills as hybrid security artifacts In Semia’s model, an agent skill is represented as a pair $s = (I, P)$, where $I$ is a structured declaration of the executable endpoints the agent may call and $P$ is a natural-language policy or instruction prescribing when and how those endpoints should fire [2605.00314]. The structured component $I$ contains action names, parameter names, declared effects, and metadata; the prose component $P$ contains English sentences describing constraints, approval rules, budget limits, or related operational policies. This decomposition is security-relevant because the prose is not merely documentation. It affects execution by specifying approval rules, exceptions, or behavioral constraints that may not be encoded in the structured interface. The paper’s illustrative “USD↔EUR signer” skill makes this explicit: one action performs `net_read`, another performs `chain_write`, while the prose states that external transfers require human approval and agent-to-agent transfers execute instantly [2605.00314]. The example demonstrates how a natural-language exemption can create exploitable behavior unless the prose-level condition is statically checked against the executable surface. A plausible implication is that Semia treats the skill as neither purely code nor purely prompt, but as a compound specification whose security properties emerge only after both halves are represented in a common formal substrate. This differs from scanners that rely only on signatures over interface declarations or code markers. ## 2. Skill Description Language (SDL) Semia’s intermediate representation is the Skill Description Language, a compact relational schema that captures the security-relevant structure of a skill as a finite set of Datalog facts [2605.00314]. SDL includes predicates for structural skeleton, data flow, annotations, secrets and barriers, documentation claims, and code-level markers. The core schema reported in the paper includes the following predicates: | Group | Predicates | Role | |---|---|---| | Skeleton | `skill(s)`, `action(a,s)`, `call(c,a,ε)`, `call_next(c₁,c₂)`, `action_param(a,name,var)` | Declares skills, actions, calls, sequencing, and parameters | | Data Flow | `call_input(c,port,var)`, `call_output(c,port,var)` | Captures variable-level input/output edges | | Annotations | `action_trigger(a,τ)`, `call_target(c,t)`, `call_target_trusted(c)`, `call_target_sensitive(c)`, `call_target_unresolved(c)` | Encodes trigger sources and target properties | | Secrets & Barriers | `secret(name)`, `secret_var(var,name)`, `secret_allowed(name,a)`, `barrier_gate(a,gate)`, `barrier_sanitize(var,point)` | Models confidentiality and approval/sanitization controls | | Documentation Claims | `doc_claim(s,κ)` | Represents claims such as `read_only`, `local_only`, `no_network`, `no_exec` | | Code-Level Markers | `call_body_obfuscated(c)`, `call_body_encoded_binary(c)`, `call_conditional(c₁,c₂)` | Captures obfuscation and control-flow cues | Several SDL predicates are particularly central to Semia’s threat model. `action_trigger(a,τ)` records whether an action is activated by `llm`, `external`, `on_import`, or `on_install`. `barrier_gate(a,gate)` records explicit human or policy barriers, where `gate ∈ {human_approval, confirmation_prompt, allowlist, budget_limit}`. `secret_var(var,name)` and `secret_allowed(name,a)` express confidentiality constraints. `doc_claim(s,κ)` enables checks for contradictions between declared behavior and operational behavior. SDL is therefore not only a normalized representation of interfaces; it is a representation of latent semantics extracted from prose. This suggests that Semia’s novelty lies less in Datalog itself than in making prose-derived operational conditions queryable in the same relational space as interface structure and data flow. ## 3. Constraint-Guided Representation Synthesis The main technical challenge in Semia is to synthesize an SDL fact base that is both structurally sound and semantically faithful to the original hybrid artifact. The paper addresses this with Constraint-Guided Representation Synthesis (CGRS), a bounded propose-verify-evaluate loop [2605.00314]. The paper restates Algorithm 1 as follows: ```text Procedure Synthesize(s, δ): Input: skill s, fidelity threshold δ Output: SDL fact base p* or ⊥ (failure) p ← ⊥ // no candidate yet η ← ∅ // no hints while True: p ← Refine(s, p, η) if p = ⊥: return ⊥ // ran out of budget or model failure if not Validate(p): η ← Diagnose(s,p) // structural hints (missing var, broken ref, etc.) continue if d(s, Verbalize(p)) < δ: return p // candidate is structurally valid & semantically faithful η ← ∅ // abandon structural hints; ask for better overall fidelity ``` A candidate fact base $p$ must satisfy structural constraints summarized as $$ I(p) \equiv I_{\text{ref}}(p) \wedge I_{\text{flow}}(p) \wedge I_{\text{auth}}(p). $$ The paper defines these components precisely. $I_{\text{ref}}$ requires that every symbol—such as an action, variable, or parameter name—is declared, with unresolved targets as the only exception. $I_{\text{flow}}$ requires that every `call_input(var)` traces back either to an action parameter or to a prior `call_output(var)`, ensuring that the data-flow graph is connected. $I_{\text{auth}}$ requires that each `barrier_gate(a,gate)` refers to a declared action and a valid gate kind [2605.00314]. Semantic faithfulness is checked by verbalizing the candidate fact base back into English and measuring its distance to the original skill prose: $$ d(s, \text{Verbalize}(p)) = 1 - \frac{\# [\text{covered semantic units}]}{\# [\text{total semantic units}] }. $$ The convergence requirement is $$ d(s, \text{Verbalize}(p)) < \delta. $$ The loop terminates with the first fact base $p^*$ such that `Validate(p*) = true` and $d(s, \text{Verbalize}(p^*)) < \delta$, or with $\bot$ if no such fact base appears within the model-call budget [2605.00314]. The paper also notes an implementation configuration with `temperature=0` and a `5-round refinement budget`. This design is significant because it separates structural validity from semantic adequacy. A candidate SDL instance can be syntactically well-formed yet semantically lossy, and CGRS treats that as an explicit synthesis failure rather than an acceptable approximation. ## 4. Reduction to Datalog reachability Once Semia has produced a faithful SDL fact base, security checking is reduced to Datalog reachability queries [2605.00314]. The paper derives predicates for data-flow reachability, taint and secrecy propagation, and control-flow reachability. The data-flow rules are given as: ```prolog data_flows(X,Y) :- action_param(A,_,X), call(A_call,A,_), call_input(A_call,_,X). data_flows(I,O) :- call(C,_,_), call_input(C,_,I), call_output(C,_,O). data_flows(O,I) :- call_output(C1,_,O), call_next(C1,C2), call_input(C2,_,I). data_flows(X,Z) :- data_flows(X,Y), data_flows(Y,Z). ``` Integrity and confidentiality labels are propagated by rules including: ```prolog var_tainted(V) :- action_param(A,_,V), action_trigger(A,T), T≠llm. var_tainted(O) :- call(C,_,E), E∈{net_read,net_write}, ¬call_target_trusted(C), call_output(C,_,O). var_tainted(D) :- var_tainted(S), data_flows(S,D), ¬barrier_sanitize(D,_). var_secret(V) :- secret(N), secret_var(V,N). var_secret(D) :- var_secret(S), data_flows(S,D). ``` Control-flow reachability is encoded as: ```prolog call_reachable(C) :- call(C,_,_), ¬call_next(_,C), ¬call_conditional(_,C). call_reachable(C) :- call_reachable(P), call_next(P,C). call_reachable(C) :- call_reachable(P), call_conditional(P,C). call_unconditional(C) :- call(C,_,_), ¬call_next(_,C), ¬call_conditional(_,C). call_unconditional(C) :- call_unconditional(P), call_next(P,C). ``` The paper gives concrete detectors formulated over these derived predicates. The Missing Human Gate (MHG) detector identifies a high-privilege call with effect in `{chain_write, proc_exec, code_eval, crypto_sign}` that is reachable without any `barrier_gate(a,human_approval)`: ```prolog MHG(s,a,c) :– action(a,s), call(c,a,ε), ε∈{chain_write,proc_exec,code_eval,crypto_sign}, ¬barrier_gate(a,human_approval). ``` Sensitive Local Resource Overreach (SLRO) detects either direct flows from `secret_var(v,sec)` into an action not listed in `secret_allowed(sec,a)`, or co-occurrence of a tainted variable and a secret in the same call when the target is not trusted [2605.00314]. The paper further states that many additional detectors—Unsanitized Context Ingestion, Implicit Egress Channels, Shadow Credentials, Behavior-Claim Contradiction, Obfuscation, Hardcoded C2, and Dormant Malicious Payload—are defined similarly as compact reachability or pattern queries over the derived predicates. This suggests a modular analysis architecture in which the synthesis problem is amortized across a family of detectors. ## 5. Empirical evaluation Semia is evaluated on `13 728 real-world skills` crawled from public registries in the OpenClaw ecosystem [2605.00314]. Of these, `10 853 had VirusTotal scans`, forming the VT-Dataset. The paper also reports a `stratified random sample of 541 skills (5 % per size stratum), hand-labeled by two authors (Cohen’s κ=0.83)`. Within that labeled sample, `301 skills (55.6 %) carry ≥1 semantic risk` and `240 skills are clean` [2605.00314]. The evaluation uses standard classification metrics: - Precision: $P = TP/(TP+FP)$ - Recall: $R = TP/(TP+FN)$ - $F_1 = 2 \cdot P \cdot R /(P+R)$ On the 541-skill sample, Semia is compared against VirusTotal and ClawScan. The reported effectiveness is as follows: | System | Precision | Recall | F1 | |---|---:|---:|---:| | VirusTotal | 89.1% | 13.6% | 23.6% | | ClawScan | 73.2% | 52.6% | 61.2% | | Semia | 84.5% | 97.7% | 90.6% | The paper also reports per-category recall with prevalence counts in parentheses: `MHG (278) 97.1 %`, `UCI (97) 95.9 %`, `SLRO (37) 94.6 %`, `IEC (23) 95.7 %`, `UDS (14) 100 %`, `DEP (10) 100 %`, and `SC (6) 100 %`, followed by the statement that all remaining categories also achieve `100 %` [2605.00314]. The summary claim is that Semia detects nearly every expert-labeled risk. In addition to benchmarked performance, the paper reports `17 critical zero-day semantic risks` confirmed by registry maintainers and responsibly disclosed. These include cases such as `shadow credentials` and `behavior-claim mismatches` [2605.00314]. These results position Semia as a high-recall auditor for semantic risks that are poorly captured by signature-based scanners. A plausible implication is that its false-negative profile is shaped more by synthesis fidelity and inlining coverage than by detector expressiveness, because once a faithful SDL is obtained the actual checks are deterministic reachability queries. ## 6. Scope, limitations, and significance The paper identifies several caveats. First, there is `corpus bias`, because the study is focused on OpenClaw and other marketplaces may exhibit different distributions or sandboxing regimes [2605.00314]. Second, `in-lining misses external-only code (e.g., very large vendor docs)`, which can break reachability chains and produce residual false negatives. Third, `MHG over-flags benign “writes” (e.g., calendar entries)` that authors judged low-impact, producing false positives. Fourth, the fidelity metric $d(s,\text{Verbalize}(p))$ is `coverage-based and may miss subtle semantic shifts`. Fifth, `LLM non-determinism` remains a concern, although the implementation fixes `temperature=0` and uses a `5-round refinement budget` [2605.00314]. The paper outlines several extensions: enriching SDL with richer gate types such as context-dependent budget rules and workflow patterns including loops and async callbacks; learning to rank refinement hints from historical failures; integrating symbolic parsing of common YAML/JSON front matter to reduce reliance on LLM-generated structural hints; adapting the approach to other ecosystems such as ChatGPT plugins and OpenAPI annotations by customizing the skill-inlining frontend; and combining Semia with runtime enforcement mechanisms such as isolation and dynamic taint-tracking for defense-in-depth [2605.00314]. Semia’s broader significance lies in its formalization of a class of artifacts that are increasingly common in agentic systems: configuration packages whose operational semantics are split across typed interfaces and free-form language. The paper’s core claim is that security queries over such artifacts can be made reproducible once the prose is translated into an auditable relational representation. This suggests a general pattern for static analysis in LLM-mediated software stacks: rather than treating natural language as unanalyzable annotation, translate it into a constrained intermediate form and subject the result to deterministic analysis. ## 7. Conceptual position within agent-security analysis Semia occupies a specific point in the design space between static analysis and LLM-based judgment. Conventional static analyzers can parse the structured half of a skill but ignore the prose; LLM-based tools can read the prose but cannot reproducibly prove end-to-end security properties such as whether tainted input reaches a high-impact sink [2605.00314]. Semia’s synthesis-plus-reachability pipeline is intended to combine semantic coverage with proof-style reproducibility. Its threat model centers on semantic risks that are induced by prose-defined conditions, approval rules, and exceptions. The paper enumerates examples including `indirect injection`, `secret leakage`, `confused deputies`, and `unguarded sinks` [2605.00314]. In this sense, Semia is not merely a malware detector or signature scanner. It is a static auditor for policy-bearing artifacts whose vulnerabilities arise from the gap between what the prose says, what the interfaces permit, and what an LLM may probabilistically infer at runtime. A plausible implication is that Semia’s principal contribution is representational: once hybrid skills are lifted into SDL with sufficient fidelity, a diverse set of semantic security properties becomes expressible as compact Datalog queries. The paper’s empirical results—`all of them auditable` over `13,728 real-world skills`, with `97.7% recall` and `90.6% F1` on expert labels—support the claim that this representation is sufficiently expressive for large-scale auditing in real-world marketplaces [2605.00314]. Source: https://www.emergentmind.com/topics/semia