Formal Security Analysis of Agent Protocol Composition

Abstract: AI agent protocols define how agents use tools, delegate work, and coordinate across software systems, but their security requirements remain incomplete and inconsistently enforced across deployments. We present AgentThread, a source-linked framework for security assurance analysis of agent protocols, from specification text to running SDKs. AgentThread contributes a layered security scope, protocol-derived checks formalized as TLA+ invariants, and a two-phase checker that compiles protocol specifications into model-checkable models and replays executable counterexamples against real SDKs through protocol adapters. For each finding, AgentThread records the source text behind the check and separates violated protocol requirements from missing recommendations, hardening gaps, and unassigned cross-protocol responsibilities. Across five emerging agent protocols, AgentThread identifies 35 specification-level findings, supports them with 80 implementation tests against production SDKs and reference servers, and finds 30 additional failures that emerge only under protocol composition. We further show that only one protocol enforces a security-relevant control in practice and no protocol assigns enforcement for cross-protocol behavior. Insecurity in agent protocols is therefore not only a specification or implementation problem, but also a responsibility gap across protocols, SDKs, and deployments.

• The paper introduces AgentThread, a formal framework linking protocol specifications to formal TLA+ checks for exposing compositional security failures in agent systems.
• It systematically analyzes five protocol layers and uncovers 35 spec-level gaps, 30 composition failures, and numerous responsibility gaps.
• The study demonstrates that explicit composition contracts and bridge-level enforcement are crucial to securing multi-agent, tool-using deployments.

Formal Security Analysis of Agent Protocol Composition

Problem Context and Motivation

The proliferation of agent protocols such as MCP, A2A, ANP, ACP, and ACP-Client has accelerated the adoption of tool-using, multi-agent, and model-integrated workflows that bridge local files, cloud services, peer communication, and developer environments. These protocols mediate delegated authority, shared context, and resource access, creating new attack surfaces whose risks are not fully captured by classical message- or channel-focused protocol verification. The paper identifies that vulnerabilities including command injection, SSRF, and arbitrary file read are insufficient to characterize the compositional failure modes observed in deployed agent systems. Specifically, failures such as indirect prompt injection and cross-server authority escalation emerge only in composed settings where runtime bridges connect multiple protocol endpoints, and no single protocol specification or implementation owns the relevant security obligation.

Figure 1: Illustrative multi-server MCP composition, where hidden instructions fetched from an attacker-controlled network endpoint propagate into local context and can steer local-file reads and network exfiltration.

The AgentThread Framework Overview

AgentThread is presented as a source-linked, formal security analysis framework designed to provide assurance across both protocol specifications and implementations, preserving provenance from specification texts through to formal checks and SDK test oracles. The analysis scope explicitly spans five protocol layers: wire-format integrity (RPC-L1), session lifecycle (RPC-L2), identity/capability binding (RPC-L3), semantic operation control (Agent-L4), and audit/accountability (Agent-L5). It models a Dolev-Yao adversary augmented to inject instructions, forge capability metadata, and amplify authority across delegation chains, matching the broader classes of practical agentic vulnerabilities observed in LLM-integrated systems.

Figure 2: AgentThread workflow—protocol artifacts are normalized into source-linked clauses and typed IR, compiled into TLA$^+$ formal checks, and replayed against implementations via protocol adapters when counterexamples are executable.

Protocol IR, Responsibility IR, and Formal Analysis Pipeline

AgentThread leverages two main intermediate representations: Protocol IR for clause, transition, and property records, and Responsibility IR for explicit responsibility and enforcement allocation. The IR design enables provenance-preserving formalization, systematic classification (spec-mandated, spec-recommended, hardening, layer-completeness), and validated automation boundaries.

The analysis pipeline proceeds as follows:

• Step 1 (Source Collection): Authoritative specification artifacts are collected, prioritizing normative spec text over schemas, SDKs, and examples. Modalities (MUST/SHOULD/MAY/absent) and actor roles are tagged.
• Step 2 (Clause Extraction): Machine-assisted extraction identifies clauses with behavioral, authority, lifecycle, and trust implications. Ambiguity flags allow nondeterministic modeling in TLA$^+$
• Step 3 (Typed IR Construction): IR records encapsulate transitions (who/when/how), properties (taxonomy labels), and responsibility (ownership/enforcement/gap).
• Step 4 (IR-to-TLA$^+$ Compilation): Syntax-directed mapping turns IR rules into quantifiable actors, guarded actions, primed updates, and classified invariants. Ambiguities are preserved as nondeterministic alternatives.

Responsibilities are tracked for source clause, modality, principal binding, enforcement, gap type, and evidence. SDK adapters translate formal traces to executable test cases or source/type-level evidence.

Empirical Study: Specification, Implementation, and Composition

An empirical study across five protocols and eight compositions yields strong quantitative results:

• Specification-level gaps: 35 protocol-level findings across 55 protocol-check cells—including content integrity, credential lifecycle, audit completeness, and fail-secure controls. No protocol passes all layer checks; audit and credential/registry integrity are systematically underspecified or unenforced.
• Implementation reproduction: 80 implementation tests corroborate 12 spec-level findings, with 30 additional failures observable only in composed settings.
• Composition failures: 30 counterexamples out of 43 composition safety obligations in 8 pairwise protocol compositions. Critical findings include bridge-driven authority escalation (MCP$\leftrightarrow$A2A), transitive trust collapse (MCP$\leftrightarrow$MCP), consent bypass (MCP$\leftrightarrow$ACP-Cap), and audit/provenance loss.
• Responsibility gaps: Only one control (ANP's DID-based identity verification) is both owned and enforced; composition boundaries are universally orphaned.

Case Study: Cross-Protocol Injection and Delegation

A motivating attack chain is rigorously analyzed: an agent runtime combines two MCP servers, one for network fetch and one for local-file/Markdown conversion. Attacker-supplied hidden text enters model context, acquires implicit authority, and triggers a file-capable tool to exfiltrate filtered result via the network-capable server—a cross-protocol failure that is invisible to single-server analysis. The composition methodology lifts layer invariants across bridge boundaries, showing that bridge actions such as context propagation, consent, and delegation contracts constitute responsibility gaps outside normative protocol control.

Implications, Limitations, and Future Directions

The study demonstrates that agent protocol insecurity is not reducible to individual specification or implementation bugs—responsibility gaps across protocols, SDKs, and deployment bridges are the dominant source of systemic failures. Local requirements and patching alone are inadequate; explicit composition contracts and bridge-level enforcement mechanisms are required. The layered methodology and provenance-preserving workflow allow reproducibility, evidence tracking, and actionable remediation as protocols evolve.

Immediate priorities for future work include:

• Formalization of composition contracts and responsibility assignment for runtime bridges and relay conductors;
• Extension of the provenance-linked workflow to triage ambiguous or conflicting protocol obligations;
• Integration with agent governance toolkits to enforce cross-protocol requirements at deployment boundaries.

This approach aligns with contemporary formal-methods research in protocol verification and conformance testing, but adapts it for agentic system architectures where semantic content, delegation, and provenance continuity are first-class security targets.

Conclusion

Agent protocols underpin emerging agentic software stacks but exhibit split security obligations owing to fragmented specification, SDK lax enforcement, and unowned composition boundaries. AgentThread systematizes provenance-linked formal checks, demonstrates that compositional failures are both common and unaddressed, and provides a generalizable workflow for layered security analysis tied to source artifacts and responsibility allocation. The results indicate that rigorous protocol-level assurance now mandates explicit composition safety, responsibility tracking, and enforcement mechanisms for the runtime bridges that join disparate protocol endpoints (2606.28690).