- The paper introduces a file-based protocol, tap, that enables collaboration between heterogeneous LLM agents by standardizing markdown file communication with YAML metadata.
- The paper demonstrates a robust methodology with per-environment configuration and workspace isolation, ensuring real-time notifications and durable audit logs.
- The paper presents empirical results showing higher defect detection rates in heterogeneous model pairings and effective recovery from transient communication failures.
File-Based Protocol for Heterogeneous LLM Agent Collaboration: An Authoritative Analysis of tap
Motivation and Background
The proliferation of LLM-based coding agents has accelerated multi-agent collaboration strategies in software engineering. Notably, prior frameworks such as ChatDev, MetaGPT, and AutoGen focus on homogeneous agent configurations — leveraging shared runtimes, central conversation servers, or unified APIs. However, such assumptions fail in practical, heterogeneous environments, where agents from disparate vendors (e.g., Anthropic's Claude and OpenAI's Codex) must coordinate without native inter-process communication or shared execution context. Existing standards such as Google A2A emphasize HTTP-based interoperability, targeting inter-organizational agent integration, but do not resolve the repository-centric, file-based requirements of local collaborative workflows.
tap Protocol Design
tap is introduced as a lightweight, file-first protocol for direct collaboration between heterogeneous LLM agents. The protocol encompasses three principal mechanisms:
- File Messages: Standardizes communication by encoding agent replies as markdown files in a designated inbox/, including YAML metadata. Delivery bifurcates: (1) file inspection by MCP tools (Tier 1), and (2) real-time notification pathways tailored for each agent (Tier 2, e.g., Claude MCP notifications, Codex WebSocket delivery).
- Per-Environment Configuration: tap orchestrates setup and configuration per agent runtime, updating Claude's .mcp.json, Codex's config.toml, and Gemini's JSON config autonomously without forcing API uniformity.
- Workspace Isolation: Each agent operates in a separate git worktree, mitigating workspace-level contention, with instance-specific logs and state tracked by unique instanceId.
The underlying rationale is that the file system is universally accessible and provides robust message canonicity and auditability. Real-time notification accelerates response latency, but all message processing is ultimately sourced from the canonical file artifact, rendering notification failures as delays rather than message loss.
Operational Results: Empirical Evaluation
tap was self-applied for 27 days over 37 agent generations in a production repository (209 PRs, 717 artifacts). Operational continuity was maintained via durable file-based logs, functioning as external memory for nonpersistent agents across generations. Agents coordinated development, review, and protocol evolution entirely through tap-mediated messaging.
Defect Detection Analysis
A salient empirical result is the differential defect/change detection rate in review artifacts:
- Heterogeneous Model Pairings: 69.8% (183/262 reviews) recorded at least one defect or requested change.
- Homogeneous Model Pairings: 53.1% (60/113 reviews) contained a defect or requested change.
Especially, Codex reviewers inspecting Claude-authored code yielded a 71.0% detection rate. These results, while confounded by reviewer role assignment and operational imbalance (later generations predominantly using headless Codex reviewers), nonetheless demonstrate that heterogeneous agent pairings broaden review perspective and enhance defect discovery rates.
Security-relevant findings emerged through both heterogeneous and homogeneous review flows, including runtime vulnerabilities (e.g., execSync shell injection, CSRF endpoints). Although the security-specific sample size was small, the operational loop enabled effective identification and remediation of critical defects.
Recovery and Failure Tolerance
Message durability is achieved via atomic file writes ahead of environment-specific delivery. Agents always read canonical files from inbox/ for processing, enabling seamless recovery after delivery disruptions (e.g., WebSocket disconnects, MCP notification loss) and end-to-end auditability.
Discussion and Implications
tap diverges from HTTP-based inter-agent standards, focusing on file system and git-mediated workflows for small-team, local repository scenarios. The protocol's design demonstrates robust message preservation, recovery from transient failures, and workspace isolation, which are essential in environments lacking centralized orchestration.
Empirical findings support the hypothesis that combining multiple vendor LLMs improves review outcomes, and that cross-execution environments amplify detection heterogeneity. However, interpretation must be tempered by operational imbalances, observational rather than controlled experimental assignment, and potential context variance across the 37 agent generations.
tap's operational records serve as external persistence in agent environments lacking internal state, enabling continuity of rationale, context, and operational history. Importantly, protocol improvements (e.g., normalization of paths, per-instance logs, canonical naming) were driven by feedback loops within self-applied tap operation.
Limitations and Future Directions
- Observational data and non-random reviewer assignment limit causal inference.
- Detection rates are artifact-based, not normalized for independent defects.
- Real-time collaboration was limited to Claude and Codex; Gemini was used only experimentally.
- The operational environment was a single organization and codebase.
- tap itself evolved throughout the evaluation, introducing potential bias in early versus late generations.
Future work will entail balanced experimental comparisons, randomized reviewer assignments, and longitudinal studies across diverse repositories and organizational practices. As LLMs and agent-enabled tooling evolve, the quality and structure of agent collaboration protocols such as tap will need revalidation and enhancement.
Conclusion
tap establishes a file-based protocol for heterogeneous LLM agent collaboration, validated through sustained self-applied operation and artifact collection. Empirical results demonstrate elevated defect detection rates for heterogeneous agent pairings, reinforcing the utility of cross-vendor and cross-environment review dynamics. tap's file-based recordkeeping ensures operational continuity and auditability, addressing key limitations in existing agent collaboration frameworks. Ongoing research should focus on isolating reviewer and model effects, scaling to broader operational contexts, and refining collaboration protocols as LLM agent capabilities mature (2606.14445).
