Latent Collaboration in Multi-Agent Systems (2511.20639v1)

Abstract: Multi-agent systems (MAS) extend LLMs from independent single-model reasoning to coordinative system-level intelligence. While existing LLM agents depend on text-based mediation for reasoning and communication, we take a step forward by enabling models to collaborate directly within the continuous latent space. We introduce LatentMAS, an end-to-end training-free framework that enables pure latent collaboration among LLM agents. In LatentMAS, each agent first performs auto-regressive latent thoughts generation through last-layer hidden embeddings. A shared latent working memory then preserves and transfers each agent's internal representations, ensuring lossless information exchange. We provide theoretical analyses establishing that LatentMAS attains higher expressiveness and lossless information preservation with substantially lower complexity than vanilla text-based MAS. In addition, empirical evaluations across 9 comprehensive benchmarks spanning math and science reasoning, commonsense understanding, and code generation show that LatentMAS consistently outperforms strong single-model and text-based MAS baselines, achieving up to 14.6% higher accuracy, reducing output token usage by 70.8%-83.7%, and providing 4x-4.3x faster end-to-end inference. These results demonstrate that our new latent collaboration framework enhances system-level reasoning quality while offering substantial efficiency gains without any additional training. Code and data are fully open-sourced at https://github.com/Gen-Verse/LatentMAS.

• The paper introduces LatentMAS, a training-free framework that enables LLM-based agents to share hidden state representations for lossless, efficient collaboration.
• The methodology leverages latent thought generation and KV-cache working memory transfer to reduce token usage by up to 83.7% while boosting accuracy by 14.6%.
• Empirical results across nine benchmarks demonstrate substantial gains in inference speed (4x faster) and semantic fidelity compared to text-based multi-agent systems.

Latent Collaboration in Multi-Agent Systems: An Expert Review

Introduction and Contextualization

The paper "Latent Collaboration in Multi-Agent Systems" (2511.20639) presents LatentMAS, a training-free multi-agent coordination framework that enables LLM-based agents to reason and communicate entirely in the latent space, obviating the need for explicit textual mediation. This work addresses foundational inefficiencies in conventional text-based multi-agent systems, proposing a pipeline that leverages internal hidden-state representations and shared working memory (via KV-caches), thus ensuring information preservation and collaborative expressiveness at a substantially reduced computational cost.

Motivation and Design Principles of LatentMAS

Current LLM-based multi-agent systems (MAS) universally rely on textual communication for message passing, which imposes three fundamental bottlenecks: limited reasoning expressiveness of discrete tokens, lossy information exchange due to serialization, and substantial computational complexity from autoregressive decoding. This paper critiques these limitations and posits the following central hypotheses:

• Latent Reasoning Expressiveness: Transformer's hidden states encapsulate semantically rich, high-dimensional information that exceeds the representational capacity of discrete tokens.
• Lossless Communication Fidelity: KV-cache-based latent working memory enables agents to share system-wise internal representations without intermediary translation to text, thereby preserving semantic integrity.
• Collaborative Complexity Reduction: Pure latent communication allows MAS to operate at higher information density per step and lower runtime overhead than text-based paradigms.

The framework is fully training-free, compatible with sequential and hierarchical MAS architectures, and agnostic to backbone scaling (validated on Qwen3-4B, 8B, 14B).

Methodological Framework

Latent Thoughts Generation

Each agent receives the input sequence (question and prompt tokens), encodes it, and performs autoregressive latent thoughts generation, iteratively producing last-layer hidden states which serve as continuous intermediate outputs. Instead of decoding tokens, the agent realigns hidden states back to the valid input distribution using a linear alignment operator $W_a$, computed via ridge regression over embedding matrices ($W_{\text{in}}$, $W_{\text{out}}$). This enables model-agnostic and distributionally robust chaining of latent reasoning steps.

Layer-wise Working Memory Transfer

After latent generation, the full layerwise KV-caches (internal key-value memories) of an agent are extracted and shared directly to the next agent, serving as a latent working memory. By concatenating caches at each transformer layer, downstream agents inherit both input context and latent thoughts from predecessors, facilitating seamless, lossless cross-agent conditioning and collaborative reasoning.

Theoretical Analyses

The paper formalizes:

• Expressiveness Gap: Latent thoughts enable $O(\frac{d_h}{\log |\mathcal{V}|})$ times higher reasoning efficiency than token-based approaches, where $d_h$ is hidden state dimensionality and $|\mathcal{V}|$ is vocabulary size. For scale reference, Qwen3-14B achieves $\sim$471$\times$ expressiveness improvement.
• Information Preservation: Theorem shows equivalence of outputs when passing either latent working memories or explicit preceding outputs, i.e., latent transfer maintains full input fidelity for downstream computation.
• Complexity Advantage: The overall steps and runtime for LatentMAS scale much more favorably than text-based MAS, especially for deep chains and wide multi-agent hierarchies.

  Figure 1: Overview of LatentMAS and its latent-only reasoning/communication pipeline across agents.

Empirical Performance

Benchmark Results

LatentMAS is evaluated on nine multi-domain tasks (math, science, commonsense, code), covering GSM8K, ARC, MedQA, MBPP-Plus, HumanEval-Plus, AIME, and GPQA benchmarks. Both sequential (planner → critic → refiner → solver) and hierarchical (domain-expert agents under summarizer) settings are considered. Performance metrics include accuracy, system throughput (token usage), and inference time.

Figure 2: LatentMAS delivers marked improvements in accuracy, runtime, and token economy across nine benchmarks and three backbone scales, under hierarchical MAS settings.

Notable empirical outcomes:

• Accuracy: LatentMAS outperforms single-model and text-based MAS baselines by up to 14.6% and delivers consistent accuracy gains across diverse tasks and MAS architectures.
• Token Usage: LatentMAS reduces output tokens by 70.8%-83.7% compared to baselines, attributable to latent-only communication between agents and minimal need for final-stage text decoding.
• Inference Speed: LatentMAS offers 4x-4.3x speedup over text-based MAS (even when baselines are vLLM-optimized), benefitting from fewer generation steps.

  Figure 3: LatentMAS achieves substantial gains in both computation time and system-wide token savings in sequential settings.

Latent Space Analysis

Figure 4: Latent thought embeddings in LatentMAS cover the semantic space of text-generated tokens and demonstrate superior diversity.

Experimental analysis establishes that last-layer hidden states generated by LatentMAS agents are largely semantically consistent with the embedding distribution of text-based MAS but occupy a broader and denser region, supporting claims of expressiveness and system-level latent reasoning.

Alignment and Ablation

Figure 5: Input-output alignment operator $W_a$ effectively counteracts activation drift, ensuring latent states project into the valid input embedding space.

Figure 6: Downstream accuracy improves by 2.3%-5.3% with $W_a$ alignment versus none, across multiple reasoning benchmarks.

Step-depth ablations show optimal accuracy at latent step ranges of 40-80, after which performance plateaus.

Implications and Theoretical Impact

LatentMAS advances MAS research across several dimensions:

• Efficiency and Scalability: By decoupling agent communication from text, it establishes a scalable substrate for system-wide reasoning and coordination, vital for large multi-agent compositions.
• Error Propagation Mitigation: Latent working memory preserves intermediate semantic structures, reducing compound reasoning errors inherent to text serialization and interpretation.
• Foundations for Latent-Optimized Training: The framework is fully training-free, yet it opens future avenues for fine-tuned latent reasoning and inter-agent alignment, potentially using gradient signals directly on cross-agent latent representations.

Practically, LatentMAS is deployable with any decoder-only transformer stack supporting KV-caching and provides a drop-in acceleration and robustness boost for agentic AI applications spanning open-domain QA, scientific discovery, code synthesis, embodied decision-making, and multi-modal systems.

Future Directions

Potential future research includes:

• Extending latent collaboration to heterogeneous agent assemblies with non-isomorphic architectures (using adapters or permutation alignment).
• Incorporating latent-space training signals to optimize agent interaction and latent reasoning protocols.
• Exploring emergent specialization and coordination in very large LatentMAS-hybrid systems well beyond current text-based collaborative limits.
• Integrating latent communication with multimodal agent architectures and tool-augmented frameworks.

Conclusion

The LatentMAS framework constitutes a systematic departure from traditional text-based communication protocols in multi-agent LLM systems, substantiating both complexity-theoretic and practical advantages of latent reasoning and collaboration. The presented theoretical and empirical analyses demonstrate robust gains in accuracy, inference efficiency, and semantic information density. These results signify a paradigm shift applicable to large-scale decentralized AI, where continuous latent-space communication pipelines may become foundational to future agentic system design and optimization.