Latent Communication in AI Systems

• Latent communication is a design pattern where agents exchange internal continuous representations instead of explicit tokens, enabling efficient, task-specific protocols.
• It leverages shared encoders, contrastive objectives, and structural constraints to achieve stable message alignment in domains like MARL, semantic image communication, and LLM systems.
• Empirical studies show improved task performance and reduced latency, though challenges remain in privacy, computational overhead, and interpretability.

Searching arXiv for papers on “latent communication” and the supplied anchor paper to ground the article in current literature. Searching arXiv for papers on “latent communication” and the supplied anchor paper to ground the article in current literature. Latent communication denotes a family of communication regimes in which agents, models, or transmitter–receiver pairs exchange learned internal representations rather than only explicit natural-language tokens, pixels, or bitstreams. Across the literature, the communicated object may be a normalized latent message in cooperative MARL, a semantic image latent transmitted through a wireless channel, a transformer KV cache or hidden-state trace exchanged between language-model agents, or a compact latent KV block used for privacy-constrained medical collaboration. What unifies these cases is that the operative communication medium is a learned continuous representation that is consumed directly by another policy, decoder, or model, rather than first being fully rendered into conventional symbolic form (Abouelyazid et al., 26 May 2026, Chen et al., 30 Apr 2025, Dery et al., 4 Jan 2026, Chen et al., 21 May 2026, Wang et al., 11 Jun 2026).

1. Conceptual scope and terminology

Within this literature, “latent communication” is not a single mechanism but a design pattern. In cooperative MARL, it refers to observation-derived latent vectors that function as messages in a shared representation space. In semantic communication for images, it refers to transmitting semantic latent features and reconstructing the signal with a generative decoder rather than preserving pixels or bitstreams. In LLM-based multi-agent systems, it typically refers to passing internal activations such as hidden states or KV caches between agents instead of text. In hybrid systems, latent communication appears as one channel among several, paired with a text channel for explicit commitments or human-readable summaries (Abouelyazid et al., 26 May 2026, Chen et al., 30 Apr 2025, Yu et al., 23 Apr 2026, Dery et al., 4 Jan 2026, Mou et al., 25 May 2026).

A common misconception is to equate latent communication with any opaque neural message. The surveyed work is more specific. Several papers insist on structure, alignment, or grounding: shared encoders and contrastive objectives in MARL; entropy-aware latent-space JSCC and diffusion-guided reconstruction in wireless semantic communication; cache alignment, reconstruction losses, and generation losses in LLM systems; or shallow-stream restrictions in collaborative driving to avoid policy entanglement (Abouelyazid et al., 26 May 2026, Chen et al., 2024, Chen et al., 11 Jun 2026, Chen et al., 21 May 2026). Another misconception is that latent communication is automatically private because it avoids explicit text. Multiple papers reject this directly by showing that raw hidden states and KV caches can remain highly reconstructable or manipulable (Xi et al., 3 Dec 2025, Asif et al., 21 May 2026, Wang et al., 11 Jun 2026).

The term also has an analytical use. In “Approaching an unknown communication system by latent space exploration and causal inference” (Beguš et al., 2023), the latent code is not an inter-agent protocol in deployed inference, but a structured internal code used to probe which properties of sperm-whale codas an unsupervised generative model finds meaningful. This suggests that latent communication can function both as an engineering substrate and as an epistemic instrument for studying external communication systems.

2. Communication substrates and message formation

A central difference across papers is the substrate from which the latent message is formed. In SCALE-COMM, each agent observes $o_i^t$, encodes it with a shared encoder, and projects it into a normalized communication space: $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$ Here $m_i^t$ is the latent communication message, and all agents share $f_\theta$ and $W_m$, so messages occupy a common manifold. The paper uses 192-dimensional embeddings and adds learnable prototypes $p_1,\dots,p_P$ to impose a discrete-like structure on the continuous space (Abouelyazid et al., 26 May 2026).

In semantic image communication, the substrate is not an observation embedding for control but a semantic image latent. LRISC maps an image $\boldsymbol{x}$ to a high-dimensional latent tensor $$\boldsymbol{z}=g_e(\boldsymbol{x};\boldsymbol{\theta}_g)$$, applies latent-space JSCC, and reconstructs with a conditional diffusion decoder: $$\boldsymbol{x} \xrightarrow{g_e} \boldsymbol{z} \xrightarrow{f_e} \boldsymbol{s} \xrightarrow{W} \hat{\boldsymbol{s}} \xrightarrow{f_d} \hat{\boldsymbol{z}} \xrightarrow{g_d} \hat{\boldsymbol{x}}_0.$$ The communicated object is therefore the latent feature $\boldsymbol{z}$, not pixels or bitstreams; the receiver uses $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$0 as conditional guidance for generative reconstruction (Chen et al., 30 Apr 2025). CASC adopts a related latent-space pipeline, but compresses a semantic latent $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$1 into a condition signal $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$2, transmits $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$3 through AWGN, and reconstructs a denoised latent $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$4 with a latent diffusion model conditioned on $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$5 (Chen et al., 2024).

In LLM systems, the dominant substrate is the transformer’s internal state. In “Latent Space Communication via K-V Cache Alignment” (Dery et al., 4 Jan 2026), the message is a layer-wise shared latent representation $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$6 obtained from a model’s K–V cache via learned adapters $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$7, then decoded into another model’s KV space by $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$8. In DiffMAS, agents append latent blocks to a shared latent trace $$z_i^t = f_\theta(o_i^t), \qquad \tilde{m}_i^t = W_m z_i^t, \qquad m_i^t = \frac{\tilde{m}_i^t}{\|\tilde{m}_i^t\|_2}.$$9, and the trace itself is the communication channel (Yu et al., 23 Apr 2026). In Interlat, the communicated object is the sequence of last-layer hidden states $m_i^t$0, interpreted as the sender’s “mind” and injected directly into the receiver’s input sequence (Du et al., 12 Nov 2025).

A notable variant is HyLaT, which explicitly separates channels. Each output is

$m_i^t$1

so elaborate cognitive content is carried by the latent segment and concise critical signals are carried by the text segment (Mou et al., 25 May 2026). This design makes the latent channel part of a broader communication protocol rather than a total replacement for text.

3. Learning aligned and usable latent channels

A recurring theme is that latent communication works only if the latent space is aligned, stable, and task-relevant. SCALE-COMM makes this explicit by treating communication as a shared latent representation learning problem and by separating communication learning from RL policy learning. Its Phase I pretrains communication with a compound self-supervised objective,

$m_i^t$2

before Phase II PPO fine-tuning, and it uses a curriculum

$m_i^t$3

to avoid the “representation vs. control” conflict (Abouelyazid et al., 26 May 2026). In this design, stability is not an emergent by-product; it is a training target.

Heterogeneous LLM communication sharpens the same issue. “See What I See, Know What I Think” (Chen et al., 11 Jun 2026) identifies a duality in dense KV-cache transfer: context-aware transfer is driven by sparse reasoning signals, whereas context-unaware transfer requires dense contextual knowledge preservation. Its method therefore uses position disentanglement, a monotone layer map $m_i^t$4, per-KV-group MLP transformations, and two-phase training: reconstruction first, then generation. The Vision Wormhole similarly treats alignment as a first-class problem. Instead of pair-specific translators, it maps heterogeneous reasoning traces into a shared universal visual space and uses a hub-and-spoke topology so that alignment complexity scales as $m_i^t$5 rather than $m_i^t$6 (Liu et al., 17 Feb 2026).

Some systems make alignment explicit through supervised distillation. HyLaT uses single-agent hybrid generation learning and multi-agent interactive co-training, together with a cross-channel hidden-state alignment loss at the pre-answer position, to force the hybrid branch to recover the same internal state that a full text explanation would have produced (Mou et al., 25 May 2026). MedLatentDx uses supervised diagnostic loss on compact latent KV blocks so that the learned latent messages are optimized for rare-disease diagnosis rather than for generic representational similarity (Wang et al., 11 Jun 2026). By contrast, LACO is deliberately training-free: it reuses pretrained VLA models and structures latent communication through Iterative Latent Deliberation, Cross-Horizon Saliency Attribution, and Shallow-Stream Knowledge Distillation rather than by updating model weights (Chen et al., 21 May 2026).

These designs support a broader interpretation. Latent communication is rarely just “send hidden states.” The effective systems in this literature usually specify at least one of the following: a compression bottleneck, a shared latent space, a structural restriction on what layers or tokens can be shared, or a task-supervised objective that determines what the latent message should preserve.

4. Representative application regimes

The literature now spans several technically distinct regimes.

Domain	Communicated latent object	Representative systems
Cooperative MARL	Normalized latent messages, prototypes, shared embeddings	SCALE-COMM (Abouelyazid et al., 26 May 2026)
Semantic image communication	Semantic latent features, condition signals, latent diffusion states	LRISC (Chen et al., 30 Apr 2025), CASC (Chen et al., 2024), latent diffusion SemCom (Pei et al., 2024)
LLM multi-agent systems	KV caches, hidden-state traces, latent traces, hybrid latent-text messages	KV Cache Alignment (Dery et al., 4 Jan 2026), DiffMAS (Yu et al., 23 Apr 2026), Interlat (Du et al., 12 Nov 2025), HyLaT (Mou et al., 25 May 2026), Vision Wormhole (Liu et al., 17 Feb 2026)
Collaborative driving	Saliency-selected shallow-layer KV caches	LACO (Chen et al., 21 May 2026)
Cross-hospital diagnosis	Compact latent KV blocks and cross-family latent alignment	MedLatentDx (Wang et al., 11 Jun 2026)
Communication analysis	Structured latent code used as a probe of an unknown external communication system	CDEV on sperm-whale codas (Beguš et al., 2023)

In MARL, the motivation is decentralized coordination under partial observability. Latent messages must remain stable under non-stationary training and should encode planning and traffic information without destabilizing the policy (Abouelyazid et al., 26 May 2026). In semantic communication for images, the motivation is efficient transmission under bandwidth and SNR constraints, where what matters is semantic consistency and perceptual quality rather than exact pixel recovery (Chen et al., 30 Apr 2025, Chen et al., 2024, Pei et al., 2024).

In LLM agent systems, latent communication is usually motivated by the inefficiency and quantization loss of text. K–V cache alignment, dense heterogeneous KV transfer, latent traces, or hybrid protocols are all proposed as ways to preserve richer internal state while avoiding repeated decode and re-encode cycles (Dery et al., 4 Jan 2026, Chen et al., 11 Jun 2026, Du et al., 12 Nov 2025, Mou et al., 25 May 2026). In collaborative driving, the motivation is real-time coordination under strict latency, bandwidth, and ego-policy stability constraints; LACO therefore uses only shallow-layer salient KV slices to avoid “agent identity confusion” (Chen et al., 21 May 2026). In the medical setting, the main constraint is regulatory: hospital agents keep private records and retrieved cases local, and only compact latent KV blocks cross institutional boundaries (Wang et al., 11 Jun 2026).

5. Empirical patterns and comparative findings

Across domains, latent communication is usually justified by one or more of three empirical claims: improved task performance, reduced communication overhead, or improved stability. In cooperative MARL, SCALE-COMM reports near-perfect success $m_i^t$7 in under 100 learning steps on Traffic-Junction, highest episode reward $m_i^t$8 on Predator-Prey, and shortest episode length $m_i^t$9 steps on Find-Goal. In the warehouse setting it reports $f_\theta$0, $f_\theta$1, $f_\theta$2, and $f_\theta$3, together with 37.20 deliveries/episode and 41.95% unassigned time under downstream PPO (Abouelyazid et al., 26 May 2026). These numbers indicate that, in this setting, representation quality and control quality move together rather than trading off.

In wireless semantic communication, the dominant pattern is a shift from pixel fidelity toward perceptual and semantic objectives. LRISC reports an average LPIPS reduction of 43.3% compared to DeepJSCC while guaranteeing semantic consistency, and it achieves the highest PSNR and lowest LPIPS across the SNRs considered (Chen et al., 30 Apr 2025). CASC reports perceptual performance comparable to DiffSC with a 51.7% reduction in inference time, while maintaining comparable perceptual performance (Chen et al., 2024). The latent diffusion SemCom system in (Pei et al., 2024) further pushes latency reduction: its EECD variant reduces denoising from 300–800 ms per image to approximately 30–45 ms per image while keeping semantic performance nearly unchanged.

In LLM systems, the gains are heterogeneous but substantial in several papers. DiffMAS reports 26.7% on AIME24 and 20.2% on GPQA-Diamond, together with consistent gains across reasoning benchmarks over single-agent inference, text-based MAS, and prior latent communication methods (Yu et al., 23 Apr 2026). “See What I See, Know What I Think” reports that dense heterogeneous KV-cache communication matches or exceeds text communication in context-aware settings at roughly 2 to 3 times lower compute and remains effective in context-unaware transfer where prior methods collapse (Chen et al., 11 Jun 2026). Interlat reports 70.48% success on seen ALFWorld tasks and 65.42% on unseen tasks for Qwen2.5-7B-Base, outperforming both CoT and text-based communication, and its trained 8-step latent communication reduces message-generation latency from 9.19 s to 0.20 s (Du et al., 12 Nov 2025). HyLaT reports 72.0 average communication tokens and 1.47 s per question, compared with 505.0 tokens and 5.47 s for TextFullT, while maintaining competitive majority-vote accuracy; its pure-latent ablation is more efficient but materially weaker, which supports the argument for a hybrid protocol (Mou et al., 25 May 2026).

In embodied and safety-critical settings, the same pattern appears but with more stringent latency constraints. LACO reports ORION latency of 430 ms versus 7802 ms for language-based communication, and for LMDrive backbones approximately 203–215 ms versus 8021–8509 ms, while improving Driving Score and Route Completion over non-collaborative, language, and visual baselines (Chen et al., 21 May 2026). In cross-hospital diagnosis, MedLatentDx reports exact OMIM accuracy of 0.64, 0.73, and 0.59 for Qwen-3-4B, Llama-3.2-3B, and MediPhi hosts respectively, with corresponding macro-F1 values of 0.61, 0.68, and 0.49, while sharply reducing reconstructable clinical content relative to raw-KV and LatentMAS baselines (Wang et al., 11 Jun 2026).

6. Limitations, security, and open questions

The literature is unusually explicit that latent communication is not an unqualified improvement over text. A first limitation is privacy and security. “Rethinking Security in Semantic Communication” shows that an on-path attacker can manipulate semantic latents through a Diffusion-based Re-encoding Attack or a Test-Time Adaptation Latent Manipulation attack, substantially changing decoded semantics while preserving natural latent-space statistics (Xi et al., 3 Dec 2025). LCGuard addresses a related problem in LLM MAS by defining leakage operationally through reconstruction of sensitive inputs from shared KV artifacts and by adversarially learning representation-level transformations that preserve task semantics while reducing reconstructable information (Asif et al., 21 May 2026). MedLatentDx reaches the same conclusion empirically in a medical setting: raw KV and LatentMAS communications nearly perfectly expose disease and phenotype attributes under reconstruction attacks, while compact latent KV blocks sharply reduce token-level F1 and attribute recovery (Wang et al., 11 Jun 2026). The general lesson is clear: avoiding explicit text does not make a latent channel safe.

A second limitation is computational overhead and systems complexity. SCALE-COMM explicitly reports approximately 77% more runtime and approximately 82% more CPU usage than lighter SSL baselines, with $f_\theta$4 time and $f_\theta$5 memory during SSL pretraining (Abouelyazid et al., 26 May 2026). Diffusion-based semantic communication systems repeatedly note the cost of iterative denoising (Chen et al., 30 Apr 2025, Pei et al., 2024). Even in LLM reasoning, “Exploring System 1 and 2 communication for latent reasoning in LLMs” concludes that under matched latent-token budgets, H2 is consistently strongest while H1 yields modest gains, yet a unified soft-embedding baseline nearly matches H2 and surpasses H1, suggesting that current dual designs mostly add compute rather than qualitatively improving reasoning (Coda-Forno et al., 1 Oct 2025). This is an important counterpoint to stronger claims in the rest of the literature.

A third limitation is interpretability and portability. Pure latent channels are efficient but opaque and often architecture-specific. HyLaT frames this as an “agent communication trilemma,” and its response is to retain a text channel for concise critical signals while offloading elaborate cognitive content to a latent channel (Mou et al., 25 May 2026). The Vision Wormhole addresses portability by repurposing the visual interface of VLMs as a universal port and by reducing pairwise alignment complexity from $f_\theta$6 to $f_\theta$7 (Liu et al., 17 Feb 2026). LACO addresses a different portability problem by remaining training-free, but only under the assumptions of pretrained VLA backbones and shallow-stream latent fusion (Chen et al., 21 May 2026).

These limitations point to a common research frontier. The strongest current systems do not merely exchange hidden states; they actively shape the latent space through contrastive alignment, distillation, structural bottlenecks, or hybrid protocols. This suggests that the future of latent communication is likely to depend less on transmitting ever-richer raw internal state and more on learning task-specific, secure, and interoperable latent protocols whose geometry is deliberately constrained rather than accidentally inherited from the backbone.