Emergent Communication between Heterogeneous Visual Agents through Decentralized Learning

Abstract: Symbols are shared, but perception is private. We study emergent communication between heterogeneous visual agents through decentralized learning, asking what visual information can become shareable when agents have different visual representations. Instead of optimizing messages through a shared external communicative objective, our agents exchange only discrete token sequences and update their own models using local perceptual evidence. This setting focuses on an underexplored aspect of emergent communication, examining whether common symbols can arise without shared perceptual access, and how the similarity between private visual spaces constrains the content and symmetry of the resulting language. We instantiate this setting in the Metropolis-Hastings Captioning Game (MHCG), where two agents collaboratively form shared captions by exchanging proposed token sequences that a listener accepts or rejects using an MH-style criterion evaluated against its own visual features. We compare three pairings of frozen visual encoders, with agents starting from randomly initialized text modules. Experiments on MS-COCO show that MHCG produces visually informative shared token sequences that outperform a no-communication baseline in cross-agent alignment, visual-feature prediction, and image-text retrieval; all cross-agent metrics decline as encoder mismatch increases. Moderate encoder heterogeneity reduces the number of shared sequences while preserving per-sequence visual specificity, whereas stronger encoder heterogeneity yields fewer, coarser, and more asymmetric sequences. Ablations show that listener-side MH acceptance is critical for avoiding degenerate token formation. These results suggest that shared symbols can arise from local perceptual evaluation alone, with visual representational similarity across encoders shaping both the content and symmetry of the resulting language.

• The paper introduces a novel decentralized learning framework using the Metropolis–Hastings Captioning Game (MHCG) where agents create shared token sequences from private perceptual evidence.
• The paper shows that moderate heterogeneity maintains high visual specificity in emergent symbols while strong heterogeneity reduces vocabulary and induces representational bias.
• The paper confirms that likelihood-based listener filtering is essential to avoid token degeneration and to ensure robust cross-agent visual information transfer.

Emergent Communication in Heterogeneous Visual Agents via Decentralized Learning

Introduction and Motivation

This work investigates the emergence of communication protocols between agents possessing heterogeneous, private visual representations, within a fully decentralized paradigm. Instead of relying on a shared external communication objective or task reward, these agents ground public, discrete token sequences (symbols) exclusively in their own private perceptual evidence. The central research question concerns which aspects of perceptual structure can be preserved in shared symbols when agents differ in their visual encoders and to what extent these emergent symbols reflect both agents' representational geometry versus privileging one agent's visual space.

Methodological Framework

The study operationalizes decentralized emergent communication using the Metropolis–Hastings Captioning Game (MHCG), an instantiation of Collective Predictive Coding (CPC) for vision–language agents. Each agent is built from a frozen visual encoder (variants of DINOv2 ViT or MAE ViT for heterogeneity) and a BLIP-style text encoder/decoder. Unlike prior work, text modules are initialized from scratch (no pre-trained language semantics), ensuring that communication codes emerge solely through decentralized interaction.

In MHCG, agents alternate in speaker/listener roles. The speaker observes an image, proposes a caption (discrete token sequence), and the listener—based on its own visual evidence—accepts or rejects the proposal using a likelihood ratio between its own generated caption and the speaker’s proposal, modeled via a probabilistic adapter (ProbVLM) in a shared latent embedding space. Each accepted caption becomes the subsequent learning target for the listener.

Figure 1: The Metropolis–Hastings Captioning Game enables two agents with private frozen visual encoders to iteratively propose and accept discrete caption sequences, operationalizing the emergence and alignment of shared symbols via representational similarity analysis.

Key to the architecture is the absence of any joint optimization or information sharing besides token exchange, forbidding shared gradients, task-level rewards, or parameter synchronization. This strictly local update regime allows examination of symbol emergence driven only by mutual, perception-grounded negotiation.

Three primary visual encoder conditions are compared:

• Homo: Both agents have DINOv2 ViT-B/14 (homogeneous).
• Hetero1: DINOv2 ViT-B/14 vs DINOv2 ViT-S/14 (moderate heterogeneity).
• Hetero2: DINOv2 ViT-B/14 vs MAE ViT-B/16 (strong heterogeneity).

Experiments are conducted on MS-COCO with independently sampled augmentations per agent. Emergent symbols are analyzed for representational alignment (RSA), visual-feature transfer, image–text retrieval, and degeneracy under ablation of the MH acceptance mechanism.

Results: Communication, Specificity, and Representational Bias

Emergent Symbols and Cross-Agent Visual Information Transfer

MHCG consistently produces shared token sequences that encode substantial visual information, outperforming a no-communication baseline (NoCom) across all cross-agent retrieval, RSA, and visual-feature prediction metrics. As encoder mismatch increases, cross-agent metrics systematically decline. However, even under strong heterogeneity (Hetero2), cross-agent transfer remains above baseline, indicating the robustness of the decentralized MHCG protocol.

Figure 3: Training dynamics illustrate higher cross-agent retrieval and token alignment under MHCG relative to controls, demonstrating non-collapsed emergent vocabularies; performance is modulated by the degree of encoder heterogeneity.

Visual Specificity and Concept Breadth

With moderate heterogeneity (Hetero1), fewer shared token sequences emerge (∼39% reduction compared to Homo), but the per-sequence visual specificity, measured as normalized visual radius in fixed measurement spaces, is preserved. This reflects a possible precision–coverage tradeoff: increased heterogeneity reduces the vocabulary of shared symbols but sharpens the semantic coherence of those that survive.

Figure 4: Across encoder conditions, matched token-sequence concepts align with compact, visually coherent clusters in each agent's visual space, though strong heterogeneity leads to broader and more object-diverse categories.

Under strong heterogeneity (Hetero2), emergent symbols are both fewer and coarser, covering larger and more visually diverse image sets, as visual radius and effective object-category entropy increase. This demonstrates that large representational gaps impair the granularity of shared semantic structure.

Representational Symmetry and Bias

Partial RSA analyses reveal that emergent communication exhibits directional biases: with increasing heterogeneity, shared token sequences reflect the visual geometry of one agent (typically Agent A with higher-capacity encoding) more than the other. This bias is minimal for moderate encoder mismatch (Δ_A ≈ 0.03) but pronounced under strong mismatch (Δ_A ≈ 0.08), and this is correlated with observable retrieval asymmetry.

Role of Listener-Side Likelihood Filtering

Ablation studies confirm that the listener-side MH likelihood-ratio filter is critical for the emergence of non-degenerate, informative tokens. Controls that accept all proposals, accept at random, or use only discriminative matching (ITM-based) either collapse to a few degenerate codes or fail to transfer significant visual information, highlighting the necessity of a generative, likelihood-based filtering mechanism for successful category formation and alignment.

Figure 6: Ablation dynamics for the Hetero1 condition show that only MH-based acceptance preserves both alignment and diversity of emergent symbols; other mechanisms result in collapse or non-informative codes.

Theoretical and Practical Implications

This work provides direct empirical evidence that decentralized negotiation—absent any explicit task- or reward-level coupling—can induce the emergence of a public symbol system that encodes and transfers visual structure between heterogeneous agents. The findings establish that shared symbols can arise exclusively through local evaluation and discrete message-passing, thus validating key tenets of collective predictive coding and decentralized Bayesian inference in multi-agent settings.

The impact of representational heterogeneity is not binary: moderate heterogeneity reduces shared symbol vocabulary but maintains per-symbol sharpness; extreme heterogeneity degrades both coverage and specificity, indicating a critical threshold where mutual intelligibility becomes unattainable. The directional bias observed in symbol formation further implies that decentralized systems are susceptible to implicit representational dominance, which must be considered in the design and audit of multi-agent AI.

Practically, these results have implications for privacy-preserving learning, modular agent deployment, and coordination in distributed systems where shared world models or gradients are impossible or undesirable. Future work should examine symbol emergence under adaptation of perceptual modules, broader encoder modalities, more complex or compositional symbol spaces, and larger agent populations.

Conclusion

This study demonstrates that heterogeneous agents with fixed, mismatched visual encoders can construct shared symbolic codes through fully decentralized interaction, driven only by perception-grounded, proposal/accept protocols. The geometry and transferability of these emergent symbols depend on the degree of representational alignment between agents, with significant implications for the scalability and equity of emergent communication systems. Listener-side MH filtering is indispensable for non-degenerate, informative symbol formation. These findings motivate further research into the dynamics of symbol emergence in more complex and adaptive multi-agent scenarios, as well as the development of diagnostic tools for representational audit and bias identification in decentralized AI systems.