EmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation

Abstract: Measuring the extent to which emergent languages encode the visual content of their inputs is an open problem. We refer to this property as visual reflection: the extent to which emergent messages preserve information about their source images that can be recovered without appeal to the speaker-listener pair that produced them. Existing metrics measure it only indirectly, through proxies such as human-defined concept inventories, natural-language captions, structural distance correlations, or Referential Game accuracy, each of which can either miss visual content the message encodes or credit content it does not. We propose EmCom-Diffusion, an evaluation framework that measures visual reflection directly: it reconstructs each input image from its emergent message and compares the reconstruction with the original image itself, rather than with human-defined targets. Concretely, it finetunes a pretrained text-to-image diffusion model on (image, emergent-message) pairs and scores visual reflection as the perceptual similarity between the reconstructed and original images, operating generatively rather than discriminatively. Instantiating it on MS-COCO with a Referential Game, we validate the metric against random and fixed-token baselines under three pretrained visual encoders, and compare it against four existing metrics (CBM, supervised translation, TopSim, and R@1). EmCom-Diffusion captures visual content the other metrics miss.

• The paper introduces EmCom-Diffusion, a novel framework that directly reconstructs images from emergent language tokens to measure visual reflection.
• It fine-tunes a pretrained Stable Diffusion model on (image, token) pairs and uses CLIP, DINOv2, and SigLIP metrics to quantify visual similarity.
• The work demonstrates that traditional metrics overlook reconstructed visual content, establishing a robust benchmark for emergent communication.

EmCom-Diffusion: A Direct Measurement of Visual Reflection in Emergent Languages

Motivation and Problem Statement

Quantifying the semantic content in emergent languages—specifically, their encoding of visual input—remains unsolved. Existing evaluation protocols, such as concept-based matching, natural language translation, topographic similarity, and Referntial Game accuracy, rely on indirect or proxy measures that may both miss and spurious credit visual content in agent-generated symbol sequences. The degree to which emergent messages actually reflect and preserve visual structure—termed visual reflection—is still inadequately captured by prevailing metrics. This limitation impedes both interpretability and a theory-grounded understanding of communicative emergence in artificial agents.

EmCom-Diffusion Framework

To address these deficiencies, the authors introduce EmCom-Diffusion, a generative evaluation framework for assessing the visual reflection of emergent languages by directly reconstructing input images from emergent messages and comparing them to the original images using perceptual metrics.

Figure 1: Overview of the EmCom-Diffusion framework—agents generate emergent messages in a referential game, a pretrained diffusion model is fine-tuned on (image, token sequence) pairs, and perceptual similarity between generated and source images quantifies reflection.

The EmCom-Diffusion protocol proceeds as follows:

1. Data Collection: Agents play a image-based referential game, generating emergent language tokens that serve as communication signals. The dataset consists of (image, message) pairs.
2. Decoder Fine-tuning: A pretrained text-to-image diffusion model (specifically, Stable Diffusion v1.5) is fine-tuned using the emergent messages as prompts for the paired images, with the generator's parameters optimized to minimize standard denoising diffusion loss conditioned on the tokens.
3. Direct Evaluation: For held-out data, the fine-tuned model generates images conditioned only on agent messages. Visual reflection is quantified by computing perceptual similarity (using multiple vision encoders: CLIP, DINOv2, and SigLIP) between the generated and original images.

Unlike discriminative or proxy-based approaches, EmCom-Diffusion is both annotation-free and generative; it does not require human-defined concept inventories or captions, nor does it rely on task performance proxy or distance-based structure.

Comparison to Existing Metrics

The paper rigorously contrasts EmCom-Diffusion with four main evaluation strategies:

• Concept-Best-Matching (CBM): Matches tokens to predefined concepts by co-occurrence, failing to credit content outside the inventory.
• Supervised Translation: Trains a seq2seq model to translate emergent messages into natural language captions, but is upper-bounded by caption content.
• Topographic Similarity (TopSim): Measures Spearman correlation between message and input distances, insensitive to structure-preserving yet non-rank aligned encodings.
• Referential Game Accuracy (R@1): Reports the accuracy of listener agents in identifying targets, but is highly sensitive to distractor sampling and can conflate communicability with semantic reflection.

The authors demonstrate that these reference- and structure-bound metrics systematically miss, under-credit, or in some cases spuriously credit visual information. Only EmCom-Diffusion directly probes the reconstructability of visual content from emergent messages—eschewing both predefined semantic anchors and structural assumptions.

Experimental Setup and Results

Game and Data Protocol

Agents are trained from scratch on MS-COCO 2017 in a referential game to produce image-conditioned messages. The emergent language is benchmarked against strong (caption-based) and null (random/fixed token) controls. Evaluation covers multiple axes:

• Image Similarity: CLIP, DINOv2, and SigLIP cosine similarity between original and generated images.
• Alignment to Natural Language: CLIP cross-modal similarity (image vs. caption).
• Image Distribution Quality: FID, Vendi, and Recall metrics.

Validation: Information Content Sensitivity

EmCom-Diffusion reliably ranks image-matching fidelity in accordance with the information content of the inputs, displaying a strict ordering:

$$\text{Random/Fixed} < \text{Emergent Language} < \text{Caption-based upper bound}$$

across all metrics.

Figure 2: Qualitative comparison—reconstructions from emergent messages closely resemble originals, while those from random tokens do not; CLIP similarity scores assess image-level match.

Random and fixed tokens result in generated images with little resemblance to original scenes, confirming that the metric is not trivially induced by priors of the generator. The emergent language reconstructions retain substantial visual correspondence.

A multi-seed analysis confirms that the measured signal reflects message-dependent content rather than sampling noise from the diffusion model.

Critical Analysis: Shortcomings of Prior Metrics

• Against Translation: When visual similarity is controlled but caption similarity is held constant, translation accuracy collapses while EmCom-Diffusion remains robust, indicating that translation is strictly bounded by caption signal and fails to capture non-captioned visual content.
• Against CBM and TopSim: When message edit distance is fixed, TopSim accuracy remains at chance and CBM degrades, whereas EmCom-Diffusion remains high—demonstrating that visual reflection is not always isomorphic to token-level distances or concept alignment.
• Against R@1: R@1 performance is shown to be heavily dependent on distractor selection. EmCom-Diffusion exhibits invariance to the distractor set, directly measuring reconstructable visual content present in the messages regardless of evaluation context.

Quantitative Summary

Empirically, the paper presents strong numerical results—e.g., DINOv2 similarity of 0.247 for emergent language versus 0.022 for random tokens; CLIP-image similarity of 0.638 for emergent language versus 0.485 for random. The referential-game-trained language achieves superior performance to random/fixed baselines but trails a stable-diffusion model fine-tuned using human-written captions (upper bound). Importantly, when other reference-based metrics become uninformative, EmCom-Diffusion still differentiates between visually similar and dissimilar cases, demonstrating its distinct discriminative power.

Implications and Future Directions

Theoretical Implications

EmCom-Diffusion enables, for the first time, a direct investigation of world-reflective structure in emergent communication without relying on human semantic inventories or communicative task performance. By operationalizing visual reflection as reconstructibility, the metric offers actionable empirical grounding for theories relating the emergence of language to the structure of perceptual domains.

Practical Use

The framework allows for principled evaluation of emergent communication agents in multimodal and language grounding tasks, providing a unified and model-agnostic approach to diagnosing semantic content. This could meaningfully inform the design of agents intended for open-world or vision-language deployment.

Future Research

Key avenues include:

• Extension to Other Communication Paradigms: Testing the framework on description, negotiation, or non-referential games.
• Domain Transfer: Evaluating on datasets with non-COCO visual statics.
• Token Attribution: Developing tools for identifying what specific aspects (object, layout, texture) of visual content are encoded by different parts of the emergent message.
• Temporal Dynamics: Longitudinal application during training to study the emergence and evolution of world-reflective structure.

Conclusion

EmCom-Diffusion is a generative, annotation-free measurement framework for visual reflection in emergent languages, bypassing the operational limits of reference-based, structure-based, and performance-based evaluation. Its combination of direct image reconstruction with perceptual metrics provides both empirical precision and theoretical clarity. The method substantiates that reconstructability from messages defines a stricter and more informative notion of visual reflection than prior proxies, offering a robust and extensible basis for future research on emergent communication and multimodal AI.

---

Reference: "EmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation" (2607.03752)