---
title: 'Referential Games: Symbolic vs. Pixel Input'
url: https://www.emergentmind.com/topics/referential-games-with-symbolic-and-pixel-input
type: topic
---

# Referential Games: Symbolic vs. Pixel Input

Referential games with symbolic and pixel input constitute a core experimental paradigm for studying the emergence and properties of artificial communication protocols under diverse input modalities. In this context, agents—generally neural networks—are tasked with cooperative, information-theoretic games in which one or both parties encode or decode referents via variable-length messages constrained by a discrete bottleneck, with raw or structured sensory input. The structure, efficiency, and compositionality of emergent languages in these settings are shaped directly by the input modality, architecture, training protocol, and explicit or implicit information bottlenecks.

## 1. Game Formalization and Modalities

Referential games in this domain are typically played between two agents, a sender (speaker) and a receiver (listener). A canonical formalization is as follows:

- Let $O$ denote the set of objects (referents).
- The sender observes a target $o_s \in O_S$, derived from $O$ via either symbolic encoding (e.g., attribute vectors, natural language definitions) or pixel-based embedding (e.g., images processed by convolutional networks) [1804.03984], [1705.10369].
- The receiver either receives a candidate set (discriminative games), a set of symbolic descriptions, or, in multimodal games, a distinct view such as raw text when the sender observes pixels [1705.10369].
- The message space $S$ is typically a discrete space, e.g., sequences of tokens from an alphabet or binary strings of configurable length and vocabulary.
- The objective for both agents is to maximize communication success, i.e., correct identification of the referent, by jointly optimizing a reward or cross-entropy signal, possibly subject to reinforcement learning constraints and entropy regularization.

Critically, games may be unidirectional (single-shot), bidirectional (multi-step dialog), or hierarchical, and may vary in the number of distractors, message length constraints, and whether sender and receiver share modality or operate across modalities (symbolic → symbolic, pixel → symbolic, etc.) [1705.10369].

## 2. Architectures and Training Regimes

Sender and receiver networks are often composed of modality-specific encoders and a shared or compositional decoder:

- Symbolic input encoders tend to be shallow MLPs or embedding averages (GloVe, word2vec) for structured attribute vectors or text.
- Pixel input encoders employ deeper CNN backbones, sometimes with attention, to extract semantically meaningful features [1804.03984], [1705.10369], [2111.01785].
- Message generation employs LSTMs or shallow affine heads (for binary vector output), often factorized across tokens/bits, and sampling is sometimes made differentiable by Gumbel-Softmax with straight-through estimators [2012.10776], [2111.01785].
- Receivers typically combine message embeddings (e.g., via LSTM/GRU or transformer) with visual or symbolic referent candidates, either scoring them (cosine similarity, softmax over dot products) or classifying into the referent space.

Agent pairs are trained with a mix of supervised classification loss, policy-gradient reinforcement learning (REINFORCE), entropy bonuses, and auxiliary baselines (for variance reduction) [1705.10369], [1804.03984]. Training often involves curriculum components, such as adaptive stopping or variable dialog length, to capture the dynamic nature of referential communication.

## 3. Metrics for Language Structure and Generalization

Evaluation of emergent protocols combines success on the nominal task with structural and generalization metrics:

| Metric                | Definition/Implementation                                       | Purpose                                 |
|-----------------------|------------------------------------------------------------------|-----------------------------------------|
| Topographic Similarity (TS) | Negative Spearman rank correlation between pairwise distances in meaning (e.g., attribute space) and message spaces (e.g., Levenshtein), computed over a batch of referents and messages [1804.03984], [2012.10776]. | Measures alignment between input similarity and message similarity (proxy for compositionality). |
| Zero-shot Compositional Accuracy | Accuracy on held-out or OOD (out-of-distribution) attribute combinations or referents not seen during training, controlled by designed splits [2012.10776], [1804.03984], [1705.10369]. | Assesses systematic generalization.     |
| Lexicon Size/Novelty | Number or proportion of unique/novel messages, especially on novel inputs [1804.03984]. | Proxy for language productivity         |
| Communication Bandwidth/Mutual Info | $I(M;X) = H(M) - H(M|X)$, mutual information between sender view and messages [1705.10369]. | Quantifies information transfer         |
| Matching Accuracy | Fraction of trials in which the receiver correctly identifies the referent, optionally at different $K$ (Accuracy@K) [1705.10369], [2111.01785], [2112.00800]. | Direct measure of functional success    |

Notably, high TS does not guarantee generalization (zero-shot), especially in visual/pixel regimes [2012.10776].

## 4. Empirical Findings and Modality Effects

### Symbolic Input
Agents operating on structured (symbolic) inputs reliably invent protocols with nontrivial lexicon size, partial compositionality, and strong topographic similarity between input and message spaces. Empirical results consistently show high in-domain accuracy, increased productivity with greater message length, and relatively robust generalization to held-out attributes when the underlying compositional structure is preserved [1804.03984], [2012.10776].

### Pixel Input
For raw-pixel regimes, performance and structure are more sensitive to task setup and architectural choices:

- Communication accuracy and lexicon size can approach symbolic levels on tasks with abundant structure in the pixel space (e.g., position-encoding games) [1804.03984].
- Compositionality (TS) is weakly or inconsistently tied to batch size, channel capacity, or message length when compared to symbolic inputs. CNNs with batch normalization tend to regularize the emergent protocols, sometimes diluting information bottleneck effects [2012.10776].
- As the input entanglement increases (e.g., pixel images with little attribute disentanglement), protocols become ad-hoc and less compositional, often overfitting to incidental features of the data or game [1804.03984], [2012.10776].
- Multi-step, bidirectional games with multimodal access (e.g., sender: image, receiver: text bag-of-words) can support systematic protocol emergence and improved OOD transfer by leveraging richer message spaces and adaptive dialog length [1705.10369].

### Bandwidth and Message Structure
Increasing discrete channel capacity (either by message length or vocabulary) generally aids transfer and generalization, but only up to a point. Overcomplete channels facilitate disentangling and compositionality in the Gumbel-Softmax setting, although excessive vocabulary size can undermine compositional reuse [2012.10776].

## 5. Specialized Variants and Practical Implementations

Several notable variants adapt the referential game formalism to richer practical scenarios:

- **PatchGame**: Communicates via discrete, ranked patch-symbols representing salient image regions; messages are sequences of sampled tokens indicating mid-level textures or object parts. The protocol that emerges is semantically meaningful (“visual words”), supports rapid matching across augmented views, and provides acceleration for downstream ViT models by enabling inference on selected patches only [2111.01785].
- **Multimodal Multi-Step Games**: Sender and receiver communicate via binary vectors across image and text modalities, taking bidirectional turns until the receiver signals sufficient information has been gathered. Rich protocols emerge, mirroring variable-length, progressively specific information exchange seen in human dialogs. Attention mechanisms further support cross-domain generalization [1705.10369].
- **Iconary**: Expands the referential game into explicitly multimodal, human-AI settings: the "Drawer" (agent or human) composes icon-based drawings of phrases, and the "Guesser" attempts to identify the phrase via text. Both roles are trainable via sequence-to-sequence LMs (e.g., T5), with explicit scaffolding to handle OOV generalization, fill-in-the-blank reasoning, and variable-length, iterative exchange [2112.00800].

## 6. Theoretical Insights and Open Questions

The degree of compositionality in emergent protocols is tightly controlled by the structure and disentanglement of the input representation. Symbolic regimes systematically produce stable, compositional encodings; pixel regimes typically yield more fragile and idiosyncratic codes, unless explicit architectural or training biases are introduced [1804.03984], [2012.10776]. Overcomplete communication channels and strong bottlenecks can support (but do not guarantee) compositionality, especially when differentiable estimators like Gumbel-Softmax are used, but the mapping from high TS to robust generalization is not reliable in pixel or multimodal environments [2012.10776].

Significant challenges remain in engineering inductive biases or environmental pressures that foster robust, human-like compositionality—especially with raw, entangled sensory inputs. Open questions include methods for disentanglement pretraining, multi-task or curriculum learning regimes, and adaptive dialog protocols that exploit variable-length, bidirectional signaling. Emerging results in Iconary and PatchGame suggest that communication across modalities via structured discrete message spaces is feasible and practically beneficial, but that performance gaps to humans (especially on generative, non-literal, and abstract tasks) persist and highlight fertile ground for future study [2112.00800], [2111.01785].

## 7. Summary Table: Core Contrasts Across Input Modalities

| Aspect                     | Symbolic Input                              | Pixel Input                                 |
|----------------------------|---------------------------------------------|---------------------------------------------|
| Protocol Compositionality  | Robust, high TS, generalization strong      | Sensitive, often low TS, generalization inconsistent |
| Role of Channel Capacity   | Bottleneck and overcompleteness boost structure | Overcomplete channels help, but to a lesser extent; vocab size effects mixed       |
| Task Generalization        | Predictable, strong on systematic tests     | Highly sensitive to train/test split, easily overfits              |
| Metrics (TS ↔ Generalization) | Strongly correlated                        | Weak or no correlation; TS cannot proxy zero-shot performance      |

These patterns underline the crucial dependence of emergent communication structure on the statistics and encoding of the input space, and the enduring challenge of scaling artificial referential games to settings that demand robust, generalizable, and interpretable language emergence [1804.03984], [2012.10776], [1705.10369], [2111.01785], [2112.00800].

Source: https://www.emergentmind.com/topics/referential-games-with-symbolic-and-pixel-input