Emergent Communication for Co-constructed Emotion Between Embodied Agents via Collective Predictive Coding

Abstract: According to the theory of constructed emotion, the brain actively forms emotion categories by integrating multimodal bodily signals, and constructs emotional experiences by using these categories to predict and interpret sensory inputs. While research has advanced in modeling individual emotion construction, the social process of co-construction-how a shared understanding of emotions emerges between individuals-remains computationally underexplored. This study investigates this process by modeling emergent communication between two embodied agents using the Metropolis-Hastings Naming Game (MHNG), grounded in the Collective Predictive Coding (CPC) framework. Our experiments, using visual, auditory, and simulated interoceptive inputs, yield two main findings. First, MHNG-based communication significantly improves the alignment, clarity, and inter-agent agreement of the learned emotion categories compared to non-communicative and non-selective baselines, with the alignment effect concentrated at the symbolic layer rather than the perceptual latent representation. Second, even when the two agents have systematically divergent interoceptive dynamics, communication still produces robust categorical alignment, with distinct, category-specific reshaping patterns of each agent's emotion categories-consistent with the constructed-emotion view that interoceptive heterogeneity is constitutive of, rather than an obstacle to, shared emotional meaning. These findings provide computational support for the co-constructionist view of emotion and extend the CPC framework from physical to socially-grounded domains.

• The paper presents a computational framework for emotion co-construction using collective predictive coding to form shared emotion categories among embodied agents.
• It employs an Inter-GMM+MVAE model paired with a Metropolis-Hastings Naming Game, aligning multimodal sensory inputs and enhancing categorical clarity.
• Empirical results demonstrate that selective communication boosts inter-agent agreement, even under interoceptive disparities, highlighting the value of embodied diversity.

Emergent Communication for Co-constructed Emotion Between Embodied Agents via Collective Predictive Coding

Constructed Emotion Theory and Computational Framework

The paper "Emergent Communication for Co-constructed Emotion Between Embodied Agents via Collective Predictive Coding" (2605.09522) investigates the computational modeling of emotion co-construction—how individuals develop shared emotion categories through social interaction. The theoretical motivation stems from the theory of constructed emotion, which posits that emotion categories are not biologically hardwired but are formed via integration of interoceptive and exteroceptive signals, shaped by prior beliefs and experiences. This view diverges from essentialist models (e.g., Ekman's basic emotions), emphasizing individual and cultural variability in emotion perception and experience.

The authors adopt predictive coding as the computational basis for emotional category formation, leveraging the idea that hierarchical generative models minimize prediction errors for both internal and external sensory streams. Extending predictive coding to the social domain, they model co-construction as a process where agents iteratively adapt their internal category mappings via communicative exchanges. This instantiates the Collective Predictive Coding (CPC) hypothesis, formalized as decentralized Bayesian inference underpinning symbol emergence.

Model Architecture: Inter-GMM+MVAE and Metropolis-Hastings Naming Game

To realize co-construction computationally, the study employs the Inter-GMM+MVAE framework, combining multimodal variational autoencoding with a joint Gaussian mixture model for category formation and a Metropolis-Hastings Naming Game (MHNG) for symbol alignment.

Each agent receives multimodal sensory input—visual (facial action units), auditory (MFCCs), and simulated interoceptive signals (core affect modeled as a stochastic Ornstein-Uhlenbeck process). These streams are fused in a shared latent space via Product-of-Experts MVAE, yielding robust unimodal clustering conducive to emergent communication. Emotion categories are modeled as discrete signs (symbols) and their assignment is governed by agent-specific GMMs.

Communication follows the MHNG protocol: agents propose and probabilistically accept or reject signs using a Metropolis-Hastings criterion, iteratively aligning their category structures without explicit feedback or access to each other's internal states.

Figure 1: Overview of two embodied agents forming and sharing emotion categories via multimodal sensory input and symbolic interaction, illustrating co-construction dynamics.

Figure 2: Graphical model of Inter-GMM+MVAE showing agent-specific GMM+MVAE modules and communication pathways.

Experimental Design and Core Affect Manipulation

The empirical evaluation uses RAVDESS as the stimulus set, generating agent observations from facial and vocal expressions, with core affect simulated dynamically for each emotion category. The experimental manipulations include: (1) presence/absence of communication (MHNG vs. baselines), and (2) interoceptive symmetry/asymmetry (agents with matching/mismatching core affect profiles, including happy-inverse, low-valence-focus, and low-arousal-focus variants).

Figure 3: Visualization of the Original core affect profile used for baseline agent embodiment.

Figure 4: Mean valence-arousal coordinates for emotion categories in the circumplex model.

Figure 5: Temporal trajectories of core affect for neutral emotion, illustrating mean-reverting stochastic simulation.

Quantitative and Qualitative Results

Communication Effects

MHNG-based communication yields substantial alignment improvements: ARI and Cohen's Kappa scores are highest in MHNG conditions, indicating strong correspondence between learned categories and reference stimuli as well as robust inter-agent agreement. Davies-Bouldin Scores are minimized, reflecting well-separated clusters and enhanced conceptual clarity. The All-acceptance baseline greatly underperforms, highlighting the necessity of selective symbol acceptance for effective emergent communication.

Latent Space Analysis

PCA and t-SNE analyses (Figure 6) reveal that MHNG primarily aligns agents' symbolic labels rather than restructuring the entire latent perceptual space. TopSim similarity scores remain largely unchanged between communication and non-communication conditions. The process thus overlays a public symbolic layer atop private, modality-specific representations.

Figure 6: PCA and t-SNE visualization of agent latent spaces under communication and non-communication scenarios, demonstrating compact clusters in MHNG.

Interoceptive Divergence

Experimental manipulations of core affect produce distinct adaptive patterns. Even with systematic interoceptive disparity (e.g., attenuated arousal sensitivity in one agent), MHNG enables robust categorical alignment (Kappa > 0.39, ARI > 0.20). The agent with reduced interoceptive differentiation sharpens its categorical structure via communication, especially for high-arousal categories (anger, fear). Simultaneously, categorical boundaries in the other agent are reshaped bidirectionally.

Figure 7: Heat map of recall illustrating enhanced emotion category differentiation for symmetric and asymmetric interoceptive profiles under MHNG.

Theoretical and Practical Implications

The dissociation between symbolic alignment and latent perceptual structure conforms to the constructionist framework, elucidating how mutual understanding is possible among agents with heterogeneous bodily experience. The selective, decentralized alignment protocol avoids forced uniformity, permitting agents to maintain embodied diversity while establishing shared conceptual references. This model offers a mechanistic foundation for symbol emergence in cognitive developmental robotics and abstract category formation in social AI systems.

Crucially, the results computationally validate the proposition that interoceptive heterogeneity is not an obstacle but an intrinsic resource for shared emotional meaning, as postulated by Barrett and Gendron. The model demonstrates the emergence of shared categories through interaction despite divergence in embodiment, reinforcing the conceptual synchrony paradigm.

Limitations and Directions for Future Research

The ecological validity of posed RAVDESS stimuli is limited compared to spontaneous expression. Interoception is simulated rather than measured from physiological data; integrating real physiological signals would improve model fidelity. The communication channel is currently limited to discrete signs; future work should incorporate continuous, multimodal signaling to more fully capture affective exchange. Scaling to populations of agents is needed to model culture-level emotional norm formation.

Conclusion

This paper establishes a computational account of emotion co-construction via collective predictive coding and emergent communication, operationalized by Inter-GMM+MVAE and MHNG. Selective communication substantially improves inter-agent alignment and categorical clarity, while symbolic coordination occurs independently of perceptual homogeneity. Asymmetric interoceptive profiles induce distinct, category-specific adaptation patterns consistent with the constructed-emotion view. The results extend CPC from physical object domains to abstract, socially-grounded emotion, laying groundwork for future research on embodied social cognition and symbol emergence in AI.