Next state prediction gives rise to entangled, yet compositional representations of objects

Abstract: Compositional representations are thought to enable humans to generalize across combinatorially vast state spaces. Models with learnable object slots, which encode information about objects in separate latent codes, have shown promise for this type of generalization but rely on strong architectural priors. Models with distributed representations, on the other hand, use overlapping, potentially entangled neural codes, and their ability to support compositional generalization remains underexplored. In this paper we examine whether distributed models can develop linearly separable representations of objects, like slotted models, through unsupervised training on videos of object interactions. We show that, surprisingly, models with distributed representations often match or outperform models with object slots in downstream prediction tasks. Furthermore, we find that linearly separable object representations can emerge without object-centric priors, with auxiliary objectives like next-state prediction playing a key role. Finally, we observe that distributed models' object representations are never fully disentangled, even if they are linearly separable: Multiple objects can be encoded through partially overlapping neural populations while still being highly separable with a linear classifier. We hypothesize that maintaining partially shared codes enables distributed models to better compress object dynamics, potentially enhancing generalization.

• The paper demonstrates that unsupervised next-state prediction enables distributed models to form linearly separable object representations without predefined object-centric priors.
• The methodology leverages auto-encoding and contrastive learning across five diverse datasets to achieve competitive or superior dynamics prediction compared to slotted models.
• The results imply that shared, partially overlapping representations can enhance generalization in AI systems, challenging the need for complete disentanglement.

Entangled Yet Compositional Representations from Next-State Prediction

The paper "Next state prediction gives rise to entangled, yet compositional representations of objects" investigates the potential for models with distributed representations to achieve compositional generalization traditionally sought through slotted models. This study emphasizes that distributed models, largely unexplored in terms of their compositional capabilities, can effectively match or surpass slotted models through unsupervised training on object interaction videos.

Overview of Findings

The study shows that distributed models can produce linearly separable object representations without object-centric priors. Surprisingly, these models often perform as well or better than slotted counterparts on tasks like image reconstruction and dynamics prediction. The key auxiliary objective facilitating this outcome is the next-state prediction, which drives the models to create separable representations without necessitating complete disentanglement.

A notable observation is that while distributed models maintain partially overlapping neural codes, they achieve high linear separability. This balance allows for efficient compression of object dynamics, potentially enhancing generalization capabilities. The study reveals that shared codes might facilitate richer generalization, enabling the use of learned dynamics across different objects.

Experimental Setup

The researchers conducted experiments across five datasets, encompassing simple block depictions to complex 3D simulations using the MOVi datasets. They utilized unsupervised learning methods including auto-encoding and contrastive learning, both in static and dynamic settings. Models were evaluated on their capacity to predict object dynamics and the linear separability of object representations.

Strong Numerical Results and Implications

Distributed models displayed superior or competitive performance in dynamics prediction tasks compared to slotted models. For example, in complex domains, distributed models achieved object separability scores significantly above chance, indicating their capability to develop systematic representations.

Dynamic data harnessed through next-state prediction substantially improved object separation, highlighting its role in representation learning. This suggests practical implications for developing efficient models that can generalize in real-world applications with dynamically interacting objects.

Theoretical and Practical Implications

The findings challenge the predominant reliance on slotted models for compositional generalization. The results underscore the potential of distributed representations to facilitate understanding in complex, dynamic environments. This has implications for both theoretical models of cognition and practical AI systems needing to interpret and act within dynamic scenes.

Future Directions

The research opens new avenues for leveraging distributed representations in AI. Future work could explore combining these models with other self-supervised learning techniques or testing scalability to naturalistic video datasets. Additionally, incorporating regularization methods could further enhance the disentanglement and generalization capabilities of distributed models.

In conclusion, the paper presents a compelling case for re-evaluating the role of distributed representations in achieving compositional generalization. By demonstrating that distributed models, when paired with dynamic objectives, can achieve separable yet entangled representations of objects, this work lays a foundation for future explorations into more efficient and adaptable AI systems.