Object-Centric Structured World Models

Updated 30 December 2025

The paper demonstrates that object-centric structured world models decompose scenes into latent object slots, boosting sample efficiency and multi-step prediction accuracy.
They integrate convolutional encoders, slot attention, and graph neural networks to model inter-object dynamics critical for tasks such as robotic manipulation and control.
The approach enhances interpretability and zero-shot transfer, though challenges remain with slot identity drift and fixed slot number limitations.

Object-centric structured world models are computational architectures that represent, track, and predict the dynamics of environments by explicitly decomposing observations into object-level entities and modeling their interactions. This approach enforces a factorization of the world state into “slots” or object representations, typically learned in a permutation-invariant manner, and leverages relational or graph-based dynamics to reason about interactions. Unlike monolithic state-space models, object-centric world models aim to match the compositional and relational structure of real-world scenes, yielding increased sample-efficiency, robustness, and generalization in prediction, planning, and control.

1. Principles and Theoretical Foundations

Object-centric world models are motivated by the compositional nature of the physical world and the need for models that can efficiently reason about multiple interacting entities. The foundational principle is to represent each scene as a set of $K$ latent “slots,” each corresponding to an object or entity, such that the slots collectively capture all decision-relevant information. The encoder $E$ maps observations $s_t$ to latent slots $\mathbf{z}_t = \{z_t^k\}_{k=1}^K$ , where each $z_t^k$ is a low-dimensional feature vector (Kipf et al., 2019, Collu et al., 8 Jan 2024, Jeong et al., 8 Mar 2025).

Provable identifiability is established under the assumptions of compositionality and irreducibility: a decoder is compositional if each pixel (observation dimension) depends on at most one slot, and mechanisms are irreducible if no object can be split into independent subparts. If the encoder/decoder pair is invertible and compositional, each ground truth object is recovered in a unique slot up to permutation and invertible reparameterization (Brady et al., 2023). This theoretical framework accounts for empirical successes in current architectures.

2. Model Architectures and Learning Methods

A canonical object-centric structured world model consists of:

Object encoder: Raw observations (usually images or video frames) are encoded using a CNN or a pre-trained foundation model, followed by a slot extraction mechanism such as feedforward masks (Kipf et al., 2019), competitive attention (Collu et al., 8 Jan 2024), or explicit segmentation (Zhang et al., 27 Jan 2025, Ferraro et al., 2023). Slot Attention applies iterative attention updates and weight-sharing to produce disentangled slot embeddings.
Dynamics module: Inter-object dynamics are modeled by a fully connected graph neural network (GNN) or a relational Transformer. Node features are the slots; message passing updates each slot by aggregating information about its neighbors and, where appropriate, the agent’s action (Kipf et al., 2019, Sancaktar et al., 2022, Ugadiarov et al., 2023).
Object-centric transition (action-conditional): The next-step state of each object slot is predicted as $z_{t+1}^k = z_t^k + \Delta z_t^k$ , where $\Delta z_t^k$ is output by the GNN as a function of current slots and actions. Actions are injected as one-hot or distributed vectors, either concatenated to specific slots or broadcast (Kipf et al., 2019, Collu et al., 8 Jan 2024, Feng et al., 4 Nov 2025).
Decoders: Slot decoders reconstruct input images or predict object attributes, often using spatial broadcast decoders or mask-weighted sum over object-specific predictions, ensuring permutation equivariance and disentanglement (Jeong et al., 8 Mar 2025, Villar-Corrales et al., 17 Feb 2025, Collu et al., 8 Jan 2024).
Losses and training: Loss functions include reconstruction loss (L2 or cross-entropy), KL divergence for variational methods, contrastive hinge loss for discriminative alignment (Kipf et al., 2019), per-slot prediction error, and explicit compositionality penalties (Brady et al., 2023). Object-centric models can be trained unsupervised, supervised, or via hybrid schemes with a small set of segmentation labels (Zhang et al., 27 Jan 2025).

3. Learning Object Structure and Relations

Unsupervised object discovery is achieved via architectural and loss-based inductive biases:

Inductive bias: Fixed number of slots; parameter-sharing across slot initializations forces competitive assignment (each slot must “explain” part of the scene) (Kipf et al., 2019, Collu et al., 8 Jan 2024).
Contrastive or predictive loss: Contrastive losses force each slot to have predictive power; empty or non-informative slots are penalized by failure to minimize the prediction or contrastive objective (Kipf et al., 2019).
Slot Attention: Competitive attention and iterative updates separate similar or duplicate objects, addressing failures of mask-based methods to disambiguate objects with similar appearance (Collu et al., 8 Jan 2024).
Relational factorization: Relational GNN/Transformer modules explicitly model pairwise/object-entity interactions and can learn interaction graphs (adjacency matrix) and dynamic factorization (Sancaktar et al., 2022, Feng et al., 4 Nov 2025).

Slot Structured World Models (SSWM) combine pre-trained object-centric encoders with GNN-based relational dynamics, achieving superior multi-step generalization compared to non-slot or mask-based baselines (Collu et al., 8 Jan 2024).

4. Downstream Applications and Empirical Performance

Object-centric structured world models yield substantial empirical benefits:

Efficient and robust exploration: Structured models with slot-wise epistemic-uncertainty (via GNN ensembles or count-based bonuses) drive targeted curiosity and early agent-object interaction, outperforming pixel-based or unstructured policy-based methods in sample efficiency and zero-shot transfer (Sancaktar et al., 2022, GX-Chen et al., 21 Aug 2024).
Manipulation and control: In robotic manipulation, slot-based world models achieve superior generalization to novel object configurations, attribute values, and tasks without retraining (Sancaktar et al., 2022, Feng et al., 4 Nov 2025, Ferraro et al., 2023). Explicit object-level reasoning yields higher success and faster learning in stacking, locomotion, and multi-object settings.
Interpretability and controllability: Slot-based models, especially those with language-conditioned prediction (e.g., TextOCVP), provide fine-grained control over predictions by editing slot-representations or textual prompts, yielding more interpretable and robust generative video and action rollouts (Villar-Corrales et al., 17 Feb 2025, Jeong et al., 8 Mar 2025).
Partially observable and uncertain environments: Structured World Belief models combine object-centric decomposition with explicit particle belief tracking, supporting robust planning, filtering, and Bayesian uncertainty quantification (Singh et al., 2021).
Generalization and transfer: Empirical studies demonstrate that object-centric models support compositional zero/few-shot transfer to scenes with unseen object types, counts, or arrangements, yielding lower sample requirements and improved robustness under distribution shift (Feng et al., 4 Nov 2025, Zhang et al., 27 Jan 2025).

Across complex domains (Sprites-World, Gym-Fetch, Franka-Kitchen, Atari, Hollow Knight), slot-based architectures consistently outperform pixel-centric or holistic alternatives on multi-step prediction (Hits@1 up to 98% vs. 20–50% for baselines at 10-step horizon (Collu et al., 8 Jan 2024, Kipf et al., 2019)), policy-learning efficiency (often >80% success in 1–2M steps (Feng et al., 4 Nov 2025)), and OOD robustness.

5. Limitations and Open Challenges

Despite empirical and theoretical strengths, object-centric world models face significant challenges:

Slot identity and permutation: Models may arbitrarily permute or swap slots, particularly for identical or highly similar objects, leading to latent “slot-identity drift” during multi-object interactions. This induces instability in downstream actor-critic training, as shown by latent-trajectory analyses revealing “representation shift” at object contact (Ferraro et al., 8 Nov 2025).
Choice of slot number: Fixed slot count $K$ must be tuned; “empty” slots are not always explicitly represented, and variable-object-number scenarios still pose architectural and loss-design challenges (Kipf et al., 2019, Collu et al., 8 Jan 2024).
Partial observability and occlusion: While belief-augmented models with particle filtering (Singh et al., 2021) or explicit permanence tracking (Singh et al., 2021, Collu et al., 8 Jan 2024) help, realistic real-world scenes involve heavy occlusion and transparency, breaking strict compositionality assumptions (Brady et al., 2023).
Background and global context: Pure slot-models may miss background context or global geometric cues needed for some tasks; hybrid approaches fuse slot and pixel or holistic features (Zhang et al., 27 Jan 2025).
Policy integration: Nontrivial drift in slot latents during contact-heavy physical interactions undermines policy learning, and naive actor-critic integration yields subpar downstream performance compared to holistic latent models (e.g., DreamerV3) (Ferraro et al., 8 Nov 2025). Smoothing strategies (e.g., slot exponential moving average) provide partial remedies.

6. Extensions and Theoretical Developments

Recent work provides rigorous guarantees of object-slot identifiability under minimal assumptions (compositional decoder, irreducible mechanisms, and invertible architectures), with new compositional regularization terms enabling diagnosability and architectural assessment (Brady et al., 2023).

Active and discriminative exploration strategies in object-centric abstract state spaces yield drastic gains in discovery and zero-shot/few-shot planning (GX-Chen et al., 21 Aug 2024), while structured, causality-aware Transformers further extend state-of-the-art in complex, object-rich RL domains (Nishimoto et al., 18 Nov 2025).

Language conditioning, cross-modal abstraction, explicit interaction learning via learned adjacency structures, and scalable computation and planning are active areas (Jeong et al., 8 Mar 2025, Villar-Corrales et al., 17 Feb 2025, Feng et al., 4 Nov 2025).

Key References