Sensorimotor World Model (SMWM)

• Sensorimotor World Models are computational architectures that learn the lawful coupling between motor commands and resulting sensory state transitions.
• They utilize methods such as probabilistic prediction, clustering, and recurrent neural networks to generate structured, action-conditioned representations.
• SMWMs enable goal-directed control and robust transfer in robotics, cognitive modeling, and machine learning through grounded, predictive modeling of sensorimotor interactions.

A Sensorimotor World Model (SMWM) is a computational or neural architecture that enables an embodied agent—biological or artificial—to learn, represent, and deploy the structured regularities linking its own motor commands to the resulting trajectories of sensory states. SMWM research transcends traditional "world modeling" by grounding representations explicitly in the space of sensorimotor contingencies: the lawful, agent-specific couplings between action and perception. SMWMs have been formulated as predictive probabilistic models, graph-structured transition systems, neural architectures with explicit action–conditionality, and self-organizing representations supporting both perception and goal-directed control. This entry systematically surveys formalizations, training methodologies, representational properties, and key applications of SMWMs, with reference to major recent contributions across robotics, cognitive modeling, and machine learning.

1. Formal Structure and Mathematical Foundations

At the core of all SMWM formulations lies an action-conditional predictive model of the form

$P(s_{t+1} \mid s_t, a_t)$

where $s_t$ is the agent's sensory state at time $t$ (possibly high-dimensional and multimodal), and $a_t$ is its motor command or action. This sensorimotor transition model can be instantiated as:

• Discrete tensor models: State–action–state tensors storing empirical transition counts or probabilities between quantized prototypes, as in minimalistic SMWMs aiming to discover objects as spatio-temporally invariant structures via clustering (Hir et al., 2018, Laflaquière et al., 2016).
• Probabilistic generative models with latent contexts: Hierarchical, context-specific forward models, where latent variables $z_t$ index the current contingency or sensorimotor regime, evolving as

$$p(z_t \mid z_{t-1}, a_{t-1}), \quad p(s_t \mid z_t)$$

and inference proceeds by minimizing a variational free energy objective (Hemion, 2016, Baltieri et al., 2019).

• Recurrent neural architectures: SMWM states generated by recurrent or memory-augmented encoders integrating sensory and action histories as

$s_t = \phi(o_{0:t}, a_{0:t-1})$

with future observations predicted from these internal codes (Kulak et al., 2018).

• Spatially structured neural fields: Continuous or discretized neural fields $h_t$ with local lateral connectivity and multiplicative motor gating, evolving under equations such as

$$h_{t+1} = h_t + \Delta t/\tau [-h_t + K * \mathrm{ReLU}(h_t) + W_{in}*I_t], \quad h_{t+1}^{(i)} \leftarrow m_i(t)h_{t+1}^{(i)}$$

preserving the topology and geometry of physical space (Nunley, 21 Feb 2026).

• Action-aligned latent models: Latent-world models in which embeddings are trained with both predictive and inverse-dynamics regularization to ensure that latent states encode the controllable degrees of freedom and prevent collapse (Ivashkov et al., 18 Jun 2026).

Across these frameworks, the common structure is an explicit mapping from sensorimotor pasts (including both observation and action history) to a state (or distribution over states), together with learning objectives that prioritize predictive sufficiency and action-relevance over veridical environmental reconstruction.

2. Learning Procedures and Algorithmic Instantiations

SMWM training proceeds via unsupervised or self-supervised extraction of sensorimotor regularities from continuous streams of agent-environment interaction:

• Random motor babbling: Naive agents explore the space of motor commands (e.g., saccades (Laflaquière, 2016, Laflaquière, 2018), limb movements (Spisak et al., 24 Apr 2025)) and empirically estimate state–action–state transition matrices from the resulting sensory outcomes.
• Clustering and discretization: High-dimensional raw sensory data are clustered (e.g., by k-means) into prototypes per receptive field or sensor patch, transforming continuous input into tractable symbolic spaces (Laflaquière, 2016, Laflaquière, 2018, Hir et al., 2018).
• Spectral graph analysis: Transition tensors are reduced to similarity graphs, with densely connected subgraphs discovered via spectral clustering to reveal invariant structures (proto-objects) (Hir et al., 2018, Laflaquière et al., 2016, Hemion, 2016).
• End-to-end neural optimization: Neural SMWMs are trained by back-propagation on predictive losses, optionally augmented by action-inversion or contrastive objectives, often in an offline, reward-free regime (Kulak et al., 2018, Ivashkov et al., 18 Jun 2026, Radosavovic et al., 2023).
• Active inference and free energy minimization: Generative/recognition pairs are optimized to minimize variational free energy, balancing predictive accuracy against model parsimony and encoding only action-relevant couplings (Baltieri et al., 2019).

Pseudocode for these algorithms typically alternates between exploration (data gathering), representation learning (clustering/transition estimation or neural optimization), and optionally planning/goal-directed rollout in the learned model.

3. Representational and Computational Properties

SMWMs routinely demonstrate the following features across environments and architectures:

• Grounding of perception in action: Only those features of sensory input that are reliably and predictably modulated by the agent’s own actions are represented; extraneous “uncontrollable” factors are actively disregarded (Ivashkov et al., 18 Jun 2026, Baltieri et al., 2019).
• Emergence of structured latent spaces: SMWMs identify controllable subspaces of the environment (e.g., object locations, agent pose, manipulable features), with the dimensionality of the latent representation matching the dimensionality of control or context (Ivashkov et al., 18 Jun 2026, Kulak et al., 2018).
• Hierarchical and context-specific encoding: Latent state discovery mechanisms (spectral clustering, PB units) separate distinct regimes of coupling (contexts), supporting compositionality in learned predictions (Hemion, 2016, Zhong et al., 2020).
• Spatial/topological fidelity: Neural fields and isomorphic models maintain pixel-wise or spatially local correspondences, supporting smooth prediction of physical phenomena and action outcomes (e.g., trajectory unfolding, body schema emergence) (Nunley, 21 Feb 2026).
• Interpretability and transfer: Learned states support transfer across tasks, environments, and morphological variation, as exhibited in large-scale robotics (RPT) (Radosavovic et al., 2023) and embodied LLMs (Varela et al., 25 May 2025).

4. Quantitative Results and Experimental Performance

SMWM models have been evaluated in domains ranging from developmental robotics and artificial perception to real-world manipulation. Key empirical findings include:

SMWM Domain/Architecture	Sample Result	Reference
Visual field grounding by saccades	100% success on foveal visual search task; high MI for correct blocks	(Laflaquière, 2016)
Predictive compaction/test MSE	1.0–2.4×10⁻³ MSE, Recurrent-SM encoder in room navigation	(Kulak et al., 2018)
Latent state object discovery	Cluster purity 100% in discrete contexts; emergence of invariant subgraphs	(Hir et al., 2018)
Inverse-dynamics SMWM (2D nav.)	99% planning success; latent PCs capture physical topology	(Ivashkov et al., 18 Jun 2026)
Multimodal LLM-robot self modeling	Mean entity-awareness score 3.27/5; ablation reveals vision/memory criticality	(Varela et al., 25 May 2025)
Robot sensorimotor pre-training	2× improvement on hardest stacking; robust zero-shot robot/lab transfer	(Radosavovic et al., 2023)
Infant mobile paradigm simulation	Δa_connected–a_unconnected ≈0.1–0.2 in <1 min; ablations confirm necessity of prediction/exploration	(Spisak et al., 24 Apr 2025)

Notably, SMWMs consistently demonstrate rapid emergence of structured and goal-relevant representations, robust transfer, and the ability to model or drive behavior even with incomplete environmental knowledge.

5. Theoretical Significance and Relation to Perception Theories

SMWM research is deeply informed by the Sensorimotor Contingencies Theory (SMCT) and the predictive processing/free energy paradigm:

• SMCT: Perception is not the direct mapping of sensory input to meaning, but the mastery of action–perception contingencies—how sensations transform as a result of the agent’s own movements (Hemion, 2016). Objects are defined as those parts of the sensorimotor flow whose internal regularities are invariant across contexts (Hir et al., 2018, Laflaquière et al., 2016).
• Predictive processing/free energy: Internal models are optimized for "actionable" prediction, not veridical or exhaustive environmental description. SMWM is the minimal generative model sufficient for goal-directed sensorimotor loop closure, as formalized via variational free energy or state transition priors (Baltieri et al., 2019).
• Perception-for-action: SMWMs implement the principle that perceptual representation should be shaped by relevance for control, producing action-aligned latent spaces and discarding distractors (Ivashkov et al., 18 Jun 2026).

These theoretical foundations explain why SMWMs can efficiently support both model-based and model-free reinforcement learning and underpin developmental phenomena such as infant contingency learning (Spisak et al., 24 Apr 2025).

6. Limitations, Open Problems, and Future Directions

Despite significant progress, challenges remain in the generalization and extension of SMWMs:

• Scalability: Many SMWM implementations operate with discretized or clustered sensory representations, limiting direct scaling to raw high-dimensional input (vision, touch).
• Continuous action spaces and online adaptation: Robust learning in continuous, unbounded motor spaces remains underexplored, as does real-time online adaptation in changing environments (Laflaquière, 2016).
• Hierarchical and semantic abstraction: Existing architectures are predominantly flat; the discovery and compression of abstract, semantic or compositional sensorimotor patterns is an open research direction (Laflaquière, 2018, Hemion, 2016).
• Robustness to occlusion/ambiguity: Identifying objects or contexts in the presence of overlapping structures, ambiguity, or partial observability requires more sophisticated hierarchical or memory-augmented strategies (Hir et al., 2018, Kulak et al., 2018).
• Integration with high-level cognition and language: SMWM-augmented LLMs point to architectures capable of integrating episodic memory, inference, and causal modeling (Varela et al., 25 May 2025). However, principled methods for merging sensorimotor grounding with symbolic and linguistic reasoning remain an area of rapid development.

Prospective advances include end-to-end neural SMWMs for continuous visual and motor streams, intrinsic-motivation-driven active exploration, hierarchical models encoding multi-scale contingencies, and architectures supporting robust model-based planning and transfer in unstructured, real-world environments.

---

Key References:

• “Autonomous Grounding of Visual Field Experience through Sensorimotor Prediction” (Laflaquière, 2016)
• “Grounding the Experience of a Visual Field through Sensorimotor Contingencies” (Laflaquière, 2018)
• “Identification of Invariant Sensorimotor Structures as a Prerequisite for the Discovery of Objects” (Hir et al., 2018)
• “Discovering Latent States for Model Learning” (Hemion, 2016)
• “Generative models as parsimonious descriptions of sensorimotor loops” (Baltieri et al., 2019)
• “Sensorimotor features of self-awareness in multimodal LLMs” (Varela et al., 25 May 2025)
• “Sensorimotor World Models: Perception for Action via Inverse Dynamics” (Ivashkov et al., 18 Jun 2026)
• “Robot Learning with Sensorimotor Pre-training” (Radosavovic et al., 2023)
• “A computational model of infant sensorimotor exploration in the mobile paradigm” (Spisak et al., 24 Apr 2025)
• “Neural Fields as World Models” (Nunley, 21 Feb 2026)