Visuo-Tactile Latent World Models

• Visuo-tactile latent world models are predictive frameworks that encode visual and tactile data into latent representations to model contact dynamics.
• They integrate modality-specific cues using mechanisms such as transformers, VAEs, and cross-modal attention to resolve ambiguities in contact states.
• These models enhance robotic manipulation tasks like grasping and pushing by improving prediction accuracy and control under occlusion and uncertainty.

Visuo-tactile latent world models are predictive or belief-updating models that encode visual and tactile observations into latent variables and use those variables to represent the evolution of contact-rich interaction. Across recent formulations, the latent state may function as an action-conditioned simulator, a Bayesian filtered belief over hidden physical properties, or a representation backbone for model-based reinforcement learning; the common premise is that vision provides global scene context, geometry, and robot kinematics, whereas tactile sensing provides local, contact-specific evidence about grasp, push, slip, deformation, and force-dependent state transitions that are weakly visible, ambiguous, or occluded in camera space (Higuera et al., 5 Feb 2026, Chen et al., 2022, Yuan et al., 24 Mar 2026, Zheng et al., 19 Mar 2026, Dutta et al., 2 Apr 2026).

1. Conceptual scope and definitions

The term does not denote a single canonical architecture. In the recent literature, it refers to a family of latent-variable systems whose unifying objective is to model contact-relevant dynamics more faithfully than vision-only approaches.

System	Latent/world-model role	Distinctive mechanism
VT-WM	Latent-state action-conditioned world model	Visual latent state $s_k$, tactile latent state $t_k$, autoregressive transformer predictor
VTT	Visuo-tactile latent state model for model-based RL	Self-attention and cross-modal attention, contact and alignment embeddings, latent heatmap
VTAM	Multimodal world modeling framework	Modality-transfer finetuning, alternating intra-view and cross-view attention, virtual force regularization
OmniVTA	Hierarchical slow-fast visuo-tactile world-model system	TactileVAE, two-stream VTWM, adaptive fusion policy, 60 Hz reflexive controller
CMLF	Deep state-space model / object-centric causal latent world model	Separate visual and tactile latent spaces, bidirectional cross-modal priors, Bayesian filtering

In "Visuo-Tactile World Models" (Higuera et al., 5 Feb 2026), the formulation is explicit: a visual latent state $s_k$ and tactile latent state $t_k$ are advanced by an action-conditioned transition,

$(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$

This makes the model a predictive latent-space simulator for imagination, rollout, and planning. By contrast, "Visuo-Tactile Transformers for Manipulation" (Chen et al., 2022) presents VTT primarily as a multimodal latent state encoder; SLAC supplies latent dynamics, reward prediction, and actor-critic learning. "VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs" (Yuan et al., 24 Mar 2026) and "OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation" (Zheng et al., 19 Mar 2026) both define predictive visuo-tactile world models more directly, while "Cross-Modal Visuo-Tactile Object Perception" (Dutta et al., 2 Apr 2026) casts the problem as Bayesian filtering over hidden physical causes.

A plausible implication is that the phrase visuo-tactile latent world model is best understood as a functional category rather than a single implementation template: what matters is that the latent state carries predictive or inferential structure about contact, not merely fused multimodal features.

2. Contact ambiguity as the central modeling problem

The central justification for visuo-tactile latent modeling is partial observability of contact. VT-WM states that vision alone is strong at global scene understanding and robot kinematics but weak at representing contact state—whether the hand is actually touching, grasping, pushing, or merely hovering near an object. It identifies characteristic vision-only failure modes under occlusion or ambiguous contact states, including objects disappearing, teleporting, or moving in ways that violate basic physics (Higuera et al., 5 Feb 2026). The paper further notes that visually similar states can have very different futures: the hand may appear nearly identical when it is not touching an object, lightly touching it, firmly grasping it, or about to lose contact.

VTT makes a related argument at the representation level. It observes that prior visuo-tactile methods often learn vector embeddings in $\mathbb{R}^n$, but such embeddings can blur fine spatial structure and locality precisely where small changes in image space correspond to contact/no-contact transitions. The motivation for token-level fusion and latent heatmaps is therefore not only multimodal aggregation, but preservation of spatially localized task relevance for manipulation (Chen et al., 2022).

VTAM extends the same diagnosis to force-sensitive interaction. It argues that modern video-action or VLA-style models are strong at semantic reasoning and long-horizon visual prediction, yet remain limited in contact-rich scenarios because visual tokens do not reliably encode incipient slip, surface compression, deformation, friction changes, or stable contact transitions. In that account, visual-only latent spaces are optimized for semantic alignment rather than physical prediction, and late tactile fusion remains brittle because the model must still infer force physics from semantic correlations (Yuan et al., 24 Mar 2026).

CMLF generalizes the observability argument from contact state to object properties. It treats geometry, pose, mass, inertia, stiffness, compliance, surface friction, and stick-slip-like behavior as latent causes that are only indirectly observable from sensory streams and are nonlinear, temporally extended, and action-dependent. Vision is informative about extrinsic properties such as shape, size, color, and surface texture; touch is informative about intrinsic properties such as stiffness, friction, mass, and compliance; neither modality alone is sufficient under occlusion, noise, or temporally sparse tactile contact (Dutta et al., 2 Apr 2026).

This suggests that, in this literature, the latent variable is not merely a compression bottleneck. It is the locus where hidden contact mechanics, object properties, and cross-modal ambiguities are resolved over time.

3. Architectural families and latent-state design

A prominent architectural pattern is the use of modality-specific encoders followed by a sequence model over latent tokens. VT-WM instantiates this pattern with a pretrained Cosmos tokenizer for RGB video and Sparsh-X for Digit 360 tactile images, collected from four Digit 360 fingertip sensors mounted on an Allegro Hand on a Franka Panda arm. The resulting tokens are augmented with sinusoidal positional embeddings, projected into a unified tensor space, concatenated along the spatial dimension, and passed to a 12-layer transformer with factorized spatio-temporal self-attention, action cross-attention, and RoPE. Spatial attention operates within a timestep, temporal attention propagates information across time, and modality-specific heads predict $\hat{s}_{k+1}$ and $\hat{t}_{k+1}$ (Higuera et al., 5 Feb 2026).

VTT extends the Vision Transformer to visuo-tactile input. An $84 \times 84 \times 3$ image is split by 2-D convolution into 36 visual patches, while a wrist-mounted force-torque sensor provides a $1 \times 6$ wrench represented as 2 tactile tokens. The concatenated tokens pass through stacked attention-fusion layers that explicitly decompose self-attention and cross-modal attention, so that vision attends to touch, touch attends to vision, and each modality also attends to itself. VTT adds three learned embeddings—contact embedding $t_k$0, alignment embedding $t_k$1, and position/modality embedding $t_k$2—and compresses the fused output to a latent vector $t_k$3. Its characteristic representational object is the latent heatmap, a spatially distributed latent representation in which attention weights across visual patches highlight task-relevant regions before and after contact (Chen et al., 2022).

VTAM begins from a pretrained video backbone rather than from separate tactile pretraining. It encodes third-person visual view, first-person visual view, and GelSight tactile stream into a shared continuous latent space using a pretrained video VAE, then processes the multimodal sequence through alternating intra-view self-attention and cross-view self-attention. The resulting latent representation conditions a conditional diffusion-based action head that predicts robot action $t_k$4, proprioceptive state $t_k$5, and a tactile-derived force proxy $t_k$6 (Yuan et al., 24 Mar 2026).

OmniVTA places a dedicated tactile latent model at the front of the pipeline. Its TactileVAE is a self-supervised variational autoencoder over 3D marker displacement, not tactile images, motivated by the claim that marker displacement directly captures the deformation field caused by contact. The encoder is a causal 3D convolution-based VAE, and the decoder is an implicit neural representation (INR): $t_k$7 On top of this tactile latent space, the VTWM uses two branches—visual and tactile—implemented as spatio-temporal diffusion transformers for future latent prediction (Zheng et al., 19 Mar 2026).

CMLF adopts a different structural decomposition. Rather than tokenize modalities into a single shared sequence, it uses separate latent spaces for vision and touch, and within each modality partitions the state into directly observable factors $t_k$8 and indirectly observable factors $t_k$9. The factorization

$s_k$0

is then embedded in a deep state-space model with inverse variational measurement models, action-conditioned transition models, latent dynamics for indirect variables, and bidirectional cross-modal prior functions $s_k$1 (Dutta et al., 2 Apr 2026).

4. Training objectives, rollout mechanisms, and control loops

VT-WM combines one-step supervision with sampled multi-step training. For a sequence of $s_k$2 frames, the teacher forcing loss is

$s_k$3

and the sampling loss over $s_k$4 autoregressive steps is defined analogously on sampled trajectories, with sampled states generated without gradients. The final objective is

$s_k$5

Planning uses the world model as a simulator inside Cross-Entropy Method (CEM) planning with a vision-only goal. The model encodes current RGB context, current touch context, and a goal RGB image; action sequences are sampled, rolled out in latent space, and scored by latent distance $s_k$6. The appendix reports a concrete CEM configuration with action dimension $s_k$7, horizon $s_k$8 seconds, frequency $s_k$9, particles $t_k$0, iterations $t_k$1, and up to 3 planning trials, after which the best action sequence is executed open-loop on the real robot (Higuera et al., 5 Feb 2026).

VTAM is trained in two stages. In Stage I, the backbone is fine-tuned to model future multimodal latent dynamics with a flow matching objective,

$t_k$2

To keep tactile information active, VTAM introduces a virtual force proxy derived from GelSight deformation: $t_k$3 with $t_k$4. Stage II then predicts a joint latent variable $t_k$5 and minimizes

$t_k$6

The paper describes the force term as a tactile regularization that prevents visual latent dominance and modality collapse in the action model (Yuan et al., 24 Mar 2026).

OmniVTA trains prediction and control as a hierarchy. Its VTWM uses a diffusion loss over future latent observations, augmented with dynamic-aware weighting and amplitude weighting so that prediction errors in rapid changes and strong tactile responses are emphasized. The downstream Adaptive Visuo-Tactile Fusion Policy (AFP) introduces a Latent Tactile Differential (LTD) Encoder: $t_k$7 where current tactile feature, predicted tactile feature, and their difference jointly represent impending contact change. A contact probability is predicted from this tactile representation, and a gating network produces normalized weights $t_k$8 and $t_k$9, with fused feature

$(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$0

The slow policy predicts 6 actions at 15 FPS, later interpolated to 60 Hz, while the Reflexive Latent Tactile Controller (RLTC) outputs high-frequency refinements at 60 Hz from aligned observed and predicted tactile latents (Zheng et al., 19 Mar 2026).

CMLF formulates learning as variational Bayesian filtering. It models $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$1, approximates the posterior with a variational distribution over $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$2, and optimizes an ELBO containing visual and tactile reconstruction terms together with KL regularization on both directly observable and indirectly observable latent states. Its filtered update combines a measurement likelihood from current observations, an action-conditioned dynamics prior, and a cross-modal prior from the other modality. In this sense, the latent state is updated rather than merely re-encoded at each step (Dutta et al., 2 Apr 2026).

5. Datasets, task regimes, and empirical findings

VT-WM is trained on a multi-task teleoperated dataset collected with a Franka Panda arm, Allegro Hand, Digit 360 fingertips, and an exocentric camera. The training set comprises eight contact-rich tasks—pick and place on a plate, reach and press a button, push, wipe with a cloth, lampshade insertion, table leg insertion, cube stacking, and scribbling with a marker—with 124 demonstrations, 112k datapoints, and average demonstration length of about 40 s; validation uses 26 demonstrations and 17k datapoints. Against a vision-only baseline, VT-WM achieves 33% better performance at maintaining object permanence and 29% better compliance with the laws of motion in autoregressive rollouts. In zero-shot real-robot planning on five tasks, it reaches up to 35% higher success rates, with reported gains of +10% on push fruits, +35% on reach push, +31% on wipe cloth, and +11% on stack cubes. In a transfer setting on place plate in dish rack, fine-tuning with only 20 demonstrations yields 77% success over 10 planning trials (Higuera et al., 5 Feb 2026).

VTT is evaluated on four PyBullet tasks—Pushing, Door-Open, Picking, and Peg-Insertion—and on a real-world block pushing task with a Franka Emika Panda arm, an ATI Gamma force-torque sensor, and two Intel RealSense D435 cameras. The reported result is that VTT outperforms Concatenation and Product of Experts (PoE) in nearly all tasks, with higher sample efficiency, higher success rate in all but Peg-Insertion, and lower variance in learning curves. The ablation study attributes performance to the cross-modal attention structure, the contact and alignment objectives, and latent compression, rather than to model size alone (Chen et al., 2022).

VTAM is evaluated on a real-robot setup with an xArm6 manipulator, parallel gripper, GelSight Mini tactile sensor, two Intel RealSense D455 cameras, and teleoperated demonstrations collected at 30 Hz. The task suite comprises potato chip pick-and-place, cucumber peeling, and whiteboard wiping. The paper reports that VTAM achieves a 90% average success rate, with 90% success on chip pick-and-place, 85% on cucumber peeling, and 95% on whiteboard wiping. It further reports that, in challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Its ablation table is unusually stark: Vision-only $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$3, Late-fusion tactile $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$4, No virtual-force regularization $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$5, and VTAM $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$6 (Yuan et al., 24 Mar 2026).

OmniVTA is built on the OmniViTac dataset, which contains 21,879 synchronized trajectories, 86 tasks, and 100+ objects across six physics-grounded interaction patterns—wiping, peeling, cutting, grasping, assembly, in-hand adjustment—and five semantic scenarios. TactileVAE is pretrained on about 1.2M tactile samples. The paper reports that TactileVAE outperforms PCA and a point-cloud autoencoder across all tasks, and that VTWM yields lower $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$7 and higher cosine similarity than UVA, exUMI, KineDex, and ForceMimic at the 2nd, 4th, and 6th latent prediction steps. In real-robot manipulation, the full system achieves the strongest success rates, including Wipe: 0.80 object diversity / 0.58 generalization / 0.60 perturbation; Peel: 0.55 / 0.48 / 0.63; Cut: 0.85 / 0.83 / 0.60; Assembly: 0.60 / 0.50 / 0.40; Grasp: 0.90 object diversity; and Adjustment: 0.65 / 0.65 (Zheng et al., 19 Mar 2026).

CMLF is evaluated on a custom dataset of 75 configurable non-rigid objects, 16 interaction configurations, 4 grip force levels, 4 linear/angular velocities, 3 repetitions per object per configuration, 48 trials per object, and 3600 trajectories total. The model is tested under property estimation, noise/corruption, and surprise-object conditions. The full bidirectional model w-CM consistently outperforms the Baseline, Joint, and wo-CM variants with lower mean inference error, faster convergence, lower steady-state error, and better robustness under corruption. The paper also reports pose estimates comparable to state-of-the-art supervised methods despite the absence of explicit pose supervision (Dutta et al., 2 Apr 2026).

6. Misconceptions, limitations, and open technical issues

A recurring misconception is that any visuo-tactile encoder constitutes a world model. The literature is more precise than that. VTT is explicitly described as primarily a multimodal latent state encoder rather than a self-contained predictive world model; SLAC supplies latent dynamics, reward prediction, and policy learning. Its limitations are correspondingly specific: tactile input is only a 6D wrist wrench, experiments are restricted to rigid-body tasks, and the auxiliary contact and alignment objectives depend on labeled or constructible positive/negative pairings (Chen et al., 2022). Conversely, VT-WM is a genuine latent dynamics model, but its planning objective remains vision-only at the goal level: tactile sensing grounds rollout dynamics, yet the goal representation is encoded only from RGB (Higuera et al., 5 Feb 2026).

A second misconception is that adding touch anywhere in the stack is sufficient. VTAM argues against late tactile injection directly, and its ablations show that raw tactile input alone is not enough: Late-fusion tactile and Vision-only both score $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$8, whereas removing virtual-force regularization drops the result to $(s_{k+1}, t_{k+1}) \sim P_\phi(s_k, t_k \mid a_k).$9. OmniVTA makes a related claim from a different angle: existing methods often treat tactile signals as passive observations rather than using them to model contact dynamics or to enable closed-loop control explicitly, and the full system outperforms its open-loop or reduced variants especially under perturbation and geometry change (Yuan et al., 24 Mar 2026, Zheng et al., 19 Mar 2026). A plausible implication is that the decisive design choice is not whether touch is present, but whether tactile information shapes predictive contact dynamics and downstream control.

A third open issue concerns cross-modal priors under distribution shift. CMLF reports that learned priors can become initially misleading on surprise objects that violate training correspondences such as height–mass, shape–stiffness, or texture–friction relationships. It also reports that delayed activation of cross-modal priors performs better than early activation, suggesting that stable unimodal representations are a prerequisite for useful cross-modal binding. The paper explicitly identifies future causal gating, improved online prior adaptation, and richer tactile tracking of geometric deformations and transient events as unresolved problems (Dutta et al., 2 Apr 2026).

Across the literature, the most stable conclusion is narrow but technically consequential: tactile sensing is most valuable not as an auxiliary feature channel, but as a latent variable source for modeling hidden contact state, future contact evolution, and uncertainty about the physical consequences of action. In that restricted sense, the visuo-tactile latent world model is a model of contact itself.