VTAM: Video-Tactile Action Model

• VTAM is a multimodal framework that integrates visual and tactile data to generate action policies for precise contact-rich manipulation.
• It employs design strategies such as lightweight conditioning, token-level tactile injection, and asymmetric attention to enhance prediction accuracy.
• Empirical results demonstrate significant improvements in task success rates for applications like peg insertion, cable plugging, and touch-sensitive assembly.

A Video-Tactile Action Model (VTAM) is a multimodal robot policy or world-action model that combines video or visual observations with tactile sensing, and often language and proprioception, to produce actions for contact-rich manipulation. The term is explicit in "VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs" (Yuan et al., 24 Mar 2026), while related work uses the same framing to describe systems such as TacFiLM, TacCoRL, OmniVTA, Tactile-WAM, and VT-WAM (Morissette et al., 15 Mar 2026, Ma et al., 10 Jun 2026, Zheng et al., 19 Mar 2026, Wu et al., 25 Jun 2026, Tian et al., 2 Jul 2026). Across these formulations, the central premise is consistent: vision alone rarely encodes the instantaneous contact geometry, local friction, compliance, shear forces, slip, jamming, or deformation patterns that determine success in insertion, assembly, peeling, wiping, grasp stabilization, and other contact-rich behaviors.

1. Origins, scope, and conceptual definition

The explicit VTAM formulation emerged in 2026 as an extension of Video-Action Models and Vision-Language-Action policies to contact-rich interaction, with touch treated as a complementary grounding signal rather than a late auxiliary input (Yuan et al., 24 Mar 2026). In this formulation, a pretrained video transformer is augmented with tactile streams, future visuo-tactile dynamics are modeled jointly, and actions are decoded through a conditional diffusion or flow-matching head. Closely related 2026 work placed the same idea in world-model form: OmniVTA described a world-model-based visuo-tactile manipulation framework that predicts short-horizon contact evolution and closes the loop with a 60 Hz tactile controller, while VT-WAM and Tactile-WAM framed the problem as joint prediction of future visual states, future tactile states, and action chunks within denoising-based World Action Models (Zheng et al., 19 Mar 2026, Tian et al., 2 Jul 2026, Wu et al., 25 Jun 2026).

Earlier work anticipated this shift before the VTAM name became common. "Action Conditioned Tactile Prediction: case study on slip prediction" stated that VTAM maps directly to ACTVP in that paper’s context, because ACTVP predicts tactile signals represented as an image and conditions those predictions on robot actions (Mandil et al., 2022). "Combining Vision and Tactile Sensation for Video Prediction" defined a best-performing fused video-tactile, action-conditioned predictor through the SPOTS family, with SPOTS-small identified as the recommended instantiation (Mandil et al., 2023). These precursors centered on prediction rather than direct manipulation success, but they established the now-standard claim that tactile signals reduce uncertainty about contact events and latent physical properties when visual scenes are ambiguous.

VTAM is therefore not a single architecture. It names a family of models in which tactile information is fused with video or visual context so that action generation is conditioned on physically informative contact state. This suggests that the distinguishing property of VTAM is not merely multimodality, but the use of tactile signals as action-relevant state for dynamics prediction, control, or both.

2. Architectural motifs and fusion strategies

One major VTAM design pattern is lightweight conditioning of a pretrained visual or VLA backbone. TacFiLM instantiates VTAM by defining a vision encoder $E_v$, tactile encoder $E_t$, optional language encoder $E_l$, a FiLM-based fusion mechanism, and an LLM action head. At time $t$, it computes

$$v_t = E_v(V_{1:t}), \qquad \tau_t = E_t(T_{1:t}), \qquad \ell = E_l(\text{text}),$$

then applies tactile-conditioned FiLM inside selected ViT blocks,

$h'_t = \gamma(\tau_t) \odot h_t + \beta(\tau_t),$

and autoregressively emits discretized robot actions with a decoder-only LLaMA 2 7B (Morissette et al., 15 Mar 2026). FiLM is inserted after normalization and before multi-head self-attention, and by default into all ViT blocks, although early, middle, and late subsets were also evaluated. The stated motivation is efficiency: FiLM adds only small MLPs for $\gamma,\beta$ and avoids increasing token length.

A second pattern uses token-level tactile injection with explicit contact-aware gating. TacCoRL augments a $\pi 0.5$-style VLA flow policy with tactile histories, encodes tactile tokens with a CNN-based temporal encoder, and applies a binary gate

$$c_t = I\!\left[\max_{r \in [t-L+1,t]} \sum_{k=1}^{K} I(|f_{r,k}|>\lambda_f) \ge m\right]$$

so that tactile tokens are removed from attention outside contact (Ma et al., 10 Jun 2026). Cross-attention updates the vision-language-proprioception context only when the gate is active. The intent is to preserve pretrained VLA behavior in non-contact phases while letting tactile evidence modulate action chunks in near-failure states.

A third pattern, characteristic of recent world-action models, is asymmetric routing of tactile influence. Tactile-WAM identifies "tactile pollution," defined as degradation of video and action prediction caused by unconstrained tactile-token injection into a visual dynamics model. Its Tactile Asymmetric Attention Mechanism combines a VideoClean mask, which blocks video-query access to tactile key/value tokens, with a touch-aware bias applied only to action-query–tactile-key attention logits (Wu et al., 25 Jun 2026). VT-WAM uses a closely related asymmetry through Asymmetric Mixture-of-Transformers attention: action tokens attend to the full tactile sequence and only to a first-frame visual anchor, tactile tokens also read the visual anchor, and visual tokens do not attend to tactile or action tokens (Tian et al., 2 Jul 2026). In both cases, tactile signals are made accessible to action generation while the visual path is intentionally protected.

A fourth pattern treats tactile sensing as another view inside a shared latent world model. The 2026 VTAM paper encodes third-person RGB, first-person RGB, and GelSight tactile streams with the same video VAE, treats tactile as "view 3," and alternates intra-view self-attention with cross-view self-attention in a 28-layer LTX-Video backbone (Yuan et al., 24 Mar 2026). OmniVTA similarly couples visual latents from SD-VAE with tactile latents from TactileVAE inside a two-stream diffusion world model, then uses predicted tactile futures in an Adaptive Fusion Policy and a 60 Hz reflexive tactile controller (Zheng et al., 19 Mar 2026). These designs move beyond tactile-conditioned policy inference and instead make contact evolution itself a prediction target.

3. Learning objectives, supervision, and optimization

Behavior cloning remains a common training regime for VTAMs built on pretrained policies. TacFiLM uses a standard autoregressive loss over discretized action tokens,

$$L_{BC}(\theta) = -\sum_t \log \pi_\theta(a_t \mid h'_t, \ell),$$

with LoRA applied to the linear layers of the augmented VLA while most of the backbone remains frozen (Morissette et al., 15 Mar 2026). The paper emphasizes that no additional multimodal pretraining is required. A dataset-oriented counterpart appears in "A Humanoid Visual-Tactile-Action Dataset for Contact-Rich Manipulation," where multimodal ACT-style imitation is formulated as continuous action prediction,

$E_t$0

with $E_t$1 or $E_t$2 regression losses and an optional alignment term

$E_t$3

for vision-tactile consistency (Kwon et al., 28 Oct 2025).

World-model VTAMs generally replace direct next-action training with denoising or flow-matching objectives over future states and actions. VT-WAM trains visual, tactile, and action experts jointly through

$E_t$4

augmented by the contact-gated AVTAG loss that penalizes visual-dominant action attention during contact phases (Tian et al., 2 Jul 2026). Tactile-WAM uses a total objective

$E_t$5

where auxiliary proxy heads supervise touch-state and touch-change signals distilled from tactile image motion (Wu et al., 25 Jun 2026). OmniVTA also uses diffusion objectives, but it adds tactile-specific dynamic-aware weighting and amplitude weighting to emphasize high-frequency contact changes and contact intensity in the tactile branch (Zheng et al., 19 Mar 2026).

TacCoRL introduces a two-stage optimization strategy that combines imitation on mixed simulated and real trajectories with PPO in a real-aligned tactile simulator. The co-training loss is

$E_t$6

followed by joint RL and supervised anchoring to real data,

$E_t$7

This is explicitly motivated by the rarity of near-failure corrective contact states in demonstrations and the risk of collecting them extensively on hardware (Ma et al., 10 Jun 2026).

The 2026 VTAM paper adds a distinct regularization mechanism: a deformation-derived virtual force proxy

$E_t$8

and a force loss

$E_t$9

embedded in the Stage II objective

$E_l$0

The stated purpose is to prevent "visual latent dominance" by ensuring that the action head must preserve tactile sensitivity to reduce the joint loss (Yuan et al., 24 Mar 2026).

4. Datasets, sensors, synchronization, and physical observability

VTAM research spans both real-robot collections and simulation-first ecosystems. TacFiLM uses a Franka Emika Panda arm with a DIGIT sensor on the gripper, an Intel RealSense RGB camera, Polymetis with libfranka over FCI at 1 kHz, and observation logging aligned at 10 Hz. Tasks include peg-in-hole with circle, square, and pentagon pegs at 2 mm and 3 mm clearance, plus USB and HDMI cable plugging (Morissette et al., 15 Mar 2026). TacCoRL uses two wrist-mounted RealSense D405 cameras, one fixed RealSense D415, and two FlexiTac-V2 tactile pads mounted on opposing gripper contact surfaces, with a total policy-facing taxel count of $E_l$1 (Ma et al., 10 Jun 2026). The 2026 VTAM paper uses two Intel RealSense D455 RGB-D cameras and a GelSight Mini on an xArm6, with all streams synchronized at 30 Hz and processed as 9-frame temporal windows at 192×256 resolution (Yuan et al., 24 Mar 2026).

Several papers define broader data resources rather than single-task evaluations. OmniViTac contributes 21,879 synchronized visuo-tactile-action trajectories across 86 tasks and 100+ objects, organized into six physics-grounded interaction patterns: Wiping, Peeling, Cutting, Grasping, Assembly, and In-hand Adjustment. Its visual streams are recorded at 30 Hz, tactile streams run at native high frequency depending on the sensor, and cross-modal timestamp alignment is reported with temporal error below 10 ms (Zheng et al., 19 Mar 2026). The humanoid dataset in "A Humanoid Visual-Tactile-Action Dataset for Contact-Rich Manipulation" contains 101.9k samples across four soft-object tasks—Towel Strong, Towel Weak, Sponge Strong, and Sponge Weak—with two visual streams and dense tactile arrays comprising 1,062 sensors per hand, for 2,124 sensors across both hands (Kwon et al., 28 Oct 2025). UniVTAC provides a simulation-driven alternative: it supports GelSight Mini, ViTai GF225, and Xense WS, contributes approximately 205,826 samples from 14 geometric indenter primitives, and pairs synthesized tactile images with privileged supervision such as marker positions, gel depth, pure contact images, and object pose (Chen et al., 10 Feb 2026).

Sensor modality strongly influences what a VTAM can represent. High-resolution visuotactile sensors such as DIGIT and GelSight provide surface deformation and slip cues under occlusion (Morissette et al., 15 Mar 2026, Yuan et al., 24 Mar 2026). Dense taxel arrays offer pressure distributions and strong or weak contact condition diversity in anthropomorphic hands (Kwon et al., 28 Oct 2025). Marker-based optical tactile systems provide displacement fields $E_l$2 whose tangential components encode shear and whose normal component correlates with pressure, which is precisely the representation exploited by OmniVTA’s tactile encoder and controller (Zheng et al., 19 Mar 2026). This suggests that VTAM design is constrained not only by model class but by the sensor’s ability to expose contact structure at the required spatial and temporal scale.

5. Empirical performance and representative results

Across insertion and plugging tasks, TacFiLM reports average success of 86.67% on ID tasks and 86.67% on OOD tasks, direct insertion rates of 31.11% and 29.33%, average maximum forces of $E_l$3 N and $E_l$4 N, and average times of $E_l$5 s and $E_l$6 s. Selected examples are more specific: on ID circle peg 3 mm, OpenVLA-OFT achieves 86.67% success and 3.33% direct insertion, while TacFiLM reaches 100.00% success and 36.67% direct insertion; on OOD HDMI, OpenVLA-OFT reaches 6.67% success, TactileConcat 13.33%, and TacFiLM 66.67% (Morissette et al., 15 Mar 2026).

TacCoRL reports that the final visuo-tactile policy achieves an average real-robot success rate of 72.5%, compared to 50.0% for the vision-only baseline, across Test Tube Insertion, Do Puzzle, Assembly #1, and Assembly #2. Per-task real success is 70%, 45%, 95%, and 80%, while the vision-only RL post-training baseline reaches 35%, 25%, 80%, and 60% (Ma et al., 10 Jun 2026). In a different world-action setting, VT-WAM achieves a 71.67% average success rate across six real-world contact-rich manipulation tasks, with 81.67% on surface-interaction tasks and 61.67% on constrained insertion tasks; the paper states gains of +26.67% over Fast-WAM and +35.84% over OmniVTLA (Tian et al., 2 Jul 2026). Tactile-WAM reports 201/450 trials and 44.7% success on ManiFeel, compared with DreamZero RGB-only WAM at 26/450 trials and 5.8%, and further reports 175/200 trials and 87.5% on the contact-centric subset (Wu et al., 25 Jun 2026).

OmniVTA reports broad gains across six interaction categories. Examples include Wipe at 0.80 on object diversity versus 0.50 for RDP, Cut at 0.85 versus 0.65, and Adjustment at 0.65 versus 0.50; removing the 60 Hz reflexive controller reduces perturbation success in Wipe from 0.60 to 0.25 (Zheng et al., 19 Mar 2026). The explicit VTAM paper reports 90% on potato chip pick-and-place, 85% on cucumber peeling, and 95% on whiteboard wiping, averaging 90% across tasks; on chip pick-and-place it improves absolute success by 80% over the $E_l$7 baseline, from 10% to 90%. Its chip ablation is especially sharp: vision-only gives 0%, late-fusion tactile gives 0%, no virtual-force regularization gives 10%, and full VTAM gives 90% (Yuan et al., 24 Mar 2026).

The broader visuo-tactile ecosystem shows parallel evidence. UniVTAC reports 48.0% average benchmark success for ACT + UniVTAC Encoder versus 30.9% for vision-only ACT and 40.5% for VITaL, together with a real-world improvement from 43.3% to 68.3% average success (Chen et al., 10 Feb 2026). OCRA reports 90% average success on three visuo-tactile tasks versus 40% without tactile, and 85% average success across four vision-only tasks (Wang et al., 15 Mar 2026). Earlier predictive models support the same general direction: ACTVP reaches MAE 0.00631, PSNR 91.8266, and SSIM 0.9965 in tactile-video prediction, outperforming PMN, CDNA, and SVG on the reported dataset (Mandil et al., 2022), while SPOTS-small improves edge-case video prediction from MAE 0.0104, PSNR 77.534, SSIM 0.9586 for SVG to MAE 0.0091, PSNR 79.258, SSIM 0.9671 in a visually ambiguous pushing benchmark (Mandil et al., 2023).

6. Limitations, misconceptions, and open directions

A recurrent misconception is that VTAM is equivalent to concatenating tactile tokens to a visual policy. Multiple papers argue against this. TacFiLM notes that concatenation increases sequence length and compute, and performance deteriorates on harder tasks (Morissette et al., 15 Mar 2026). Tactile-WAM identifies tactile pollution as a failure mode in which unconstrained tactile-token injection degrades video and action prediction, and its ablations show that naively adding future tactile tokens reduces success from 5.8% to 1.3% before VideoClean and touch-aware bias are introduced (Wu et al., 25 Jun 2026). The VTAM ablation in chip grasping reaches the same conclusion from another angle: late-fusion tactile remains at 0% without predictive visuo-tactile world modeling and force-aware regularization (Yuan et al., 24 Mar 2026).

A second misconception is that effective VTAMs necessarily require large-scale tactile-language pretraining. TacFiLM states that no additional multimodal pretraining is required (Morissette et al., 15 Mar 2026). The 2026 VTAM paper states that its modality-transfer finetuning uses no tactile-language paired data and no independent tactile pretraining (Yuan et al., 24 Mar 2026). TacCoRL likewise emphasizes improvement without requiring large-scale tactile pretraining or extensive real-world contact exploration, though it does rely on a real-aligned simulator and a real-data anchor (Ma et al., 10 Jun 2026). A plausible implication is that the critical design question is not simply data volume, but how tactile information is routed into prediction and control.

Current limitations are also consistent across papers. TacFiLM evaluates on insertion and cable plugging on a single robot and sensor, and explicitly notes potential failures from sensor noise, latency, or timestamp misalignment (Morissette et al., 15 Mar 2026). TacCoRL identifies setup effort for digital twins, controller identification, and tactile calibration, and notes that highly deformable objects or fluids remain difficult to simulate (Ma et al., 10 Jun 2026). OmniVTA notes sensitivity to tactile calibration and alignment, cross-sensor domain shift, and possible prediction drift under large disturbances (Zheng et al., 19 Mar 2026). VT-WAM and Tactile-WAM both report that search, exploration, regrasping, and long-horizon recovery remain harder than contact correction itself (Tian et al., 2 Jul 2026, Wu et al., 25 Jun 2026). The humanoid soft-object dataset similarly reports that dense tactile features clearly separate task distributions in t-SNE, yet action MAE differences between dense and sparse inputs remain modest under current optimization, indicating that high-dimensional tactile signals are still difficult to exploit fully (Kwon et al., 28 Oct 2025).

Open directions described in the literature include extending TacFiLM beyond OpenVLA-OFT to other backbones such as $E_l$8 (Morissette et al., 15 Mar 2026), improving tactile encoders through graph-based or spatiotemporal models on dense hand arrays (Kwon et al., 28 Oct 2025), scaling simulation-driven visuo-tactile data generation and benchmarking (Chen et al., 10 Feb 2026), and integrating broader multimodal world modeling with stronger language grounding, preference learning, or multi-sensor proprioception (Ma et al., 10 Jun 2026). Taken together, these works indicate that VTAM has evolved from action-conditioned tactile-video prediction into a broader class of visuo-tactile world-action systems in which the decisive issue is how tactile evidence is made structurally available to action generation exactly when contact state becomes the dominant hidden variable.