ViTaS: Visuo-Tactile Fusion for Robotic Manipulation

Updated 5 July 2026

ViTaS is a visuo-tactile framework that fuses visual and tactile cues using soft contrastive learning and a CVAE to align and complement sensory inputs.
It employs separate CNN encoders and an alternating modality updating strategy to overcome limitations of direct concatenation in multi-modal fusion.
ViTaS demonstrates superior performance in simulation and real-world tasks, enhancing robotic manipulation under occlusion and contact-rich conditions.

ViTaS, short for Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning, is a visuo-tactile representation-learning framework for robotic manipulation that incorporates both visual and tactile information to guide the behavior of an agent. It is designed for settings in which prior visuomotor methods either rely mainly on vision and struggle in contact-rich, occluded, or self-occluded scenarios, or fuse visual and tactile inputs by direct concatenation, which fails to fully exploit the alignment and complementarity between the two modalities. ViTaS learns fused representations for both reinforcement learning (RL) and imitation learning (IL) by combining Soft Fusion Contrastive Learning with a CVAE module and training the resulting representation jointly with a downstream policy loss (Tian et al., 12 Feb 2026).

1. Problem formulation and conceptual basis

ViTaS is motivated by two observations. First, tactile information plays a crucial role in human manipulation tasks and is especially relevant for robotic manipulation under partial observability. Second, many existing approaches focus primarily on aligning visual and tactile features, while the integration mechanism tends to be direct concatenation. The framework therefore treats visual and tactile inputs as both aligned and complementary: aligned when they describe the same underlying scene or contact state, and complementary when one modality is degraded, ambiguous, or occluded (Tian et al., 12 Feb 2026).

The method’s central claim is that direct concatenation is too weak a fusion strategy for multi-modal manipulation. In the reported formulation, concatenation simply stacks features, does not ensure that the visual and tactile embeddings are aligned, does not exploit the fact that similar visual states may correspond to similar tactile states even if they are not from the exact same timestep, and cannot infer missing information in self-occluded situations. ViTaS addresses these deficiencies by separating representation learning into an alignment component and a complementarity component, then coupling both to policy learning (Tian et al., 12 Feb 2026).

Operationally, the pipeline uses separate CNN encoders for visual and tactile observations, applies soft fusion contrastive learning to align the modalities, uses a CVAE to reconstruct the current visual observation from fused visuo-tactile features, and feeds the learned representation to a policy network. This architecture reflects a deliberate rejection of single-stage fusion. A plausible implication is that ViTaS treats perception and control as a joint representation-learning problem rather than as a late-fusion policy design problem.

2. Soft Fusion Contrastive Learning

The contrastive component is defined over a trajectory

$\Gamma = \{o_i, t_i\}_{i=1}^{N},$

where $o_i$ is the visual observation and $t_i$ is the tactile observation at timestep $i$ . Visual and tactile observations are encoded by $f_o(\cdot)$ and $f_t(\cdot)$ , respectively. Unlike conventional cross-modal contrastive learning, which uses only the exact paired sample at the same timestep as the positive, ViTaS uses softly selected positives derived from a local semantic neighborhood (Tian et al., 12 Feb 2026).

In the first stage, for a visual observation $o_i$ , the method retrieves its top- $K$ most similar visual samples and uses the corresponding tactile embeddings as positive samples for $t_i$ , while all other tactile samples are treated as negatives. In this stage, only the tactile encoder is updated and the visual encoder is frozen. In the second stage, the roles are swapped: the visual encoder is updated using tactile-neighbor positives derived from the tactile modality. This alternating schedule is intended to avoid modality imbalance and prevent one encoder from dominating the learning dynamics (Tian et al., 12 Feb 2026).

Two departures from standard instance discrimination are central. The first is soft positive mining: ViTaS does not restrict positives to the exact pair $(o_i, t_i)$ , but instead identifies the top- $o_i$ 0 semantically similar samples in the anchor modality and uses their dual counterparts as positives. The second is alternating modality updating, which periodically changes which encoder is optimized. The reported ablation states that $o_i$ 1 corresponds to conventional contrastive learning, that ViTaS with $o_i$ 2 performs best, and that too large a value of $o_i$ 3 degrades performance. This supports the stated view that a small soft neighborhood is beneficial, whereas overly loose neighborhoods weaken the contrastive signal (Tian et al., 12 Feb 2026).

The method is also positioned against time contrastive learning. The reported argument is that temporally adjacent frames are not always the best semantic positives. This suggests that ViTaS is explicitly designed to preserve cross-modal semantic similarity even when similar manipulation states recur at different times in a trajectory.

3. CVAE-based feature integration and modality complementarity

Contrastive alignment alone is not the full mechanism. ViTaS adds a Conditional Variational Autoencoder (CVAE) to leverage the complementary nature of vision and touch, especially in occluded or partially observable manipulation. The CVAE reconstructs the current visual frame from a fused visuo-tactile condition constructed from the concatenated visuo-tactile feature and passed through a projector $o_i$ 4 (Tian et al., 12 Feb 2026).

The stated purpose of the CVAE is threefold: it enforces cross-modal consistency, allows tactile information to compensate for missing or corrupted visual information, and is especially helpful in self-occluded scenes, such as cases in which the robot hand blocks the camera view. During training, the CVAE encoder, decoder, projector, and the modality encoders are jointly optimized. At inference time, only the image and tactile encoders are retained (Tian et al., 12 Feb 2026).

This component is explicitly about complementarity, not merely reconstruction fidelity. The reported design forces the fused latent representation to remain useful even when one modality is degraded. In the paper’s framing, the CVAE is what moves ViTaS beyond pure cross-modal alignment: contrastive learning captures correspondence, whereas the CVAE encourages the latent state to encode information that can fill in missing visual detail from tactile cues.

A recurrent misconception in visuo-tactile learning is that alignment alone is sufficient. ViTaS is constructed against that view. The reported ablations state that removing either the soft fusion contrastive module or the CVAE hurts performance substantially, indicating that alignment and complementarity are treated as distinct requirements rather than interchangeable design choices (Tian et al., 12 Feb 2026).

4. Joint objective, optimization, and implementation details

The full training objective is

$o_i$ 5

where $o_i$ 6 balances the soft fusion contrastive loss, $o_i$ 7 balances the CVAE loss, and $o_i$ 8 is the task policy loss. In RL, the policy term is $o_i$ 9; in IL, it is the Diffusion Policy loss. The reported default coefficients are $t_i$ 0 and $t_i$ 1 (Tian et al., 12 Feb 2026).

Several implementation details clarify how the framework is intended to operate in practice. ViTaS uses separate CNN encoders for visual and tactile observations rather than a unified encoder, because the ablation indicates that a unified encoder performs poorly and that vision and touch have distinct statistical structures. The method alternates contrastive learning across modalities to avoid imbalance, and the top- $t_i$ 2 positive mining is described as central to the design. In simulation, some tasks use built-in tactile modules, while others use gripper-mounted tactile maps of size $t_i$ 3. In the real-world experiments, the tactile encoder is adapted to the $t_i$ 4 tactile format (Tian et al., 12 Feb 2026).

The framework is trained jointly with downstream policy learning and is presented as a representation-learning plug-in for both RL and IL. This suggests a modular usage pattern: the visuo-tactile representation is not an auxiliary pretraining artifact, but an online component of the manipulation learner.

5. Experimental evaluation and reported performance

ViTaS is evaluated in both simulation and real-world manipulation. In simulation, the reported setup covers 12 simulated tasks in 5 environments, including Gymnasium dexterous hand rotation, Robosuite contact-rich manipulation, Insertion, Mobile Catching, and Block Spinning. The listed tasks include Insertion, Door opening, Lift, Pen rotate, Dual arm lift, Mobile catch, Egg rotate, Block rotate, Block spin, Insertion with noise, Lift with cap, and Lift with can. Most simulation tasks are trained with PPO, while harder tasks such as Box Stack, Wiping, and Assembly are also evaluated with Diffusion Policy in imitation learning (Tian et al., 12 Feb 2026).

In the real world, ViTaS is tested on three robotic manipulation tasks using a Galaxea-R1 humanoid robot: Dual Arm Clean (DAC), Table Pick Place (TPP) with TPP-1 and TPP-3, and Fridge Pick Place (FPP). The method uses only the head camera for vision and a 3D-ViTac tactile sensor on the gripper, with 100 expert trajectories per task collected via teleoperation (Tian et al., 12 Feb 2026).

The principal baselines include M3L, VTT, PoE, Concat, MViTac, and ConViTac; for imitation-learning comparisons, the method is also compared against CNN-based Diffusion Policy and Transformer-based Diffusion Policy. The reported aggregate results are summarized below.

Setting	ViTaS result	Reported comparators
Simulation average success	91.4	MVT 70.3, CVT 71.5, M3L 46.6, PoE 35.7, VTT 47.5, Concat 40.6
IL average	60.4	DP w/ CNN 38.2, DP w/ Transformer 33.4
Real-world average success	46.0	DP 30.0

The reported findings emphasize that ViTaS is especially strong on Egg Rotate, Block Rotate, Block Spin, and Mobile Catch, which are identified as tasks where tactile and visual cues must be fused effectively. The method is also reported to remain strong when object shape changes, targets are randomized, and Gaussian noise is added to observations, with much smaller performance degradation than the baselines. In the real world, the result is described as notable because ViTaS uses fewer cameras than Diffusion Policy yet still does better (Tian et al., 12 Feb 2026).

6. Limitations, interpretation, and name ambiguity

The reported limitations are specific. First, high-dynamic precise manipulation in the real world remains challenging due to robot physical capability and RL bottlenecks, with pen spinning given as an example. Second, deformable object manipulation is described as underexplored and requiring further study. These caveats are important because they qualify the empirical claims: ViTaS is presented as effective for a broad range of contact-rich manipulation settings, but not as a complete solution to all visuomotor regimes (Tian et al., 12 Feb 2026).

The paper’s own ablations attribute the method’s performance to three factors: tactile information is crucial; separate encoders matter; and both soft fusion and the CVAE are necessary. This suggests that ViTaS should not be reduced to a generic multimodal policy with an added tactile stream. Its distinctiveness lies in the combination of soft positive mining, alternating cross-modal optimization, and reconstruction-driven exploitation of complementarity.

The name ViTaS also requires disambiguation across research domains. In voice-user-interface testing, Vitas is the state-of-the-art model-based testing framework for VPA apps and serves as the main prior-art baseline for Elevate, which improves over it by using an LLM to recover semantic information that Vitas misses (Li et al., 2024). In vision transformer architecture search, ViTAS refers to Vision Transformer Architecture Search, a NAS framework built around cyclic weight sharing and identity shifting (Su et al., 2021). In multimodal radiology summarization, ViTAS stands for Visual-Text Attention Summarizer, a pipeline for the FINDINGS → IMPRESSION task on MIMIC-CXR (Naznin et al., 31 Mar 2026). A plausible implication is that citations using the string “ViTaS” or “ViTAS” should be interpreted by expansion and domain, not by acronym alone.