---
title: 'ViTaS: Visuo-Tactile Fusion for Robotic Manipulation'
url: https://www.emergentmind.com/topics/vitas
type: topic
---

# ViTaS: Visuo-Tactile Fusion for Robotic Manipulation

ViTaS, short for **Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning**, is a visuo-tactile representation-learning framework for robotic manipulation that incorporates both visual and tactile information to guide the behavior of an agent. It is designed for settings in which prior visuomotor methods either rely mainly on vision and struggle in **contact-rich**, **occluded**, or **self-occluded** scenarios, or fuse visual and tactile inputs by **direct concatenation**, which fails to fully exploit the **alignment** and **complementarity** between the two modalities. ViTaS learns fused representations for both **reinforcement learning (RL)** and **imitation learning (IL)** by combining **Soft Fusion Contrastive Learning** with a **CVAE** module and training the resulting representation jointly with a downstream policy loss [2602.11643].

## 1. Problem formulation and conceptual basis

ViTaS is motivated by two observations. First, tactile information plays a crucial role in human manipulation tasks and is especially relevant for robotic manipulation under partial observability. Second, many existing approaches focus primarily on aligning visual and tactile features, while the integration mechanism tends to be direct concatenation. The framework therefore treats visual and tactile inputs as both **aligned** and **complementary**: aligned when they describe the same underlying scene or contact state, and complementary when one modality is degraded, ambiguous, or occluded [2602.11643].

The method’s central claim is that direct concatenation is too weak a fusion strategy for multi-modal manipulation. In the reported formulation, concatenation simply stacks features, does not ensure that the visual and tactile embeddings are aligned, does not exploit the fact that similar visual states may correspond to similar tactile states even if they are not from the exact same timestep, and cannot infer missing information in self-occluded situations. ViTaS addresses these deficiencies by separating representation learning into an alignment component and a complementarity component, then coupling both to policy learning [2602.11643].

Operationally, the pipeline uses separate **CNN encoders** for visual and tactile observations, applies soft fusion contrastive learning to align the modalities, uses a **CVAE** to reconstruct the current visual observation from fused visuo-tactile features, and feeds the learned representation to a policy network. This architecture reflects a deliberate rejection of single-stage fusion. A plausible implication is that ViTaS treats perception and control as a joint representation-learning problem rather than as a late-fusion policy design problem.

## 2. Soft Fusion Contrastive Learning

The contrastive component is defined over a trajectory
$$
\Gamma = \{o_i, t_i\}_{i=1}^{N},
$$
where $o_i$ is the visual observation and $t_i$ is the tactile observation at timestep $i$. Visual and tactile observations are encoded by $f_o(\cdot)$ and $f_t(\cdot)$, respectively. Unlike conventional cross-modal contrastive learning, which uses only the exact paired sample at the same timestep as the positive, ViTaS uses **softly selected positives** derived from a local semantic neighborhood [2602.11643].

In the first stage, for a visual observation $o_i$, the method retrieves its **top-$K$ most similar visual samples** and uses the corresponding tactile embeddings as positive samples for $t_i$, while all other tactile samples are treated as negatives. In this stage, **only the tactile encoder is updated** and the visual encoder is frozen. In the second stage, the roles are swapped: the visual encoder is updated using tactile-neighbor positives derived from the tactile modality. This alternating schedule is intended to avoid modality imbalance and prevent one encoder from dominating the learning dynamics [2602.11643].

Two departures from standard instance discrimination are central. The first is **soft positive mining**: ViTaS does not restrict positives to the exact pair $(o_i, t_i)$, but instead identifies the **top-$K$ semantically similar samples** in the anchor modality and uses their dual counterparts as positives. The second is **alternating modality updating**, which periodically changes which encoder is optimized. The reported ablation states that **$K=1$ corresponds to conventional contrastive learning**, that **ViTaS with $K=10$ performs best**, and that too large a value of $K$ degrades performance. This supports the stated view that a small soft neighborhood is beneficial, whereas overly loose neighborhoods weaken the contrastive signal [2602.11643].

The method is also positioned against **time contrastive** learning. The reported argument is that temporally adjacent frames are not always the best semantic positives. This suggests that ViTaS is explicitly designed to preserve cross-modal semantic similarity even when similar manipulation states recur at different times in a trajectory.

## 3. CVAE-based feature integration and modality complementarity

Contrastive alignment alone is not the full mechanism. ViTaS adds a **Conditional Variational Autoencoder (CVAE)** to leverage the complementary nature of vision and touch, especially in occluded or partially observable manipulation. The CVAE reconstructs the current visual frame from a fused visuo-tactile condition constructed from the concatenated visuo-tactile feature and passed through a projector $f_\phi(\cdot)$ [2602.11643].

The stated purpose of the CVAE is threefold: it enforces **cross-modal consistency**, allows tactile information to compensate for missing or corrupted visual information, and is especially helpful in **self-occluded** scenes, such as cases in which the robot hand blocks the camera view. During training, the CVAE encoder, decoder, projector, and the modality encoders are jointly optimized. At inference time, **only the image and tactile encoders are retained** [2602.11643].

This component is explicitly about **complementarity**, not merely reconstruction fidelity. The reported design forces the fused latent representation to remain useful even when one modality is degraded. In the paper’s framing, the CVAE is what moves ViTaS beyond pure cross-modal alignment: contrastive learning captures correspondence, whereas the CVAE encourages the latent state to encode information that can fill in missing visual detail from tactile cues.

A recurrent misconception in visuo-tactile learning is that alignment alone is sufficient. ViTaS is constructed against that view. The reported ablations state that removing either the soft fusion contrastive module or the CVAE hurts performance substantially, indicating that alignment and complementarity are treated as distinct requirements rather than interchangeable design choices [2602.11643].

## 4. Joint objective, optimization, and implementation details

The full training objective is
$$
\mathcal{L} = \lambda \mathcal{L}_{\tt{CON}} + \mu \mathcal{L}_{\tt{VAE}} + \mathcal{L}_{\tt{policy}},
$$
where $\lambda$ balances the soft fusion contrastive loss, $\mu$ balances the CVAE loss, and $\mathcal{L}_{\tt{policy}}$ is the task policy loss. In **RL**, the policy term is $\mathcal{L}_{\tt{PPO}}$; in **IL**, it is the **Diffusion Policy** loss. The reported default coefficients are **$\lambda = 1$** and **$\mu = 0.1$** [2602.11643].

Several implementation details clarify how the framework is intended to operate in practice. ViTaS uses **separate CNN encoders** for visual and tactile observations rather than a unified encoder, because the ablation indicates that a unified encoder performs poorly and that vision and touch have distinct statistical structures. The method alternates contrastive learning across modalities to avoid imbalance, and the **top-$K$ positive mining** is described as central to the design. In simulation, some tasks use built-in tactile modules, while others use gripper-mounted tactile maps of size **$32\times 32\times 3$**. In the real-world experiments, the tactile encoder is adapted to the **$16\times 16\times 1$** tactile format [2602.11643].

The framework is trained jointly with downstream policy learning and is presented as a representation-learning plug-in for both RL and IL. This suggests a modular usage pattern: the visuo-tactile representation is not an auxiliary pretraining artifact, but an online component of the manipulation learner.

## 5. Experimental evaluation and reported performance

ViTaS is evaluated in both simulation and real-world manipulation. In simulation, the reported setup covers **12 simulated tasks in 5 environments**, including **Gymnasium** dexterous hand rotation, **Robosuite** contact-rich manipulation, **Insertion**, **Mobile Catching**, and **Block Spinning**. The listed tasks include **Insertion**, **Door opening**, **Lift**, **Pen rotate**, **Dual arm lift**, **Mobile catch**, **Egg rotate**, **Block rotate**, **Block spin**, **Insertion with noise**, **Lift with cap**, and **Lift with can**. Most simulation tasks are trained with **PPO**, while harder tasks such as **Box Stack**, **Wiping**, and **Assembly** are also evaluated with **Diffusion Policy** in imitation learning [2602.11643].

In the real world, ViTaS is tested on **three robotic manipulation tasks** using a **Galaxea-R1 humanoid robot**: **Dual Arm Clean (DAC)**, **Table Pick Place (TPP)** with **TPP-1** and **TPP-3**, and **Fridge Pick Place (FPP)**. The method uses only the **head camera** for vision and a **3D-ViTac** tactile sensor on the gripper, with **100 expert trajectories per task** collected via teleoperation [2602.11643].

The principal baselines include **M3L**, **VTT**, **PoE**, **Concat**, **MViTac**, and **ConViTac**; for imitation-learning comparisons, the method is also compared against **CNN-based Diffusion Policy** and **Transformer-based Diffusion Policy**. The reported aggregate results are summarized below.

| Setting | ViTaS result | Reported comparators |
|---|---:|---|
| Simulation average success | **91.4** | MVT 70.3, CVT 71.5, M3L 46.6, PoE 35.7, VTT 47.5, Concat 40.6 |
| IL average | **60.4** | DP w/ CNN 38.2, DP w/ Transformer 33.4 |
| Real-world average success | **46.0** | DP 30.0 |

The reported findings emphasize that ViTaS is especially strong on **Egg Rotate**, **Block Rotate**, **Block Spin**, and **Mobile Catch**, which are identified as tasks where tactile and visual cues must be fused effectively. The method is also reported to remain strong when **object shape changes**, **targets are randomized**, and **Gaussian noise is added to observations**, with much smaller performance degradation than the baselines. In the real world, the result is described as notable because ViTaS uses **fewer cameras** than Diffusion Policy yet still does better [2602.11643].

## 6. Limitations, interpretation, and name ambiguity

The reported limitations are specific. First, **high-dynamic precise manipulation** in the real world remains challenging due to robot physical capability and RL bottlenecks, with **pen spinning** given as an example. Second, **deformable object manipulation** is described as underexplored and requiring further study. These caveats are important because they qualify the empirical claims: ViTaS is presented as effective for a broad range of contact-rich manipulation settings, but not as a complete solution to all visuomotor regimes [2602.11643].

The paper’s own ablations attribute the method’s performance to three factors: tactile information is crucial; separate encoders matter; and both **soft fusion** and the **CVAE** are necessary. This suggests that ViTaS should not be reduced to a generic multimodal policy with an added tactile stream. Its distinctiveness lies in the combination of soft positive mining, alternating cross-modal optimization, and reconstruction-driven exploitation of complementarity.

The name **ViTaS** also requires disambiguation across research domains. In **voice-user-interface testing**, **Vitas** is the **state-of-the-art model-based testing framework for VPA apps** and serves as the main prior-art baseline for **Elevate**, which improves over it by using an LLM to recover semantic information that Vitas misses [2407.02791]. In **vision transformer architecture search**, **ViTAS** refers to **Vision Transformer Architecture Search**, a NAS framework built around cyclic weight sharing and identity shifting [2106.13700]. In **multimodal radiology summarization**, **ViTAS** stands for **Visual-Text Attention Summarizer**, a pipeline for the **FINDINGS → IMPRESSION** task on **MIMIC-CXR** [2603.29901]. A plausible implication is that citations using the string “ViTaS” or “ViTAS” should be interpreted by expansion and domain, not by acronym alone.

Source: https://www.emergentmind.com/topics/vitas