Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

Abstract: Visuo-Tactile policies leveraging optical tactile sensors have shown great promise in contact-rich manipulation. These sensors achieve high spatial resolution and multi-dimensional force sensing by utilizing an internal camera to monitor the deformation of their elastic gel surface, thereby indirectly inferring tactile cues. Despite their advantages, extracting fine-grained contact states necessary for contact-rich manipulation remains an open challenge. Existing methods typically use either raw images or cumulative motion fields to represent tactile cues. However, both are prone to perception ambiguity. Raw tactile images mainly capture appearance changes, while cumulative motion fields only reflect the aggregate gel deformation. Consequently, distinct fine-grained contact states can exhibit highly similar patterns, making it difficult to explicitly distinguish subtle contact variations. To address this issue, we explore the dynamic priors of tactile motion and discover that the correlation between transient and cumulative motion can explicitly distinguish fine-grained contact states. Based on this insight, we propose a motion-aware tactile representation to facilitate contact-rich manipulation. Beyond tactile representation, effective fusion of tactile and visual modalities is also critical. Most existing fusion methods either directly concatenate features from each modality or train modality-specific networks separately and fuse their outputs. However, these strategies struggle to simultaneously model cross-modal interactions and preserve modality-specific characteristics. In this work, we take advantage of the Mixture-of-Transformers architecture and propose a unified modality-aware visuo-tactile policy that captures cross-modal complementarity while maintaining modality-specific properties.

• The paper introduces Tactile Motion Correlation, a dot product of transient and cumulative tactile motion that distinguishes contact, release, stability, sliding, and force changes across three optical sensors.
• The paper combines TMC with a Mixture-of-Transformers policy that preserves modality-specific processing while enabling shared attention across vision, tactile appearance, motion, and proprioception.
• The paper’s ViTacMotor system achieves the highest success rates on four bimanual manipulation tasks, reaching 86.7% for whiteboard erasing and 60.0% for pencil sharpening, while ablations confirm benefits from both TMC and modality-aware fusion.

Motivation and problem statement

Optical tactile sensors infer contact information by imaging the deformation of a markered elastic gel surface. Although these sensors provide high spatial resolution and multi-dimensional force sensing, the paper identifies a persistent representational gap: existing visuo-tactile policies encode tactile input either as raw gel images or as cumulative motion fields (optical flow from the initial non-contact frame to the current frame), and both encodings suffer from perception ambiguity. In particular, fine-grained contact states that matter for manipulation—making contact versus releasing contact—can produce nearly identical raw appearances and nearly identical cumulative deformation patterns, so a policy cannot reliably distinguish them from either representation alone.

The paper's central observation is that the relationship between two motion fields resolves this ambiguity. Transient motion, the optical flow between consecutive tactile frames, captures instantaneous deformation; cumulative motion captures aggregate deformation relative to the no-contact state. Their correlation is state-dependent: during making contact the two fields are aligned; during stable contact transient motion nearly vanishes while cumulative motion persists; during release the fields point in opposite directions; during sliding, the pixel-wise dot product shows a characteristic positive–negative separation that encodes the sliding direction. This yields a discriminative, physically interpretable signal that neither field provides in isolation.

The Tactile Motion Correlation representation

The proposed Tactile Motion Correlation (TMC) representation is computed with an efficient optical flow estimator (DIS, running at 600 Hz on a single CPU core at 1024×436 resolution). Given the initial frame $T_0$, previous frame $T_{t-1}$, and current frame $T_t$, transient and cumulative motion fields are estimated as $\mathrm{Flow}(T_{t-1}, T_t)$ and $\mathrm{Flow}(T_0, T_t)$, and the representation is their pixel-wise dot product:

$$Corr_t(x) = \mathbf{M}^{\mathrm{tran}}_t(x) \cdot \mathbf{M}^{\mathrm{cumu}}_t(x)$$

The paper claims, and supports empirically, three properties of this representation. First, its sign and magnitude separate fine-grained contact states: positive during making contact, near zero during no contact or stable contact, negative during release, and spatially split during sliding. Second, its magnitude during making contact is positively correlated with the applied contact force, so the representation implicitly carries force information without a force sensor. Third, it is sensor-agnostic: the correlation analysis is validated on three optical sensors with different marker layouts (random dense markers from InTac S1 and DM-Tac W2, and regularly arranged sparse markers from Xense), and the discriminative patterns hold across all of them. This contrasts with learning-based tactile representations such as AnyTouch and Sparsh, which encode transient and cumulative changes implicitly in a latent space; TMC instead computes an explicit, physically interpretable quantity.

A limitation of the analysis is that the force-correlation claim is established only for the making-contact regime using the sensor SDK's force readings; the relationship under other contact regimes is not quantified.

Unified yet modality-aware fusion with Mixture-of-Transformers

The second contribution concerns how tactile and visual information are combined. The paper characterizes existing strategies as either feature-level (concatenation or simple attention), which risks one modality dominating and ignores modality-specific structure, or decision-level (independent policies per modality with output aggregation), which forfeits cross-modal interaction. ViTacMotor instead adopts the Mixture-of-Transformers (MoT) architecture: query, key, value, output projections, feed-forward networks, and layer normalizations are all modality-specific, while self-attention is computed jointly over all modalities. This preserves per-modality inductive structure while allowing cross-modal attention.

The observation space comprises wrist and third-view RGB images, three tactile frames ($T_0$, $T_{t-1}$, $T_t$), and proprioception. Notably, the motion correlation map $Corr_t$ is treated as a distinct modality from tactile appearance, reflecting their heterogeneous physical characteristics, and is encoded by a separate ResNet. The fused embeddings feed a transformer decoder that autoregressively predicts action chunks, trained with a $T_{t-1}$0-VAE objective (L1 action prediction loss plus KL regularization, $T_{t-1}$1) in the style of ACT. Training takes about 5 hours per task on a single RTX 3090 Ti, indicating modest compute requirements.

Experimental results

The method is evaluated on a bimanual Agilex platform across four real-world contact-rich tasks: tube collection and whiteboard erasing (dense-marker sensors), and lightbulb insertion and pencil sharpening (sparse-marker sensor), with 15 trials per method and 35–60 demonstrations per task. Baselines include ACT, Diffusion Policy, ACT with tactile images (feature-level concatenation), Policy Consensus (decision-level fusion), and TactileACT.

Method	Tube Collection	Lightbulb Insertion	Whiteboard Erasing	Pencil Sharpening
ACT	53.3	13.3	46.7	33.3
DP	40.0	6.7	40.0	33.3
ACT + T	60.0	26.7	60.0	40.0
Policy Consensus	53.3	26.7	66.7	33.3
TactileACT	60.0	40.0	73.3	46.7
ViTacMotor	73.3	40.0	86.7	60.0

All values are success rates in percent. The largest margins appear on whiteboard erasing (+13.3 over the strongest baseline) and pencil sharpening (+13.3), where the force-correlation property of TMC is argued to enable stable pressure modulation and shear-force-aware correction. On lightbulb insertion, ViTacMotor ties TactileACT at 40.0, so the advantage is not uniform across tasks. The authors attribute visual-only failures to occlusion during in-hand phases and to the absence of force feedback, which is consistent with the staged success-rate breakdown showing degradation at insertion and erasing sub-stages.

Ablations isolate both components. Removing TMC drops success from 86.7 to 73.3 (whiteboard erasing) and from 73.3 to 66.7 (tube collection); replacing the MoT fusion with concatenation drops performance to 66.7 and 60.0, respectively, and cross-attention fusion performs worst on tube collection (60.0). TMC is also shown to be a drop-in improvement: substituting it for raw images or cumulative motion as the tactile input to ACT and Diffusion Policy improves success on whiteboard erasing from 60.0 to 73.3 and from 53.3 to 66.7, respectively. Notably, cumulative motion alone degrades ACT relative to raw images (53.3 vs. 60.0), reinforcing the paper's claim that cumulative fields are insufficient. Robustness experiments with temporally varying uniform and non-uniform lighting and object substitutions (different eraser/ink, tube replaced by a cylinder) show the policy completing tasks under these perturbations, and the results replicate on a second dense-marker sensor.

Limitations and open questions

The paper concedes that generalization remains limited under extreme visual disturbances or significant object pose variations: in the tube collection task, the policy grasps the tube under position changes and severe visual perturbation but fails during insertion. The force-correlation property of the dot product is demonstrated only for the making-contact case. Evaluation is confined to four tasks with a fixed bimanual platform, 15 trials per condition, and no statistical significance testing, so the margins—particularly the tie on lightbulb insertion—should be interpreted with that sample size in mind. The claim that TMC could serve as a general, sensor-bridging tactile representation is supported only across three optical sensors with marker-based gels; extension to marker-free or non-optical sensors is left open. Whether the explicit dot-product representation outperforms learned latent representations (e.g., AnyTouch-style encoders) on the same tasks is not directly tested.

Conclusion

The paper makes two concrete contributions: an explicit, physically interpretable tactile representation—the dot product of transient and cumulative gel motion—that disambiguates fine-grained contact states and correlates with contact force, and an MoT-based visuo-tactile policy that fuses vision, tactile appearance, motion correlation, and proprioception with modality-specific parameters under shared attention. Across four real-world contact-rich tasks the combined system outperforms feature- and decision-level fusion baselines, and TMC improves existing policies when embedded directly. The main open question the work leaves is generalization beyond its demonstrated tasks, sensors, and disturbance regimes, particularly for insertion under large object pose variation.