Diffusion Models for Touch Prediction

• Diffusion models for touch prediction are probabilistic generative models that leverage iterative denoising and cross-modal conditioning to capture the inherent ambiguity and multimodality of tactile signals.
• They utilize architectures such as latent diffusion, conditional U-Nets, and transformer-based denoisers to effectively map and translate signals across diverse tactile sensor domains.
• These models enable advances in sim-to-real transfer, contact localization, and visuo-tactile synthesis, achieving superior metrics like high SSIM and low MSE in experimental benchmarks.

Diffusion models for touch prediction are a class of probabilistic generative models that have dramatically expanded the ability to synthesize, translate, and infer tactile signals across a variety of touch sensors and physical substrate representations. By iteratively denoising samples in a high-dimensional space, these models capture the intrinsic ambiguity, multimodality, and intricate correlations of tactile percepts, enabling applications ranging from cross-modal tactile translation and sim-to-real sensory adaptation to robust contact localization and visuo-tactile synthesis. The adoption of diffusion paradigms has facilitated the modeling of physically complex, sensor-specific, and inherently uncertain tactile signals with a rigor and generality unmatched by previous discriminative or GAN-based approaches.

1. Mathematical Foundations of Diffusion Models for Touch

Diffusion models for touch prediction are based on the denoising diffusion probabilistic model (DDPM) paradigm. In this framework, a forward process gradually perturbs an initial clean sample (e.g., a tactile image, a set of contact points, or an object pose) via a Markov chain, injecting Gaussian noise at each step: $$q(y_t|y_{t-1}) = \mathcal{N}(y_t;\sqrt{1-\beta_t}y_{t-1},\beta_t\,I)$$ for a noise schedule $\{\beta_t\}$. The reverse denoising process is learned by a neural network $\epsilon_\theta$, which predicts the noise at each step given the current noisy observation and one or more conditioning signals (such as a vision-based tactile image, sensor readings, or proprioceptive data). The reverse conditional typically takes the form: $$p_\theta(y_{t-1}|y_t,\,c) = \mathcal{N}(y_{t-1};\mu_\theta(y_t,t,c),\sigma_t^2 I)$$ where the mean $\mu_\theta$ is analytically derived from the predicted noise.

Training objectives are dominated by the simplified denoising score-matching loss: $$\mathcal{L}_{\mathrm{simple}} = \mathbb{E}_{y_0,\epsilon \sim \mathcal{N}(0,I), t}[ \| \epsilon - \epsilon_\theta(y_t, t, c) \|^2 ]$$ This approach unifies the probabilistic modeling of inherently ambiguous inverse problems in touch, such as inferring contact locations from distributed signals (Han et al., 10 Feb 2025, Maric et al., 16 Jun 2025), generating realistic tactile images from simulated data (Higuera et al., 2023, Lin et al., 2024), or mapping between heterogeneous sensor domains (Rodriguez et al., 2024).

2. Architectural Strategies and Conditioning Mechanisms

Contemporary diffusion models for touch prediction employ a variety of domain-specific architectures catering to input and output modality:

• Latent Diffusion (LDM): High-dimensional signals such as tactile or visual images are first encoded into a lower-dimensional latent space (e.g., via VAE or VQ-GAN) where denoising operates more efficiently (Yang et al., 2023, Rodriguez et al., 2024).
• Conditional U-Nets: Standard choices for image or grid-based signal generation, often integrating FiLM, AdaIN, or cross-attention at every resolution to inject conditioning signals (e.g., force vectors, images, proprioceptive features) (Higuera et al., 2023, Lin et al., 2024).
• MLP/Transformer-Based Denoisers: For structured predictions such as contact points in 3D or sequential trajectory features, feedforward MLPs or MADT (Motion-Aware Denoising Transformer) layers are used, with specialized mechanisms for temporal or egomotion-aware conditioning (Ma et al., 2024, Han et al., 10 Feb 2025).
• Cross-Modal Conditioning: Conditioning paths encode source signals (images, force-torque arrays, etc.) via pretrained ResNets or MLPs, then inject features into the diffusion denoiser either by channel-wise concatenation (when spatial correspondence exists) or cross-attention (when abstract semantic alignment is required) (Rodriguez et al., 2024, Lin et al., 2024, Yang et al., 2023).

The architectural modularity is essential for mapping between disparate sensor types, e.g., from force-torque measurements to 3D contact locations (Han et al., 10 Feb 2025), tactile-to-image generation for sim-to-real transfer (Higuera et al., 2023), or cross-modal translation between high-resolution vision-based tactile sensors (Rodriguez et al., 2024).

3. Cross-Modal Touch Prediction and Sensor Translation

Diffusion models have shown unique capability in translating touch signals across heterogeneous sensors—a task critical for general-purpose robotic manipulation and perception.

"Touch2Touch" (Rodriguez et al., 2024) exemplifies this: a latent diffusion model is trained to map between the GelSlim (RGB) and Soft Bubble (depth) tactile sensor domains, using paired data collected via a KUKA arm and toolset. The GelSlim signal is encoded via a ResNet-50, and the Soft Bubble depth is compressed by a shallow VAE. The network is conditioned by concatenating upsampled GelSlim features at every diffusion step in the latent U-Net. The resulting framework enables:

• Visual reconstruction metrics: PSNR ≈ 26.1 dB, SSIM ≈ 0.62, FID ≈ 61.6 on held-out contacts.
• Downstream functional transfer: in-hand pose estimation and manipulation can be performed as if a different sensor modality had been used, with accuracy nearly matching ground-truth Soft Bubble estimation.
• Ablation analysis demonstrates that both cross-modal alignment (via conditioning) and output renormalization are critical for functional fidelity.

A notable implication is the facilitation of sensor-agnostic robot manipulation policies via generative touch translation.

4. Tactile Image Generation and Sim-to-Real Bridging

Diffusion models have been exceptionally effective at sim-to-real transfer for vision-based tactile sensors, where physical data acquisition is prohibitive.

In "Learning to Read Braille" (Higuera et al., 2023), a conditional DDPM translates simulated gel deformation depth maps (from TACTO) to realistic RGB tactile images mimicking the DIGIT sensor's illumination profile. The conditional U-Net receives both the noisy sample and the simulated depth map as input. Key findings:

• The model reaches SSIM = 0.908 and MSE = 36.0 on held-out Braille frames, outperforming cGAN baselines.
• Zero-shot transfer is enabled: a classifier trained only on diffusion-generated synthetic data, fine-tuned with 20% real samples, achieves 75.7% accuracy—superior to simulation or GAN-based translation with the same supervision.
• The model captures subtle color gradients and small raised features that GANs typically miss.

Similarly, "Vision-based Tactile Image Generation via Contact Condition-guided Diffusion Model" (Lin et al., 2024) conditions the diffusion denoiser on a combined real-object RGB image and six-axis force vector, yielding:

• 60.6% reduction in MSE, 38.1% reduction in marker displacement error over prior simulators.
• Transferability across different illumination regimes and sensor architectures.
• Faithful recovery of sub-millimeter textures and load-dependent deformation patterns.

These results confirm that diffusion models provide the necessary diversity, fidelity, and invariance for bridging the sim-to-real gap across tactile tasks.

5. Touch-Informed Inference and Multimodal Sampling

Diffusion models are naturally suited for touch-informed inference tasks characterized by multimodality and geometric constraint.

The "Contact Diffusion Model" (CDM) (Han et al., 10 Feb 2025) solves multi-contact point localization for torque/force-sensor-equipped robots. The DDPM learns to sample plausible point clouds of contact locations conditioned on noisy proprioceptive signatures and historical diffusion outputs, incorporating signed distance fields (SDF) to enforce surface constraints. Salient results:

• Single contact: M-RMSE = 0.44 cm over 100 real trials.
• Dual contact: M-RMSE = 1.24 cm.
• Ablation shows SDF channeling is essential for on-surface predictions (0.29 cm surface error with SDF vs. 0.89 cm without).
• Inference is performed in 15.97 ms using a 10-step DDIM sampler.

For inverse touch inference, "Diffusion-based Inverse Observation Model for Artificial Skin" (Maric et al., 16 Jun 2025) trains a diffusion network to sample from the posterior over detected object pose $q^o$ conditioned on CySkin tactile activations $z^{tax}$. Compared to SDF-manifold projection, diffusion models halve pose estimation error and require only one-third the number of physical contacts for particle filter convergence in simulated manipulation.

A plausible implication is that diffusion models enable robust, diverse hypothesis generation for ambiguously observed contact states, a substantial advance over traditional single-mode or deterministic inference mechanisms.

6. Visuo-Tactile and Egocentric Sequence Synthesis

The cross-modal expressive power of diffusion models extends into visuo-tactile synthesis and temporal prediction. "Generating Visual Scenes from Touch" (Yang et al., 2023) leverages a latent diffusion model to generate scene images from GelSight tactile signals (and vice versa), as well as stylized images and shading reconstructions. Unique features include:

• Cross-attentional conditioning between contrastively-aligned touch and vision embeddings.
• Tunable denoising for control of content vs. style preservation.
• Superior FID (48.7), SSIM (0.76), and material classification agreement versus Pix2Pix and cycleGAN baselines.

In sequential interaction prediction, "Diff-IP2D" (Ma et al., 2024) introduces a diffusion-based paradigm for 2D hand-object interaction forecasting from egocentric video, mitigating compounding autoregressive error by reconstructing all future slots jointly in latent space. This model:

• Anchors past features and denoises future blocks in parallel, enforcing temporal and contextual consistency.
• Integrates egomotion features via multi-head cross-attention to accommodate first-person dynamics.
• Outperforms autoregressive baselines on joint hand trajectory/future affordance prediction tasks.

This evidences the capacity of diffusion to capture both fine local tactile structure and global temporal or multimodal context.

7. Limitations, Practicalities, and Future Directions

While diffusion models for touch prediction have surpassed previous approaches in fidelity, generality, and sample diversity, several practical constraints persist:

• Inference speed can be a bottleneck (e.g., ~16 ms for CDM (Han et al., 10 Feb 2025), or 23 s/batch for full-resolution image generation (Higuera et al., 2023)), though advances in latent-space diffusion and DDIM-based schemes are mitigating factors.
• Most frameworks require substantial, well-registered paired datasets (real and simulated or cross-sensor), highlighting a data collection cost for new sensor types or environments.
• Force magnitude prediction and higher degree-of-freedom contacts are relatively unexplored; current models focus on localization given sensed geometry or appearance, not full contact wrench inference (Han et al., 10 Feb 2025).
• Domain adaptation and transfer to novel objects or non-uniform backgrounds remain as active research challenges.
• Adopting unified architectures for simultaneous translation between multiple touch sensor types, or for seamless sim-to-real-to-sim transfer, stands as a promising direction (Rodriguez et al., 2024).

A salient outcome is that continuous progress in architectural innovations (e.g., hybrid DDPM–GANs, transformer-based denoisers), better conditioning strategies (e.g., multi-sensor, force, and vision fusion), and principled dataset generation will likely lead to even broader applicability and efficiency of diffusion models for tactile prediction in robotics and material perception.