EgoPressDiff: Multimodal Video Diffusion for Egocentric UV-Domain Hand-Pressure Estimation

Abstract: Estimating hand-surface contact pressure from an egocentric view is crucial for AR/VR devices, robotic imitation, and ergonomic analysis. Existing methods often discretize pressure signal and process frames independently, leading to quantization errors and temporal inconsistencies. We present \emph{EgoPressDiff}, a conditional video diffusion framework that generates UV-pressure maps from visual input. The core of our approach is a multi-modal conditioning strategy, introducing a PoseNet and a Vertex Encoder to efficiently extract features from hand pose and 3D mesh vertices. These signals, along with depth information, guide the generative process to ensure the pressure fields are physically grounded. To effectively fuse these heterogeneous features, we further propose a Distribution-Calibrated Spatial Layer, which aligns their statistical properties before combination. Evaluated on the EgoPressure ego-view setting, EgoPressDiff achieves state-of-the-art results, improving Volumetric IoU by over 34\% relative to prior baseline, while reducing MAE and maintaining high temporal accuracy. Our project page is at https://egopressdiff.github.io/.

• The paper introduces a continuous, diffusion-based approach for estimating hand pressure from egocentric video input.
• It leverages a novel DC Spatial Layer to fuse pose, depth, RGB, and geometric features for temporally coherent UV-pressure predictions.
• Quantitative results on the EgoPressure dataset show a 34% improvement in Volumetric IoU and enhanced temporal accuracy over previous methods.

EgoPressDiff: Multimodal Video Diffusion for Egocentric UV-Domain Hand-Pressure Estimation

Problem Formulation and Motivation

Estimating hand-surface contact pressure from egocentric visual input is an open problem with direct implications in AR/VR interaction, ergonomic assessment, and robotic imitation. Traditional approaches either rely on tactile sensors, which are intrusive and cumbersome, or vision-based pipelines that quantize pressures and treat each frame independently, resulting in incomplete representations, quantization error, and poor temporal consistency. Key prior works such as PressureVision and PressureFormer have demonstrated the utility of vision-only approaches but are fundamentally limited by discrete classification schemas and frame-level modeling.

EgoPressDiff Model Overview

EgoPressDiff formulates hand pressure estimation as a conditional spatiotemporal video generation problem, replacing discrete frame-wise estimation with continuous, physically-grounded UV-pressure prediction. The model generates temporally coherent pressure maps in the UV domain, which correspond directly to hand surfaces modeled by MANO. By leveraging a diffusion-based architecture, EgoPressDiff produces high-fidelity, temporally stable, and physically plausible pressure sequences from single egocentric RGB video streams.

Figure 1: EgoPressDiff generates dynamic hand pressure maps from egocentric RGB input, producing temporally coherent UV-pressure fields for 3D contact visualization on the MANO mesh.

The model’s generative process is conditioned on multiple complementary control signals: hand pose, 3D vertices, depth, and RGB. Each input stream is encoded via modality-dedicated subnetworks—PoseNet for hand skeleton, VAE for depth/UV-pressure, a CLIP-based encoder for image streams, and a custom Vertex Encoder for hand geometry.

Figure 2: The EgoPressDiff pipeline integrates pose, depth, RGB, and geometric features using modality-specific encoders and a DC Spatial Layer to generate temporally-consistent UV-pressure predictions.

Multimodal Fusion: Distribution-Calibrated Spatial Layer

A core innovation is the DC Spatial Layer, which solves the challenging problem of fusing heterogeneous features from fundamentally different imaging and geometric domains. The DC Spatial Layer replaces the standard cross-attention conditioning in diffusion U-Nets, leveraging parallel cross-attention branches for image and vertex embeddings. These outputs are aligned at the distributional level (channel-wise mean and variance) before fusion, ensuring statistical compatibility. This approach outperforms naïve concatenation, as validated by ablation.

Figure 3: The DC Spatial Layer calibrates latent distribution statistics prior to fusing image and vertex embeddings, ensuring stable cross-modal feature integration.

Training Objective and Implementation

EgoPressDiff is trained with a weighted reconstruction loss that focuses optimization on UV regions overlapping the hand, leveraging a binary UV mask that up-weights reconstruction error in physically meaningful regions. Supervision targets are denoised UV-pressure maps obtained from the EgoPressure dataset, which provides dense ground-truth measurements mapped to MANO mesh geometry.

Control signals (depth, pose, geometry) are generated with state-of-the-art methods (e.g., Video Depth Anything, EgoPressure annotation pipeline). The training employs end-to-end learning from randomly initialized PoseNet and Vertex Encoder, and SVD-pretrained U-Net initialization.

Quantitative and Qualitative Evaluation

On the EgoPressure dataset, EgoPressDiff achieves state-of-the-art results, with a Volumetric IoU improvement of over 34% relative to prior baselines in the egocentric setting and leading results in MAE and Temporal Accuracy as well. The method demonstrates robust performance across all metrics by fully leveraging temporal context, geometric priors, and multimodal fusion.

Qualitative assessments highlight that, unlike previous per-frame classification models, EgoPressDiff accurately distinguishes fine-grained contact events (e.g., middle finger contact in isolation) and produces smoother pressure gradients, closely matching ground truth.

Figure 4: EgoPressDiff produces accurate and smooth UV-pressure maps, outperforming baselines in correctly localizing contact points and representing pressure distributions.

Ablation and Architectural Analysis

A comprehensive ablation analysis underscores the criticality of each control signal and the DC Spatial Layer. Removing depth cues leads to a drastic drop in Temporal Accuracy (from 94% to 82%) and other key metrics, demonstrating its necessity for physical plausibility. Omitting the DC Spatial Layer is shown to be more detrimental than removing vertex features entirely, validating the model’s feature calibration approach. Loss-based hand-region weighting (UV mask) is also essential for maximizing spatial accuracy.

Implications and Future Directions

EgoPressDiff demonstrates the efficacy of continuous, temporally-aware conditional video diffusion models for dense hand pressure estimation, setting a new performance frontier in this domain. Its architectural choices emphasize the importance of cross-modal feature fusion and distributional calibration for high-fidelity generative tasks. Practical implications include scalable contact/pressure acquisition from visual streams for AR/VR systems and data-efficient simulation for imitation learning and robotics.

Current limitations are rooted in the dataset’s bias toward simple contact patterns, restricting generalization to complex real-world manipulations. Immediate research directions include extending EgoPressDiff to more diverse grasp scenarios and activities, scaling models with stronger priors (e.g., language or task-level context), and developing robust adaptation mechanisms for multi-contact and environmental variability.

Conclusion

EgoPressDiff marks a significant step in vision-based hand pressure estimation, reframing the problem by exploiting video diffusion methods and calibrated multimodal fusion. The framework yields state-of-the-art egocentric pressure estimation by combining physically-informed priors with temporally-consistent generation. The methods and findings constitute a strong foundation for future advances in human-centric sensing and physically-aware AR/VR systems.

(2606.06872)