EgoPHI: Estimating Contact and Force from Egocentric Vision

Abstract: Understanding hand-object interaction from egocentric vision is essential for modeling how people physically engage with the surrounding world. Yet reasoning about physically grounded interaction requires estimating the forces acting on hands and objects, beyond localizing contact. We present EgoPHI, the first method that jointly estimates dense contact maps and 3D force distributions on hand and object meshes from a single monocular RGB image and object geometry. To address the lack of scalable ground-truth force annotations, we introduce a physics-based simulation pipeline that augments existing hand-object datasets with dense per-vertex force supervision. EgoPHI then learns dense 3D contact and force on interacting hand and articulated object meshes, extending vision-based force estimation beyond image-space or planar settings. Our evaluation on in-distribution and out-of-distribution benchmarks shows that EgoPHI improves force estimation over existing approaches while generalizing to unseen datasets. To evaluate sim-to-real transfer, we constructed two physical objects that capture dense object contact and force magnitude and used them to record a dataset of interactions from eight participants across diverse touch and grasp types. Our results demonstrate that EgoPHI recovers meaningful 3D contact and force distributions in simulated, out-of-distribution, and real-world settings, advancing egocentric hand-object understanding from contact localization toward physically grounded interaction reasoning.

• The paper introduces EgoPHI, a vision-only method that jointly predicts dense per-vertex contact maps and 3D normal-force distributions on hand and articulated-object meshes from a single monocular RGB image with a known object template.
• The paper combines SOFA-based physics simulation, projected ViT features, graph and cross-mesh attention, and iterative object-pose refinement, reducing hand-force MAE from 6.62 to 4.03 N on ARCTIC and improving out-of-distribution results on H2O.
• The paper validates sim-to-real transfer with instrumented objects, achieving force MAEs of 0.48 N on a cube and 1.35 N on a cylinder, while identifying limitations in temporal modeling, curved geometry, material variation, and tangential-force estimation.

EgoPHI addresses a gap in egocentric hand–object understanding: while contact localization from vision is well studied, the physical forces underlying interaction have only been estimated in image space or on planar surfaces. The paper introduces the first method that jointly predicts dense per-vertex contact maps and 3D force distributions over both hand meshes and an articulated object mesh, given a single monocular RGB image and the object's template geometry (2608.13014).

Problem setting and motivation

The authors argue that contact alone is insufficient for physically grounded reasoning about manipulation: stability, control, and intent are governed by forces. Inferring forces from egocentric RGB is difficult because of occlusion, weak visual intensity cues (e.g., skin blanching), and ambiguous depth in single views. Prior force-estimation work is restricted to 2D image-space pressure prediction or flat surfaces; no existing model estimates contact or force on the manipulated object itself. EgoPHI therefore defines a new problem setting—3D force estimation on non-planar rigid and articulated objects from vision alone, without tactile sensing at test time.

Physics-based supervision

Because dense ground-truth force annotations are unavailable at scale, the paper's central enabler is a simulation pipeline built on SOFA. Recorded hand and object meshes are replayed as quasi-rigid bodies with penalty-based soft springs that activate within a proximity zone, producing per-vertex restoring forces $\mathbf{f}_i = -k\, d_i\, \mathbf{n}_i$ with a fixed stiffness $k=10$. A binary contact mask filters spurious non-contact forces before supervision. Notably, the authors validate this simulated supervision against real measurements from their instrumented objects: mean simulated forces closely match FTIR-based readings (e.g., cube: 1.61 N vs. 1.67 N), with per-frame MAE of 0.25–0.36 N and Spearman correlations of .604–.760. This validation is important because it substantiates the claim that simulation can substitute for scarce real force labels.

Architecture

EgoPHI uses a ViT-B/16 backbone whose patch features are sampled at mesh vertices projected into the image, fused with 3D coordinates via MLPs. Reasoning proceeds through stacked Graph-Based Interaction Blocks combining intra-mesh graph attention, inter-mesh cross-attention among left hand, right hand, and object, and feed-forward refinement. An Iterative Refinement Module (IRM) estimates the object pose in camera coordinates through three incremental updates (6D rotation plus translation), conditioned on hand centroids and RoI-aligned features. A second Interaction Block stack then predicts per-vertex contact probability, normalized force magnitude, and unit direction; predicted forces are multiplied by contact probabilities to enforce consistency. Training combines weighted BCE contact losses (with class-balanced, dataset-level, and smoothness regularizers adopted from HACO), magnitude and direction losses, and pose losses.

Quantitative results

Trained exclusively on ARCTIC, EgoPHI is evaluated in-distribution on ARCTIC s05 and out-of-distribution on H2O s4_ego. Against HACO fine-tuned for force regression, EgoPHI reduces hand-force MAE from 6.62 to 4.03 N on ARCTIC and achieves lower MAE/RMSE with higher volumetric IoU on H2O despite never seeing that dataset. Against PressureVision projected into 2D, EgoPHI substantially improves F1 (.271 vs. .128) and IoU (.157 vs. .068) on ARCTIC. Two caveats deserve emphasis: absolute contact scores remain low (F1 < 0.2, IoU < 0.15), reflecting sparse ground-truth contact regions; and HACO retains higher hand-contact recall, attributable to its training across 14 datasets.

The IRM ablation shows consistent gains—on ARCTIC, 2D IoU rises from 0.021 to 0.157 and volumetric IoU from 1.6 to 15.4—and improves vertex alignment by over 70% (MPV dropping from 64 to 19 cm on ARCTIC). An error-isolation experiment with ground-truth poses attributes residual misalignment primarily to egocentric object pose estimation under occlusion rather than to label noise. A further experiment reveals a tradeoff when fine-tuning on HAMER-estimated hand poses: hand-force metrics improve (MAE 4.03 → 3.32 N) but hand-contact F1 degrades (0.190 → 0.128), because SOFA supervision anchored to ground-truth geometry becomes inconsistent with predicted vertex positions.

Real-world evaluation

For sim-to-real transfer, the authors fabricated two optically instrumented objects—a cube and a cylinder—using frustrated total internal reflection: an LED strip beneath 8 mm acrylic illuminates contacting skin, with intensity calibrated against a scale up to 9.81 N. Eight participants produced roughly 2000 synchronized egocentric/contact frames across diverse grips. On these objects, EgoPHI recovers object contact (F1 of .121–.151) with low force errors (MAE 0.48 N on the cube, 1.35 N on the cylinder)—lower than its ARCTIC errors, though the authors note this may partly reflect limited object variety. This constitutes the first experimental evidence that a vision-only system can estimate contact and 3D forces on non-planar real objects.

Limitations

The paper is explicit about several constraints. The model processes frames independently, discarding temporal structure inherent to force dynamics. It requires a known object template mesh, limiting applicability to unconstrained video. Uniform stiffness in simulation preserves relative force distributions but cannot capture heterogeneous or compliant materials. Only normal contact forces are modeled; tangential forces would require friction-aware simulation and additional sensing, though the authors suggest post-hoc inference via Newton's second law using object motion. Global force equilibrium cannot be enforced without object properties such as mass. Finally, both real-world objects are geometrically simple (piecewise planar and developable surfaces); performance on truly curved objects remains untested.

Conclusion

EgoPHI extends egocentric hand–object perception from contact localization toward dense 3D force estimation on hands and articulated objects, enabled by physics-simulated supervision validated against real optical force measurements. Its quantitative gains over specialized baselines, out-of-distribution generalization, and real-world results establish a credible basis for force-aware egocentric perception, while leaving open robust occluded pose recovery, temporal consistency, and material-dependent modeling.