Address open challenges in egocentric physical perception

Develop methods for robust object-pose recovery under occlusion, temporally consistent force estimation, and material-dependent interaction modeling in egocentric physical perception.

Background

EgoPHI estimates hand–object contact and 3D forces from a single egocentric RGB frame, using known object geometry and physics-simulated supervision. The discussion identifies several limitations of this frame-based formulation: it discards temporal structure, assumes uniform stiffness across objects, models only normal contact forces, and lacks object properties such as mass for enforcing global force equilibrium.

The authors explicitly identify robust object-pose recovery under occlusion, temporal force consistency, and material-dependent interaction modeling as unresolved challenges. These issues matter because egocentric views frequently occlude the manipulated object, forces evolve dynamically during interaction, and real objects may have heterogeneous or compliant material properties that are not represented by the current simulation.

References

At the same time, the remaining errors highlight open challenges in egocentric physical perception, including robust object pose recovery under occlusion, temporal force consistency, and modeling material-dependent interaction.

— EgoPHI: Estimating Contact and Force from Egocentric Vision  (2608.13014 - Ilic et al., 13 Aug 2026) in Conclusion; Discussion and Limitations

Failures frequently coincide with incorrect estimates of the asymmetric peg's orientation under occlusion, while grasp instability can arise from the small contact patch between the rounded Allegro fingertips and the rounded side of the peg under fast arm motion. Fingertip tactile sensing on Flexiv-Sharpa mitigates contact ambiguity, but robust object-centric perception under occlusion remains open.

— ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning  (2608.19182 - Lee et al., 19 Aug 2026) in Section Limitations