EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Video

Abstract: Humans naturally understand object physics through everyday interactions, but faithfully predicting complex deformable dynamics, such as elastic materials and fabrics, remains a major challenge for computer vision and robotics. We present EgoPhys, a framework that constructs deformable physical digital twins from egocentric RGB-only video using generalizable priors. EgoPhys overcomes the limitations of existing methods to enable controllable deformable digital twin generation from egocentric videos by distilling per-object inverse-physics solutions into a compact codebook, enabling prediction of dense spring stiffness fields for unseen objects without per-spring test-time optimization. Trained with generalizable priors from diverse egocentric interactions, EgoPhys outperforms baselines in reconstruction, future prediction, and zero-shot generalization. To support training and evaluation, we curate an egocentric interaction dataset covering diverse deformable objects, scenes, and manipulation styles. We deploy EgoPhys on a real xArm6 robot, demonstrating that a digital twin initialized from a single egocentric human play video can serve as an internal world representation to aid in deformable-object planning, highlighting egocentric RGB observations as a scalable path toward real-to-sim pipelines.

• The paper introduces a novel method for building digital twins of deformable objects from egocentric video using a prototype-based codebook that eliminates test-time optimization.
• It leverages state-of-the-art trackers and 3D lifting models to reconstruct temporally coherent 4D point clouds and achieve superior performance in metrics like Chamfer distance and IoU.
• The approach enables zero-shot generalization and efficient sim-to-real transfer for robotic manipulation, significantly reducing configuration error in physical tasks.

Authoritative Summary of "EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Video" (2606.16202)

Motivation and Problem Formulation

The paper addresses the challenge of constructing simulatable physics models of deformable objects, focusing specifically on creating digital twins from egocentric RGB-only video. Existing approaches require controlled capture environments, depth sensing, or multi-view setups, thereby limiting scalability and accessibility. EgoPhys eliminates these constraints by leveraging common human-object interaction video and generalizable physical priors. The key technical question: Can deformable physical digital twins with accurate and generalizable dynamics be generated from monocular egocentric video, without test-time optimization or specialized sensors?

Methodology

EgoPhys operates via a pipeline that extracts temporally coherent 4D point clouds from wearable egocentric video using state-of-the-art trackers and 3D lifting models (notably VGGT [11]). Using this reconstructed geometry, a coarse spring-mass simulator is initialized via CMA-ES-based inverse physics, capturing global parameters and graph topology.

The central innovation is the distillation of dense per-object spring stiffness fields into a compact, state-conditioned material codebook. This codebook is trained to predict spring stiffnesses for unseen objects by mapping local static and dynamic features to prototype mixtures under tension/compression. The prototype bottleneck regularizes the model and dramatically reduces per-object refinement, replacing test-time optimization with efficient inference that leverages shared material priors. The codebook is dynamic and sign-aware, enabling asymmetrical response to tension and compression states.

Experimental Design and Evaluation

A new dataset is curated, consisting of egocentric manipulation videos across various object types (plush toys, towels, bags) and manipulation modalities. Evaluation employs multiple metrics:

• Physical Consistency: Chamfer Distance (CD), Track Error (TE), Intersection-over-Union (IoU)
• Visual Consistency: PSNR, SSIM, LPIPS

EgoPhys is benchmarked against adapted physics-based baselines (PhysTwin [9], Spring-Gaus [18]), all fed the same egocentric-derived observations.

Numerical Results

• Reconstruction and Resimulation: EgoPhys yields substantial improvements in Chamfer distance, track error, and IoU over baselines. For future prediction, performance metrics indicate more reliable deformation and motion tracking under egocentric deformations.
• Zero-shot Generalization: In held-out scenarios, EgoPhys achieves CD = 0.0319, TE = 0.0621, IoU = 57.9%. These outperform PhysTwin (CD = 0.0457, TE = 0.0818, IoU = 51.6%) despite PhysTwin’s reliance on dense per-scene optimization.
• Ablation Studies: The prototype codebook (dynamic K = 4) delivers optimal compactness-accuracy tradeoffs, outperforming direct MLP regression and static codebooks. Notably, the learned prior bypasses test-time per-spring refinement, enhancing inference speed and preserving accuracy.
• Sim-to-Real Transfer: Using EgoPhys digital twins as forward models in an MPPI planner, waypoints are transferred to an xArm6 robot. Quantitative results show up to 77.6% reduction in configuration error (Chamfer distance) for deformable object tasks (pulling and lifting). Deformation patterns are consistent between simulation and real robot execution, demonstrating practical utility for real-to-sim pipelines from human video.

Theoretical and Practical Implications

The EgoPhys framework redefines deformable real-to-sim pipelines for manipulation, making physical modeling accessible from unconstrained, wearable RGB data. By amortizing the stiffness prediction and distilling into reusable material prototypes, the approach overcomes the limitations of task-specific per-scene optimization. The codebook-based representation serves as a shared transferable prior, enabling zero-shot generalization across novel objects and interaction modes.

Practically, this enables scalable downstream planning and control for robotics, reducing the need for manual parameter tuning or instance-specific physical calibration. The deployment on robot manipulation validates sim-to-real feasibility even with challenging cloth-like and plush objects, which are underrepresented in existing real-to-sim evaluation frameworks.

Theoretically, the design highlights the efficacy of prototype-based latent representations for modeling complex material response in deformable dynamics, suggesting further generalizability and transfer capacity as egocentric datasets scale. The results point to the utility of dynamic conditioning in motion prediction and support the principle that material priors derived from human interaction can be leveraged for embodied AI.

Future Directions

Current limitations include modest scale of the egocentric object dataset and restricted diversity of materials and manipulation tasks. Extension to broader datasets, longer interaction horizons, and complex contact-rich skills are essential for practical deployment in diverse manipulation contexts. Closed-loop planning, richer contact modeling, and real-robot evaluation will be critical for advancing the framework.

Future developments may focus on: (1) scaling the codebook and prior to encompass universal material models, (2) integration with advanced 3D generation and vision stacks, and (3) rigorous benchmarking across a variety of deformable tasks (including autonomous skill learning and transfer).

Conclusion

EgoPhys presents an effective method for learning generalizable deformable-object physics models from egocentric human-interaction video without specialized sensors or optimization overhead. The codebook mechanism enables rapid, zero-shot construction of digital twins for novel objects and manipulation scenarios, and its efficacy is validated both in simulation and on real robot hardware. This work lays a foundation for scalable, real-to-sim pipelines in robotics and embodied AI, suggesting new directions for leveraging human interaction data as a source of physical knowledge.