Complex interactions with deformable objects and human collaboration remain unexplored

Extend the VisualMimic hierarchical sim-to-real framework—consisting of a task-agnostic low-level keypoint tracker trained from human motion data and a task-specific high-level visuomotor keypoint generator—to support complex humanoid loco-manipulation involving deformable objects and collaboration with humans, enabling execution from egocentric visual and proprioceptive inputs without external object state estimation.

Background

VisualMimic integrates egocentric visual perception with hierarchical whole-body control for humanoid robots. A low-level keypoint tracker learns dexterity priors from human motions, and a high-level policy generates keypoint commands from vision and proprioception; the framework enables zero-shot sim-to-real transfer on diverse loco-manipulation tasks.

Despite strong performance on rigid-object tasks such as pushing, lifting, and kicking, the paper explicitly notes that more complex interaction scenarios, specifically those involving deformable objects and collaboration with humans, have not yet been explored within VisualMimic’s current design, motivating future extensions to these domains.