MeshMimic: Geometry-Aware Humanoid Motion

• MeshMimic is a geometry-aware framework that integrates scene reconstruction and reinforcement learning for autonomous, terrain-adaptive humanoid motion.
• It employs a multi-stage pipeline combining π³ depth estimation, ViTDet-SAM segmentation, and joint alignment to create metrically consistent human and environment geometry.
• The system retargets human motion to robots and applies reinforcement learning with asymmetric PPO to achieve robust real-world performance on complex parkour tasks.

MeshMimic is a geometry-aware framework for humanoid motion learning that unifies 3D scene reconstruction and reinforcement learning (RL) to enable complex terrain-aware humanoid behaviors directly from monocular RGB video. Addressing the limitations of traditional motion capture (MoCap) pipelines—namely the absence of geometric context and high acquisition costs—MeshMimic introduces methods for inferring coupled “motion–terrain” interactions and transferring them to physics-based humanoid agents, thereby offering a scalable real-to-sim-to-real solution for autonomous robot evolution in unstructured environments (Zhang et al., 17 Feb 2026).

1. Geometry-Aware Reconstruction

MeshMimic employs a multi-stage pipeline to jointly reconstruct both the dynamic human trajectory and the 3D geometry of the surrounding scene:

• Scene Reconstruction via π³: The pipeline applies π³ [Wang et al. 2025] to estimate per-frame depth maps $D^t$, camera poses $[R^t \mid \mathbf t^t]$, and intrinsics $K$. Large static surfaces are modeled as planar polygonal primitives, denoising the raw geometry while preserving contact-sensitive terrain features.
• Human Segmentation and Reconstruction: Using a ViTDet–SAM2 cascade, the system detects and tracks the subject through binary silhouettes $M^t$ and 2D pose keypoints $\mathbf u^{2D}_j(t)$. This data conditions SAM3D-Body, which fits a SMPL-X model per frame, yielding shape $\boldsymbol\beta$, local pose $\boldsymbol\theta^t$, translation $\mathbf t^t$, and orientation $\phi^t$.
• Joint Alignment and Losses: Since π³ and SAM3D provide geometry in different coordinate frames and scales, MeshMimic applies joint optimization. Let $\mathcal V_h^t$ be the SMPL-X mesh vertices and $\mathcal P^t$ the scene point cloud. The objective:

$$L_{\rm recon} = \lambda_{\rm align}\,L_{\rm align} +\lambda_{c}\,L_{c} +\lambda_{p}\,L_{p} +\lambda_{sm}\,L_{sm} +\lambda_{fs}\,L_{fs}$$

with terms: - $L_{\rm align}$: 2D joint reprojection and Chamfer distance alignment - $L_c$: Contact anchoring between human mesh and terrain - $L_p$: Penetration penalty via truncated TSDF and Huber loss - $L_{sm}$: Smoothness in global translation - $L_{fs}$: Foot-snapping for near-surface foot joints

This reconstruction stage produces metrically consistent human and scene geometry, suitable for physically plausible retargeting.

2. Kinematic Consistency Optimization

After reconstruction, per-frame SMPL-X outputs may be temporally inconsistent. MeshMimic addresses this by formulating a sequential quadratic programming (SQP)-style kinematic optimization over the pose trajectory $\{ \mathbf q_t \}$:

$$J_{\rm kin} = \sum_{t=1}^T \| \mathbf q_t - f(\mathcal V_h^t) \|^2 + \alpha_{\rm vel} \sum_{t=2}^T \| \mathbf q_t - \mathbf q_{t-1} \|^2$$

Here, $f(\mathcal V_h^t)$ extracts joint angles from reconstructed meshes, while the second term regularizes velocity for temporal coherence. The result is a smoothed, robot-operable joint trajectory that accurately tracks observed human motion in the reconstructed scene (Zhang et al., 17 Feb 2026).

3. Contact-Invariant Retargeting

MeshMimic introduces a “MeshRetarget” method for transferring reconstructed human trajectories to morphologically distinct humanoid robots while maintaining discovered human–scene contacts:

• An interaction mesh collects corresponding human and robot keypoints, augmented by points from proximate terrain regions.
• Retargeting minimizes Laplacian deformation energy, constrained by collision avoidance (enforced via TSDF queries), robot joint and velocity limits, and stance-foot anchoring.
• If minor penetrations persist, a global translation correction is applied, using the TSDF gradient at robot contacts to iteratively offset the robot mesh away from terrain collisions, guaranteeing clearance above a safety threshold.

This procedure ensures that retargeted motions preserve the critical physical interactions originally observed between human and environment.

4. Reinforcement Learning for Terrain-Aware Control

The retargeted, contact-annotated motion and terrain data creates an immersive dataset for reinforcement learning:

• Policy Architecture: An asymmetric Proximal Policy Optimization (PPO) setup is adopted. Actor observations comprise reference joint positions, torso pose errors, proprioceptive signals, and action histories. The critic receives additional privileged scene geometry.
• Reward Structure: Rewards enforce torso and body tracking, penalize action rates and soft joint limit violations, and employ automatic failure detection upon large deviations.
• Training Regime: Initial pre-training leverages approximately 50 hours of non-interactive motion, followed by fine-tuning across 8 diverse scene-interactive tasks (stepping, vaulting, climbing) in both simulation and physical robots. Deployment occurs at 50 Hz onboard (NVIDIA Jetson Orin) in reconstructed scenes.

Empirically, this approach enables robust transfer to the real world, achieving high success rates in previously unstructured or complex terrains (Zhang et al., 17 Feb 2026).

5. Empirical Results and Evaluation

MeshMimic demonstrates the following performance on subsets of the SLOPER4D benchmark:

Metric	MeshMimic	Improvement vs. VideoMimic
WA-MPJPE (mm)	94.32	–15.9%
W-MPJPE (mm)	518.98	–25.5%
Chamfer distance (m)	0.61	–18.7%

On eight parkour-style tasks, the full system achieves the highest mean training reward in IsaacLab, with up to 100% real-world success rates on simpler tasks and approximately 80–100% on multi-contact, scene-interactive scenarios. An ablation study shows that supplementing with global torso position increases long-horizon success by 20% at a minor cost to performance on rapid, short tasks (Zhang et al., 17 Feb 2026).

6. Limitations and Future Prospects

Current limitations include susceptibilities of monocular depth priors to occlusion and lighting, leading to occasional errors in contact prediction due to fragmentary geometry. Longest-sequence reconstructions employ off-board MoCap for torso localization; future developments are planned for onboard visual–inertial fusion for global reference via exteroception. The present retargeting assumes static scenes; support for dynamic objects or deforming surfaces would necessitate a time-varying contact model.

Ongoing research directions involve a fully closed-loop perception–planning–control pipeline capable of generalizing to previously unseen terrain, dynamic obstacles, and extended parkour challenges directly from video observation (Zhang et al., 17 Feb 2026).