SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework

Abstract: Building humanoid robots capable of generalizable whole-body loco-manipulation in the real world remains a fundamental challenge. Existing methods either rely on laborious task-specific reward engineering, rigidly replay reference motions that fail to generalize, or depend on costly teleoperation that limits scalability. While human videos capture diverse human behaviors, motion priors inferred from them are inherently imperfect, suffering from occlusion, contact artifacts, and retargeting errors that render them unsuitable for direct policy learning. To address this, we present SUGAR, a scalable data-driven framework that converts diverse human videos into deployable humanoid loco-manipulation skills, without any task-specific reward engineering or reference-motion conditioning at inference. SUGAR proceeds in three stages. First, a fully automated pipeline extracts kinematic interaction priors including human-object motion trajectories and contact labels from unstructured human videos. Second, a privileged physics-based refiner uses a unified mimic reward and progressive state pool to transform imperfect priors into physically feasible, high-fidelity skills. Third, refined skills are distilled into a hierarchical autonomous policy consisting of a command generator and a command tracker. We evaluate SUGAR on six representative loco-manipulation tasks in simulation and real-world humanoid hardware. Our method substantially outperforms reference-tracking baselines, and performance scales clearly with the amount of human video data. It also achieves zero-shot real-world transfer with reliable closed-loop execution, autonomous failure recovery, and stable long-horizon performance under external perturbations. Project Page: https://tianshuwu.github.io/sugar-humanoid/

• The paper introduces a novel three-stage pipeline that automates extraction, physics-based refinement, and hierarchical policy distillation from human videos.
• The paper demonstrates superior performance across six tasks with high success rates, showcasing effective sim-to-real transfer and robustness to disturbances.
• The paper highlights how integrating RL-based refinement and domain randomization minimizes manual effort while ensuring high-fidelity, generalizable control policies.

SUGAR: A Data-Driven Framework for Generalizable Humanoid Loco-Manipulation from Human Videos

Motivation and Context

Efficient acquisition of robust, generalizable whole-body loco-manipulation skills on real-world humanoid platforms remains a major open challenge. Traditional policy learning from scratch in simulation is constrained by intensive task-specific reward engineering, laborious environment design, and poor generalization. Reference motion tracking attains high-fidelity reproduction of actions but rigidly ties policy generalization to recorded trajectories, limiting adaptation to novel instances and situations. Teleoperation-based approaches amass high-quality data but fundamentally scale poorly due to high requirements on human labor and specialized hardware.

SUGAR (Scalable hUman-Video-driven Generalizable humAnoid loco-manipulation leaRning) addresses this gap by leveraging unstructured human videos as a scalable data source for learning whole-body robot skills while systematically mitigating the intrinsic noise, occlusions, and physical implausibility in video-derived kinematic motion priors (2605.20373).

Figure 1: Sugar enables generalizable real-world humanoid loco-manipulation from diverse human videos. Six representative tasks are shown, including robust performance under human disturbances.

Methodological Framework

Automated Extraction and Refinement of Kinematic Interaction Priors

SUGAR operates in three tightly coupled stages:

1. Automated Kinematic Prior Extraction: The system reconstructs kinematic trajectories for both the human and manipulated object in monocular videos. It employs SAMBody for human mesh recovery, SAMObj for object mesh generation and scaling, and FoundationPose for object 6D pose estimation. Contact events are programmatically labeled using VLM-based prompting or velocity heuristics, eliminating manual annotation and creating a dataset of noisy but semantically rich human–object interaction priors.
2. Physics-Based Privileged Refinement: Direct policy learning from the extracted kinematic priors is ineffective due to physical implausibility (e.g., penetrations, noisy contacts). SUGAR introduces a privileged RL-based refiner trained with a unified mimic reward comprising tracking, interaction, and regularization components. The refiner leverages a "progressive state pool" for robust initialization and incorporates extensive domain randomization and perturbation during training to produce physically feasible, high-fidelity demonstration trajectories.
3. Hierarchical Policy Distillation: The refined demonstrations are distilled into a deployable policy comprised of a high-level diffusion-based command generator (intent synthesis) and a robust low-level command tracker. This two-tiered scheme separates planning from dynamic execution, facilitating robust closed-loop behavior and transfer to unseen tasks and conditions.

   Figure 2: Overview of Sugar’s architecture, highlighting extraction, refinement, and robust policy training.

   

   Figure 3: Detailed depiction of the training pipeline: refiner, tracker, and generator components and data flow.

Experimental Analysis

Benchmarks and Performance Evaluation

SUGAR is evaluated on six challenging whole-body loco-manipulation tasks involving coordinated locomotion and contact-rich manipulation: Carry Box, Push Box, Kick Box, Pick Bottle, Stand Bottle, and Sit Chair. Each task uses 100 human videos for training and 30 for testing, with deployment on the Unitree G1 humanoid platform.

Quantitative results strongly indicate SUGAR's superiority over state-of-the-art reference-tracking baselines (Resmimic, HDMI), particularly in generalization to unseen initial and target states and in tasks requiring high interaction fidelity. The system achieves high success rates (89.5–98.8% on train, 69.6–99.6% on test; see paper) and low final object position error, substantially outperforming baselines that fail to generalize or solve tasks robustly from noisy video.

Performance scales monotonically with training data volume, demonstrating effective utilization of large unstructured datasets and confirming the benefits of the modular refinement framework.

Figure 4: Success rates as a function of training data size, highlighting strong scaling properties for both train and test conditions.

Ablation Studies and Component Analysis

Component ablation reveals key findings:

• Refinement (w/o Refiner): Eliminating the refiner leads to severe performance drops, supporting the necessity of physically consistent skill generation given noisy priors.
• Progressive State Pool Initialization (PSPI): Replacing PSPI with standard or reference-only initialization reduces stability and task success, signifying the importance of diverse, physically validated starting states.
• Interaction Rewards: Removing interaction-specific rewards causes failures in contact-intensive tasks, as the model no longer enforces essential contact constraints.
• Interaction Robustness Enhancement: Omitting perturbation during training yields overfit, brittle policies that cannot tolerate real-world disturbances.

  Figure 5: Qualitative results on the Carry Box task. The method succeeds where ablations (w/o IR, w/o IRE) exhibit failure or reduced robustness.

Real-World Deployment and Robustness

Policies trained entirely in simulation are deployed zero-shot on the real robot, with success rates exceeding 70% on all tasks and up to 99.2% in Pick Bottle. The policy demonstrates:

• Autonomous Recovery: It can handle out-of-distribution and error states, recovering from execution failures without external intervention.

  Figure 6: The system autonomously recovers from task failure, continuing execution successfully.

• Robustness to Disturbances: The learned policy endures significant external physical disturbances (e.g., human interference) and maintains closed-loop operation in the real world.

  Figure 7: The humanoid robot maintains robust execution under heavy external interference.

• Zero-Shot Generalization: The approach generalizes to unseen objects of varying shapes, masses, and appearances in the real world with no finetuning, suggesting capture of transferable control strategies.

  Figure 8: The policy exhibits zero-shot generalization to novel objects during real-world deployment.

Theoretical and Practical Implications

SUGAR provides a robust and fully automated pathway from large-scale unstructured human video data to autonomous, generalizable humanoid manipulation policies. The introduction of the privileged RL-based refiner and hierarchical policy drastically reduces manual engineering and annotation bottlenecks, while grounding behavior in true physical interaction principles. The systematic integration of domain randomization and robustness enhancement ensures closed-loop real-world viability and adaptive behavior under uncertainty.

Practically, these results pave the way for scalable low-cost skill acquisition for humanoids, enabling rapid iteration as public video repositories expand. Theoretically, SUGAR’s architecture bridges the sim-to-real gap for whole-body loco-manipulation, laying groundwork for future research in general-purpose, vision- and language-conditioned robot learning, and data-efficient transfer.

Conclusion

SUGAR establishes a scalable, robust solution for converting raw human videos into deployable, generalizable humanoid loco-manipulation skills. Its novel three-stage pipeline—automated extraction, privileged RL-based refinement, and hierarchical policy distillation—systematically addresses the limitations of prior work, achieves strong empirical performance, and demonstrates sim-to-real transfer, robust recovery, and generalization. Future work may target fine-grained skill learning, further data utilization efficiency improvements, and integration of rich sensory modalities such as vision and language for broader real-world deployment scenarios.

(2605.20373)