GRAIL: Humanoid Loco-Manipulation Framework

• The paper introduces a fully digital pipeline that fuses 3D asset synthesis, video foundation model priors, and 4D human–object reconstruction to scale humanoid loco-manipulation.
• It employs robust sim-to-real transfer techniques, achieving high success rates with 84% pick-up and 90% stair-climbing on Unitree G1 robots.
• Comparative analysis reveals that GRAIL outperforms traditional frameworks by offering modular, physically consistent control policies through interaction-aware optimization.

GRAIL: Generating Humanoid Loco-Manipulation

GRAIL refers to a collection of frameworks and digital pipelines for generating, training, and deploying humanoid loco-manipulation behaviors—complex coordinated motions involving both bipedal locomotion and fine-grained whole-body object interaction. Across its most recent instantiations, the GRAIL paradigm emphasizes the use of privileged scene representations, large-scale synthetic data generation (notably via 3D assets and video foundation models), interaction-centric 4D human–object reconstruction, and modular sim-to-real pipelines. This results in scalable, generalizable, and physically consistent control policies for humanoid robots, realized without the need for extensive real-world teleoperation or instrumentation. The term "GRAIL" emerged in the context of this unified, simulation-driven approach to large-scale humanoid skill development (Xie et al., 3 Jun 2026).

1. Fully-Digital Data Generation Pipeline

The core innovation of GRAIL is an end-to-end workflow that remains entirely virtual until robot deployment. This pipeline consists of three primary modules:

• 3D Assets & Scene Synthesis: Diverse object meshes are sourced from large-scale datasets (e.g., Robocasa, ComAsset, OMOMO, Hunyuan3D). Two main scene templates are procedurally generated: an empty floor and a furnished room. Each object is physically simulated into a stable pose and placed according to affordance class, with precise knowledge of object geometry, metric scale, and camera intrinsics/extrinsics.
• Video Foundation Model (VFM) Priors: Given a rendered first frame and an automatically generated caption, a static-camera VFM (e.g., Kling v2.5) generates high-resolution, multi-second human–object interaction videos. All camera and scale parameters are preserved, directly linking 3D simulation assets to generated video.
• HOI Reconstruction and Retargeting: The synthetic video is processed with foundation models to recover metric 4D (space-time) trajectories for both the human (SMPL-X parametric body model) and object. These trajectories are retargeted to the robot’s kinematic chain (e.g., Unitree G1) using General Motion Retargeting (GMR), yielding dense action references for learning downstream policies (Xie et al., 3 Jun 2026).

This privileged setup eliminates the scale ambiguity, appearance mismatches, and depth inconsistency typical in "in-the-wild" video reconstruction, providing reliable pseudo-demonstrations across a broad range of humanoid tasks.

2. Interaction-Aware 4D Human–Object Reconstruction

GRAIL's 4D HOI recovery pipeline operates as follows:

• Initial Per-frame Estimation: GENMO is used for per-frame SMPL-X pose estimation; WiLoR refines MANO hand parameterization; FoundationPose tracks object pose, with all assets parameterized in known camera and world spaces.
• Joint Optimization: All human and object parameters are refined via an interaction-aware loss function:

$$L = \lambda_\mathrm{kp} L_\mathrm{kp} + \lambda_\mathrm{proj} L_\mathrm{proj} + \lambda_\mathrm{depth} L_\mathrm{depth} + \lambda_\mathrm{cont} L_\mathrm{cont} + \lambda_\mathrm{reg} L_\mathrm{reg}$$

where terms account for 2D keypoint reprojection, object image alignment, depth matching (MODGE-2 + SAM2), hand–object contact consistency, and kinematic regularization. Depth gaps in contact regions are penalized using view-space Chamfer distance.

• Morphology Anchoring: By fixing the character’s body shape to match the robot and using true 3D scene geometry, the optimizer avoids common pathologies of scale mismatch and unphysical contacts, resulting in high-fidelity, robot-executable interaction trajectories (Xie et al., 3 Jun 2026).

3. Policy Learning: Task-General Loco-Manipulation Trackers

From the retargeted kinematic labels, the GRAIL framework trains two specialized, task-general control modules:

• Object-Aware Latent Adaptor: This adapts a base whole-body policy (SONIC backbone) to manipulation by conditioning on the current/future object pose, shape code (BPS), hand–object contact features, and proprioception. The adaptor outputs a residual correction in latent space, plus hand primitives. Rewards sum motion tracking, object tracking, grasp metrics, and regularization, with learning accomplished by PPO while freezing the base policy network.
• Scene-Aware Tracker: For non-flat or non-trivial terrain (stairs, slopes, sitting), SONIC is augmented with an 11x11 local height map encoded via a small CNN, fused with proprioception and previous internal state. Rewards emphasize accurate tracking and contact feasibility, with further regularization on ankle accelerations and joint velocities. Fine-tuning is performed via PPO, amortizing over large pools of scene-conditioned trajectories.

Both modules are trained on thousands of unique sequences per family and are amortized for efficient adaptation (Xie et al., 3 Jun 2026).

4. Sim-to-Real Transfer and Evaluation

For deployment, GRAIL distills physics-based policies into egocentric visual policies that operate on real robot sensor data:

• Visual Policy Distillation: Behavioral cloning or diffusion policy learning is performed, mapping head-mounted RGB streams to latent tokens for SONIC at 10 Hz. Extensive domain randomization covers lighting, textures, camera calibration noise, and image perturbation, following practices like those in VIRAL. The policy is validated in simulation before transfer to real robot hardware (Xie et al., 3 Jun 2026).
• Empirical Results: On the Unitree G1 platform, GRAIL-trained policies achieve 84% mean success on diverse object pick-up tasks (both seen and unseen objects) and 90% on stair-climbing, without real-robot demonstration data. In simulation, GRAIL outperforms baselines such as HDMI and ResMimic by substantial margins (81.4% vs ~48–49% skill success rates; significantly lower object pose and MPJPE-L errors).

The overall dataset comprises over 20,000 sequences (covering pick-up, manipulation, sitting, and terrain traversal), each 5–10 s at 24 fps, with generation requiring ~14 min per sequence for full pipeline execution.

5. Comparative Context: Methodological Innovations

GRAIL distinguishes itself from prior humanoid loco-manipulation frameworks via:

• Scale and Generality: Data is produced entirely virtually at scale, avoiding costly teleoperation and bypassing the sim-to-real gap typical in small-scale or object-specific demonstration-based methods.
• Interaction Priors from VFMs: The system leverages advanced video diffusion models for realistic human–object demonstration synthesis under known camera and geometry priors, which is not handled by classic LDM-only or mocap-based approaches (Taouil et al., 23 Apr 2025).
• Modularity: The entire framework is partitioned into well-defined stages (asset assembly, VFM sampling, 4D optimization, retargeting, modular policy adaptation) and connects to the robot only at final deployment. Adaptors can be swapped or rapid-fine-tuned for new task families.
• Physical Executability: By anchoring reconstruction and policy learning in privileged geometry, the generated behaviors maintain high contact and actuation feasibility, as demonstrated by both simulated and real-robot evaluations. Ablations show that omitting object-aware observations or the latent adaptor severely degrades skill success.

6. Dataset Composition and Experimental Metrics

A summary of GRAIL's key dataset and performance metrics is provided below:

Component	Scale/Result	Reference
Object Meshes	1,000 unique meshes, 4 catalogs	(Xie et al., 3 Jun 2026)
Terrains	1,000 layouts (stairs, curbs, slopes)	(Xie et al., 3 Jun 2026)
Sequence Count	20,000+ sequences (~5–10 s @ 24 fps each)	(Xie et al., 3 Jun 2026)
4D HOI Executability	88.9% (vs. 10.5–24.0% baseline)	(Xie et al., 3 Jun 2026)
Real G1 Pick-up	84% (seen), 80% (unseen, per-object range)	(Xie et al., 3 Jun 2026)
Real G1 Stairs	90% success (20 trials, visually guided)	(Xie et al., 3 Jun 2026)

Primary limitations include requirements for high-quality assets and cooperative VFM priors, along with reconstruction difficulties under occlusion or rapid motion. Approximately 20% of raw generated videos are filtered for tracking difficulty or inconsistency.

7. Contributions, Open Challenges, and Directions

GRAIL demonstrates that human-like, physically consistent humanoid loco-manipulation skills—including trajectory generation, grasping, and dynamic traversal—can be synthesized, learned, and deployed entirely from virtual data, scaling across task types without real robot trial data. The approach introduces interaction-aware 4D optimization routines, specialized object/scene-aware control adaptors, and effective sim-to-real transfer protocols.

Open challenges include handling new motion/task families (requiring further tracker adaptation or fine-tuning), enhancing robustness for occluded or highly dynamic scenes, and integrating more advanced scene understanding (e.g., joint vision–language planning) to replace fixed task plans. Additional research is warranted on generalizing the paradigm to less-structured and higher-DOF interaction environments and refining VFM sampling and filtering for broader motion diversity (Xie et al., 3 Jun 2026).