Learning Sim-to-Real Humanoid Locomotion in 15 Minutes (2512.01996v1)

Abstract: Massively parallel simulation has reduced reinforcement learning (RL) training time for robots from days to minutes. However, achieving fast and reliable sim-to-real RL for humanoid control remains difficult due to the challenges introduced by factors such as high dimensionality and domain randomization. In this work, we introduce a simple and practical recipe based on off-policy RL algorithms, i.e., FastSAC and FastTD3, that enables rapid training of humanoid locomotion policies in just 15 minutes with a single RTX 4090 GPU. Our simple recipe stabilizes off-policy RL algorithms at massive scale with thousands of parallel environments through carefully tuned design choices and minimalist reward functions. We demonstrate rapid end-to-end learning of humanoid locomotion controllers on Unitree G1 and Booster T1 robots under strong domain randomization, e.g., randomized dynamics, rough terrain, and push perturbations, as well as fast training of whole-body human-motion tracking policies. We provide videos and open-source implementation at: https://younggyo.me/fastsac-humanoid.

• The paper demonstrates that scalable off-policy RL (FastSAC and FastTD3) can train robust, high-dimensional humanoid control policies in just 15 minutes on a single RTX 4090.
• The methodology leverages large-batch, high-throughput simulations with advanced normalization and minimalist reward design to overcome domain randomization challenges.
• Experimental results validate sim-to-real transfer by achieving reliable locomotion and motion-tracking on Unitree G1 and Booster T1 platforms.

Rapid Sim-to-Real Humanoid Locomotion with Fast Off-Policy RL

Introduction

"Learning Sim-to-Real Humanoid Locomotion in 15 Minutes" (2512.01996) systematically demonstrates that scalable off-policy deep RL—specifically, the FastSAC and FastTD3 variants—can enable robust, high-dimensional humanoid control policies to be trained end-to-end in the domain-randomized simulation in just 15 minutes on a single RTX 4090 GPU. The work targets the longstanding bottleneck in sim-to-real cycles for humanoids, where environment mismatch and the high dimensionality of whole-body control interact with sample inefficiency. The authors propose a streamlined recipe involving algorithmic, architectural, and reward design adaptations, validated on both simulated and physical Unitree G1 and Booster T1 platforms across locomotion and motion-tracking tasks.

Figure 1: Summary of results showing 15-min training of robust humanoid control on a single GPU and the advantage over PPO in whole-body tracking with massive environments.

Methods

The recipe centers on off-policy RL and leverages FastSAC (entropy-regularized) and FastTD3 (deterministic), each enhanced for large-batch, high-throughput parallel simulation. The following design decisions are key:

• Algorithmic hyperparameters: The paper shows that layer normalization markedly improves training stability in both FastSAC and FastTD3 for high-D control (Figure 2c), while averaging twin Q-values outperforms minimum (clipped) double-Q learning (Figure 2a), with overestimation bias well-controlled via tuning.
• Exploration: In FastSAC, restricting the stochasticity ($\sigma_{max}$) and low initial entropy temperature ($\alpha$) stabilizes maximum entropy exploration in the presence of strong domain randomization.
• Environmental parallelism: The approach scales environments (upwards of 16k) to maximize gradient steps per second, exploiting off-policy data reuse and speeding up convergence, especially in complex tracking tasks (Figure 2f).
• Normalization: Observation and layer normalization provide superior gradient flow and policy stability at scale, superseding simple observation normalization (Figure 2c).
• Action bounds: Joint-limit-aware action bounds for the Tanh policy head simplify tuning and minimize torque under-utilization.

Figure 2: Ablations showing improvements from algorithmic changes for stability, convergence, and sample efficiency.

In reward design, the approach explicitly minimizes complexity: less than 10 terms for each task. Locomotion control uses only well-motivated tracking, stability, and regularization terms, with a curriculum on penalty weights and symmetry augmentation. Whole-body tracking builds on BeyondMimic's minimalist structure.

Experimental Results

Locomotion and Domain Randomization

The paper rigorously evaluates on velocity-tracking locomotion for G1 and T1 humanoids, with policies trained under randomized dynamics, terrains, and perturbations. Training consistently completes within 15 minutes (single RTX 4090, batch size 8k), demonstrating that PPO fails to match wall-time or robustness under strong randomization. Push perturbations at 1–3s intervals (Push-Strong) are handled by FastSAC and FastTD3 but not by PPO (Figure 3).

Figure 3: Velocity tracking learning and sim2real results in challenging randomized domains, with FastSAC+FastTD3 outperforming PPO.

Algorithmic Improvements

Upgrades to FastSAC—including curated normalization, exploration, and critic-averaging—address prior instability and outperform the previous FastSAC setup (Figure 4).

Figure 4: Quantitative improvement from the refined FastSAC design over baseline.

Whole-Body Tracking

Large-scale experiments on human motion tracking demonstrate that FastSAC and FastTD3 are competitive or superior to PPO in training time and final performance (Figure 5), scalable with more environments and GPU resources. Notably, FastSAC demonstrates superior learning on temporally extended tasks (dance motion). Policies are robust to diverse domain randomization.

Figure 5: Whole-body tracking task results, highlighting the scalability and performance boost with FastSAC/FastTD3.

Sim-to-real deployment is validated on Unitree G1 hardware, maintaining full-body control over long horizons (Figure 6), verifying transfer of behavior learned entirely via the proposed recipe.

Figure 6: Real-world deployment of FastSAC-trained WBT controllers (Dance, Box-lifting, Push) on Unitree G1.

Discussion and Implications

This work substantiates that, with appropriate normalization, network design, and reward minimalism, off-policy RL can outperform on-policy methods like PPO for high-DOF humanoid sim2real transfer—contradicting entrenched practice. The training regime’s scalability suggests that further acceleration is possible with increased hardware and environment count. The minimalist reward design enables rapid hyperparameter search and robust transfer.

Practically, this regime supports fast iteration in sim-to-real projects, facilitating agile adaptation to simulation-real mismatches via rapid retraining—crucial for bipedal robotics. Theoretically, the results motivate further paper of high-dimensional function approximation, sample-efficient RL under heavy domain randomization, and new normalization techniques for continuous control.

Future AI research directions following from this work include:

• Generalizing the presented recipe to vision-based and multi-contact manipulation tasks.
• Merging foundation model policy architectures with the proposed fast RL loop for x-embodiment.
• Developing advanced curriculum learning and reward generation methods leveraging LLMs for further abstraction and task diversity.

Conclusion

The paper establishes that full-body sim-to-real humanoid controllers can be trained in minutes with off-policy RL, robust to strong domain randomization, through a principled recipe and architectural choices. The approach closes the practical gap between massive simulator throughput and iterative sim-to-real robot learning. The open-sourced implementation offers a reproducible and extensible foundation for the robotics community to advance general-purpose and rapid-adaptation humanoid AI (2512.01996).