Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Abstract: In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop $π^n$MPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.

• The paper presents an efficient MPC-guided RL method that integrates MPC-derived trajectories into structured rewards to enhance humanoid locomotion and manipulation.
• It leverages a batched, GPU-accelerated πⁿMPC solver to achieve lower tracking RMSE, enhanced push-recovery, and improved contact constraint adherence.
• The approach enables fast RL inference without online MPC, demonstrating robust hardware transfer on complex tasks such as heavy payload handling.

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Motivation and Problem Statement

This paper addresses the synthesis of robust, agile humanoid locomotion and manipulation policies by integrating model predictive control (MPC) guidance into reinforcement learning (RL), circumventing excessive training overhead and practical bottlenecks inherent in previous MPC–RL frameworks. While MPC offers physically grounded prediction and explicit constraint satisfaction, RL has demonstrated increased robustness and generalization in embodied agent settings. The challenge is to efficiently deliver predictive, constraint-aware guidance from MPC during training-time RL rollouts, enabling scalable learning of whole-body behaviors and transfer to hardware, all without incurring deployment-time MPC computation.

MPC-RL Architecture and Centroidal Dynamics Guidance

The core architecture (Figure 1) injects centroidal-dynamics MPC trajectory supervision into RL episodes via structured rewards. During training, the batched $\pi^n$MPC solver generates per-environment MPC rollouts defining reference landmarks for center-of-mass (CoM), linear/angular momentum, ground reaction forces (GRF), and footstep locations. Prediction-confidence weighting is implemented: near-horizon MPC landmarks receive higher reward weights, tapering for longer-horizon predictions where dynamics mismatch and numerical integration errors predominate. This reward structure fuses model-based signals into bounded, shaped objectives, replacing or augmenting traditional RL reward stacks.

Figure 1: MPC-guided RL architecture leveraging parallel-in-horizon batched MPC solver to supply predictive references and structured reward signals.

During deployment, the trained policy utilizes only proprioceptive and command signals and requires no MPC, ensuring hardware suitability and fast response. Centroidal-dynamics modeling enables explicit reasoning over contacts, momentum processes, and external disturbances for both locomotion and manipulation scenarios.

$\pi^n$MPC: Batched Parallel-in-Horizon GPU Solver

A substantive technical contribution is the development of $\pi^n$MPC, a velocity-form ADMM solver exploiting parallelization across prediction horizons and environments, fully construction-free and memory-efficient (Figure 2). The solver operates directly on time-varying dynamics matrices, sidestepping symbolic QP assembly and pre-compilation typical of traditional sparse QP MPC approaches. Variable splitting and batch-friendly updates yield linear algebraic closed forms, harnessing GPU acceleration. Empirical benchmarks (Figure 2) demonstrate superior scalability and VRAM utilization compared to SOTA solvers (qpth, qpax, CusADi, consensus ADMM), facilitating parallel batch sizes and long-horizon solves that are otherwise prohibitive.

Figure 2: Comparative scalability and solve times for batched GPU MPC solvers, highlighting $\pi^n$MPC's advantages in VRAM efficiency and long-horizon parallelization.

Moreover, Nesterov acceleration is incorporated for expedited convergence, further mitigating training overhead. The solver adapts online to time-varying robot dynamics without loss of performance, critical for RL environments with randomizations and non-stationary contacts.

Empirical Performance: Locomotion and Manipulation Results

Robust evaluation is performed in both simulation and hardware, with policies trained using PPO atop massive parallel batched environments. The default configuration employs a $N=10$ horizon at $10$ Hz, chosen via systematic ablation studies (Table: tracking RMSE, max recoverable push force, constraint satisfaction rate) optimizing velocity tracking, push-recovery, and compliance.

Numerical results show MPC-RL policies achieve:

• Lower RMSE in command velocity tracking, both in-distribution and out-of-distribution
• Significant improvement in push-recovery, with higher recoverable external forces across multiple directions
• Substantially improved contact constraint satisfaction (friction cone, contact wrench cone) compared to RL-only
• Faster reward progression and training efficiency, even when parallel MPC imposes extra computation per iteration

Model-based reward ablations (next-step, averaged, tapered confidence weighting) confirm the horizon-tapered landmark schedule is optimal for trading off structure exploitation vs. sharpness in guidance.

Hardware transfer is showcased in Figure 3: treadmill locomotion, handling unknown payloads, push-recovery, heavy object manipulation (pushing 290 kg = 829% robot mass payload). The trained policy consistently demonstrates robust adaptation in the presence of disturbances and variable task parameters.

Figure 3: Training and deployment scenes illustrating MPC-RL policies in diverse locomotion and manipulation tasks including treadmill, push-recovery, and payload handling.

Simulation snapshots and velocity tracking plots validate improved performance on loco-manipulation tasks, notably heavy box pushing with low-friction contacts (Figure 4). MPC-RL policies discover physically plausible solutions unattainable with pure RL reward design, especially in settings requiring explicit contact force optimization.

Figure 4: Simulation snapshots of box-pushing tasks, highlighting the physical consistency achieved by MPC-guided RL policy relative to pure RL.

Practical and Theoretical Implications

The methodology demonstrates that training-time integration of high-fidelity, constraint-aware predictive optimization into massively parallel RL is systematically tractable, yielding not only improved physical realism but also increased robustness on real hardware. This architecture (MPC signals only at training, RL-only inference) aligns with practical constraints—fast deployment, onboard memory limits, and sensor noise robustness. Broad applicability to loco-manipulation tasks is established, moving beyond pure locomotion. Theoretical implications center on the improved sample efficiency and structured exploration guarantee conferred by model-derived reward shaping, as well as the possibility of extending parallel-in-horizon batch MPC solvers to encompass nonlinear dynamics and more complex contact formulations. Key limitations are the reliance on centroidal abstraction and prescribed contact sequences.

Future Directions

Strategic avenues include:

• Transitioning to nonlinear MPC guidance with neural augmentation to capture full-order dynamics and richer interactive environments
• Expanding to multi-modal contact schedules and adaptive prediction horizons for unstructured terrain and manipulation
• Integrating learned or data-driven dynamic abstractions to inform MPC rollouts
• Quantifying sample efficiency scaling and hardware transferability across varied morphologies and task domains
• Extending batch MPC-RL frameworks for multi-agent and cooperative manipulation tasks

Conclusion

The presented approach establishes an efficient, scalable MPC-guided RL paradigm for humanoid locomotion and manipulation, combining construction-free, parallel-in-horizon MPC solvers and structured prediction-confidence reward schedules. The method achieves superior numerical performance, practical hardware transfer, and improved learning dynamics, suggesting that structured predictive optimization is a viable foundation for future scalable RL approaches in embodied AI.