Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

Abstract: In this paper, we propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning (RL) framework. A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data. Specifically, the MPC cost-to-go is evaluated along recorded state-action trajectories, and the parameters are updated to minimize the discrepancy between MPC-predicted values and measured returns. This formulation enables efficient gradient-based learning while avoiding the computational burden of repeatedly solving the MPC problem during training. The proposed method is validated in simulation using a commercial humanoid platform. Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances compared with manually tuned baselines.

• The paper introduces a cost-matching MPC framework that aligns surrogate cost-to-go with measured long-horizon returns to improve humanoid locomotion control.
• It leverages gradient-based, rollout-evaluated parameter updates to adapt cost and model parameters without repeated in-loop optimizations.
• Empirical results demonstrate faster recovery from disturbances, reduced actuator effort, and enhanced closed-loop performance under model errors.

Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

Overview of the CM-MPC Framework

This paper addresses the performance limitations of conventional Model Predictive Control (MPC) for humanoid locomotion stemming from model mismatch, suboptimal cost shaping, and the high computational burden of naive reinforcement learning (RL) integration. The proposed Cost-Matching MPC (CM-MPC) framework leverages data-driven adjustments to cost and model parameters by directly aligning the surrogate cost-to-go of a parameterized MPC with measured long-horizon returns. Critically, the method eschews the need for repeated MPC optimization during learning, instead enabling efficient gradient-based updates via rollout-based evaluation of the MPC's value function.

Figure 1: The Cost-Matching MPC-RL framework integrates an on-policy MPC controller with data-driven cost and model adaptation while maintaining a hierarchical structure typical for humanoid architectures.

The approach maintains the constraint-handling and interpretability benefits of MPC, exploits closed-loop trajectory data for adaptation, and is instantiated on a centroidal dynamics-based OCP for bipedal locomotion.

Core Methodology: Data-Driven Cost and Model Adaptation

The foundation lies in parameterizing both the predictive model and the cost function of the MPC. The cost-matching loss minimizes the squared error between the surrogate cost-to-go (the cumulative discounted cost of the MPC over an $N$-step horizon plus terminal penalty) and the actual long-term return measured along executed trajectories. This objective is efficiently differentiated by rolling out the parameterized dynamics and cost offline, obviating the need for "solve-in-the-loop" during each learning update.

Constraint violations encountered during the forward rollout are incorporated via quadratic penalties in the cost-to-go, allowing the parameterization to cover both physical limitations (e.g., joint bounds, friction cones) and the practical efficacy of the controller as realized through the tracking stack.

Notably, the cost parameterization is expressive: diagonal quadratic weights for reference tracking, base motion shaping, CoM stabilization, swing-foot motion, and torque regularization are all adapted. The centroidal dynamics model is augmented with learnable gain parameters to absorb discrepancies between the simplified planning model and high-fidelity actuator-tracking dynamics.

Analysis of Value Matching and Training Dynamics

The CM-MPC training loop alternates between collecting new closed-loop trajectories and updating parameter vectors to reduce the cost-to-go mismatch. Early in training, as reported in validation, the mean-squared-error between the surrogate value and measured returns reduces sharply, with diminishing improvement as convergence is approached.

Figure 2: Closed-loop validation showing the convergence of the value mismatch MSE between MPC-predicted values and measured long-horizon returns.

Block-wise analysis reveals significant adaptation in terminal cost weights, which grow to effectively proxy future long-term consequences beyond the short receding horizon of the controller. Gains on torque regularization also increase, limiting impulsive actuation, while reference-tracking weights for swing feet decrease, affording more compliance in aggressive terrain or under disturbances.

Joint distributions of surrogate and measured values corroborate that after training, the residuals tighten, and the RMSE reduces, confirming a high-fidelity alignment between MPC computation and realized closed-loop performance.

Figure 3: Value-matching diagnostics pre- and post-training show concentrated density around the identity line, indicative of improved value alignment.

Robustness to Model Errors and External Disturbances

A key empirical contribution is the evaluation of robustness under systematic disturbances—a set of timed lateral and yaw torque pulses applied to the robot during ongoing reference-tracking locomotion. Here, the learned CM-MPC controller exhibits accelerated settling to nominal operation after each disturbance, with notably smoother actuation and reduced high-frequency jitter.

Figure 4: Visualization of representative push disturbances affecting the locomotion platform, illustrating the challenge for robust balance and trajectory recovery.

Statistical metrics collected over six disturbance trials demonstrate:

• Faster settling post-disturbance (31% reduction in settling time)
• Reduced post-push peak error and IAE (Integral Absolute Error)
• Lower control effort and smoother command profiles
• Comparable momentum tracking error, implying that high-level objectives are not sacrificed for mere smoothness

Representative time series show that CM-MPC agents recover the nominal performance band more rapidly, with less aggressive corrective actuation compared to manually tuned baselines.

Figure 5: Closed-loop response of the nominal MPC and the CM-MPC under disturbance; the latter achieves earlier settling with reduced actuator effort and smoother corrective action.

Theoretical and Practical Implications

From an RL perspective, the cost-matching update is a stochastic minimization with convergence guarantees to a first-order stationary point under mild smoothness and bounded-variance assumptions. Notably, constraint satisfaction is separated in train/deploy: soft-penalty violations during learning, strict enforcement at test time, preserving feasibility guarantees and safety margins.

The cost-matching paradigm imparts several key practical benefits:

• Short-Horizon Compensation: By learning terminal penalties aligned to long-term return, myopic behaviors from limited-MPC-horizon control are significantly mitigated.
• Robust Policy Adaptation: The framework leads to policies tailored to systematic modeling errors, including tracking-controller abstractions and unmodeled compliance, increasing reliability in real-world deployment.
• Disturbance Rejection: Explicitly leveraging closed-loop data, CM-MPC enhances the capacity to gracefully handle unforeseen external forces.
• Efficient Training: All learning iterations scale linearly with horizon length $N$ and are "solver-free", in contrast to traditional approaches requiring full constrained optimization per update.

Future Directions

The demonstrated method applies readily to closed-loop systems with open data access and explicit OCP structure. Extensions could involve richer model classes (e.g., neural-augmented dynamics), structured cost regularization for multi-task locomotion, or further integration with safety-critical control primitives. Adaptation across contact sequences or real-world online tightening against acting uncertainties also presents a clear path for future experimentation.

Conclusion

This work establishes cost matching as a practical mechanism for reconciling MPC structure with the empirical realities of high-dimensional robot locomotion. Empirical evidence demonstrates both improved value fidelity and consistently enhanced closed-loop performance, especially in regimes dominated by disturbances and systematic model error. The CM-MPC approach thus represents an efficient, theoretically sound, and easily deployable advancement for robust humanoid control architectures.