Humanoid Whole-Body Controllers

• Humanoid whole-body controllers are algorithms that compute physically and kinematically feasible motor commands for robots under multi-contact and torque constraints.
• They integrate optimization frameworks like quadratic programming with reinforcement and supervised learning to ensure fast, safe, and robust motion control.
• Recent advances incorporate multi-agent interactions, sim-to-real transfers, and teleoperation systems to enhance expressive and adaptive humanoid movements.

Humanoid whole-body controllers (WBCs) are specialized algorithms and software frameworks that compute physically and kinematically feasible motor commands for all degrees of freedom (DoFs) of a humanoid robot, enabling the execution of coordinated, expressive, and robust movements across diverse environments and tasks. WBCs operate under the constraints imposed by underactuation, high-dimensionality, multi-contact scenarios, and physical limitations such as torque, joint bounds, and stability requirements. Recent advancements have extended WBCs far beyond single-agent autonomy, incorporating interactive multi-humanoid and human-object interaction, sim-to-real learning pipelines, robust and expressive motor control, task versatility, teleoperation interfaces, and foundational model architectures.

1. Fundamental Principles and Mathematical Foundations

Humanoid WBCs solve a constrained optimal control problem at high frequency (typically ∼1 kHz), integrating the robot's full-body rigid-body dynamics, kinematic constraints, contact interactions, prioritized task objectives, and safety-critical conditions. The canonical formulation is a quadratic program (QP):

$$\min_{\ddot q,\,\tau,\,\lambda}\;\sum_{i=1}^T\|W_i\,e_i(\ddot q)\|^2 + \epsilon_\tau\|\tau\|^2 + \epsilon_\lambda\|\lambda\|^2$$

subject to dynamics,

$$M(q)\ddot q + C(q,\dot q)\dot q + g(q) = S\tau + J_c^T\lambda$$

physical constraints (torque, joint, contact, friction cone), and task-space objectives (e.g., operational-space tracking, balance, manipulation) (Yuan et al., 25 Jun 2025). Recent works also embed high-relative-degree safety sets via acceleration-based exponential control barrier functions (A–ECBFs) directly into the QP, guaranteeing forward invariance of arbitrary user-defined safety sets (joint limits, self-collision, ZMP, momentum bounds) (Paredes et al., 2023).

2. Modalities: From Classic Optimization to Learning-Based Control

WBCs encompass a spectrum of methodologies, including:

• Analytical and optimization-based approaches: Reduced-mass and inertia-constrained models with explicit geometric relations enable real-time whole-body motion planning, supporting dynamic tasks (e.g. kicking) at time scales <100 μs (Ficht et al., 2020). Hierarchical QP frameworks, e.g. for WIP (wheeled inverted pendulum) humanoids, separately control wheel dynamics (CoM position via MPC or DDP) and body pose/manipulation via task-prioritized QPs (Zafar et al., 2018).
• Supervised and imitation learning: Tracking-based controllers are pre-trained on diverse motion capture datasets, with goal- or trajectory-conditioned outputs, supporting expressive motion tracking, keypoint imitation, and upper/lower body decoupling (Ji et al., 17 Dec 2024). Transformer-encoded motion retargeting generalizes to heterogeneous morphologies and arbitrary MoCap sources (Yao et al., 13 Aug 2025).
• Reinforcement learning (RL): Model-free RL policies, often in PPO or TD-MPC2 frameworks, discover robust, human-like whole-body coordination (e.g. push recovery (Ferigo et al., 2021), versatile locomotion (Xue et al., 5 Feb 2025), and hierarchical world-model control for visual-motion tasks (Hansen et al., 28 May 2024)). Hierarchies are common: low-level RL tracks precise motor targets, while high-level policies select behavioral modes, switch between goal vs. safety/recovery, or synthesize action from vision and language (Lin et al., 2 Mar 2025, Hansen et al., 28 May 2024, Xue et al., 16 Jun 2025).

3. Interactive, Multi-Agent, and Contact-Aware Control

Traditional WBCs suffer from the “isolation issue,” failing to model inter-agent contacts or physically grounded interactive behaviors. Harmanoid (Liu et al., 11 Oct 2025) introduced dual-humanoid controllers that retarget interacting SMPL human meshes to robot skeletons via contact-aware mesh collision detection and centroid alignment. Critical stages include:

• Explicit optimization for inter-agent contact centroid alignment and regularized shape fitting.
• State/action/observation spaces for each robot include intended and realized contact masks, proprioceptive summaries, and partner state.
• Interaction-driven RL controllers (PPO) exploit cross-agent keypoint error rewards and contact compliance terms to enforce realistic, non-interpenetrating contacts.
• Online curriculum adjusts reward weights for seamless transition from single-body fidelity to multi-body coordination, yielding success rates of 0.92 on contact-rich sequences—far exceeding single-agent baselines (ExBody 0.00, HOVER 0.20).

This framework generalizes to arbitrary multi-humanoid and interactive tasks by parameterizing new contact masks and keypoint sets, preserving core optimization and RL machinery.

4. Decomposition and Hierarchical Architectures

Decoupling control tasks—especially in high-DoF robots—alleviates dimensionality curses and improves fault tolerance. The JAEGER framework (Ding et al., 10 May 2025) explicitly separates lower body (locomotion, balance, root velocity commands) and upper body (pose tracking) into independent controllers (lower: gated Transformer-XL; upper: 3-layer MLP). Curriculum learning (RL → supervised init → joint RL) stabilizes convergence. Intervention training, as in HugWBC (Xue et al., 5 Feb 2025), supports real-time external upper-body control via teleoperation or vision input without destabilizing locomotion.

Predictive motion priors (PMP) with CVAE encoding (Lu et al., 10 Dec 2024) capture future upper-body manipulations and inform lower-body RL policies for robust loco-manipulation under remote teleoperation or human demonstration, outperforming monolithic RL.

5. Sim-to-Real Transfer and Generalization

Robustness and transferability constitute central performance metrics for WBCs. Pipelines utilize domain randomization (masses, friction, actuator delays, contact inactivity) (Dugar et al., 30 Jul 2024, Weng et al., 20 Sep 2025, Xue et al., 16 Jun 2025) and teacher-student distillation (DAgger) to bootstrap policies that operate from sparse onboard sensors (e.g. head and hand pose, IMUs), compensating for missing privileged signals (Ji et al., 17 Dec 2024, He et al., 13 Jun 2024). Key strategies include:

• Masked observation and action spaces supporting partial- or multi-modal control (Dugar et al., 30 Jul 2024).
• History-based networks (sequence of proprioceptive frames) for stability in absence of global state (He et al., 13 Jun 2024, Ji et al., 17 Dec 2024).
• RL curriculum and ablation analysis confirming that staged learning of robust standing, walking, and manipulation actions accelerates convergence and maintains low failure rates (< 1%) across diverse tasks and disturbances.

Unified frameworks (e.g. GBC, HOVER) leverage large-scale retargeted MoCap, modular or mask-conditioned policies, and distillation for transfer across morphologies and downstream tasks (He et al., 28 Oct 2024, Yao et al., 13 Aug 2025).

6. Extensions: Vision-Language, Teleoperation, and Foundation Models

Recent developments position humanoid WBCs as foundation models—BFMs—pretrained on massive behavior corpora and generalizable to zero-shot or few-shot adaptation via task embedding or latent reward injection (Yuan et al., 25 Jun 2025). Key design features include:

• Latent vision-language action spaces (LeVERB (Xue et al., 16 Jun 2025)), in which high-level vision-LLMs generate structured “verb” commands decoded in real time by low-level WBC policies for comprehensive task coverage and cross-modal generalization.
• Hierarchical world models using visual observations (RGB or egocentric) and model-based planning (Hansen et al., 28 May 2024), producing high-fidelity, human-preferred motions in navigation, hurdle-bounding, and obstacle avoidance.
• Whole-body teleoperation systems (CHILD (Myers et al., 31 Jul 2025), OmniH2O (He et al., 13 Jun 2024)) supporting joint-level control via reconfigurable leader devices and adaptive force feedback.
• Integration of explicit safety barriers, e.g. ZMP constraints, ECBFs, and human behavior imitation via adversarial discriminators (Lin et al., 2 Mar 2025, Paredes et al., 2023). Robust RL formulations minimize reward under worst-case dynamic perturbations, maintaining feasibility and style fidelity.

7. Open Challenges and Future Directions

Persistent challenges for humanoid WBCs include sim-to-real domain gaps, limited motion datasets, and embodiment generalization. Future directions comprise:

• Scaling multimodal BFMs to vision, touch, and language.
• Hierarchical integration with LLMs and high-level planners.
• Rapid adaptation, benchmarking, and safe certification for real-world deployment.
• Multi-agent and interactive group dynamics.

By unifying analytical optimization, reinforcement learning, foundation modeling, and modular architectures, humanoid whole-body controllers now support interactive, expressive, robust, and transferable motor intelligence at scale (Liu et al., 11 Oct 2025, Yuan et al., 25 Jun 2025, Ji et al., 17 Dec 2024, Lin et al., 2 Mar 2025, Yao et al., 13 Aug 2025).