Manipulation-Centric Whole-Body Controllers

• Manipulation-centric whole-body controllers are unified frameworks that integrate locomotion and manipulation into a single control strategy.
• They employ a mix of model-based MPC, reinforcement learning, and hierarchical QP methods to optimize full-body motion and contact dynamics.
• These controllers enhance robotic performance by ensuring seamless base-arm coordination, robust contact handling, and efficient energy utilization.

Manipulation-centric whole-body controllers enable robots—especially humanoids, mobile manipulators, and legged platforms—to perform complex manipulation tasks in coordination with their base, limbs, and the environment. These controllers unify motion planning, control, and contact modeling across the entire robot body, allowing for robust, efficient, and versatile loco-manipulation in dynamic and unstructured settings. State-of-the-art frameworks span model-based optimization (e.g., bilevel MPC with Bézier parameterization), end-to-end reinforcement learning of unified policies, hierarchical task-priority stacks, policy distillation from full-body or human-motion oracles, and generative goal-driven control architectures. Manipulation-centric approaches are distinguished by their direct coupling of base and arm/hand motion, integration of environmental force constraints, and explicit capability for seamless transition among control modes.

1. Unified Abstractions and Problem Formulations

Manipulation-centric control adopts abstractions that treat whole-body motion as unified tracking or optimization over all degrees of freedom, eschewing the hierarchical separation of locomotion and manipulation. For example, controllers like HOVER cast all low-level humanoid tasks—walking, whole-body manipulation, tabletop object handling—as full-body kinematic motion imitation problems, parameterized by a stack of 3D keypoint positions, joint angles, and root velocities or orientations (He et al., 28 Oct 2024). Similarly, frameworks such as MaskedManipulator frame the control problem as a goal-conditioned Markov Decision Process (MDP), where high-level sparse goals on hand, head, pelvis, or object future poses are mapped to full-body PD-controller targets and synthesized motions (Tessler et al., 25 May 2025). RL-based unified policies for legged manipulators optimize joint-level commands to maximize cumulative rewards that blend manipulation, locomotion, and energy penalties, learning motor synergies reminiscent of biological systems (Fu et al., 2022). In model-based schemes for mobile dual-arm manipulation and omnidirectional wheel-legged robots, the planning horizon is split between task-centric object motion and whole-body trajectory refinement, both solved as optimal control or MPC problems with contact/dynamics constraints (Du et al., 30 Oct 2024, Chen et al., 17 Sep 2025).

2. Architectural and Algorithmic Methodologies

The dominant methodologies span several paradigms:

• Multi-mode Masked Policies (Distillation): The HOVER framework uses an oracle trained to imitate full-body kinematic trajectories via RL on human MoCap, followed by policy distillation with DAgger, producing a student policy capable of arbitrarily combining upper-body, lower-body, and hand-specific manipulation tasks via binary masks (control modes and sparsity selection) (He et al., 28 Oct 2024).
• Bilevel MPC with Bézier Parameterization: Dual-arm mobile manipulation is controlled using two MPCs—MPC-T for long-horizon, object-centric Bézier-curve planning in SE(3), and MPC-W for short-horizon joint-space motion generation with predictive admittance control and hard robot constraints. Bézier parameterization drastically reduces variable count, improves smoothness, and enables real-time feasibility (Du et al., 30 Oct 2024).
• Whole-body Inverse Dynamics MPC: Legged loco-manipulation employs multi-shooting MPC that directly optimizes joint torques by enforcing full-order rigid-body dynamics, contact force constraints, and task-space end-effector objectives. All force/motion objectives and robot-environment contacts are handled in a single predictive layer, with online solvers exploiting block sparsity (Molnar et al., 24 Nov 2025).
• Goal-conditioned Generative Policies: MaskedManipulator trains physics-based trackers and then distills them into transformer-based, generative policies (including diffusion and C-VAE architectures) capable of filling in long-horizon, under-specified goals, such as sequential chaining of object-hand interaction tasks (Tessler et al., 25 May 2025).
• Hierarchical Stack-of-Tasks and Task-priority QP: Several works implement recursive prioritized stacks (joint limits, self-collision, torso/base tracking, end-effector tasks) via QP optimization or analytic nullspace projectors to achieve real-time blending of manipulation and locomotion objectives, ensuring safety constraints are always respected (Arduengo et al., 2019, Teng et al., 2021).
• Impedance and Admittance Control: Approaches such as whole-body Cartesian impedance render desired stiffness and compliance between base and arm, embedding double-mass–spring–damper models in QP or MPC formulations for compliant manipulation and force interaction (Risiglione et al., 2022, Du et al., 30 Oct 2024).

3. Contact Modeling, Constraints, and Compliance

Manipulation-centric whole-body control requires explicit modeling and enforcement of contact constraints:

• Contact Wrenches and Friction Cones: Full-order dynamic models incorporate ground reaction forces from feet, wheels, prongs, and manipulator end-effectors; friction cones, unilaterality, and slip-free kinematics are either hard-constrained (via QP/MPC) or softly penalized in optimization (Wolfslag et al., 2020, Chen et al., 17 Sep 2025).
• Prong-Augmented Robustness: Attaching prongs to the robot's body augments manipulation robustness by increasing rejectable wrench polytopes (measured by SUF metric), enabling new loco-manipulation tasks such as obstacle clearance and box lifting with lower joint-torque requirements and higher disturbance rejection (Wolfslag et al., 2020).
• Predictive Admittance Control: Real-time, compliant control is achieved by embedding simplified admittance laws as dynamical equality constraints, coupling external force deviations with motion responses parameterized analytically over Bézier basis (Du et al., 30 Oct 2024).
• Soft vs. Hard Constraints: Model-based optimization frameworks often relax hard constraints (e.g., collision margins, joint torques) via barrier or hinge penalties, enhancing feasibility under unmodeled disturbances (Mittal et al., 2021).

4. Policy Learning, Sim2Real Transfer, and Generative Control

Several frameworks leverage reinforcement learning, policy distillation, and generative methods to implement manipulation-centric control:

• Unified RL Policies: Learning joint policies over all DoFs yields seamless synergies between locomotion and manipulation, with reward decomposition (e.g., advantage mixing) to avoid local minima in training and curriculum-based adaptation to environmental variability (Fu et al., 2022).
• Distillation and Multi-mode Generalization: Oracle policies trained on full-body human-motion imitation are distilled into robust, mask-programmable student networks. These can be programmed via binary masks to perform pure manipulation, pure locomotion, or mixed loco-manipulation tasks, without retraining (He et al., 28 Oct 2024).
• Cross-embodiment and Portability: Frameworks such as UMI-on-Legs decouple high-level visuomotor manipulation policies from embodiment-specific whole-body controllers, enabling zero-shot transfer of tabletop policies collected with handheld grippers to legged mobile platforms by tracking desired end-effector trajectories in task frames (Ha et al., 14 Jul 2024).
• Sim2Real Adaptation: Domain randomization (mass, friction, CoM offset, motor strength) and online adaptation modules close the gap between simulation and deployment, especially in RL-based schemes. Policy architectures often include onboard latent encoders tracking privileged supervisory signals (Fu et al., 2022, Ze et al., 5 May 2025).
• Teleoperation and Goal Injection: Some systems (TWIST, MaskedManipulator) support direct whole-body teleoperation by mapping human MoCap or high-level goal tokens into reference targets, reconstructed by end-to-end learned policies with motion retargeting and low-latency tracking (Ze et al., 5 May 2025, Tessler et al., 25 May 2025).

5. Evaluation, Performance Metrics, and Comparative Analyses

Manipulation-centric controllers are benchmarked across simulation and hardware experiments:

• Metric Domains: Controllers are evaluated by success rate, mean per-joint position error (MPJPE), survival rate under disturbance, end-effector pose/orientation accuracy, torque/energy consumption, and workspace volume (He et al., 28 Oct 2024, Pan et al., 26 Mar 2024, Fu et al., 2022).
• Comparisons: Unified, multi-mode, and bilevel optimization controllers consistently outperform specialist baselines across dozens of tracked metrics, including handling real-world tasks such as tabletop manipulation, dynamic obstacle avoidance, box pushing/pulling, and object-hand manipulation in unstructured environments (He et al., 28 Oct 2024, Molnar et al., 24 Nov 2025, Mittal et al., 2021).
• Robustness Under Disturbance: Enhanced controllers achieve higher disturbance rejection, lower base deflection, and improved force-tracking during contact-rich manipulation and environmental interaction tasks, validated in hardware push recovery and object manipulation scenarios (Wolfslag et al., 2020, Molnar et al., 24 Nov 2025).
• Cross-Task Generalization: MaskedManipulator diffusion and C-VAE policies generalize to teleoperation-style conditioning and long-horizon object-goal chaining, demonstrating superior success rates and emergent behaviors in sequential pick-and-place, bimanual lifting, and real-time goal change tasks (Tessler et al., 25 May 2025).

6. Open Challenges and Future Directions

Despite substantial advances, notable challenges remain:

• Scalability of Real-Time Optimization: Computational demands for high-DoF, contact-rich whole-body MPC remain a concern; Bézier parameterization and feasibility-driven DDP reduce variable count but require further optimization for massive parallelism (Du et al., 30 Oct 2024, Chen et al., 17 Sep 2025).
• Contact Estimation and Model Fidelity: Performance degrades under uncertainty in friction, contact timing, and force estimation; future work is directed at online parameter estimation and learning-based residual models for robustness (Chen et al., 17 Sep 2025, Fu et al., 2022).
• Collision Avoidance and Safety: Soft constraint relaxation can produce transient violations; incorporating convex approximations for self-collision and environment collision remains an active area (Chen et al., 17 Sep 2025, Mittal et al., 2021).
• Hardware Validation: Several paradigms remain at the level of simulation; experimental evaluation and validation across diverse platforms are needed to fully establish generality (Risiglione et al., 2022).
• Integration with Perception: Fully manipulation-centric control must combine real-time perception feedback, environment state estimation, and high-dimensional control in unknown, dynamic environments (Mittal et al., 2021).

In sum, manipulation-centric whole-body controllers redefine the paradigm for robot motor intelligence, providing unified, adaptive, and robust control of both locomotion and manipulation through advanced modeling, policy learning, and efficient optimization. They achieve state-of-the-art performance across demanding tasks and settings, while exposing new research directions in real-time optimization, compliance, generalization, and embodied intelligence.