PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes

Abstract: Humanoid robots hold great potential for diverse interactions and daily service tasks within human-centered environments, necessitating controllers that seamlessly integrate precise locomotion with dexterous manipulation. However, most existing whole-body controllers lack exteroceptive awareness of the surrounding environment, rendering them insufficient for stable task execution in complex, unstructured scenarios.To address this challenge, we propose PILOT, a unified single-stage reinforcement learning (RL) framework tailored for perceptive loco-manipulation, which synergizes perceptive locomotion and expansive whole-body control within a single policy. To enhance terrain awareness and ensure precise foot placement, we design a cross-modal context encoder that fuses prediction-based proprioceptive features with attention-based perceptive representations. Furthermore, we introduce a Mixture-of-Experts (MoE) policy architecture to coordinate diverse motor skills, facilitating better specialization across distinct motion patterns. Extensive experiments in both simulation and on the physical Unitree G1 humanoid robot validate the efficacy of our framework. PILOT demonstrates superior stability, command tracking precision, and terrain traversability compared to existing baselines. These results highlight its potential to serve as a robust, foundational low-level controller for loco-manipulation in unstructured scenes.

• The paper introduces a unified RL framework that integrates attention-driven terrain perception with a MoE-based motor control, reducing tracking errors and stumbles.
• The cross-modal context encoder fuses proprioceptive history and LiDAR maps to enhance foothold placement and adaptability across diverse terrains.
• Experimental results validate PILOT's robust performance in both simulation and real deployments, enabling stable locomotion and dexterous manipulation.

Unified Perceptive Loco-Manipulation with PILOT: An Expert Review

Introduction and Motivation

The paper "PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes" (2601.17440) presents a single-stage RL framework for humanoid robots, addressing the persistent challenge of integrating perceptive locomotion and dexterous whole-body manipulation in unstructured environments. While previous controllers exhibited shortcomings in terrain adaptation and multi-task coordination—particularly under highly variable or non-planar conditions—the proposed PILOT framework unifies perceptive sensing with a high-dimensional whole-body control policy, supporting robust performance across challenging, real-world tasks.

Architectural Overview

PILOT achieves integrated control through two principal innovations: a cross-modal context encoder and a MoE-based unified actor network. The cross-modal encoder fuses prediction-based proprioceptive features with attention-derived perceptive representations, constructed from egocentric LiDAR elevation maps. This combination yields enhanced terrain awareness, facilitating temporally consistent foothold placement and minimizing adverse interactions with irregular surfaces. The MoE actor network enables coordinated motor skill deployment, allowing seamless transitions between locomotion and manipulation modes, even as sensory dimensionality and control objectives change.

Figure 1: The PILOT framework fuses multi-scale terrain perception with proprioceptive context for skill selection and torque generation using a MoE policy network.

PILOT’s state representation combines proprioceptive history (joint positions, velocities, base velocities, previous actions) and a high-resolution local terrain map, enabling real-time adaptation to environment changes. Goal specification is achieved through a procedurally generated command vector, covering linear and angular velocities, base height, torso orientation, and upper-body joint targets, ensuring full workspace coverage.

Key Components

Cross-modal Context Encoding

A prediction-based history encoder implicitly models robot dynamics, yielding rich proprioceptive state representation. Attention-based terrain encoding, leveraging a PointNet-inspired architecture, extracts global and local geometric features; cross-attention mechanisms then synthesize state-conditioned foothold assessments. Empirical evaluations demonstrate critical improvements in adaptability and stumble minimization when attention-based encoding is used.

Unified Whole-Body Control via Mixture-of-Experts

PILOT deploys a MoE policy, consisting of a gating network and four expert subnetworks accepting fused sensory inputs. The gating network modulates expert activations in response to dynamic task demands, illustrated by task-dependent activation intensities across different motion regimes.

Figure 2: Expert activation heatmap shows dynamic gating in six distinct motion modes.

Upper-body outputs are delivered as residual corrections to target joint configurations, enabling fine-tuned manipulation without negating underlying references. Reward structure combines command tracking fidelity with explicit regularization terms penalizing physical infeasibilities.

Curriculum and Training

A progressive curriculum strategy facilitates robust skill acquisition. The command space is incrementally expanded from local, conservative goals to full workspace exploration. Sampling heuristics for base height and arm configurations bias early training toward stable behaviors and gradually increase operational range, mitigating distribution shift and catastrophic forgetting.

Experimental Evaluation

Simulation Benchmarks

Quantitative simulation results indicate consistently lower tracking errors across velocity, height, torso orientation, and arm positions compared to state-of-the-art baselines (HOMIE, FALCON, AMO), even on less challenging terrains. The ablation studies, conducted on complex terrains, demonstrate the essentiality of both the attention-based encoder and the MoE policy for safe traversal—removal of these components dramatically increases stumble frequency and impairs precision.

Real-world Validation

Direct zero-shot deployment on the Unitree G1 humanoid illustrates transferable robustness. In teleoperated object transport tasks, the policy maintains stable locomotion and precise manipulation while ascending/descending stairs or traversing high steps with payloads. The system handles coupled dynamic disturbances (asymmetric mass, shifting CoM) without sacrificing gait stability. Across five trials with varying payloads, a 100% success rate is reported, with no stumble events.

Figure 3: Real-world stair and platform traversal during payload transport using PILOT on the Unitree G1 robot.

Hierarchical Autonomous Tasks

Autonomous box-lifting tasks are executed through RL-trained high-level policies issuing structured commands to the low-level PILOT controller. The robot reliably executes navigation, grasping, and recovery sequences without teleoperation, confirming PILOT as a stable abstraction for hierarchical policy stacking.

Figure 4: Autonomous box approach, lift, and recovery powered by PILOT as the low-level tracking controller.

Implications and Future Directions

The proven robustness and generalizability of PILOT advance the field of humanoid control, particularly for deployment in unstructured, multi-modal environments where blind policies fail. The architectural modularity supports seamless integration into hierarchical planners, SLAM-based navigation, and vision-guided task execution. The reliance on native exteroception and attention-based fusion mitigates the fragility of purely kinematics-driven or motion-capture-based approaches.

Practical deployments may leverage PILOT for service robotics, search-and-rescue, or assistive technology in dynamic urban environments. Theoretical exploration into improved expert gating, meta-learning for skill synthesis, and self-supervised perceptual learning could further enhance adaptability and generalization. Future work will likely focus on end-to-end visuomotor policy stacks and extending autonomy for manipulation-intensive scenarios, removing dependencies on external sensory tracking or manual command specification.

Conclusion

PILOT provides a technically rigorous, unified RL framework for perceptive loco-manipulation in humanoid robotics, outperforming prior controllers in tracking accuracy, stability, and terrain adaptability. The integration of multi-scale attention-driven perception and MoE-based whole-body policy learning establishes PILOT as a foundational low-level controller for advanced autonomous behaviors in complex, unstructured environments.