LadderMan: Robust Robotic Ladder Climbing

• LadderMan is a system of advanced robotics and algorithms that robustly combines hybrid motion tracking, RL distillation, and perception for ladder climbing and manipulation.
• It employs a two-stage learning pipeline and sim-to-real transfer strategies to achieve over 95% simulation and 90% real-world success rates.
• The dual-agent formulation and integration of vision foundation models enable stable on-ladder operations and precise telemanipulation under challenging conditions.

LadderMan refers to a collection of advanced robotics and algorithmic systems explicitly targeting the challenge of robust, perception-driven ladder climbing and ladder-based manipulation for humanoid robots. Climbing ladders presents nontrivial difficulties in both perception and control, due to the sparsity and discontinuity of footholds/handholds, whole-body kinematic complexity, and the high risk of instability from perception noise or contact loss. The “LadderMan” system provides a scalable, unified approach to enable humanoid robots to climb a wide range of ladder geometries—including variations in inclination and rung spacing—while supporting on-ladder manipulation under severe physical constraints. This article provides an in-depth technical exposition of the LadderMan architecture, learning pipeline, sim-to-real transfer solutions, experimental validation, and prospects for future development (Zhao et al., 4 Jun 2026).

1. System Modules and Architectural Overview

LadderMan is organized into four principal modules:

• Perception: A front-end producing egocentric depth images $D_t \in \mathbb{R}^{H\times W}$, based on stereo camera and vision foundation models (VFMs) for geometric awareness.
• Motion Tracking Experts (Stage I): State-based climbing expert policies $\pi^{\rm expert}_{\phi,z}$ covering a grid of ladder geometries (inclination $\phi$, spacing $z$). Hybrid motion tracking is employed using a single reference motion and extracting stable lower-body patterns while adapting upper-body contacts.
• Policy Distillation and Unified Visuomotor Policy (Stage II): Expert policies are distilled into a unified depth-based visuomotor policy $\pi^{\rm visual}(a_t|o_t)$ via hybrid imitation-reinforcement learning.
• On-Ladder Manipulation (Dual-Agent): A dual-agent setup decomposes policy into lower-body stabilization ($\pi^l$) and upper-body manipulation ($\pi^u$), enabling stable teleoperation during manipulation tasks under ladder constraints.

Notably, the system leverages vision foundation models to bridge the sim-to-real gap in depth perception and includes robust architectural elements for high-frequency policy inference on real-world platforms.

2. Learning Pipeline: Hybrid Motion Tracking and Policy Distillation

2.1 Stage I: Hybrid Motion Tracking

LadderMan trains a collection of expert climbing policies for varied ladder geometries using reinforcement learning objectives mediated by hybrid reference tracking. The key RL objective for each expert is:

$$\max_\theta \mathbb{E}\left[ \sum_{t=0}^T \gamma^t r_t \right]$$

where the reward $r_t$ is a composite of:

• Tracking reward ($r^{\rm tracking}_t$): Penalizes deviation from reference lower-body trajectories, allowing upper-body contact adaptation.
• Contact reward ($\pi^{\rm expert}_{\phi,z}$0): Encourages stable contacts at designated times.
• Task reward ($\pi^{\rm expert}_{\phi,z}$1): Penalizes deviation from target progress along the ladder.
• Regularization: Penalizes control effort and instability.

A curriculum of domain randomization on ladder geometry, friction, and dynamics is applied, with training performed in high-parallelism simulated environments.

2.2 Stage II: Policy Distillation—Hybrid Imitation + RL

The multiple expert policies are merged via a hybrid objective combining PPO-based RL with a KL-regularized imitation term:

$\pi^{\rm expert}_{\phi,z}$2

where $\pi^{\rm expert}_{\phi,z}$3 is the standard PPO surrogate loss, and $\pi^{\rm expert}_{\phi,z}$4 is a KL-divergence between the unified policy and grid-expert policies:

$\pi^{\rm expert}_{\phi,z}$5

The annealing schedule for $\pi^{\rm expert}_{\phi,z}$6 transitions from strong imitation at initialization to pure RL for robustness and generalization across the ladder space.

3. Depth-Based Visuomotor Policy and Sim-to-Real Transfer

3.1 Policy Architecture

The unified climbing policy receives as input:

• A proprioceptive vector $\pi^{\rm expert}_{\phi,z}$7.
• A processed depth map $\pi^{\rm expert}_{\phi,z}$8 via three-layer convolutional encoder.

Actions $\pi^{\rm expert}_{\phi,z}$9 represent PD target joint angles, output by a deep MLP. Value prediction is accomplished with an additional critic MLP.

PPO with dual-clip surrogate loss is employed for on-policy updates. Domain randomization over depth noise, camera extrinsics, dropout, and rung-focused masking is central to sim-to-real transfer.

3.2 Vision Foundation Model (VFM) Integration

For real-world deployment, raw stereo depth is replaced by Fast-FoundationStereo—delivering high-fidelity, artifact-free depth without extra fine-tuning loss. Calibration includes disabling the IR emitter and auto-exposure, and adjusting for stereo consistency. High-annulus collision meshes are adopted for physical fidelity, especially for extremities (shins, hands, feet).

4. On-Ladder Manipulation: Dual-Agent Formulation

A two-agent decomposition supports manipulation while maintaining balance:

• Lower-body agent ($\phi$0): Stabilizes contacts and controls the pelvis, subject to climbing reward and stability criteria.
• Upper-body agent ($\phi$1): Tracks user-specified teleoperation targets for the arms/hands/waist, using Ada-style upper-body tracking reward and subject to manipulation objectives.

The combined action vector is $\phi$2. Training uses separate PPO updates for each subpolicy, and simulated teleoperation targets are drawn from motion capture datasets. Instability penalty terms for large COM excursions and slip are enforced in the reward.

5. Experimental Evaluation

5.1 Simulation Benchmarks

Success rates for ladder climbing across a matrix of $\phi$3 show that LadderMan achieves greater than 95% success for central regions (24–28 cm rung spacing, 55°–65° inclination). Blind motion-tracking baselines fail catastrophically outside narrow reference conditions. Hybrid motion tracking significantly outperforms DeepMimic for off-reference cases.

5.2 Real-World Zero-Shot Transfer

Deployment on the Unitree G1 humanoid with stereo depth and VFM achieves robust zero-shot ladder climbing. In controlled trials (10 per geometry), LadderMan attains 6–9/10 success, whereas policies trained without RL, VFM, or rung-focused masking (RFM) fail in all or nearly all attempts. RFM is critical for avoiding overfitting to simulated appearance and enabling generalization.

5.3 On-Ladder Manipulation Tasks

The dual-agent policy supports 90% success rate in teleoperation tasks (painting, bulb replacement, box handover) compared to 30% for TWIST2 baseline. Lower-body stabilization is preserved during upper-body manipulation.

6. Implementation and System Details

• Robot: Unitree G1 (29 DoF, 1.3 m height).
• Sensors: Intel RealSense D435i (stereo), processed via Fast-FoundationStereo on NVIDIA RTX 4090, streamed to Jetson Orin.
• Compute: Policy inference at 50 Hz on Jetson Orin.
• Training: IsaacSim with 4096 parallel envs on single NVIDIA L40S.
• Policy: PPO learning rate $\phi$4, batch size 2048, 8000 epochs (Stage I), 4000 epochs (Stage II). KL imitation weight annealed from 1.0 downward.

7. Limitations and Future Directions

LadderMan does not address fully vertical (90°) ladders or highly dynamic ladder morphologies. Manipulation is limited by end-effector dexterity, with fine telemanipulation still out of scope. Planned future work includes extension to vertical/rotating ladders, tactile integration, dextrous hands, and online adaptation to dynamically mapped environments (Zhao et al., 4 Jun 2026).

---

LadderMan establishes a rigorous, scalable approach for hierarchical learning and transfer of visuomotor climbing and manipulation on ladders. By integrating hybrid tracking, RL-based distillation, robust perception, and dual-agent decomposition, robust sim-to-real performance is achieved on real humanoid hardware with generalization across challenging ladder geometries.