RL Framework for Quadrupedal Wall Climbing

• The paper introduces an RL control architecture integrating actor, critic, and state estimation with a physics-based magnetic adhesion model for robust wall climbing.
• A three-phase curriculum—covering crawling, gravity rotation, and adhesion uncertainty—ensures adaptive gait generation and resilience to partial contacts and stochastic failures.
• Empirical results show near 90% success in simulation and hardware transfer, outperforming MPC baselines and paving the way for advanced sim-to-real applications.

Reinforcement learning frameworks for quadrupedal wall-climbing address the joint challenges of robust whole-body control, vertical surface adherence, adaptive gait generation, and resilience to actuation or contact failures. Recent studies have formalized wall-climbing as a sequential decision problem, introducing integrated solutions that combine physics-based contact modeling, curriculum learning, and advanced policy architectures. The objective is a locomotion controller that reliably coordinates magnetic adhesion, leg movement, and body posture—even under uncertainties such as stochastic attachment failure or incomplete wall contact.

1. RL Control Architecture and Observation Design

A core component is the control policy, trained using Proximal Policy Optimization (PPO) within a physically realistic simulation environment (e.g., RaiSim). The controller typically comprises three neural networks:

• Policy (Actor): Outputs leg joint torques and binary magnetic adhesion commands per foot.
• Critic: Estimates future returns over extended horizons.
• State Estimator: Reconstructs privileged signals including base velocity, foot heights, and per-foot contact probabilities.

The observation space includes joint positions/velocities, historical joint targets, body orientation, angular velocities, relative foot positions, and a clock input (leg phase encoding by sine/cosine features):

$$\text{o}_{\text{clock}} = [\sin(\phi_i), \cos(\phi_i)] \quad \forall i \in \{1,\ldots,4\}$$

where

$$\phi_i = \left( \frac{2\pi}{T} t + \frac{\pi}{2} i \right),\ T = \text{gait cycle period}$$

Auxiliary imitation signals penalize magnet activation during leg swing and reward temporally correct turning ON of the magnet during stance.

The physics-based foot adhesion model is directly embedded in the simulation so the RL policy must adapt to partial contacts, air-gap sensitivity, and probabilistic failures.

2. Physics-Based Magnetic Adhesion Modeling

Magnetic adhesion is modeled with four sequential constraints:

1. Contact Recognition: The state estimator provides a contact confidence signal; contact is registered when $\tilde{c}_{\text{foot}} \geq 0.5$.
2. Magnet Activation: Magnetic force is applied only if $a_{\text{magnet}} \geq 0.5$.
3. Stochastic Attachment: A random variable $X \sim U(0,1)$ is compared against a scheduled probability $\text{Prob}_{\text{attach}}$; adhesion succeeds only if $X \leq \text{Prob}_{\text{attach}}$.
4. Geometric Alignment: The EPM surface $S_{\text{EPM}}$ must match the wall surface $S_{\text{wall}}$; any misalignment or air gap ($>1~\mathrm{mm}$) severely reduces holding force.

Adhesion retention, detachment, and recovery metrics are measured in simulation and hardware, confirming the impact of probabilistic failures and geometric constraints on vertical locomotion.

3. Three-Phase Curriculum Training Strategy

A phased curriculum is used to stabilize policy learning while gradually exposing the robot to increasing difficulty:

Phase	Environment	Adhesion Model	Gravity Vector	Prob_attach Schedule
1. Crawl Gait	Flat ground	Disabled	Horizontal	-
2. Gravity Rotation	Rotating surface	Enabled	$0 \rightarrow \pi/2$	1.0
3. Adhesion Uncertainty	Vertical wall	Enabled	$\pi/2$	$1.0 \rightarrow 0.85$

Phase 1: Learn stable crawling without adhesion forces. Auxiliary reward guides magnet actuation timing.

Phase 2: Activate the adhesion model and rotate the gravity vector towards vertical:

$$\theta(t) = \min\left\{ \frac{\pi}{2}, \max\left\{ 0, \frac{\pi}{2} \cdot \frac{t-1200}{20000} \right\} \right\}$$

Phase 3: Inject stochastic adhesion failures by linearly decreasing $\text{Prob}_{\text{attach}}$:

$$\text{Prob}_{\text{attach}}(t) = 1.0 - 0.15 \cdot \frac{\min(\max(t-21200, 0), 13800)}{13800}$$

Simultaneously, RL reward terms and auxiliary penalties teach slip recovery and stable vertical crawling.

4. Robustness to Partial Attachment and Comparison to Baseline Methods

Simulation studies quantify adhesion retention, early ablation termination, recovery rates, and velocity tracking errors. Even with $\text{Prob}_{\text{attach}}$ reduced to 0.85, overall success rates remain near 90%, demonstrating rapid recovery from detachment events.

A Model Predictive Control (MPC) baseline, assuming perfect adhesion, fails immediately when contact is lost. In contrast, the RL controller shifts the body, redistributes loads, and reattempts adhesion. Ablations confirm the necessity of both the physics-based adhesion model and stochastic attachment for robust locomotion.

5. Hardware Validation and Real-World Transfer

The RL-trained controller is deployed on an untethered magnetic climbing robot equipped with electropermanent magnets (EPMs) capable of up to 697 N holding force. Hardware trials on vertical steel surfaces demonstrate:

• Stable vertical crawling motion under partial contact, repeated detachment, and real-time surface misalignment.
• Magnet ON/OFF commands are synchronized to contact confidence inputs from proprioceptive sensors.
• Actual recovery strategies (e.g., body shift and reattachment) are consistent with simulation results.

Sim-to-real transfer is achieved through extensive domain randomization of joint gains, friction coefficients, sensor noise, and control lags during training. The staged curriculum and adhesion uncertainty generalize robustly to hardware, overcoming reality gaps due to imperfect actuation or unmodeled surface properties.

6. Significance and Underlying Principles

The RL framework for wall climbing incorporates physics-grounded contact models, explicit handling of adhesion uncertainty, curriculum-based transfer from easy conditions to difficult vertical climbs, and high-dimensional observation spaces including temporal gait encoding. The actor-critic architecture and auxiliary timing signals ensure correct synchrony of leg bracing and magnet actuation. Empirical comparisons to MPC show clear superiority for robust recovery and sustainable vertical locomotion.

Key mathematical constraints of the adhesion model (contact recognition, magnet activation threshold, stochastic failure scheduling, and geometric alignment) are central for resilience to degraded adhesion. The curriculum progression and domain randomization collectively enable effective sim-to-real transitions in complex ferromagnetic environments.

A plausible implication is that the design principles observed here—staged curriculum, explicit generative contact modeling, and RL-based control—can be extended to non-magnetic adhesion modalities (e.g., vacuum, microspines) and other forms of vertical climbing, provided the physical phenomena are similarly represented in the learning framework.

7. Future Directions and Open Challenges

Current frameworks exhibit robustness to stochastic failures and incomplete contact in magnetic environments, but further research is needed to address the scalability to non-planar surfaces, contact-rich transitions (e.g., moving from wall to ceiling), and real-time adaptation to unmodeled surface properties. Integration of higher-fidelity tactile sensing, online estimation of adhesion state, and multi-modal adhesion control (combining magnetic and frictional effects) remain important open problems.

Advances in sim-to-real transfer—driven by domain randomization, deep state estimation, and staged curriculum learning—point towards RL-based controllers for quadrupedal robots capable of robust climbing in more diverse industrial and unknown environments.