Visual CPG-RL: Learning Central Pattern Generators for Visually-Guided Quadruped Locomotion (2212.14400v2)

Abstract: We present a framework for learning visually-guided quadruped locomotion by integrating exteroceptive sensing and central pattern generators (CPGs), i.e. systems of coupled oscillators, into the deep reinforcement learning (DRL) framework. Through both exteroceptive and proprioceptive sensing, the agent learns to coordinate rhythmic behavior among different oscillators to track velocity commands, while at the same time override these commands to avoid collisions with the environment. We investigate several open robotics and neuroscience questions: 1) What is the role of explicit interoscillator couplings between oscillators, and can such coupling improve sim-to-real transfer for navigation robustness? 2) What are the effects of using a memory-enabled vs. a memory-free policy network with respect to robustness, energy-efficiency, and tracking performance in sim-to-real navigation tasks? 3) How do animals manage to tolerate high sensorimotor delays, yet still produce smooth and robust gaits? To answer these questions, we train our perceptive locomotion policies in simulation and perform sim-to-real transfers to the Unitree Go1 quadruped, where we observe robust navigation in a variety of scenarios. Our results show that the CPG, explicit interoscillator couplings, and memory-enabled policy representations are all beneficial for energy efficiency, robustness to noise and sensory delays of 90 ms, and tracking performance for successful sim-to-real transfer for navigation tasks. Video results can be found at https://youtu.be/wpsbSMzIwgM.

• The paper presents a method combining visual feedback and CPGs with DRL to synchronize rhythmic movement and obstacle avoidance.
• Results show that memory-enabled LSTM networks enhance energy efficiency and robustness compared to memory-free models.
• The study demonstrates that explicit interoscillator couplings improve sim-to-real transfer by managing sensor delays up to 90 ms.

An Analysis of Visual CPG-RL for Visually-Guided Quadruped Locomotion

The integration of Central Pattern Generators (CPGs) with deep reinforcement learning (DRL) provides novel insights into controlling visually-guided quadruped locomotion, as presented in the paper "Visual CPG-RL: Learning Central Pattern Generators for Visually-Guided Quadruped Locomotion." This paper explores the interaction between exteroceptive sensory input and CPGs to develop robust navigation strategies for quadruped robots, evaluated under various conditions.

Framework Overview

The authors propose a method that intersects exteroceptive and proprioceptive sensing to teach robotic quadrupeds to integrate rhythmic locomotion and obstacle avoidance. Using a reinforcement learning framework, the robot modulates CPGs—systems of coupled oscillators—to navigate while maintaining coordination. A central theme is understanding the role of explicit interoscillator couplings, testing memory-enabled vs. memory-free neural network architectures, and investigating the effect of sensory delays.

Scientific Questions and Findings

1. Interoscillator Couplings: The paper finds that coupling between CPG oscillators enhances the robustness of sim-to-real transfer. This conclusion draws from empirical data showing improved navigation capabilities in dynamic and cluttered environments. This addresses the hypothesis that such couplings can better accommodate high-dimensional sensory inputs.
2. Memory-Enabled vs. Memory-Free Policies: The paper investigates whether Long Short-Term Memory (LSTM) networks, which embody a biological parallel with memory processing, outperform multilayer perceptron (MLP) networks. Results indicate that memory-enabled networks significantly enhance energy efficiency and robustness, particularly during real-world application, emphasizing the advantage of incorporating memory to handle temporal dependencies in sensory inputs.
3. Handling of Sensory Delays: The paper observes that quadruped robots can effectively manage sensory delays of up to 90 ms with the proposed CPG architecture, aligning with findings on sensorimotor delays in biological systems. Such robustness is critical for real-world applicability where sensory data is subject to latency and noise.

Implications and Future Directions

The methodology substantiates CPGs as effective intermediary layers in the control hierarchy of legged robots, facilitating real-time adaptation to environmental variables while maintaining energy-efficient locomotion. This paper enriches the understanding of biologically-inspired robotics, suggesting that further integration of neural control models can enhance robotic adaptability.

Additionally, the findings indicate potential advancements in robotic systems operating in unstructured environments, promoting adaptive locomotion solutions which could be beneficial in areas such as search and rescue, planetary exploration, and wildlife monitoring.

Future research may focus on optimizing CPG parameters to understand their broader applications, particularly in more complex terrains and cluttered environments. Additionally, exploring the dynamic adjustment of coupling strengths and enhancing perception via advanced sensor technologies could further realize the potential of quadruped robotics in practical scenarios.

The integration of CPGs and visual feedback in DRL frameworks opens new avenues for biologically-inspired control systems, underscoring a promising direction in the robotics field where theoretical advancements are immediately applicable in intricate robotic systems.