Learning Agile Swimming: An End-to-End Approach without CPGs (2409.10019v2)

Abstract: The pursuit of agile and efficient underwater robots, especially bio-mimetic robotic fish, has been impeded by challenges in creating motion controllers that are able to fully exploit their hydrodynamic capabilities. This paper addresses these challenges by introducing a novel, model-free, end-to-end control framework that leverages Deep Reinforcement Learning (DRL) to enable agile and energy-efficient swimming of robotic fish. Unlike existing methods that rely on predefined trigonometric swimming patterns like Central Pattern Generators (CPG), our approach directly outputs low-level actuator commands without strong constraints, enabling the robotic fish to learn agile swimming behaviors. In addition, by integrating a high-performance Computational Fluid Dynamics (CFD) simulator with innovative sim-to-real strategies, such as normalized density calibration and servo response calibration, the proposed framework significantly mitigates the sim-to-real gap, facilitating direct transfer of control policies to real-world environments without fine-tuning. Comparative experiments demonstrate that our method achieves faster swimming speeds, smaller turn-around radii, and reduced energy consumption compared to the state-of-the-art swimming controllers. Furthermore, the proposed framework shows promise in addressing complex tasks, paving the way for more effective deployment of robotic fish in real aquatic environments.

• The paper demonstrates that model-free DRL using the SAC algorithm can directly output low-level actuator commands for agile, energy-efficient swimming.
• It integrates a high-performance CFD simulator with innovative sim-to-real strategies to rapidly train policies that transfer effectively to real-world underwater scenarios.
• Experimental results show superior performance compared to CPG-based controllers, achieving faster speeds, lower energy consumption, and precise maneuvering.

Learning Agile Swimming: An End-to-End Approach without CPGs

Overview

The research presented in “Learning Agile Swimming: An End-to-End Approach without CPGs” by Xiaozhu Lin, Xiaopei Liu, and Yang Wang introduces an innovative framework for the control of bio-mimetic robotic fish. This framework distinguishes itself by employing a model-free end-to-end learning methodology using Deep Reinforcement Learning (DRL) to enable agile and energy-efficient swimming without relying on predefined patterns such as Central Pattern Generators (CPGs). The proposed approach directly outputs low-level actuator commands, facilitating the robotic fish to learn swimming behaviors autonomously. Integrating a high-performance Computational Fluid Dynamics (CFD) simulator with novel sim-to-real strategies, the framework shows potential for significant advancements in underwater robotics.

Methodology

The research tackled the complex control challenges inherent to robotic fish by leveraging high-efficiency computational tools and sophisticated reinforcement learning algorithms. The core methodology can be summarized in several key components:

1. Model-Free DRL: The paper employs DRL, specifically the Soft Actor-Critic (SAC) algorithm, to train policies directly controlling the low-level actuator commands. This is a departure from traditional methods that utilize trigonometric functions to predefine swimming patterns.
2. CFD Simulator: FishGym, a high-performance CFD simulator, is utilized to create a virtual environment for training. Its GPU-based Lattice Boltzmann Method (LBM) model facilitates accelerated simulations, enabling extensive exploration of the robotic fish dynamics.
3. Sim-to-Real Transfer: To bridge the gap between simulation and real-world application, the authors introduce techniques such as normalized density matching and actuator response matching. These techniques aim to align the simulated dynamics closely with the physical characteristics of the robotic fish.
4. Robust State and Action Space: The state space captures linear and angular velocities along with joint angles and velocities, whereas the action space comprises desired joint angles. This comprehensive design ensures the Markov property and allows for adaptive policy adjustments.
5. Reward Function: A well-defined reward function balances approach rewards, energy penalties, and terminal rewards, guiding the robotic fish to agile and efficient swimming behaviors.

Experimental Validation

The framework underwent rigorous testing through both simulation and real-world experiments, validating its effectiveness:

• Training and Performance: The DRL agent trained in FishGym showed rapid improvement, achieving agile swimming capabilities within a reasonable time frame. The expected return stabilized after approximately 1,000 episodes, indicating a successful learning process.
• Real-world Transfer: The zero-shot transfer of trained policies to real-world scenarios was effective. Tasks such as sharp U-turns and multi-waypoint tracking demonstrated high consistency between simulated and physical performances.
• Comparison with CPG-based Baselines: Experimental results highlighted the superiority of the proposed framework over traditional CPG-PID controllers, showcasing faster speeds, reduced energy consumption, and smaller turning radii.

Implications and Future Directions

The implications of this research are significant for both practical applications and theoretical advancements in robotics:

• Practical Applications: The ability to train agile and efficient control policies without predefined motion patterns broadens the applicability of robotic fish in environmental monitoring, underwater exploration, and interaction with aquatic life.
• Theoretical Advancements: The successful sim-to-real transfer without fine-tuning demonstrates the potential of innovative calibration techniques and robust reinforcement learning frameworks.
• Future Developments: While the current methodology shows promising results, future research could focus on enhancing training efficiency and task-specific learning. By avoiding redundant learning phases, the development of control strategies can be further accelerated.

The research represents a notable step forward in robotic fish control, emphasizing the potential for model-free, end-to-end learning methodologies in complex fluid-coupled environments. This framework, with its advanced simulation tools and innovative calibration techniques, sets a new precedent for the deployment of autonomous underwater vehicles in real-world applications.