Anchored Learning for On-the-Fly Adaptation -- Extended Technical Report (2301.06987v2)

Abstract: This study presents "anchor critics", a novel strategy for enhancing the robustness of reinforcement learning (RL) agents in crossing the sim-to-real gap. While RL agents can be successfully trained in simulation, they often encounter difficulties such as unpredictability, inefficient power consumption, and operational failures when deployed in real-world scenarios. We identify that naive fine-tuning approaches lead to catastrophic forgetting, where policies maintain high rewards on frequently encountered states but lose performance on rarer, yet critical scenarios. Our method maximizes multiple Q-values across domains, ensuring high performance in both simulation and reality. Evaluations demonstrate that our approach enables behavior retention in sim-to-sim gymnasium tasks and in sim-to-real scenarios with racing quadrotors, achieving a near-50% reduction in power consumption while maintaining controllable, stable flight. We also contribute SwannFlight, an open-source firmware for testing adaptation techniques on real robots.