Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning

Abstract: Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called Adaptive $Q$-Network (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several $Q$-functions, each one trained with different hyperparameters, which are updated online using the $Q$-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari $2600$ games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability.

• The paper introduces Adaptive Q-Network (AdaQN), a novel deep reinforcement learning method that dynamically selects hyperparameters during training to address non-stationarity without additional environment interaction.
• AdaQN utilizes multiple Q-functions updated online, selecting the one with the smallest approximation error as a shared target by comparing TD-errors to enhance performance.
• Empirical validation on MuJoCo tasks demonstrates AdaQN's superior sample-efficiency, robustness, and performance compared to static configurations, showing its effectiveness in creating adaptive hyperparameter policies.

Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning

The paper introduces Adaptive $Q$-Network (AdaQN), a novel approach to Automated Reinforcement Learning (AutoRL) that addresses the challenges of non-stationarity in deep reinforcement learning (RL) by dynamically selecting hyperparameters on-the-fly during the training process. This approach mitigates the sensitivity of deep RL algorithms to hyperparameter settings without necessitating additional environment interactions.

Core Contributions

AdaQN distinguishes itself from traditional AutoRL methods by centering on the dynamic adaptation of hyperparameters in response to the ever-shifting optimization landscape within RL. The paper details a strategy where multiple $Q$-functions, each trained with distinct hyperparameters, are employed. These $Q$-functions are updated online, leveraging the $Q$-function with the smallest approximation error as a shared target. This novel selection mechanism is orthogonal to various critic-based RL algorithms and involves comparing the TD-errors of different $Q$-networks to select the optimal one at each target update stage. This method theoretically ensures a minimized sum of approximation errors over the training process, leading to better performance than static hyperparameter configurations.

Theoretical Underpinnings

The paper provides a rigorous theoretical foundation for AdaQN, rooted in minimizing the sum of approximation errors across Bellman iterations as suggested by reinforcement learning theory, which is crucial for enhancing performance guarantees. By evaluating the projection error rather than the more traditional Bellman error, the authors argue convincingly that their method can dynamically select better-performing hyperparameter schedules during the training process, thus avoiding pitfalls like local minima or excessive divergence.

Empirical Validation

Empirically, the authors validate AdaQN on various control tasks within the MuJoCo simulator. The results exhibit AdaQN's superior sample-efficiency, robustness, and overall performance against both individual static hyperparameter setups and exhaustive grid search methods. Notably, AdaQN consistently matches or surpasses the best performance attained by individual hyperparameter configurations provided as input, demonstrating its effectiveness in crafting adaptive hyperparameter policies that respond aptly to the dynamic RL environment.

Implications and Future Work

AdaQN opens several avenues for future exploration in adaptive systems within machine learning. The research marks a move towards more autonomous and intelligent RL systems, which can self-regulate their learning processes according to task-specific requirements without manual tuning. This innovation has implications for deploying RL in real-world applications where the adaptability of learning algorithms can substantially enhance reliability and efficiency. Future developments might involve integrating broader hyperparameter classes, including network architectures and environment-specific parameters, thus broadening scope and applicability.

Conclusion

The paper contributes a methodologically sound and empirically validated approach to AutoRL through AdaQN, providing an insightful mechanism for handling RL's inherent non-stationarities. It aligns with evolving trends in machine learning towards adaptive and automated systems that can effectively reduce the need for extensive manual intervention. On balance, the demonstration of effectiveness in handling multiple MuJoCo control problems signifies a promising step toward more autonomous and sample-efficient RL methodologies.