DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization (2112.04716v1)

Abstract: Despite overparameterization, deep networks trained via supervised learning are easy to optimize and exhibit excellent generalization. One hypothesis to explain this is that overparameterized deep networks enjoy the benefits of implicit regularization induced by stochastic gradient descent, which favors parsimonious solutions that generalize well on test inputs. It is reasonable to surmise that deep reinforcement learning (RL) methods could also benefit from this effect. In this paper, we discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the offline deep RL setting, leading to poor generalization and degenerate feature representations. Our theoretical analysis shows that when existing models of implicit regularization are applied to temporal difference learning, the resulting derived regularizer favors degenerate solutions with excessive "aliasing", in stark contrast to the supervised learning case. We back up these findings empirically, showing that feature representations learned by a deep network value function trained via bootstrapping can indeed become degenerate, aliasing the representations for state-action pairs that appear on either side of the BeLLMan backup. To address this issue, we derive the form of this implicit regularizer and, inspired by this derivation, propose a simple and effective explicit regularizer, called DR3, that counteracts the undesirable effects of this implicit regularizer. When combined with existing offline RL methods, DR3 substantially improves performance and stability, alleviating unlearning in Atari 2600 games, D4RL domains and robotic manipulation from images.

• The paper reveals that implicit regularization in TD-learning causes feature co-adaptation, degrading performance in offline deep RL.
• The authors introduce DR3, an efficient explicit regularizer that minimizes similarity between state-action representations to stabilize learning.
• Empirical results across Atari, robotic, and D4RL tasks demonstrate DR3's ability to improve performance and robustness in offline RL scenarios.

Overview of "DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization"

The paper presents an analysis of implicit regularization effects in value-based deep reinforcement learning (RL) and proposes an explicit regularizer, DR3, to improve the performance and stability of offline RL algorithms. The paper addresses the challenges posed by implicit regularization when training Q-value networks via temporal difference (TD) learning, particularly in offline settings where a static dataset is used for training.

Implicit Regularization in Deep RL

While implicit regularizers induced by stochastic gradient descent (SGD) aid generalization in supervised learning, the authors identify a detrimental implicit regularization effect in offline deep RL. This effect results in feature co-adaptation, where representations of state-action pairs in BeLLMan updates become overly similar, leading to poor generalization and degraded performance. Empirical observations reveal increasing feature dot products in TD-learning, particularly with out-of-sample actions, a phenomenon not easily mitigated by existing methods like CQL or REM.

Theoretical Analysis

The paper provides a thorough theoretical characterization of implicit regularization in TD-learning. Building on existing SGD theories for overparameterized settings, the authors derive an implicit regularizer for TD fixed points. A key insight is that TD-learning favors solutions that maximize the similarity of feature representations for consecutive state-action pairs appearing in BeLLMan backups. This co-adaptation potentially destabilizes the learning process and aligns with empirical findings.

DR3: The Explicit Regularizer

Inspired by the theoretical insights, DR3 is proposed to counteract the adverse effects of implicit regularization. It minimizes the similarity between feature representations of state-action pairs, thereby promoting stability and enhancing performance. DR3 is designed to be computationally efficient, primarily focusing on the dot-product similarity of last-layer features in a neural network.

Empirical Evaluation

Extensive evaluations on offline RL benchmarks demonstrate that DR3 enhances performance and stability across various methods. Key results include:

• Substantial improvement in interquartile mean normalized scores across 17 Atari games, with DR3 reducing feature co-adaptation and performance degradation.
• Enhancements in robotic manipulation tasks using image-based inputs, where DR3 fosters quicker learning and higher average performance.
• Improvements in complex D4RL tasks, including antmaze and kitchen domains, where DR3 significantly surpasses CQL, demonstrating robustness to longer training periods.

Implications and Future Directions

The findings suggest that managing implicit regularization is crucial in extending the robustness and effectiveness of deep RL. DR3 exemplifies how theoretically-inspired explicit regularization can enhance existing RL architectures. Future work may explore adapting DR3 to online RL scenarios and investigating its scalability in larger, more complex environments.

Conclusion

The research provides valuable insights into the implicit dynamics of TD-learning in offline settings. By addressing feature co-adaptation through DR3, the authors contribute a practical solution to improve the stability and efficacy of value-based deep RL algorithms, heralding a step forward in refining RL methodologies.