Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn

Abstract: Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature of RL training. One source of the challenges in RL is that output predictions can churn, leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, how churn occurs and impacts RL remains under-explored. In this work, we start by characterizing churn in a view of Generalized Policy Iteration with function approximation, and we discover a chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias of policy value in actor-critic methods. We then propose a method to reduce the chain effect across different settings, called Churn Approximated ReductIoN (CHAIN), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings, as well as a scaling setting.

• The paper presents the CHAIN method, which reduces value and policy churn to improve DRL stability and efficiency.
• It formalizes churn using NTK formulations, revealing how chain effects in value and policy updates bias the learning process.
• Experimental results across various environments demonstrate enhanced sample efficiency and stable learning through reduced network output changes.

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn

The paper "Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn" by Hongyao Tang and Glen Berseth addresses the understudied phenomenon of value and policy churn in deep reinforcement learning (DRL). Churn is defined as the unintended changes in the network outputs for states and actions that are not included in the batch after each network update. This paper proposes a formal characterization of churn and introduces a method for mitigating its adverse effects, named Churn Approximated ReductIoN (CHAIN).

Characterization of Churn

The authors characterize churn through the lens of Generalized Policy Iteration (GPI) with function approximation. In this framework, value churn and policy churn are interwoven during the evaluation and policy improvement steps of GPI. Value churn affects the accuracy of the value function, while policy churn influences the policy's action selection stability. The paper identifies a "chain effect" where churn in value estimation and policy improvement compounds over iterations, leading to biased learning dynamics.

Specifically, the paper defines value churn and policy churn as follows:

• Value Churn ($\mathcal{C}_{Q}(\theta,\theta', B_\text{ref})$): The unintended changes in the Q-value network's output after parameter updates.
• Policy Churn ($\mathcal{C}_{\pi}(\phi,\phi', B_\text{ref})$): The unintended changes in the policy network's output after parameter updates.

These churns are quantified using NTK (Neural Tangent Kernel) formulations to express how parameter updates influence value and policy outputs.

Impact and Implications

The chain effect identified describes a cycle where churn in value and policy functions biases subsequent parameter updates. This cyclical amplification can destabilize the learning process, reduce sample efficiency, and hinder the convergence of DRL algorithms.

The paper further details several learning issues caused by the chain effect:

• Greedy Action Deviation: In value-based methods like DQN, the action distribution can be significantly altered due to churn, causing instability.
• Trust Region Violation: In policy gradient methods like PPO, churn can violate the trust region assumption, leading to erratic policy updates.
• Dual Bias: In actor-critic methods, churn in both value and policy functions introduces compounded biases, stearing the training unfavorably.

Each of these issues underscores the need for mechanisms to manage and reduce churn in DRL.

Proposed Method: CHAIN

To mitigate churn, the authors introduce the CHAIN method, which employs regularization terms to reduce unintended changes in network outputs for states and actions outside the current training batch. The CHAIN regularization terms are defined as:

• $$L_{\text{QC}}(\theta_t, B_\text{ref}) = \frac{1}{|B_\text{ref}|} \sum_{(\bar{s}, \bar{a}) \in B_\text{ref}} ( Q_{\theta_t}(\bar{s}, \bar{a}) - Q_{\theta_{t-1}}(\bar{s}, \bar{a}) )^2$$
• $$L_{\text{PC}}(\phi_t, B_\text{ref}) = \frac{1}{|B_\text{ref}|} \sum_{\bar{s} \in B_\text{ref}} d_{\pi}(\pi_{\phi_t}(\bar{s}), \pi_{\phi_{t-1}}(\bar{s}))$$

The regularization is applied directly in the loss functions of value and policy networks, making CHAIN easy to integrate into existing DRL algorithms.

Experimental Results

The efficacy of the CHAIN method was validated on a variety of environments, including MinAtar, OpenAI MuJoCo, DeepMind Control Suite, and D4RL. Key observations from the experiments include:

• Significant reduction in both value and policy churn across different DRL settings.
• Improved sample efficiency and final performance in both online and offline RL scenarios.
• Enhanced scalability of DRL agents, achieving superior learning performance with wider or deeper networks.

Future Directions

The paper speculates on several future directions to further the understanding and control of churn in DRL:

• A more rigorous theoretical analysis of the chain effect under specific assumptions.
• Exploring the relationship between network architecture, experience replay strategies, and churn.
• Development of adaptive methods to automatically tune regularization parameters based on the learning dynamics.

By addressing the chain effect of value and policy churn, this paper contributes to the robustness and efficiency of DRL algorithms. The proposed CHAIN method provides a practical solution that can be readily adopted in various DRL frameworks, leading to more stable and effective learning processes. This study opens a pathway to deeper insights into the dynamics of neural network training within the RL domain, with potential implications for improved AI systems across a spectrum of applications.