Policy Churn Regularization (PCR) in RL

• PCR is a policy stabilization technique that penalizes abrupt changes in policy outputs via a KL-divergence term on reference states.
• It is applied in frameworks like GRPOformer and CHAIN to control policy drift, reduce oscillations, and improve convergence stability.
• Empirical results indicate that PCR enhances performance metrics in HPO and reinforcement learning while preserving beneficial exploratory behavior.

Searching arXiv for recent PCR-related papers and adjacent churn-regularization work. Use the arXiv search tool to look up papers on “Policy Churn Regularization”, “chain effect of value and policy churn”, “The Phenomenon of Policy Churn”, “GRPOformer PCR”, and “continual churn reduction”. Policy Churn Regularization (PCR) is a policy-stabilization regularizer that penalizes abrupt changes in a learned policy across consecutive updates. In the strict usage introduced within GRPOformer, PCR is a KL-divergence penalty between the old policy and the updated policy on reference states drawn from the earlier portion of the historical optimization trajectory, added to the Group Relative Policy Optimization objective to “enhance the stability of GRPO training” (Guo et al., 21 Sep 2025). In adjacent deep reinforcement learning literature, the same acronym denotes the policy-side churn reduction component of CHAIN, where output changes on a disjoint reference batch are penalized to control off-batch policy drift (Tang et al., 2024). Across these usages, the shared purpose is to dampen policy oscillations, smooth policy evolution, and limit unnecessary policy drift while preserving task performance.

1. Conceptual basis and definitions

PCR is rooted in the broader notion of policy churn, namely unintended or excessively rapid policy change during learning. In GRPOformer, policy churn refers to “large, abrupt fluctuations in the policy across consecutive updates,” meaning large changes in the action probabilities output by the Transformer policy from one iteration to the next (Guo et al., 21 Sep 2025). In CHAIN, churn is defined more generally as unintended change in network outputs on states or state-action pairs not included in the current mini-batch update; for the policy network, the per-state quantity is

$$c_{\pi}(\phi,\phi^{\prime}, \bar s) = \pi_{\phi^{\prime}}(\bar s) - \pi_{\phi}(\bar s),$$

with batch-level policy churn

$$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$

This makes PCR a form of function-space regularization: it constrains policy outputs, not merely parameters (Tang et al., 2024).

A related but distinct formulation appears in value-based reinforcement learning, where policy churn is the rapid change of the greedy policy induced by changing value estimates. The per-state change between policies $\pi$ and $\pi'$ is measured by total variation,

$$W(\pi, \pi' \mid s) := \frac{1}{2}\sum_a \left|\pi(a|s) - \pi'(a|s)\right|,$$

and the state-distribution-weighted aggregate is

$$\mathbb{W}(\pi, \pi') := \mathbb{E}_{s\sim d}[W(\pi, \pi' \mid s)].$$

For greedy deterministic policies, this reduces to the fraction of states in which the $\arg\max$ action changes (Schaul et al., 2022).

The practical motivation is that churn can be both large and persistent. In the DQN-style Atari setting studied in “The Phenomenon of Policy Churn,” the greedy action changes in about 10% of states after a single gradient update, and substantial churn can remain even after performance stabilizes (Schaul et al., 2022). PCR is therefore aimed at suppressing policy movement that is not essential for improvement.

2. Core mathematical formulations

In GRPOformer, hyperparameter optimization is modeled as conditional sequence generation over task information $\tau$, hyperparameter configuration space $\mathcal{H}$, and historical trajectory

$$\mathcal{T} = \{(h_1, r_1), (h_2, r_2), \dots, (h_t, r_t)\},$$

where $$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$0 is the $$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$1-th hyperparameter configuration and $$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$2 is its evaluation result. The Transformer models

$$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$3

GRPO supplies the policy-optimization backbone through group-relative advantages

$$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$4

and PCR adds an explicit regularizer

$$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$5

where $$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$6 is a set of reference states chosen from the earlier portion of the historical optimization trajectory. The final policy loss is

$$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$7

PCR thus enters as a direct KL penalty on policy drift between consecutive policies (Guo et al., 21 Sep 2025).

In CHAIN, PCR is the policy-side churn reduction term within a broader framework that also includes value churn reduction. The policy regularizer is

$$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$8

where $$\mathcal{C}_{\pi}(\phi,\phi^{\prime}, B_{\text{ref}}) = \frac{1}{|B_{\text{ref}}|}\sum_{\bar s \in B_{\text{ref}}} |c_\pi(\phi,\phi^{\prime},\bar s)|.$$9 is mean squared error for deterministic policies and KL divergence for stochastic policies. The policy objective becomes

$\pi$0

To reduce manual tuning, CHAIN proposes

$\pi$1

using running means of the main policy loss and the churn regularizer (Tang et al., 2024).

A closely aligned formulation appears in C-CHAIN, which does not use the exact term PCR but penalizes output change on a disjoint reference batch. Its churn-reduction loss is

$\pi$2

with an adaptive policy coefficient reported as

$\pi$3

The underlying mechanism is the same in spirit: minimize off-batch functional drift caused by the current update (Tang et al., 31 May 2025).

3. Optimization rationale and learning dynamics

The immediate motivation for PCR in GRPOformer is that clipped updates alone do not eliminate instability. Although GRPO is PPO-like and critic-free, the authors state that training can still exhibit drastic policy oscillations; in hyperparameter optimization, the Transformer can overreact to recent rewards and change its sampling behavior too aggressively. PCR is therefore introduced to constrain the policy on earlier trajectory states, reduce sudden shifts in action probabilities, smooth policy evolution, improve convergence stability, and preserve the benefits of group-relative learning while reducing churn (Guo et al., 21 Sep 2025).

CHAIN generalizes this rationale into a dynamical account of reinforcement learning with function approximation. Its central claim is a chain effect of churn: parameter updates cause value and policy churn; churn then causes deviations in the action gradient and policy value; those deviations bias following parameter updates. The paper summarizes this causal loop as “(U.I) Parameter updates cause the value and policy churn, (U.II) which further leads to deviations in the action gradient and policy value; (U.III) the churns and the deviations then bias following parameter updates” (Tang et al., 2024). Under this view, PCR is not merely a smoothness prior; it is a device for breaking a feedback loop that compounds instability across generalized policy iteration.

C-CHAIN connects the same problem to plasticity loss through the Neural Tangent Kernel. It defines churn as “network output variability for out-of-batch data induced by mini-batch training” and argues that loss of plasticity is accompanied by exacerbation of churn due to gradual rank decrease of the NTK matrix. The paper further claims that reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively (Tang et al., 31 May 2025). This gives PCR-style regularization an additional interpretation as a mechanism for preserving representational adaptability in continual RL.

A crucial qualification is that not all churn is necessarily harmful. “The Phenomenon of Policy Churn” argues that churn can be a beneficial but overlooked form of implicit exploration, and reports that reducing churn by acting with a lagged target network can reduce performance in some settings (Schaul et al., 2022). This suggests that PCR cannot be treated as a universally monotone objective: it must trade off stabilization against the exploratory role of policy movement.

4. Architectural settings and algorithmic uses

The most explicit named PCR formulation appears in GRPOformer, a framework for hyperparameter optimization that combines three components: a Transformer-based hyperparameter generator, Group Relative Policy Optimization, and Policy Churn Regularization. The Transformer ingests task information $\pi$4, the hyperparameter configuration space $\pi$5, and the historical optimization trajectory $\pi$6, then generates the next hyperparameter configuration. Each iteration generates a group of candidate hyperparameter configurations, evaluates them, appends them to the trajectory, and updates the policy with GRPO plus PCR (Guo et al., 21 Sep 2025). Within this framework, PCR is tightly integrated with the sequential, trajectory-conditioned nature of HPO.

In CHAIN, PCR is one component of a broader churn-reduction framework rather than the entire method. For PPO, CHAIN is used as policy churn reduction (PCR) only. For TD3 and SAC, the framework can apply value churn reduction, policy churn reduction, or both together as double churn reduction. For offline methods such as IQL and AWAC, PCR and value churn reduction are treated as plug-ins for the corresponding actor and critic networks (Tang et al., 2024). CHAIN is therefore a family-level deployment pattern for PCR-style regularization across policy-based, actor-critic, and offline RL.

Related continual-learning work uses policy regularization mechanisms that are conceptually close to PCR even when the term itself is absent. COPR regularizes the current policy based on the historically optimal distribution to mitigate catastrophic forgetting, fitting the current policy to a re-normalized optimal distribution and constraining drift on replay samples with KL terms against historical optimal policies (Zhang et al., 2023). A later COPR formulation casts the same idea as constrained optimization with Lagrangian Duality, dynamically regularizing the current policy based on the historically optimal policy in continual RLHF settings (Zhang et al., 2024). These methods address long-horizon policy drift across tasks rather than short-horizon policy oscillation within a task, but the functional logic is closely related.

5. Empirical evidence

The clearest direct evidence for PCR as a named module comes from the GRPOformer ablation study on OpenML. The evaluation covers 6 machine learning models—XGBoost, SVM, Logistic Regression, Random Forest, Neural Network, and Histogram-based Gradient Boosting—across 6 datasets, for 36 HPO scenarios. Metrics are BtR, MP, MnP, and MnR. In the ablation, the full GRPOformer reports BtR $\pi$7, MP $\pi$8, MnP $\pi$9, and MnR $\pi'$0; removing PCR yields BtR $\pi'$1, MP $\pi'$2, MnP $\pi'$3, and MnR $\pi'$4; removing GRPO yields BtR $\pi'$5, MP $\pi'$6, MnP $\pi'$7, and MnR $\pi'$8 (Guo et al., 21 Sep 2025). The reported interpretation is that PCR improves stability, convergence behavior, and overall HPO performance.

The broader CHAIN experiments indicate that policy churn reduction remains useful outside HPO. On PPO benchmarks in MuJoCo and DeepMind Control, CHAIN reduces policy churn substantially and improves performance in most tasks, especially Ant and HalfCheetah. On TD3 and SAC, PCR often gives the stronger performance gains relative to value churn reduction, while double churn reduction is not always additive. In offline RL, CHAIN improves scores on most tasks for IQL and AWAC. In scaling experiments with widened and deepened PPO networks, CHAIN consistently improves performance across most scale settings, while naive scaling degrades performance (Tang et al., 2024).

C-CHAIN provides complementary empirical support from the continual-learning perspective. It is evaluated on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar. The reported findings are that vanilla PPO often collapses after early tasks, C-CHAIN improves average performance over Vanilla and often over other baselines, it gets the best aggregate score in the ProcGen table, and it maintains higher NTK rank and lower off-diagonal values than vanilla PPO (Tang et al., 31 May 2025). Although the paper does not name the regularizer PCR, its results support the more general claim that reducing policy churn improves stability and continual adaptability.

6. Distinctions, misconceptions, and related regularization families

PCR should not be conflated with CHAIN as a whole. In the terminology of “Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn,” CHAIN is the overall framework, VCR is value churn reduction, PCR is policy churn reduction, and DCR is the joint application of both (Tang et al., 2024). This distinction matters because some results attributed to “churn reduction” arise from critic-side or joint regularization rather than policy regularization alone.

The term PCR is also not used uniformly across the literature. Some papers propose closely aligned mechanisms without using the name. PPO-RPE explicitly constrains the latest policy to the baseline one through relative Pearson divergence and is described as a policy-update regularization method with a clear minimization target, but it is framed as “proximal policy optimization with relative Pearson divergence” rather than PCR (Kobayashi, 2020). Jacobian conditioning regularization similarly adds a conditioning penalty to PPO or TRPO and is motivated by policy stability and generalization, but it is formulated through sensitivity of policy outputs to state perturbations, not through a direct policy-churn objective (Asadulaev et al., 2019).

A further misconception is that all churn reduction problems are policy-learning problems. In supervised learning, the related notion is predictive churn, defined as disagreement between retrained classifiers on test samples. “Locally Adaptive Label Smoothing for Predictive Churn” reduces churn by modifying targets through local soft labels in logit space rather than penalizing policy change between successive updates (Bahri et al., 2021). This is best regarded as an implicit stability regularizer, not PCR in the reinforcement-learning sense.

Finally, PCR is not equivalent to “minimize all policy change.” The policy-churn literature explicitly notes that large churn can coexist with policy null spaces and that churn may support implicit exploration (Schaul et al., 2022). A plausible implication is that effective PCR must be selective: strong enough to suppress harmful off-batch drift and oscillation, but not so strong that it removes beneficial exploratory movement or makes the policy excessively rigid. This tension is visible in current formulations. GRPOformer does not specify the exact numerical value of $\pi'$9, the size of the reference-state subset $$W(\pi, \pi' \mid s) := \frac{1}{2}\sum_a \left|\pi(a|s) - \pi'(a|s)\right|,$$0, or a separate PCR-specific schedule (Guo et al., 21 Sep 2025), while CHAIN notes that automatic coefficient tuning is helpful but not fully universal across domains (Tang et al., 2024).