---
title: Primacy Bias in Deep RL
url: https://www.emergentmind.com/topics/primacy-bias-in-deep-reinforcement-learning
type: topic
---

# Primacy Bias in Deep RL

Primacy bias in deep reinforcement learning (RL) is the empirically grounded tendency for neural network–based agents, when trained continually or with high replay ratios, to overfit their earliest experiences and consequently lose the ability to adapt to new data or tasks. This effect manifests across value-based and actor–critic methods, on both stationary and non-stationary task streams, and is associated with a loss of plasticity in the neural function approximator. Modern research describes and quantifies primacy bias, analyzes network-level mechanisms, and proposes algorithmic modifications to prevent plasticity collapse and restore continual learning capacity [2205.07802][2303.07507][2412.07224][2507.02712][2502.00802].

## 1. Formal Definitions and Theoretical Foundations

In deep RL with experience replay, the primacy bias is the property that early transitions in the replay buffer receive a disproportionate number of gradient updates, especially as the buffer grows. For a buffer of $N$ samples and a replay ratio $R$ (number of SGD updates per environment step), the expected number of updates involving the $i$th sample (added at time $i$) is approximately $c_i(T)=R\,[\ln T-\ln i]$ up to time $T$, so the earliest experiences dominate the optimization trajectory [2205.07802][2507.02712].

In continual RL, where an agent cycles through tasks or environments without resets, primacy bias arises as "experiences seen early in training exert an outsized and enduring influence on the network parameters, to the point that the agent stops adapting to new tasks or environment variations” [2303.07507]. The parameterization of the Q-function or policy becomes inert in regions required for later tasks; gradients with respect to new data vanish due to dead activations or ill-conditioned weights [2303.07507][2412.07224].

The phenomenon can also be formalized via the Fisher Information Matrix (FIM): early experiences induce high-curvature regions in parameter space, locking network weights so that subsequent updates are less effective (“memorization phase” followed by a sharp drop and plateau in the trace of the FIM, signaling reduced plasticity) [2502.00802].

## 2. Empirical Manifestations and Diagnostics

Primacy bias is consistently observed in standard off-policy deep RL with high replay ratios, in continual learning scenarios, and with non-stationary reward/task streams. The following metrics and indicators are reported:

- **Performance Degradation:** Episodic returns on revisited tasks in continual RL settings decline over successive visits, relative to a reset baseline [2303.07507].
- **Gradient Collapse:** The $\ell_p$-norms of gradients, especially in critical layers, fall by over an order of magnitude as the agent cycles through tasks; $\|\nabla_\theta L\|_p\rightarrow 0$ indicates the network becomes untrainable [2303.07507].
- **Activation Sparsity:** The fraction of active post-activation units plummets in downstream MLP layers (to $<1\%$ in some cases), corresponding to “activation collapse” and cessation of learning [2303.07507].
- **Plasticity Loss:** In sequential-task regimes, success rate on new tasks typically drops to near zero and recovers poorly [2412.07224].
- **Replay Bias Quantification:** The primacy ratio (sampling weight of the earliest $p$ fraction of data over the latest) sharply exceeds one in unmitigated agents, reflecting increased sampling of initial experiences [2205.07802][2507.02712].

Table: Typical Primacy Bias Signatures Across Scenarios

| Manifestation              | Metric or Signal                                  | Reference        |
|----------------------------|---------------------------------------------------|------------------|
| Early experience overfit   | $c_i(T)\propto R[\ln T-\ln i]$                    | [2205.07802]     |
| Gradient collapse          | Diminishing $G_p = \langle \|\nabla_\theta L\|_p\rangle$ | [2303.07507]     |
| "Dead" activations         | Activation sparsity $>99\%$ in value/adv layers   | [2303.07507]     |
| Loss of task plasticity    | Declining per-task return, poor recovery          | [2412.07224]     |

## 3. Mechanistic Analyses: Network Dynamics and Replay

The primacy bias is driven by a combination of replay buffer effects, optimizer dynamics, and neural function approximation properties:

- **Replay Buffer Chronology:** In uniform replay with fixed-rate updates, early samples persist longer and thus dominate learning. The expected “usage count” of the oldest transition is $\Theta(R\log N)$ per environment step, versus $O(1)$ for recent samples [2507.02712][2205.07802].
- **Optimizer Effects:** Under high update-to-data ratios (UTD), Adam’s second-moment estimates amplify spurious large gradients on early or out-of-distribution actions, driving Q-values to diverge and locking weights to early modes (“value divergence”) [2403.05996].
- **Activation Collapse:** ReLU-based networks in continual settings develop extreme sparsity, with most downstream units clamped to zero; no gradient is passed, so representations needed for new tasks never emerge [2303.07507].
- **Weight Matrix Conditioning:** During sequential task learning, weight-matrix rank and stable rank decrease, neuron-weight correlations rise, and input–output Jacobian variance increases, resulting in a network less able to update along novel task directions [2412.07224].
- **Parameter Geometry:** Early data create high-curvature regions (FIM eigenvalues increase), rendering weights locally “sticky” and later innovations ineffective unless plasticity is explicitly restored [2502.00802].

## 4. Algorithmic Remedies and Regularization Approaches

Several mitigation strategies have been introduced and empirically validated:

- **Partial or Periodic Resets:** Re-initializing parts or all of the network at fixed intervals (without resetting the buffer) “breaks” the effect of primacy bias, allowing rapid relearning of relevant representations while leveraging stored data [2205.07802][2310.20287]. The method improves sample efficiency but can cause performance collapses immediately after reset unless ensemble averaging is employed [2310.20287].
- **Deep Ensemble Resetting:** Resetting individual members of a policy or value ensemble sequentially and aggregating actions using the Q-value of the oldest head removes abrupt collapse after reset and provides both diversity and continual access to prior experience [2310.20287].
- **Activation Function Replacement:** Swapping ReLU for Concatenated ReLU (CReLU), which guarantees at least half of units remain active, restores gradient flow, prevents complete activation sparsity, and maintains performance across task switches [2303.07507].
- **Weight Matrix Regularization:** Parseval regularization penalizes deviations from orthogonality in weight matrices, preserving isometry and thus plasticity. Networks retain high stable rank, near-zero neuron-weight correlation, and uniform Jacobian variance under sequential tasks [2412.07224].
- **Fisher-Guided Selective Forgetting (FGSF):** Periodically injecting FIM-shaped noise into parameters selectively disrupts early “memorization modes” and reestablishes capacity to represent new data, especially effective for the critic [2502.00802].
- **Experience Replay Decay (ER Decay):** Assigns time-decaying weights to replay buffer contents so that older transitions are sampled with exponentially decreasing probability. This bounds every transition’s usage to $O(1)$ per environment step and flattens the replay-driven training skew [2507.02712].
- **Network Expansion:** Dynamically enlarging the critic by adding new residual blocks rejuvenates plasticity after capacity collapse, as fresh parameters are unexposed to overfit early data [2507.02712].

## 5. Primacy Bias in Model-Based RL and Continual Learning

In model-free RL, agent parameter primacy dominates; resetting the agent network disrupts overfit to initial experience. In model-based RL, however, “primacy” most strongly affects the world model—the learned dynamics function. Overfitting to early buffer transitions in the world model impairs subsequent policy performance, even if the agent’s networks remain flexible [2310.15017]. Resetting only the world model’s last layers, on a schedule tuned to the model UTD ratio, preserves adaptation and yields significant gains in continuous- and discrete-control domains [2310.15017].

In continual RL, primacy bias is tightly linked with loss of plasticity: agents show sensitivity to task order, declining improvement on successive tasks, and catastrophic forgetting (failure to learn or recover on new tasks). Orthogonalization of weight matrices [2412.07224] or invariant feature activations [2303.07507] are central for retaining plasticity in continual regimes.

## 6. Comparative Empirical Insights and Benchmarks

A body of experiments across Atari 100k, DeepMind Control Suite, MuJoCo, MetaWorld, CARL, and HumanoidBench demonstrates that uncorrected primacy bias severely degrades final return, learning speed, and adaptability. For example, in off-policy RL, ER Decay and network expansion (“Forget+Grow”/FoG) yields state-of-the-art normalized return (0.92), surpassing strong baselines SimBa, BRO, and TD-MPC2 (0.69–0.76) over 40+ tasks [2507.02712]. Parseval regularization delivers an absolute 20pp increase in continual RL success on MetaWorld 20–10, a 50% gain in median GridWorld success rate, and a 10–15pp gain on CARL tasks [2412.07224]. FGSF leads to 25–50 % improvement in complex DMC tasks over standard and reset-based SAC [2502.00802]. Reset Deep Ensemble methods eliminate performance collapses and improve safety (halving constraint violations in Safe RL settings) versus vanilla resets [2310.20287].

## 7. Practical Recommendations and Future Directions

Best practices for mitigating primacy bias include:

1. Monitor replay sampling ratios and plasticity-related metrics (gradient norms, activation sparsity, weight matrix rank, FIM trace) to detect early onset of bias.
2. Implement periodic partial resets or ensemble-based resets, tuning interval and layers for network size and task nonstationarity [2205.07802][2310.20287].
3. Employ architectural modifications (CReLU, Parseval regularization, network expansion) to maintain continuous activation and avoid capacity collapse [2303.07507][2412.07224][2507.02712].
4. Use replay and optimizer schemes (ER Decay, OFN, FIM-based noise, Adam parameter tuning) that decouple the domination of early data [2403.05996][2507.02712][2502.00802].
5. In model-based RL, prioritize regular re-initialization of the world model, not the agent policy, to ensure adaptation to new buffer evidence [2310.15017].
6. Adjust replay ratios, buffer sizes, and task shuffling to balance stability and adaptability; avoid excessive replay of early transitions.

The collective results suggest that overcoming primacy bias is fundamental for robust, sample-efficient continual and off-policy deep RL. Ongoing research pinpoints the control of experience replay, maintenance of parameter plasticity, and architectural regularization as key design axes for future algorithm development [2205.07802][2303.07507][2412.07224][2507.02712][2502.00802].

Source: https://www.emergentmind.com/topics/primacy-bias-in-deep-reinforcement-learning