Replay Divergence: Mechanisms and Implications

• Replay divergence is a multi-context term defining discrepancies between nominal and replayed data across reinforcement learning, cyber-physical security, and lifelong learning applications.
• In reinforcement learning, diverse replay enhances zero-shot generalisation, while stale or high-TD-error samples lead to off-policy and gradient mismatches.
• In cyber-physical and continual learning settings, controlled divergence via metrics like KL, TV, or JS supports detection of replay attacks and preserves data diversity.

to=arxiv_search diýenjson {"query":"id:(Weltevrede et al., 2023) OR id:(Zaman et al., 2020) OR id:(Jiang et al., 2024) OR id:(Bao et al., 2021) OR id:(Naha et al., 2020) OR id:(Cicek et al., 2021) OR id:(Saglam et al., 2022) OR id:(Sharshar et al., 10 Jun 2026) OR id:(Li et al., 17 Mar 2026)","max_results":10,"sort_by":"relevance"}【อ่านข้อความเต็มanalysis to=arxiv_search code omitted ႏjson {"results":[{"arxiv_id":"(Weltevrede et al., 2023)","title":"The Role of Diverse Replay for Generalisation in Reinforcement Learning","authors":["Samuel M Allen","Diana Borsa","Gregory Farquhar","Mark Rowland","Tim Genewein"],"abstract":"In reinforcement learning (RL), key components of many algorithms are the exploration strategy and replay buffer. These strategies regulate what environment data is collected and trained on and have been extensively studied in the RL literature. In this paper, we investigate the impact of these components in the context of generalisation in multi-task RL. We investigate the hypothesis that collecting and training on more diverse data from the training environments will improve zero-shot generalisation to new tasks. We motivate mathematically and show empirically that generalisation to tasks that are \"reachable'' during training is improved by increasing the diversity of transitions in the replay buffer. Furthermore, we show empirically that this same strategy also shows improvement for generalisation to similar but \"unreachable'' tasks which could be due to improved generalisation of the learned latent representations.","categories":["cs.LG","stat.ML"],"published":"2023-06-09T17:44:34Z","pdf_url":"http://arxiv.org/pdf/([2306.05727](/papers/2306.05727))v2"},{"arxiv_id":"([2004.02257](/papers/2004.02257))","title":"Security Analysis and Fault Detection Against Stealthy Replay Attacks","authors":["S. H. Mousavian","F. Pasqualetti","M. Zamani"],"abstract":"This paper investigates the security issue of the data replay attacks on the control systems. The attacker is assumed to interfere with the control system process in a steady-state case. The problem is presented as the standard way to attack, which is storing measurements and replay ing them in further times to the system. The controller is assumed to be the LQG controller. The main novelty in this paper can be stated as proposing a different attack detection criterion by using the K-L divergence method to cover more general control system problems with these attacks and with higher-order dynamics. Also, there exists a packet-dropout feature in transmitting the data as another contribution of the paper. Formulations and numerical simulations prove the effectiveness of the newly proposed attack detection procedure by having a quick response to occurred attacks. Although, in previous approaches, the trade-off between attack detection delay or LQG performance was significant, in this approach it is proved that the difference in this trade-off is not considered in early moments when the attack happens since the attack detection rate is rapid and thus, these attacks can be stopped with defense strategies in the first moments with the proposed attack detection criterion.","categories":["eess.SY"],"published":"2020-04-05T13:07:52Z","pdf_url":"http://arxiv.org/pdf/([2004.02257](/papers/2004.02257))v1"},{"arxiv_id":"([2408.16999](/papers/2408.16999))","title":"A Tighter Convergence Proof of Reverse Experience Replay","authors":["Hua Su","Fengxuan Wu"],"abstract":"In reinforcement learning, Reverse Experience Replay (RER) is a recently proposed algorithm that attains better sample complexity than the classic experience replay method. RER requires the learning algorithm to update the parameters through consecutive state-action-reward tuples in reverse order. However, the most recent theoretical analysis only holds for a minimal learning rate and short consecutive steps, which converge slower than those large learning rate algorithms without RER. In view of this theoretical and empirical gap, we provide a tighter analysis that mitigates the limitation on the learning rate and the length of consecutive steps. Furthermore, we show theoretically that RER converges with a larger learning rate and a longer sequence.","categories":["math.OC","cs.LG"],"published":"2024-08-30T14:45:16Z","pdf_url":"http://arxiv.org/pdf/([2408.16999](/papers/2408.16999))v2"},{"arxiv_id":"([2111.01865](/papers/2111.01865))","title":"Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replay","authors":["Murat Bektas","Hikmet Yildirim Sinir"],"abstract":"The experience replay mechanism allows agents to use the experiences multiple times. In prior works, the sampling probability of the transitions was adjusted according to their importance. Reassigning sampling probabilities for every transition in the replay buffer after each iteration is highly inefficient. Therefore, experience replay prioritization algorithms recalculate the significance of a transition when the corresponding transition is sampled to gain computational efficiency. However, the importance level of the transitions changes dynamically as the policy and the value function of the agent are updated. In addition, experience replay stores the transitions are generated by the previous policies of the agent that may significantly deviate from the most recent policy of the agent. Higher deviation from the most recent policy of the agent leads to more off-policy updates, which is detrimental for the agent. In this paper, we develop a novel algorithm, Batch Prioritizing Experience Replay via KL Divergence (KLPER), which prioritizes batch of transitions rather than directly prioritizing each transition. Moreover, to reduce the off-policyness of the updates, our algorithm selects one batch among a certain number of batches and forces the agent to learn through the batch that is most likely generated by the most recent policy of the agent. We combine our algorithm with Deep Deterministic Policy Gradient and Twin Delayed Deep Deterministic Policy Gradient and evaluate it on various continuous control tasks. KLPER provides promising improvements for deep deterministic continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training.","categories":["cs.LG"],"published":"2021-11-02T15:36:52Z","pdf_url":"http://arxiv.org/pdf/([2111.01865](/papers/2111.01865))v2"},{"arxiv_id":"([2111.07511](/papers/2111.07511))","title":"Lifelong Vehicle Trajectory Prediction Framework Based on Generative Replay","authors":["Bingqing Chen","Xingyuan Zhang","Jianyu Wang","Jingliang Du","Yousong Zhu","Yilong Ren","Shengbo Eben Li"],"abstract":"Accurate trajectory prediction of vehicles is essential for reliable autonomous driving. To maintain consistent performance as a vehicle driving around different cities, it is crucial to adapt to changing traffic circumstances and achieve lifelong trajectory prediction model. To realize it, catastrophic forgetting is a main problem to be addressed. In this paper, a divergence measurement method based on conditional Kullback-Leibler divergence is proposed first to evaluate spatiotemporal dependency difference among varied driving circumstances. Then based on generative replay, a novel lifelong vehicle trajectory prediction framework is developed. The framework consists of a conditional generation model and a vehicle trajectory prediction model. The conditional generation model is a generative adversarial network conditioned on position configuration of vehicles. After learning and merging trajectory distribution of vehicles across different cities, the generation model replays trajectories with prior samplings as inputs, which alleviates catastrophic forgetting. The vehicle trajectory prediction model is trained by the replayed trajectories and achieves consistent prediction performance on visited cities. A lifelong experiment setup is established on four open datasets including five tasks. Spatiotemporal dependency divergence is calculated for different tasks. Even though these divergence, the proposed framework exhibits lifelong learning ability and achieves consistent performance on all tasks.","categories":["cs.CV","cs.RO"],"published":"2021-11-15T06:14:50Z","pdf_url":"http://arxiv.org/pdf/([2111.07511](/papers/2111.07511))v2"},{"arxiv_id":"([2209.00532](/papers/2209.00532))","title":"Actor Prioritized Experience Replay","authors":["William Doherty","Zhongqiang Ren","Alistair Letcher","Nikolay Yakovenko","Timothy P. Lillicrap","Amy Zhang","Maren Awad","Robert Calandra"],"abstract":"A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical studies indicate that it considerably underperforms actor-critic algorithms in continuous control. We theoretically show that actor networks cannot be effectively trained with transitions that have large TD errors. As a result, the approximate policy gradient computed under the Q-network diverges from the actual gradient computed under the optimal Q-function. Motivated by this, we introduce a novel experience replay sampling framework for actor-critic methods, which also regards issues with stability and recent findings behind the poor empirical performance of PER. The introduced algorithm suggests a new branch of improvements to PER and schedules effective and efficient training for both actor and critic networks. An extensive set of experiments verifies our theoretical claims and demonstrates that the introduced method significantly outperforms the competing approaches and obtains state-of-the-art results over the standard off-policy actor-critic algorithms.","categories":["cs.LG"],"published":"2022-09-01T17:41:52Z","pdf_url":"http://arxiv.org/pdf/([2209.00532](/papers/2209.00532))v3"},{"arxiv_id":"([2012.10748](/papers/2012.10748))","title":"Sequential detection of Replay attacks","authors":["Aritra Mitra","Suraj Anand","Shreyam Dey","Aditya Mahajan"],"abstract":"One of the most studied forms of attacks on the cyber-physical systems is the replay attack. The statistical similarities of the replay signal and the true observations make the replay attack difficult to detect. In this paper, we have addressed the problem of replay attack detection by adding watermarking to the control inputs and then performed resilient detection using cumulative sum (CUSUM) test on the joint statistics of the innovation signal and the watermarking signal. We derive the expression of the Kullback-Liebler divergence (KLD) between the two joint distributions before and after the replay attack, which is asymptotically inversely proportional to the detection delay. We perform structural analysis of the derived KLD expression and suggest a technique to improve the KLD for the systems with relative degree greater than one. A scheme to find the optimal watermarking signal variance for a fixed increase in the control cost to maximize the KLD under the CUSUM test is presented. We provide various numerical simulation results to support our theory. The proposed method is also compared with a state-of-the-art method.","categories":["eess.SY","cs.SY","math.OC"],"published":"2020-12-19T05:53:50Z","pdf_url":"http://arxiv.org/pdf/([2012.10748](/papers/2012.10748))v2"},{"arxiv_id":"([2606.12655](/papers/2606.12655))","title":"Amnesia: A Stealthy Replay Attack on Continual Learning Dreams","authors":["Tobias Zörgiebel","Nils Schöne","Chhabi Biswas","Mohammadmahdi Soltanolkotabi","Mario Fritz","Yingzhen Li"],"abstract":"Continual learning (CL) models often use experience replay to reduce catastrophic forgetting, but their robustness to replay sampling interference remains underexplored. Existing CL attacks alter inputs or training pipelines (poisoning/backdoors) and rarely include explicit auditable constraints, limiting realism. Here, auditability means a monitor can verify compliance from sampler-visible telemetry - e.g., logged replay index/label statistics - by checking that the realized replay class histogram stays close to a nominal baseline and that replay rate is unchanged per batch and/or over a rolling window. We study a limited-privilege insider who controls only replay index selection, not pixels, labels, or model parameters, while staying within auditable limits such as queue priorities. We introduce Amnesia, a replay composition attack that maximizes degradation under two budgets: a visibility budget delta bounding the TV/KL divergence from a nominal class histogram p0, and a mass budget f fixing the replay rate. Amnesia has two steps: (i) compute lightweight class utilities, such as EMA loss or confidence, to tilt p0 toward harmful classes; and (ii) project the tilt back into the delta-ball using efficient KL (exponential tilt) or TV (balanced mass redistribution) optimizers. A windowed scheduler enforces rolling audits. Across challenging CL benchmarks and strong replay baselines, Amnesia consistently lowers final accuracy (ACC) and worsens backward transfer (-BWT). The KL variant delivers high impact while remaining largely undetected under multiple audit schemes, including per-batch and rolling-window checks. The TV variant is more damaging but easier to detect, especially under tight per-class constraints. These results expose index-only replay control as a practical, auditable threat surface in CL systems and establish a principled impact-visibility trade-off.","categories":["cs.LG","cs.CR"],"published":"2026-06-10T14:48:39Z","pdf_url":"http://arxiv.org/pdf/([2606.12655](/papers/2606.12655))v1"},{"arxiv_id":"([2603.16157](/papers/2603.16157))","title":"DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay","authors":["Zhiyuan Yuan","Xiaolin Xiao","Nanjun Jiang","Wenhao Gong","Yujiu Yang"],"abstract":"While Reinforcement Learning (RL) enhances LLM reasoning, on-policy algorithms like GRPO are sample-inefficient as they discard past rollouts. Existing experience replay methods address this by reusing accurate samples for direct policy updates, but this often incurs high computational costs and causes mode collapse via overfitting. We argue that historical data should prioritize sustaining diversity rather than simply reinforcing accuracy. To this end, we propose Dynamic Jensen-Shannon Replay (DyJR), a simple yet effective regularization framework using a dynamic reference distribution from recent trajectories. DyJR introduces two innovations: (1) A Time-Sensitive Dynamic Buffer that uses FIFO and adaptive sizing to retain only temporally proximal samples, synchronizing with model evolution; and (2) Jensen-Shannon Divergence Regularization, which replaces direct gradient updates with a distributional constraint to prevent diversity collapse. Experiments on mathematical reasoning and Text-to-SQL benchmarks demonstrate that DyJR significantly outperforms GRPO as well as baselines such as RLEP and Ex-GRPO, while maintaining training efficiency comparable to the original GRPO. Furthermore, from the perspective of Rank-k token probability evolution, we show that DyJR enhances diversity and mitigates over-reliance on Rank-1 tokens, elucidating how specific sub-modules of DyJR influence the training dynamics.","categories":["cs.LG","cs.CL"],"published":"2026-03-17T11:03:53Z","pdf_url":"http://arxiv.org/pdf/([2603.16157](/papers/2603.16157))v1"}]} Replay divergence is a context-dependent term for discrepancies induced, measured, or controlled through replayed data. In reinforcement learning, it may denote insufficient replay diversity, off-policy mismatch between stale replayed behavior and the current policy, or replay-induced divergence between approximate and true policy gradients. In cyber-physical security, it denotes a Kullback–Leibler divergence between nominal and replayed innovation statistics under replay attacks. In continual and lifelong learning, it denotes either a conditional distribution shift across tasks or an auditable deviation between a nominal replay histogram and the realized replay composition (Weltevrede et al., 2023, Naha et al., 2020, Bao et al., 2021, Sharshar et al., 10 Jun 2026).

1. Scope of the term

Across the cited literature, replay divergence is not a single universally standardized scalar. The term covers several mathematically distinct objects, each tied to a different replay mechanism and a different failure mode.

Context	Replayed object	Divergence notion
Multi-task RL generalisation	Transitions in a replay buffer	Diversity or reachable state-action coverage
Off-policy actor-critic RL	Batches or TD-prioritized transitions	KL-based policy mismatch or gradient mismatch
Reverse replay theory	Consecutive replayed subsequences	Contractivity and stability of the reverse backup operator
Cyber-physical security	Replayed sensor measurements	KLD between nominal and replayed innovation statistics
Lifelong trajectory prediction	Replayed trajectories across cities	Conditional KL divergence between $p(Y|X)$ across tasks
Auditable continual learning	Replay index or class composition	TV/KL deviation from a nominal replay histogram
RL with verifiable rewards	Historical successful trajectories	JS-regularized distributional anchoring to prevent mode collapse

In the RL generalisation setting, the relevant quantity is not a formal scalar replay-divergence metric; diversity is operationalised through broader state-action coverage and by whether replay remains diverse over training. In replay-attack detection, by contrast, divergence is explicitly a KLD-based detection statistic. In lifelong prediction, divergence is a conditional Kullback–Leibler divergence over future trajectories given context. In auditable continual learning, divergence is a constrained TV or KL distance between replay class histograms. In RL with verifiable rewards, the central regularizer is Jensen–Shannon divergence rather than direct replay imitation (Zaman et al., 2020, Saglam et al., 2022, Li et al., 17 Mar 2026).

2. Coverage-based replay divergence in reinforcement learning

A coverage-based notion of replay divergence is developed in multi-task zero-shot transfer. The central claim is that replay diversity matters for zero-shot generalisation: if training explores more broadly and samples more broadly from replay, the learned policy tends to generalise better to new tasks. The formal distinction is between ordinary RL and multi-task zero-shot transfer. In singleton RL, there is usually no reason to care about states that the optimal policy never visits during training. In zero-shot transfer, however, the test task may require competence in parts of the state space that were not important for maximizing training return.

The formal object is the reachable set $S_r(\mathcal{M}|_{S_0^{train}})$, defined as all states $s_r$ for which there exists a sequence of actions that gives non-zero probability of reaching $s_r$ in the training MDP. A closure property holds: if a state $s$ is reachable, and another state $s'$ is reachable from $s$, then $s'$ is also reachable. This motivates ZSPT-R, where $S_0^{test} \subset S_r(\mathcal{M}|_{S_0^{train}})$. In that case, if a policy is optimal on the whole reachable set, then it is optimal for the reachable test tasks as well. The operational implication is that broader replay coverage should make the agent optimal on a larger portion of the reachable state space and thereby improve transfer (Weltevrede et al., 2023).

The replay mechanism used to instantiate this idea is also specific. Standard buffer-uniform replay is contrasted with an $(S,A)$-uniform replay strategy that samples mini-batches uniformly over the state-action pairs present in the buffer,

$S_r(\mathcal{M}|_{S_0^{train}})$0

rather than uniformly over stored transitions. In experiments, diversity is also increased by slowing the decay of $S_r(\mathcal{M}|_{S_0^{train}})$1-greedy exploration. Because the replay buffer is a large FIFO buffer with size equal to the full $S_r(\mathcal{M}|_{S_0^{train}})$2-step training horizon, earlier exploratory data is not discarded, so slower decay means more diverse replay.

The empirical study uses a small fully observable 4-room grid world adapted from MiniGrid. The observation is a $S_r(\mathcal{M}|_{S_0^{train}})$3 tensor centered on the agent, with channels encoding the agent, next-move location, walls, and goal. The task variation comes from room topology, agent location, agent direction, and goal location. The key metric is the fraction of reachable states where the greedy policy

$S_r(\mathcal{M}|_{S_0^{train}})$4

is optimal. Longer exploration, and hence more diverse replay, produces a policy that is optimal on a larger fraction of the reachable state space, while training return remains largely unchanged. On a 100% reachable test set, more diverse replay improves zero-shot performance and correlates with higher optimality over reachable states. The same trend appears on a 0% reachable test set, where the theory does not directly apply; the proposed explanation is improved latent representations. A linear-probe/fine-tuning analysis suggests that FC1 learns a latent representation that already generalises to both reachable and unreachable states, and that the quality of this representation depends on replay diversity.

3. Replay order, off-policyness, and gradient mismatch

A second line of work treats replay divergence as an optimisation or stability problem rather than a coverage problem. In reverse experience replay, the relevant question is whether replay order preserves contraction. Reverse Experience Replay samples a consecutive subsequence of length $S_r(\mathcal{M}|_{S_0^{train}})$5 and updates the Q-function in reverse order. The governing matrix is

$S_r(\mathcal{M}|_{S_0^{train}})$6

Earlier theory required the restrictive condition $S_r(\mathcal{M}|_{S_0^{train}})$7. A tighter analysis replaces the earlier contraction bound with

$S_r(\mathcal{M}|_{S_0^{train}})$8

valid under the broader condition $S_r(\mathcal{M}|_{S_0^{train}})$9. Here replay divergence is not a named scalar but a question of whether long reverse sweeps remain contractive rather than destabilizing. The analysis does not prove general divergence of RER; it strengthens the contraction argument and shows that larger learning rates and longer replayed sequences can still be controlled in linear MDP Q-learning (Jiang et al., 2024).

In off-policy deterministic actor-critic methods, replay divergence is often a mismatch between replayed actions and the current actor. Batch Prioritizing Experience Replay via KL Divergence constructs, for each sampled batch, a Gaussian “Batch Generating Policy” from the deviation between stored actions and current-policy actions, then scores the batch by

$s_r$0

The selected batch is the one with the minimum KL score among $s_r$1 candidate batches. The intended effect is selection-based off-policy correction: rather than importance weighting, the replay mechanism prefers a batch that is most likely generated by the most recent policy. This formulation treats replay divergence as off-policy divergence induced by stale replay data (Cicek et al., 2021).

A related but sharper claim appears in actor-prioritized replay for actor-critic methods. There the concern is not only stale behavior, but the fact that PER samples high-TD-error transitions, and those are precisely transitions on which the critic may be unreliable enough that the actor’s approximate policy gradient diverges from the true gradient. The core chain is

$s_r$2

The remedy is asymmetric replay: inverse prioritized sampling for the actor, prioritized replay for the critic, and a shared uniformly sampled subset to preserve actor-critic coupling. In this usage, replay divergence is a replay-induced divergence between the gradient computed under $s_r$3 and the gradient that would be obtained under $s_r$4 (Saglam et al., 2022).

4. Statistical replay divergence in cyber-physical security

In cyber-physical systems, replay divergence has a more literal and explicit statistical meaning. A replay attack stores legitimate measurements during nominal steady-state operation and later re-injects those stale measurements. In a steady-state LQG/Kalman setting, the defender compares the nominal innovation sequence

$s_r$5

with the replayed innovation sequence

$s_r$6

For Gaussian innovations, the KLD is written as

$s_r$7

This KLD is the replay-divergence quantity: if replayed and nominal innovations are statistically indistinguishable, the divergence is zero; if replay changes the innovation covariance, it becomes positive. A central theoretical result is that if

$s_r$8

is stable, then the extra replay term vanishes asymptotically and $s_r$9. A plain KL test alone is therefore ineffective against a pure replay attack in steady state unless additional excitation or modified control logic is used. To make the attack detectable, a small random excitation

$s_r$0

is added to the control input, which drives the replay divergence away from zero at the cost of an LQG performance penalty (Zaman et al., 2020).

A sequential variant formulates the same idea at the level of quickest change detection. The system adds a private watermark,

$s_r$1

and runs a CUSUM test on the joint statistics of the innovation and the watermark. Under normal operation, the innovation is uncorrelated with the watermark; under replay, the attacked innovation contains the term $s_r$2, so the post-attack joint covariance acquires a nonzero cross-covariance. The resulting KLD between the pre-attack and post-attack joint Gaussian laws is the replay-divergence statistic governing delay: $s_r$3 For systems with relative degree greater than one, the direct coupling vanishes, and the proposed remedy is to include a delayed watermark $s_r$4 in the test. In this literature, replay divergence is therefore the per-sample information gain that separates attack from nominal operation (Naha et al., 2020).

5. Continual learning, generative replay, and diversity-preserving replay

In lifelong vehicle trajectory prediction, replay divergence appears first as a task-difference diagnostic. Different cities and road sections induce different spatiotemporal dependencies between observed history $s_r$5 and future trajectory $s_r$6. Those differences are formalised by the conditional Kullback–Leibler divergence

$s_r$7

The conditioning variable is dimension-normalized by selecting the $s_r$8 closest vehicles and using the target history together with spectral features of a Laplacian matrix,

$s_r$9

Mixture Density Networks approximate each task’s $s$0, and Monte Carlo sampling approximates the CKLD because exact KL between GMMs is not analytically tractable. This divergence is then used to interpret continual-learning difficulty: fine tuning performs better for low-CKLD task pairs and worse for high-CKLD pairs. Generative replay is implemented with a conditional GAN, R2GAN, and two merging strategies, Longterm-Data-Merge and Longterm-Temporal-Merge, so that replayed trajectories can be mixed with new data and catastrophic forgetting can be alleviated (Bao et al., 2021).

In auditable continual learning, replay divergence is elevated from a diagnostic statistic to an explicit stealth constraint. If $s$1 is the nominal replay-buffer class histogram and $s$2 is the attack-chosen replay class distribution, the audit checks

$s$3

The attack objective is

$s$4

The KL optimizer has the exponential-tilt form

$s$5

while the TV optimizer is a two-sided water-filling or mass-redistribution procedure. The replay rate is fixed by a mass budget $s$6, and rolling-window audits are enforced by a residual-budget scheduler. Empirically, the KL variant is more stealthy under per-batch and rolling-window checks, whereas the TV variant is more damaging but easier to detect (Sharshar et al., 10 Jun 2026).

In RL with verifiable rewards, a further shift occurs: replay divergence is linked to mode collapse rather than forgetting or attack detection. Dynamic Jensen-Shannon Replay treats historical data as a reference distribution that should preserve diversity rather than as extra positive data for direct gradient updates. The total objective is

$s$7

where $s$8 is a replay-based Jensen–Shannon regularizer estimated from stored token-level log-probabilities. The buffer is FIFO with a Max Age $s$9, retaining only recent successful samples,

$s'$0

The paper reports that policy entropy drops very quickly early in training, usually within about the first 20 steps, so DyJR raises the target fill rate from $s'$1 to $s'$2 in the initial phase. The best result occurs at $s'$3, while larger values such as 16, 32, and 64 reduce accuracy. In this setting, replay divergence is the tendency of naive replay to overfit to a narrow set of successful chains; JS regularization is introduced as a bounded, symmetric constraint to mitigate that collapse (Li et al., 17 Mar 2026).

6. Recurring distinctions and limitations

A recurring misconception is to treat replay divergence as a universal replay-buffer pathology with a single metric. The literature does not support that simplification. In one setting the relevant object is reachable state-action coverage; in another it is the KL separation between nominal and replayed innovations; in another it is CKLD between conditional trajectory distributions; in another it is a TV/KL audit statistic on replay composition; and in yet another it is a gradient-level mismatch induced by high-TD-error replay (Weltevrede et al., 2023, Naha et al., 2020, Bao et al., 2021, Saglam et al., 2022, Sharshar et al., 10 Jun 2026).

A second distinction concerns what replay is meant to accomplish. In multi-task RL, replay diversity is valuable because zero-shot transfer may require competence in states outside the optimal training trajectory, and training return can remain largely unchanged even while reachable-space optimality and zero-shot performance improve. This suggests that replay should not be evaluated only by sample efficiency or training reward. At the same time, the reachability-based theory is explicitly restricted to ZSPT-R; for unreachable tasks the representation-learning explanation remains empirical rather than fully formalised (Weltevrede et al., 2023).

A third distinction concerns detectability and controllability. In steady-state replay attacks, a pure KL test can collapse to zero divergence when the relevant closed-loop quantity is stable; detectability requires excitation, watermarking, or modified control logic. Conversely, in auditable continual learning, divergence is not merely observed after the fact but bounded by design through visibility budgets and rolling audits (Zaman et al., 2020, Sharshar et al., 10 Jun 2026).

Several limitations are also domain-specific. The reverse-replay convergence theory is developed for linear MDPs with bounded features, zero inherent Bellman error, a covariance lower bound under a stationary distribution, and target-network updates; it does not settle deep nonlinear function approximation. CKLD-based task comparison depends on MDN density estimation and Monte Carlo approximation. DyJR’s argument is tied to recent successful trajectories and a dynamic reference distribution rather than a long-lived archive. These limitations do not invalidate the concept, but they make clear that replay divergence is best understood as a family of formally distinct discrepancies arising wherever replay changes the effective training, estimation, or detection distribution (Jiang et al., 2024, Bao et al., 2021, Li et al., 17 Mar 2026).