Dual Replay with Experience Enhancement (DREE)

• The paper's main contribution is the SODACER framework that integrates dual replay buffers with self-organizing clustering to balance rapid adaptation and historical diversity.
• It introduces adaptive mechanisms including redundancy pruning, variance forgetting, and Gaussian cluster merging to optimize memory efficiency and accelerate convergence.
• The approach demonstrates improved learning speed, reduced cost variance, and enhanced safety through integration with control barrier functions and the Sophia optimizer.

Dual Replay with Experience Enhancement (DREE) refers to a reinforcement learning (RL) methodology that augments conventional experience replay with two synergistic memory buffers, enabling both rapid adaptation to recent trajectories and principled retention of diverse historical knowledge. The Self-Organizing Dual-buffer Adaptive Clustering Experience Replay (SODACER) framework is a concrete and comprehensive instantiation of DREE, grounded in a dual-buffer architecture that leverages self-organizing clustering, adaptive redundancy pruning, and integrated control-theoretic and optimization mechanisms for safe and scalable optimal control in nonlinear and dynamic environments (Amirabadi et al., 10 Jan 2026).

1. Architectural Foundations of Dual Replay

SODACER implements DREE through the interplay of two distinct experience buffers:

• Fast-Buffer (FB): A compact queue designed for immediate assimilation and replay of the most recent transitions $S_{\rm new} = (x_t, u_t)$. Sampling predominantly from FB yields low-bias, high-variance gradients and enables the RL agent to adjust rapidly to evolving system dynamics.
• Cluster Buffer (CB): A slow, long-term memory that eschews storing raw experiences in favor of dynamically organized Gaussian clusters. Each cluster $\mathcal{C}_j$ maintains a centroid $\mathbf{c}_j$, variance $\sigma_j^2$, and count $N_j$, representing local statistics over a segment of the historical state–action space.

By interleaving mini-batches that draw from both FB and representative members of CB, the architecture tightly couples plasticity (via recency sensitivity) to stability (via persistent diversity), offering a direct solution to the bias–variance trade-off endemic to off-policy RL methodologies.

2. Self-Organizing Adaptive Clustering Mechanism

CB’s design is predicated on an adaptive Gaussian clustering algorithm that ensures memory compactness without information loss:

1. Cluster Assignment and Creation: When an aged sample $S_{\rm old}$ leaves FB, its association to each cluster is measured via the Gaussian membership:

$$\mu_{\mathcal{C}_j}(S_{\rm old}) = \exp\left(-\frac{\|S_{\rm old} - \mathbf{c}_j\|^2}{2\sigma_j^2}\right).$$

If no existing cluster exceeds a threshold $\Gamma_{\rm th}$ in membership, a new cluster is initialized.

2. Cluster Update: If a cluster $\mathcal{C}_j$ is most representative, its parameters are adaptively updated:

$$\mathbf{c}_j \leftarrow \frac{N_j\mathbf{c}_j + S_{\rm old}}{N_j + 1}, \quad N_j \leftarrow N_j + 1, \quad \sigma_j \leftarrow \sigma_j(1 + \beta).$$

1. Adaptive Pruning and Merging:
   • Clusters with variance below $\sigma_{\rm th}$ are pruned.
   • Pairs $(i, j)$ with centroids satisfying $$\|\mathbf{c}_i - \mathbf{c}_j\| < \gamma \max(\sigma_i, \sigma_j)$$ are merged ($\gamma\approx0.32$), aggregating centroids, counts, and retaining maximal variance.
2. Variance Forgetting:

$$\sigma_k \leftarrow \sigma_k \, \sigma_0 \frac{1}{\rho}\left(1 - \frac{N_k}{\sum_i N_i}\right)$$

ensures cluster widths adapt globally to the evolving occupancy of the buffer.

This mechanism dynamically preserves salient, high-density regions of the experience space while jettisoning redundancy, maintaining both diversity and tractability over time.

3. Experience Enhancement and Replay Pipeline

The operational lifecycle of experiences in SODACER comprises:

• Immediate entry into FB upon sampling $(x_t, u_t)$.
• Migration to CB upon aging, triggering cluster assignment (creation or refinement).
• Periodic cluster structure adaptation (variance amplification, merging, pruning).
• Mini-batch assembly by combining fresh transitions from FB with cluster-based samples (centroids and members, proportionally to cluster size) from CB.

The following condensed pseudocode outlines the replay and update algorithm:

Initialize W_0, FB, empty CB, m_0=v_0=0 for t = 0,1,2,... Observe x_t, compute u_t via current policy Execute u_t, form S_new, push to FB Pop S_old from FB; compute μ_Cj(S_old) If max_j μ_Cj ≤ Γ_th: create new cluster Else: update winning cluster, adjust σ Apply variance forgetting, prune, merge clusters B ← FB ∪ CB batch Compute ∇_W J(W) on B m_t = β_1 m_{t-1} + (1-β_1)∇J v_t = β_2 v_{t-1} + (1-β_2)(∇J)^2 Bias-correct: m̂_t, v̂_t W_{t+1} = W_t - η m̂_t / (sqrt(v̂_t) + ε_0) end for

This sequence is designed to maximize both learning speed and memory efficiency.

4. Integration with Control Barrier Functions and Sophia Optimizer

Safety and stability are enforced within SODACER through the following integrations:

• Control Barrier Functions (CBFs): Constrain each action $u(t)$ via

$$\frac{\partial h}{\partial x} \left(f(x) + g(x) u\right) + \tilde{\alpha}(h(x)) \geq 0,$$

guaranteeing $h(x(t)) \geq 0$ for all $t$ (i.e., state and input constraints are always respected). This is operationalized either through action projection or Hamiltonian penalty terms during optimization.

• Sophia Optimizer: An adaptive second-order optimizer that computes dynamic step sizes using both first and second moments of the gradient (as shown in the pseudocode above), yielding superior convergence behavior compared with Adam or SGD. Sophia’s scheme adapts learning rates and curvature online, further stabilizing the dual-buffer update process.

5. Empirical Evaluation and Benchmarking Results

Validation on a nonlinear Human Papillomavirus (HPV) transmission model—with multiple control inputs and safety constraints—confirms SODACER’s efficacy. The approach was benchmarked against:

Method	Friedman Ranking	Convergence Improvement	Cost Variance Reduction
Random Experience Replay (RER)	2.80	Reference	Reference
Clustering-Based Experience Replay (CBER)	2.20	–	–
SODACER (DREE) + Sophia	1.00	30–40% faster in “Scenario 5”	>50% lower in “Scenario 5”

Over 200 randomized runs across five distinct cost scenarios $f_1$–$f_5$, SODACER achieved:

• Lowest mean cost across all scenarios
• Most rapid convergence (fewest gradient steps to tolerance)
• Superior bias–variance balance (narrowest span of outcomes)
• Best overall Friedman-test ranking (lower is better)

This substantiates SODACER’s advantages in both sample efficiency and learning stability (Amirabadi et al., 10 Jan 2026).

6. Benefits and Practical Limitations

SODACER’s DREE methodology yields the following:

• Rapid policy refinement through immediate replay of high-variance, recent experiences (FB).
• Memory efficient, diverse retention of historical patterns (CB) via clustering, redundancy pruning, and structural adaptation.
• Guaranteed safety with CBFs at every control step.
• Fast, stable learning with Sophia’s second-order adaptivity.

Potential limitations include the computational overhead of online clustering—though offset by aggressive pruning—and hyperparameter sensitivity (e.g., $\Gamma_{\rm th}$, $\sigma_{\rm th}$, $\beta$, $\rho$). A plausible implication is that for very high-dimensional state–action spaces, further investigation into scalable clustering and automated hyperparameter selection is warranted.

In summary, SODACER exemplifies the DREE paradigm: a dual-memory, experience-enhancing strategy for RL that ensures robust, scalable, and safe learning in complex optimal control domains (Amirabadi et al., 10 Jan 2026).