---
title: 'Slow Surprise Replay: SuRe & EMA Dual-Learner'
url: https://www.emergentmind.com/topics/slow-surprise-replay-sure-plus-ema-dual-learner
type: topic
---

# Slow Surprise Replay: SuRe & EMA Dual-Learner

Slow Surprise Replay (SuRe plus EMA dual-learner) is a method for continual finetuning of large language models (LLMs) designed to mitigate catastrophic forgetting in sequential task learning. It integrates surprise-prioritised replay—storing sequences with high negative log-likelihood ("surprise")—with a dual-learner architecture employing exponential moving average (EMA) consolidation of LoRA adapters. This approach addresses both the selection and integration failure modes in continual learning, achieving state-of-the-art performance in benchmarks with a large number of tasks [2511.22367].

## 1. Continual Learning Failure Modes: Selection and Integration

Continual learning seeks to train models sequentially on tasks $D_1, \dots, D_T$ without catastrophic forgetting. The expected forgetting $\mathcal{F}$ of the "slow" model decomposes into two non-vanishing error terms:

- **Selection Error**: This arises from mismatch between the true past-task distribution $P_{1:t-1} = \tfrac{1}{t-1} \sum_{k<t} P_k$ and the replay buffer distribution $q$. It is bounded by an integral probability metric $D_{\mathcal F_{\mathrm{loc}}}(P_{1:t-1}, q)$, quantifying how the replay buffer fails to sample representative experiences from previous tasks.

- **Integration Error**: This represents variance in stochastic gradient descent (SGD) updates that can overwrite old knowledge. Lemma 2 in [2511.22367] shows that consolidating past updates via EMA reduces this variance by a factor $\approx (1-\beta)$. The slow parameter update is defined as:
  $$
  \theta_t^{\mathrm{slow}} = \beta \theta_{t-1}^{\mathrm{slow}} + (1-\beta) \theta_t^{\mathrm{fast}}, \quad \beta \in (0,1),
  $$
  yielding a bound for suboptimality that decays with higher $\beta$ and larger replay windows.

Selection and integration errors must both be addressed for robust continual learning, as neither vanishes in isolation given finite buffer and steps.

## 2. Surprise-Prioritised Replay (SuRe)

SuRe addresses the selection error by storing the most "surprising" sequences as measured by the sequence-level average negative log-likelihood under the current model. For sequence $z_i = (z_{i,1}, \dots, z_{i,T_i})$:
$$
S_\theta(z_i) = -\frac{1}{T_i} \sum_{t=1}^{T_i} \log p_\theta(z_{i,t} \mid z_{i,<t}, x_i).
$$
High-surprise examples have higher per-example gradient norms and contribute more significantly to gradient geometry; thus, prioritising them in the replay buffer minimizes $D_{\mathcal F_{\mathrm{loc}}}(P, q)$. Buffer management enforces per-task quotas. Let $S$ denote total buffer capacity (2% of total data), and after $d$ tasks, each receives $m_i = \lfloor S/d \rfloor$ slots. On arrival of new dataset $D_d$, $S_\theta(z)$ is computed for each $z \in D_d$, the top $m_d$ samples are retained as candidates $\mathcal C$, and $M$ is updated, pruning to maintain $S$ examples.

### Buffer Management Pseudocode

```plaintext
Procedure UpdateBuffer(M, C, S):
  Remove from M any old examples of the current task
  Let M' = M ∪ C
  If |M'| > S:
    Drop oldest or lowest-surprise examples until |M'| = S
  Return M'

Procedure SampleReplay(M, B_rep):
  Return UniformSample(M, size=B_rep)
```
This selection process ensures continual representation of high-impact past data across tasks.

## 3. Dual-Learner LoRA with Exponential Moving Average

The architecture attaches dual LoRA adapters to each attention layer weight ($W_Q$, $W_V$):

- **Fast LoRA ($\theta^{\mathrm{fast}}$)**: Updated by SGD on the cross-entropy loss over current and replay batches.
- **Slow LoRA ($\theta^{\mathrm{slow}}$)**: Not directly optimised; receives updates via EMA using the fast LoRA weights.

During training, the fast head minimises the expected cross-entropy over the mixed batch:
$$
\mathcal{L}_{\mathrm{CE}} = \mathbb{E}_{z \in B_{\mathrm{mix}}} \left[ -\log p_{\theta^{\mathrm{fast}}}(z) \right].
$$
The slow head is consolidated:
$$
\theta_t^{\mathrm{slow}} = \beta \theta_{t-1}^{\mathrm{slow}} + (1-\beta) \theta_t^{\mathrm{fast}}.
$$
At inference, only the base model and $\theta^{\mathrm{slow}}$ adapter are used, incurring minimal overhead.

## 4. EMA Consolidation and Hyperparameters

Generally, the EMA consolidation is:
$$
\phi_t = \alpha \phi_{t-1} + (1-\alpha) \theta_t, \quad \alpha \in (0,1),
$$
where $\alpha$ near 1 (e.g., $\alpha=0.995$) produces an effective averaging window $\approx 1/(1-\alpha) \approx 200$ steps, substantially reducing variance.

Key hyperparameters:

| Parameter              | Value                         | Definition/Role           |
|------------------------|------------------------------|---------------------------|
| Buffer size ($S$)      | 2% of total dataset          | Replay memory             |
| Per-task quota ($m_i$) | $\lfloor S/d \rfloor$        | Buffer entries per task   |
| Replay frequency ($k$) | 2                            | Steps per replay batch    |
| Batch sizes            | $B_{\mathrm{cur}}=64$, $B_{\mathrm{rep}}=32$ | Current and replay       |
| Learning rate          | $\eta=10^{-3}$               | T5-Large fast adapter     |
| LoRA rank/$\alpha_{\mathrm{LoRA}}$ | $r=8$, $32$       | Adapter structure         |
| EMA rate ($\beta$)     | 0.995                        | EMA decay                 |
| Dropout                | 0.1                          | Regularisation            |

Surprise is computed once per sequence either before or after each task, and the buffer is updated post-training as a stability measure.

## 5. Algorithmic Summary

The complete training process for task $d$ is as follows:

1. Compute candidates $\mathcal C=$ TopKSamples($D_d$, score=$S_\theta$, K=$m_d$).
2. Update buffer $M \leftarrow$ UpdateBuffer($M$, $\mathcal C$, $S$).
3. For each SGD step $s$ on $D_d$:
   - Sample $B_{\mathrm{cur}}$ from $D_d$.
   - Every $k^{th}$ step, supplement with replay batch $B_{\mathrm{rep}}$ from $M$; otherwise, use $B_{\mathrm{cur}}$ only.
   - Update $\theta^{\mathrm{fast}}$ via SGD on cross-entropy over $B_{\mathrm{mix}}$.
   - Merge into $\theta^{\mathrm{slow}}$ via EMA.

This workflow enforces dense replay of impactful past sequences and consolidates knowledge in a manner robust to SGD noise and task drift.

## 6. Empirical Performance

On two benchmarks (averaged over three random task orders):

- **Standard CL (4 tasks):**
  - Reservoir replay: 76.9% average accuracy
  - Surprise Replay: 77.2%
  - Slow Surprise Replay: **78.1%**
- **Large Number of Tasks (15 tasks):**
  - Reservoir: 69.1%
  - Surprise Replay: 72.1%
  - Slow Surprise Replay: **75.1%**

This represents an improvement of up to +5 percentage points on LNT over prior state-of-the-art. Ablation studies show resilience under small buffer sizes (e.g., with 300 examples, Slow Surprise achieves ≈75% vs. random replay ≈70%) and infrequent replay. Negative forgetting is observed, indicating increased retention or even gains on prior tasks [2511.22367].

## 7. Advantages, Limitations, and Deployment Considerations

**Advantages:**
- SuRe reduces selection error via surprise-based prioritisation, storing high-impact samples.
- EMA dual-learner minimises integration error, yielding low-variance slow-weights for deployment.
- Complementary structure, each targeting a distinct component in the additive forgetting bound.
- Minimal inference overhead, as only the slow adapter is retained at runtime.

**Limitations:**
- Requires known task boundaries and explicit per-task quotas.
- Involves an extra forward pass per task to compute surprise, adding a marginal computational cost.
- Currently operates offline; extension to task-free or fully online continual learning remains open.

**Deployment:** The method is lightweight, with buffer size scaling sublinearly (as 2% of data), LoRA adapters contributing a moderate parameter overhead, and is architecture-agnostic (applicable to vision, speech, etc.).

Slow Surprise Replay establishes surprise-prioritised buffer policies and EMA dual-learner architectures as effective, scalable mechanisms for continual LLM finetuning, substantiating the value of statistically motivated replay and low-variance integration on challenging benchmarks [2511.22367].

Source: https://www.emergentmind.com/topics/slow-surprise-replay-sure-plus-ema-dual-learner