Reptile-style Meta-Experience Replay

• The paper introduces MER, a framework that integrates experience replay with a Reptile-inspired meta-learning objective to promote gradient alignment and mitigate catastrophic forgetting.
• MER employs a two-loop optimization strategy where an inner loop mixes new and replayed examples with AdamW updates and an outer loop applies Reptile-style meta-updates with minimal overhead.
• Empirical results demonstrate that MER improves loss retention, reduces forgetting, and enhances downstream performance across LLM models without significant compute or memory costs.

Reptile-style Meta-Experience Replay (MER) is a scalable continual learning framework for LLMs that combines experience replay with a Reptile-inspired meta-learning objective to promote gradient alignment across new and previously-seen data. The technique is motivated by the inefficiency of retraining LLMs from scratch when new data become available. By maintaining knowledge and mitigating catastrophic forgetting during continual pre-training, MER improves loss retention and generalization without incurring substantial compute or memory overhead (Abbes et al., 3 Aug 2025).

1. Meta-Learning Objective and Gradient Alignment

MER frames continual pre-training as an unconstrained meta-learning problem. Let $k$ denote a chunk of consecutive mini-batches $\{B_1, \dots, B_k\}$, each of size $N$. Building on the formulation of Riemer et al. (2019), MER seeks parameter updates $\theta$ that not only minimize the standard loss $L(B_i; \theta)$ on each batch but also encourage positive dot products between the gradients of loss computed on new and replayed examples. The formal objective is:

$$\min_\theta\; \mathbb{E}_{B_1,\dots,B_k}\left[ 2\sum_{i=1}^k \left( L(B_i;\theta) - \sum_{j=1}^{i-1} \beta\, \nabla_\theta L(B_i;\theta) \cdot \nabla_\theta L(B_j;\theta)\right)\right]$$

Here:

• $L(B_i; \theta)$ is the cross-entropy loss on batch $B_i$.
• $$\nabla_\theta L(B_i; \theta)\cdot\nabla_\theta L(B_j;\theta)$$ measures alignment (transfer) or interference.
• There are no hard constraints; the gradient dot product is a regularization term.

The negative penalty on anti-aligned gradients directly leads to gradient alignment, which incentivizes parameter updates beneficial across both new and replayed examples.

2. MER Algorithm: Inner and Outer Loops

MER implements a two-loop optimization strategy:

Inner loop: The primary training step samples a batch $B$ as a mixture of $\{B_1, \dots, B_k\}$0 new examples (from the data stream $\{B_1, \dots, B_k\}$1) and $\{B_1, \dots, B_k\}$2 replayed examples (from a reservoir buffer $\{B_1, \dots, B_k\}$3). Standard AdamW updates are applied:

$\{B_1, \dots, B_k\}$4

with batch size $\{B_1, \dots, B_k\}$5, learning rate $\{B_1, \dots, B_k\}$6 (cosine-warm-restarted schedule), and replay rate $\{B_1, \dots, B_k\}$7.

Outer loop: Every $\{B_1, \dots, B_k\}$8 mini-batches, a Reptile-style meta-update is performed:

$\{B_1, \dots, B_k\}$9

where $N$0 is the meta-learning rate (typically $N$1). $N$2 and $N$3 respectively denote parameters $N$4 steps before and after inner-loop updates.

Replay Buffer: Updated via on-disk reservoir sampling, $N$5 maintains a uniform sample of all observed examples, unconstrained except by disk capacity.

3. Implicit Gradient Alignment via Reptile Interpolation

In contrast to methods that explicitly project gradients (e.g., GEM, PCGrad), MER achieves alignment implicitly. The Reptile meta-objective (see above) introduces an effective regularization that promotes positive transfer and suppresses interference without gradient projection. Specifically:

• The regularizer $N$6 strengthens transfer across batches.
• Larger chunk size $N$7 amplifies meta-regularization, trading off between compute and alignment strength.
• Key hyperparameters for tuning this behavior are $N$8 (AdamW step size), $N$9 (outer loop period), and $\theta$0 (meta-update rate).

A plausible implication is that Reptile-style interpolation, as opposed to explicit alignment, simplifies implementation and minimizes compute/memory overheads versus prior approaches.

4. Efficient Implementation and Pseudocode

MER exhibits minimal computational and memory overhead. The only additional step is an $\theta$1 vector interpolation every $\theta$2 batches.

High-level pseudo-algorithm:

$L(B_i; \theta)$1 Overhead for replay at $\theta$3 is $\theta$4 that of pure streaming, and $\theta$5 at $\theta$6. Meta-update cost is negligible. Asynchronous disk prefetching hides replay I/O latency.

5. Empirical Results: Forgetting, Retention, and Transfer

MER was evaluated on continual pre-training of LLMs (Spectra, Llama-based) at four scales (99M, 560M, 1B, 6B parameters) across 100B tokens in sequential language tasks (English $\theta$7 French $\theta$8 German, with a five-task extension adding Arabic and Japanese). Experiments used a batch size of 4096, AdamW optimizer, and linear warmup followed by cosine decay.

Key findings for the 560M parameter model and 3-task setting (English $\theta$9 French $L(B_i; \theta)$0 German):

Method	Retained Loss	Forgetting Score	Downstream Avg (HellaSwag/PiQA/PubMedQA)
No Replay	≈ 3.30	High	60.7
25% Replay	≈ 2.40	Reduced	66.4
50% Replay	≈ 2.29	Lower Still	—
25% Replay + Reptile	≈ 2.35	Forgetting Halved	67.5
50% Replay + Reptile	≈ 2.24	Lowest	—
560M Joint i.i.d.	—