--- title: Reptile-style Meta-Experience Replay url: https://www.emergentmind.com/topics/reptile-style-meta-experience-replay type: topic --- # Reptile-style Meta-Experience Replay Reptile-style Meta-Experience Replay (MER) is a scalable continual learning framework for large language models (LLMs) that combines experience replay with a Reptile-inspired meta-learning objective to promote gradient alignment across new and previously-seen data. The technique is motivated by the inefficiency of retraining LLMs from scratch when new data become available. By maintaining knowledge and mitigating catastrophic forgetting during continual pre-training, MER improves loss retention and generalization without incurring substantial compute or memory overhead [2508.01908]. ## 1. Meta-Learning Objective and Gradient Alignment MER frames continual pre-training as an unconstrained meta-learning problem. Let $k$ denote a chunk of consecutive mini-batches $\{B_1, \dots, B_k\}$, each of size $N$. Building on the formulation of Riemer et al. (2019), MER seeks parameter updates $\theta$ that not only minimize the standard loss $L(B_i; \theta)$ on each batch but also encourage positive dot products between the gradients of loss computed on new and replayed examples. The formal objective is: \[ \min_\theta\; \mathbb{E}_{B_1,\dots,B_k}\left[ 2\sum_{i=1}^k \left( L(B_i;\theta) - \sum_{j=1}^{i-1} \beta\, \nabla_\theta L(B_i;\theta) \cdot \nabla_\theta L(B_j;\theta)\right)\right] \] Here: - $L(B_i; \theta)$ is the cross-entropy loss on batch $B_i$. - $\nabla_\theta L(B_i; \theta)\cdot\nabla_\theta L(B_j;\theta)$ measures alignment (transfer) or interference. - There are no hard constraints; the gradient dot product is a regularization term. The negative penalty on anti-aligned gradients directly leads to gradient alignment, which incentivizes parameter updates beneficial across both new and replayed examples. ## 2. MER Algorithm: Inner and Outer Loops MER implements a two-loop optimization strategy: **Inner loop:** The primary training step samples a batch $B$ as a mixture of $(1-\alpha)N$ new examples (from the data stream $p(x|t)$) and $\alpha N$ replayed examples (from a reservoir buffer $M$). Standard AdamW updates are applied: \[ \theta \leftarrow \theta - \beta \, \hat\nabla_\theta L(B; \theta) \] with batch size $N$, learning rate $\beta$ (cosine-warm-restarted schedule), and replay rate $\alpha \in \{0, 0.25, 0.5\}$. **Outer loop:** Every $k$ mini-batches, a Reptile-style meta-update is performed: \[ \theta \leftarrow \theta_{\mathrm{before}} + \epsilon(\theta_{\mathrm{after}} - \theta_{\mathrm{before}}) \] where $\epsilon$ is the meta-learning rate (typically $0.1$). $\theta_{\mathrm{before}}$ and $\theta_{\mathrm{after}}$ respectively denote parameters $k$ steps before and after inner-loop updates. **Replay Buffer:** Updated via on-disk reservoir sampling, $M$ maintains a uniform sample of all observed examples, unconstrained except by disk capacity. ## 3. Implicit Gradient Alignment via Reptile Interpolation In contrast to methods that explicitly project gradients (e.g., GEM, PCGrad), MER achieves alignment implicitly. The Reptile meta-objective (see above) introduces an effective regularization that promotes positive transfer and suppresses interference without gradient projection. Specifically: - The regularizer $-2\beta\sum_{i