---
title: Replaying Pre-Training Data
url: https://www.emergentmind.com/topics/replaying-pre-training-data
type: topic
---

# Replaying Pre-Training Data

Replaying pre-training data refers to the explicit reuse or reintroduction of data from a model’s original (or prior) pre-training distribution during later stages of training—such as continual pre-training, fine-tuning, or even self-supervised representation learning—either to mitigate catastrophic forgetting or, more recently, to enhance adaptation and data efficiency on target domains. This technique spans a wide range of neural network paradigms, from language modeling and computer vision to reinforcement learning, and is motivated by the observation that continually updating a model without revisiting its foundational data often results in the degradation of general capabilities or poor transfer to new tasks. Replaying pre-training data may involve streaming, reservoir sampling, buffer-based strategies, or proportional mixing in multi-objective losses. The appropriate use and scheduling of replayed data is subject to theoretical, empirical, and domain-specific considerations, affecting both target and generalization performance.

## 1. Theoretical Foundations and Objectives

The core rationale for replaying pre-training data is to navigate the stability–plasticity dilemma intrinsic to continual learning systems: protecting previously-acquired knowledge while simultaneously enabling plastic adaptation to new distributions. Theoretical treatments in continual supervised and self-supervised learning regimes formalize this as minimizing a risk (or loss) on target data while controlling for drift in performance on the original pre-training distribution. In fine-tuning, for example, combining the target empirical risk $F_n(\theta)$ with a weighted pre-training risk $H_m(\theta)$ yields an aggregate objective of the form
\[
\min_{\theta} \, \alpha F_n(\theta) + (1-\alpha) H_m(\theta)
\]
where $\alpha \in (0,1]$ controls emphasis on the target task. The excess risk bound on the target task can be tightened by replaying pre-training samples that are similar (in gradient or embedding space) to the target domain [2111.12292]. In continual pre-training (CPT) for language models, data replay is essential for preventing catastrophic forgetting of general capabilities—conceptually formalized as minimizing
\[
L_{\mathrm{CPT}}(\theta) = (1-r) L_{\mathrm{tgt}}(\theta) + r L_{\mathrm{src}}(\theta)
\]
with $r$ as the replay ratio [2407.02118, 2407.17467]. In self-supervised streaming, replay or parameter regularization similarly bridges the gap to joint training [2104.12081].

## 2. Replay Mechanisms and Scheduling Strategies

Multiple replay mechanisms are employed, varying by domain and training context:

- **Proportional Mixing**: Interleaving batches from pre-training and new datasets with fixed or adaptive ratios. For CPT, optimal replay ratios are empirically set between 0.1–0.3 for general tasks, balancing retention and adaptation. In fine-tuning, values as high as 0.5–0.6 are optimal when target domain data is exceedingly scarce [2603.04964, 2407.02118].

- **Curriculum Schedules and Replay Timing**: Replay may be constant throughout, or scheduled for specific epochs/steps. In continued language model pretraining, a phase transition is triggered when the learning rate reaches a fraction of its maximum—switching from a "general blend" to a "QA blend" to optimize transfer and retention [2407.07263].

- **Buffer-Based Streaming**: Streaming regimes implement replay using reservoir buffers, FIFO sliding windows, or prioritized sampling, maintaining a subset of old data for efficiency. Sizes are typically capped at 5–10% of the cumulative stream under practical memory constraints [2104.12081].

- **Similarity-Driven or Task-Selective Replay**: In vision, optimal transport and embedding-based selection isolate pre-training subsets most aligned with the target to be replayed, rather than indiscriminately sampling from the whole source corpus [2111.12292].

- **Regularized Replay**: In parameter-efficient fine-tuning (e.g., LoRA adapters), KL-regularization toward the base model can be paired with data replay, often of approximate web-text proxies, further stabilizing representations while allowing efficient instruction adaptation [2512.22337].

## 3. Data-Efficiency and Target-Task Performance Gains

Empirical studies consistently report that replaying pre-training data not only prevents forgetting but also markedly improves data efficiency for the target task, even in less-related or low-overlap scenarios:

| Study                | Data-Efficiency Gain | Replay Ratio ρ  | Regime                 | Scale      |
|----------------------|---------------------|-----------------|------------------------|------------|
| [2603.04964]         | up to 1.87×         | 0.5–0.6         | Fine-tune, 150M        | 4M target / 4B total |
| [2603.04964]         | up to 2.06×         | 0.3–0.8         | Mid-training           | 4M / 4B    |
| [2407.02118]         | Target accuracy ≈1–2× | 0.1–0.3       | CPT, cross-lingual     | 1.4B–5B    |
| [2407.07263]         | +9% rel. acc.       | Hybrid (phased) | Continued pretrain, LLM| 15B        |
| [2111.12292]         | +1%–3% top-1        | UOT-based       | Fine-tuning, vision    | ≲100M      |
| [2512.22337]         | ~3–7× less forgetting| 1–3             | LoRA SFT, replay+KL    | 1.5–14B    |

In large-scale LLM adaptation, replaying pre-training data increased web navigation success by 4.5% and Basque QA accuracy by 2% [2603.04964].

Replay is maximally beneficial when target domain data is rare or underrepresented in pre-training. The marginal utility diminishes as more target data is incorporated early in training, or as the domain gap narrows.

## 4. Trade-offs, Mixture Laws, and Predictive Scaling

A central challenge is optimizing the mixture ratio between replayed and new data to simultaneously (a) preserve base model competence (measured on general benchmarks) and (b) maximize target domain adaptation. The CMR ("Critical Mixture Ratio") scaling law models this trade-off explicitly, fitting power laws to general and domain losses as functions of tokens and mixture ratio, then solving for the largest replay fraction that maintains general loss within a user-specified budget [2407.17467].

Let $L_{\mathrm{gen}}(R, T)$ and $L_{\mathrm{dom}}(R, T)$ be general and domain loss after $T$ tokens at mixture $R$. The CMR is computed by fitting these as
\[
\Delta L_{\mathrm{dom}}(R, T) = \alpha_1(R) T^{s_1} + \beta_1(R)
\]
\[
\Delta L_{\mathrm{gen}}(R, T) = \alpha_2(R) T^{s_2} + \alpha_3(R) T^{s_3} + \beta_2(R)
\]
then solving for $R = R^*$ that maintains general loss below threshold after $T$ tokens.

For example, with $T=20\text{B}$ tokens, CMRs between 30–48% domain-data are optimal for various LLM scales [2407.17467]. This law transforms scheduling from heuristic to analytically predictable, reducing resource wastage.

## 5. Risks, Pathologies, and Controversies

Although replay is widely effective, it is not universally benign. Recent theoretical work demonstrates that in over-parameterized linear and nonlinear settings, small or adversarially-chosen replay buffers can increase—rather than decrease—forgetting, depending critically on the principal angle between subspaces of tasks [2506.04377]. If task subspaces are neither orthogonal nor fully overlapping, there exists a danger zone (principal angle ≈ π/4) where replay can exacerbate parameter drift, especially for small buffer sizes or poorly-chosen samples. Empirical results confirm non-monotonic and sometimes negative effects of replay in both linear regression and neural network tasks. These results indicate that replay’s efficacy depends on both the relationship between tasks and the adequacy of replay buffer size and diversity. 

Empirical recommendations include:
- Avoiding overly small or randomly-selected replay subsets when domain/task geometry is uncertain;
- Curating replay buffers to ensure coverage of high-leverage directions or task modes;
- Monitoring validation curves for non-monotonicity with respect to buffer size [2506.04377].

## 6. Application Domains and Practical Guidelines

Replay of pre-training data is deployed across major modalities:

- **Language Modeling**: Replaying general corpus tokens preserves out-of-domain abilities during domain specialization. Practical ratios are 10–30%, and CMR scaling allows domain-specific adaptation within generality constraints [2407.02118, 2407.17467, 2603.04964].

- **Computer Vision**: Fine-tuning on small or fine-grained datasets benefits from replaying selected pre-training classes or clusters that maximize representation similarity, especially with information-theoretic subset selection (e.g., unbalanced optimal transport) [2111.12292].

- **Self-Supervised Learning (SSL)**: Streaming pre-training with replay buffers closes nearly all the performance gap to joint training, with typical buffer sizes at 5–10% of the cumulative stream. For distribution shifts between chunks, replay plus regularization (e.g., MAS, EWC) is robust [2104.12081].

- **Parameter-Efficient Adaptation**: LoRA and similar adapters used in LLMs are susceptible to forgetting, even for small instruction-tuning runs. Combining KL-regularization with replay from approximate pre-training distributions nearly eliminates forgetting at modest compute overhead [2512.22337].

- **Data Recycling**: When high-quality organic data is scarce, synthetic replay-like generation (e.g., RL-rewarded rephrasers) can multiply data efficiency by 2–3× [2510.10681].

General guidelines include:
- Always retain (or synthesize) access to representative slices of base data for replay.
- Set replay ratios empirically within 10–50%, adjusting for domain disparity and data scarcity.
- Prefer subset or representation-based selection over uniform random sampling when feasible.
- Combine replay with regularization or KL-penalties in parameter-restricted fine-tuning.
- Use scaling laws to optimize mixture and efficiently allocate limited compute [2407.17467].

## 7. Extensions, Open Problems, and Future Directions

Emerging directions include dynamic or context-dependent curriculum replay—placing high-quality data at minima of the training re-evaluation curve (TREC) to boost LLM transfer [2509.25380]—and reward-driven data rephrasers for corpus augmentation [2510.10681]. Data-recycling pipelines are being extended to new modalities and to hybrid approaches that interleave programmatic and generative replay. In continual self-supervised settings, synthetic replay (i.e., model-generated proxies) is being developed for privacy-constrained medical imaging domains [2512.19213], though full method details remain underreported.

A persistent open question is the principled adaptation of replay strategies to mixture of experts, sparse networks, and extreme low-data or ultra-high-disparity regimes. Additionally, the theory of harmful replay and mechanisms to detect or remedy it remain active research areas. The optimal scheduling and buffer management in highly nonstationary online settings, and the translation of scaling-law approaches to multi-task or multi-modal environments, will be important for generalizing and automating replay schedules.

---
**References:**
- [2111.12292], [2104.12081], [2407.02118], [2407.17467], [2407.07263], [2506.04377], [2603.04964], [2512.22337], [2509.25380], [2510.10681], [2512.19213]

Source: https://www.emergentmind.com/topics/replaying-pre-training-data