Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

Abstract: The training paradigm of LLMs has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but under-explored question. Through a novel observation of model's "Memorization Window" signals derived from loss retention dynamics and downstream evaluation scores, we propose "Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit). While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data.

• The paper introduces a memorization window framework that adaptively schedules data reuse to balance learning efficacy and prevent overfitting.
• It defines retention and generalization windows, supported by empirical experiments, to determine the optimal intervals for re-exposing high-quality data.
• Experimental results show that strategically spaced data replays yield learning signals equivalent to fresh data injection while minimizing redundancy.

Memorization-Guided Data Reuse: A Principled Framework for Efficient LLM Training

Introduction

The paradigm shift from single-epoch to multi-epoch training for LLMs is motivated by the limited supply of high-quality (HQ) data amid scaling demands. This paper, "Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training" (2607.04969), formalizes the necessity for sophisticated data reuse strategies, introduces the concept of the Memorization Window, and empirically characterizes when repeated exposure to training samples remains beneficial versus when it induces overfitting. It advocates a principled approach of adaptive scheduling guided by memorization signals, moving past heuristic repetition limits and fixed global shuffling.

Memorization Window Theory

The memorization window is defined via two axes:

• Retention-based window ($\tau_{\text{ret}}$): Quantifies the interval over which a model retains memorized information, measured via the loss retention gap (difference between loss at first exposure and loss after continued training, excluding the sample).
• Generalization-based window ($\tau_{\text{gen}}$): Denotes the horizon at which downstream evaluation scores peak before overfitting sets in.
• Effective window ($\tau^*$): The safe reuse interval, formalized as $$\tau^* = \min(\tau_{\text{ret}}, \tau_{\text{gen}})$$, defining when re-exposing samples is maximally informative.

This framework enables analytical scheduling for data replay: re-exposing HQ data only when the model approaches forgetting, maximizing the learning signal while minimizing redundancy and overfitting.

Experimental Validation

The proposed framework is validated via loss-based and downstream evaluation studies:

• Retention window estimation: Rollback loss experiments on a 100M parameter Transformer trained on FineWeb indicate substantial decay of the loss retention gap within 1–5B tokens, empirically situating $\tau_{\text{ret}}$ in this range, independent of model capability across training stages.
• Generalization window estimation: Mixed HQ/LQ training on OpenMathInstruct2 and FineWeb, varying epoch sizes, reveals accuracy peaks at distinct points depending on the interval between HQ sample re-exposures. Larger epoch sizes delay both the peak and onset of overfitting, confirming the memorization-window hypothesis.

Notably, sufficient spacing between HQ repetitions leads to a validation loss descent indistinguishable from fresh data injection, underlining that rereading forgotten tokens is equivalent in learning utility to new tokens. However, excessive repetition leads to saturation without further gains, marking the effective horizon for reuse.

Data Scheduling and Practical Implications

This memorization-aware regime provides actionable guidelines for multi-epoch training:

• Set epoch size $N \geq \tau^*$ to prevent over-memorization.
• Limit total epochs $E \leq E_{\text{max}}$ (determined by generalization plateau).
• Incorporate LQ data to regularize and enhance distributional diversity, as shown by performance leaps even with minimal LQ admixture.

Scheduling based on memorization windows, rather than random shuffling, enables optimal replay timing for each data source, maximizing efficiency with scarce HQ corpora. Deterministic data loading and adaptive replay based on per-sample memory signals further augment flexibility and reproducibility.

Theoretical and Scaling Considerations

Key claims and scaling implications include:

• Model size: Larger models retain information longer, necessitating more spaced repetition to avoid overfitting, implying unique data requirements scale with model capacity.
• Architecture: Retention properties may differ across Transformer, linear attention, or retrieval-augmented models, requiring fine-grained profiling for each.
• Training phase: Memorization window consistency across training stages suggests dynamic scheduling; early phases tolerate tighter reuse, late phases require increased spacing.
• Data complexity: Memorization varies across domains and entropy levels, indicating per-source scheduling and interference management in mixtures.

Connections to Grokking and Continual Learning

The memorization window naturally connects to grokking dynamics—secondary performance spikes observed after overfitting—suggesting multi-phase learning regimes that may be intentionally exploited via repetition scheduling. Retention window also offers mechanisms for replay buffer strategies in continual learning, preventing catastrophic forgetting using forgetting boundary-based replay.

Limitations and Future Directions

Current analysis is constrained by domain (mathematical reasoning), model scale (100M), and data types. Generalizing memorization window scaling laws requires systematic studies across architectures, larger models, mixed domains, and diverse hyperparameter regimes.

Online estimation of memorization window during training, adaptive curriculum integration, and mixture-aware scheduling remain open for investigation.

Conclusion

This work establishes the Memorization Window as a theoretically principled and empirically validated criterion for adaptive data reuse in LLM pretraining. Scheduling repetitions based on loss retention and generalization dynamics allows practitioners to optimize the efficiency of limited HQ data, reducing overfitting and maximizing learning signal. By moving toward memorization-aware training, the study delineates clear practical guidelines for epoch sizing and replay frequency and lays groundwork for future research in scalable, domain-sensitive, and architecture-adaptive optimization of LLMs.