Data-efficient pre-training by scaling synthetic megadocs

Abstract: Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near $1.48\times$ data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from $1.48\times$ to $1.80\times$ at $32$ generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.

• The paper demonstrates that synthetic rephrasing reduces real data requirements by 1.48× while lowering validation loss (from 3.55 to 3.41).
• It introduces megadoc strategies that concatenate multiple rephrasings, improving long-context modeling and yielding up to a 5% accuracy gain on benchmarks.
• It reveals that careful mixing of synthetic and real data, combined with ensembling, leads to enhanced generalization across both short and long documents.

Data-Efficient LLM Pre-Training via Synthetic Megadoc Scaling

Motivation and Problem Setting

The paper addresses the scarcity of high-quality web text for LLM pre-training in the face of rapidly escalating compute budgets. As compute grows faster than data availability, optimizing data efficiency—achieving minimal validation loss per real token used—becomes strategically essential. The focus is the data-limited regime: training overparameterized LLMs (here, 300M parameters) on just 200M real tokens, with essentially infinite training compute. The core question is: how far can synthetic data augmentation and novel document-level manipulations push data efficiency, validation loss, and downstream generalization?

Synthetic Data Augmentation and Scaling Paradigms

The authors begin with synthetic rephrasing, generating multiple paraphrases ("rephrases") per real document using a strong external model (Llama 3.1 8B Instruct). Each real doc is transformed into $G$ distinct rephrasings, multiplying the effective data size. Crucially, synthetic and real data are not merged indiscriminately: a careful mixing procedure is employed, tuning both the ratio and the number of epochs over the real data to avoid overfitting to synthetic noise.

Figure 1: Scaling synthetic data by increasing rephrasing count per document and introducing megadoc strategies significantly improves validation loss compared to simply using real data.

Monotonically increasing $G$ directly improves i.i.d. validation loss, plateauing near $G = 32$ rephrasings/doc, with best 300M models reaching 3.41 loss (compared to the 3.55 baseline). Data scaling laws imply that this synthetic-augmented approach requires $1.48\times$ less real data to reach the same loss, a substantial gain in the compute-constrained regime. The loss gains are reflected in downstream generalization, with up to 5% accuracy improvement on PIQA, SciQ, and ARC Easy.

Figure 2: Validation loss and downstream benchmark scores both improve and then plateau as the number of rephrasings per document increases, supporting the effectiveness but eventual limits of simple synthetic scaling.

Megadoc Construction: Long-Context Synthesis Algorithms

Simple synthetic rephrasing shuffles all generated variants as i.i.d. short-form texts, missing deeper inter-generational structure. The paper proposes "megadocs," where all $G$ rephrasings (or other generations) of a given real document are concatenated into a single long sequence—potentially exceeding the context window—either via:

• Stitched Rephrasing: Concatenate $G$ synthetic rephrasings (and optionally the original real doc) into a megadoc.
• Latent Thoughts: Chunk real documents and, for each split, use the generator to interpolate explicit rationales ("thoughts") that tie prefixes and suffixes, stretching the document with generated, interpretable reasoning traces.

  Figure 3: Visualization of the different megadoc strategies—simple rephrasing, stitched rephrasing, and latent thoughts—demonstrates how synthetic data is organized at the document level.

On top of boosting long-context modeling (as measured by loss on arXiv CS papers and other long-form validation sets), both stitched rephrasing and latent thoughts provide further i.i.d. loss reduction ($1.64\times$ and $1.80\times$ data efficiency, respectively), outperforming basic rephrasing. The scaling of megadocs is particularly favorable—the performance improvement with increasing $G$ shows little plateauing compared to the baseline.

Figure 4: Comparison of various synthetic data scaling strategies demonstrates clear benefit for megadoc-based algorithms in both i.i.d. and long-context loss.

Figure 5: Loss and benchmark gains with growing generation counts are more pronounced and less plateaued for megadoc approaches versus simple rephrasing.

Importantly, megadocs allow more real data epochs and higher synthetic mixing ratios before overfitting, increasing the total number of effective training steps and thus the utilization of the limited real data budget.

Figure 6: Megadocs preserve the scaling benefit under longer training regimes—when controlling for number of steps, megadocs retain a constant advantage.

Interaction with Model Ensembles

The analysis extends to ensembles of synthetically-pretrained models. Unlike self-distillation (using model-generated "student labels" as targets), where ensemble generalization matches single-model asymptotes, augmenting with synthetic data via megadocs compounds with ensembling, lowering ensemble loss asymptotes by at least 0.12 absolute loss versus the all-real-data baseline.

Figure 7: Ensembling magnifies the loss gains of synthetic data pre-training, with megadocs continuing to lower the ensemble asymptote unlike standard self-distillation.

Ablations and Hyperparameter Studies

The study devotes thorough attention to training protocol optimization:

• Convexity in Hyperparameters: Exhaustive grid search confirms that tuning mixing and epoching is critical for monotonic improvements.
• Inclusion of Real Docs in Synthetic Streams: Inserting originals into the synthetic pool further decreases loss.
• Cross-Document Attention: Allowing tokens from different docs within a megadoc to attend to each other is justified under heavy synthetic augmentation, in contrast to traditional masking for data efficiency.

Model and Generator Scaling Implications

Ablations with larger students (1.5B parameters) confirm that data efficiency gains from synthetic methods continue to increase with student capacity, and are not mere artifacts of teacher-student capability imbalance. Experiments and prior work also support the feasibility of continued self-improvement: even when the data generator is matched in data to the student (as in self-distillation or bootstrapped training), synthetic augmentation can yield non-trivial gains, provided sufficient compute and careful data mixing.

Figure 8: Scaling the student model (from 300M to 1.5B parameters) increases the loss reductions obtained from all synthetic data methods.

Figure 9: Joint scaling of student and generator demonstrates conditions for optimal self-improvement as evidenced by downstream QA accuracy.

Generalization: Long-Context and Document-Type Robustness

Megadoc strategies are effective not only for long-context evaluation but also improve short-context document modeling, suggesting broader generalization benefit rather than a pure long-range dependence effect.

Figure 10: On genuinely long-context domains (GitHub code, Wikipedia), megadocs yield larger, less-plateaued loss reductions compared to simple rephrasing.

Figure 11: Megadoc improvements persist across both short and long document types, demonstrating universality of the approach.

Theoretical and Practical Implications

The findings situate data-efficient large-scale language modeling at a conjuncture: with internet-scale data nearing exhaustion for LLMs, synthetic data augmentation based on structured megadoc construction provides a clear, scalable avenue for further improvements. The empirical evidence dispels concerns of overfitting or loss of diversity inherent in brute-force synthetic augmentation, provided parameters (mixing, epoching) and document-level structure are appropriately tuned.

Practically, these results advocate for synthetic megadoc construction across the LLM pipeline, especially in resource-constrained or open-source settings where compute can be traded for real data scarcity, and for ensembles, which benefit synergistically from megadoc pre-training. On the theory front, the interaction between document structure, synthesis algorithms, long-context scaling, and generalization points to a rich design space for optimizing data representations well beyond naive i.i.d. augmentation.

Conclusion

This work establishes that scaling synthetic data generation—and in particular building megadocs that tie together multiple generations or interpolated thoughts per real document—significantly increases data efficiency for LLM pre-training. These methods improve i.i.d. loss, long-context robustness, and downstream accuracy, with gains persisting for both single models and ensembles, and scaling with model size. As the supply of high-quality web text plateaus, such structured synthetic augmentation will play a dominant role in future LLM pre-training strategies.

(2603.18534)