Synthetic bootstrapped pretraining (2509.15248v1)

Abstract: We introduce Synthetic Bootstrapped Pretraining (SBP), a LLM (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter model on up to 1T tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers a significant fraction of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases -- SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.

• The paper introduces a novel SBP framework that synthesizes new training data by leveraging inter-document correlations across semantically related documents.
• It employs nearest neighbor pairing, conditional synthesizer tuning, and joint training with synthetic data to achieve up to 49% of Oracle QA improvements at large scales.
• Comprehensive ablation studies and benchmark evaluations validate SBP’s potential to overcome data constraints in language model pretraining.

Synthetic Bootstrapped Pretraining: Leveraging Inter-Document Correlations for LLM Scaling

Introduction

The paper introduces Synthetic Bootstrapped Pretraining (SBP), a novel framework for LLM (LM) pretraining that explicitly models and exploits inter-document correlations within a fixed pretraining corpus. Unlike standard next-token prediction, which treats documents as independent samples, SBP identifies semantically related document pairs, trains a conditional synthesizer to generate one document from another, and synthesizes a large corpus of new documents for joint training. This approach is motivated by the impending "scaling wall" in LM development, where the availability of high-quality, unique internet text is rapidly depleting, necessitating more efficient utilization of existing data.

Figure 1: SBP data synthesis pipeline: (1) semantic pairing, (2) conditional model training, (3) large-scale synthesis for joint training.

Methodology

Data-Constrained Pretraining Setup

SBP operates under a data-constrained regime, where the LM is trained on a fixed, deduplicated document collection $\mathcal{D}_{\text{pre}}$. The standard pretraining objective maximizes the marginal likelihood of each document, ignoring structural similarities and latent concept sharing across documents. SBP augments this by:

1. Nearest Neighbor Pairing: Documents are embedded (using Qwen3-0.6B-Embedding) and paired via approximate nearest neighbor (ANN) search (ScaNN) based on inner-product similarity, with a threshold to ensure semantic relevance.

   Figure 2: ScaNN system architecture for scalable distributed similarity search.

2. Synthesizer-Tuning: A conditional LM is trained to maximize $p_\theta(d_2|d_1)$ over curated pairs, initialized from the baseline checkpoint. This step encourages the model to learn diverse, high-entropy mappings rather than deterministic paraphrasing.
3. Data Synthesis: The trained synthesizer samples new documents by drawing seeds from $\mathcal{D}_{\text{pre}}$ and generating related texts, yielding a synthetic corpus $\mathcal{S}_{\text{pre}}$ for joint training.

Implementation Details

• Architecture: 3B-parameter Llama 3 variant, 26 layers, 4096 context window, custom BPE tokenizer.
• Dataset: 582M deduplicated, cleaned documents (482B tokens) from DCLM.
• Benchmarks: Held-out perplexity (OpenWebText2, LAMBADA), QA accuracy (ARC-Challenge, ARC-Easy, SciQ, Winogrande, TriviaQA, WebQS, MMLU).
• Compute: Experiments at 200B and 1T training token scales, with compute-matched baselines (repetition, oracle upper bound).

Empirical Results

SBP consistently outperforms the repetition baseline and closes a substantial fraction of the gap to the oracle upper bound (which has access to 20x more unique data). At 200B-scale, SBP achieves on average 42% of the oracle's QA accuracy improvement; at 1T-scale, this rises to 49%. Notably, SBP's gains are benchmark-dependent, with some tasks (e.g., WebQS, TriviaQA) showing larger relative improvements.

Figure 3: Distributional analysis of paired data at 200B-scale, showing relevance and duplication rates as a function of similarity threshold.

Figure 4: SBP performance as a function of synthetic token ratio at 200B-scale; optimal mixture yields maximal benchmark gains.

Figure 5: SBP scaling at 1T-scale; performance plateaus for some benchmarks, but ARC-Challenge and Winogrande continue to benefit.

Qualitative analysis reveals that synthesized documents are not mere paraphrases but often abstract latent concepts from the seed and generate new narratives, genres, or perspectives. Quantitative metrics (repetition, duplication, non-factuality, pair-irrelevance, pair-copying) confirm that SBP-generated texts maintain diversity and relevance, with factuality improving as the synthesizer is trained on more data.

Statistical Foundations

Theoretical analysis frames SBP within a Bayesian hierarchical concept model. Documents are generated by first sampling latent concepts $c$ and then generating texts conditioned on $c$. Standard pretraining learns the marginal $P(d)$, while SBP's synthesizer-tuning models the posterior $P(d_2|d_1)$, implicitly inferring shared concepts and enabling the LM to learn conditional dependencies missed by the marginal objective. This mechanism is analogous to self-distillation, where the synthesizer acts as a teacher, and the final LM training as a student.

Implementation Trade-offs and Scaling

SBP's effectiveness depends on the quality of document embeddings, the similarity threshold for pairing, and the mixture ratio of synthetic to real data. Ablation studies show that joint training on a mixture yields optimal results, with diminishing returns for excessive repetition or synthetic data. SBP's benefits are orthogonal to model scaling; experiments with a 6B-parameter model under fixed compute show complementary gains.

Resource requirements are substantial: nearest neighbor search over 60M documents, large-scale synthesis (vLLM), and joint training on up to 1T tokens. However, SBP is compatible with existing LM infrastructure and can be integrated into standard pretraining pipelines.

Implications and Future Directions

SBP demonstrates that inter-document correlations are a valuable, underutilized signal for LM scaling, especially as unique high-quality data becomes scarce. The approach is robust to the inductive biases of transformer architectures and leverages their scalability. Future work includes:

• Using LM activations for document embedding, enabling fully self-contained SBP pipelines.
• Parametric analysis of SBP scaling laws to characterize its efficiency relative to standard pretraining.
• Extending SBP to other modalities (e.g., code, multimodal data) and exploring alternative forms of weak supervision.

Conclusion

Synthetic Bootstrapped Pretraining provides a principled, empirically validated framework for augmenting LM pretraining via synthetic data that encodes inter-document correlations. SBP achieves significant improvements over repetition baselines, approaching oracle performance with limited unique data. Its Bayesian interpretation elucidates the mechanism by which LMs can infer and exploit latent concepts, offering a scalable path forward as the field confronts the limitations of internet-scale data.