BeyondWeb: Synthetic Data for LLM Pretraining

• BeyondWeb is a synthetic data generation framework that transforms high-quality web text into diverse, rephrased forms to enhance LLM pretraining.
• It systematically applies multiple rephrasing strategies and filters outputs before blending synthetic and real data at a 40%/60% ratio.
• The framework demonstrates improved benchmark accuracy and faster convergence compared to pure web pretraining and prior synthetic corpora.

BeyondWeb is a synthetic data generation framework for LLM pretraining designed to address the “data wall,” the point at which simply increasing web-scale data quantity yields diminishing returns. It is formulated as a rephrasing-driven pipeline rather than a purely generator-driven corpus construction method: a high-quality web subset is selected, multiple rephrasing strategies are applied, outputs are filtered, and the resulting synthetic data are mixed with real web data at a 40%/60% ratio. In the reported evaluations, BeyondWeb is positioned against pure web pretraining and prior synthetic corpora, including Cosmopedia and Nemotron-CC’s high-quality synthetic subset, Nemotron-Synth, and is described as improving both average benchmark accuracy and convergence speed in trillion-scale pretraining settings (Maini et al., 14 Aug 2025).

1. Problem Formulation and Paradigms

The framework is motivated by the observation that recent LLM pretraining results show a limit to gains from scaling data quantity alone. BeyondWeb therefore targets synthetic data as a means of extending scarce high-quality web text. Its stated objective is to jointly optimize three factors that determine per-token learning: information density, style alignment, and generation diversity.

The paper distinguishes two synthetic-data paradigms. In the generator-driven paradigm, exemplified by Cosmopedia, large models are prompted de novo. In the source-rephrasing paradigm, exemplified by WRAP and Nemotron-CC, existing web documents are rewritten into structured, high-value forms. BeyondWeb belongs to the second class. It is explicitly described as selecting a high-quality web subset (“HQ Web”), applying multiple rephrasing strategies such as question–answer pairs, pedagogical style, and content restructuring, and then filtering and mixing synthetic data with real data. This framing suggests that BeyondWeb treats synthetic pretraining data not as unconstrained generation, but as controlled transformation of selected web sources (Maini et al., 14 Aug 2025).

2. End-to-End Pipeline and Training Objective

The pipeline takes as input an HQ Web corpus $D_{\text{web}}$ selected via DataComp-LM methods, a rephraser model $G$ such as Llama-3.1-8B, Mistral-7B, or Phi-4, a strategy set $$S = \{\text{format\_transform}, \text{style\_modify}, \text{content\_restructure}, \ldots\}$$, a target synthetic token budget $N_{\text{syn}}$, and a score threshold $\tau$. Synthetic examples are generated by repeatedly sampling a document from $D_{\text{web}}$, sampling a rephrasing strategy from a weighted multinomial over $S$, prompting $G$ with that strategy and document, and retaining the result only if $\text{Score}(d_{\text{syn}}) \ge \tau$.

The scoring component is described as a simple per-token quality score, for example normalized negative log-probability under $G$ or a classifier score. After filtering, the synthetic corpus $G$0 is blended with real web data to form the pretraining mixture

$G$1

Training then optimizes the standard cross-entropy objective

$G$2

This design makes the synthetic component a controlled augmentation of web pretraining rather than a replacement. The mixture fraction $G$3 synthetic and $G$4 web is therefore a central operational choice in the framework (Maini et al., 14 Aug 2025).

3. Reported Performance Relative to Web and Prior Synthetic Corpora

The principal quantitative comparison is average accuracy across 14 benchmarks, combining 0-shot and 5-shot evaluation, at three model scales.

Scale and token budget	Average accuracy (%)	Key comparison
1B, 1T tokens	RPJ 50.7 / Cosm. 52.2 / N-Synth 54.3 / BeyondWeb 57.4	BeyondWeb: +6.7 pp vs RPJ, +5.2 pp vs Cosm., +3.1 pp vs N-Synth
3B, 180B tokens	RPJ 53.5 / Cosm. 55.8 / N-Synth 58.8 / BeyondWeb 60.8	BeyondWeb: +7.3 pp vs RPJ, +5.0 pp vs Cosm., +2.0 pp vs N-Synth
8B, 180B tokens	RPJ 56.6 / Cosm. 58.6 / N-Synth 61.1 / BeyondWeb 63.7	BeyondWeb: +7.1 pp vs RPJ, +5.1 pp vs Cosm., +2.6 pp vs N-Synth

The abstract summarizes these gains as up to 5.1 percentage points over Cosmopedia and 2.6 percentage points over Nemotron-Synth when averaged across the 14-benchmark suite. Convergence speed is reported as a second major advantage. For the 8B model, RPJ reaches 56.6% at 180B tokens, whereas BeyondWeb reaches the same accuracy in 23.2B tokens, corresponding to 7.7× faster training. Nemotron-Synth reaches 61.1% at 180B tokens, and BeyondWeb reaches that level in 66.2B tokens, corresponding to 2.7× faster training. A notable Pareto result is that a 3B BeyondWeb model at 180B tokens reaches 60.8%, surpassing all but one 8B baseline trained on the same token budget; the abstract also states that a 3B model trained for 180B tokens on BeyondWeb outperforms an 8B model trained for the same token budget on Cosmopedia (Maini et al., 14 Aug 2025).

4. Principal Determinants of Synthetic Data Quality

The ablations identify three core levers: seed-data quality, style alignment, and generation diversity. The first is seed-data quality. Rephrasing HQ Web into HQ Synth and mixing it with HQ Web yields 49.2%, whereas LQ Synth + HQ Web yields 48.6%; the baseline LQ Web + HQ Web mixture is 45.6%. The stated takeaway is that the quality of source data matters more than novelty.

The second lever is style alignment. Web data are reported as only about 3.7% conversational. Upsampling conversational content from 3.7% to 10%, 20%, and 50% yields 5-shot accuracy of 43.2%, 43.5%, 44.0%, and 44.1%, respectively. The paper therefore reports a +0.9 pp gain at 50% conversational content relative to the original 3.7%, while also stating that gains saturate beyond about 20% conversational content.

The third lever is generation diversity. Single-strategy synthetic data, including methods such as QA-WRAP or Cosmopedia, are reported to plateau early in long training runs. In the controlled diversity scaling study, single-strategy methods flatten out by about 100–200B tokens, whereas BeyondWeb’s mixed strategies continue climbing past 1T tokens. The paper’s synthesis is that no single factor suffices: only simultaneous optimization of seed quality, style matching, and diversity allows BeyondWeb to “breach the data wall.” This suggests that the framework’s reported advantage is emergent from interaction effects among several design choices rather than from any single transformation heuristic (Maini et al., 14 Aug 2025).

5. Controlled Experiments on Summarization and the Data Wall

A central controlled comparison concerns whether simple knowledge distillation via summarization is sufficient. In the summarization experiment, simple summarization of web text with an 8B model yields 46.8% average accuracy, equal to Cosmopedia at 46.8%. BeyondWeb’s focused rephrasing reaches 50.4%, a gain of +3.7 pp over summarization. Within the paper’s framing, this is evidence that rephrasing strategy matters substantially and that not all synthetic transformations produce comparable pretraining value.

A second controlled comparison tests the data wall directly. Full Web with 20B unique tokens yields 46.2%. Repeating 10B tokens twice yields 45.5%. A continuation condition with 10B real tokens and 10B naive continuation also yields 46.2%. BeyondWeb reaches 50.4%, which is +4.2 pp over the full-data condition. The interpretation given in the summary is that only strategic synthetic data can surpass the data wall.

These experiments also delimit a common misconception. The results do not support the claim that any synthetic expansion of web data is sufficient. The paper instead states that naive approaches can yield modest improvements, potentially at great cost, while well-executed methods can yield much larger improvements. In that sense, BeyondWeb is presented not as evidence for synthetic data in the abstract, but as evidence for a specific regime of source selection, targeted rephrasing, filtering, and mixture design (Maini et al., 14 Aug 2025).

6. Generator Family, Rephraser Scale, and Broader Interpretation

The paper reports that synthetic-data quality is relatively robust across several generator families. Four generators—OLMo-2-7B, Phi-4-14B, Mistral-7B, and Llama-3.1-8B—produce synthetic data that trains to 48.9–49.9%, corresponding to +3.4 to +4.4 pp over RPJ. The variance across these generators is reported as less than 1 pp, and the authors explicitly state that there is no positive correlation between the generator’s own accuracy and synthetic-data quality.

Rephraser size shows a saturation pattern. With Llama-3.2 generators, moving from 1B to 3B to 8B yields BeyondWeb accuracies of 47.3%, 48.8%, and 49.2%, respectively. The increase from 1B to 3B is +1.5 pp, whereas the increase from 3B to 8B is only +0.4 pp. The accompanying recommendation is that a 3B rephraser suffices in most production settings.

The broader conclusion is explicitly non-monocausal. The paper states that “there’s no silver bullet for generating high-quality synthetic pretraining data” and that the best outcomes require jointly optimizing many factors, a task requiring “rigorous science and practical expertise.” A plausible implication is that BeyondWeb should be understood less as a fixed recipe than as an empirical synthesis of interacting design constraints: source quality, style matching, diversity, filtering, and computational efficiency. Within that framing, its significance lies in the reported speed–accuracy trade-off for LLM pretraining and in the claim that moderate-sized rephrasers can be sufficient when these factors are jointly optimized (Maini et al., 14 Aug 2025).