Synthetic Data Pretraining

• Synthetic data pretraining is a strategy that employs rendered, simulated, or rephrased data to overcome limitations in real-data collection.
• It enables precise control over labels, structure, and domain coverage, enhancing model supervision in tasks like image recognition and language modeling.
• Recent studies show that target-aware synthetic data selection and quality control can yield performance gains comparable to traditional real-data pretraining.

In the literature summarized here, synthetic data pretraining denotes pretraining or continual pretraining on data that are rendered, simulated, interpolated, translated, rephrased, or otherwise synthesized before downstream adaptation. The motivating pressures recur across modalities: real corpora and image collections are expensive to curate, pose privacy, usage-rights, and ethical issues, and become especially limiting in low-resource and data-bound regimes; synthetic data is therefore used to increase controllability over labels, structure, domain coverage, and nuisance variation, while attempting to preserve the transfer advantages of conventional real-data pretraining (Law et al., 2022, Yang et al., 2024, Maini et al., 14 Aug 2025, Manoj et al., 13 Nov 2025, Yu et al., 18 May 2026).

1. Scope, motivation, and problem formulations

Synthetic data pretraining is not a single method but a family of strategies for replacing, supplementing, or restructuring the source distribution seen during pretraining. In computer vision, it is motivated by the dependence of standard pipelines on large real-image corpora such as ImageNet, together with the observation that rendering and procedural generation can provide unlimited images, exact provenance, and explicit control over viewpoint, occlusion, illumination, and object scale (Law et al., 2022, Nakamura et al., 2024, Salehi et al., 24 Jun 2025). In language modeling, the same idea appears under data-bound scaling: the problem is no longer only compute, but the fact that available organic text falls far short of scaling demands, so synthetic transformations are used to help models more thoroughly learn from limited organic data without introducing external information (Yu et al., 18 May 2026).

A second recurring formulation is target-aware source construction. Rather than pretraining on the union of all available source datasets, several papers ask whether the target domain is known in advance and whether one can synthesize a source that is geometrically or semantically closer to that target. This appears explicitly in optimal-transport interpolation of labeled datasets, in simulator-parameter selection for downstream tasks, and in task-specific synthetic corpora for domains such as cloth-changing person re-identification, 3D vision-language learning, or myocardial infarction detection from ECGs (Fan et al., 2023, Zhao et al., 2024, Yang et al., 2024, Naghashyar, 29 Jun 2025).

A third motivation is annotation asymmetry. Synthetic pipelines are especially attractive when the target supervision is expensive but some generative mechanism already exposes latent structure “for free”: exact-instance correspondences in rendered scenes, simulator-provided object IDs and scene graphs, SQL execution answers for tables, or deterministic encrypted entities in privacy-sensitive corpora (Law et al., 2022, Yang et al., 2024, Jiang et al., 2022, Liu et al., 9 Jan 2026). This suggests that synthetic pretraining is often most effective when it does not merely imitate raw data volume, but converts hidden structure into trainable supervision.

2. Major synthesis paradigms

One major paradigm is simulation and rendering. In object detection, SOLID generates synthetic images from unlabelled 3D models with optimized scene arrangement and pretrains a detector on an instance detection task: given a query image depicting an object, detect all instances of the exact same object in a target image (Law et al., 2022). In 3D vision-language pretraining, SynVL3D constructs a synthetic scene-text corpus with 10,000 indoor synthetic scenes and more than 1 million textual descriptions, exploiting simulator metadata for object categories, 3D bounding boxes, segmentation masks, scene graphs, and view parameters (Yang et al., 2024). In cloth-changing person re-identification, CCUP builds a controllable Unreal Engine pipeline with 6,000 identities, 1,179,976 images, 100 cameras, and 26.5 average outfits per identity, explicitly separating identity-defining physiological parameters from clothing variation (Zhao et al., 2024). In biosignals, synthetic ECG pretraining uses a physiology-aware simulator with tunable MI morphology and realistic noise, adapting NeuroKit2 to a 12-lead setting with limb and precordial transfer matrices (Naghashyar, 29 Jun 2025).

A second paradigm is procedural or formula-driven generation. “Scaling Backwards: Minimal Synthetic Pre-training?” shows that pretraining can be driven by a dataset constructed from a single fractal with perturbations, using the locally integrated empirical distribution

$$p_{\Delta}(x, y) = \frac{1}{|\mathcal{R}_{\Delta}|} \int_{\mathcal{R}_{\Delta}} \delta(x - I_{\bm{\epsilon}}) \delta(y - \bm{\epsilon}) d \bm{\epsilon}$$

and the locally perturbed cross entropy

$$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$

The paper argues that subtle shape perturbations of a single structured source can be sufficient for transfer (Nakamura et al., 2024). “Stylized Structural Patterns for Improved Neural Network Pre-training” pushes this further with improved neural fractals defined by

$z_{n+1} = g(z_n) + c, \qquad z_0 = 0,$

adaptive Monte Carlo rendering, escape-time coloring, and later reverse stylization from a small set of real images (Salehi et al., 24 Jun 2025).

A third paradigm is dataset-space interpolation and source engineering. The optimal-transport formulation in “Generating Synthetic Datasets by Interpolating along Generalized Geodesics” treats datasets as probability distributions and defines a target-aware synthetic source

$$P_a := \left(\sum_{i=1}^m a_i T_i^*\right)_\sharp Q,$$

with coefficients chosen by minimizing an approximate projection objective in OTDD geometry (Fan et al., 2023). This is not generic augmentation; it is a continuous optimization over a convex hull of source datasets conditioned on the target. The same target-specific logic appears in Task2Sim, whose abstract states that different downstream tasks are favored by different configurations of simulation parameters and that there is “no one-size-fits-all solution” (Mishra et al., 2021).

A fourth paradigm is source-grounded text synthesis. This includes document rephrasing and reformatting from existing corpora, relation-conditioned document synthesis, SQL-to-question generation, and synthetic dialogues or hidden-thought prefixes. SBP learns a conditional model over related documents and generates a synthetic corpus by sampling a seed document $d_1$ and then a synthetic document $d_2 \sim p_{\theta_{\text{st}}}(\cdot \mid d_1)$ (Yang et al., 17 Sep 2025). BeyondWeb is a source-rephrasing system that transforms a high-quality subset of DCLM into more information-dense and stylistically diverse text (Maini et al., 14 Aug 2025). SynPro likewise uses rephrasing and reformat operations over organic data, while MIND converts OpenWebMath documents into synthetic multi-turn dialogues, and Reasoning CPT prepends synthetic hidden-thought sequences to original texts (Yu et al., 18 May 2026, Akter et al., 2024, Ishibashi et al., 15 May 2025). OmniTab synthesizes table QA examples by sampling SQL over tables, converting SQL into natural-language questions, and executing the SQL to obtain answers (Jiang et al., 2022).

3. Pretraining objectives and transfer protocols

The objectives used with synthetic data are diverse and often differ from the downstream task only minimally. In target-aware dataset interpolation, the synthetic object is the pretraining dataset itself, after which a model is pretrained on $\widehat P_a$ and fine-tuned on target-domain labeled data (Fan et al., 2023). In Reasoning CPT, the objective remains standard autoregressive continual pretraining,

$$\mathcal{L}_{\text{CPT}}(\theta) = -\mathbb{E}_{X \sim \mathcal{D}} \left[ \sum_{t=1}^{L} \log p_\theta(x_t \mid x_{<t}) \right],$$

but the sequence is changed from $X=S$ to

$$X = \texttt{<start\_of\_thought>} \oplus H \oplus \texttt{<end\_of\_thought>} \oplus S,$$

where $$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$0 is a synthetic hidden-thought sequence (Ishibashi et al., 15 May 2025). In SynPro, the generator itself is optimized with a composite reward

$$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$1

with $$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$2, $$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$3, and $$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$4, explicitly making synthesis model-aware rather than static (Yu et al., 18 May 2026).

Synthetic pretraining also frequently uses task-specific supervision exposed by the generator. SOLID replaces semantic category prediction with instance detection and trains with a MoCo-style contrastive loss over query and target embeddings, combined with bounding-box regression and RPN loss (Law et al., 2022). SynVL3D combines standard Masked Language Modeling, Masked Object Modeling, and Scene-Sentence Matching with Object Relationship Prediction, Multi-level Region-Word Alignment, and View-aggregated Region-Word Alignment (Yang et al., 2024). OmniTab combines a natural-data masked denoising loss,

$$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$5

with synthetic QA loss

$$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$6

and SQL supervision

$$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$7

reflecting its claim that natural data teaches alignment while synthetic data teaches reasoning (Jiang et al., 2022).

A recurring pattern is the combination of self-supervision and synthetic labels. In ECG pretraining, recurrent and transformer classifiers are pretrained with supervised synthetic labels, self-supervised masked autoencoding, and a joint reconstruction-classification objective (Naghashyar, 29 Jun 2025). In code pretraining, Arctic-SnowCoder uses a three-phase curriculum: 500B standard-quality code tokens, 50B high-quality real tokens, and a final 5B synthetic-data phase generated by Llama-3.1-70B-Instruct from top-percentile seeds (Wei et al., 2024). In the analysis paper on simpler synthetic tasks, even very generic synthetic seq2seq objectives such as the Set function recover much of the benefit of natural pretraining, implying that useful transfer can arise from surprisingly weak or abstract synthetic supervision (Wu et al., 2022).

4. Empirical evidence across modalities

The empirical record is heterogeneous but substantial. In detection, SOLID shows that label-free synthetic pretraining can be competitive with real-image pretraining: with Mask R-CNN (2fc), FPN, ResNet-50, fine-tuned on COCO under the standard 1x schedule, it obtains 41.5 box AP and 37.5 mask AP, and under 2x reaches 42.8 and 38.6 (Law et al., 2022). In low-data detection, with only 10% of COCO train2017, SOLID reaches 26.3 AP, compared with 17.8 for random initialization and 22.6 for ImageNet (Law et al., 2022). In minimal procedural vision pretraining, 1p-frac achieves 84.2 on CIFAR-100 and 89.0 on ImageNet-100 with a single perturbed fractal, while ViT-B pretrained with 1p-frac and fine-tuned on ImageNet-1k reaches 82.1 (Nakamura et al., 2024). “Stylized Structural Patterns” reports that pretraining EDM2 on its synthetic dataset yields an 11% reduction in FID, a 20% decrease in autoencoder reconstruction error, and that a ViT-S model achieves over a 10% improvement in ImageNet-100 accuracy relative to existing synthetic datasets (Salehi et al., 24 Jun 2025).

In 3D and multimodal settings, SynVL3D uses 10K indoor synthetic scenes and 1M descriptions for pretraining and reports downstream gains to 67.5 on visual grounding, 72.1 [email protected] on dense captioning, and 27.6 EM@1 on ScanQA when all proposed fine-grained tasks are used (Yang et al., 2024). In target-aware vision transfer, OTDD barycentric interpolation reaches 22.60 \pm 1.01\% raw accuracy on Oxford-IIIT Pet, compared with 18.24 \pm 3.42\% for Caltech101 alone, 14.88 \pm 0.57\% for pooling all sources, and 11.71 \pm 1.65\% for no transfer (Fan et al., 2023). In cloth-changing person re-identification, CCUP pretraining improves TransReID from 45.9 / 48.2 to 58.9 / 59.0 on PRCC, from 73.1 / 74.3 to 83.3 / 83.1 on VC-Clothes, and from 21.2 / 14.8 to 28.8 / 19.4 on NKUP (Zhao et al., 2024).

Language-model evidence is equally varied. At 200B-scale, SBP improves average QA by +2.17 over a strong repetition baseline, versus +5.09 for an oracle with much more unique data; at 1T-scale, SBP yields +0.74 versus oracle +1.50, leading the authors to characterize the method as recovering roughly 47% on average of the oracle’s QA benefit across scales (Yang et al., 17 Sep 2025). BeyondWeb improves average benchmark accuracy by up to 5.1 pp over Cosmopedia and 2.6 pp over Nemotron-Synth, and reaches baseline 8B RedPajama performance after only 23.2B tokens, a 7.7× speedup (Maini et al., 14 Aug 2025). SynPro, operating in a data-bound regime, unlocks 3.7–5.2x the effective tokens of repetition and slightly surpasses the unique-data oracle at 1.1B scale with average 0.5584 versus 0.5561 (Yu et al., 18 May 2026). In low-resource multilingual pretraining, BhashaKritika contributes approximately 540B filtered tokens across ten Indic languages and produces continued-pretraining models broadly competitive with web-data baselines, with gains on tasks such as GSM8K, TriviaQA, ARC Easy, Winogrande, Indic Sentiment, and Indic Copa, though not uniformly on Indic XNLI and Indic XParaphrase (Manoj et al., 13 Nov 2025).

Task-specific language synthesis also shows strong gains. Arctic-SnowCoder’s final 5B synthetic phase lifts HumanEval+ from 21.3 to 28.0, MBPP+ from 34.7 to 42.9, EvoEval from 12.8 to 18.0, and BigCodeBench from 12.3 to 19.4 (Wei et al., 2024). MIND-OWM raises GSM8K by +13.42%, MATH by +2.30%, MMLU by +4.55%, MMLU-STEM by +4.28%, and General Reasoning by +2.51% relative to raw-data pretraining (Akter et al., 2024). Reasoning CPT improves Gemma2-9B from 65.8 to 69.1 overall on MMLU with STEM hidden thoughts, and the gap over standard CPT grows to about 8 points on the hardest problems (Ishibashi et al., 15 May 2025). OmniTab’s omnivorous pretraining yields an absolute gain of 16.2% in the 128-shot setting and 2.7% in the full setting on WikiTableQuestions (Jiang et al., 2022). In privacy-preserving continual pretraining, encrypted synthetic data raises the Chinese Judge case score from 0.388 to 0.611, while unencrypted synthetic data reaches 0.68, showing both feasibility and an accuracy cost for encryption (Liu et al., 9 Jan 2026). In interleaved summarization, pretraining on synthetic interleaved texts allows hier2hier transfer to reach 45.56 ROUGE-1 F1 and 15.35 ROUGE-2 F1 on AMI, outperforming the cited two-step system by about 22% on ROUGE-1 F1 (Karn et al., 2021).

5. Quality control, target adaptation, and domain shift

A central theme is that synthetic data quality is not reducible to raw volume. Several papers explicitly reject naive union or naive generation. The OT interpolation work shows that pooling all source datasets can be substantially suboptimal, while target-aware generalized geodesics yield better transfer (Fan et al., 2023). BeyondWeb argues that there is “no single silver bullet,” and its gains come from jointly optimizing source quality, style, diversity, prompting, and filtering rather than from synthetic generation alone (Maini et al., 14 Aug 2025). SynPro operationalizes this with explicit quality, faithfulness, and data influence rewards, updating the generator whenever reference loss saturates so that synthetic data remains aligned with what the model has yet to absorb (Yu et al., 18 May 2026).

This emphasis on quality control produces elaborate filtering pipelines. BhashaKritika combines script and language detection, metadata consistency checks, heuristic thresholds on word count and repetition, KenLM perplexity filtering, and a FastText quality classifier (Manoj et al., 13 Nov 2025). SBP filters nearest-neighbor document pairs with similarity above 0.75, removes outputs with repeated 13-token shingles, and still reports non-factual content of 15.1% at 200B-scale and 8.7% at 1T-scale, making factuality a first-order concern rather than a side issue (Yang et al., 17 Sep 2025). OmniTab’s SQL2NL stage shows that generator confidence is a weak selector, whereas verification with a QA model against execution-derived answers is materially more effective (Jiang et al., 2022). Arctic-SnowCoder defines “high-quality” not as cleanliness alone, but as alignment with downstream application distributions, using a BERT-style quality annotator trained on high-quality code and instruction data (Wei et al., 2024).

Domain shift remains a major obstacle, particularly in multimodal settings. SynVL3D explicitly introduces synthetic-to-real domain adaptation with vision, language, and joint discriminators because synthetic scenes and text differ from real downstream data; the ablation shows that removing adaptation drops visual grounding from 67.5 to 53.1, dense captioning from 72.1 to 58.7, and QA from 27.6 to 23.7 (Yang et al., 2024). Reverse stylization in neural fractal pretraining is another response to the same problem: a small, license-free real-image set is used to lower the distributional gap measured by KID while preserving synthetic structure (Salehi et al., 24 Jun 2025). More generally, these results suggest that successful synthetic data pretraining often depends as much on adaptation and filtering as on the initial synthesis step.

6. Limitations, controversies, and open directions

The literature is explicit that synthetic data pretraining is not uniformly superior and not universally transferable. BeyondWeb states that there is “no silver bullet” and shows that naive continuation or single-style rewriting yields only modest gains despite large generation cost (Maini et al., 14 Aug 2025). SBP shows that pair irrelevance and hallucination remain serious problems, especially at smaller scale (Yang et al., 17 Sep 2025). The privacy-preserving encrypted-synthetic framework uses deterministic encryption in AES ECB mode, which preserves repeated entity identity but leaks equality and frequency patterns and therefore does not provide semantic security (Liu et al., 9 Jan 2026). In ECG pretraining, gains are largest in low-data settings and become small or negligible when real data are abundant (Naghashyar, 29 Jun 2025). In CCUP, synthetic pretraining is consistently helpful overall, but CCUP is not best on every metric in every setting; for example, ClonedPerson gives the best FIRe$$\mathcal{L}_{\Delta} = -\mathbb{E}_{x, y \sim p_{\Delta}} \left[ \log p_{\theta}(y | x) \right].$$8 result on VC-Clothes (Zhao et al., 2024).

A related controversy concerns what synthetic data is actually teaching. The analysis of simpler synthetic tasks reports that LIME attains an average of 67% of the benefits of natural pretraining, the generic Set function attains 65%, and parameter statistics alone attain 39%, suggesting that some benefits of pretraining may arise from generic sequence-processing or initialization effects rather than task-specific synthetic semantics (Wu et al., 2022). MIND, by contrast, argues that for mathematical reasoning, synthetic data must add complementary explanatory structure rather than merely paraphrase the source (Akter et al., 2024). These are not direct contradictions: a plausible implication is that the effective granularity of synthetic supervision is highly task dependent.

Several boundary conditions are already visible. Synthetic dialogues that help mathematical reasoning can be ineffective or harmful for programming benchmarks because they disrupt linear structure and remove critical formatting such as indentation (Akter et al., 2024). Synthetic image pretraining that matches or exceeds real-data baselines under full fine-tuning does not automatically dominate on frozen-backbone or object-centric natural-image tasks (Nakamura et al., 2024). BhashaKritika shows that native generation, translation, and prompt language must often be chosen per language rather than globally (Manoj et al., 13 Nov 2025). SynVL3D and ECG pretraining both show that synthetic pretraining can require dedicated downstream adaptation rather than being plug-and-play (Yang et al., 2024, Naghashyar, 29 Jun 2025).

The broad direction of the field is toward more task-aligned, target-aware, and quality-controlled synthesis. This includes continuous optimization over dataset space, faithfulness-constrained source transformations, hybrid self-supervised and supervised pretraining objectives, synthetic corpora with simulator-provided fine-grained annotations, and small real-data anchors used for style transfer or adaptation (Fan et al., 2023, Yu et al., 18 May 2026, Yang et al., 2024, Salehi et al., 24 Jun 2025). The common lesson is not that synthetic data can simply replace real data, but that synthetic pretraining becomes competitive when the synthesis mechanism exposes structure, preserves faithfulness where needed, and is explicitly matched to the transfer regime.