Pre-training from Scratch (PTS) Overview

• Pre-training from Scratch (PTS) is a family of training regimes that initializes models randomly and employs various strategies such as direct training, self-supervised pretraining, and curriculum methods.
• PTS variants demonstrate competitive performance in tasks like object detection, information retrieval, and speech-to-text by tailoring optimization and data engineering techniques.
• PTS redefines reliance on external priors by reconstructing them from target data, with success contingent on optimization scaffolding, data scale, and specific task demands.

Pre-training from Scratch (PTS) denotes a family of training regimes in which a model is initialized randomly rather than inherited from a generic checkpoint. In contemporary usage, however, the term does not refer to a single protocol. It can mean direct optimization of the downstream task from random weights, self-supervised pretraining on the target corpus followed by adaptation, curriculum-based staged pretraining before transfer, or target-data-only self-pretraining used to provide data-driven priors for fair architectural comparison. The literature therefore does not support a single verdict on PTS: in some settings it matches or exceeds conventional pretraining, whereas in others it incurs decisive penalties in convergence, generalization, or retention of general capability (He et al., 2018, Lassance et al., 2023, Peña et al., 7 Jul 2026).

1. Definitions and operational scope

The literature uses the label “PTS” in several non-identical ways.

Protocol in the literature	What is initialized from zero	Representative instantiations
Direct target-task training	Entire model trained on the downstream task from random initialization	COCO detection (He et al., 2018), visuo-motor Learning-from-Scratch baseline (Hansen et al., 2022), unsupervised source separation (Saijo et al., 2023)
Target-corpus pretraining	Model pretrained from random initialization on the collection of interest, then adapted	IR on MSMARCO, Mr. TyDi, and TripClick (Lassance et al., 2023), SE text adaptation (Peña et al., 7 Jul 2026), Turkish legal encoders (Uğur et al., 22 Jan 2026)
Curriculum or staged PTS	Model trained sequentially on simpler or auxiliary stages before transfer	Developmental PreTraining edge $\rightarrow$ shape $\rightarrow$ Imagenette (Rajesh et al., 2023), OpenBA UL2 $\rightarrow$ length-adaptation $\rightarrow$ Bilingual Flan (Li et al., 2023)
Target-data self-pretraining for evaluation	Same downstream data used for self-supervised pretraining to provide a data-driven initialization	Long Range Arena self pretraining (Amos et al., 2023)

This heterogeneity matters because claims about PTS are often conditional on what is being compared. In software-engineering language modeling, PTS is explicitly defined as training a LLM entirely from random initialization on a software-engineering corpus only, with the same causal objective as the base family (Peña et al., 7 Jul 2026). In information retrieval, PTS refers to MLM-style pretraining from random initialization on the target retrieval collection itself, followed by retrieval fine-tuning (Lassance et al., 2023). In object detection, by contrast, the key comparison is often between random initialization on the target task and ImageNet-based initialization (He et al., 2018).

A recurrent consequence is that “from scratch” is not synonymous with “without priors.” Some papers rebuild priors from target data through denoising, MLM, or staged curricula rather than importing them from external corpora. Others study whether such priors are unnecessary, or even misleading, for the target task.

2. Optimization strategies that make PTS viable

Successful PTS regimes are rarely naive random-initialization baselines. In detection, training from scratch becomes reliable only after specific optimization changes. “Rethinking ImageNet Pre-training” identifies proper normalization and longer schedules as the crucial adjustments: Group Normalization or SyncBN replaces frozen BN, and schedules extend from $2\times = 180k$ iterations to as long as $11\times$ while keeping the standard Mask R-CNN learning-rate recipe (He et al., 2018). “Rethinking Training from Scratch for Object Detection” further reconfigures the problem as target-dataset pretraining: a full detector is pretrained on low-resolution images from the detection dataset using larger batch sizes and ordinary BN, then fine-tuned at the usual high resolution; the paper reports that $3\times$ pre-training plus $1\times$ fine-tuning is generally sufficient (Li et al., 2021).

Speech and speech-adjacent domains required more explicit optimization scaffolding. In end-to-end speech-to-text translation, scratch training was improved by reducing BPE size, deepening the encoder to 12 layers while keeping a 6-layer decoder, increasing $d_{ff}$ to 4096, using post-LN, adding translation-based CTC regularization, and introducing parameterized distance penalty for locality-aware self-attention (Zhang et al., 2022). In multimodal speech recognition, MSRS treats sparsity as an optimization mechanism: it learns a binary mask through a continuous relaxation with distinct forward and backward temperatures, stabilizes that mask early—typically around 3 epochs—and then either restarts dense training from masked weights or continues with only nonzero weights (Fernandez-Lopez et al., 2024).

Other PTS variants rely on curricula rather than purely optimizer-level interventions. Developmental PreTraining uses a two-phase curriculum in which a network first learns edge detection on BIPEDv2 and then 9-class shape recognition on a 2D geometric shapes dataset before transfer to Imagenette (Rajesh et al., 2023). OpenBA uses a three-stage training strategy—UL2 pre-training, length-adaptation, and Bilingual Flan training—to train a 15B bilingual asymmetric seq2seq model from scratch on 380B tokens (Li et al., 2023). Mecellem’s Turkish legal encoders likewise show that checkpoint selection itself can be part of the PTS recipe: the models are trained from random initialization on 112.7B Turkish-dominant tokens, but the selected checkpoints are those with best downstream retrieval rather than lowest MLM loss (Uğur et al., 22 Jan 2026).

3. Empirical regimes in which PTS is competitive or superior

In vision, some of the strongest evidence for PTS comes from object detection. On COCO val2017, Mask R-CNN + FPN + GN trained from random initialization matched or slightly exceeded ImageNet-pretrained counterparts: ResNet-50 achieved 41.3 bbox AP versus 41.1, and ResNet-101 achieved 42.7 versus 42.3. A larger ResNeXt-152 configuration reached 50.9 bbox AP on COCO object detection without external data (He et al., 2018). Direct detection pre-training reported 38.2 bbox AP / 34.7 mask AP for ImageNet 1x, 41.5 / 37.0 for Direct(P3x) 1x, and 41.7 / 37.3 for Direct(P4x) 1x, while accelerating the pre-training phase by more than 11x (Li et al., 2021).

In information retrieval, target-only pretraining from random initialization is especially strong for first-stage ranking. On MSMARCO sparse retrieval, MLM+FLOPS 6L trained only on the target collection reached 0.382 MRR@10, compared with 0.373 for DistilBERT and 0.367 for BERT; on dense retrieval, MLM+FLOPS 12L reached 0.352 versus 0.342 and 0.347. The same study reports strong results on TripClick and Mr. TyDi, including target-collection pretraining competitive with SciBERT and PubMedBERT, and a scratch-pretrained SPLADE model beating reproduced MContriever on Arabic, Russian, and Japanese retrieval (Lassance et al., 2023). In Turkish legal NLP, Mecellem’s ModernBERT-based encoders trained from scratch on 112.7B tokens achieved top-3 Turkish retrieval leaderboard placements; the detailed extraction further reports rank 1 for the 403M retriever and rank 2 for the 155M model after post-training, with 92.36% production efficiency for the smaller system (Uğur et al., 22 Jan 2026).

Speech, translation, and control offer additional but more domain-contingent successes. “Revisiting End-to-End Speech-to-Text Translation From Scratch” reports an average BLEU of 25.1 on MuST-C, 22.7 BLEU on MuST-C En-De versus 22.9 for an ASR-pretrained comparison built on the same improved architecture, and 5.8 BLEU on Kosp2e Ko-En versus 5.9 for the benchmark paper’s pretraining-based result (Zhang et al., 2022). MSRS reports 21.1% WER for large visual speech recognition and 0.9% WER for audiovisual speech recognition on LRS3 while reducing training time by at least $2\times$ (Fernandez-Lopez et al., 2024). In unsupervised source separation, from-scratch RemixIT and Self-Remixing improved WSJ-mix SISDR from 8.8 for MixIT to 10.3 (Saijo et al., 2023). In visuo-motor control, a shallow ConvNet trained from scratch with random shift augmentation was competitive with frozen PVR, MVP, and R3M across 17 tasks in 4 domains, particularly at 100 demonstrations (Hansen et al., 2022).

4. Negative results, failure modes, and low-data exceptions

The strongest negative evidence against PTS appears in domain adaptation of LLMs. In software-engineering text adaptation, PTS means training from random initialization on SE text only under the standard causal LM objective. Under both constant-token and compute-matched budgets, continual pre-training usually performs better on domain and general benchmarks, and PTS becomes competitive only for small LMs under a token-rich budget (Peña et al., 7 Jul 2026). The paper’s overall recommendation is therefore to use CPT rather than PTS for most SE adaptation scenarios.

Curriculum-based PTS does not automatically transfer. Developmental PreTraining learned its internal tasks successfully—the edge-detection model reached a stable low value in about 10–15 epochs and the shape-recognition model reached near-perfect accuracy in about 10 epochs—but on Imagenette the DPT and vanilla randomly initialized models converged to similar training accuracy, DPT did not converge faster, and the authors conclude that the pre-trained weights may have held the model back (Rajesh et al., 2023). The likely mechanism proposed in the paper is overfitting to phase-specific data rather than acquisition of broadly transferable structure.

Even studies favorable to PTS identify hard low-data limits. On COCO subsets, random initialization remained comparable at 10k images with 25.9 AP versus 26.0 for pre-training, but failed at 1k images with 3.5 AP, or 5.4 AP after scratch-specific tuning, versus 9.9 AP for the pretrained model (He et al., 2018). Direct detection pre-training also did not beat ImageNet pre-training on PASCAL VOC (Li et al., 2021). In low-resource speech-to-intent learning, a pretrained ASR encoder substantially improved an NMF-based decoder, yet could worsen a capsule-network decoder on Patcor because the frozen ASR features did not preserve enough timing or word-order information for that architecture (Wang et al., 2021).

A related caution concerns evaluation methodology itself. “Never Train from Scratch” argues that random initialization plus supervised downstream training can grossly overestimate differences between long-sequence architectures. On Long Range Arena, a vanilla Transformer averaged 53.66 when trained from scratch, but 81.34 with masked self pretraining and 81.81 with causal self pretraining on the downstream data itself; with rotary embeddings and masked SPT, Transformers reached 86.21 versus 86.09 for S4 (Amos et al., 2023). In that setting, “scratch” is not treated as a neutral baseline but as a protocol that suppresses data-driven priors.

5. PTS as structure search, sparsity, and data engineering

A substantial branch of the literature treats PTS as more than a question of weight inheritance. “Pruning from Scratch” freezes randomly initialized weights, learns channel gates under sparsity regularization, thresholds them to satisfy a FLOPS budget, and then trains the resulting architecture from scratch. On ImageNet, the paper reports, for example, a ResNet-50 0.85x model with 76.7 top-1 at 3.0G FLOPS versus a 76.1 baseline at 4.1G FLOPS, and claims at least $\rightarrow$0 speedup on CIFAR-10 and at least $\rightarrow$1 on ImageNet for structure search (Wang et al., 2019). The central argument is that pre-training an over-parameterized model is unnecessary for finding a good pruned structure, and may even reduce structural diversity.

Small-language-model compression studies refine this point. “Small LLMs: Pruning vs. Training from Scratch” analyzes pruning as an initialization strategy rather than merely a compression method. Under equal training token budget, pruned initialization consistently outperformed random initialization at 50% pruning across six methods; however, when the scratch baseline was given the full 250B-token budget consumed by pretrain-prune-retrain, coarse structured pruning could be matched or surpassed, while unstructured sparse pruning often retained an advantage (Xu et al., 12 Jun 2026). The comparison thus depends strongly on whether fairness is defined by retraining tokens alone or by the total tokens consumed by the full pipeline.

Data augmentation can play a similar role to inherited weights. Thinking augmented Pre-Training concatenates each document with an automatically generated thinking trajectory and then applies standard next-token prediction; the paper reports a $\rightarrow$2 improvement in data efficiency and, for a 3B parameter model, post-training gains of over 10% on several challenging reasoning benchmarks (Wang et al., 24 Sep 2025). OpenBA’s staged UL2/length-adaptation/Bilingual Flan curriculum and Mecellem’s retrieval-aware checkpoint selection show the same general trend from a different angle: scratch pretraining increasingly depends on corpus design, stage ordering, and downstream-aware model selection rather than on random initialization alone (Li et al., 2023, Uğur et al., 22 Jan 2026).

6. Evaluation protocols, misconceptions, and research outlook

One persistent misconception is that PTS is a single baseline. The literature instead distinguishes pure downstream training, target-corpus self-supervised pretraining, target-dataset pretraining, and stage-wise curricula. Conclusions can reverse depending on whether the comparison is constant-token, compute-matched, or full-pipeline budget matched, and depending on whether the “scratch” baseline includes self-supervised reconstruction of the target data before supervision (Peña et al., 7 Jul 2026, Xu et al., 12 Jun 2026, Amos et al., 2023).

A second misconception is that external pretraining is either universally necessary or universally dispensable. The surveyed results do not support either extreme. Detection, first-stage IR, several speech-translation settings, and strong visuo-motor baselines show that in-domain data and careful optimization can close or reverse the gap to conventional pretraining (He et al., 2018, Lassance et al., 2023, Zhang et al., 2022, Hansen et al., 2022). Conversely, software-engineering adaptation, very low-data vision, and timing-sensitive speech tasks show that inherited priors or continual pretraining can remain decisive (Wang et al., 2021).

This suggests that the central scientific question in PTS is not simply whether random initialization can work, but which priors can be reconstructed from target data, under what optimization budget, and at what cost to generality. The surveyed work repeatedly points to four control variables: the scale and representativeness of the target corpus, optimization scaffolding such as normalization or learned sparsity, explicit accounting of total token or compute budget, and data-engineering mechanisms that make difficult examples more learnable (Li et al., 2021, Fernandez-Lopez et al., 2024, Wang et al., 24 Sep 2025). Under that interpretation, PTS is less a rejection of pretraining than a redefinition of where the prior comes from: external checkpoints, target data, structure search, or synthetic reasoning traces.