Pretraining in Machine Learning

• Pretraining involves optimizing a model on a preliminary objective, using the pretrained parameters as a starting point for fine-tuning on a specific task or domain.
• Key purposes include improving optimization, transferring representations, accelerating convergence, and enabling transfer across tasks, domains, and environments.
• Pretraining methods vary widely, including unsupervised, self-supervised, supervised, and knowledge-based techniques, each suited to different applications and data types.

Pretraining is the optimization of a model on a preliminary objective, dataset, or knowledge source before adaptation to a downstream task. The pretrained parameters serve as an initialization, representation, prior, policy, skill repertoire, or relational structure for subsequent fine-tuning or task-specific inference. Depending on the domain, pretraining may be unsupervised, self-supervised, supervised, weakly supervised, instruction-based, physics-informed, multimodal, or based on formal prior knowledge. Its purposes include improving optimization, regularizing solutions, transferring representations, reducing downstream data requirements, accelerating convergence, improving robustness, and enabling transfer across tasks, domains, modalities, and environments. These purposes are not interchangeable: pretraining can improve generalization while harming strongly mismatched tasks, and a lower pretraining loss does not necessarily imply better downstream transfer.

1. Conceptual foundations and historical development

The conventional deep-learning pipeline consists of pretraining a model on a source objective and subsequently fine-tuning it on a target objective. In its simplest form, a model with parameters $\theta$ is first optimized on a pretraining dataset $\mathcal{D}_{\mathrm{pre}}$ and then initialized from the resulting parameters for optimization on $\mathcal{D}_{\mathrm{down}}$. The source and target objectives may be identical in form, as in direct physics-prediction transfer for neural operators, or substantially different, as in masked language modeling followed by relation classification.

Early accounts of pretraining emphasized two mechanisms. The first is a regularization effect: pretraining places parameters in a region associated with a useful solution rather than an arbitrary solution reached from random initialization. The second is an optimization effect: the initialization may already be near a low-objective solution, making subsequent supervised optimization easier. An additional interpretation is that pretraining changes the geometry of hidden representations. In sigmoid networks, unsupervised layer-wise pretraining was associated with selectively active, moderately sparse hidden units and progressively greater class separation across layers (Li et al., 2013).

Layer-wise pretraining trains shallow components sequentially and composes them into a deep model. Restricted Boltzmann machine pretraining and autoencoder pretraining follow this pattern: the first layer is trained on the input, its representation becomes the input for the next layer, and the complete network is later fine-tuned. In an RBM, visible and hidden units are connected bipartitely, with conditional hidden activation

$p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$

where $f$ is the sigmoid function. The resulting encoder-decoder structure can be interpreted as adaptive sparse coding. Unlike conventional sparse coding, which explicitly penalizes hidden activations with an $L_1$ term, RBM pretraining learns feature detectors that respond to present features and suppress absent or weakly relevant features. The resulting representation is not maximally sparse everywhere: lower layers preserve overlapping shared structure, whereas higher layers become more selective and class-separable.

The scope of pretraining subsequently expanded beyond models with clean layer decompositions. Knowledge-transfer pretraining trains an entire complex student model against soft outputs produced by a simpler teacher, then fine-tunes the student with hard labels. For teacher logits $z_i$ and temperature $T$, the teacher distribution is

$p_i=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)}.$

The student minimizes cross-entropy with this distribution rather than with a one-hot target. This approach was designed for recurrent models whose hidden-to-hidden and cross-layer connections make layer-wise decomposition difficult. In speech recognition, a DNN teacher pretraining an LSTM student improved the reported two-layer LSTM WER from $12.34\%$ under direct training to $\mathcal{D}_{\mathrm{pre}}$0 after transfer pretraining and hard-label fine-tuning (Tang et al., 2015).

More recent approaches treat pretraining itself as a learning-to-learn problem. Rather than optimizing only a proxy task, meta-learning pretraining optimizes the performance of an initialization after several adaptation steps. With inner-loop updates $\mathcal{D}_{\mathrm{pre}}$1, the objective is of the form

$\mathcal{D}_{\mathrm{pre}}$2

Ordinary multi-task pretraining is recovered when the meta-train depth is zero. In text representation experiments, moderate meta-train depths improved downstream performance and early fine-tuning behavior relative to BERT-style pretraining, although the principal Transformer experiments initialized from official BERT parameters (Lv et al., 2020).

2. Pretraining objectives and mechanisms

Pretraining objectives determine which information a model is encouraged to preserve, compress, predict, or organize. The central distinction is between objectives that directly approximate the downstream task and proxy objectives whose relation to downstream performance is indirect.

Reconstruction and masked prediction

Masked language modeling predicts tokens hidden from a corrupted sequence:

$\mathcal{D}_{\mathrm{pre}}$3

Masked image modeling applies an analogous principle to image patches. In MAE, $\mathcal{D}_{\mathrm{pre}}$4 of image patches are masked, the encoder processes only the visible $\mathcal{D}_{\mathrm{pre}}$5, and a decoder reconstructs the masked content. In tabular learning, reconstruction recovers original feature values from corrupted inputs, while mask prediction identifies which columns were corrupted. These objectives are self-supervised because targets are derived from the input rather than from human labels.

Masked prediction can induce structure that is not explicitly specified by the loss. Analysis of BERT-6 found that accuracy per token increases with token frequency and Transformer depth. Token confusions become concentrated within finite groups of strong substitutes, including directions, months, ordinal numbers, and semantically related verbs. This organization becomes sharper toward the output layer, while downstream FewRel and DBpedia classification accuracy and signal-to-noise ratio increase across layers (Tzach et al., 3 Sep 2025).

Contrastive and relational objectives

Contrastive pretraining brings representations of related views or states together and separates unrelated examples. In visuo-tactile pretraining, visual and tactile encoders map observations from the same timestep to nearby embeddings while using temporally separated observations as negatives. The resulting visual encoder improved a vision-only USB-plugging ACT policy from $\mathcal{D}_{\mathrm{pre}}$6 to $\mathcal{D}_{\mathrm{pre}}$7 success, despite tactile observations being discarded during policy training and deployment (George et al., 2024).

Graph and hypergraph pretraining can incorporate task descriptions into relational propagation. Instruction-based Hypergraph Pretraining uses natural-language instructions to specify the semantic purpose of high-order relations. Its Prompt Hypergraph Convolution first aggregates nodes into hyperedges, fuses the aggregate with an instruction embedding, and propagates the prompted relation back to nodes:

$\mathcal{D}_{\mathrm{pre}}$8

The method is designed to reduce the discrepancy between pretraining and downstream tasks and to handle unseen context nodes (Yang et al., 2024).

Task-aware and target-aware objectives

When target labels are available, using them during pretraining can be beneficial. In tabular learning, supervised loss on corrupted inputs, reconstruction or mask prediction combined with supervised loss, and target-conditioned corruption were compared across eleven datasets and three neural architectures. The most stable approach was reconstruction plus supervised loss, while target-conditioned mask prediction significantly improved over ordinary mask prediction. Pretraining gains were largest for vanilla MLPs, although embedding-based architectures also benefited. Target-aware pretrained MLPs surpassed GBDTs on Gesture, California Housing, Otto, and Higgs in the reported comparisons (Rubachev et al., 2022).

In commonsense reasoning, a pretrained masked-language-model head can be reused directly rather than replaced by a randomly initialized classifier. The Sequence Scoring Method converts premise–hypothesis pairs into full text and ranks candidates according to the probability assigned to masked premise tokens. On COPA, four-gram premise scoring achieved $\mathcal{D}_{\mathrm{pre}}$9 zero-shot accuracy, and fine-tuned scoring achieved $\mathcal{D}_{\mathrm{down}}$0 accuracy with a substantially lower variation across random restarts than a randomly initialized classifier (Tamborrino et al., 2020).

Physics-aware and domain-specific objectives

For speech translation, curriculum pretraining progresses from ASR transcription to frame-based masked language modeling and bilingual lexicon translation. The lower encoder layers are trained for source-language transcription and understanding, while upper layers learn source-to-target lexical mapping. Removing the ASR phase reduced expanded LibriSpeech BLEU from $\mathcal{D}_{\mathrm{down}}$1 to $\mathcal{D}_{\mathrm{down}}$2, and jointly training all objectives in one phase produced $\mathcal{D}_{\mathrm{down}}$3 BLEU, supporting sequential rather than simultaneous training (Wang et al., 2020).

For neural operators, direct future-state prediction generally transferred more reliably than vision-inspired temporal classification, jigsaw, or patch-order objectives. Derivative prediction was especially useful for short-horizon autoregression, while physics-informed contrastive learning was competitive in selected architecture–PDE combinations. The study concluded that objective–architecture matching and physics-preserving augmentation were more important than importing generic computer-vision objectives unchanged (Zhou et al., 2024).

3. Data, domain alignment, and the pretraining distribution

Pretraining quality depends not only on the objective but also on the source data distribution. Dataset size, annotation quality, modality alignment, linguistic function, visual content, and target similarity can interact in ways that make raw scale an unreliable proxy for transfer value.

In visio-linguistic pretraining, COCO Captions, VQA-P, and Conceptual Captions were compared across VQA, VizWiz, SNLI-VE, and MM-IMDB. COCO contains approximately 200,000 images with five human-written captions per image; VQA-P uses the same COCO visual domain but question–answer language; Conceptual Captions contains approximately 3.1 million available web-mined image–caption pairs with noisier text and a different visual distribution. A smaller, cleaner, or more task-relevant dataset often outperformed the much larger Conceptual Captions corpus. VQA-P was particularly effective for question-oriented transfer, while COCO was strong for visually similar tasks. Pretraining harmed MM-IMDB, a strongly mismatched movie-poster and plot task. These results show that pretraining does not work automatically across domains (Singh et al., 2020).

Synthetic annotation can improve the usefulness of a large but mismatched image collection. In the CCG experiment, captions generated by a COCO-trained captioning model replaced raw Conceptual Captions alt-text. CCG improved VQA-D performance over raw Conceptual Captions for both VisualBERT and ViLBERT, although it did not dominate on every downstream task. The result illustrates that annotation style and image–text alignment can matter as much as corpus size.

Conditional pretraining makes this dependence explicit by selecting a target-relevant subset of a larger source dataset. A target-representation method clusters target features and retains source images near target clusters. A domain-classifier method trains a binary classifier to distinguish target images from source images, then retains source images with high target-domain probability. On ImageNet-derived subsets, filtering improved supervised transfer by up to approximately $\mathcal{D}_{\mathrm{down}}$4 percentage points and unsupervised transfer by up to approximately $\mathcal{D}_{\mathrm{down}}$5 percentage points over random selection. In object detection and segmentation, gains over random subset selection reached approximately $\mathcal{D}_{\mathrm{down}}$6. Lowering source-pretraining resolution from $\mathcal{D}_{\mathrm{down}}$7 to $\mathcal{D}_{\mathrm{down}}$8 reduced training cost by roughly $\mathcal{D}_{\mathrm{down}}$9–$p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$0 while generally preserving transfer performance (Chakraborty et al., 2020).

The same principle applies to large-scale weakly supervised vision training. MAE pre-pretraining uses label-free masked autoencoding before weakly supervised hashtag-based pretraining on billions of images. MAE initialization improved weakly supervised transfer across model sizes from $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$1 million to $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$2 billion parameters. A $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$3-billion-parameter MAWS model outperformed a $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$4-billion-parameter WSP-only model in the reported linear-probe comparison, and the effect persisted after billions of weakly supervised image samples (Singh et al., 2023).

In reinforcement learning, the source distribution consists of trajectories rather than independent examples. The usefulness of pretraining data depends on behavior policies, environment dynamics, exploration coverage, action semantics, and controllability. Curiosity-driven pretraining may focus on irreducible stochasticity, skill discovery may produce distinguishable but useless behaviors, and offline skill extraction is bounded by the quality and support of the available trajectories (Xie et al., 2022).

4. Transfer mechanisms across domains and modalities

Pretraining transfers different objects depending on the learning problem. These include hidden representations, initialization geometry, output heads, Gaussian-process priors, policies, skills, world models, node embeddings, and physical operators.

Representational and initialization transfer

In ordinary supervised learning, the pretrained backbone is commonly retained while a task-specific head is initialized or trained. Frozen-base evaluations distinguish transferable representations from improvements attributable mainly to end-to-end optimization. In multimodal learning, VQA-P produced especially strong frozen VisualBERT representations on VQA-D, VizWiz, and SNLI-VE, while end-to-end fine-tuning sometimes favored other source datasets. Thus, the checkpoint with the lowest pretraining validation loss need not have the best frozen representation.

In continual pretraining, the model is updated sequentially using self-supervised objectives and later adapted to downstream tasks. MLM-based continual pretraining on arXiv abstracts caused little measurable forgetting on tweet sentiment and QNLI, whereas supervised ImageNet classification pretraining on sequential iNaturalist experiences produced substantial CORe50 degradation. BEiT, trained with masked image modeling, showed substantially less forgetting than supervised ResNet and ViT models. The evidence associates self-supervised objectives with more stable representations, although the experiments do not isolate every confounding variable (Cossu et al., 2022).

Policies, skills, and decision-making priors

Reinforcement-learning pretraining can transfer exploratory policies, skills, behavior priors, successor features, world models, or visual and language representations. Skill discovery introduces a latent variable $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$5 and trains a policy $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$6 to produce distinguishable behavior, often by maximizing a mutual-information objective. Offline skill extraction compresses trajectory segments into latent temporally extended actions, allowing a downstream high-level policy to select skills rather than primitive actions. Successor features separate dynamics-related features from reward weights, supporting rapid adaptation when reward functions vary but environment dynamics remain shared.

Pretraining can also use privileged modalities. In visuo-tactile imitation learning, tactile data shaped a visual encoder during pretraining and were then removed. This improved vision-only policies because the visual representation acquired features correlated with contact and insertion states. However, tactile-only policies did not uniformly benefit, and task-specific data, fixed initial positions, sensor wear, and tactile-pad mismatch limited generalization (George et al., 2024).

Priors over functions and optimization problems

Pretraining need not produce a neural representation. HyperBO pretrains a Gaussian-process prior from observations of many related optimization functions. The learned mean, covariance, and noise parameters are then held fixed during target Bayesian optimization. Its negative-log-likelihood and KL-based variants were evaluated on 23 hyperparameter-tuning tasks. On more than $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$7 of tasks, the KL variant was at least $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$8 times faster than the best non-HyperBO alternative in terms of function evaluations, while the NLL variant was at least $p(h_j=1\mid x)=f\left(\sum_i W_{ji}x_i+b_j\right),$9 times faster on more than $f$0 of tasks (Wang et al., 2022).

In feedback-alignment networks, pretraining can operate without meaningful data. Random Gaussian inputs and independent random labels modify forward weights so that they become aligned with fixed random feedback matrices. This reduced alignment angles, accelerated subsequent learning, lowered effective rank, reduced generalization gaps, and improved transformed-MNIST and USPS performance in the reported small networks. The mechanism is structural rather than semantic: random-noise pretraining prepares credit assignment and representation geometry rather than learning task content (Cheon et al., 2024).

Formal prior knowledge can also be converted into knowledge prototypes. Informed pretraining trains a model on a small set of examples generated from graphs, templates, geometric structures, simulations, or equations before fine-tuning on real data. Experiments on GTSRB and MNIST reported improvements of up to approximately $f$1 percentage points in small-data settings and approximately $f$2 percentage points in the smallest MNIST-to-USPS robustness setting. Layer-transfer analysis attributed the strongest prototype-transfer effects to mid-to-late layers, in contrast to ImageNet pretraining, whose largest effects were in earlier layers (Rueden et al., 2022).

5. Evaluation, benefits, and limitations

The effects of pretraining are best evaluated through multiple criteria rather than a single downstream score. Relevant measures include final accuracy or error, convergence speed, early adaptation, data efficiency, robustness, frozen-feature transfer, low-shot performance, zero-shot performance, forgetting, and computational cost.

Pretraining can improve optimization even when final training-set accuracy is not higher. Knowledge-transfer pretraining improved speech-recognition WER and cross-validation accuracy without necessarily maximizing training frame accuracy. Meta-learning pretraining improved early fine-tuning curves, consistent with an initialization designed to adapt rapidly rather than merely minimize a proxy loss. In tabular learning, clean-data fine-tuning after corrupted-input pretraining was essential; omitting it often degraded performance.

Pretraining can also improve robustness and low-data behavior. MAWS combined MAE’s spatial structure with weakly supervised semantic information and improved fully fine-tuned, linear-probe, low-shot, zero-shot, video, detection, and segmentation results. In neural operators, gains were largest with scarce downstream data and when downstream PDE distributions resembled the pretraining distribution. In RL, the relevant metric is often area under the learning curve or interaction efficiency rather than only final return.

However, high pretraining performance does not guarantee useful transfer. Lower masked-image-modeling loss on Conceptual Captions did not imply better multimodal downstream performance. In language, self-pretraining on downstream inputs sometimes matched or exceeded BookWiki pretraining, but BookWiki remained better on MNLI, Jigsaw Toxicity, and Discovery. In commonsense benchmarks, high scores on SWAG and HellaSwag were partly explained by hypothesis-only artifacts. In neural operators, vision-inspired objectives could substantially harm forecasting despite successful pretraining-task optimization.

Negative transfer arises when the source objective or distribution imposes an inappropriate inductive bias. Strongly mismatched multimodal pretraining hurt MM-IMDB. Conditional data filtering can discard generic diversity if the future target distribution is unknown. Self-supervised RL representations may preserve visually salient but control-irrelevant variables. Pretrained policies may bias exploration toward unsuitable behaviors, and frozen representations may underfit when downstream dynamics or action spaces differ.

Computational cost is another limitation. Meta-learning pretraining requires approximately $f$3 batches per outer update and becomes substantially more expensive as meta-train depth increases. Tabular pretraining can require roughly an order of magnitude more computation than training from scratch. MAE pre-pretraining adds an additional stage, although matched-compute experiments reported up to approximately $f$4 FLOP efficiency gains. Neural-operator pretraining is not free and must be amortized across downstream tasks. Filtering reduces the number of source examples, while lower resolution reduces computation per example, but filtering itself incurs representation-learning or domain-classifier costs.

Evaluation can also conflate representation retention with optimization ease. Continual-pretraining studies use forgetting-control datasets and subsequent fine-tuning, so measured retention reflects both information preserved in the representation and the ease with which it can be reactivated. Frozen-base evaluation is more restrictive but can underestimate useful knowledge that becomes accessible after adaptation.

6. Contemporary directions and open problems

Several research directions extend the pretraining paradigm beyond conventional source-to-target fine-tuning.

Objective alignment remains central. Meta-learning pretraining explicitly optimizes post-adaptation performance; speech-translation curricula organize objectives by transcription, source-language understanding, and bilingual mapping; neural-operator studies favor direct physics prediction; graph pretraining injects textual task instructions into message passing; and tabular work uses target-aware corruption. These approaches suggest that the relationship between pretraining and downstream objectives should be designed rather than assumed.

Data selection and mixture design are increasingly important. Domain similarity, annotation quality, task structure, and controllability can matter more than raw data volume. Conditional source filtering, synthetic captions, VQA-style question data, and sequential conditional pretraining provide mechanisms for concentrating useful information while reducing cost. Open questions include how to quantify source–target similarity, how to avoid losing broad robustness, and how to adapt data mixtures as downstream tasks change.

Continual and lifelong pretraining seeks to update models with new language, visual domains, skills, or environments while retaining prior competence. Self-supervised continual objectives appear more resistant to forgetting than supervised task objectives in the reported language and vision experiments, but order dependence, contradictory information, long streams, and realistic task-free online settings remain unresolved.

Multimodal and privileged-sensor pretraining can transfer information across modalities even when one modality is absent at inference. Visuo-tactile pretraining demonstrates task-specific privileged information transfer; image-only MAE initialization improves later weakly supervised semantic training; and RL systems combine visual, language, trajectory, action, and reward tokens. The principal challenges are grounding, modality alignment, data heterogeneity, and interference between unrelated objectives.

Scientific and structured-data pretraining requires objectives that preserve domain laws and high-order relations. Physics-aware neural-operator objectives, informed knowledge prototypes, hypergraph instructions, and successor features all encode structure beyond generic feature reconstruction. Future systems may combine direct prediction, derivative constraints, symmetry-preserving augmentation, formal knowledge, and instruction-conditioned relational propagation.

Mechanistic understanding remains incomplete. Studies of sparse hidden activations, token-confusion clusters, effective rank, alignment geometry, CKA, and layer-specific transfer provide partial explanations for why pretraining works. They suggest that pretraining can organize representations, improve credit assignment, reduce effective complexity, and create task-relevant priors. None establishes a universal mechanism. The observed benefits depend on architecture, objective, data distribution, downstream adaptation, and evaluation protocol.

The general principle is therefore conditional rather than universal: pretraining is effective when the upstream process produces information, geometry, or behavior aligned with the downstream bottleneck. It may transfer low-level features, semantic representations, task-aware initialization, structured priors, policies, skills, or physical dynamics. Its success should be assessed not by pretraining loss alone, nor by raw dataset scale, but by downstream adaptation, transfer under distribution shift, computational efficiency, robustness, and retention of useful knowledge.