Pretrain-then-Finetune Paradigm

• The pretrain-then-finetune paradigm is a sequential machine learning strategy that first learns broad representations from large-scale data and later adapts to specialized tasks.
• It employs techniques such as parameter-efficient tuning, structured adaptation, and knowledge distillation to optimize performance and reduce computational demands.
• Its efficacy is demonstrated by improved data efficiency, enhanced robustness in low-data regimes, and superior generalization across diverse application domains.

The pretrain-then-finetune paradigm denotes a sequential machine learning strategy wherein a model is first trained on a large, typically generic or multi-domain dataset (“pretraining”), acquiring broad and general representations, and is subsequently adapted (“finetuned”) on a smaller, target-specific dataset to optimize performance for a downstream task. This approach underpins state-of-the-art methodologies in natural language processing, vision, recommendation, biomedical question answering, code understanding, structured retrieval, clinical predictions, and large-scale decision-making. Central to its efficacy is the construction of robust, general-purpose feature extractors during pretraining, followed by parameter or architectural adaptation tailored to downstream data and objectives.

1. Conceptual Foundations

In the pretrain-then-finetune workflow, pretraining leverages massive, often self-supervised, corpora to endow models with general knowledge. For example, in BERT-style masked language modeling (MLM), the objective is to learn parameters $\theta$ minimizing

$$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$

where $M$ denotes masked token indices (Wang et al., 2024).

Subsequent finetuning adapts all or a subset of parameters to the supervised target task via

$$L_{\mathrm{fine}}(\theta) = \mathbb{E}_{(x', y') \sim D_{\mathrm{fine}}}[\ell_{\mathrm{task}}(f(\theta; x'), y')]$$

with $\ell_{\mathrm{task}}$ denoting the task-specific loss, e.g., cross-entropy for classification (Wang et al., 2024, Wang et al., 2021). Separation of objectives ensures that knowledge extracted from large generic data (language, visual, or cross-modal) is harnessed in low-resource or specialized settings.

The formalism holds across architectures: Transformers (Wang et al., 2024, Xu et al., 2021), graph neural networks (Xu et al., 2024), dual-encoders (You et al., 19 Sep 2025), and domain-specific variants for code, EHR, or recommender systems (Wang et al., 2021, Liu et al., 2024).

2. Methodological Implementation Patterns

a. Model, Loss, and Optimization

Models typically comprise a shared backbone $g_\theta$ and a small task-specific head $h_\phi$ as $f(\theta; x) = h_\phi \circ g_\theta(x)$ (Xu et al., 2021). Pretraining adjusts $\theta$ using unsupervised or weakly supervised tasks such as MLM, next-sentence prediction, contrastive objectives, or item prediction in recommendation (Wang et al., 2024, Liu et al., 2024, Yuan et al., 2023). During finetuning, either the entire model or only certain layers/heads are updated on target supervision.

A crucial hyperparameter is the learning rate schedule: large and decaying during pretraining, then sharply reduced for finetuning, to avoid “catastrophic forgetting” of pretraining knowledge (Wang et al., 2024, Wang et al., 2021).

Variants include:

• Partial adaptation (“feature extraction”): backbone frozen, only head trained (Peters et al., 2019).
• Parameter-efficient tuning (adapters, prompt-tuning, BitFit, LoRA): only a small set of new or sidechain parameters updated during finetuning (Yang et al., 2022, Liu et al., 2024).
• Structured adaptation: adaptation through prompts, task-specific sub-networks, graph-specific components, or group-balanced reweighting (Liu et al., 2024, Xu et al., 2024).

b. Knowledge Distillation and Compression

Compression via pruning, sparse masks, or knowledge distillation is integrated into the pretrain-then-finetune workflow for deployment efficiency. For example, SparseBERT combines sparse pruning with task/representation-level distillation, enforcing the student to mimic both general and task knowledge via loss terms:

$$L_{\mathrm{distil}}(\theta, m) = L_{\mathrm{emb}} + L_{\mathrm{att}} + L_{\mathrm{hid}} + L_{\mathrm{prd}}$$

and iterative pruning (Xu et al., 2021). Similarly, progressive distillation (SPD) uses gradual module grafting and multi-level KD to avoid overfitting in highly sparse models (Huang et al., 2021).

3. Analytical Theory and Empirical Phenomena

Theoretical analyses decompose the generalization error in pretrain-then-finetune as a competition between domain-gap (mismatch between pretrain prior and task), sample efficiency (scaling with number of task instances $$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$0), and model approximation properties (Zhang et al., 3 Feb 2026, Liu et al., 2021). For instance, in multi-task decision settings, the excess estimation error is bounded as:

$$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$1

where $$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$2 (finetune generalization) decreases as $$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$3, and $$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$4 (pretraining bias) is controlled by prior alignment (Zhang et al., 3 Feb 2026).

Analytic theory for linear diagonal networks demonstrates that careful choice of initialization scale determines whether the model operates in feature-reuse, feature-refinement, or mixed regimes, with substantial impact on downstream sample efficiency (Anguita et al., 23 Feb 2026).

Empirically, pretrain-then-finetune:

• Provides strong gains in data efficiency, especially with limited labeled data (Wang et al., 2024, Wang et al., 2021, Lv et al., 19 May 2025).
• Supports rapid learning in reinforcement learning, particularly when self-reference or memory-augmented mechanisms are added (Zhao et al., 2023).
• Yields robustness and stability, especially in low-data regimes and for complex compositional tasks (Tamborrino et al., 2020).

4. Variants and Extensions

Multiple recent works introduce structured or domain-adapted extensions:

• Prompt-then-finetune: introduces shared and private prompts, freezing pretrained prompts while adapting only task-specific ones, shown to improve stability and accuracy in cross-domain recommendation (Liu et al., 2024).
• Retrieval-augmented: fuses representations with retrieved external knowledge (e.g., biomedical image-text pairs) via attention-based modules, enhancing performance in knowledge-intensive domains (Yuan et al., 2023).
• Group-balanced reweighting and smoothness-inducing regularization: in clinical prediction, these strategies maintain balanced accuracy across subgroups and prevent catastrophic forgetting of representations learned on “basic” feature sets (Xu et al., 2024).
• Hybridization with semi-supervised learning: recent analyses show that few-shot finetuning of pretrained vision-LLMs can outperform classical SSL, except in low-resolution or semantically-coarse data regimes (Lv et al., 19 May 2025).

Furthermore, in code understanding, pretrain-then-finetune workflows are enhanced via semantic-preserving transformations and curricular pacing of augmented data (Wang et al., 2021).

5. Practical Protocols and Empirical Results

Optimal pretrain-then-finetune practice depends on factors such as task similarity, data scale, and computational constraints. For BERT and other Transformer models, recommendations include (Peters et al., 2019, Wang et al., 2024):

• Use end-to-end finetuning (unfreeze all layers) for sentence-pair or similar downstream tasks.
• When data is extremely limited or mismatch is strong, use feature extraction with a large task-specific head.
• Perform parameter-efficient head pretraining (“EH-FT”) with techniques such as LoRA or Prefix-tuning before full finetuning to improve stability and convergence (Yang et al., 2022).
• Integrate knowledge distillation and progressive pruning during finetuning for aggressive compression without excessive overfitting (Xu et al., 2021, Huang et al., 2021).

Quantitative summaries:

• SparseBERT compresses BERT by $$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$5 (in parameters and FLOPs) with only $$L_{\mathrm{pre}}(\theta) = \mathbb{E}_{x \sim D_{\mathrm{pre}}}\left[-\sum_{i \in M} \log P_\theta(x_i | x_{\setminus M})\right]$$61–2% accuracy drop on GLUE (Xu et al., 2021).
• Retrieval-augmented pretrain-finetune (RAMM) gains up to 1.9 percentage points in biomedical VQA and outperforms prior state-of-the-art (Yuan et al., 2023).
• Pretrained VLMs (with prompt-tuning) achieve higher accuracy than SSL methods when labels are scarce, except for low-res and non-semantic datasets (Lv et al., 19 May 2025).
• Layerwise fine-tuning and preservation of pretrained classification layers yield 3–6 point gains for transfer in vision (Shermin et al., 2019).

6. Domain-Specific and Security Implications

The paradigm is highly adaptable but presents challenges:

• In LLMs, “pretrain-to-finetune” establishes a persistent inheritance of adversarial vulnerabilities. Adversarial suffixes crafted on pretrained models remain effective post-finetuning, with transferability encoded linearly in the hidden representations; advanced attacks (PGP) exploit this for highly effective black-box jailbreaks (Tan et al., 14 Dec 2025).
• In unsupervised RL, pretraining with self-reference modules stabilizes intrinsic reward and prevents catastrophic forgetting, achieving state-of-the-art sample efficiency (Zhao et al., 2023).
• For structured retrieval (hierarchical IR), decoupled pretraining on short-distance pairs plus finetuning on hard, long-distance pairs dramatically increases long-range recall (from 19% to 76% at distance 8 in WordNet) (You et al., 19 Sep 2025).
• In clinical and multi-modal settings, specialized architectures for handling heterogeneity and external knowledge are crucial—pretrain-then-finetune with hypergraph transformers and group-regularization preserves subgroup performance and prevents forgetting (Xu et al., 2024).

7. Extensions, Limitations, and Open Directions

The pretrain-then-finetune paradigm is not without limitations:

• Where sufficient labeled data are available, the marginal benefit may be reduced, and training from scratch approaches may close the gap (Liu et al., 2021).
• Effective deployment demands alignment between pretraining data distributions and downstream task domains to minimize domain-gap–induced generalization error (Zhang et al., 3 Feb 2026).
• Safety risks are heightened when pretrained models are publicly released, due to vulnerability inheritance in downstream finetuned versions (Tan et al., 14 Dec 2025).

Open challenges include formal characterizations of optimal adaptation hyperparameters, automated data/parameter selection for hybrid or multi-task settings, and development of robust safety-focused finetuning methods.

---

Key Papers Referenced

Domain/Problem	Work	arXiv id
NLP / Pruning	Rethinking Network Pruning	(Xu et al., 2021)
Cross-domain Recommendation	MCRPL	(Liu et al., 2024)
Code Understanding	Bridging Pre-trained Models	(Wang et al., 2021)
Inductive Transfer (NLP)	To Tune or Not to Tune?	(Peters et al., 2019)
Learning Mechanism NLP	Learning Mechanism Underlying NLP Pre-Training	(Tzach et al., 3 Sep 2025)
Efficient Head Tuning	Parameter-Efficient Tuning	(Yang et al., 2022)
Generalization Theory (Vision)	Improved Fine-Tuning …	(Liu et al., 2021)
Vision Transfer/Classifier	Enhanced Transfer Learning ImageNet Class. Layer	(Shermin et al., 2019)
RL / Self-reference	Augmenting Unsupervised RL with Self-Reference	(Zhao et al., 2023)
Progressive Pruning	Sparse Progressive Distillation	(Huang et al., 2021)
Clinical/EHR	Hypergraph Transformer PT-FT	(Xu et al., 2024)
LLM Security/Jailbreak	One Leak Away	(Tan et al., 14 Dec 2025)
SSL vs Pretrain-Finetune	Unlabeled Data or Pre-trained Model	(Lv et al., 19 May 2025)
Hierarchical Retrieval	Hierarchical Retrieval: Geometry & PT-FT	(You et al., 19 Sep 2025)
Biomedical VQA	RAMM: Retrieval-Augmented Pretrain-Finetune	(Yuan et al., 2023)
Decision Optimization	LLM-Inspired Pretrain-Then-Finetune	(Zhang et al., 3 Feb 2026)
Theory – Inductive Bias	Theory of Pretraining Shaping Bias	(Anguita et al., 23 Feb 2026)
Commonsense / Headless FT	Pre-training Is (Almost) All You Need	(Tamborrino et al., 2020)

These works collectively elaborate the principles, analytic underpinnings, practical recipes, and evolving landscape of the pretrain-then-finetune paradigm across domains.