Two-Stage Pre-Training Regime

• Two-stage pre-training regime is a sequential learning strategy that first builds general representations and then refines models for specific domains.
• It leverages diverse objectives such as masked language modeling, denoising autoencoding, and contrastive learning to transfer and adapt knowledge.
• Empirical studies show up to 27% compute savings and enhanced performance in language, vision, and multi-modal applications.

A two-stage pre-training regime is a sequential learning strategy in which a model, or its constitutive modules, are pre-trained in two distinct phases, often with different objectives, data modalities, or structural constraints. This approach has emerged as a dominant paradigm across language, vision, and audio domains, combining disparate inductive biases or exploiting curriculum-style knowledge transfer for improved task transfer, efficiency, or robustness.

1. Definitions and Rationale

A two-stage pre-training regime first pre-trains a portion or all of a model using a task- or modality-specific objective, often on large-scale or weakly labeled data, to imbue the network with general-purpose representations. In the second stage, the pre-trained parameters are either transferred, reused, or further adapted in a new architecture or under a new training objective targeting the final application domain. This regime leverages the complementary strengths of different objectives (e.g., masked language modeling for structure, sequence denoising for generation), mitigates catastrophic forgetting by staged unfreezing or adapter-based freezing, and, in many empirical studies, results in both improved compute efficiency and superior transfer performance compared to single-stage or end-to-end pre-training.

The regime is instantiated not only in classic encoder-decoder architectures—pre-training encoders before or in conjunction with decoders—but also in multi-modal, distillation, self-supervised, and topology-aware frameworks.

2. Prototypical Methodologies

Several archetypal two-stage pre-training recipes have been validated:

2.1 Sequential Encoder-Decoder Pre-training

In multilingual sequence modeling, a common recipe is:

1. Pre-train an encoder with masked language modeling (MLM), e.g., RoBERTa-style [MASK]-token prediction over a multilingual corpus.
2. Attach a decoder and continue pre-training as a sequence-to-sequence (seq2seq) denoising autoencoder, freezing the encoder in early updates and unfreezing it later to permit cross-attention adaptation (Soltan et al., 2023).

This regime achieves the same generalization as from-scratch training at 27% lower compute and is robust both for discriminative (token classification, NER) and generative (semantic parsing, summarization) tasks.

2.2 Multi-Modal and Curriculum Pre-training

In multi-modal domains, two-stage pre-training is leveraged to align representations between modalities and then specialize them:

• Stage 1: Contrastive learning (e.g., InfoNCE) is used to align patch-level representations across RGB-D or image–text pairs (Jamal et al., 2024, Chen et al., 2023).
• Stage 2: Masked autoencoding, denoising, or video-text reasoning further adapts these representations to dense prediction or temporal tasks.

For example, VideoOFA first pre-trains on massive image–text pairs casting multiple vision–language tasks as seq2seq, then adapts to video using captioning, video-text matching, and frame order modeling (Chen et al., 2023).

2.3 Progressive Distillation

Student-teacher distillation frameworks exploit two-stage regimes to bridge capacity gaps:

• Stage 1: Distill from a large teacher to an intermediate “teacher assistant” (TA) model via hidden state and logit matching.
• Stage 2: Distill from the TA to a narrow-and-deep student, again using the same objectives (Yao et al., 2023).

This bridges representation mismatches and improves downstream accuracy and efficiency versus direct teacher→student distillation.

2.4 Unsupervised and Self-supervised Extensions

In speech, unsupervised two-stage pre-training methods separately exploit unpaired speech and text:

• Stage 1: Acoustic pre-training by masked-feature prediction on raw speech signals (MSE on masked chunks).
• Stage 2: Linguistic pre-training by synthesizing paired data (TTS) and cross-entropy training of the encoder-decoder (Fan et al., 2019).

This integrates acoustic and linguistic priors, achieving robust ASR generalization across low-resource and cross-lingual tasks.

2.5 Specialized Domains and Adaptive Schemes

Two-stage regimes appear in recommendation systems (contrastive ID embedding→full CTR model; (Hsu et al., 26 Aug 2025)), topology-aware medical imaging (SDF pre-training→persistent-homology fine-tuning; (Wu et al., 14 Mar 2025)), and few-shot relation extraction (masked span modeling→span-level contrastive alignment; (Guo et al., 18 May 2025)).

3. Representative Training Objectives and Architectures

Pre-training tasks and network architectures vary by domain and stage:

Stage	Objective	Network Configuration
Stage 1	MLM, InfoNCE, SDF regression, MSE, distillation	Encoder-only/Minimal model
Stage 2	Seq2Seq denoising, autoencoding, fine-tuning	Encoder+Decoder, adapters,
	cross-modal denoising, contrastive SCL	mixture-of-experts, dynamic
		adapters/head

Algorithmic highlights:

• MLM: $$L_\text{MLM} = - \mathbb{E}_{x \sim D} \sum_{i \in M} \log P(x_i|x_{\setminus M})$$
• Seq2Seq denoising: $$L_\text{seq2seq} = -\mathbb{E}_{x \sim D} \sum_{t=1}^{|y|} \log P(y_t | y_{<t}, corr(x))$$
• Patch-level InfoNCE: $$\mathcal{L} = -\frac{1}{n} \sum_i \log \frac{\exp(s_{i,i}/\tau)}{\sum_{k\neq i}[\exp(s_{i,k}/\tau)+\exp(s_{k,i}/\tau)]}$$
• SDF regression: $$L_\text{SDF} = \frac{1}{N} \sum_{i=1}^N (f_\theta(I;x_i) - SDF_{gt}(x_i))^2$$
• Distillation: $L_\text{layer} + L_\text{pred}$ (mean squared hidden state error + soft CS between logits)
• Span-level contrastive: $$L_{scl} = -\log \frac{\exp(sim(h_a, h_p)/\tau)}{\exp(sim(h_a, h_p)/\tau) + \exp(sim(h_a, h_n)/\tau)}$$

Staged freezing (e.g., freezing the encoder then unfreezing during seq2seq) and adapter-based tuning are common mechanisms for preventing catastrophic forgetting while allowing later adaptation (Soltan et al., 2023, Hao et al., 4 Sep 2025). Fine-tuning hyperparameters and loss weights are task-specific but typically preserve lower learning rates and regularization for transferred modules.

4. Empirical Outcomes and Efficiency Gains

Empirical studies demonstrate that two-stage regimes offer consistent advantages:

• Efficiency Gains: Sequential MLM→Seq2Seq reduces compute by 27% (from 15.0 to 11.0 TU) while matching performance on classification, sequence labeling, and generative tasks (Soltan et al., 2023).
• Task Performance: In sequence labeling, an MLM-initialized encoder substantially outperforms one extracted from a seq2seq model (e.g., mATIS++ SL: 61.6 vs. 44.3); staged denoising closes the gap for generative tasks without sacrificing discriminative accuracy.
• Domain Adaptation and Generalization: In multi-modal and recommendation settings, first-stage contrastive pre-training on large-scale or broad-coverage data provides robust, generalized representations for efficient downstream fine-tuning (Jamal et al., 2024, Hsu et al., 26 Aug 2025).
• Data Efficiency: In medical imaging and dense prediction, two-stage regimes yield improved AUC, Dice, and clDice metrics over single-stage pre-training, especially in low-data regimes (Wang et al., 2024, Wu et al., 14 Mar 2025).
• Improved Optimization: SDF-based pre-training in topology-aware segmentation speeds up convergence and reduces the reliance on expensive topological penalties during fine-tuning (Wu et al., 14 Mar 2025).
• Distillation Gaps Bridged: Two-stage student–TA–teacher distillation consistently yields higher accuracy and architectural flexibility at a fixed parameter budget (Yao et al., 2023).

5. Variants, Ablations, and Limitations

Variants of two-stage regimes have been systematically evaluated:

• Decoder depth reduction and variant masking schedules often fail to improve over the canonical regime (Soltan et al., 2023).
• Attention fusion (decoders attending to all encoder layers) provides no additional benefit over standard cross-attention (Soltan et al., 2023).
• Global distillation via weight carryover, as opposed to explicit loss terms, is sufficient to transfer knowledge between stages in multi-modal domains (Jamal et al., 2024).
• Contrastive learning pre-training is robust against overfitting in ID-embedding settings even under multi-epoch schedules; binary cross-entropy baselines degenerate after the first epoch (Hsu et al., 26 Aug 2025).

A limitation is that the performance gain of each stage is task- and domain-dependent, and optimal hyper-parameter schedules for freezing, unfreezing, and loss weighting may not transfer across applications. Detailed ablatons demonstrate that removing either stage can significantly degrade final task accuracy (Guo et al., 18 May 2025).

6. Theoretical and Practical Implications

Two-stage pre-training fundamentally enables curriculum-like knowledge transfer, modular reuse of models, and computational efficiency across varied domains. Analysis of why the regime works emphasizes:

• Preservation of inductive biases and beneficial priors from simpler or more data-rich objectives.
• Prevention of catastrophic forgetting via progressive unfreezing or staged adapter training.
• Faster convergence and improved data efficiency for both discriminative and generative tasks, enabled by richer intermediate representations.

This regime formalizes the process of building up invariant, generalizable representations before specializing to downstream or domain-specific settings, often with minimal architectural modifications and strong empirical gains.

7. Cross-Domain Applicability and Future Prospects

Originally developed in natural language processing, two-stage pre-training now underpins best practices for vision-language integration, multi-modal fusion, recommender system embedding, medical imaging, few-shot learning, and topology-preserving segmentation. Its foundations in modularity, staged curriculum, and representational transfer suggest continued applicability as models scale and new modalities are integrated. The regime remains a basis for research into optimal curriculum schedules, domain adaptation, and the integration of learned priors with downstream task-specific constraints.

Key references include "Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models" (Soltan et al., 2023), "VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation" (Chen et al., 2023), and "A Two-Stage Progressive Pre-training using Multi-Modal Contrastive Masked Autoencoders" (Jamal et al., 2024).