Unified Pre-Training Tasks

• Unified pre-training tasks are methodologies where a single neural network is trained with multi-modal objectives to support diverse tasks like text, vision, and speech.
• They leverage architectures such as unified transformers, mixture-of-experts, and modality-agnostic tokenizers to enable efficient parameter sharing and cross-modal alignment.
• Empirical results show these models achieve state-of-the-art benchmarks with improved parameter efficiency, facilitating robust transfer learning across tasks.

Unified pre-training tasks are methodologies and model architectures designed to enable a single neural network to support diverse downstream tasks—such as understanding and generation—across one or more modalities (text, vision, audio, code, molecular graphs, or combinations thereof). Unlike traditional pre-training, which often specializes models for a narrow set of tasks or modalities, unified pre-training seeks to develop architectures, objectives, and data pipelines that enable broad generalization, efficient parameter sharing, and streamlined transfer learning. Unified pre-trained models support both discriminative and generative paradigms, fostering cross-task knowledge transfer and reducing the need for task-specific models and objectives.

1. Foundational Model Architectures

Unified pre-training models typically employ versatile neural frameworks that can process multiple modalities and task types under a shared set of parameters and operations. Several architectural paradigms have emerged:

• Unified Transformer Networks: Many vision-language and LLMs, such as Unified VLP (Zhou et al., 2019) and UniLMv2 (Bao et al., 2020), use a single Transformer stack with specialized self-attention masks or prefix tokens to switch between encoder, decoder, or encoder–decoder roles. Both bidirectional (for understanding) and autoregressive (for generation) flows are supported.
• Mixture-of-Experts or Modular Blocks: Models such as VLMo (Bao et al., 2021) introduce modality-specific experts (e.g., V-FFN for vision, L-FFN for language, VL-FFN for fusion) within Transformer blocks. Routing logic enables shared or specialized processing according to the task and modality configuration.
• Cross-Modal Tokenization and Alignment: Architectures like Uni-Perceiver (Zhu et al., 2021) and LayoutLMv3 (Huang et al., 2022) use unified input tokenization and modality-agnostic Transformer encoders, allowing text, images, videos, and other modalities to be embedded and represented in a common latent space.
• Encoder–Decoder Frameworks: Many unified speech models (e.g., SpeechT5 (Ao et al., 2021), UniWav (Liu et al., 2 Mar 2025)), as well as vision-language and code models, rely on encoder–decoder designs with unified representations and modality-specific pre/post-processing layers.

2. Pre-Training Objectives and Task Formulation

Unified pre-training relies on custom formulations of multi-task, multi-modal learning objectives, often realized via masking or multi-view self-supervision:

• Masked and Sequence-to-Sequence Objectives: Unified VLP (Zhou et al., 2019) and UniLMv2 (Bao et al., 2020) employ both bidirectional masked objectives (cloze-style, as in BERT) and unidirectional sequence-to-sequence/auto-regressive objectives, distinguished through self-attention mask manipulation.
• Contrastive and Alignment Losses: Cross-modal contrastive learning (as in UniVL (Luo et al., 2020), CLIP-inspired frameworks (Shao et al., 2023)), forces alignment between different modalities by maximizing similarity for true pairs and minimizing it for distractors.
• Reconstruction and Prediction Losses: Tasks include reconstructing masked out atomic/molecular structure features (Zhu et al., 2022, Ding et al., 2023), masked frames in video (Luo et al., 2020), or document patches (Huang et al., 2022), fostering rich, localized representations.
• Auxiliary Cross-Modal or Generation Tasks: Objectives such as program comment generation from ASTs (Guo et al., 2022), canonicalization in molecular representations (Ding et al., 2023), or speech-to-text/speech-to-phoneme generation (Tang et al., 2022) supplement primary language or vision tasks with additional supervision.

Commonly, the overall pre-training loss is a weighted sum of multiple objectives. For example, in LayoutLMv3 (Huang et al., 2022):

$$L = L_{\mathrm{MLM}} + L_{\mathrm{MIM}} + L_{\mathrm{WPA}}$$

where $L_{\mathrm{MLM}}$ is masked language modeling, $L_{\mathrm{MIM}}$ is masked image modeling, and $L_{\mathrm{WPA}}$ is the word-patch alignment loss.

3. Masking, Conditioning, and Attention Control

Central to many unified pre-training frameworks is the explicit control of the context available to different parts of the model:

• Self-Attention Masking: The sole difference between bidirectional and sequence-to-sequence pre-training in Unified VLP (Zhou et al., 2019) is the self-attention mask $M$: positions are blocked to enforce causal (autoregressive) or full-context prediction. This mechanism is extended in UniLMv2 (Bao et al., 2020) to support pseudo-masked (partially autoregressive) modeling, using explicit mask and pseudo-mask tokens.
• Prefix Adapters and Token Control: UniXcoder (Guo et al., 2022) uses special prefix tokens and attention masks for encoder-only, decoder-only, and encoder-decoder modes, providing a flexible approach to code generation and understanding without redundant model duplication.
• Selective Masking Regimes: UniMASK (Carroll et al., 2022) demonstrates that in sequential decision-making, changes to the masking scheme (which tokens are hidden and must be predicted) correspond to shifting between behavior cloning, reward-conditioning, and other inference tasks, all realized under the same Transformer.

4. Modality and Scale Bridging

Unified pre-training increasingly addresses the challenges of bridging across both modalities and data scale:

• Multi-Modal Tokenizers: Systems like Uni-Perceiver (Zhu et al., 2021) and XDoc (Chen et al., 2022) employ modality-agnostic or adaptive tokenizers and embedding layers, allowing everything from plain text to 2D document layouts or XPath web features to be represented in the same space.
• Granularity-Adjustable Encodings: AdaMR (Ding et al., 2023) establishes "granularity-adjustable" tokenization, switching between atomic-level and substructure-level representations for molecules by controlling tokenizer dropout.
• Cross-Scale Pre-Training and Differentiable Rendering: UniPre3D (Wang et al., 11 Jun 2025) applies differentiable Gaussian splatting to render both object-level and scene-level 3D point clouds, achieving pixel-level supervision and bridging the scale diversity inherent in 3D data.

5. Empirical Performance and Benchmarking

Unified models set or approach state-of-the-art results across a wide spectrum of benchmarks:

Domain	Unified Model	Key Benchmarks	Highlights
Vision-Language	Unified VLP (Zhou et al., 2019)	COCO, Flickr30k, VQA 2.0	BLEU@4 ≈ 36.5, METEOR ≈ 28.4, VQA Acc ≈ 71%
Video+Language	UniVL (Luo et al., 2020)	YouCook2, COIN, CrossTask	Recall@1 = 28.9 (retrieval), BLEU-4 > 17 (captioning)
Language	UniLMv2 (Bao et al., 2020)	SQuAD, GLUE, CNN/DailyMail	SotA NLU and NLG, unified training
Code	PLBART (Ahmad et al., 2021), UniXcoder (Guo et al., 2022)	Code search/generation/translation	Outperforms CodeBERT/GraphCodeBERT on most tasks
Speech	SpeechT5 (Ao et al., 2021), UniWav (Liu et al., 2 Mar 2025)	ASR/TTS/ST	ASR and TTS metrics on par with task-specific models
Document AI	LayoutLMv3 (Huang et al., 2022), UDoc (Gu et al., 2022), XDoc (Chen et al., 2022)	FUNSD, DocVQA, RVL-CDIP	State-of-the-art or highly competitive
3D Vision	UniPre3D (Wang et al., 11 Jun 2025)	ScanObjectNN, ScanNet, S3DIS	Outperforms all prior 3D pre-training approaches

Benchmarking demonstrates that unified training can match or surpass the performance of prior task- or modality-specialized pre-training schemes, even at reduced parameter overhead (e.g., XDoc matches independent models at 36.7% of the total parameter count).

6. Implications, Applications, and Future Directions

Unified pre-training transforms development, deployment, and generalization properties of foundation models:

• Parameter Efficiency and Simplified Deployment: Sharing backbones across modalities or tasks removes the need for training and maintaining multiple large separate networks (Chen et al., 2022, Zhu et al., 2021).
• Data Efficiency and Knowledge Transfer: Unified objectives facilitate effective few-shot learning and prompt-based adaptation, as demonstrated in vision-language (Liu et al., 2021), customer service dialogue (He et al., 2022), and perception (Zhu et al., 2021).
• Cross-Modal and Cross-Task Generalization: Many models, e.g., Uni-Perceiver and LayoutLMv3, show "zero-shot" or prompt-tuned success on tasks and domains not explicitly present during pre-training, demonstrating broad representational generality.
• Challenges: Trade-offs in optimizing for both discriminative and generative tasks remain; e.g., increasing auto-regressive mask ratio can degrade understanding accuracy (Liu et al., 2021). Unified architectures typically require careful balancing of losses and architecture flexibility (e.g., mixture-of-modal-experts in VLMo).
• Research Directions: Richer granularity in tokenization (Ding et al., 2023), advanced masking/conditioning paradigms (Carroll et al., 2022, Bao et al., 2020), interactive multi-modal fusions (Wang et al., 11 Jun 2025), and cross-scale or cross-format transfer (Wang et al., 11 Jun 2025, Chen et al., 2022), as well as methods to disentangle task- or modality-specific factors (Liu et al., 2 Mar 2025), are active research areas.

Unified pre-training tasks thus provide an architectural, objective-driven, and data-centric foundation for developing large-scale models capable of handling a spectrum of complex tasks and modalities within a single, efficient, and easily extensible framework.