Multi-Task Pretraining (MTP): Overview

• Multi-Task Pretraining (MTP) is a transfer-oriented approach that trains a shared model on multiple tasks using diverse supervision signals to improve downstream performance.
• MTP leverages shared architectures, joint loss formulations, and dynamic task weighting to balance heterogeneous objectives and reduce task discrepancies.
• Empirical results across vision, language, robotics, and other domains demonstrate that MTP can significantly enhance performance, sometimes achieving up to 20–50% closer expert returns with fewer training steps.

Multi-task pretraining (MTP) denotes a transfer-oriented pretraining regime in which a shared model is optimized on multiple tasks or objectives before downstream adaptation. In the cited literature, the term covers reward-free control pretraining, supervised dense-prediction pretraining, multilingual and multimodal self-supervision, parameter-efficient prompt transfer, and multimodal representation learning for vision, language, speech, robotics, molecules, and 3D geometry. Across these settings, MTP is used to reduce task discrepancy between pretraining and fine-tuning, to exploit shared structure across heterogeneous supervision signals, and to improve transfer under limited downstream data or distribution shift (Sun et al., 2023, Ni et al., 2020, Wang et al., 2023, Wang et al., 2024, Di et al., 20 Jan 2026, Liu et al., 23 Jul 2025).

1. Scope and historical development

The modern MTP literature represented here begins with dialogue and multilingual multimodal pretraining in 2020. "Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning" defines four unsupervised mask-and-recover tasks—Reference Prediction, Word Prediction, Role Prediction, and Sentence Generation—and applies them to multi-role dialogue understanding (Wang et al., 2020). In the same year, "M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training" combines multilingual masked language modeling with multimodal masked language modeling, masked region modeling, and visual-linguistic matching in a single-stream multilingual image-text encoder (Ni et al., 2020).

Subsequent work broadens MTP along two axes. One axis is scale: Pinterest’s billion-image visual system uses a single shared backbone trained jointly for weakly supervised classification, single-label classification tasks, and retrieval objectives, with deployment in a production visual shopping system (Beal et al., 2021). Prompt-based language modeling extends the idea to parameter-efficient transfer: "Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning" distills a single transferable prompt from multiple source tasks and adapts it through low-rank multiplicative updates (Wang et al., 2023), while "Task Compass: Scaling Multi-task Pre-training with Task Prefix" introduces learned task-prefix embeddings and a joint supervised-plus-MLM objective across 40 datasets (Zhang et al., 2022).

The other axis is domain diversification. MTP is applied to sequential decision making via SMART’s reward-free control-centric objective (Sun et al., 2023), to remote sensing via joint semantic segmentation, instance segmentation, and rotated object detection on SAMRS (Wang et al., 2024), to xLSTM vision through autoregression, depth estimation, and segmentation (Huang et al., 2024), to robot control with weakly aligned video-text pairs and latent continuous action representations (Shi et al., 30 Jan 2026), to ADME property prediction through a probabilistic latent-variable objective (Xue et al., 9 Jun 2026), and to point cloud understanding through token reconstruction, point reconstruction, and cross-modal contrastive learning (Liu et al., 23 Jul 2025). This distribution of applications suggests that MTP is not a single algorithmic template but a general design pattern for pretraining transferable representations.

2. Shared architectural motifs

A recurrent architectural pattern is a shared encoder or backbone coupled to multiple task realizations. The simplest form is a shared trunk with task-specific heads. Pinterest’s system trains a single high-capacity ViT-Base backbone, with each task using its own classification or metric head (Beal et al., 2021). Remote-sensing MTP uses a shared encoder $f_\theta$ and three decoder heads—UperNet for semantic segmentation, Mask-R-CNN for instance segmentation, and Oriented-RCNN for rotated detection (Wang et al., 2024). MTV likewise uses a shared ViT backbone with separate heads for vision-language contrastive learning, self-supervised distillation and masked feature prediction, grounding, and depth estimation (Di et al., 20 Jan 2026). MMPT for point clouds follows the same pattern: one Transformer encoder with three parallel pretraining heads for Token-Level Reconstruction, Point-Level Reconstruction, and Multi-Modal Contrastive Learning (Liu et al., 23 Jul 2025).

Other systems collapse multiple tasks into a more unified generation or control interface. OFA-X builds on the OFA sequence-to-sequence Transformer, maps images and text into a common token space, and uses a single shared decoder to generate both answer tokens and free-form explanations, separated by the word “because,” with no task-specific heads (Plüster et al., 2022). SMART’s Control Transformer produces observation and action embeddings during pretraining, but downstream control is handled by attaching a lightweight policy head $\pi_\phi$ to the current-observation embedding (Sun et al., 2023). CARE builds on a Prismatic-7B vision-language backbone, extracts a continuous latent action representation from “action-placeholder” token positions, and couples it to a frame decoder and a keypoint-track decoder during pretraining (Shi et al., 30 Jan 2026).

A separate line of work implements task sharing through conditioning rather than explicit decoders. MPT keeps the pretrained LLM frozen and transfers via a shared prompt matrix $P^{\!*}$ plus low-rank task-specific updates (Wang et al., 2023). Task Compass prepends a learned $\texttt{[Prefix\_}j\texttt{]}$ token for each task, and the MLM objective also masks and predicts prefixes, forcing prefix embeddings to encode task-specific signals (Zhang et al., 2022). Taken together, these systems show that MTP can be realized through shared backbones, shared generation interfaces, or shared conditioning parameters.

Setting	Shared component	Task realization
SMART	Control Transformer	Forward, inverse, and masked hindsight control objectives
OFA-X	Shared encoder-decoder Transformer	Answer and explanation generation in one decoder
Pinterest visual system	ViT-Base backbone	Classification and retrieval heads
Remote sensing MTP	Shared encoder $f_\theta$	Segmentation, instance, and rotated-detection decoders
MPT	Shared prompt $P^{\!*}$	Low-rank task-specific prompt updates
MMPT	Shared point-cloud Transformer	TLR, PLR, and multi-modal contrastive heads

3. Objective design and optimization regimes

The core mathematical form of MTP is usually a joint objective built by summing task losses. OFA pretraining uses

$L_{\text{total}}=\sum_{t=1}^{T}\lambda_t L_t,$

with $T=8$ vision-language pretraining tasks and task weights set roughly proportional to the inverse of dataset size (Plüster et al., 2022). M3P uses equal-weight accumulation of multilingual masked language modeling, multimodal code-switched masked language modeling, masked region modeling, and visual-linguistic matching:

$$L_{\text{total}}=L_{xMLM}+L_{MC\text{-}MLM}+L_{MC\text{-}MRM}+L_{MC\text{-}VLM}.$$

The compact three-way form with $\lambda_1=\lambda_2=\lambda_3=1$ is also used in that framework (Ni et al., 2020).

Control-oriented MTP modifies the constituent tasks rather than the summation principle. SMART defines

$\pi_\phi$0

where forward and inverse dynamics capture short-term control and the masked hindsight loss captures long-horizon controllability (Sun et al., 2023). MAL uses

$\pi_\phi$1

and reports that $\pi_\phi$2 gave the best transfer performance (Huang et al., 2024). Remote-sensing MTP also uses an equal-weight formulation, setting all $\pi_\phi$3 to $\pi_\phi$4 across semantic segmentation, instance box loss, instance mask loss, and rotated object detection (Wang et al., 2024). MTV reports that uniform weighting of vision-language, SSL, and dense losses yielded stable convergence and that no task-specific tuning was needed (Di et al., 20 Jan 2026).

Not all formulations use manually fixed scalar weights. Task Compass combines supervised multi-task loss and masked-language-modeling regularization through

$\pi_\phi$5

(Zhang et al., 2022). CARE uses Kendall et al.’s uncertainty-weighted loss to balance next-frame feature prediction and keypoint-trajectory prediction without manual weights (Shi et al., 30 Jan 2026). In ADME modeling, Contrastive KERMT explicitly rejects auxiliary-loss interpretation: reconstruction, contrastive discrimination, and chemistry-specific supervision enter as unit-weighted log-probability factors in a single probabilistic latent-variable objective (Xue et al., 9 Jun 2026). A common consequence is that MTP objective design is inseparable from assumptions about what should be transferable: local dynamics, global semantics, spatial geometry, discourse structure, chemistry priors, or task identity.

4. Task mixtures, supervision sources, and data construction

MTP is defined as much by task composition as by model architecture. OFA’s pretraining mixture includes Image Captioning, Grounded Captioning, Visual Question Answering, Image–Text Matching, Visual Grounding, Object Detection, Image Infilling, and Text Infilling, all cast as text-to-text examples with natural-language prompts (Plüster et al., 2022). M3P combines Wikipedia text in 100 languages with English and code-switched image-caption pairs, using Panlex-based replacement to expose the model to direct non-English language-vision alignment (Ni et al., 2020). Task Compass scales supervised task-aware pretraining to 40 datasets drawn from GLUE, Rainbow, LexGLUE, domain-specific classification, multiple-choice QA, and miscellaneous tasks (Zhang et al., 2022).

Several systems derive task mixtures from large-scale weak or pseudo supervision. Pinterest constructs “Annotations-1.3B” through annotation selection, visual concreteness filtering, clustering long-tail labels, and L1 interest restriction, yielding 1.329 B unique images, 18 K cluster labels, and $\pi_\phi$6 labels per image on average (Beal et al., 2021). MTV uses DataComp images augmented with captions, OWLv2-plus-RAM++ grounding pairs, and Depth Anything V2 depth maps, effectively turning each image into a jointly supervised sample without human dense annotation (Di et al., 20 Jan 2026). Remote-sensing MTP uses SAMRS, constructed by applying the Segment-Anything Model to DOTA-V2, DIOR, and FAIR1M, producing 105 090 images and 1 668 241 instance masks with dense labels for three pretraining tasks (Wang et al., 2024).

In sequential and embodied domains, task mixtures arise from trajectories or demonstrations rather than static labels. SMART pretrains on DeepMind Control Suite trajectories collected from cartpole-swingup, hopper-hop, cheetah-run, walker-stand, and walker-run, with both random rollouts and exploratory SAC replay data (Sun et al., 2023). CARE pretrains on video-text pairs and then fine-tunes with only 3% of RT-1 for action supervision (Shi et al., 30 Jan 2026). MAL stages its pretraining across ImageNet-1K, NYU Depth v2, and ADE20K (Huang et al., 2024). Contrastive KERMT uses unlabeled corpora ranging from 11 M ZINC15+ChEMBL molecules to a larger $\pi_\phi$7 M ZINC15 superset and ADME-adjacent molecular augmentations (Xue et al., 9 Jun 2026). These examples show that MTP does not presuppose uniform supervision; instead, it assembles complementary tasks from whatever signals a domain affords.

5. Transfer behavior and empirical effects

A central empirical claim of the MTP literature is improved transfer efficiency. In sequential decision making, SMART yields a warm-start advantage of approximately 20–50% closer to expert return on seen tasks, often reaches 90% of expert with 2× fewer gradient steps, and remains resilient when pretrained on random data, outperforming scratch by 30–40% on average (Sun et al., 2023). In remote sensing, MTP improves or matches strong MAE or ImageNet-22K baselines across scene classification, horizontal and rotated detection, semantic segmentation, and change detection on 14 datasets (Wang et al., 2024). MAL shows consistent gains over ViL baselines—$\pi_\phi$8 for tiny, $\pi_\phi$9 for small, and $P^{\!*}$0 for base ImageNet-1K Top-1 classification—and improves ADE20K mIoU from 46.3% to 47.5% for the small backbone (Huang et al., 2024).

In language and multimodal generation, OFA-X reports new state-of-the-art task accuracy on VQA-X, e-SNLI-VE, and VCR, and human judges preferred its explanations over the ground truth in two of the three evaluated datasets (Plüster et al., 2022). Dialogue MTP improves all downstream tasks studied there, including a macro-F$P^{\!*}$1 boost of +17.4% on CDD classification and +53% to +192% BLEU-4 improvement on Controversy Focus Generation over a strong hierarchical S2S baseline (Wang et al., 2020). MPT shows that parameter-efficient transfer can itself be multitask-pretrained: on GLUE and SuperGLUE with T5-Base it reaches 85.6% and 74.1% average accuracy while tuning approximately 77.6K parameters per task, or 0.035% of a 220M-parameter T5-Base model (Wang et al., 2023). Task Compass reaches 89.0 average validation accuracy on Rainbow and 89.7 with Tailor, and reports human-parity leaderboard results on HellaSwag and $P^{\!*}$2NLI (Zhang et al., 2022).

At industrial and foundation-model scale, Pinterest reports a 36% improvement in top-1 relevance and a 23% improvement in click-through volume after deployment of the Transformer-based visual system (Beal et al., 2021). MTV reports that, for ViT-B/16 at 10 M samples, adding SSL, grounding, and depth to a VL-only baseline yields total gains of +13.5% on ImageNet-1k, +12.2% on COCO image-to-text retrieval, +14.2 on ADE20K, and a reduction of 0.131 in NYUv2 RMSE (Di et al., 20 Jan 2026). In robotics, CARE reaches 77.7% average success rate on LIBERO, exceeding CoMo at 69.2% and LAPA at 64.3%, while also improving interpretability and reducing shortcut learning according to LP-MSE, semantic accuracy, and S-PCFC (Shi et al., 30 Jan 2026). In ADME prediction, Contrastive KERMT improves over KERMT by 7.6% on Biogen, 9.9% on ExpansionRX, and 9.5% on ChEMBL-MT when averaged over significantly improved endpoints (Xue et al., 9 Jun 2026). In 3D understanding, MMPT improves or matches strong baselines across classification, segmentation, detection, and point-cloud completion, including 93.9% on ModelNet40 and 86.4% on ScanObjectNN PB-T50-RS (Liu et al., 23 Jul 2025).

6. Misconceptions, limitations, and open questions

A common simplification is to equate MTP with fully supervised joint training over related tasks. The surveyed literature is broader: it includes reward-free control pretraining (Sun et al., 2023), unsupervised dialogue mask-and-recover objectives (Wang et al., 2020), multilingual and multimodal self-supervision (Ni et al., 2020), weakly aligned video-text pretraining for robot control (Shi et al., 30 Jan 2026), and pseudo-supervised dense vision targets from expert models (Di et al., 20 Jan 2026). Another simplification is to assume that MTP requires task-specific heads. OFA-X uses a single decoder for answer and explanation generation (Plüster et al., 2022), whereas Pinterest, remote-sensing MTP, MAL, MTV, and MMPT rely on task-specific heads or decoders (Beal et al., 2021, Wang et al., 2024, Huang et al., 2024, Di et al., 20 Jan 2026, Liu et al., 23 Jul 2025).

Negative transfer remains a persistent concern. Task Compass explicitly states that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks, motivating task-prefix guidance and prefix-correlation analysis (Zhang et al., 2022). Several works respond through better balancing or decomposition: OFA used inverse-dataset-size weighting during pretraining (Plüster et al., 2022), MPT uses shared prompt distillation with low-rank task-specific updates (Wang et al., 2023), CARE uses learned task-uncertainty parameters (Shi et al., 30 Jan 2026), and Contrastive KERMT replaces ad hoc auxiliary-loss weights with a unified probabilistic objective (Xue et al., 9 Jun 2026). This suggests that MTP’s benefits depend not only on adding tasks but on controlling interference.

The literature also identifies domain-specific limits. SMART notes that all experiments are in simulated continuous-control benchmarks with compact image observations, that fine-tuning still requires tens of thousands of frames, and that online-RL finetuning of transformer models remains computationally heavy and sample-inefficient relative to lightweight MLP policies (Sun et al., 2023). OFA-X distinguishes plausibility from faithfulness and notes that current human ratings do not measure whether the model truly used the generated reasons; it also reports dataset biases in e-SNLI-VE (Plüster et al., 2022). M3P observes that excessive code-switching can introduce noise and hurt convergence (Ni et al., 2020). Remote-sensing MTP reports that starting MTP from random initialization underperforms strong single-task initialization and that the advantage shrinks when downstream training is very long or very large (Wang et al., 2024). Pinterest notes that new tasks may require re-balancing of $P^{\!*}$3 weights and that long-tail and domain shift remain challenges (Beal et al., 2021). Accordingly, open directions in the surveyed work include dynamic task weighting, continual addition of new tasks without catastrophic forgetting, broader multimodal coverage, and scaling MTP to more heterogeneous task ecologies (Sun et al., 2023, Wang et al., 2024, Di et al., 20 Jan 2026).