---
title: 'Multi-Task Pretraining (MTP): Overview'
url: https://www.emergentmind.com/topics/multi-task-pretraining-mtp
type: topic
---

# Multi-Task Pretraining (MTP): Overview

Multi-task pretraining (MTP) denotes a transfer-oriented pretraining regime in which a shared model is optimized on multiple tasks or objectives before downstream adaptation. In the cited literature, the term covers reward-free control pretraining, supervised dense-prediction pretraining, multilingual and multimodal self-supervision, parameter-efficient prompt transfer, and multimodal representation learning for vision, language, speech, robotics, molecules, and 3D geometry. Across these settings, MTP is used to reduce task discrepancy between pretraining and fine-tuning, to exploit shared structure across heterogeneous supervision signals, and to improve transfer under limited downstream data or distribution shift [2301.09816] [2006.02635] [2303.02861] [2403.13430] [2601.13886] [2507.17533].

## 1. Scope and historical development

The modern MTP literature represented here begins with dialogue and multilingual multimodal pretraining in 2020. "Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning" defines four unsupervised mask-and-recover tasks—Reference Prediction, Word Prediction, Role Prediction, and Sentence Generation—and applies them to multi-role dialogue understanding [2003.04994]. In the same year, "M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training" combines multilingual masked language modeling with multimodal masked language modeling, masked region modeling, and visual-linguistic matching in a single-stream multilingual image-text encoder [2006.02635].

Subsequent work broadens MTP along two axes. One axis is scale: Pinterest’s billion-image visual system uses a single shared backbone trained jointly for weakly supervised classification, single-label classification tasks, and retrieval objectives, with deployment in a production visual shopping system [2108.05887]. Prompt-based language modeling extends the idea to parameter-efficient transfer: "Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning" distills a single transferable prompt from multiple source tasks and adapts it through low-rank multiplicative updates [2303.02861], while "Task Compass: Scaling Multi-task Pre-training with Task Prefix" introduces learned task-prefix embeddings and a joint supervised-plus-MLM objective across 40 datasets [2210.06277].

The other axis is domain diversification. MTP is applied to sequential decision making via SMART’s reward-free control-centric objective [2301.09816], to remote sensing via joint semantic segmentation, instance segmentation, and rotated object detection on SAMRS [2403.13430], to xLSTM vision through autoregression, depth estimation, and segmentation [2412.10730], to robot control with weakly aligned video-text pairs and latent continuous action representations [2601.22467], to ADME property prediction through a probabilistic latent-variable objective [2606.11508], and to point cloud understanding through token reconstruction, point reconstruction, and cross-modal contrastive learning [2507.17533]. This distribution of applications suggests that MTP is not a single algorithmic template but a general design pattern for pretraining transferable representations.

## 2. Shared architectural motifs

A recurrent architectural pattern is a shared encoder or backbone coupled to multiple task realizations. The simplest form is a shared trunk with task-specific heads. Pinterest’s system trains a single high-capacity ViT-Base backbone, with each task using its own classification or metric head [2108.05887]. Remote-sensing MTP uses a shared encoder $f_\theta$ and three decoder heads—UperNet for semantic segmentation, Mask-R-CNN for instance segmentation, and Oriented-RCNN for rotated detection [2403.13430]. MTV likewise uses a shared ViT backbone with separate heads for vision-language contrastive learning, self-supervised distillation and masked feature prediction, grounding, and depth estimation [2601.13886]. MMPT for point clouds follows the same pattern: one Transformer encoder with three parallel pretraining heads for Token-Level Reconstruction, Point-Level Reconstruction, and Multi-Modal Contrastive Learning [2507.17533].

Other systems collapse multiple tasks into a more unified generation or control interface. OFA-X builds on the OFA sequence-to-sequence Transformer, maps images and text into a common token space, and uses a single shared decoder to generate both answer tokens and free-form explanations, separated by the word “because,” with no task-specific heads [2212.04231]. SMART’s Control Transformer produces observation and action embeddings during pretraining, but downstream control is handled by attaching a lightweight policy head $\pi_\phi$ to the current-observation embedding [2301.09816]. CARE builds on a Prismatic-7B vision-language backbone, extracts a continuous latent action representation from “action-placeholder” token positions, and couples it to a frame decoder and a keypoint-track decoder during pretraining [2601.22467].

A separate line of work implements task sharing through conditioning rather than explicit decoders. MPT keeps the pretrained language model frozen and transfers via a shared prompt matrix $P^{\!*}$ plus low-rank task-specific updates [2303.02861]. Task Compass prepends a learned $\texttt{[Prefix\_}j\texttt{]}$ token for each task, and the MLM objective also masks and predicts prefixes, forcing prefix embeddings to encode task-specific signals [2210.06277]. Taken together, these systems show that MTP can be realized through shared backbones, shared generation interfaces, or shared conditioning parameters.

| Setting | Shared component | Task realization |
|---|---|---|
| SMART | Control Transformer | Forward, inverse, and masked hindsight control objectives |
| OFA-X | Shared encoder-decoder Transformer | Answer and explanation generation in one decoder |
| Pinterest visual system | ViT-Base backbone | Classification and retrieval heads |
| Remote sensing MTP | Shared encoder $f_\theta$ | Segmentation, instance, and rotated-detection decoders |
| MPT | Shared prompt $P^{\!*}$ | Low-rank task-specific prompt updates |
| MMPT | Shared point-cloud Transformer | TLR, PLR, and multi-modal contrastive heads |

## 3. Objective design and optimization regimes

The core mathematical form of MTP is usually a joint objective built by summing task losses. OFA pretraining uses
$$
L_{\text{total}}=\sum_{t=1}^{T}\lambda_t L_t,
$$
with $T=8$ vision-language pretraining tasks and task weights set roughly proportional to the inverse of dataset size [2212.04231]. M3P uses equal-weight accumulation of multilingual masked language modeling, multimodal code-switched masked language modeling, masked region modeling, and visual-linguistic matching:
$$
L_{\text{total}}=L_{xMLM}+L_{MC\text{-}MLM}+L_{MC\text{-}MRM}+L_{MC\text{-}VLM}.
$$
The compact three-way form with $\lambda_1=\lambda_2=\lambda_3=1$ is also used in that framework [2006.02635].

Control-oriented MTP modifies the constituent tasks rather than the summation principle. SMART defines
$$
J_{\rm pre}(\theta)=L_{fwd}+L_{inv}+L_{mask},
$$
where forward and inverse dynamics capture short-term control and the masked hindsight loss captures long-horizon controllability [2301.09816]. MAL uses
$$
L_{\text{total}}=\alpha L_{AR}+\beta L_{depth}+\gamma L_{seg},
$$
and reports that $\alpha=1.0,\beta=1.0,\gamma=1.0$ gave the best transfer performance [2412.10730]. Remote-sensing MTP also uses an equal-weight formulation, setting all $\lambda$ to $1.0$ across semantic segmentation, instance box loss, instance mask loss, and rotated object detection [2403.13430]. MTV reports that uniform weighting of vision-language, SSL, and dense losses yielded stable convergence and that no task-specific tuning was needed [2601.13886].

Not all formulations use manually fixed scalar weights. Task Compass combines supervised multi-task loss and masked-language-modeling regularization through
$$
\mathcal L=\mathcal L_{\text{mtl}}+\lambda \mathcal L_{\text{mlm}},\quad \lambda=0.1
$$
[2210.06277]. CARE uses Kendall et al.’s uncertainty-weighted loss to balance next-frame feature prediction and keypoint-trajectory prediction without manual weights [2601.22467]. In ADME modeling, Contrastive KERMT explicitly rejects auxiliary-loss interpretation: reconstruction, contrastive discrimination, and chemistry-specific supervision enter as unit-weighted log-probability factors in a single probabilistic latent-variable objective [2606.11508]. A common consequence is that MTP objective design is inseparable from assumptions about what should be transferable: local dynamics, global semantics, spatial geometry, discourse structure, chemistry priors, or task identity.

## 4. Task mixtures, supervision sources, and data construction

MTP is defined as much by task composition as by model architecture. OFA’s pretraining mixture includes Image Captioning, Grounded Captioning, Visual Question Answering, Image–Text Matching, Visual Grounding, Object Detection, Image Infilling, and Text Infilling, all cast as text-to-text examples with natural-language prompts [2212.04231]. M3P combines Wikipedia text in 100 languages with English and code-switched image-caption pairs, using Panlex-based replacement to expose the model to direct non-English language-vision alignment [2006.02635]. Task Compass scales supervised task-aware pretraining to 40 datasets drawn from GLUE, Rainbow, LexGLUE, domain-specific classification, multiple-choice QA, and miscellaneous tasks [2210.06277].

Several systems derive task mixtures from large-scale weak or pseudo supervision. Pinterest constructs “Annotations-1.3B” through annotation selection, visual concreteness filtering, clustering long-tail labels, and L1 interest restriction, yielding 1.329 B unique images, 18 K cluster labels, and $\sim 2.88$ labels per image on average [2108.05887]. MTV uses DataComp images augmented with captions, OWLv2-plus-RAM++ grounding pairs, and Depth Anything V2 depth maps, effectively turning each image into a jointly supervised sample without human dense annotation [2601.13886]. Remote-sensing MTP uses SAMRS, constructed by applying the Segment-Anything Model to DOTA-V2, DIOR, and FAIR1M, producing 105 090 images and 1 668 241 instance masks with dense labels for three pretraining tasks [2403.13430].

In sequential and embodied domains, task mixtures arise from trajectories or demonstrations rather than static labels. SMART pretrains on DeepMind Control Suite trajectories collected from cartpole-swingup, hopper-hop, cheetah-run, walker-stand, and walker-run, with both random rollouts and exploratory SAC replay data [2301.09816]. CARE pretrains on video-text pairs and then fine-tunes with only 3% of RT-1 for action supervision [2601.22467]. MAL stages its pretraining across ImageNet-1K, NYU Depth v2, and ADE20K [2412.10730]. Contrastive KERMT uses unlabeled corpora ranging from 11 M ZINC15+ChEMBL molecules to a larger $\approx 208$ M ZINC15 superset and ADME-adjacent molecular augmentations [2606.11508]. These examples show that MTP does not presuppose uniform supervision; instead, it assembles complementary tasks from whatever signals a domain affords.

## 5. Transfer behavior and empirical effects

A central empirical claim of the MTP literature is improved transfer efficiency. In sequential decision making, SMART yields a warm-start advantage of approximately 20–50% closer to expert return on seen tasks, often reaches 90% of expert with 2× fewer gradient steps, and remains resilient when pretrained on random data, outperforming scratch by 30–40% on average [2301.09816]. In remote sensing, MTP improves or matches strong MAE or ImageNet-22K baselines across scene classification, horizontal and rotated detection, semantic segmentation, and change detection on 14 datasets [2403.13430]. MAL shows consistent gains over ViL baselines—$+0.5$ for tiny, $+0.8$ for small, and $+1.0$ for base ImageNet-1K Top-1 classification—and improves ADE20K mIoU from 46.3% to 47.5% for the small backbone [2412.10730].

In language and multimodal generation, OFA-X reports new state-of-the-art task accuracy on VQA-X, e-SNLI-VE, and VCR, and human judges preferred its explanations over the ground truth in two of the three evaluated datasets [2212.04231]. Dialogue MTP improves all downstream tasks studied there, including a macro-F$_1$ boost of +17.4% on CDD classification and +53% to +192% BLEU-4 improvement on Controversy Focus Generation over a strong hierarchical S2S baseline [2003.04994]. MPT shows that parameter-efficient transfer can itself be multitask-pretrained: on GLUE and SuperGLUE with T5-Base it reaches 85.6% and 74.1% average accuracy while tuning approximately 77.6K parameters per task, or 0.035% of a 220M-parameter T5-Base model [2303.02861]. Task Compass reaches 89.0 average validation accuracy on Rainbow and 89.7 with Tailor, and reports human-parity leaderboard results on HellaSwag and $\alpha$NLI [2210.06277].

At industrial and foundation-model scale, Pinterest reports a 36% improvement in top-1 relevance and a 23% improvement in click-through volume after deployment of the Transformer-based visual system [2108.05887]. MTV reports that, for ViT-B/16 at 10 M samples, adding SSL, grounding, and depth to a VL-only baseline yields total gains of +13.5% on ImageNet-1k, +12.2% on COCO image-to-text retrieval, +14.2 on ADE20K, and a reduction of 0.131 in NYUv2 RMSE [2601.13886]. In robotics, CARE reaches 77.7% average success rate on LIBERO, exceeding CoMo at 69.2% and LAPA at 64.3%, while also improving interpretability and reducing shortcut learning according to LP-MSE, semantic accuracy, and S-PCFC [2601.22467]. In ADME prediction, Contrastive KERMT improves over KERMT by 7.6% on Biogen, 9.9% on ExpansionRX, and 9.5% on ChEMBL-MT when averaged over significantly improved endpoints [2606.11508]. In 3D understanding, MMPT improves or matches strong baselines across classification, segmentation, detection, and point-cloud completion, including 93.9% on ModelNet40 and 86.4% on ScanObjectNN PB-T50-RS [2507.17533].

## 6. Misconceptions, limitations, and open questions

A common simplification is to equate MTP with fully supervised joint training over related tasks. The surveyed literature is broader: it includes reward-free control pretraining [2301.09816], unsupervised dialogue mask-and-recover objectives [2003.04994], multilingual and multimodal self-supervision [2006.02635], weakly aligned video-text pretraining for robot control [2601.22467], and pseudo-supervised dense vision targets from expert models [2601.13886]. Another simplification is to assume that MTP requires task-specific heads. OFA-X uses a single decoder for answer and explanation generation [2212.04231], whereas Pinterest, remote-sensing MTP, MAL, MTV, and MMPT rely on task-specific heads or decoders [2108.05887] [2403.13430] [2412.10730] [2601.13886] [2507.17533].

Negative transfer remains a persistent concern. Task Compass explicitly states that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks, motivating task-prefix guidance and prefix-correlation analysis [2210.06277]. Several works respond through better balancing or decomposition: OFA used inverse-dataset-size weighting during pretraining [2212.04231], MPT uses shared prompt distillation with low-rank task-specific updates [2303.02861], CARE uses learned task-uncertainty parameters [2601.22467], and Contrastive KERMT replaces ad hoc auxiliary-loss weights with a unified probabilistic objective [2606.11508]. This suggests that MTP’s benefits depend not only on adding tasks but on controlling interference.

The literature also identifies domain-specific limits. SMART notes that all experiments are in simulated continuous-control benchmarks with compact image observations, that fine-tuning still requires tens of thousands of frames, and that online-RL finetuning of transformer models remains computationally heavy and sample-inefficient relative to lightweight MLP policies [2301.09816]. OFA-X distinguishes plausibility from faithfulness and notes that current human ratings do not measure whether the model truly used the generated reasons; it also reports dataset biases in e-SNLI-VE [2212.04231]. M3P observes that excessive code-switching can introduce noise and hurt convergence [2006.02635]. Remote-sensing MTP reports that starting MTP from random initialization underperforms strong single-task initialization and that the advantage shrinks when downstream training is very long or very large [2403.13430]. Pinterest notes that new tasks may require re-balancing of $\lambda_i$ weights and that long-tail and domain shift remain challenges [2108.05887]. Accordingly, open directions in the surveyed work include dynamic task weighting, continual addition of new tasks without catastrophic forgetting, broader multimodal coverage, and scaling MTP to more heterogeneous task ecologies [2301.09816] [2403.13430] [2601.13886].

Source: https://www.emergentmind.com/topics/multi-task-pretraining-mtp