Rep-MTL: Representation-Level Task Saliency

• Rep-MTL is a multi-task learning framework that leverages representation-level task saliency for effective hard parameter sharing.
• It introduces two regularizers—Task-specific Saliency Regulation (TSR) and Cross-task Saliency Alignment (CSA)—to preserve task-specific features and promote complementary learning.
• Empirical evaluations on benchmarks and power-law analyses show improved performance and efficiency compared to traditional optimizer-centric approaches.

Searching arXiv for Rep-MTL and closely related multi-task optimization papers for grounding and citations. Rep-MTL is a multi-task learning method that operates on representation-level task saliency rather than modifying the optimizer itself. In the formulation introduced in "Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning" (Wang et al., 28 Jul 2025), the method is designed for hard parameter sharing, where a shared backbone representation $Z$ must support multiple task objectives simultaneously. Its central premise is that task interactions are expressed directly in the shared representation space, and that these interactions can be steered by two additional regularizers: Task-specific Saliency Regulation (TSR), which penalizes overly entropic task saliency, and Cross-task Saliency Alignment (CSA), which aligns sample-wise saliency patterns across tasks. The resulting objective is optimizer-agnostic, adds only two regularization terms to the usual multi-task loss, and is evaluated on both task-shift and domain-shift benchmarks (Wang et al., 28 Jul 2025).

1. Conceptual basis

Rep-MTL is motivated by a limitation of standard multi-task learning with hard parameter sharing: the shared backbone representation must simultaneously serve multiple objectives $\ell_1,\ldots,\ell_T$, and conflicting update signals can induce negative transfer. The method is positioned against existing Multi-Task Optimization techniques that work primarily at the optimizer level through loss scaling or direct gradient manipulation. In the account given for Rep-MTL, these optimizer-centric strategies include loss-rescaling methods such as EW, GLS, RLW, UW, DWA, IMTL-L, IGBv2, and FAMO, as well as gradient-oriented methods such as MGDA, GradNorm, PCGrad, GradDrop, GradVac, CAGrad, MTAdam, Nash-MTL, MetaBalance, MoCo, Aligned-MTL, FairGrad, and STCH; IMTL and DB-MTL are listed as hybrid baselines (Wang et al., 28 Jul 2025).

The distinctive claim of Rep-MTL is that the shared representation space already contains information about both conflict and complementarity. Instead of attempting only to resolve gradient conflict, it computes each task’s saliency map in the shared representation and uses that signal in two ways. First, it discourages diffuse task saliency, which is intended to preserve task-specific features. Second, it explicitly aligns sample-wise saliency structure across tasks, which is intended to promote complementary information sharing. This yields a complementary perspective on multi-task optimization rather than a replacement for off-the-shelf optimizers (Wang et al., 28 Jul 2025).

A plausible implication is that Rep-MTL reframes multi-task interaction as a representational allocation problem: some parts of the shared feature space should remain task-salient, while others should become sites of controlled cross-task sharing. That interpretation is consistent with the method’s decomposition into TSR and CSA, though the paper does not provide a formal convergence theorem (Wang et al., 28 Jul 2025).

2. Computational pipeline and saliency construction

Rep-MTL assumes parameters $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$, where $\theta_s$ denotes the shared backbone and each $\theta_t$ a task-specific decoder. For an input batch $X\in\mathbb R^{B\times 3\times H\times W}$, the backbone produces a shared feature tensor

$$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$

Each task head then produces a prediction $\hat Y_t = H_{\theta_t}(Z)$ and an associated loss $\ell_t(\theta_s,\theta_t)$ (Wang et al., 28 Jul 2025).

The defining intermediate quantity is the representation-level task saliency

$$S_t \in \mathbb R^{B\times C\times H'\times W'} \coloneqq \nabla_Z \ell_t(\theta_s,\theta_t).$$

Rep-MTL extracts these saliency maps by back-propagating each task loss through $\ell_1,\ldots,\ell_T$0. Those saliency tensors are then used to construct the two additional regularizers (Wang et al., 28 Jul 2025).

The per-batch workflow is described as follows. First, the model performs a forward pass through the shared encoder and task decoders. Second, it computes the per-task losses. Third, it back-propagates each task loss through the shared representation to obtain $\ell_1,\ldots,\ell_T$1. Fourth, it computes TSR from normalized task saliency entropies over spatial locations. Fifth, it computes CSA from sample-wise channel-affinity structures using an NT-Xent–style contrastive objective. Finally, it back-propagates the total loss and updates $\ell_1,\ldots,\ell_T$2 and $\ell_1,\ldots,\ell_T$3 using any standard optimizer such as Adam (Wang et al., 28 Jul 2025).

This pipeline is notable because it leaves the optimizer untouched. The method is therefore characterized as a lightweight, optimizer-agnostic regularization layer over standard multi-task training, rather than an alternative optimizer or gradient surgery procedure (Wang et al., 28 Jul 2025).

3. Objective function and regularization terms

TSR begins by aggregating saliency across channels:

$\ell_1,\ldots,\ell_T$4

For each spatial index, the method forms a $\ell_1,\ldots,\ell_T$5-way distribution across tasks and computes its entropy. The TSR loss is the average of these entropies over all spatial positions:

$\ell_1,\ldots,\ell_T$6

In the stated interpretation, low-entropy saliency distributions concentrate each task’s update on task-specific regions and thereby reduce antagonistic interference (Wang et al., 28 Jul 2025).

CSA is defined through channel-affinity matrices. For each task and sample,

$\ell_1,\ldots,\ell_T$7

An anchor is formed by spatial pooling of saliency across tasks and constructing an affinity matrix from that pooled representation. After $\ell_1,\ldots,\ell_T$8 normalization and flattening, the method treats $\ell_1,\ldots,\ell_T$9 as positive pairs and $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$0 for $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$1 as negatives, and applies an NT-Xent loss:

$\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$2

The stated role of CSA is to surface feature interactions that benefit multiple tasks simultaneously (Wang et al., 28 Jul 2025).

The integrated objective is

$\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$3

In all experiments, the paper sets equal weights $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$4, corresponding to the equal-weighting policy. Hyperparameters $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$5 and $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$6 are chosen by simple grid search, and values in $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$7 are reported to exhibit stable performance; the main tables use $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$8 (Wang et al., 28 Jul 2025).

The method does not provide formal convergence theorems. Its theoretical discussion is instead organized around two qualitative observations: concentrated task saliency preserves task-specific learning, and contrastive saliency alignment promotes cross-task complementarity (Wang et al., 28 Jul 2025).

4. Benchmarks, training protocol, and quantitative results

Rep-MTL is evaluated on four benchmarks covering both task-shift and domain-shift settings (Wang et al., 28 Jul 2025).

Benchmark	Tasks or domains	Architecture
NYUv2	3 tasks: segmentation, depth, normals	DeepLabV3+ with ResNet-50 backbone
Cityscapes	2 tasks: segmentation, depth	same architecture
Office-31	3 domains as tasks: Amazon, DSLR, Webcam	ResNet-18
Office-Home	4 domains	ResNet-18

The reported dataset sizes are 795 train / 654 val for NYUv2 and 2975 train / 500 test for Cityscapes. Training details common across all experiments are Adam with base learning rate $\theta=\{\theta_s,\theta_1,\ldots,\theta_T\}$9, weight decay $\theta_s$0, and a halfway learning-rate halving for the vision tasks. Batch sizes are 8 for NYUv2 and 64 for Cityscapes and the Office benchmarks. The losses are cross-entropy for segmentation and classification, $\theta_s$1 for depth, and cosine for normals (Wang et al., 28 Jul 2025).

The paper reports the following benchmark-level gains. On NYUv2, average $\theta_s$2 is $\theta_s$3 for EW, $\theta_s$4 for DB-MTL, and $\theta_s$5 for Rep-MTL (EW), with $\theta_s$6 values $\theta_s$7, $\theta_s$8, and $\theta_s$9, respectively. On Cityscapes, $\theta_t$0 is $\theta_t$1 for EW, $\theta_t$2 for DB-MTL, and $\theta_t$3 for Rep-MTL (EW). On Office-31, average accuracy gains are $\theta_t$4 for EW, $\theta_t$5 for DB-MTL, and $\theta_t$6 for Rep-MTL (EW). On Office-Home, the corresponding gains are $\theta_t$7, $\theta_t$8, and $\theta_t$9 (Wang et al., 28 Jul 2025).

Within the study’s comparison frame, these results are used to argue that the method remains effective even when paired with basic equal weighting. That is an important point because the empirical gains are not contingent on specialized loss weighting or gradient projection schemes (Wang et al., 28 Jul 2025).

5. Empirical interpretation: power-law exponents, ablations, and efficiency

Rep-MTL’s empirical analysis extends beyond task metrics to a power-law exponent analysis of layer spectra. The paper fits each layer’s weight correlation matrix spectrum as $X\in\mathbb R^{B\times 3\times H\times W}$0 and states that well-trained networks lie in $X\in\mathbb R^{B\times 3\times H\times W}$1, with higher $X\in\mathbb R^{B\times 3\times H\times W}$2 indicating over-parameterization and lower $X\in\mathbb R^{B\times 3\times H\times W}$3 indicating under-training. This diagnostic is used to characterize the balance between shared representation learning and task-specific learning (Wang et al., 28 Jul 2025).

On NYUv2, the backbone exponents are reported as $X\in\mathbb R^{B\times 3\times H\times W}$4 for EW, $X\in\mathbb R^{B\times 3\times H\times W}$5 for GradNorm, $X\in\mathbb R^{B\times 3\times H\times W}$6 for DB-MTL, and $X\in\mathbb R^{B\times 3\times H\times W}$7 for Rep-MTL, where the Rep-MTL value is described as the lowest and as corresponding to the best cross-task sharing. For the three decoder heads, EW yields $X\in\mathbb R^{B\times 3\times H\times W}$8, DB-MTL yields $X\in\mathbb R^{B\times 3\times H\times W}$9, and Rep-MTL yields $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$0; the Rep-MTL configuration is described as low and balanced, corresponding to the best mitigation of negative transfer (Wang et al., 28 Jul 2025).

The ablation study separates the contributions of TSR and CSA. TSR alone yields a modest gain of $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$1 $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$2 on NYUv2 and is reported to improve the balance of decoder power-law exponents. CSA alone yields $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$3 $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$4 and lowers the backbone $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$5 into the optimal range. The full method, combining TSR and CSA, yields $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$6, outperforming either component individually. The paper interprets this division of labor as follows: CSA improves shared learning in the backbone, while TSR preserves effective per-task training in the task-specific heads (Wang et al., 28 Jul 2025).

The efficiency discussion reports that Rep-MTL is approximately 26% faster than Nash-MTL and approximately 12% faster than FairGrad, despite requiring saliency-gradient computations. It is also reported to maintain positive $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$7 over $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$8 and to show stable gains for learning rates in $$Z = E_{\theta_s}(X)\in\mathbb R^{B\times C\times H'\times W'}.$$9 (Wang et al., 28 Jul 2025).

A common misconception in multi-task optimization is that negative transfer can be handled only by explicit gradient conflict resolution. The reported results for Rep-MTL do not eliminate the relevance of optimizer-centric methods, but they do show that representation-level regularization can act as an effective complementary mechanism (Wang et al., 28 Jul 2025).

6. Scope, naming, and disambiguation

In the machine-learning context represented by the 2025 arXiv paper, Rep-MTL denotes a representation-level saliency method for multi-task learning (Wang et al., 28 Jul 2025). The label should be distinguished from similarly named but technically unrelated uses.

One such neighboring usage is RepMT-SAC, a multi-task reinforcement learning framework built on spectral MDP decomposition and soft actor-critic. That method assumes a task-invariant feature map $\hat Y_t = H_{\theta_t}(Z)$0 and a task encoder $\hat Y_t = H_{\theta_t}(Z)$1 such that rewards factorize as $\hat Y_t = H_{\theta_t}(Z)$2, and it is evaluated on quadrotor trajectory-following tasks with source, in-distribution, and out-of-distribution splits. Its reported gains include outperforming baselines by up to 30%, faster convergence, 100% success on source and in-distribution tasks zero-shot, and 100% success on out-of-distribution tasks after only 10% of original training steps (Naveen et al., 11 Jun 2026). Despite the similar prefix, this is a reinforcement-learning transfer framework rather than a multi-task optimization method.

A second unrelated usage appears in the algebraic and topological literature, where “Rep-MTL” names a representation theorem for srDL- and MTL-algebras. In that setting, the theorem constructs the extended Priestley dual of an srDL-algebra from the Stone space of its Boolean skeleton, the GMTL-space of its radical, and a family of closure and action maps, yielding an isomorphism between the algebra and the clopen up-sets of a rotation space (Fussner et al., 2018). This sense belongs to many-valued logic and duality theory rather than machine learning.

For contemporary arXiv discussions of multi-task learning, Rep-MTL therefore refers specifically to the representation-level task saliency framework with TSR and CSA (Wang et al., 28 Jul 2025). Its technical significance lies in treating multi-task interaction as a property of the shared representation geometry, and in showing empirically that such geometry can be regularized without altering the underlying optimizer.