Constrained Multi-Task Representation Learning

• Constrained Multi-Task Representation Learning (CMTRL) is a family of methods that imposes explicit constraints like compute budgets, sparsity, and safety to manage cross-task interference and resource usage.
• It encompasses diverse mechanisms—such as slimmable encoder-decoders, structured sparsity, contrastive modularization, and low-rank formulations—across supervised, reinforcement, and decentralized settings.
• Empirical evidence shows that CMTRL methods can outperform dense baselines by reducing sample complexity and ensuring safety while maintaining high performance on multiple benchmarks.

Searching arXiv for the cited CMTRL-related papers and topic variants to ground the article in current literature. Constrained Multi-Task Representation Learning (CMTRL) denotes a class of multi-task learning formulations in which a shared representation is learned under explicit constraints on capacity, sparsity, module diversity, safety, rank, or communication. In the cited literature, these constraints appear as a total compute budget $B$ and relative task-importance weights $\{\alpha_t\}$ for slimmable encoder-decoder models, structured or group sparsity for selecting task-specific branching depths, contrastive separation of shared modules in multi-task reinforcement learning, stage-wise conservativeness constraints in linear bandits, and hard low-rank constraints in decentralized federated regression (Aich et al., 2023, Upadhyay et al., 2024, Lan et al., 2023, Lin et al., 12 May 2026, Kang et al., 29 Dec 2025). Taken together, the papers suggest that CMTRL is best understood not as a single canonical algorithm, but as a family of constraint-aware representation-learning schemes for controlling negative transfer, deployment cost, sample complexity, or safety.

1. Formal scope and problem statements

A prominent supervised formulation uses a shared encoder and task-specific decoders whose channel widths are slimmable. For an $N$-task problem, each input image $x$ produces outputs $\{y^{(1)},\dots,y^{(N)}\}$, and the super-model $\mathcal{S}$ consists of encoder weights $\theta_{\mathrm{enc}}$ and decoder weights $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$. A runtime configuration is written as

$$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$

where each width multiplier $\omega\in(0,1]$ retains $\{\alpha_t\}$0 channels from a full layer of width $\{\alpha_t\}$1. User constraints are a total compute budget $\{\alpha_t\}$2 and relative task-importance weights $\{\alpha_t\}$3, and the weighted multi-task loss is

$\{\alpha_t\}$4

with inference-time optimization posed as minimizing the weighted loss subject to $\{\alpha_t\}$5 (Aich et al., 2023).

A second supervised formulation, identified with Layer-Optimized Multi-Task (LOMT), begins from structured sparsity over backbone groups. With backbone parameters $\{\alpha_t\}$6, task-specific head parameters $\{\alpha_t\}$7, and disjoint groups $\{\alpha_t\}$8, the penalty is

$\{\alpha_t\}$9

Phase 1 solves, separately for each task,

$N$0

and Phase 2 constructs a hard-shared multi-task model by attaching task $N$1's decoder at the deepest layer $N$2 whose channels remain nonzero. Joint optimization then uses uncertainty weighting,

$N$3

following Kendall et al. (2018) (Upadhyay et al., 2024).

In multi-task reinforcement learning, CMTRL appears as constrained modular representation learning. CMTA uses a Mixture-of-Experts with $N$4 encoders $N$5, a task-specific encoder $N$6, and a unidirectional LSTM producing temporal state $N$7. Attention weights

$N$8

combine expert outputs as $N$9, and the final representation is $x$0. The task loss is

$x$1

and the multi-task objective averages this over tasks (Lan et al., 2023).

The term CMTRL is used explicitly as a framework in conservative linear bandits. There are $x$2 stochastic linear bandit tasks in $x$3, each with unknown parameter $x$4, reward

$x$5

and stage-wise safety constraint

$x$6

where $x$7 is the known baseline reward. Shared structure is imposed through a rank-$x$8 matrix

$x$9

and the objective is to minimize cumulative regret subject to safety (Lin et al., 12 May 2026).

A related decentralized formulation studies multi-task linear regression with shared low-rank structure. Task $\{y^{(1)},\dots,y^{(N)}\}$0 has data $\{y^{(1)},\dots,y^{(N)}\}$1, and the common objective is

$\{y^{(1)},\dots,y^{(N)}\}$2

or, equivalently, a rank-constrained formulation with $\{y^{(1)},\dots,y^{(N)}\}$3. Here the constraint is both representational and distributed: recovery must proceed in a diffusion-based decentralized and federated fashion (Kang et al., 29 Dec 2025).

2. Constraint taxonomy

The cited work spans several distinct kinds of constraints.

Domain	Constraint	Core mechanism
Supervised MTL	Total compute budget $\{y^{(1)},\dots,y^{(N)}\}$4 and relative task-importance weights $\{y^{(1)},\dots,y^{(N)}\}$5	Slimmable shared encoder and task-specific decoders; CIKD; runtime search
Supervised MTL	Structured or group sparsity	Sparse single-task probing; deepest active layer $\{y^{(1)},\dots,y^{(N)}\}$6; LOMT branching
Multi-task RL	Module distinctiveness	InfoNCE-style contrastive loss and temporal attention
Conservative linear bandits	Stage-wise safety and rank-$\{y^{(1)},\dots,y^{(N)}\}$7 structure	Safe-AltGDmin with safe exploration and greedy epochs
Decentralized federated regression	Rank-$\{y^{(1)},\dots,y^{(N)}\}$8 structure and decentralized consensus	Dif-AltGDmin with diffusion and QR projection

These formulations constrain different objects: channel widths, layer activity patterns, expert functions, admissible actions, or feasible parameter subspaces. This suggests that CMTRL is organized less by application area than by the location of the constraint in the learning stack. In some papers the constraint is architectural, as in width multipliers or task-specific branch points; in others it is geometric, as in low-rank factorization; in still others it is operational, as in stage-wise safety or communication-limited diffusion (Aich et al., 2023, Upadhyay et al., 2024, Lan et al., 2023, Lin et al., 12 May 2026, Kang et al., 29 Dec 2025).

A related distinction concerns what is being protected. Compute-budget formulations protect deployability. Structured sparsity and contrastive modularization protect against negative transfer or representational collapse. Conservative bandits protect safety at every round. Decentralized low-rank recovery protects communication efficiency while preserving identifiability. The common feature is that the representation is not merely shared; it is shared under conditions that restrict how cross-task information can be used.

3. Core algorithmic patterns

In controllable slimmable MTL, training aims to make every sub-architecture in a predefined set $\{y^{(1)},\dots,y^{(N)}\}$9 useful after a single training run. The objective is

$\mathcal{S}$0

with Sandwich-Rule sampling of one largest configuration, one smallest configuration, and several random intermediate configurations. Configuration-Invariant Knowledge Distillation (CIKD) enforces representation invariance by comparing channel-averaged encoder feature maps from the full-width “parent” and slimmer “children,” without back-propagating into the teacher path. At inference time, a two-stage search maps user constraints to width settings: decoder widths are assigned by

$\mathcal{S}$1

and then an evolutionary loop searches encoder widths within the compute budget using an offline-trained accuracy predictor $\mathcal{S}$2 (Aich et al., 2023).

In LOMT, the first phase is sparse single-task learning with proximal-gradient updates,

$\mathcal{S}$3

and closed-form group-thresholding zeroes out trivial channel groups. After convergence, each backbone layer is inspected; if every channel in layer $\mathcal{S}$4 is driven exactly to zero, the layer is marked inactive, and $\mathcal{S}$5 is chosen as the deepest layer not marked inactive for task $\mathcal{S}$6. The second phase builds a hard-shared network whose backbone runs full depth but whose task-specific decoders branch at their task-specific $\mathcal{S}$7 values. Training then proceeds with learned uncertainty weights $\mathcal{S}$8 (Upadhyay et al., 2024).

CMTA combines modular decomposition with temporal routing. Expert $\mathcal{S}$9 produces query $\theta_{\mathrm{enc}}$0, the positive key is $\theta_{\mathrm{enc}}$1, and negatives are the other experts' outputs at time $\theta_{\mathrm{enc}}$2. The per-expert InfoNCE-style term is

$\theta_{\mathrm{enc}}$3

Temporal attention uses the LSTM hidden state and task identity to compute expert weights at every time step, so routing is finer-grained than task-level module assignment. Optimization uses an off-policy SAC backbone, a shared replay buffer of size $\theta_{\mathrm{enc}}$4, batch size $\theta_{\mathrm{enc}}$5, Adam learning rates $\theta_{\mathrm{enc}}$6, discount $\theta_{\mathrm{enc}}$7, and best-run $\theta_{\mathrm{enc}}$8; training runs for $\theta_{\mathrm{enc}}$9 M steps per task with evaluation every $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$0 steps and averaging over $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$1 seeds (Lan et al., 2023).

Safe-AltGDmin for conservative bandits divides the horizon into $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$2 epochs via a doubling schedule. Epoch $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$3 performs safe exploration using actions of the form

$\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$4

with $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$5, while later epochs use greedy actions

$\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$6

After each epoch, low-rank estimation alternates between ordinary least squares in $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$7 and projected gradient steps in $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$8, with a QR-based orthonormalization. Initialization is spectral, based on a truncated sum $\{\theta_{\mathrm{dec}^{(t)}}\}_{t=1}^N$9, and the data are partitioned into $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$0 disjoint subsets to decouple noise across gradient steps (Lin et al., 12 May 2026).

Dif-AltGDmin follows the same alternating projected gradient descent and minimization pattern, but in a diffusion-based decentralized setting. On node $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$1, the min-step solves

$$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$2

for each local task. The local gradient

$$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$3

is followed by a scaled GD step, diffusion consensus using a doubly-stochastic matrix $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$4, and QR projection to restore orthonormality. The algorithm is designed so that one consensus step per GD iteration suffices rather than many (Kang et al., 29 Dec 2025).

4. Empirical behavior across domains

On standard supervised MTL benchmarks, controllable slimmable architectures report improvements in controllability and memory overhead. On NYUD-v2 with a ResNet-34 backbone and budgets of $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$5 GMACs versus $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$6 GMACs, the reported hypervolume values at $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$7 GMACs are $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$8 and $$\omega=\bigl(\omega_{\mathrm{enc}},\omega_{\mathrm{dec}^{(1)}},\dots,\omega_{\mathrm{dec}^{(N)}}\bigr),$$9, a $\omega\in(0,1]$0 pp gain. On PASCAL-Context with MobileNetV2, at approximately $\omega\in(0,1]$1 GMACs the reported values are $\omega\in(0,1]$2 versus $\omega\in(0,1]$3 for CDMA, a $\omega\in(0,1]$4 pp difference. On CIFAR100-MTL with ResNet-9, at approximately $\omega\in(0,1]$5 GMACs the reported values are $\omega\in(0,1]$6 versus $\omega\in(0,1]$7, described as on par, but with only $\omega\in(0,1]$8 M extra control parameters versus CDMA’s $\omega\in(0,1]$9 M (Aich et al., 2023).

For structured-sparsity-based LOMT, the NYU-v2 benchmark reports averages over $\{\alpha_t\}$00 seeds for four tasks: semantic segmentation, depth, normals, and edge. Dense MTL yields Seg. IoU $\{\alpha_t\}$01, Depth MAE $\{\alpha_t\}$02, Norms CS $\{\alpha_t\}$03, and Edge MAE $\{\alpha_t\}$04, while LOMT reports $\{\alpha_t\}$05, $\{\alpha_t\}$06, $\{\alpha_t\}$07, and $\{\alpha_t\}$08, respectively. On CelebAMask-HD, dense MTL reports Seg. IoU $\{\alpha_t\}$09 and average attribute accuracy approximately $\{\alpha_t\}$10, while LOMT reports Seg. IoU $\{\alpha_t\}$11 and average attribute accuracy approximately $\{\alpha_t\}$12 (Upadhyay et al., 2024).

In Meta-World multi-task RL, CMTA reports “max smoothed” success rates of $\{\alpha_t\}$13 on MT10-Fixed, $\{\alpha_t\}$14 on MT10-Mixed, $\{\alpha_t\}$15 on MT50-Fixed, and $\{\alpha_t\}$16 on MT50-Mixed. These are compared with CARE at $\{\alpha_t\}$17, $\{\alpha_t\}$18, $\{\alpha_t\}$19, and $\{\alpha_t\}$20, respectively, and with single-task SAC at $\{\alpha_t\}$21 on MT10-Fixed and $\{\alpha_t\}$22 on MT10-Mixed. Removing contrastive loss lowers MT10-Mixed from $\{\alpha_t\}$23 to $\{\alpha_t\}$24 and MT50-Mixed from $\{\alpha_t\}$25 to $\{\alpha_t\}$26, while removing temporal context yields a further approximately $\{\alpha_t\}$27–$\{\alpha_t\}$28 drop in the mixed setting (Lan et al., 2023).

In conservative linear bandits, Safe-AltGDmin is compared with trace-norm convex relaxation, per-task Thompson Sampling with safe set elimination, and Method-of-Moments alternating estimation. On synthetic data with $\{\alpha_t\}$29, $\{\alpha_t\}$30, $\{\alpha_t\}$31, $\{\alpha_t\}$32 actions per task, $\{\alpha_t\}$33 rounds in $\{\alpha_t\}$34 epochs, and $\{\alpha_t\}$35 averaged trials, the reported findings are that regret grows sublinearly, estimation error $\{\alpha_t\}$36 is smaller by an order of magnitude versus baselines, cumulative regret satisfies $\{\alpha_t\}$37 trace-norm, MoM, and only CMTRL and TS achieve zero constraint violations. On Movielens-100K, CMTRL attains the lowest regret while never violating safety (Lin et al., 12 May 2026).

For decentralized low-rank multi-task regression, the reported empirical evidence consists of numerical simulations validating the performance of Dif-AltGDmin and comparisons with benchmark algorithms. The emphasis is less on benchmark tables than on constructive guarantees and the claim that the method is fast and communication-efficient in the diffusion-based decentralized setting (Kang et al., 29 Dec 2025).

5. Theoretical properties and representation effects

Several papers articulate representational effects that are specific to the imposed constraint. CIKD is intended to make shared-encoder representations independent of runtime width configurations, and its reported role is to avoid catastrophic performance drops in the smallest submodels. The structured sparsity phase in LOMT is described as probing how deep each task truly needs to go in a shared encoder, with the result that tasks need not compete over irrelevant deep features. In CMTA, the contrastive term prevents module collapse and redundancy, while temporal attention permits different sub-phases of a task to recruit different skills, addressing both across-task conflicts and within-task conflicts (Aich et al., 2023, Upadhyay et al., 2024, Lan et al., 2023).

The conservative bandit formulation supplies the strongest theorem-level guarantees among the cited works. If $\{\alpha_t\}$38, the step size is chosen as

$\{\alpha_t\}$39

and the epoch lengths and iteration counts satisfy the stated lower bounds, then with probability at least $\{\alpha_t\}$40 all actions in every epoch satisfy the safety constraint. Theorems 2 and 3 give exponential subspace contraction with rates $\{\alpha_t\}$41 in epoch $\{\alpha_t\}$42 and $\{\alpha_t\}$43 in later epochs. Theorem 4 states

$\{\alpha_t\}$44

and in particular

$\{\alpha_t\}$45

which scales with the latent rank $\{\alpha_t\}$46 rather than $\{\alpha_t\}$47. The paper also states that sharing $\{\alpha_t\}$48 reduces per-task sample complexity from $\{\alpha_t\}$49 to $\{\alpha_t\}$50 (Lin et al., 12 May 2026).

The decentralized regression formulation provides an informal theorem of exactness and complexity. With probability at least $\{\alpha_t\}$51, after

$\{\alpha_t\}$52

$\{\alpha_t\}$53

and

$\{\alpha_t\}$54

the output on every node satisfies $\{\alpha_t\}$55, and each task-parameter error is at most $\{\alpha_t\}$56. The total sample requirement is

$\{\alpha_t\}$57

whose leading term $\{\alpha_t\}$58 is described as near-optimal for recovering a rank-$\{\alpha_t\}$59 $\{\alpha_t\}$60 matrix. Computation per GD iteration is $\{\alpha_t\}$61 overall, and the communication analysis is explicit in $\{\alpha_t\}$62, node degree, and spectral gap. A notable claim is that the diffusion-based scheme has a much weaker requirement on $\{\alpha_t\}$63 than prior Dec-AltGDmin, so sparser graphs suffice (Kang et al., 29 Dec 2025).

These results point to a recurring CMTRL pattern: explicit constraints are used to convert vague desiderata such as “task specificity,” “safety,” or “efficiency” into optimization variables and feasible sets that admit either direct search procedures or finite-sample guarantees.

6. Limitations, misconceptions, and open directions

A common misconception is to treat CMTRL as synonymous with one specific mechanism, usually low-rank factorization or pruning. The cited literature shows a broader picture: CMTRL may refer to slimmable width control, structured sparsity and task-specific branch selection, contrastive modular separation with temporal routing, conservative action constraints in bandits, or decentralized low-rank projected optimization (Aich et al., 2023, Upadhyay et al., 2024, Lan et al., 2023, Lin et al., 12 May 2026, Kang et al., 29 Dec 2025). Another misconception is that constraints are introduced only to trade performance for feasibility. In the reported experiments, constrained designs often outperform dense or less constrained baselines, although the effect depends on the benchmark and the nature of the constraint.

The limitations are correspondingly heterogeneous. The slimmable architecture work states that it has only been tested on CNN backbones; adapting to Vision Transformers would require slim-attention heads or MLP expansions. It also states that task correlations or conflicts are not explicitly modeled during search, and that the evolution search still carries a small runtime cost if user constraints change frequently. The structured sparsity work notes that adding a new task requires one more sparse run and a new branch, and it reports that larger gains in Phase 2 arise when the initial per-task sparsity pattern is stronger. In conservative bandits, future work is identified around unknown baseline rewards, non-Gaussian contexts, general convex safety constraints, non-linear shared representations via deep networks and safe policy-gradient methods, and distributed variants. In decentralized regression, a centralized simplification is available by replacing diffusion with a fusion center, which removes dependence on network connectivity.

Open technical directions are stated with unusual clarity. For controllable MTL, the paper asks whether the discrete search can be made gradient-based via a continuous surrogate such as Gumbel-softmax, and how to incorporate resource elasticity without re-running the evolution loop, possibly through a hypernetwork mapping $\{\alpha_t\}$64. For structured sparsity, the extension recipe is to choose a backbone, define group partitions, grid-search $\{\alpha_t\}$65, extract $\{\alpha_t\}$66, and retrain the LOMT model with uncertainty weighting. For safe low-rank CMTRL, the proposed extensions include constrained contextual bandits and representation learning in sequential decision-making beyond linear bandits. For decentralized CMTRL, the principal open systems question concerns how low-rank sharing, consensus speed, and communication efficiency interact as the network becomes sparse.