Cross-Task Recalibration in Multi-Task Learning

• Cross-task recalibration is the process of explicitly modeling inter-task influences to mitigate negative transfer and knowledge conflicts in multi-task settings.
• Recent approaches utilize selective task grouping, top-down feature modulation, and trust-region projection to optimize shared and task-specific parameter updates.
• Empirical studies show that these recalibration strategies improve performance metrics by aligning gradient directions and reducing interference across tasks.

Cross-task recalibration, as an Editor’s term, denotes explicit mechanisms for adjusting how multiple tasks influence a shared model, a merged model, or an adaptation process so that inter-task structure is modeled rather than left to naïve joint updates. Across recent work, the motivating pathologies recur under different names: negative transfer in multi-task learning, knowledge conflicts in model merging, unsynchronized task behavior during test-time training, and biased optimization with catastrophic forgetting in federated continual learning. Representative formulations include selective grouping during joint optimization, top-down feature modulation, trust-region projection of task vectors, label-free LoRA merge-weight calibration, masked-latent synchronization under domain shift, and server-side adaptive model recalibration (Jeong et al., 17 Feb 2025, Levi et al., 2020, Sun et al., 25 Jan 2025, Lee et al., 27 Mar 2026, Jeong et al., 10 Jul 2025, Qi et al., 23 Mar 2025, Jeong et al., 2024).

1. Problem setting and conceptual scope

In multi-task learning, a single network parameterized by $\Theta=(\Theta_s,\Theta_1,\ldots,\Theta_K)$ simultaneously minimizes task-specific losses $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$, often through an objective of the form

$$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$

The central difficulty is that different tasks may “pull” the shared parameters $\Theta_s$ in conflicting directions, so improving one task’s loss can degrade another’s; this is the canonical formulation of negative transfer (Jeong et al., 17 Feb 2025).

The same structural issue appears in other regimes. In training-free model merging, the addition of one task vector can harm another task after merging; this is formalized as knowledge conflicts (Sun et al., 25 Jan 2025). Under domain shift, conventional test-time training may adapt one task in a way that does not align with the requirements of other tasks, producing unsynchronized task behavior (Jeong et al., 10 Jul 2025). In federated continual learning, asynchronous task streams can produce biased optimization while the absence of raw-data replay exacerbates catastrophic forgetting (Qi et al., 23 Mar 2025).

A recurring theme is that the locus of recalibration differs across methods. Some methods recalibrate optimization over shared and task-specific parameters; others recalibrate feature activations, task vectors, LoRA merge weights, masked latent predictions, or hypernetwork updates. This suggests that cross-task recalibration is not a single algorithmic primitive but a family of interventions defined by where inter-task structure is enforced.

2. Optimization-time recalibration in joint multi-task training

A direct optimization-time formulation is given by "Selective Task Group Updates for Multi-Task Optimization" (Jeong et al., 17 Feb 2025). Instead of asking how much each task should influence the shared network, the method asks which subsets of tasks should be updated together, and in what order. In each mini-batch, the $K$ tasks are dynamically partitioned into $M$ groups $G_1,\ldots,G_M$, and the model performs $M$ sequential backpropagation steps, one per group, updating both shared and task-specific parameters. The central compatibility measure is proximal inter-task affinity:

$$B_{G\to k}^t = 1-\frac{L_k(z^t,\Theta_{s|G}^{t+1},\Theta_k^{t+1})} {L_k(z^t,\Theta_s^t,\Theta_k^t)}.$$

Because both $\Theta_s$ and $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$0 are updated, this quantity is intended to capture the immediate influence of a group’s update on another task more faithfully than affinity measures that only inspect $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$1. Positive affinity triggers merging, while negative mutual affinity within a group triggers splitting. On an 11-task benchmark, the grouping typically settles on an average $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$2 groups, with the paper reporting $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$3. The theoretical analysis connects higher affinity to better gradient alignment and shows that sequential group updates still descend a weighted sum of group losses, converging to Pareto-stationary points. Empirically, on Taskonomy’s 11-task suite with ViT-L, joint SGD scores $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$4 in $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$5 while the selective grouping approach attains $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$6; on NYUD-v2 and PASCAL-Context the reported gains are 3–5 percentage points in $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$7.

A related but distinct formulation appears in "Quantifying Task Priority for Multi-Task Optimization" (Jeong et al., 2024). That work argues that earlier gradient-manipulation methods are sub-optimal because they cannot accurately determine the individual contributions of each parameter across tasks. It introduces task priority through connection strength, computed from the magnitude of parameters on task-specific back-propagation paths, and organizes optimization into two phases. In Phase 1, sequential gradient steps are used to learn priority structure. In Phase 2, the method conserves this priority by grouping channels according to the task with the highest normalized connection strength and projecting conflicting gradients away from the reference gradient:

$$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$8

The paper reports $$L_1(\Theta_s,\Theta_1),\ldots,L_K(\Theta_s,\Theta_K)$$9 on NYUD-v2 with HRNet-18, compared with a baseline GD value of $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$0, and $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$1 on PASCAL-Context with HRNet-18, compared with $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$2 for GD. Theoretical claims are framed in terms of recovering new Pareto-optimal solutions beyond those reached by prior approaches.

Taken together, these two works shift optimization-time recalibration beyond scalar loss weighting and beyond gradient surgery confined to shared parameters. One emphasizes adaptive grouping and update order; the other emphasizes parameter-level priority and conflict projection.

3. Cross-task feature recalibration by top-down control

"Multi-Task Learning by a Top-Down Control Network" introduces a feature-space realization of cross-task recalibration through a dedicated control network (Levi et al., 2020). The architecture contains a main recognition network BU2, a second bottom-up stream BU1, and an inverted top-down stream TD. The input image is fed to BU1 and BU2 in parallel. BU1 produces intermediate feature maps $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$3, while a one-hot task vector is linearly embedded into a task embedding that initializes TD. At each layer, TD receives an additive lateral input from BU1 and produces spatially varying scale and bias maps $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$4 and $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$5, which modulate BU2 as

$$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$6

This modulation is both channel-wise and spatial-wise, and it depends jointly on image content, spatial location, and the selected task.

The reported interpretation is that the control stream “prunes” or “amplifies” those channels and spatial locations most useful for the current task. Ablations indicate that task-selectivity emerges sharply: when the wrong task code is supplied, off-task activations fall to chance level. The task-selectivity matrices show a selectivity index of approximately 37 for 9-class MNIST, whereas channel-wise modulation has an index of approximately 8. When trained with an auxiliary localization loss, the TD stream also learns spatial masks that highlight task-relevant regions.

The empirical results situate recalibration as an architectural alternative to task branching. On Multi-MNIST by-location with 9 tasks, Branching / uniform multi-loss reaches 74.8%, channel-wise modulation 76.6%, and ControlNet 88.1%, with relative parameter count $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$7 a baseline LeNet. On CLEVR with 40 tasks, channel-wise modulation reaches 87.1% and ControlNet 96.8%; with 1645 tasks, the corresponding numbers are 60.4% and 88.8%. On Celeb-A with 40 attributes, channel-wise modulation reaches 90.06% and ControlNet 90.46%. The method therefore treats recalibration not as a correction applied after learning, but as a layerwise task-conditioned transformation of the shared backbone throughout the forward pass.

4. Parameter-space recalibration in training-free model and adapter merging

A different line of work treats cross-task recalibration as a parameter-space operation performed without additional gradient-based multi-task training. "Task Arithmetic in Trust Region" starts from standard task arithmetic, in which task vectors are defined by $$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$8 and merged as

$$\Theta^*=\arg\min_\Theta \sum_{i=1}^K w_i\,L_i(\Theta_s,\Theta_i).$$9

The paper formalizes knowledge conflict as the performance drop on task $\Theta_s$0 caused by adding task $\Theta_s$1 and attributes the conflict primarily to components of task vectors aligned with gradients of task-specific losses at $\Theta_s$2 (Sun et al., 25 Jan 2025). To restrict merging to safe directions, it defines a trust region through constraints

$\Theta_s$3

where $\Theta_s$4, and projects each task vector into that region. In closed form, when $\Theta_s$5,

$\Theta_s$6

The practical algorithm uses feature-based approximations and layer-wise projections. On eight vision-classification tasks with ViT-B/32, plain Task Arithmetic reports average accuracy 69.1 and the full trust-region clipped method reports 76.1; on ViT-L/14, the corresponding values are 84.5 and 86.7.

"Label-Free Cross-Task LoRA Merging with Null-Space Compression" applies a related recalibration logic to LoRA adapters, again without labels and without output-based surrogates (Lee et al., 27 Mar 2026). A LoRA update is written as $\Theta_s$7, and the key signal is the null space of the down-projection factor $\Theta_s$8. For a feature vector $\Theta_s$9, the per-layer null-space ratio is

$K$0

Averaging over LoRA-equipped layers and unlabeled samples yields $K$1, which decreases as fine-tuning proceeds; the paper reports a tight inverse correlation between $K$2 and validation loss or accuracy across both classification and regression. Merge weights $K$3 are then optimized so that the merged LoRA update

$K$4

minimizes the average null-space ratio over tasks, using only unlabeled inputs and feature activations. The reported results are 92.0% average normalized performance on twenty heterogeneous vision tasks, 92.3% on six NLI benchmarks, and 82.7% on six VLM tasks. Ablations further report that applying NSC to just the last quarter of attention layers, or only the output projection, retains more than 99% of full-model performance while cutting compute by 75%, and that first-token scoring in the VLM setting matches full-sequence performance at 82.7% while being 7× faster.

These two works show that recalibration can be performed entirely in parameter space. In TATR, the relevant object is the task vector and its gradient-aligned components. In NSC, it is the geometry of LoRA down-projections and the preservation of adapter signal across heterogeneous task types.

5. Synchronizing tasks during test-time adaptation

Under domain shift, cross-task recalibration has been formulated as a test-time objective rather than a training-time or merging-time procedure. "Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Training" defines a shared encoder representation $K$5, task-specific latents $K$6, and masked versions $K$7 generated with the same mask across tasks (Jeong et al., 10 Jul 2025). The core discrepancy is

$K$8

where $K$9 is a divergence such as KL or $M$0. The stated goal is to drive $M$1 at test time so that all tasks adapt in lock-step.

The auxiliary mechanism is the Task Behavior Synchronizer (TBS), a small Vision-Transformer trained on the source domain to predict all main-task labels from masked task latents. At test time, labels are unavailable, so the method minimizes a pseudo-label prediction loss between TBS outputs from masked latents and the main network outputs from full latents:

$M$2

The encoder, the task projections, and the TBS are updated using this objective. Proposition 1 provides an upper-bound argument in which minimizing masked-versus-unmasked divergence reduces an upper bound on the true supervised loss in the target domain under two assumptions: preservation of task relation across source and target, and sufficient TBS training.

The reported benchmarks are Taskonomy $M$3 NYUD-v2 and Taskonomy $M$4 PASCAL-Context. For Taskonomy $M$5 NYUD-v2, the summary table gives Baseline at SemSeg mIoU 29.3, Depth RMSE 1.179, Normal mErr 61.3, Edge RMSE 0.1443, and $M$6; TENT yields 40.4, 1.056, 56.0, 0.1441, and $M$7; TIPI yields 48.1, 1.029, 55.7, 0.1440, and $M$8; S4T yields 59.3, 1.053, 45.3, 0.1441, and $M$9. Synchronization-specific trajectory measures are also reported: Step Variance, Dynamic-Time-Warping, and Cosine-Similarity. S4T yields the lowest SV, the lowest DTW, and the highest CS, indicating that adaptation trajectories across tasks become more tightly coupled.

6. Recalibration in federated continual learning

In federated continual learning, recalibration is applied to server-side updates under asynchronous task streams. "Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for FCL" proposes a Dynamic Allocation Hypernetwork, $G_1,\ldots,G_M$0, that maps a task identity $G_1,\ldots,G_M$1 to model weights, together with an Adaptive Model Recalibration (AMR) module that prevents historical task mappings from being overwritten while fusing repeated observations of the same task according to similarity (Qi et al., 23 Mar 2025).

For a new client model $G_1,\ldots,G_M$2 with task identity $G_1,\ldots,G_M$3, the hypernetwork first minimizes

$G_1,\ldots,G_M$4

which yields a candidate change $G_1,\ldots,G_M$5. To preserve historical mappings, the method adds

$G_1,\ldots,G_M$6

and defines $G_1,\ldots,G_M$7. When the same task reappears later, the server compares the historical prototype $G_1,\ldots,G_M$8 and the new client model $G_1,\ldots,G_M$9 by Jensen–Shannon divergence and sets

$M$0

then uses $M$1 to weight the old and new anchors inside a recalibration loss $M$2.

The role of AMR is stated in three ways. The regularization $M$3 penalizes candidate changes that move historical task outputs away from stored prototypes, mitigating forgetting. The same regularizer also suppresses conflicting gradient components when different clients contribute asynchronous tasks, thereby correcting biased optimization. The similarity weight $M$4 determines how much emphasis to place on the old prototype and the new client update for the same task at different time steps.

All experiments are on the AMOS abdominal CT segmentation benchmark with 15 organs and four clients with dynamic, asynchronous task streams. Global mean Dice comparisons report FedAvg at 0.018 average, FBL at 0.209, FedWeIT at 0.697, FedSpace at 0.750, FedDAH–DAHyper at 0.695, FedDAH–$M$5 at 0.327, FedDAH–$M$6 at 0.442, FedDAH full at 0.806, Local Only at 0.616, and Centralized at 0.820. The paper highlights that FedDAH improves over FedSpace by +5.4%, that removing $M$7 collapses performance to approximately 0.33, and that removing $M$8 drops performance to approximately 0.44. In this setting, cross-task recalibration is inseparable from continual retention and server-side aggregation.

7. Conceptual distinctions, misconceptions, and open directions

A common misconception is that cross-task recalibration is equivalent to loss reweighting or gradient surgery on shared parameters. The surveyed work does not support that restriction. Selective task grouping explicitly targets the coupled learning of shared and task-specific parameters rather than shared parameters alone (Jeong et al., 17 Feb 2025). Top-Down Control Networks recalibrate intermediate feature maps by spatially varying scale and bias fields conditioned on task and image content (Levi et al., 2020). TATR and NSC operate without further joint training and relocate recalibration to parameter-space merging (Sun et al., 25 Jan 2025, Lee et al., 27 Mar 2026). S4T places the mechanism at test time under domain shift, and FedDAH places it on the server side of federated continual learning under asynchronous task streams (Jeong et al., 10 Jul 2025, Qi et al., 23 Mar 2025).

A second misconception is that recalibration always requires labels or direct access to model outputs. NSC is explicitly label-free and output-agnostic, and its objective depends only on unlabeled inputs, feature activations, and the geometry of LoRA down-projections (Lee et al., 27 Mar 2026). TATR is training-free and reuses only the pre-trained model, the fine-tuned task parameters, and a small number of unlabeled exemplars per task (Sun et al., 25 Jan 2025). By contrast, S4T does rely on a source-trained synchronization branch and pseudo-labels from the main network at test time, illustrating that recalibration strategies differ sharply in supervision assumptions (Jeong et al., 10 Jul 2025).

Open problems are likewise heterogeneous. For top-down control, the optimal depth and width of the TD stream are not fully explored; extending the method beyond classification to segmentation, detection, or image generation remains open, as does a recurrent BU–TD feedback loop (Levi et al., 2020). For task-priority optimization, extending connection-strength definitions beyond ConvNets to transformers or dynamic graph architectures is identified as a limitation, and sufficient conditions for nonconvex Pareto-optimality remain open (Jeong et al., 2024). For S4T, extension to sequential tasks such as video or to continual-learning settings is proposed through temporal or hierarchical dependencies (Jeong et al., 10 Jul 2025). For selective task grouping, the main trade-off is computational: each batch requires $M$9 forward/backward passes rather than one, although the reported runtime remains between loss-based and gradient-based methods because typically $$B_{G\to k}^t = 1-\frac{L_k(z^t,\Theta_{s|G}^{t+1},\Theta_k^{t+1})} {L_k(z^t,\Theta_s^t,\Theta_k^t)}.$$0 (Jeong et al., 17 Feb 2025).

The literature therefore presents cross-task recalibration not as a settled doctrine but as a unifying perspective on how task relations can be made operational. The specific operational form may be sequential grouping, channel-and-spatial modulation, trust-region projection, null-space-based merge weighting, masked-latent synchronization, or hypernetwork regularization. What remains constant is the premise that multi-task performance improves when inter-task relations are modeled explicitly rather than treated as incidental by-products of a shared architecture.