Gradient-Based Task Analysis

• Gradient-based task analysis is a framework that uses gradients, update vectors, and task vectors to measure task compatibility, conflict, and latent structure.
• Techniques such as cosine similarity, per-task clipping, and gradient decomposition are employed to diagnose gradient conflicts and balance multi-task learning.
• The approach also benefits meta-learning, task grouping, and continual learning by leveraging local optimization geometry for efficient task representation and transfer.

Gradient-based task analysis denotes the use of gradients, gradient-derived update vectors, and local optimization geometry to characterize tasks, quantify compatibility or interference among them, and control shared learning dynamics. In the cited literature, the analyzed object may be a task gradient on shared parameters, a task update vector in meta-learning, a task vector produced by finetuning, a gradient embedding used for task identification, or a low-dimensional projection of per-sample gradients. These signals are used to diagnose gradient conflict, detect gradient norm bias, estimate pairwise or higher-order task affinity, infer latent task identity, and approximate multitask optimization without exhaustive retraining (Zhang et al., 2023, Eshratifar et al., 2018, Li et al., 2024, Hummos et al., 2024).

1. Formal objects and analytical primitives

A common starting point is the task gradient on shared parameters. For task $k$, one formulation uses

$\mathbf{g}_k = \nabla_\theta L_k,$

with pairwise affinity estimated by cosine similarity

$$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$

In that formulation, $G_{ij} > 0$ indicates synergy or shared mechanisms, $G_{ij} < 0$ indicates conflict, and $G_{ij} \approx 0$ indicates little shared structure (Zhang et al., 9 Apr 2026).

A second primitive is the task update vector. In meta-learning, after inner-loop adaptation from $\theta$ to $\theta_i$, Gradient Agreement defines

$g_i = \theta - \theta_i,$

and weights each task by its inner-product agreement with the batch: $$w_i = \frac{\sum_{j \in T} (g_i^T g_j)}{\sum_{k \in T}\left|\sum_{j \in T}(g_k^T g_j)\right|}, \qquad \sum_{i\in T}|w_i|=1.$$ The outer-loop update then becomes a signed, agreement-sensitive aggregation rather than an unweighted average (Eshratifar et al., 2018).

A third primitive is the task vector from finetuning. For task $\mathbf{g}_k = \nabla_\theta L_k,$0, the task vector after $\mathbf{g}_k = \nabla_\theta L_k,$1 epochs is

$\mathbf{g}_k = \nabla_\theta L_k,$2

Under full-batch gradient descent, the one-epoch case is exact: $\mathbf{g}_k = \nabla_\theta L_k,$3 For multiple epochs, the cited analysis states that the discrepancy from multitask gradient descent is a curvature-controlled perturbation of order $\mathbf{g}_k = \nabla_\theta L_k,$4, with an explicit bound for feed-forward networks (Zhou et al., 22 Aug 2025).

These formulations differ in their immediate target—compatibility, balancing, inference, or merging—but they share a common premise: task structure is legible in the differential signal induced by each task. This suggests that gradient-based task analysis is less a single method than a family of local geometric probes.

2. Shared-backbone multi-task optimization

In shared-backbone multi-task learning, one recurring failure mode is imbalance among task gradients. TBGC identifies this explicitly as a “gradient norm bias problem”: in the reported setting, detection produces the largest backbone gradient norm, segmentation is intermediate, and classification is smallest, so the backbone becomes “overly biased” toward detection. Figure 1 plots backbone gradient norms over 900 iterations and shows the detection curve consistently above the others. Vanilla clipping is written as

$\mathbf{g}_k = \nabla_\theta L_k,$5

which preserves relative dominance after summation. TBGC instead clips each task independently and then rescales each task so that its backbone gradient norm has the same max-norm scale $\mathbf{g}_k = \nabla_\theta L_k,$6, before summing the rescaled task gradients. The reported ablation moves from 77.50 overall with vanilla clipping to 79.82 with per-task clipping only (TBGC*) and 80.45 with full TBGC; task scores change from 86.78/56.67/89.06 to 87.88/60.17/93.31 for detection/segmentation/classification (Zhang et al., 2023).

Other methods analyze conflict directionally rather than norm-wise. GDOD forms a mini-batch gradient matrix, computes an orthogonal basis for the span of task gradients via SVD, projects each task gradient into that basis, and separates coordinates into task-shared and task-conflict components according to sign consistency across tasks. The update retains only the shared part,

$\mathbf{g}_k = \nabla_\theta L_k,$7

and the paper proves convergence to the optimal value in the convex case and to a stationary point in the non-convex case under $\mathbf{g}_k = \nabla_\theta L_k,$8 (Dong et al., 2023).

CoGrad treats the central quantity not as alignment per se but as inter-task loss reduction. It defines transfer from task $\mathbf{g}_k = \nabla_\theta L_k,$9 to task $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$0 as the reduction in task $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$1's loss caused by a virtual update using task $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$2's gradient: $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$3 Differentiation yields an auxiliary term governed by $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$4, and the coordinated gradient becomes

$$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$5

The Hessian-vector product is approximated by $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$6 with $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$7, which makes the procedure gradient-based and elementwise (Yang et al., 2023).

A separate line of work modifies the shared update by meta-learning-style adaptation. “Multitask Learning with Single Gradient Step Update for Task Balancing” updates shared parameters for each task by a temporary inner step,

$$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$8

then updates the original shared parameters using the sum of losses evaluated after that one-step adaptation: $$G_{ij} = \frac{\mathbf{g}_i \cdot \mathbf{g}_j}{\|\mathbf{g}_i\| \|\mathbf{g}_j\|}.$$9 The paper reports that this balances tasks at the gradient level rather than by direct loss reweighting (Lee et al., 2020).

Gradient analysis has also been coupled to loss-landscape geometry. “Improving Multi-task Learning via Seeking Task-based Flat Regions” adapts Sharpness-aware Minimization to MTL by decomposing each shared SAM gradient into a low-loss component and a flatness component,

$G_{ij} > 0$0

aggregating the two components separately across tasks, and then summing the aggregated directions. The paper reports reductions in gradient conflict and improved robustness to perturbation, together with gains on Multi-MNIST, CelebA, CityScapes, and NYUv2 (Phan et al., 2022).

Taken together, these methods distinguish several analytically different sources of difficulty: unequal backbone norm scale, incompatible directions, poor transfer geometry, and sharp shared minima. A plausible implication is that “task analysis” in optimization should not be reduced to cosine similarity alone.

3. Task affinity estimation and task grouping

Several papers use gradients not to repair joint training after task selection, but to estimate which tasks should be trained together in the first place. A strong version of this idea appears in work on sample overlap. “Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning” argues that gradient conflict analysis has an unstated requirement: tasks must be evaluated on the same training instances for gradient similarity to reflect task structure. Overlap is defined as

$G_{ij} > 0$1

The paper reports a phase transition: below about 30% overlap, gradient-task correlations are weak and not statistically significant; above about 40%, they become strong, with a fitted sigmoid inflection at $G_{ij} > 0$2 and $G_{ij} > 0$3. It recommends $G_{ij} > 0$4 overlap for reliable analysis and reports that standard benchmarks such as MoleculeNet ($G_{ij} > 0$5) and TDC (8–14%) fall far below that regime (Zhang et al., 9 Apr 2026).

Grad-TAG estimates task affinity without repeated retraining. It first trains a base model on all tasks to obtain $G_{ij} > 0$6, then linearizes the model around that point: $G_{ij} > 0$7 The resulting surrogate treats gradients as features in logistic regression, after Johnson–Lindenstrauss projection to low dimension. From the estimated affinity matrix, the method solves a semidefinite program that maximizes average within-cluster density. The reported estimates are within 2.7% distance to the true affinities while using 3% of FLOPs in full training; on a graph with 21M edges and 500 labeling tasks, the paper reports estimates within 5% distance using 112 GPU hours (Li et al., 2024).

A different grouping framework uses sample-wise optima at initialization. “Efficient Task Grouping Through Samplewise Optimisation Landscape Analysis” defines, for sample $G_{ij} > 0$8, a one-step optimum

$G_{ij} > 0$9

and for tasks $G_{ij} < 0$0 a Sample-wise Convergence-based Affinity score

$G_{ij} < 0$1

These affinities are then refined by a graph attention network and clustered by a Gaussian mixture model. The paper reports a five-fold speed enhancement compared to previous state-of-the-art methods across 8 datasets (Thakur et al., 2024).

In NLP auxiliary-task selection, GradTS uses gradients of Transformer attention heads from single-task experiments. For each task, it accumulates absolute gradients at each head, normalizes them, and forms a head-ranking matrix. Task relatedness is then measured by Kendall’s $G_{ij} < 0$2 between the primary-task ranking and each auxiliary-task ranking. GradTS-fg adds instance-level filtering with a threshold of Kendall’s $G_{ij} < 0$3, while GradTS-thres uses a task-level threshold of Kendall’s $G_{ij} < 0$4. On 8 GLUE classification tasks, the paper reports that GradTS-trial costs 6.73% less time than AUTOSEM-p1, and GradTS-fg costs 21.32% less time than AUTOSEM, with comparable GPU usage (Ma et al., 2021).

These results formalize task affinity at several levels: shared-sample gradient alignment, local linearized transfer, sample-wise optimum proximity, and architecture-specific salience patterns. The common theme is that affinity is estimated from local differential behavior rather than from exhaustive joint training.

4. Meta-learning, task representation, and gradient-derived features

In meta-learning, task analysis often appears as weighting or representing tasks by their update geometry. Gradient Agreement is a direct example: tasks whose update vectors align with the batch consensus receive positive weight, while tasks that anti-align may receive negative weight. The paper derives the weighting rule from a regularized bilevel objective and reports improvements over MAML and Reptile, including 54.80% versus 48.70% and 49.97% on miniImageNet 5-way 1-shot, and 73.27% versus 63.11% and 65.99% on 5-way 5-shot (Eshratifar et al., 2018).

Grad2Task represents a few-shot NLP task by Fisher-information-derived gradient features computed with respect to bottleneck adapter parameters, which are described as “< 0.5% of the number of parameters in BERT.” For a support subset, it samples labels from the base model and accumulates squared gradients into a task feature $G_{ij} < 0$5, then encodes per-layer gradient features with an RNN: $G_{ij} < 0$6 An adaptation network converts these task embeddings into FiLM-like scale and shift parameters for the adapters. On the reported benchmark, Grad2Task is best overall, with average scores 44.11, 46.18, and 47.68 in the 4-shot, 8-shot, and 16-shot settings, compared with 42.76, 44.90, and 46.82 for ProtoNet-BN (Wang et al., 2022).

Gradient-based inference of abstract task representations pushes the idea further by treating the task representation itself as a latent variable. The method is cast in a variational inference and expectation-maximization framework, with one-step inference

$G_{ij} < 0$7

where

$G_{ij} < 0$8

The paper reports that, on MNIST, GBI attains 85.46% accuracy, compared with 81.91% for a pure generative model, 88.51% for iterative optimization, and 91.94% for exhaustive likelihood evaluation; for OOD detection, it reports AUCROC $G_{ij} < 0$9, compared with $G_{ij} \approx 0$0 for classifier softmax max and $G_{ij} \approx 0$1 for likelihood regret (Hummos et al., 2024).

Task vectors provide another representation-theoretic bridge. “On Task Vectors and Gradients” proves that one epoch of full-batch gradient descent yields a task vector exactly equal to a negative gradient step, and argues empirically across seven vision benchmarks that the first-epoch gradient dominates the finetuning trajectory in both norm and direction. The paper reports that one-epoch task vectors often perform comparably to fully converged vectors in model merging, and interprets task arithmetic as approximate multitask learning (Zhou et al., 22 Aug 2025).

At the level of individual examples rather than whole tasks, “Gradients as Features for Deep Representation Learning” uses the gradient of the model output with respect to upper backbone parameters as a feature: $G_{ij} \approx 0$2 combined with activation features in a linear model that is a first-order approximation to fine-tuning. The paper reports that the full model improves over activation-only features and, in the ImageNet-pretrained transfer setting, outperforms fine-tuning on VOC07 and COCO2014 (Mu et al., 2020).

Across these formulations, gradients serve as descriptors of task identity, local adaptability, and downstream transfer structure. This suggests a broader reinterpretation of task analysis: not only as conflict diagnosis, but as representation learning over task-conditioned derivatives.

5. Continual learning, replay-free inference, and task-agnostic selection

In continual settings, gradient-based task analysis is often used for task inference when task identity is unavailable at test time. “Efficient Expansion and Gradient Based Task Inference for Replay Free Incremental Learning” addresses class-incremental learning by generating $G_{ij} \approx 0$3 augmented views of a test sample, producing pseudo-labels by majority vote across augmentations, and defining an entropy-weighted loss

$G_{ij} \approx 0$4

For each candidate task model, it computes a gradient embedding and predicts the task by the smallest normalized gradient norm: $G_{ij} \approx 0$5 The method uses the last two convolutional layers and the fully connected layer, with “mean filters” compression reducing gradient cardinality by about 99.98%. The paper reports an absolute gain of 4.1% over the vanilla entropy baseline on CIFAR100/5, and supplementary task-prediction accuracy rising from 57.7 for entropy to 61.9 for the full method (Roy et al., 2023).

Prompt-based continual learning uses similar ideas at the prompt level. GRID computes, for each stored prompt $G_{ij} \approx 0$6, the average gradient norm on the new task: $G_{ij} \approx 0$7 defines a threshold

$G_{ij} \approx 0$8

retains prompts above threshold, and aggregates low-gradient prompts into

$G_{ij} \approx 0$9

The paper reports 66.7% prompt storage reduction, substantial backward-transfer improvements, and reductions in forgotten tasks of roughly 76.7%–83.0%, summarized in the abstract as “up to 80%” (Tiwari et al., 19 Jul 2025).

Gradient-based task analysis has also been extended from continual learning to continual learning-unlearning. UG-CLU defines an oracle solution on the remaining dataset and approximates it by KL-divergence minimization on output distributions. The resulting update is decomposed into four components: remaining-data preservation, learning, unlearning, and saliency modulation. The paper further introduces a remain-preserved manifold constraint and a fast-slow weight adaptation mechanism to approximate Hessian-adjusted directions efficiently, and it evaluates both task-aware and task-agnostic CLU settings (Huang et al., 21 May 2025).

In these settings, gradients are treated not merely as optimization signals but as compatibility tests: the correct task, prompt, or retained knowledge state is the one that requires the least corrective change, or the one whose induced change can be modulated without damaging retained behavior.

6. Assumptions, caveats, and contested interpretations

A central caveat is that alignment is not equivalent to transfer. CoGrad explicitly argues that methods focusing on directional alignment, magnitude balancing, or Pareto-style compromise can be too blunt because shared modules contain both general and specific knowledge; maximizing cosine similarity alone may crowd out task-specific information. Its proposed target is measured cross-task loss reduction rather than raw geometric similarity (Yang et al., 2023).

A second caveat is that gradient affinity estimates may be uninterpretable when tasks are evaluated on disjoint inputs. The sample-overlap analysis states that, in the zero-overlap case, gradient similarity carries no information about true task relationship, and that below roughly 30% overlap the observed correlations are statistically indistinguishable from noise (Zhang et al., 9 Apr 2026).

A third caveat concerns manipulability. “Gradient-based Analysis of NLP Models is Manipulable” shows that a model can be merged with a Facade whose predictions are nearly uniform but whose gradients dominate attribution. The paper defines token attribution as

$\theta$0

and demonstrates manipulation of saliency maps, input reduction, and HotFlip on text classification, NLI, QA, and Biosbias, while leaving predictions essentially unchanged. Integrated Gradients is described as more robust than simple Gradient or SmoothGrad, and black-box methods such as LIME, Anchors, and SHAP are not affected by this specific attack (Wang et al., 2020).

Method-specific caveats also recur. In TBGC, per-task clipping without backbone rescaling already gives “decent performance,” so the backbone-oriented step is helpful but not strictly necessary for improvement; the multi-branch augmentation is only applied to detection and segmentation, not classification; and the paper notes that in the competition context segmentation was not evaluated in Leaderboard B, which gave the team a practical advantage (Zhang et al., 2023). In GradTS, task-ranking thresholds are tuned empirically and the method is built around Transformer attention heads, so transfer to non-Transformer architectures is not directly addressed (Ma et al., 2021).

These caveats delimit the scope of gradient-based task analysis rather than negating it. The cited literature supports a narrower interpretation: gradients can be informative task signals when the measured object, sampling regime, and inference context are well specified, but neither faithfulness nor transferability is automatic.