Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning

Abstract: Multi-task learning shows strikingly inconsistent results -- sometimes joint training helps substantially, sometimes it actively harms performance -- yet the field lacks a principled framework for predicting these outcomes. We identify a fundamental but unstated assumption underlying gradient-based task analysis: tasks must share training instances for gradient conflicts to reveal genuine relationships. When tasks are measured on the same inputs, gradient alignment reflects shared mechanistic structure; when measured on disjoint inputs, any apparent signal conflates task relationships with distributional shift. We discover this sample overlap requirement exhibits a sharp phase transition: below 30% overlap, gradient-task correlations are statistically indistinguishable from noise; above 40%, they reliably recover known biological structure. Comprehensive validation across multiple datasets achieves strong correlations and recovers biological pathway organization. Standard benchmarks systematically violate this requirement -- MoleculeNet operates at <5% overlap, TDC at 8-14% -- far below the threshold where gradient analysis becomes meaningful. This provides the first principled explanation for seven years of inconsistent MTL results.

• The paper demonstrates that gradient-based task affinity is meaningful only when there is sufficient sample overlap (≥40%) between tasks.
• It provides rigorous theoretical analysis and empirical evidence on molecular benchmarks to quantify how gradient similarity reflects true task structure.
• Gradient similarity is shown to predict multi-task learning benefits, guiding effective task grouping and reducing negative transfer.

Information-Theoretic Foundations and Limitations of Gradient-Based Task Affinity in Multi-Task Learning

Introduction and Problem Statement

The paper "Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning" (2604.07848) provides a rigorous framework to diagnose the interpretability of gradient-based task affinity metrics in Multi-Task Learning (MTL), specifically within the context of molecular property prediction. The work identifies a previously unstated, yet indispensable, information-theoretic requirement: gradient-based analysis of task similarity is meaningful if and only if tasks are measured on sufficiently overlapping samples. This perspective directly addresses pervasive inconsistencies in the MTL literature, notably clarifying why gradient-based analyses and derived methods yield unreliable or even contradictory results on standard molecular benchmarks such as MoleculeNet and TDC.

Gradient Conflict as an Indicator of Task Structure

Gradient alignment, typically measured by the cosine similarity of task gradients, has been commonly used as a proxy for mechanistic task relationships in MTL. The core assumption is that synergy manifests as positive gradient similarity, antagonism as negative, and independence as near-zero alignment. Prior work has generally treated gradient conflict as an optimization artifact to be minimized or circumvented, rather than a statistical signal to interpret task structure.

The central conceptual contribution here is the formal demonstration that gradient comparisons only meaningfully reflect inter-task structure in the presence of substantial sample overlap. When sample overlap is minimal or absent, observed gradient relationships are confounded by distributional shift, leading to fundamentally uninterpretable metrics.

Figure 1: Sample overlap determine the interpretability of gradient similarity—gradients computed on shared samples encode mechanistic relationship, but on disjoint samples are confounded by distributional shift.

Quantifying the Sample Overlap Requirement

Through comprehensive empirical analyses spanning six molecular datasets (covering toxicity, side effect, pharmacokinetics, kinase selectivity, and quantum chemistry), the authors systematically degrade the instance overlap between task pairs and observe a sharp phase transition in the informativeness of gradient affinity measures. The primary results can be summarized as:

• Below 30% instance overlap, the correlation between gradient affinity and true task relationship is non-significant ($r < 0.25$).
• Above 40% overlap, the correlation becomes strong ($r > 0.65$) and empirically reliable.
• The transition is well-approximated by a sigmoid function with an inflection point at 29.7% overlap ($R^2 = 0.94$), providing a practical quantitative threshold.

  Figure 2: Across datasets, the gradient-task empirical correlation undergoes a phase transition at ≈30% overlap, distinguishing unreliable and reliable regimes for gradient-based affinity estimation.

Notably, standard molecular multi-task benchmarks often operate with median overlaps far below this critical threshold (MoleculeNet $<$5\%, TDC 8–14\%), explaining the heterogeneity and lack of reproducibility of performance improvements reported using gradient-based methods.

Information-Theoretic Model and Theoretical Results

A central theoretical contribution is the precise characterization of the mutual information between gradient-based affinity and the true underlying task relationship. The paper proves that, under i.i.d. sampling, the mutual information is bounded by the overlap fraction $\alpha$ between two task sample sets; crucially, with zero overlap, mutual information is also zero, implying gradient affinity is completely uninformative in this regime.

The empirical phase transition is explained as a signal-to-noise ratio problem: in the presence of partial overlap, the observed gradient is a mixture of a "true signal" term (from overlapping samples) and noise stemming from the unshared (and potentially distributionally shifted) samples. The sharpness of the empirical transition is quantitatively matched by a variance decomposition model.

Biological, Physical, and Practical Validation

The validity and generality of the sample-overlap constraint are demonstrated through several lines of evidence:

• On Tox21, SIDER, and QM9 (all 100% overlap), the Pearson correlation between gradient and empirical task correlation matrices reaches up to $r = 0.94$ (Spearman $\rho = 0.97$ for SIDER).
• Subdividing panels into disjoint (cross-domain) groups (e.g., Tox21+ADME), the gradient-task correlation drops to near zero.
• In kinase selectivity, gradient-based affinity not only captures synergistic but also antagonistic task groups, provided overlap is sufficient.
• Hierarchical clustering on gradient matrices recovers known biological and physical pathway structure with ARI up to 0.65 and NMI up to 0.95 at high annotation granularity.

  Figure 3: Full gradient (G) and empirical (E) correlation matrices for Tox21, reordered to visualize the fine-grained recovery of biological structure.

Moreover, gradient similarity is predictive of MTL benefit: high-similarity task pairs on Tox21 exhibit $+2.3\%$ performance improvement when trained jointly, while low-similarity pairs experience negative transfer ($-1.8\%$ on average). Using a gradient similarity threshold ($G \geq 0.10$) eliminates 77% of negative transfer cases while retaining 85% of beneficial ones.

Figure 4: Gradient similarity robustly predicts positive or negative transfer in MTL and enables effective data-driven task grouping for improved performance.

Gradient groupings derived from the affinity matrix outperform random groupings by 3–4% in AUC, offering a scalable, data-driven workflow for MTL task selection.

Implications and Recommendations

The findings establish multiple practical and conceptual guidelines:

• Gradient-based affinity estimation and related optimization methods are not reliable for task selection unless the 40% overlap criterion is met.
• Most popular public molecular benchmarks do not satisfy this requirement, explaining the field's inconsistent empirical findings.
• Reliable affinity estimation is possible within 20 epochs, facilitating computationally efficient task grouping or early stopping strategies.
• Learned representations (GCN, GAT, CNN) yield robust affinity matrices ($r > 0.65$0–$r > 0.65$1 correlation across architectures), whereas fixed fingerprints (ECFP) do not, emphasizing the importance of representation learning in MTL affinity diagnostics.

Theoretical implications extend beyond molecular ML, potentially impacting MTL in vision, NLP, and reinforcement learning, but further empirical work is required to calibrate the sample overlap threshold in these settings.

Limitations and Directions for Future Research

Several limitations are recognized:

• The 40% threshold is empirically derived for molecular data; transferability to other application domains remains to be established.
• The mathematical model assumes that out-of-overlap gradients contribute only independent noise, which may not always hold if sample selection correlates with task semantics in complex ways.
• Low-overlap settings, prevalent in real-world bioactivity matrices, are outside the operational scope; future research should address strategies like overlap-augmentation, anchor selection, or hybrid data-knowledge integration.
• Both gradient and empirical correlation matrices are computed from data with potential circularity in low-overlap regimes, although synthetic validation mitigates this concern.

Conclusion

This work resolves a core ambiguity in multi-task learning by tightly linking the information-theoretic validity of gradient-based affinity estimates to task-instance overlap. The implications are immediate and substantial: to interpret or exploit gradient-based task similarities, practitioners must construct datasets or panels with $r > 0.65$240% overlap per task pair, or else employ alternative techniques. The supplied guidelines and open-source implementation set new standards for principled MTL benchmark and experiment design. Future research should focus on overlap-efficient transfer diagnostic strategies and generalization of the theoretical framework to diverse MTL settings.

(2604.07848)