Cross-Task Predictability

• Cross-task predictability is the quantification of how knowledge, parameters, and representations derived from one task enhance performance on another.
• It employs metrics such as relative transfer gain, nDCG ranking, Top-k regret, and OTCE to evaluate and predict effective knowledge sharing.
• Empirical findings indicate that refined methods like token-wise maximum similarity and joint Bayesian estimation yield significant gains in multi-task and transfer learning.

Cross-task predictability refers to the extent to which knowledge, parameters, or representations derived from one task (or set of tasks) enable the accurate or improved prediction, adaptation, or grouping for another task. This concept is fundamental in transfer learning, multi-task learning, and meta-learning, and is formalized in diverse ways across neural, probabilistic, and structural paradigms. Robust quantification and mechanism design for cross-task predictability enables principled task selection, transferability estimation, grouping for joint learning, and automated allocation of computational resources. Recent literature systematically investigates not only statistical and representational metrics for cross-task predictability, but also the architectural, statistical, and computational conditions under which it translates into empirical performance gains.

1. Formal Definitions and Metrics of Cross-Task Predictability

Cross-task predictability is typically operationalized via transfer gain, ranking-quality, regret, affinity, or transferability metrics:

• Relative Transfer Gain is the percent improvement in target task performance from initializing or transferring from a source task $T_s$:

$$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$

where $M(\cdot)$ denotes task-specific evaluation metric (e.g., accuracy, F1) (Lin et al., 2024).

• Ranking Quality (nDCG) quantifies the alignment of a predicted source-task ranking (based on a task similarity or embedding score) with a ground-truth ranking sorted by $M(T_s \rightarrow T_t)$. Normalized Discounted Cumulative Gain up to $p=|S|$ tasks is used:

$$\mathrm{nDCG}(R_{\text{pred}}, R_{\text{true}}) = \frac{\mathrm{DCG}(R_{\text{pred}})}{\mathrm{DCG}(R_{\text{true}})}$$

with DCG based on transfer metric (Lin et al., 2024).

• Top-$k$ Regret captures the regret from selecting among only the top $k$ candidate sources under a proxy, rather than the true best:

$$\mathrm{Regret@k} = \frac{M(s^* \rightarrow T_t) - \max_{s \in S_k} M(s \rightarrow T_t)}{M(s^* \rightarrow T_t)}$$

(Lin et al., 2024).

• Affinity for MTL Grouping is defined as the relative improvement in loss when tasks $t_i, t_j$ are grouped:

$$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$0

(Ayman et al., 2023).

• OTCE Metric for cross-domain/cross-task transfer combines feature-space distance (Wasserstein) and conditional entropy of matched source/target labels under optimal transport:

$$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$1

where $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$2 is Wasserstein distance of features, $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$3 is conditional entropy of target given source labels (Tan et al., 2021).

• Predictor Combination Predictability measures how well a target task predictor $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$4 can be explained as a (linear or GP-based) function of reference predictors $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$5:

$$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$6

where $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$7 is the least-squares regressor; nonlinear $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$8 uses GP regression (Kim et al., 2020).

2. Representational and Algorithmic Approaches

A. Embedding-Based Similarity and Predictability

• Task Embeddings: Derived from fine-tuned parameter vectors (e.g., soft-prompt embeddings in T5), such as $$\Delta_{\mathrm{rel}}(T_s \rightarrow T_t) = 100\% \cdot \frac{M(T_s \rightarrow T_t) - M(T_t)}{M(T_t)}$$9 for prompt tokens, where similarity is given via cosine (Lin et al., 2024).
• Token-wise Maximum Similarity: Instead of mean, use maximum similarity over tokens to better capture fine-grained alignment:

$M(\cdot)$0

This metric more effectively predicts transferability for tasks with distributed token-level semantics (Lin et al., 2024).

• Optimal Transport with Conditional Entropy (OTCE): Combines geometric alignment of feature distributions and semantic alignment of label distributions via conditional entropy computed under the optimal transport plan (Tan et al., 2021).

B. Predictive Modeling of Group Affinity and Transfer

• Affinity Predictors for Automated Grouping: Small neural networks estimate expected MTL gain (affinity) based on features per task or task pair (variance, size, STL learning curve gradient, inter-task weight dot-product) (Ayman et al., 2023).
• Traveling Observer Model (TOM): Embeds all scalar variables (across disjoint tasks) in a common space, with a universal prediction function $M(\cdot)$1, facilitating transfer among tasks with disjoint input/output spaces (Meyerson et al., 2020).

C. Structural and Consistency Methods

• Inference-Path Invariance and Consistency Energy: Represent tasks as nodes in an inference graph with learned neural mapping functions; enforce that multiple inference paths map to the same output—a property monitored by the unsupervised Consistency Energy metric, which tracks predictability and can serve as an OOD detector (Zamir et al., 2020).
• Cross-Task Consistency Loss: Mold predictors between task outputs (e.g., $M(\cdot)$2) to enforce self-consistency, theoretically bounding the gap between indirect/transferred and direct predictions (Nakano et al., 2021).

3. Empirical Findings and Practical Applications

Cross-task predictability enables:

• Task selection for transfer learning: Token-wise maximum prompt similarity (as opposed to mean or textual similarity) improves the normalized DCG for predicting beneficial intermediate tasks from ≈0.793 (text embedding) to 0.825 (Max) (Lin et al., 2024).
• Automated grouping in MTL: The predictor-driven search finds lower joint test losses than clustering, full pooling, or exhaustive pairwise search, e.g., on the School and Landmine datasets (Ayman et al., 2023).
• Unsupervised or out-of-distribution confidence estimation: Consistency Energy correlates with true supervised error ($M(\cdot)$3), and achieves ROC-AUC $M(\cdot)$4 for OOD detection (Zamir et al., 2020).
• Dataset selection and modular fine-tuning: Statistical properties of source datasets (label entropy $M(\cdot)$5, output length, dependency relation sensitivity) are more predictive of transfer performance than semantic/task similarity; $M(\cdot)$6 for transfer gain regressions (Krishna et al., 17 Sep 2025).
• Feature fusion and zero/few-shot selection: OTCE yields an absolute ranking error $M(\cdot)$7 (DomainNet) for source selection and consistently outperforms LEEP/H-score in Pearson correlation with transfer accuracy (DomainNet: $M(\cdot)$8, Office31: $M(\cdot)$9) (Tan et al., 2021).

Predictability Metric	Mathematical Definition	Primary Use/Strength
nDCG (ranking)	see above	Source task selection
Top-$M(T_s \rightarrow T_t)$0 regret	see above	Practical transfer scenario loss bounds
Consistency Energy	see above	Unsupervised error/OOD estimation
OTCE	see above	Cross-domain/cross-task model selection
Affinity predictor	see above	Automated grouping for MTL
GP/Lin. predictability	see above	Joint denoising, reference selection

4. Limitations, Variance Sources, and Open Challenges

• Seed and Task Instability: Transfer gains and rankings can exhibit high variance across random seeds, especially for low-resource settings (e.g., COPA task: relative gain between +7.7% and +26.8%) (Lin et al., 2024).
• Reasoning/Multiple-Choice Tasks: Even the strongest transferability predictors (token-wise Max) underperform data-size heuristics for reasoning-style targets (e.g., HellaSWAG, COPA) (Lin et al., 2024).
• Negative Transfer in MTL: Arbitrary grouping may hurt performance; affinity-based partitioning is essential (Ayman et al., 2023).
• Hyperparameter Sensitivity: Token-wise and prompt-based approaches are sensitive to N (number of tokens), optimizer parameters, and stability under different model architectures (Lin et al., 2024).
• Interpretability and Specialization–Generality Tradeoff: Specializing predictors may degrade performance on unrelated or held-out tasks, especially in graph foundation models using task-tree pretraining (Wang et al., 2024).
• Representation Choice: Model-agnostic features may not suffice; layer- or neuron-level activation matching and richer metadata may further improve cross-task predictability (Lin et al., 2024).

5. Connections to Theory and Unified Perspectives

• Theoretical Transfer Bounds: For task-tree pretraining, excess downstream risk is explicitly upper-bounded by the pretraining reconstruction error plus distribution shift and Rademacher complexity terms (Wang et al., 2024).
• PAC Learnability under Constraints: Formal analysis in knowledge-constrained self-training shows that given a correct, discriminating task-output compatibility constraint $M(T_s \rightarrow T_t)$1, cross-task self-training enables PAC-learnability under noise (0907.0784).
• Joint Bayesian Estimation in Predictor Combination: Enhancing joint predictability (via GP-based objectives and automatic reference selection through kernel marginal likelihood) yields strictly stronger performance compared to pairwise-only or single-predictor denoising (Kim et al., 2020).

6. Future Directions

• Extending token-wise, task-graph, or tree-based approaches to richer model classes and architectures, e.g., Transformer activations or neuron-level weights (Lin et al., 2024, Wang et al., 2024).
• Incorporating additional task properties (label topology, task difficulty) and meta-information for more robust and generalized predictability (Lin et al., 2024, Krishna et al., 17 Sep 2025).
• Enabling continual or dynamic multi-task selection, rather than static one-shot choice, and exploring positive/negative transfer patterns in low-resource and highly heterogeneous settings (Lin et al., 2024).
• Defining and evaluating unified task-trees or computation trees to capture all levels of task structure—node, edge, and graph—bridging the domain-agnostic and structure-aware paradigms (Wang et al., 2024).
• Combining cross-task predictability with test-time training, self-supervised adaptation, and out-of-distribution robustness mechanisms (Jeong et al., 10 Jul 2025).

7. Summary of Impact and Significance

Cross-task predictability serves as a critical axis for improving knowledge transfer, task selection, model composition, and robust automation in multi-task and transfer learning. While simple task- or data-size heuristics are often competitive for some domains, embedding-based and token-wise similarity predictors yield measurable gains in ranking quality and absolute target performance. Nonetheless, task heterogeneity, data scarcity, and architectural idiosyncrasies remain central challenges. Joint modeling of statistical (feature, label) structure and learned task representations shows particular promise for predicting and controlling the flow of information among tasks, setting the stage for principled foundation models and automated multi-task systems (Lin et al., 2024, Krishna et al., 17 Sep 2025, Ayman et al., 2023, Wang et al., 2024, 0907.0784).