Cross-Task Spillover

• Cross-task spillover is the transfer of learned features or biases from one task to improve performance in another, even when the tasks have differing distributions or objectives.
• It employs mechanisms such as constraint-driven optimization, policy guidance, and demonstration alignment to achieve favorable bias-variance tradeoffs and measurable performance gains.
• This concept is pivotal in applications ranging from multi-task and continual learning to reinforcement learning and in-context LLM adaptation, supported by both theoretical guarantees and empirical evidence.

Cross-task spillover refers to the explicit transfer, propagation, or sharing of information, features, or inductive biases acquired in one task to improve learning, inference, or generalization in other tasks—especially where tasks are only weakly coupled and may have non-identical distributions, objectives, or data regimes. This phenomenon is exploited across diverse domains such as multi-task and continual learning, in-context learning for LLMs, reinforcement learning, and social-behavioral modeling. Cross-task spillover is both a mechanism and a target of algorithmic design, with rigorous mathematical characterizations available in recent literature.

1. Formalizations and General Mechanisms

Multiple frameworks instantiate cross-task spillover by relaxing the standard multi-task paradigm (where joint learning shares model parameters) to allow calibrated, often constraint-driven, information transfer between tasks. In the constrained multi-task optimization setup (Agorio et al., 17 Nov 2025), each task $t$ has its own parameter vector $\theta_t$, with joint estimation over tasks constrained to ensure $\|\theta_t - \theta_g\|_2 \leq \varepsilon$ for a global centroid $\theta_g$, or via pairwise/taskwise similarity constraints. The Karush-Kuhn-Tucker (KKT) conditions reveal explicit interpolation between task-local estimators $\hat\theta_t$ (unconstrained) and the global centroid, parameterized by Lagrange multipliers $\mu_t$, with spillover emerging as high-variance, data-poor tasks are pulled toward estimates of confidently learned, data-rich tasks.

In policy-transfer regimes such as Cross-Task Policy Guidance (CTPG) in reinforcement learning (He et al., 9 Jul 2025), cross-task spillover is realized at the behavioral level: guide policies $\Pi_i^g(j|s)$ select among all policies $\pi_j$ for a given state $s$, thus directly leveraging the control skills of "source" tasks to improve policy trajectories in "target" tasks. Sophisticated gating mechanisms filter beneficial policies and block unnecessary guidance, dynamically optimizing the direction and strength of spillover.

In LLM in-context learning, techniques like CrossICL (Gao et al., 30 May 2025) operationalize cross-task spillover by reusing demonstrations from source tasks—aligned by representation similarity and prompt transformation—to boost target task performance, even across significant task gaps.

2. Theoretical Guarantees and Analytical Results

Theoretical results quantify spillover efficacy in simplified statistical regimes. Under multivariate Gaussian mean-estimation (Agorio et al., 17 Nov 2025), cross-task constrained learning yields a mean squared error (MSE) strictly lower than both the per-task and global consensus estimators for some constraint width $\varepsilon>0$:

$\theta_t$0

Proof exploits the structure introduced by the constraints, showing that even a small relaxation from consensus enables favorable bias-variance tradeoffs and that tasks with less data benefit disproportionately from the information provided by others.

In multi-agent social contexts, analytical results via extended pair-approximation ODEs (Khoo et al., 2017) establish that cross-layer spillover in games has a nonmonotonic effect: moderate spillover maximizes cooperation, but too much is detrimental—a result quantified by computing the critical $\theta_t$1 at which cooperation peaks and rigorous bifurcation diagrams for system dynamics.

3. Task- and Domain-specific Instantiations

Multi-task and Continual Learning

• In class-incremental learning, cross-task spillover is essential for the learner to develop features that discriminate among classes never seen together during training (Soutif--Cormerais et al., 2021). The degree of spillover depends critically on the use of cross-task discriminative losses (i.e., global cross-entropy over all classes) rather than per-task objectives.
• Cross-task consistency frameworks (Nakano et al., 2021) introduce encoder-decoder architectures with explicit consistency losses, enforcing alignment between task outputs via auxiliary transfer networks, yielding measurable gains in interdependent downstream metrics (e.g., mIoU and depth estimation error in vision). Theoretical bounds guarantee that expected consistency loss is bounded and at least as good as standard approaches.

LLMs and In-Context Learning

• CrossICL (Gao et al., 30 May 2025) demonstrates that LLMs exhibit robust spillover, with the aligned use of source-task demonstrations improving average ROUGE-L by +2.8 percentage points over self-ICL baselines on 875 Super-NI NLP tasks. Progressive adaptation steps—query rewriting, format refinement, source-guided label generation—are required to manage task-gap-induced interference, typified by misleading task type, output format confusion, and label prior shifts.
• Retrieval-augmented adaptation methods such as ReCross (Lin et al., 2022) use a combination of dense retrieval and reranking over upstream multi-task training data to enhance model adaptation on unseen tasks, concretely yielding 3-8 percentage point gains in zero-shot generalization, demonstrating transferable cross-task skills even in the absence of explicit supervision.

Reinforcement Learning

• CTPG (He et al., 9 Jul 2025) enables explicit policy-level spillover, where learned guide policies pick among all task policies, with value-based filters ensuring only beneficial cross-task behavior is reused. Experimental results on MetaWorld and HalfCheetah benchmarks show that CTPG consistently boosts both final return and sample efficiency, especially in settings with heterogeneous task difficulties.
• Model-based RL with cross-task transfer (XTRA) (Xu et al., 2022) leverages pretraining of world models across many tasks and concurrent finetuning on held-out targets, yielding empirical spillover reflected in mean performance improvements of 23% over standard baselines and up to 71% on specific games. Adaptive task weighting is deployed to filter negative transfer and maximize cross-task benefits during finetuning.

Social Models and Evolutionary Games

• Cross-context spillover is rigorously defined in multiplex games (Khoo et al., 2017), where strategies transfer across network layers (e.g., repeated and one-shot donation games), with the intensity and mechanism of spillover (neighbor imitation, self-comparison, context interference) directly modulating the evolution of cooperation. Analytical and simulation studies reveal double-edged, nonmonotonic effects and the emergence of bistability, with critical dependence on initial cooperation levels, spillover mode competition, and layer weighting.

4. Empirical Evidence and Evaluation

Empirical validation is provided across domains:

Domain	Method	Spillover Metric/Gain	Reference
Supervised/statistical	Constrained cross-learning	MSE lower than per-task & consensus; ±2000%→0.07% error	(Agorio et al., 17 Nov 2025)
NLP, LLMs, ICL	CrossICL	Avg. ROUGE-L +2.8pp over self-ICL	(Gao et al., 30 May 2025)
NLP, few/zero-shot tasks	CrossTaskMoE	ARG +2.6% (few-shot), +5.6% (zero-shot)	(Ye et al., 2022)
Multi-task, class-IL	FT-BAL+CTF	Accuracy gains up to +5.7 over no-cross-task features	(Soutif--Cormerais et al., 2021)
Reinforcement learning	CTPG	+10–20 pts success / +0.5–1.0k return	(He et al., 9 Jul 2025)
Model-based RL	XTRA	Mean return +23%/+71% on held-out Atari games	(Xu et al., 2022)
Vision (MTL)	XTasC-Net	Δ̄ +4.09 (NYU), +1.58 (Cityscapes) over baselines	(Nakano et al., 2021)
Multiplex games	ODE/simulations	Cooperation peaks at $\theta_t$2; bistability, double-edged	(Khoo et al., 2017)

These demonstrate consistent spillover gains, with the precise magnitude depending on alignment mechanisms, task similarity, transfer architecture, domain, and data richness.

5. Modes and Limitations of Spillover

The spillover effect depends on multiple design choices and domain idiosyncrasies:

• Constraint strength (in parameter or output space) controls the magnitude of spillover: tight coupling approaches consensus, while loose constraints reduce to per-task learning (Agorio et al., 17 Nov 2025).
• In LLM ICL, task-gap-induced interference is a major limiting factor: spillover can be negative if demonstration alignment is naive. Task description and input similarity are the most predictive selection criteria (Gao et al., 30 May 2025).
• In RL, excessively similar tasks may yield limited marginal spillover as optimal policies are already learned; overextended guide horizons or poor filtering can cause out-of-distribution behavior (He et al., 9 Jul 2025).
• In class-incremental learning, limited exemplar memory can bottleneck cross-task discrimination, even if spillover is encouraged (Soutif--Cormerais et al., 2021); most reported "forgetting" is due not to catastrophic interference but to the challenge of scaling discrimination over a growing label set.

6. Interpretability and Analysis of Spillover

Several architectures facilitate introspection and causal attribution of spillover:

• Task-level mixture-of-experts (MoE) routers (Ye et al., 2022) naturally cluster tasks by skill/domain, and ablating individual experts causally demonstrates which source-task knowledge underpins transfer to specific targets.
• Cross-task consistency frameworks (Nakano et al., 2021) make spillover explicit through auxiliary transfer nets; theoretical analysis confirms that cross-task prediction consistency tightens error bounds and improves learning efficiency.
• Pair-approximation ODEs in social settings reveal detailed phase diagrams of system dynamics under varying spillover intensities, supporting direct design of effective interventions (Khoo et al., 2017).

7. Practical Guidelines and Future Directions

Emerging consensus in the literature suggests several best practices for harnessing cross-task spillover:

• Explicitly design loss functions or architectural constraints that promote information transfer while preserving task-specificity (Agorio et al., 17 Nov 2025, Nakano et al., 2021).
• When reusing demonstrations or examples (NLP/LLMs), apply selection and adaptation strategies prioritizing input-label mapping fidelity and input distribution similarity. Always apply progressive adaptation; raw demonstration transfer can induce catastrophic interference (Gao et al., 30 May 2025, Lin et al., 2022).
• Monitor and maximize representation quality for cross-task features, particularly in class-incremental regimes where task IDs are unavailable at test time (Soutif--Cormerais et al., 2021).
• In RL and control, deploy gating and adaptive weighting mechanisms to prevent negative transfer and maximize guidance from tasks with mature, high-confidence policies or models (He et al., 9 Jul 2025, Xu et al., 2022).
• Track and report domain-specific spillover metrics (ARG, accuracy gain, MSE, sample efficiency) to quantify transfer benefits and avoid confounded interpretations of loss/accuracy improvements.

Limitations remain, particularly regarding scalability of constraint learning, interpretability of transfer pathways in large-scale models, and principled selection of transfer source tasks or experts. Future work is directed toward foundation models and architectures explicitly optimized for maximal and controllable cross-task spillover, both at the algorithmic and theoretical levels.

---

References: (Agorio et al., 17 Nov 2025, Gao et al., 30 May 2025, Bigoulaeva et al., 2022, He et al., 9 Jul 2025, Lin et al., 2022, Ye et al., 2022, Xu et al., 2022, Nakano et al., 2021, Soutif--Cormerais et al., 2021, Khoo et al., 2017)