Cross-Learning: Methods, Theory, & Applications

• Cross-Learning is a family of strategies that integrates task-specific models with a shared representation to balance specialization and generalization.
• It employs mechanisms such as centrality constraints, cross-regularization, and experience reuse to enable controlled information sharing across tasks.
• The concept is applied in settings from multi-task and continual learning to reinforcement and federated learning, yielding measurable gains in accuracy and efficiency.

Searching arXiv for recent and foundational papers on cross-learning to ground the article. Cross-learning denotes a family of learning strategies in which information acquired in one task, domain, context, client, language, layer, embodiment, or learning process is explicitly reused to improve another. In the literature, the term does not refer to a single canonical algorithm. Rather, it appears as constrained multi-task estimation, cross-regularization across representations, cross-domain and continual adaptation, contextual feedback sharing, and cross-task experience reuse. Across these formulations, a recurring objective is to retain task-specific specialization while exploiting related structure for better generalization, data efficiency, or knowledge preservation (Cervino et al., 2020, Cervino et al., 2020, Balseiro et al., 2018, Qi et al., 2024).

1. Scope and recurrent design pattern

A consistent pattern across cross-learning methods is controlled coupling. Instead of either training each task independently or collapsing all tasks into a single shared model, these methods introduce an explicit mechanism that keeps task-specific models “close” to a shared object, or aligns predictions, features, or trajectories across related learning problems. In some papers the shared object is a central parameter vector or policy; in others it is a prototype set, a cross-layer prediction distribution, a learning trajectory, or a retrieved experience trace (Cervino et al., 2020, Aravind et al., 2022, Flennerhag et al., 2018, Li et al., 29 May 2025).

This suggests that cross-learning is best understood as an intermediate regime between separability and consensus. That interpretation is explicit in several constrained formulations, where a centrality parameter determines whether the method degenerates to independent learning or to full sharing (Cervino et al., 2020, Cervino et al., 2020, Aravind et al., 2022, Agorio et al., 17 Nov 2025).

Regime	Coupling object	Representative papers
Constrained multi-task learning	Central parameter or policy	(Cervino et al., 2020, Cervino et al., 2020, Aravind et al., 2022)
Representation-level cross-learning	Features, prototypes, layers, residual branches	(Luo et al., 2019, Gao et al., 2023, Jou et al., 2016)
Cross-domain or continual transfer	Domains, languages, tasks over time	(Carvalho et al., 2024, M'hamdi et al., 2022, Nooralahzadeh et al., 2020)
Decision and agent systems	Context feedback, experience pools, embodiments	(Balseiro et al., 2018, Li et al., 29 May 2025, Tan et al., 28 Oct 2025)

A common misconception is that cross-learning is merely parameter sharing. The surveyed work shows otherwise. Some methods exchange models across clients, some align distributions between layers, some constrain policies in an RKHS, and some optimize over learning-process geometry rather than final weights. The commonality lies in explicit reuse of related information under a mechanism that preserves some degree of task individuality (Qi et al., 2024, Luo et al., 2019, Flennerhag et al., 2018).

2. Constrained optimization and the centrality principle

The most explicit formalization of cross-learning appears in multi-task supervised learning, reinforcement learning, and apprenticeship learning. In supervised learning, the task-specific parameters $\theta_i$ are coupled to a central parameter $\theta_g$ through a proximity constraint:

$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$

As stated in "Multi-task Supervised Learning via Cross-learning" (Cervino et al., 2020), $\epsilon \to 0$ yields consensus learning, large $\epsilon$ yields task-specific learning, and intermediate $\epsilon$ yields cross-learning. The paper also provides a Gaussian two-variable illustration in which, for $\epsilon=\epsilon_0$, the MSE of cross-learning is at most $(3/4)\,\sigma^2 / M$, and reports an Office-Home experiment where consensus achieves about 34% accuracy, agnostic about 24%, and the best cross-learning achieves almost 38% (Cervino et al., 2020).

The same centrality principle is extended in "Cross-Learning from Scarce Data via Multi-Task Constrained Optimization" (Agorio et al., 17 Nov 2025), which formulates joint estimation as

$$\begin{aligned} \{\hat{\theta}_t\}, \theta_g = \arg\min_{\{\theta_t\}, \theta_g} &\frac{1}{T}\sum_{t=1}^T \frac{1}{N_t} \sum_{i=1}^{N_t} \ell(y_i, f(x_i, \theta_t)) \ \text{subject to:} \quad & \|\theta_t - \theta_g\| \leq \epsilon, \quad \forall t . \end{aligned}$$

The paper states that there always exists a value of $\epsilon$ where cross-learning achieves strictly lower MSE than both separable and consensus estimators under Gaussian noise, and extends the framework to functional constraints on model outputs. Its empirical illustrations include SIR curve fitting, where with $\theta_g$0 the exact peak day for Argentina is predicted and the peak count error is reduced to 0.07%, and Office-Home image classification, where cross-learning achieves 44.5% accuracy compared to 35.6% for consensus and 31.9% for separate training (Agorio et al., 17 Nov 2025).

In multi-task reinforcement learning, "Multi-task Reinforcement Learning in Reproducing Kernel Hilbert Spaces via Cross-learning" (Cervino et al., 2020) constrains continuous policies $\theta_g$1 to remain close to a central policy $\theta_g$2 in an RKHS:

$\theta_g$3

The paper gives a projected policy gradient algorithm, a relaxed projection with a closed-form solution, convergence to a near-optimal solution with high probability, and navigation experiments in which cross-learned policies outperform agnostic and consensus policies on unseen related environments (Cervino et al., 2020).

An analogous policy-centric formulation appears in "Cross apprenticeship learning framework: Properties and solution approaches" (Aravind et al., 2022), where one policy is learned per environment while all policies remain close to a central policy $\theta_g$4 in $\theta_g$5-norm. The paper derives properties of the optimizers as $\theta_g$6 varies, provides a convex outer approximation using McCormick envelopes, and demonstrates in a windy gridworld that high $\theta_g$7 yields environment-specific performance, $\theta_g$8 yields a single shared policy, and small but nonzero $\theta_g$9 can improve cross-environment generalization without eliminating specialization (Aravind et al., 2022).

Taken together, these works establish a mathematically crisp version of cross-learning: a tunable interpolation between bias-reducing pooling and variance-reducing specialization. A plausible implication is that much of the later literature can be read as replacing explicit norm constraints by architectural, probabilistic, or retrieval-based couplings.

3. Representation-level cross-learning in vision, federated learning, and graph prediction

A second major lineage treats cross-learning as representation alignment across images, layers, views, or clients. In fine-grained visual categorization, "Cross-X Learning for Fine-Grained Visual Categorization" (Luo et al., 2019) introduces two regularizers: the cross-category cross-semantic regularizer $$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$0, which encourages attention features from the same excitation module to be highly correlated across images while decorrelating different modules, and the cross-layer regularizer $$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$1, which matches prediction distributions between layers through KL divergence. The combined objective is

$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$2

The reported results include 86.4 on NABirds with SENet-50, 87.7 on CUB-Birds with ResNet-50, 94.6 on Stanford Cars with ResNet-50, and 92.7 on FGVC-Aircraft with SENet-50, all without multi-crop, multi-stage, or part annotations (Luo et al., 2019).

Cross-learning between semantic levels is also central in "Self-Supervised Representation Learning with Cross-Context Learning between Global and Hypercolumn Features" (Gao et al., 2023). CGH constructs instance-relation distributions separately from global features and hypercolumn features, then enforces cross-context consistency through

$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$3

The paper reports 70.5% top-1 linear evaluation on ImageNet-1K after 200 epochs for CGH and 72.3% for CGH-pred, compared with 69.3% for ReSSL and 67.5% for MoCo-v2; it also reports 82.6 AP$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$4 on VOC detection and 39.0 AP$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$5 on COCO (Gao et al., 2023).

Cross-learning can also be embedded directly into network topology. "Deep Cross Residual Learning for Multitask Visual Recognition" (Jou et al., 2016) augments multitask residual networks with cross-residual connections,

$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$6

so that related task branches exchange shortcut information. The resulting multitask cross-residual network has >40% less parameters than multiple specialized single-task networks and improves detection performance by about 10.4% over a standard multitask residual network without cross-residuals (Jou et al., 2016).

At larger scale, "X-Learner: Learning Cross Sources and Tasks for Universal Visual Representation" (He et al., 2022) performs cross-learning across heterogeneous tasks and multiple data sources. Its Expansion stage uses task-specific sub-backbones and reconciliation layers,

$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$7

while the Squeeze stage distills the expanded system into a compact backbone. A single X-Learner model shows gains of 3.0%, 3.3% and 1.8% over current pretrained models on 12 downstream datasets for classification, object detection and semantic segmentation (He et al., 2022).

Related cross-learning mechanisms appear in domain generalization and structured prediction. "Self-Supervised Learning Across Domains" (Bucci et al., 2020) combines supervised classification with self-supervised jigsaw and rotation prediction on the same images; on PACS with AlexNet, DeepAll obtains 71.81% average accuracy, while the multi-task method with Jigsaw+Rotation reaches 74.08% (Bucci et al., 2020). "Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers" (Berger et al., 2024) introduces a regularized edge sampling loss, dual-level domain adaptation over image and graph features, and a projection function $$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$8 for 2D$$\begin{aligned} \min_{\{\theta_i\}, \theta_g} \quad & \sum_{i=1}^N \mathbb{E}_{p_i(x,y)}[\ell(y, f(x, \theta_i))] \ \text{subject to}\quad & \|\theta_i - \theta_g\| \leq \epsilon, \quad \forall i . \end{aligned}$$93D transfer; it consistently outperforms standard transfer learning and self-supervised pretraining on retinal and whole-brain vessel graph extraction (Berger et al., 2024).

In heterogeneous federated learning, "Cross-Training with Multi-View Knowledge Fusion for Heterogenous Federated Learning" (Qi et al., 2024) treats cross-learning as client-level model exchange guided by multi-view knowledge. FedCT combines Consistency-Aware Knowledge Broadcasting, Multi-View Knowledge-Guided Representation Learning, and Mixup-Based Feature Augmentation. The fused prototype

$\epsilon \to 0$0

anchors local retraining in both global and personalized knowledge. The paper reports that FedCT alleviates knowledge forgetting from both local and global views, outperforms FedAvg, MOON, FedExg, and PGCT across CIFAR-10/100, TinyImagenet, and VireoFood172, especially under high heterogeneity $\epsilon \to 0$1, and reaches target accuracy in fewer rounds (Qi et al., 2024).

These methods collectively show that cross-learning at the representation level is not restricted to one architectural motif. It can be instantiated through regularization, shortcut connectivity, prototype fusion, domain adversarial alignment, or teacher-student consistency.

4. Cross-learning under sequential, continual, and meta-transfer settings

When tasks or domains arrive sequentially, the central issue is no longer only sharing but also preservation. "Towards Cross-Domain Continual Learning" (Carvalho et al., 2024) introduces CDCL for continual unsupervised domain adaptation. Its compact convolutional-transformer network combines inter- and intra-task cross-attention, center-aware pseudo-labeling, and rehearsal memory. The overall loss is

$\epsilon \to 0$2

The paper derives a target-error bound involving source-domain error, feature distribution discrepancies $\epsilon \to 0$3, and KL divergence for memory-rehearsed label distributions, and reports that CDCL outperforms all continual learning baselines in task-incremental setups on target-domain accuracy $\epsilon \to 0$4 and forgetting $\epsilon \to 0$5, with especially strong behavior on challenging continual UDA such as DomainNet (Carvalho et al., 2024).

A multilingual analogue is given by "Cross-lingual Lifelong Learning" (M'hamdi et al., 2022), which defines a Cross-lingual Continual Learning evaluation paradigm with preservation, accumulation, generalization, and final performance:

$\epsilon \to 0$6

with additional metrics $\epsilon \to 0$7 and $\epsilon \to 0$8. The benchmark shows that Naive Seq FT suffers the worst forgetting, that model expansion and experience replay are most effective for preservation, and that lower forgetting does not necessarily imply better zero-shot transfer. This is one of the clearest statements in the literature that cross-learning in sequential settings is governed by a stability-plasticity trade-off rather than by a monotone “more sharing is better” rule (M'hamdi et al., 2022).

Meta-learning supplies a different route to cross-learning. "Zero-Shot Cross-Lingual Transfer with Meta Learning" (Nooralahzadeh et al., 2020) adapts MAML to multilingual NLI and QA. Using auxiliary languages during meta-training, X-MAML improves zero-shot and few-shot performance across 15 languages on XNLI and improves average F1 on MLQA by up to +1.47% for XLM and XLM-R variants. The paper’s error analysis further notes that typological feature correlation can partly explain when parameter sharing learned via meta-learning is beneficial (Nooralahzadeh et al., 2020).

"Transferring Knowledge across Learning Processes" (Flennerhag et al., 2018) generalizes the object of transfer from parameters to trajectories. Leap associates each task with a manifold and seeks an initialization minimizing expected path length:

$\epsilon \to 0$9

Because the meta-gradient uses only information observed during ordinary training, the paper describes the method as computable on the fly at negligible cost. Leap outperforms finetuning, MAML/FOMAML, and Reptile on computer-vision tasks and transfers to Atari environments with millions of gradient steps (Flennerhag et al., 2018).

This body of work broadens cross-learning from simultaneous multi-task estimation to sequential adaptation. The shared lesson is that transfer is useful only insofar as previously acquired structure remains accessible without overwhelming the current task.

5. Context feedback and strategic online learning

In online decision-making, cross-learning often refers to cross-context observability. "Contextual Bandits with Cross-learning" (Balseiro et al., 2018) extends classical contextual bandits by allowing action $\epsilon$0 chosen in context $\epsilon$1 to reveal rewards for other contexts $\epsilon$2. Under complete cross-learning, where $\epsilon$3 contains all contexts, the paper gives regret

$\epsilon$4

removing the dependence on the number of contexts $\epsilon$5 that appears in the classical $\epsilon$6 rate. Under partial cross-learning, the bounds depend on graph quantities such as clique cover number $\epsilon$7 and maximum acyclic subgraph number $\epsilon$8; for stochastic rewards the paper gives $\epsilon$9, and for adversarial rewards with stochastic contexts it gives $\epsilon$0 (Balseiro et al., 2018).

The paper proposes UCB1-Cross for stochastic rewards, EXP3-Cross for adversarial rewards with known context distribution, and EXP3-U-Cross for adversarial rewards with unknown context distribution. Its motivating application is repeated first-price auctions, where observing the outcome for a chosen bid can reveal the reward that would have resulted under different private values. On real auction data from an ad exchange running first-price auctions, the cross-learning algorithms outperform traditional contextual bandit algorithms (Balseiro et al., 2018).

This formulation is structurally different from centrality-constrained multi-task learning, but the conceptual link is direct: information generated in one local situation can update beliefs in several related situations. In that sense, contextual bandits make explicit a general principle that is implicit elsewhere in the literature.

6. Cross-task experience, cross-space transfer, and embodied systems

Recent large-model work uses cross-learning to denote explicit reuse of solved-task traces. "Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration" (Li et al., 29 May 2025) introduces MAEL, where agents in a graph-structured collaboration network store reward-annotated step tuples $\epsilon$1 in individual experience pools. At inference, each agent retrieves task-relevant experiences with a reward-weighted score,

$\epsilon$2

and uses them as few-shot exemplars. The paper reports that MAEL$\epsilon$3 improves HumanEval from 80.0 to 90.0 and SRDD from 56.0% to 76.4%, that MAEL$\epsilon$4 cuts tokens by 49% on CommonGen-Hard, and that dialogue rounds to convergence on SRDD drop by 26% (Li et al., 29 May 2025).

A broader embodied formulation appears in "BLM$\epsilon$5: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning" (Tan et al., 28 Oct 2025). BLM$\epsilon$6 uses a two-stage paradigm: Stage I injects embodied knowledge into an MLLM via supervised fine-tuning on corpora such as RoboVQA, AgiBot, HoloAssist, BridgeData V2, EgoPlan, and ShareRobot; Stage II freezes the MLLM and trains a shared Diffusion Transformer policy through an intent-bridging interface that compresses MLLM hidden states for control. The model is trained on a cross-embodiment demonstration suite spanning four robot embodiments and six tasks, and the paper reports $\epsilon$7 gains in digital tasks and $\epsilon$8 in physical tasks over four model families. On physical-space benchmarks, BLM$\epsilon$9 reaches 75.8% average success rate, compared with 75.4% for GR00T-N1.5 and 67.4% for $\epsilon=\epsilon_0$0 (Tan et al., 28 Oct 2025).

These recent systems make one final point about the term’s evolution. In earlier work, cross-learning usually denoted a coupling constraint or a regularizer. In large-model and agentic settings, it increasingly denotes a memory mechanism over prior task executions. The underlying rationale is unchanged: solved instances are not isolated training events but reusable structure.

Across the literature, the strongest empirical results typically occur when sharing is explicit but limited. Fully mixed sharing can reduce specialization, as shown by the degradation of identity-weight cross-residuals in multitask residual networks (Jou et al., 2016). Purely local or purely global transfer can also be suboptimal, as shown by the benefits of fused local and global prototypes in FedCT (Qi et al., 2024). Continual multilingual learning further shows that preservation, accumulation, and zero-shot generalization do not move together (M'hamdi et al., 2022). Cross-learning is therefore best regarded not as a single method, but as a design principle for specifying what should be shared, between which units, under what constraint, and with what retention mechanism.