Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning

Abstract: Class-Incremental Learning (CIL) requires a learning system to learn new classes while retaining previously learned knowledge. However, in real-world scenarios such as autonomous driving, a system trained on urban roads in sunny weather may later need to operate in rural or highway environments with different traffic patterns and weather conditions. This requires the model not only to overcome catastrophic forgetting, but also to effectively handle domain shifts. In this paper, we propose CrOss-sample Relational Fusion (CORF), a unified framework to address domain shift and catastrophic forgetting simultaneously. To enhance generalizability, we perform selective refinement of training samples by leveraging spatial contribution maps to highlight semantically informative regions. Furthermore, we incorporate predictive confidence to adaptively weigh samples, thereby facilitating the learning of domain-agnostic representations. To alleviate forgetting, we propose a cascaded distillation framework that captures cross-sample relational dependencies across multiple feature hierarchies, enabling multi-grained knowledge transfer from previous tasks. CORF can be seamlessly integrated into existing CIL algorithms to enhance their generalizability, achieving competitive performance across various benchmark datasets. Code is available at https://github.com/LAMDA-CL/TMM26-CORF .

• The paper introduces CDCIL, a unified framework that addresses catastrophic forgetting and domain shift through cross-sample relational fusion.
• It employs Dual-Sensitive Refinement and Hierarchical Kernel-Based Distillation to enhance intra-class alignment and preserve inter-sample relationships.
• Experimental results show significant accuracy gains over baselines across OfficeHome, PACS, and DomainNet, demonstrating the framework's scalability and robustness.

Cross-Sample Relational Fusion for Unified Domain Generalization and Class-Incremental Learning

Introduction

The paper "Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning" (2605.08839) presents a principled approach for simultaneously tackling catastrophic forgetting and domain shift within continual learning systems. Addressing the widely recognized gap between Class-Incremental Learning (CIL)—focused on memory stability amid evolving class distributions—and Domain Generalization (DG)—focused on robust generalization to unseen domains—the authors introduce Cross-Domain Class-Incremental Learning (CDCIL) as a unified problem setting. CDCIL encapsulates the dual challenges of incremental class acquisition and domain shift, thus demanding learning systems that are both forgetting-resistant and capable of domain-agnostic representation.

Figure 1: CDCIL involves incremental introduction of novel classes sourced from multiple seen domains, with inference generalized to all encountered classes across both seen and unseen domains.

CDCIL: Problem Formulation and Evaluation

CDCIL redefines the learning protocol: models are exposed to sequential tasks, each comprising samples from multiple seen domains, and incrementally incorporate novel classes. During inference, evaluation protocols require correct prediction over all previously encountered classes across both seen and unseen domains, as visualized by domain-rotation evaluations.

Figure 2: CDCIL evaluation protocol employing domain rotation on OfficeHome, reporting averaged final-stage accuracy across seen and unseen domains.

Methodology: Cross-Sample Relational Fusion (C\ORF)

C\ORF is architected with two synergistic, modular components: Dual-Sensitive Refinement (DSR) and Hierarchical Kernel-Based Distillation (HKD).

Dual-Sensitive Refinement (DSR)

DSR leverages spatial contribution maps (via Grad-CAM) and per-sample predictive confidence to selectively synthesize auxiliary training samples. High-confidence samples undergo blending with same-class counterparts from differing domains in low-contribution regions, facilitating domain-agnostic intra-class alignment. Conversely, low-confidence samples are blended with different-class instances from the same domain in high-contribution regions, assigned to auxiliary classes to prevent semantic corruption. This selective sample fusion yields an auxiliary set that, when optimized jointly with the original dataset, enforces compact intra-class and well-separated inter-class representations across domains.

Figure 3: C\ORF overview. Left: DSR constructs auxiliary samples via confidence-sensitive, contribution-guided blending; Right: HKD aligns relational topology across feature hierarchies by kernel-based distillation.

Hierarchical Kernel-Based Distillation (HKD)

HKD extends distillation from logit-level to relational alignment across feature hierarchies, capturing both global and localized sample interactions. Kernel maps are computed via cosine and student-$t$ similarity to encode inter-sample topologies. The distillation loss aggregates symmetric divergences between old and current kernel-induced relational distributions at selected layer pairs, ensuring structural consistency and geometry preservation in the evolving feature space. HKD thus mitigates catastrophic forgetting by maintaining fine-grained relational cues through incremental adaptation.

Experimental Evaluation

C\ORF demonstrates versatility through seamless integration with diverse CIL baselines, including FineTune, Replay, iCaRL, DER, MEMO, FOSTER, and others. Comprehensive evaluations are conducted on OfficeHome, DomainNet, and PACS, using domain-rotation protocols for balanced assessment across seen (SD) and unseen (UD) domains, together with harmonic mean (HM) for model robustness.

Numerical Results

C\ORF consistently outperforms baselines across all datasets. Notably, when appended to DER, accuracy improvements reach +3.7 points HM on OfficeHome (13 tasks), +7.5 points HM on PACS, and +10.9 points HM on DomainNet. Ablation studies confirm additive effects from DSR and HKD, with joint optimization further bolstering SD, UD, and HM metrics.

Figure 4: Ablation study on OfficeHome (seen domains) showing isolated and combined gains by C\ORF components.

Figure 5: Sensitivity analysis of C\ORF under varying $\alpha$ (DSR blending ratio) and $\beta$ (HKD distillation weight), demonstrating stability across a wide hyperparameter range.

Feature Analysis

Feature similarity matrices show sharp reductions in inter-class similarity under C\ORF, indicating improved class separability and enhanced domain discrimination. Kernel selection experiments underscore the imperative to combine cosine and student-$t$ kernels for robust relational preservation amid domain shift.

Applicability and Compatibility

C\ORF is primarily designed for convolutional backbones but retains stable behavior when integrated with prompt-based transformer architectures (L2P, DualPrompt) under ViT backbones, albeit with reduced improvement due to limited backbone parameter updates. Resource consumption remains practically tractable, with moderate increases in training time and memory but no extra trainable parameters.

Implications and Future Prospects

C\ORF establishes a generalizable paradigm for CDCIL. Practically, this enables systems in real-world applications (e.g., autonomous driving, robotics) to incrementally expand capability over novel classes and environments without catastrophic knowledge decay or domain bias. Theoretically, the cross-sample fusion and hierarchical relational distillation offer scalable primitives for broader continual- and lifelong-learning frameworks.

Future directions include revising C\ORF to accommodate more aggressive domain shifts, leveraging generative replay, augmenting relational cues in transformer architectures, and scaling to larger, more diverse incremental settings. The modularity of C\ORF allows for rapid integration as a regularization or augmentation mechanism across established and emerging continual learning pipelines.

Conclusion

This work delivers a unified, technically rigorous approach for continual class acquisition under domain shifts by synergistically combining confidence-guided sample refinement and kernel-based relational distillation. The strong empirical performance and modular design position C\ORF as a robust solution to CDCIL, offering improved generalizability and resistance to forgetting across both seen and unseen domains.