Source-Free Cross-Domain Continual Learning

• Source-free cross-domain continual learning is a framework that adapts models to successive target domains without source samples, mitigating catastrophic forgetting.
• It leverages diverse methodologies such as teacher-student setups, replay synthesis, and parameter-efficient prompting to maintain stability under domain and label shifts.
• The approach addresses challenges like pseudo-label corruption and representational collapse by using tailored loss functions and consistency-driven updates.

Source-free cross-domain continual learning studies how a model pre-trained on a labeled source domain can be adapted to a sequence of target domains when source-domain samples are unavailable during adaptation. In the literature, this umbrella includes source-free domain adaptation to unlabeled targets, continual source-free adaptation across multiple target domains, joint task and domain shift, universal domain adaptation with changing label spaces, and task-agnostic inference without domain-identity hints. Across these variants, the central objective is stable plasticity under domain shift: acquiring new domain-specific competence while limiting catastrophic forgetting of previously learned domains, classes, or zero-shot capabilities (Agrawal et al., 12 Sep 2025, Furqon et al., 2 Oct 2025, Toldo et al., 2022, Schlachter et al., 16 Jan 2026, Xu et al., 2024).

1. Problem setting and taxonomic scope

A basic formulation is Source-Free Domain Adaptation (SFDA), which adapts a model to a target domain without access to the data of the source domain. SCoDA places this formulation in a continual setting by initializing both teacher $f_\theta$ and student $f_\psi$ from a model $h$ pre-trained on source data via self-supervised learning and then reusing the adapted model for further adaptation to future, novel target domains (Agrawal et al., 12 Sep 2025). CoSDA similarly studies continuous adaptation to multiple target domains without access to source data or prior domain IDs, with explicit emphasis on preserving performance across all seen domains (Feng et al., 2023). C-SUDA formulates continual source-free unsupervised domain adaptation as adaptation to target domains with varying distributional shifts while maintaining performance on the unseen source (Ahmed et al., 2023).

Other formulations enlarge the problem along the label-space axis. LwS addresses continual semantic segmentation under both task and domain shift: at each step $t$, a new class set $\mathcal{C}_t$ and a new domain $\mathcal{X}_t$ are introduced, supervision is available only for $\mathcal{C}_t$ on $\mathcal{X}_t$, and data from that step is discarded after training (Toldo et al., 2022). GMM-COMET extends source-free continual adaptation to continual SF-UniDA, where the model must adapt sequentially to a stream of multiple different unlabeled target domains and the target label space may differ from the source label space, including PDA, ODA, and OPDA (Schlachter et al., 16 Jan 2026).

A further extension removes domain-identity information at inference. RAIL introduces Cross-domain Task-Agnostic Incremental Learning (X-TAIL), where a learner must incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint (Xu et al., 2024). In multimodal settings, CL-CrossVQA defines sequential cross-domain task-incremental learning for VQA, with no access to source data after task transition and mostly disjoint answer spaces per task (Zhang et al., 2022).

These settings are not interchangeable. Some assume fixed task semantics with changing input distributions, some introduce new classes or answers, and some require explicit unknown-class rejection. A plausible implication is that “source-free cross-domain continual learning” is better understood as a family of source-restricted sequential adaptation problems rather than a single benchmark protocol.

2. Principal failure modes

The most recurrent failure mode is catastrophic forgetting. CoSDA states that previous SFDA approaches suffer forgetting on the source domain due to the lack of data and reports a trade-off between adaptation gain and forgetting loss (Feng et al., 2023). REFEREE sharpens this diagnosis by introducing the notion of double catastrophic forgetting (DCF): in continual streams in both source and target domains, both sides may suffer forgetting (Furqon et al., 2 Oct 2025). In CL-CrossVQA, sequential fine-tuning yields strongly negative backward transfer, confirming severe forgetting in cross-domain multimodal adaptation (Zhang et al., 2022).

A second failure mode is domain-shift-induced pseudo-label corruption. REFEREE identifies noisy pseudo-labels as a central challenge because target adaptation relies solely on pseudo-labels generated from models trained on source data, which are especially error-prone under domain shift (Furqon et al., 2 Oct 2025). LwS makes a related point for segmentation: old domain data is unavailable, and distribution shifts interfere with classical distillation or rehearsal for knowledge retention (Toldo et al., 2022). GMM-COMET addresses this in continual SF-UniDA by combining pseudo-labeling with out-of-distribution detection and adaptive thresholds for known, unknown, and ignored samples (Schlachter et al., 16 Jan 2026).

A third issue concerns the geometry of transferred representations. SCoDA argues that prevailing SFDA methods align instance-level features using cosine similarity over $L_2$-normalized feature vectors and thereby discard crucial geometric information about the latent manifold of the source model (Agrawal et al., 12 Sep 2025). CP-Prompt identifies an analogous representational deficit in cross-modal DIL: existing methods cause high forgetting rates by lacking intra-domain knowledge extraction and an inter-domain common prompting strategy (Feng et al., 2024). The shared diagnosis is that pointwise matching alone is often insufficient under continual domain shift.

These failure modes interact. Domain shift degrades pseudo-label quality; noisy pseudo-labels destabilize adaptation; unstable adaptation accelerates forgetting; and representational collapse can further reduce transfer to unseen domains. This suggests that robust systems require both a stable temporal mechanism and a domain-aware representational mechanism.

3. Methodological families

Representative systems can be grouped by the mechanism used to preserve prior knowledge while adapting to new domains.

Method family	Representative papers	Core mechanism
Mean-teacher / dual-speed adaptation	SCoDA, CoSDA, GMM-COMET	EMA teacher, consistency, stabilized pseudo-labeling
Replay by transformation or synthesis	LwS, C-SUDA	Stylized replay or synthetic source-style images
Prompting and analytic classifiers	REFEREE, CP-Prompt, RAIL	Prompt isolation, frozen backbones, analytic updates

Teacher–student methods use a slowly evolving teacher as long-term memory. SCoDA departs from standard SFDA by replacing supervised source pre-training with fully self-supervised initialization, such as BYOL, and by combining instance-level feature matching with a Space Similarity Loss for geometric manifold alignment (Agrawal et al., 12 Sep 2025). CoSDA employs a dual-speed optimized teacher–student pair: the student is updated rapidly on the current target domain using consistency and mutual information losses, while the teacher is updated slowly via EMA and serves as the global model (Feng et al., 2023). GMM-COMET integrates GMM-based pseudo-labeling inside a mean teacher framework and adds consistency losses for robustness over long adaptation sequences (Schlachter et al., 16 Jan 2026).

A second family reconstructs or approximates prior domains without replaying source data. LwS stores, for each domain, the average low-frequency amplitude spectrum window as a compact style signature, then generates oldly-stylized images by applying past-domain style to current-domain content (Toldo et al., 2022). C-SUDA instead synthesizes class-conditioned source-style images by optimizing target-initialized images against the frozen source model with cross-entropy, total variation, and BatchNorm feature regularization (Ahmed et al., 2023). Both approaches avoid raw source replay while reintroducing source- or past-domain structure into the training signal.

A third family minimizes parameter drift by freezing most of the model. REFEREE is rehearsal-free and gradient-free during target adaptation: a source-pretrained ViT branch is fused with a frozen CLIP branch, while only class statistics are updated through uncertainty-aware KLDA (Furqon et al., 2 Oct 2025). CP-Prompt tunes only prompts—about $0.22\%$ of parameters per domain—using common prompts for inter-domain knowledge and personalized Prefix-One prompts inserted into attention keys and values for intra-domain adaptation (Feng et al., 2024). RAIL freezes the CLIP backbone, learns a recursive ridge-regression adapter, and uses a training-free fusion module so that zero-shot predictions are preserved on unseen domains without reference data (Xu et al., 2024).

These methodological families are complementary rather than exclusive. CoSDA is explicitly described as plug-and-play with other SFDA mechanisms, and REFEREE combines prompt-like test-time adaptation with analytic classification. A plausible implication is that future systems may hybridize EMA stabilization, structured pseudo-labeling, and parameter-efficient adaptation.

4. Canonical objectives and update rules

SCoDA formalizes continual source-free adaptation with a composite objective

$f_\psi$0

where $f_\psi$1 aligns each sample’s student feature with the teacher feature, and $f_\psi$2 aligns activation patterns for each feature dimension across the batch, preserving geometric manifold structure. Its teacher update is

$f_\psi$3

with EMA momentum $f_\psi$4 such as $f_\psi$5 (Agrawal et al., 12 Sep 2025).

CoSDA uses mixup to contract features toward domain centroids,

$f_\psi$6

followed by a consistency loss on the mixed sample and a mutual-information term. The student objective is

$f_\psi$7

This design is intended to improve robustness on hard domains and to reduce abrupt forgetting under continuous adaptation (Feng et al., 2023).

LwS decomposes continual segmentation under class and domain shift into four sub-tasks: cross-entropy for new classes on self-stylized new-domain data, cross-entropy for new classes on oldly-stylized data, pseudo-label-based distillation for old classes on the new domain, and distillation on oldly-stylized data for old classes on old domains. The total loss is

$f_\psi$8

The formulation explicitly separates old versus new classes and old versus new domains (Toldo et al., 2022).

REFEREE quantifies pseudo-label reliability through entropy-weighted fusion of a source-pretrained branch and a VLM branch. With fused prediction $f_\psi$9, the sample weight is

$h$0

and these weights enter the updates of class means and the shared covariance used by KLDA (Furqon et al., 2 Oct 2025). GMM-COMET similarly combines pseudo-labeling and stability, with the overall objective

$h$1

where the contrastive, entropy, source-consistency, and student–teacher consistency terms are all defined on unlabeled target batches (Schlachter et al., 16 Jan 2026).

In VLM-based task-agnostic continual learning, RAIL uses a training-free fusion rule

$h$2

applied only when CLIP predicts a seen class; if CLIP predicts an unseen class, zero-shot CLIP output is used directly (Xu et al., 2024). This sharply differs from continual SFDA objectives that update a student network end-to-end.

5. Empirical landscape

The empirical record is heterogeneous and benchmark-dependent. On Office-Home with three tasks, Art $h$3 Clipart/Product/Real World, SCoDA reports an average accuracy of $h$4, compared with $h$5 for Source-Only, $h$6 for SHOT, $h$7 for NRC, $h$8 for AaD, and $h$9 for LFTL. The same study reports a $t$0 absolute improvement over pre-adaptation and states that source performance drops less than $t$1 during adaptation (Agrawal et al., 12 Sep 2025). On DomainNet Infograph $t$2 Clipart, however, the reported numbers are SHOT $t$3, NRC $t$4, AaD $t$5, CoSDA $t$6, and SCoDA $t$7, indicating that gains are not uniform across transfer pairs (Agrawal et al., 12 Sep 2025).

CoSDA emphasizes the transferability-forgetting trade-off. On DomainNet single-target adaptation it reports average target accuracy and source accuracy drop of $t$8 for SHOT, $t$9 for DaC, $\mathcal{C}_t$0 for GSFDA, $\mathcal{C}_t$1 for CoTTA, and $\mathcal{C}_t$2 for CoSDA, with CoSDA (+NRC) at $\mathcal{C}_t$3. In multi-target sequential adaptation, CoSDA reports BWT $\mathcal{C}_t$4 on DomainNet and $\mathcal{C}_t$5 on OfficeHome, and CoSDA + NRC improves BWT from $\mathcal{C}_t$6 to $\mathcal{C}_t$7 on DomainNet (Feng et al., 2023).

C-SUDA focuses on source retention under continual source-free UDA. On PACS, for Art $\mathcal{C}_t$8 Cartoon, source performance after SUDA is $\mathcal{C}_t$9 and after C-SUDA is $\mathcal{X}_t$0, while target accuracy remains above $\mathcal{X}_t$1. On VisDA-C, average target accuracy is $\mathcal{X}_t$2, and source performance increases from $\mathcal{X}_t$3 with SUDA to $\mathcal{X}_t$4 with C-SUDA. On DomainNet, C-SUDA reports average accuracy $\mathcal{X}_t$5 versus $\mathcal{X}_t$6 on source recall after SUDA (Ahmed et al., 2023).

In segmentation, LwS reports that the relative performance gap to a fully supervised oracle, $\mathcal{X}_t$7, is reduced by more than $\mathcal{X}_t$8 compared to the best CIL/DIL competitor across various datasets and step orders. On Cityscapes $\mathcal{X}_t$9 BDD $\mathcal{C}_t$0 IDD, it reduces final-step average performance gap relative to oracle from more than $\mathcal{C}_t$1 for PLOP and UCD to about $\mathcal{C}_t$2, and it is more than $\mathcal{C}_t$3 better in generalization mIoU to unseen Mapillary than state-of-the-art methods (Toldo et al., 2022).

REFEREE reports especially large margins despite not using source samples for adaptation. It gives $\mathcal{C}_t$4 average accuracy on VisDA versus $\mathcal{C}_t$5 for CLAMP, $\mathcal{C}_t$6 on Office-31 versus $\mathcal{C}_t$7 for CLAMP, and $\mathcal{C}_t$8 on large-scale DomainNet versus $\mathcal{C}_t$9 for CLAMP and $\mathcal{X}_t$0 for CDCL. Its ablations report a $\mathcal{X}_t$1 drop when the CLIP branch is removed, $\mathcal{X}_t$2 without frequency-aware augmentation, and $\mathcal{X}_t$3 without uncertainty weighting (Furqon et al., 2 Oct 2025).

Parameter-efficient and multimodal methods yield a different empirical profile. CP-Prompt reports $\mathcal{X}_t$4 average accuracy and average forgetting $\mathcal{X}_t$5 on CDDB-Hard, $\mathcal{X}_t$6 on CORe50, and $\mathcal{X}_t$7 on DomainNet, while tuning only about $\mathcal{X}_t$8 of parameters per domain (Feng et al., 2024). RAIL reports state-of-the-art performance in both X-TAIL and existing multi-domain task-incremental learning settings and states that its zero-shot accuracy on unseen classes remains identical to the frozen CLIP (Xu et al., 2024). CL-CrossVQA, by contrast, shows that replay-based methods such as ER and DERPP mitigate forgetting substantially, adapters almost entirely prevent forgetting in single-stream models, and ALBEF is the most resistant architecture, with sequential ViLT exhibiting BWT around $\mathcal{X}_t$9 (Zhang et al., 2022).

6. Extensions, distinctions, and open directions

A recurrent distinction in the literature is between source-free and rehearsal-free adaptation. REFEREE is explicitly rehearsal-free and stores only class statistics; LwS is source-free but stores compact average style per domain; CP-Prompt is source-free but retains prompts; CrossWorld-CL is annotation-free and privacy-safe because replay uses public auxiliary ImageNet images rather than downstream data (Furqon et al., 2 Oct 2025, Toldo et al., 2022, Feng et al., 2024, Kuchibhotla et al., 24 Nov 2025). Accordingly, source-free does not imply zero memory, and the memory object may be a style tensor, prompt, mean and covariance estimate, or public auxiliary set.

Another distinction concerns the target label space. Much of continual SFDA assumes the source label space remains relevant, but continual SF-UniDA explicitly handles missing, extra, and overlapping target classes through OOD detection and unknown-class assignment (Schlachter et al., 16 Jan 2026). This broadens the field from “adaptation under domain shift” to “adaptation under simultaneous domain and category shift.”

Recent work also extends the problem beyond standard image classification. CL-CrossVQA studies cross-domain continual learning for VQA with Vision-and-Language Pre-trained Models (Zhang et al., 2022). RAIL addresses preservation of VLM zero-shot ability without reference datasets or domain-identity hints (Xu et al., 2024). CrossWorld-CL goes further to Annotation-Free Class-Incremental Learning, where new classes arrive without labels and external world knowledge from ImageNet is used for supervision and replay (Kuchibhotla et al., 24 Nov 2025). These developments suggest that source-free cross-domain continual learning is becoming a design principle for broader continual foundation-model adaptation.

Several open directions follow directly from these results. One is the balance between adaptation and stability: CoSDA explicitly reports a trade-off between adaptation gain and forgetting loss (Feng et al., 2023). A second is pseudo-label reliability under severe shift, which drives designs such as uncertainty-aware weighting, hard pseudo-label distillation, GMM likelihoods, and dual-branch fusion (Furqon et al., 2 Oct 2025, Toldo et al., 2022, Schlachter et al., 16 Jan 2026). A third is domain-agnostic inference: X-TAIL removes task IDs, and CL-CrossVQA shows large architecture-dependent differences in forgetting under cross-domain streams (Xu et al., 2024, Zhang et al., 2022). A plausible implication is that the next phase of the field will be defined less by source-free adaptation alone than by simultaneous constraints: no source data, no domain hints, evolving label spaces, and preservation of zero-shot or open-world competence.