---
title: Source-Free Cross-Domain Continual Learning
url: https://www.emergentmind.com/topics/source-free-cross-domain-continual-learning
type: topic
---

# Source-Free Cross-Domain Continual Learning

Source-free cross-domain continual learning studies how a model pre-trained on a labeled source domain can be adapted to a sequence of target domains when source-domain samples are unavailable during adaptation. In the literature, this umbrella includes source-free domain adaptation to unlabeled targets, continual source-free adaptation across multiple target domains, joint task and domain shift, universal domain adaptation with changing label spaces, and task-agnostic inference without domain-identity hints. Across these variants, the central objective is stable plasticity under domain shift: acquiring new domain-specific competence while limiting catastrophic forgetting of previously learned domains, classes, or zero-shot capabilities [2509.09935] [2510.01649] [2210.07016] [2601.11161] [2406.18868].

## 1. Problem setting and taxonomic scope

A basic formulation is Source-Free Domain Adaptation (SFDA), which adapts a model to a target domain without access to the data of the source domain. SCoDA places this formulation in a continual setting by initializing both teacher $f_\theta$ and student $f_\psi$ from a model $h$ pre-trained on source data via self-supervised learning and then reusing the adapted model for further adaptation to future, novel target domains [2509.09935]. CoSDA similarly studies continuous adaptation to multiple target domains without access to source data or prior domain IDs, with explicit emphasis on preserving performance across all seen domains [2304.06627]. C-SUDA formulates continual source-free unsupervised domain adaptation as adaptation to target domains with varying distributional shifts while maintaining performance on the unseen source [2304.07374].

Other formulations enlarge the problem along the label-space axis. LwS addresses continual semantic segmentation under both task and domain shift: at each step $t$, a new class set $\mathcal{C}_t$ and a new domain $\mathcal{X}_t$ are introduced, supervision is available only for $\mathcal{C}_t$ on $\mathcal{X}_t$, and data from that step is discarded after training [2210.07016]. GMM-COMET extends source-free continual adaptation to continual SF-UniDA, where the model must adapt sequentially to a stream of multiple different unlabeled target domains and the target label space may differ from the source label space, including PDA, ODA, and OPDA [2601.11161].

A further extension removes domain-identity information at inference. RAIL introduces Cross-domain Task-Agnostic Incremental Learning (X-TAIL), where a learner must incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint [2406.18868]. In multimodal settings, CL-CrossVQA defines sequential cross-domain task-incremental learning for VQA, with no access to source data after task transition and mostly disjoint answer spaces per task [2211.10567].

These settings are not interchangeable. Some assume fixed task semantics with changing input distributions, some introduce new classes or answers, and some require explicit unknown-class rejection. A plausible implication is that “source-free cross-domain continual learning” is better understood as a family of source-restricted sequential adaptation problems rather than a single benchmark protocol.

## 2. Principal failure modes

The most recurrent failure mode is catastrophic forgetting. CoSDA states that previous SFDA approaches suffer forgetting on the source domain due to the lack of data and reports a trade-off between adaptation gain and forgetting loss [2304.06627]. REFEREE sharpens this diagnosis by introducing the notion of double catastrophic forgetting (DCF): in continual streams in both source and target domains, both sides may suffer forgetting [2510.01649]. In CL-CrossVQA, sequential fine-tuning yields strongly negative backward transfer, confirming severe forgetting in cross-domain multimodal adaptation [2211.10567].

A second failure mode is domain-shift-induced pseudo-label corruption. REFEREE identifies noisy pseudo-labels as a central challenge because target adaptation relies solely on pseudo-labels generated from models trained on source data, which are especially error-prone under domain shift [2510.01649]. LwS makes a related point for segmentation: old domain data is unavailable, and distribution shifts interfere with classical distillation or rehearsal for knowledge retention [2210.07016]. GMM-COMET addresses this in continual SF-UniDA by combining pseudo-labeling with out-of-distribution detection and adaptive thresholds for known, unknown, and ignored samples [2601.11161].

A third issue concerns the geometry of transferred representations. SCoDA argues that prevailing SFDA methods align instance-level features using cosine similarity over $L_2$-normalized feature vectors and thereby discard crucial geometric information about the latent manifold of the source model [2509.09935]. CP-Prompt identifies an analogous representational deficit in cross-modal DIL: existing methods cause high forgetting rates by lacking intra-domain knowledge extraction and an inter-domain common prompting strategy [2407.21043]. The shared diagnosis is that pointwise matching alone is often insufficient under continual domain shift.

These failure modes interact. Domain shift degrades pseudo-label quality; noisy pseudo-labels destabilize adaptation; unstable adaptation accelerates forgetting; and representational collapse can further reduce transfer to unseen domains. This suggests that robust systems require both a stable temporal mechanism and a domain-aware representational mechanism.

## 3. Methodological families

Representative systems can be grouped by the mechanism used to preserve prior knowledge while adapting to new domains.

| Method family | Representative papers | Core mechanism |
|---|---|---|
| Mean-teacher / dual-speed adaptation | SCoDA, CoSDA, GMM-COMET | EMA teacher, consistency, stabilized pseudo-labeling |
| Replay by transformation or synthesis | LwS, C-SUDA | Stylized replay or synthetic source-style images |
| Prompting and analytic classifiers | REFEREE, CP-Prompt, RAIL | Prompt isolation, frozen backbones, analytic updates |

Teacher–student methods use a slowly evolving teacher as long-term memory. SCoDA departs from standard SFDA by replacing supervised source pre-training with fully self-supervised initialization, such as BYOL, and by combining instance-level feature matching with a Space Similarity Loss for geometric manifold alignment [2509.09935]. CoSDA employs a dual-speed optimized teacher–student pair: the student is updated rapidly on the current target domain using consistency and mutual information losses, while the teacher is updated slowly via EMA and serves as the global model [2304.06627]. GMM-COMET integrates GMM-based pseudo-labeling inside a mean teacher framework and adds consistency losses for robustness over long adaptation sequences [2601.11161].

A second family reconstructs or approximates prior domains without replaying source data. LwS stores, for each domain, the average low-frequency amplitude spectrum window as a compact style signature, then generates oldly-stylized images by applying past-domain style to current-domain content [2210.07016]. C-SUDA instead synthesizes class-conditioned source-style images by optimizing target-initialized images against the frozen source model with cross-entropy, total variation, and BatchNorm feature regularization [2304.07374]. Both approaches avoid raw source replay while reintroducing source- or past-domain structure into the training signal.

A third family minimizes parameter drift by freezing most of the model. REFEREE is rehearsal-free and gradient-free during target adaptation: a source-pretrained ViT branch is fused with a frozen CLIP branch, while only class statistics are updated through uncertainty-aware KLDA [2510.01649]. CP-Prompt tunes only prompts—about $0.22\%$ of parameters per domain—using common prompts for inter-domain knowledge and personalized Prefix-One prompts inserted into attention keys and values for intra-domain adaptation [2407.21043]. RAIL freezes the CLIP backbone, learns a recursive ridge-regression adapter, and uses a training-free fusion module so that zero-shot predictions are preserved on unseen domains without reference data [2406.18868].

These methodological families are complementary rather than exclusive. CoSDA is explicitly described as plug-and-play with other SFDA mechanisms, and REFEREE combines prompt-like test-time adaptation with analytic classification. A plausible implication is that future systems may hybridize EMA stabilization, structured pseudo-labeling, and parameter-efficient adaptation.

## 4. Canonical objectives and update rules

SCoDA formalizes continual source-free adaptation with a composite objective
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{cos}} + \lambda \mathcal{L}_{\text{space}},
$$
where $\mathcal{L}_{\text{cos}}$ aligns each sample’s student feature with the teacher feature, and $\mathcal{L}_{\text{space}}$ aligns activation patterns for each feature dimension across the batch, preserving geometric manifold structure. Its teacher update is
$$
\theta_{t+1} \leftarrow m \theta_t + (1 - m)\psi_{t+1},
$$
with EMA momentum $m$ such as $0.999$ [2509.09935].

CoSDA uses mixup to contract features toward domain centroids,
$$
\tilde{x} = \lambda x_i + (1-\lambda) x_j,\qquad \tilde{y} = \lambda y_i + (1-\lambda) y_j,
$$
followed by a consistency loss on the mixed sample and a mutual-information term. The student objective is
$$
\ell_{\psi} = \ell_{\text{cons}} + \alpha\,\ell_{\text{MI}}.
$$
This design is intended to improve robustness on hard domains and to reduce abrupt forgetting under continuous adaptation [2304.06627].

LwS decomposes continual segmentation under class and domain shift into four sub-tasks: cross-entropy for new classes on self-stylized new-domain data, cross-entropy for new classes on oldly-stylized data, pseudo-label-based distillation for old classes on the new domain, and distillation on oldly-stylized data for old classes on old domains. The total loss is
$$
\mathcal{L}_{tot} =
\mathcal{L}_{ce}^{\tilde{n}} +
\lambda_{ce}^{\tilde{o}}\mathcal{L}_{ce}^{\tilde{o}} +
\lambda_{kd}^{\tilde{n}}\mathcal{L}_{kd}^{\tilde{n}} +
\lambda_{kd}^{\tilde{o}}\mathcal{L}_{kd}^{\tilde{o}}.
$$
The formulation explicitly separates old versus new classes and old versus new domains [2210.07016].

REFEREE quantifies pseudo-label reliability through entropy-weighted fusion of a source-pretrained branch and a VLM branch. With fused prediction $\hat{p}_i$, the sample weight is
$$
\omega_i = 1 - \frac{\mathcal{H}(\hat{p}_i)}{\log |\mathcal{C}|},
$$
and these weights enter the updates of class means and the shared covariance used by KLDA [2510.01649]. GMM-COMET similarly combines pseudo-labeling and stability, with the overall objective
$$
\mathcal{L} = \mathcal{L}_c + \lambda_1 \mathcal{L}_e + \lambda_2 \mathcal{L}_{con,src} + \lambda_3 \mathcal{L}_{con,mt},
$$
where the contrastive, entropy, source-consistency, and student–teacher consistency terms are all defined on unlabeled target batches [2601.11161].

In VLM-based task-agnostic continual learning, RAIL uses a training-free fusion rule
$$
\hat{\mathbf{y}}_{fs} = (1-\beta)\,\hat{\mathbf{y}}_{ad} + \beta\,\hat{\mathbf{y}}_{zs},
$$
applied only when CLIP predicts a seen class; if CLIP predicts an unseen class, zero-shot CLIP output is used directly [2406.18868]. This sharply differs from continual SFDA objectives that update a student network end-to-end.

## 5. Empirical landscape

The empirical record is heterogeneous and benchmark-dependent. On Office-Home with three tasks, Art $\rightarrow$ Clipart/Product/Real World, SCoDA reports an average accuracy of $86.59\%$, compared with $47.63$ for Source-Only, $71.73$ for SHOT, $73.33$ for NRC, $73.57$ for AaD, and $86.17$ for LFTL. The same study reports a $+11.79\%$ absolute improvement over pre-adaptation and states that source performance drops less than $0.2\%$ during adaptation [2509.09935]. On DomainNet Infograph $\rightarrow$ Clipart, however, the reported numbers are SHOT $14.82$, NRC $18.01$, AaD $47.12$, CoSDA $40.97$, and SCoDA $37.68$, indicating that gains are not uniform across transfer pairs [2509.09935].

CoSDA emphasizes the transferability-forgetting trade-off. On DomainNet single-target adaptation it reports average target accuracy and source accuracy drop of $56.20\% \mid 24.76\%$ for SHOT, $51.94\% \mid 14.89\%$ for DaC, $48.47\% \mid 8.86\%$ for GSFDA, $45.10\% \mid 3.25\%$ for CoTTA, and $51.63\% \mid 6.52\%$ for CoSDA, with CoSDA (+NRC) at $53.72\% \mid 7.61\%$. In multi-target sequential adaptation, CoSDA reports BWT $-8.6$ on DomainNet and $-2.24$ on OfficeHome, and CoSDA + NRC improves BWT from $-39.48$ to $-8.44$ on DomainNet [2304.06627].

C-SUDA focuses on source retention under continual source-free UDA. On PACS, for Art $\rightarrow$ Cartoon, source performance after SUDA is $58.1\%$ and after C-SUDA is $81.0\%$, while target accuracy remains above $84\%$. On VisDA-C, average target accuracy is $84.6\%$, and source performance increases from $39.7\%$ with SUDA to $45.1\%$ with C-SUDA. On DomainNet, C-SUDA reports average accuracy $51.0\%$ versus $29.2\%$ on source recall after SUDA [2304.07374].

In segmentation, LwS reports that the relative performance gap to a fully supervised oracle, $\bar{\Delta}_t$, is reduced by more than $12\%$ compared to the best CIL/DIL competitor across various datasets and step orders. On Cityscapes $\rightarrow$ BDD $\rightarrow$ IDD, it reduces final-step average performance gap relative to oracle from more than $40\%$ for PLOP and UCD to about $29\%$, and it is more than $5\%$ better in generalization mIoU to unseen Mapillary than state-of-the-art methods [2210.07016].

REFEREE reports especially large margins despite not using source samples for adaptation. It gives $85.1\%$ average accuracy on VisDA versus $30.7\%$ for CLAMP, $90.7\%$ on Office-31 versus $79.7\%$ for CLAMP, and $32.7\%$ on large-scale DomainNet versus $13.3\%$ for CLAMP and $2.8\%$ for CDCL. Its ablations report a $-16\%$ drop when the CLIP branch is removed, $-1.3\%$ without frequency-aware augmentation, and $-0.7\%$ without uncertainty weighting [2510.01649].

Parameter-efficient and multimodal methods yield a different empirical profile. CP-Prompt reports $93.65\%$ average accuracy and average forgetting $-0.25$ on CDDB-Hard, $90.67\%$ on CORe50, and $73.35\%$ on DomainNet, while tuning only about $0.22\%$ of parameters per domain [2407.21043]. RAIL reports state-of-the-art performance in both X-TAIL and existing multi-domain task-incremental learning settings and states that its zero-shot accuracy on unseen classes remains identical to the frozen CLIP [2406.18868]. CL-CrossVQA, by contrast, shows that replay-based methods such as ER and DERPP mitigate forgetting substantially, adapters almost entirely prevent forgetting in single-stream models, and ALBEF is the most resistant architecture, with sequential ViLT exhibiting BWT around $-42.5$ [2211.10567].

## 6. Extensions, distinctions, and open directions

A recurrent distinction in the literature is between source-free and rehearsal-free adaptation. REFEREE is explicitly rehearsal-free and stores only class statistics; LwS is source-free but stores compact average style per domain; CP-Prompt is source-free but retains prompts; CrossWorld-CL is annotation-free and privacy-safe because replay uses public auxiliary ImageNet images rather than downstream data [2510.01649] [2210.07016] [2407.21043] [2511.19344]. Accordingly, source-free does not imply zero memory, and the memory object may be a style tensor, prompt, mean and covariance estimate, or public auxiliary set.

Another distinction concerns the target label space. Much of continual SFDA assumes the source label space remains relevant, but continual SF-UniDA explicitly handles missing, extra, and overlapping target classes through OOD detection and unknown-class assignment [2601.11161]. This broadens the field from “adaptation under domain shift” to “adaptation under simultaneous domain and category shift.”

Recent work also extends the problem beyond standard image classification. CL-CrossVQA studies cross-domain continual learning for VQA with Vision-and-Language Pre-trained Models [2211.10567]. RAIL addresses preservation of VLM zero-shot ability without reference datasets or domain-identity hints [2406.18868]. CrossWorld-CL goes further to Annotation-Free Class-Incremental Learning, where new classes arrive without labels and external world knowledge from ImageNet is used for supervision and replay [2511.19344]. These developments suggest that source-free cross-domain continual learning is becoming a design principle for broader continual foundation-model adaptation.

Several open directions follow directly from these results. One is the balance between adaptation and stability: CoSDA explicitly reports a trade-off between adaptation gain and forgetting loss [2304.06627]. A second is pseudo-label reliability under severe shift, which drives designs such as uncertainty-aware weighting, hard pseudo-label distillation, GMM likelihoods, and dual-branch fusion [2510.01649] [2210.07016] [2601.11161]. A third is domain-agnostic inference: X-TAIL removes task IDs, and CL-CrossVQA shows large architecture-dependent differences in forgetting under cross-domain streams [2406.18868] [2211.10567]. A plausible implication is that the next phase of the field will be defined less by source-free adaptation alone than by simultaneous constraints: no source data, no domain hints, evolving label spaces, and preservation of zero-shot or open-world competence.

Source: https://www.emergentmind.com/topics/source-free-cross-domain-continual-learning