Self-supervised Continual Domain Adaptation

• Self-supervised Continual Domain Adaptation (SCoDA) is a research area that adapts models to sequential, unlabeled domain shifts by leveraging self-supervised signals alongside continual learning.
• Methods in SCoDA employ techniques like teacher–student frameworks, dual-network dynamics, and geometric alignment to simultaneously adapt and preserve prior domain performance.
• SCoDA strategies mitigate catastrophic forgetting through mechanisms such as latent replay, EMA teacher averaging, and synthetic rehearsal, ensuring robust performance across evolving domains.

Self-supervised Continual Domain Adaptation (SCoDA) denotes a family of adaptation settings and methods in which a model is exposed to a sequence of domain shifts, receives little or no labeled supervision during adaptation, and must preserve competence on previously encountered data rather than specializing only to the current domain. In the vision literature, closely related formulations include self-supervised continual domain adaptation, lifelong self-supervised domain adaptation, and continual source-free domain adaptation; in recent language-model work, the same term has also been extended to continual pretraining on unlabeled in-domain text while preserving instruction-following capability. Across these variants, the recurring technical problem is the joint management of domain discrepancy and catastrophic forgetting under sequential, label-scarce, or source-free constraints (Thota et al., 2022, Agrawal et al., 12 Sep 2025, Feng et al., 2023, Zhang et al., 23 Oct 2025).

1. Problem formulation and scope

In the lifelong self-supervised formulation, a sequence of domains $D_1, \ldots, D_T$ arrives in a stream, and at time $t$ the learner only has access to unlabeled samples from $D_t$, optionally together with a bounded memory $M$ containing latent representations saved from earlier domains. The objective is to learn an encoder that adapts to each new domain while retaining performance on earlier domains, without access to labels. This setting sits at the intersection of UDA, CL, and self-supervised learning, and it is explicitly distinguished from test-time adaptation because training continues as new data arrive rather than only at inference (Thota et al., 2022).

A second major formulation is continual source-free domain adaptation. Here the source dataset is not available during adaptation; only a source-trained model is accessible. In the 2025 paper titled "SCoDA," the initialization is further changed from a supervised source model to a model pre-trained entirely via self-supervision, and adaptation proceeds over unlabeled target domains using a teacher–student framework. CoSDA studies the same source-free continual regime, but begins from a source-trained model and emphasizes the adaptation–forgetting trade-off under sequential unlabeled targets with no domain ID at test time (Agrawal et al., 12 Sep 2025, Feng et al., 2023).

The literature also uses the term more broadly. GRCL studies continual domain adaptation with a labeled source domain and a sequence of unlabeled target domains, combining supervised source anchoring with target-side contrastive learning and constrained updates. IKnow transfers the idea to instruction-tuned LLMs: only unlabeled test-time domain text is available before queries arrive, the original base model weights are unavailable, and the goal is to strengthen domain-specific semantic representations without degrading instruction-following capability (Su et al., 2020, Zhang et al., 23 Oct 2025).

This suggests that SCoDA is best understood as a research area rather than a single protocol. The common denominator is continual adaptation under distribution shift with self-supervised or label-free adaptation signals, but the precise assumptions differ in whether a labeled source domain exists, whether source weights are supervised or self-supervised, and whether replay, synthesis, or teacher–student stabilization is permitted.

2. Recurrent design patterns in SCoDA methods

Several architectural motifs recur across the literature.

Method	Setting	Distinctive mechanism
LLEDA	Self-supervised continual domain adaptation	Dual networks, latent replay, MMD-based DA1/DA2 (Thota et al., 2022)
CoSDA	Continual source-free DA	Dual-speed teacher–student, mixup consistency, mutual information (Feng et al., 2023)
SCoDA	Source-free continual adaptation with SSL initialization	EMA teacher–student, instance matching, Space Similarity Loss (Agrawal et al., 12 Sep 2025)
GRCL	Continual DA with labeled source and unlabeled targets	Contrastive learning with gradient regularization and memory (Su et al., 2020)
C-SUDA	Continual source-free UDA	Pseudo-label refinement plus class-conditioned source-style synthesis (Ahmed et al., 2023)
IKnow	Continual adaptation for instruction-tuned LLMs	Instruction-formatted self-supervision with MTP, MPP, and NL↔KG (Zhang et al., 23 Oct 2025)

One dominant pattern is the use of asymmetric dynamics. LLEDA instantiates a fast-adapting DA network and a slow-learning SSL network, explicitly motivated by Complementary Learning Systems. CoSDA and the 2025 SCoDA paper instead use teacher–student pairs in which the student changes quickly and the teacher evolves by EMA. Although the mechanisms differ, all three designs implement a stability–plasticity split between rapid accommodation of the current domain and slower retention of accumulated structure (Thota et al., 2022, Feng et al., 2023, Agrawal et al., 12 Sep 2025).

A second pattern is that continual adaptation is rarely reduced to pointwise feature matching alone. LLEDA couples self-supervised learning with domain alignment at two levels; the 2025 SCoDA paper augments instance-level cosine alignment with manifold-aware Space Similarity; CoSDA combines teacher consistency with mutual-information regularization; GRCL constrains updates so that adaptation gradients do not increase source or memory losses. These methods differ in objective form, but each attempts to preserve structural properties of the representation space rather than merely matching current predictions (Thota et al., 2022, Agrawal et al., 12 Sep 2025, Feng et al., 2023, Su et al., 2020).

A third pattern concerns memory. LLEDA stores block-1 latent representations rather than raw pixels; GRCL stores exemplar images and a feature bank; CoSDA and the 2025 SCoDA paper avoid replay buffers entirely; C-SUDA synthesizes source-style images instead of replaying source data. The field therefore contains both memory-based and memory-free branches, and source-free privacy constraints are addressed either by compressed latent replay or by entirely avoiding stored source samples (Thota et al., 2022, Su et al., 2020, Feng et al., 2023, Ahmed et al., 2023).

3. Objectives and optimization strategies

LLEDA combines self-supervised representation learning with domain adaptation losses. With VICReg as the default SSL backbone, the embedding loss is

$$l(z_i, z_j) = \lambda s(z_i, z_j) + \mu [\upsilon(z_i) + \upsilon(z_j)] + \nu [c(z_i) + c(z_j)],$$

and the overall SSL objective is

$$L_{\mathrm{ssl}} = \sum_{I \in D} \sum_{t_i,t_j \sim T} l(z_i, z_j).$$

For adaptation, LLEDA uses MMD both for intra-domain cross-network alignment $\mathrm{MMD}(s_4, d_4)$ and for cross-domain alignment between current data and replayed memory on the transformed feature $d_4' = d_4 \otimes s_4$. The total loss is

$$L_{\mathrm{total}} = L_{\mathrm{ssl}} + \alpha L_{\mathrm{DA1}}(S_d) + \beta L_{\mathrm{DA1}}(S_m) + \gamma L_{\mathrm{DA2}}(S_d, S_m),$$

with DA losses backpropagated through both networks except the frozen lower layers (Thota et al., 2022).

The 2025 method titled "SCoDA" retains the standard instance-wise cosine alignment

$$L_{\mathrm{inst}} = 1 - \frac{1}{B}\sum_{i=1}^{B}\frac{\langle F_{\theta,i},F_{\psi,i}\rangle}{\|F_{\theta,i}\|_2 \|F_{\psi,i}\|_2},$$

but adds a Space Similarity Loss that aligns row vectors of the transposed feature matrices: $t$0 The composite objective is

$t$1

with $t$2 in the experiments. The stated motivation is that cosine alignment over $t$3-normalized features can erase geometric information carried by the latent manifold, whereas the row-wise space term preserves cross-sample activation structure at $t$4 cost rather than the $t$5 cost of Gram-matrix alignment (Agrawal et al., 12 Sep 2025).

CoSDA uses teacher-generated pseudo-label distributions under mixup and augments consistency learning with mutual-information regularization. The student objective is

$t$6

where $t$7 is the cross-entropy between mixed teacher labels and student predictions, and $t$8 encourages confident per-sample predictions while preventing collapse to a few classes. GRCL instead formulates continual adaptation as a constrained optimization problem: it minimizes a contrastive objective subject to nonnegative gradient inner products with the source classification loss and the memory classification loss, then solves a two-variable dual quadratic program to obtain the closest feasible update to the raw contrastive gradient (Feng et al., 2023, Su et al., 2020).

Other SCoDA-style systems use different self-supervised signals. C-SUDA deliberately relies on pseudo-label refinement and deep inversion rather than feature-space distillation, while IKnow reformulates self-supervision into instruction–response objectives such as Masked Token Prediction, Masked Phrase Prediction, NL→KG extraction, and KG→NL verbalization, all optimized as conditional negative log-likelihood in dialogue format (Ahmed et al., 2023, Zhang et al., 23 Oct 2025).

4. Mechanisms for mitigating catastrophic forgetting

Replay is the most explicit anti-forgetting mechanism in LLEDA. The framework stores latent representations extracted at block 1 of both networks, freezes layers below block 1 after the first domain’s training, and interleaves memory minibatches with current minibatches every iteration. The stated purpose is to preserve latent compatibility, avoid “aging” of stored features, and reduce gradient interference across domains. Empirically, stronger DA2 weighting improves retention across larger domain shifts, and increasing the latent memory per domain from 100 to 250 improves average accuracy on Digits from 86.6 to 89.5 (Thota et al., 2022).

Teacher–student averaging is the main stabilization mechanism in CoSDA and the 2025 SCoDA paper. In the latter, teacher parameters are updated as

$t$9

with $D_t$0 used as the momentum parameter, and BatchNorm statistics are also updated via EMA. The paper explicitly attributes improved robustness over a stream of domains to this slowly changing target. CoSDA similarly updates the teacher and BatchNorm statistics via EMA, but at epoch scale with a cosine-scheduled momentum increasing from approximately $D_t$1 to $D_t$2, making the teacher the new global model after each target domain (Agrawal et al., 12 Sep 2025, Feng et al., 2023).

GRCL addresses forgetting through gradient constraints rather than teacher averaging. By requiring the projected update $D_t$3 to satisfy $D_t$4 and $D_t$5, the method aims, under first-order Taylor approximation, to prevent increases in source cross-entropy and memory classification loss. This ties retention to update geometry rather than to stored teacher parameters (Su et al., 2020).

C-SUDA replaces explicit replay with rehearsal by synthesis. After refining pseudo-labels, it selects low-entropy target samples and optimizes them with a frozen source model using cross-entropy, total variation regularization, and BatchNorm-statistics regularization to synthesize class-conditioned source-style images. The final model is initialized from the source model, the fully connected layers are frozen, and the feature extractor is trained on the mixture of refined real target samples and synthetic source-style samples. This design is intended to preserve the source decision boundary without source data (Ahmed et al., 2023).

For instruction-tuned LLMs, IKnow argues that catastrophic forgetting is triggered by naively applying plain NTP or MLM to an instruction-tuned checkpoint. Its mitigation strategy is not replay or explicit distillation, but instruction-style formatting, conservative learning rates, short training horizon, LoRA when needed, and early stopping at an “inflection” epoch. A plausible implication is that, in language-model SCoDA, preservation of the interaction format itself functions as a form of alignment regularization (Zhang et al., 23 Oct 2025).

5. Evaluation protocols and empirical behavior

Evaluation protocols vary, but all emphasize final performance after sequential adaptation. LLEDA uses cycles: the model is trained sequentially on each domain in a predefined order, then the encoders are frozen and a linear classifier is trained on the fixed representations, after which accuracy is reported on all domains seen in the cycle together with the average. The primary metric reported is average accuracy across domains in a cycle, although the paper notes standard continual-learning metrics such as forgetting $D_t$6, backward transfer (BWT), and forward transfer (FWT). Under this protocol, LLEDA-250 reports 89.5 on Digits, 62.1 on Office-Home, 90.3 on Office-Caltech, and 92.6 on ImageCLEF-DA, outperforming the listed unlabeled baselines in those settings (Thota et al., 2022).

The 2025 SCoDA paper uses standard SFDA evaluation by top-1 accuracy on the target domain. On Office-Home for Ar→Cl, Ar→Pr, and Ar→Rw, it reports a Source-Only average of 47.63, a CoSDA average of 70.28, and an SCoDA post-adaptation average of 86.59, with individual results of 75.60, 93.62, and 90.54. On DomainNet I→C, it reports Source-Only 13.43, CoSDA 40.97, AaD 47.12, and SCoDA 37.68, explicitly noting that on this large shift SCoDA improves over weak baselines but lags behind AaD and CoSDA (Agrawal et al., 12 Sep 2025).

GRCL reports both ACC and BWT over sequential target streams. The published values are 85.34 ± 0.10 ACC and $D_t$7 BWT on Digits, 37.74 ± 0.13 ACC and $D_t$8 BWT on DomainNet, and 87.23 ± 0.06 ACC with $D_t$9 BWT on Office-Caltech. The paper attributes these outcomes to the combination of contrastive learning with gradient regularization, and reports that larger memory improves ACC with gains saturating between 1024 and 2048 exemplars per domain (Su et al., 2020).

CoSDA emphasizes the adaptation–forgetting trade-off. In multi-target sequential adaptation, the paper reports BWT of approximately $M$0 on the DomainNet chain Real→Infograph→Clipart→Painting→Sketch→Quickdraw and approximately $M$1 on the OfficeHome chain Art→Clipart→Product→Real. It also reports that integrating CoSDA into NRC improves BWT from $M$2 to $M$3 on DomainNet while increasing average adaptation accuracy by at least 12% across the middle domains, and that adding CoSDA to AaD improves BWT from $M$4 to $M$5 with about 11% average accuracy gain on the same mid-chain targets (Feng et al., 2023).

C-SUDA evaluates both target accuracy and retained source accuracy after adaptation. On PACS averaged across pairs, SUDA reports Tg = 77.2 and Sc = 46.3, whereas C-SUDA reports Tg = 77.1 and Sc = 78.0, indicating markedly better source retention with nearly unchanged target performance. On DomainNet, SUDA* source accuracy after adaptation is 29.2 and C-SUDA* is 62.6. For LLM adaptation, IKnow reports that instruction-style pretraining tasks outperform naive NTP in 19/24 cases, and that knowledge-focused tasks outperform MTP in 8/16 cases; for example, on Llama-3.2-3B-Instruct, NLKG improves RepliQA from 34.71 to 36.83, and MPP improves SciQAG from 39.52 to 40.11 (Ahmed et al., 2023, Zhang et al., 23 Oct 2025).

6. Limitations, misconceptions, and research directions

A common misconception is that SCoDA is synonymous with pseudo-label self-training. The literature does not support that equivalence. LLEDA uses self-supervised objectives and MMD alignment without source labels; the 2025 SCoDA method explicitly avoids pseudo-labeling and classification heads during adaptation; CoSDA depends on teacher-generated pseudo-label distributions rather than hard target labels; C-SUDA, by contrast, does use pseudo-label refinement and synthesis. The methodological spectrum is therefore broader than pseudo-labeling alone (Thota et al., 2022, Agrawal et al., 12 Sep 2025, Feng et al., 2023, Ahmed et al., 2023).

Another misconception is that all continual DA methods in this area are source-free and label-free in exactly the same way. GRCL still uses supervised source loss during continual adaptation, whereas the 2025 SCoDA paper assumes a source model pre-trained entirely via self-supervision and forbids source samples during adaptation. IKnow removes not only source data but also access to the original base model weights, showing that SCoDA assumptions have already expanded beyond the original vision formulations (Su et al., 2020, Agrawal et al., 12 Sep 2025, Zhang et al., 23 Oct 2025).

The limitations reported in the papers are correspondingly heterogeneous. LLEDA may underperform with extremely small memory capacity, highly non-stationary shifts, poor augmentations, or poor MMD kernel choice. The 2025 SCoDA paper notes that extreme shifts such as DomainNet I→C can challenge geometry alignment alone, that label shift is not explicitly addressed, and that transformer-based DINO can suffer collapse or severe forgetting without centering and sharpening. CoSDA remarks that MI can harm datasets with many semantically similar classes if weighted too strongly and that poor early teacher predictions can degrade pseudo-label quality. C-SUDA depends on source-model quality and on the quality of pseudo-label refinement, while IKnow depends on parse quality, has only been tested on datasets such as RepliQA and SciQAG, and uses the full test corpus rather than true per-sample test-time training (Thota et al., 2022, Agrawal et al., 12 Sep 2025, Feng et al., 2023, Ahmed et al., 2023, Zhang et al., 23 Oct 2025).

The research directions named in the literature are similarly diverse. For LLEDA and related vision systems, the stated open challenges include adaptive replay scheduling, prototype-based or compressed memories, kernel selection for MMD in complex shifts, and integration with test-time adaptation. The 2025 SCoDA paper proposes semi-supervised target adaptation, multi-source or multi-target alignment, online SSL updates, alternative geometric metrics such as Gram or Laplacian alignment, and transformer compatibility through DINO-specific stabilizers. IKnow points toward combined objective training, broader task coverage, and larger-scale LLM validation. Taken together, these proposals indicate that future SCoDA systems are likely to combine geometric preservation, continual stabilization, and modality-specific self-supervision rather than relying on any single recipe (Thota et al., 2022, Agrawal et al., 12 Sep 2025, Zhang et al., 23 Oct 2025).