Semantic-Enriched Continual Adaptation (SECA)

Updated 7 July 2026

SECA is a continual adaptation framework that uses textual semantic priors to guide knowledge transfer and refine classifiers across sequential tasks.
It integrates Semantic-Guided Adaptive Knowledge Transfer (SG-AKT) with Semantic-Enhanced Visual Prototype Refinement (SE-VPR) to balance stability and plasticity.
The architecture minimizes catastrophic forgetting by structuring adaptation around semantic relations, demonstrating strong performance on various benchmarks.

Searching arXiv for recent and relevant papers on SECA and related continual adaptation methods. Semantic-Enriched Continual Adaptation (SECA) is a continual learning framework that uses textual semantic priors to guide knowledge transfer and classifier refinement over a stream of tasks. In the specific sense established by “Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning” (He et al., 3 Aug 2025), SECA is designed for class-incremental learning with CLIP and addresses the stability–plasticity dilemma by combining Semantic-Guided Adaptive Knowledge Transfer (SG-AKT) with Semantic-Enhanced Visual Prototype Refinement (SE-VPR). In a broader research sense, the term also usefully names a family of methods in which semantic structure—textual priors, hierarchical labels, semantic maps, or class-aware latent models—organizes continual or online adaptation so that new knowledge can be incorporated while catastrophic forgetting is controlled. This broader interpretation is supported, though not uniformly named, across work on continual semantic segmentation, source-free domain adaptation, robotic adaptation, and semantic skill libraries (Shenaj et al., 2022, Wu et al., 2019, Stan et al., 2024, Frey et al., 2021, Blum et al., 2021, Wang et al., 6 Jun 2026).

1. Definition and scope

In its canonical formulation, SECA is a CLIP-based continual learning method for class-incremental learning (CIL) in which tasks arrive sequentially as

$\mathcal{D} = \{\mathcal{D}^1, \dots, \mathcal{D}^S\},$

with disjoint label spaces

$\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$

and inference after task $S$ must be performed over all seen classes $\mathcal{Y}^{1:S}$ without task identifiers (He et al., 3 Aug 2025). The core SECA setting is rehearsal-free, although a replay-augmented variant, SECA++, is also described (He et al., 3 Aug 2025).

The distinctive feature of SECA is that it does not treat prior knowledge as an undifferentiated constraint. Instead, it uses the structured nature of textual priors in CLIP to decide which historical knowledge is relevant to a new instance and to regularize the geometry of the visual classifier through inter-class semantic relations (He et al., 3 Aug 2025). This differs from semantics-agnostic distillation, in which all prior knowledge is transferred without regard to semantic relevance, and from purely visual prototype methods, in which classifier structure is learned without explicit semantic relations (He et al., 3 Aug 2025).

A broader SECA reading is also justified by adjacent literature. “Continual Coarse-to-Fine Domain Adaptation in Semantic Segmentation” explicitly characterizes its setting as “Semantic-Enriched Continual Adaptation” in the sense that hierarchical semantic relations are built into losses and initialization (Shenaj et al., 2022). “Online Continual Domain Adaptation for Semantic Image Segmentation Using Internal Representations” uses class-conditional latent modeling as a compact semantic memory for source-free adaptation (Stan et al., 2024). “Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations” and “Self-Improving Semantic Perception for Indoor Localisation” use semantic maps or task-aligned semantic affordances as self-supervised signals for continual adaptation in robotics (Frey et al., 2021, Blum et al., 2021). “Continual Quadruped Robots Coordination via Semantic Skill Discovery” instantiates a retrieve–adapt–update process in a shared semantic embedding space for continual multi-robot skill acquisition (Wang et al., 6 Jun 2026). These lines of work do not define a single unified paradigm, but they collectively support the interpretation that SECA denotes continual adaptation in which semantics act as an organizing prior.

2. Core architecture in CLIP-driven SECA

The CLIP-driven SECA framework builds on frozen CLIP encoders and parameter-efficient adaptation. The visual encoder is $\mathcal{F}_\mathrm{V}(\cdot)$ , the text encoder is $\mathcal{F}_\mathrm{T}(\cdot)$ , and both are frozen; trainable components consist of task-specific textual prompts, task-shared visual adapters, semantic projectors used in SG-AKT, and a prototype semantic projector used in SE-VPR (He et al., 3 Aug 2025).

For task $s$ , the text branch constructs prompt-conditioned class representations by forming a token sequence

$\mathbf{E}_y^s = [\mathbf{P}^s]_1[\mathbf{P}^s]_2 \cdots [\mathbf{P}^s]_M[\mathrm{CLASS}_y],$

and then computing

$\mathcal{F}_\mathrm{T}(y; \mathbf{P}^s) = \mathcal{F}_\mathrm{T}(\mathbf{E}_y^s).$

On the visual side, adapters are inserted after FFN blocks in the ViT: $\mathbf{h}_{l+1} = \hat{\mathbf{h}_l} + \mathrm{FFN}(\hat{\mathbf{h}_l}) + \mathcal{A}_l(\hat{\mathbf{h}_l}),$ yielding the final visual feature $\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 0 (He et al., 3 Aug 2025).

The baseline text classifier for current-task labels $\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 1 is

$\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 2

with task loss

$\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 3

SECA augments this hybrid parameter-efficient baseline with two semantic modules: SG-AKT for relevance-aware backbone transfer and SE-VPR for semantically regularized visual classification (He et al., 3 Aug 2025).

This architecture belongs to a larger trend in continual learning toward frozen backbones with lightweight adaptive modules. “Adapter-Enhanced Semantic Prompting for Continual Learning” likewise combines semantic-guided prompts, adapters, and prompt selection in a lightweight continual learning framework intended to reduce catastrophic forgetting and memory growth (Yin et al., 2024). The details available for that method are abstract-level rather than full-paper-level, but the overlap suggests a convergent design logic: preserve a strong pretrained representation, encode semantic structure in lightweight modules, and route or regularize adaptation with semantic cues (Yin et al., 2024).

3. Semantic-Guided Adaptive Knowledge Transfer

SG-AKT addresses a specific limitation of conventional continual distillation: historical knowledge is usually transferred without considering semantic relevance, which can introduce interference from unrelated tasks (He et al., 3 Aug 2025). SECA instead maintains a fixed-capacity pool of frozen historical adapters,

$\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 4

and computes historical visual features for the current image $\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 5 under each adapter,

$\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 6

For the ground-truth class $\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 7, it also collects prompt-conditioned textual embeddings from all tasks seen so far,

$\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 8

These text and visual features are projected into a shared semantic space using $\mathcal{Y}^s \cap \mathcal{Y}^{s'} = \varnothing,\quad s \neq s',$ 9 and $S$ 0, and their alignment defines an adapter relevance score

$S$ 1

A softmax with sharpness parameter $S$ 2 converts these scores into instance-adaptive weights,

$S$ 3

from which SECA forms an aggregated teacher representation

$S$ 4

The aggregation space is trained with

$S$ 5

and the actual transfer objective is a semantically guided distillation loss,

$S$ 6

The stop-gradient operator ensures that the aggregated distribution acts as a teacher rather than co-adapting with the student feature $S$ 7 (He et al., 3 Aug 2025).

This mechanism makes semantic relevance an explicit control variable for stability–plasticity trade-offs. A plausible implication is that SG-AKT can be interpreted as a semantics-conditioned mixture-of-experts over historical visual states, except that the “experts” are frozen adapters rather than full models. That interpretation is consistent with the observed comparison against no distillation, frozen-CLIP distillation, last-model distillation, and simple feature averaging, where semantic-aware aggregation is reported to be superior (He et al., 3 Aug 2025).

The same principle appears in other domains with different implementations. In CCDA, teacher supervision is constrained by the hierarchical mapping $S$ 8 between coarse and fine labels, so that only semantically valid transfer paths are allowed (Shenaj et al., 2022). In source-free semantic segmentation with internal representations, class-conditional GMMs serve as semantics-aware latent anchors rather than using an unconditional global feature target (Stan et al., 2024). In Conquer, semantic retrieval determines which skill adapter is reused for a new task, replacing indiscriminate transfer by semantics-conditioned reuse (Wang et al., 6 Jun 2026). These are not identical algorithms, but they reflect the same design idea: continual transfer should be gated by semantic structure.

SE-VPR addresses a second problem in CLIP-based continual learning: a text-only classifier retains strong generalization but limited plasticity, while a visual classifier can adapt more flexibly but lacks the rich semantic structure encoded in text embeddings (He et al., 3 Aug 2025). SECA therefore refines visual prototypes using inter-class semantic relations extracted from the text space.

For each class $S$ 9, SECA defines a coarse visual prototype

$\mathcal{Y}^{1:S}$ 0

using the frozen CLIP visual encoder (He et al., 3 Aug 2025). It then obtains a prompt-conditioned text embedding

$\mathcal{Y}^{1:S}$ 1

projects it with $\mathcal{Y}^{1:S}$ 2,

$\mathcal{Y}^{1:S}$ 3

and builds a semantic affinity matrix

$\mathcal{Y}^{1:S}$ 4

The refined prototype for class $\mathcal{Y}^{1:S}$ 5 is then

$\mathcal{Y}^{1:S}$ 6

In effect, each visual prototype is semantically smoothed by the coarse prototypes of related classes (He et al., 3 Aug 2025).

The resulting visual classifier is

$\mathcal{Y}^{1:S}$ 7

with loss

$\mathcal{Y}^{1:S}$ 8

and an old-class prototype consistency regularizer

$\mathcal{Y}^{1:S}$ 9

These terms stabilize old visual prototypes while allowing semantically structured refinement (He et al., 3 Aug 2025).

This module is notable because it uses text not merely as a separate classifier but as a geometry prior over the visual prototype space. A plausible implication is that SE-VPR functions as a graph-based smoothing operator where the graph is induced by textual relations rather than empirical co-occurrence. That perspective aligns SECA with other semantically structured continual methods. In CCDA, the hierarchy of labels defines allowable redistribution of classifier mass from coarse classes to children through both knowledge distillation and weight initialization (Shenaj et al., 2022). In robotic semantic mapping, voxel-wise posteriors and map-consistency labels encode semantic structure that is projected back into model training, though not in prototype form (Frey et al., 2021, Blum et al., 2021).

5. Objective, training dynamics, and empirical performance

The full SECA objective for task $\mathcal{F}_\mathrm{V}(\cdot)$ 0 is

$\mathcal{F}_\mathrm{V}(\cdot)$ 1

The coefficient $\mathcal{F}_\mathrm{V}(\cdot)$ 2 is gradually increased with task index, reflecting the larger burden of preserving historical knowledge as the task stream lengthens (He et al., 3 Aug 2025). After each task, the current adapters are copied into the adapter pool, utility scores are updated with momentum using the relevance scores $\mathcal{F}_\mathrm{V}(\cdot)$ 3, and pruning keeps the pool at fixed capacity (He et al., 3 Aug 2025).

At inference time, SECA uses a hybrid classifier that combines the refined visual prototype classifier with the average of all task-specific text classifiers: $\mathcal{F}_\mathrm{V}(\cdot)$ 4 Core SECA is replay-free, while SECA++ adds Gaussian feature replay to strengthen the multi-modal classifier without storing real images (He et al., 3 Aug 2025).

The reported evaluation covers ImageNetR, ImageNetA, and CIFAR-100 in both 10-split and 20-split settings, using CLIP ViT-B/16 and the standard “Last” and “Avg” metrics (He et al., 3 Aug 2025). On 10-split ImageNetR and ImageNetA, replay-free SECA achieves Last/Avg of 83.18/88.58 and 65.09/74.45 respectively, while SECA++ further improves to 83.41/88.75 and 65.77/74.65 (He et al., 3 Aug 2025). On 10-split CIFAR-100, SECA achieves 79.79/86.70 and SECA++ reaches 81.59/87.80 (He et al., 3 Aug 2025). The paper reports that replay-free SECA surpasses strong replay-based baselines such as VPT-NSP on 10-split ImageNetR and ImageNetA, and that SECA++ is SOTA or near-SOTA across the evaluated settings (He et al., 3 Aug 2025).

A concise summary of representative numbers is useful.

Benchmark	Variant	Last / Avg
10S-ImageNetR	SECA	83.18 / 88.58
10S-ImageNetA	SECA	65.09 / 74.45
10S-CIFAR100	SECA	79.79 / 86.70
10S-ImageNetR	SECA++	83.41 / 88.75
10S-ImageNetA	SECA++	65.77 / 74.65
10S-CIFAR100	SECA++	81.59 / 87.80

Ablations reported in the same work indicate that SE-VPR contributes larger gains than SG-AKT in several settings, but the combination performs best, suggesting that semantic-aware transfer and semantic-aware classifier refinement are complementary rather than redundant (He et al., 3 Aug 2025).

Outside the CLIP-CIL formulation, the idea of semantic-enriched continual adaptation appears in several technically distinct forms.

In semantic segmentation under label refinement, CCDA introduces a multi-stage setting in which classes evolve from coarse to fine under domain shift. It combines supervised source learning, maximum-squares UDA on target data, coarse-to-fine knowledge distillation on target images, and coarse-to-fine weight initialization. The hierarchy $\mathcal{F}_\mathrm{V}(\cdot)$ 5 explicitly encodes which fine labels inherit from which coarse labels, and the distillation constraint enforces that the sum of child probabilities match the parent class probability (Shenaj et al., 2022). This is a direct form of semantic enrichment because the label ontology, rather than a flat class list, determines how knowledge is preserved and transferred.

In semantic segmentation under changing environments, ACE adapts to a sequence of unlabeled target domains by stylizing source images to match target feature statistics via AdaIN, training a generator/decoder, and replaying compact style statistics from prior domains to prevent forgetting (Wu et al., 2019). The paper stores 1024-dimensional style vectors per environment and uses them to regenerate source content in past styles for replay. Its memory is style-centric rather than class-centric, so it does not fully qualify as explicit semantic memory, but the adaptation target is semantic segmentation and the distillation acts on semantic maps (Wu et al., 2019). This suggests a weaker form of semantic enrichment in which semantics remain implicit in the prediction task rather than explicitly structured in memory.

A more explicit memory of semantic structure appears in source-free online domain adaptation for segmentation using internal representations. There, decoder features are clustered by class and modeled with class-conditional GMMs,

$\mathcal{F}_\mathrm{V}(\cdot)$ 6

and target latent features are aligned to this semantic source surrogate via sliced Wasserstein distance while a classifier is regularized on sampled GMM latent features (Stan et al., 2024). This is a source-free, online, semantics-enriched adaptation strategy because memory is ոչ raw data but a class-aware latent model.

In robotics, continual semantic adaptation is frequently grounded in spatial structure. “Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations” accumulates 2D predictions into a 3D semantic voxel map, renders pseudo-labels back into 2D via ray tracing, and adapts a Fast-SCNN model with experience replay from the pre-training dataset. Reported results on ScanNet show average adaptation accuracy rising from 51.5% to 56.6% and mean mIoU from 23.2% to 26.6%, while replay preserves source-domain performance far better than naive fine-tuning (Frey et al., 2021). “Self-Improving Semantic Perception for Indoor Localisation” similarly uses map consistency to generate binary foreground–background pseudo-labels and reports average improvements of 60% in segmentation and 10% in localisation accuracy relative to a fixed model, with replay identified as the most effective mechanism against catastrophic forgetting under noisy pseudo-labels (Blum et al., 2021). In both cases, semantics are task-aligned and operational rather than merely descriptive: the learned labels directly affect mapping or localization performance.

A different but relevant branch is continual test-time adaptation under contextual and semantic domain shifts. The evaluation study comparing BN, TENT, and CoTTA finds that test-time adaptation performs better and forgets less on contextual shifts than on semantic shifts, that TENT is stronger in short-term adaptation, and that CoTTA is better in longer runs, while BN remains the most robust baseline (Kerssies et al., 2022). This study does not propose a semantic-enriched algorithm, but it clarifies why such methods may be needed: purely statistical adaptation struggles when the shift is semantic rather than merely contextual (Kerssies et al., 2022).

Finally, Conquer extends the SECA idea beyond perception to multi-robot coordination. It formulates continual learning over a task stream of Dec-POMDPs, represents each skill by LoRA adapters and a lightweight head on a frozen Self-Allies-Goal backbone, retrieves initializations using semantic descriptors in a 1024-dimensional text embedding space, and updates a skill library by merging or inserting skills based on semantic distance (Wang et al., 6 Jun 2026). On its benchmark, Conquer reports a final average success rate of 95.6%, forward transfer of 11.3%, and backward transfer of 0.0%, illustrating how semantic retrieval plus parameter isolation can support continual adaptation without catastrophic forgetting (Wang et al., 6 Jun 2026). This is not the same problem class as CLIP-CIL or continual segmentation, but it demonstrates the generality of semantically organized continual adaptation.

7. Conceptual significance, limitations, and misconceptions

The main conceptual significance of SECA is that it replaces uniform transfer with semantics-conditioned transfer. In the CLIP-based formulation, textual priors are anti-forgetting signals and structural priors at once: they decide which old adapters matter for a new sample, and they shape the geometry of visual prototypes (He et al., 3 Aug 2025). In related segmentation and robotics work, semantic hierarchies, class-conditional latent models, 3D semantic maps, and semantic skill descriptors play analogous roles (Shenaj et al., 2022, Stan et al., 2024, Frey et al., 2021, Wang et al., 6 Jun 2026).

A common misconception is that “semantic” in continual adaptation simply means that the downstream task is semantic segmentation or classification. The literature indicates a narrower and more technical meaning. A method is semantically enriched when semantic structure is encoded in the adaptation mechanism itself: for example, through text embeddings, class hierarchies, class-conditional density models, task descriptors, or semantically organized memory. ACE, for instance, is semantic in its output task but stores only style statistics; the paper explicitly notes that it lacks explicit class-wise semantic memory (Wu et al., 2019). By contrast, CCDA and CLIP-driven SECA use semantic relations in losses and parameter initialization directly (Shenaj et al., 2022, He et al., 3 Aug 2025).

A second misconception is that semantic enrichment automatically eliminates forgetting. The evidence is more qualified. In CLIP-driven SECA, performance gains are strong, but they come with added adapter-pool inference cost during training and quadratic prototype-affinity computation over seen classes (He et al., 3 Aug 2025). In continual test-time adaptation, semantic shifts remain harder than contextual shifts even for advanced online methods (Kerssies et al., 2022). In robotics, replay remains necessary to stabilize learning under noisy pseudo-labels despite the presence of semantically enriched signals (Frey et al., 2021, Blum et al., 2021). This suggests that semantics improves the inductive bias for transfer and retention but does not replace the need for stability mechanisms such as replay, parameter isolation, or conservative update rules.

Several limitations recur across the literature. CLIP-driven SECA depends on meaningful class names and a text encoder whose geometry reflects useful inter-class relations; domains with weak textual semantics may reduce its effectiveness (He et al., 3 Aug 2025). CCDA assumes a predefined hierarchy $\mathcal{F}_\mathrm{V}(\cdot)$ 7, which requires expert design and may not generalize automatically across datasets (Shenaj et al., 2022). Internal-representation methods using class-conditional GMMs depend on the quality of latent clustering and do not naturally model long target-domain sequences unless memory is expanded (Stan et al., 2024). Robotic 2D–3D adaptation methods depend on static geometry and reliable depth or pose estimation, and semantic maps can reinforce consistent model mistakes if the initial network is systematically wrong (Frey et al., 2021). Conquer notes that semantic closeness is not perfectly aligned with physical transferability, so retrieval in embedding space is only a heuristic proxy for skill reuse value (Wang et al., 6 Jun 2026).

Taken together, these works suggest a plausible general definition: Semantic-Enriched Continual Adaptation is a family of continual or online learning methods in which semantic structure is embedded into memory, transfer, routing, or classifier geometry so that adaptation over time is guided by meaning rather than by raw parameter proximity or distributional alignment alone. In the strictest and most fully specified sense, however, SECA denotes the CLIP-based framework with SG-AKT and SE-VPR introduced in 2025 (He et al., 3 Aug 2025).