Semantic-Guided Adaptive Knowledge Transfer

• Semantic-Guided Adaptive Knowledge Transfer is an architectural paradigm that employs structured semantic signals to guide adaptive reuse of prior knowledge under distribution shift and data sparsity.
• It leverages mechanisms like semantically-guided representation and knowledge distillation to compute instance-dependent transfer policies, reducing negative transfer and enhancing stability.
• SG-AKT extends beyond continual learning to HDR reconstruction and recommendation by using semantic priors and adaptive attention to refine outputs and improve performance metrics.

Searching arXiv for papers on “Semantic-Guided Adaptive Knowledge Transfer” and closely related terminology to ground the article in current literature. Semantic-Guided Adaptive Knowledge Transfer (SG-AKT) designates a class of transfer mechanisms in which semantic structure is used to determine what prior knowledge should be reused, how strongly it should be transferred, and which historical signals should be suppressed. In the literature, the term is used explicitly in CLIP-based continual learning, where textual label semantics guide both current-task representation learning and cross-task distillation (Yu et al., 2024), and in SECA, where class-wise textual cues score the relevance of historical visual adapters and aggregate them into an instance-conditioned teacher (He et al., 3 Aug 2025). The same designation is also used in HDR reconstruction, where semantic priors extracted in the SDR domain are transferred back into an HDR model through self-distillation and feature alignment (Yan et al., 19 Mar 2025). Related systems in cross-disciplinary and cross-course knowledge tracing, as well as long-tail recommendation, do not always use the exact name SG-AKT, but they instantiate closely related principles: semantic group structure, semantic IDs, or semantic concept relations are used as routing priors when direct overlap or dense supervision is limited (Deng et al., 25 Nov 2025, Han et al., 14 May 2025, Yan et al., 22 May 2026).

1. Core idea and recurring design pattern

Across domains, SG-AKT is characterized by three recurrent ingredients. First, it introduces a semantic prior that is external to the immediate prediction signal. Second, it converts that prior into a transfer policy, typically by weighting, routing, or softening supervision. Third, it uses the resulting semantic bias to improve the plasticity–stability tradeoff, reduce negative transfer, or compensate for missing evidence.

A compact view of the main variants is given below.

Setting	Semantic signal	Adaptive transfer mechanism
Class-incremental continual learning	CLIP text embeddings of class labels	SG-RL soft labels and SG-KD semantic distillation
Rehearsal-free continual learning in SECA	Task-prompted class-wise textual embeddings	Relevance-weighted aggregation over historical adapters
HDR reconstruction	FastSAM semantic priors in SDR space	Self-distillation plus feature alignment through SKAM
Cross-disciplinary knowledge tracing	Clustered student knowledge-state categories	Category-enhanced Mixture-of-Experts and adversarial alignment
Cross-course knowledge tracing	LLM-predicted concept relations and LLM summaries	Graph propagation plus contrastive alignment
Long-tail recommendation	MLLM-aligned semantic IDs via RQ-VAE	Head-to-tail asymmetric transfer with activity-aware gating

The semantic signal itself varies substantially. In CLIP-based continual learning, it is derived from normalized text embeddings of class names (Yu et al., 2024). In SECA, it is a collection of task-specific textual embeddings produced from multiple prompts and used as a semantic reference over historical adapters (He et al., 3 Aug 2025). In HDR reconstruction, it is a multi-scale semantic prior extracted by FastSAM before the representation head and fused through an FPN (Yan et al., 19 Mar 2025). In educational transfer settings, semantics may instead be induced from clustered learner states or LLM-discovered concept relations (Deng et al., 25 Nov 2025, Han et al., 14 May 2025).

This suggests that SG-AKT is best understood not as a single canonical algorithm, but as an architectural principle: semantic structure acts as a control signal for transfer under distribution shift, task evolution, or data sparsity.

2. CLIP-based SG-AKT in class-incremental continual learning

In class-incremental continual learning, SG-AKT is introduced to address catastrophic forgetting by using the semantic information encoded in class labels rather than relying only on visual features (Yu et al., 2024). The method starts from CLIP, where the image encoder produces normalized image features $I$, the text encoder produces normalized text features $T$, and the image–text logit is

$P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$

The method fine-tunes the last block of the CLIP vision encoder, which is reported to give the best adaptation (Yu et al., 2024).

The framework has two modules: Semantically-guided Representation Learning (SG-RL) and Semantically-guided Knowledge Distillation (SG-KD). SG-RL operates within the current task. For task $t$ with label set $C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$, the text encoder yields normalized embeddings $\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$, and the intra-task similarity matrix is

$$S_{i,j}^{t \leftrightarrow t} = [T_i^\top T_j]_{K \times K}.$$

Instead of one-hot supervision, the method forms semantically-guided soft labels

$$C_{i,j}^{sg} = \frac{\exp(\alpha \cdot S_{i,j}^{t \leftrightarrow t})} {\sum_{k=1}^{K}\exp(\alpha \cdot S_{i,k}^{t \leftrightarrow t})},$$

and optimizes

$$\mathcal{L}_{SG-RL} = D_{KL}(\hat{y} \,\|\, C^{sg}).$$

This replaces rigid label assignment with a semantic distribution over current-task classes (Yu et al., 2024).

SG-KD operates across tasks. It computes inter-task semantic similarity between current labels and old labels through text embeddings: $$S_{i,j}^{t-1 \leftrightarrow t} = T_i \cdot O_j^\top.$$ From this, it defines a semantic transfer prior

$T$0

and combines ordinary distillation with semantic alignment: $T$1 The total objective is

$T$2

The adaptive element lies in the fact that the transfer structure is recomputed whenever new label sets arrive. Intra-task semantics define the current soft labels, while inter-task semantics redefine which previous classes are most relevant to each new class. On CIFAR100 in the 10-step split, the method reports 80.1 last accuracy and an improvement over the previous best by 11.4 points; it also reports top-1 last-accuracy gains of 7.9 points on imagenet_subset and 6.5 points on ImageNet1000, with 3.1 points improvement in top-5 accuracy on ImageNet1000 (Yu et al., 2024).

3. Instance-adaptive SG-AKT in SECA

A later formulation appears in Semantic-Enriched Continual Adaptation (SECA), where SG-AKT is the backbone-side selective distillation mechanism for rehearsal-free class-incremental learning (He et al., 3 Aug 2025). The problem formulation is more selective than conventional continual distillation: the objective is not to preserve all prior knowledge, but to retain only historical visual knowledge that is semantically relevant to the current sample.

For a new sample $T$3, the method constructs two sets of features. First, it passes the image through each historical adapter in the cached pool

$T$4

producing

$T$5

Second, it extracts task-specific textual embeddings for the class label from all task prompts: $T$6

These two modalities are projected into a shared space using learnable projectors $T$7 and $T$8, after LayerNorm $T$9. The relevance score for adapter $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$0 is

$P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$1

These scores are normalized into a soft attention distribution over the adapter pool: $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$2 The aggregated representation is then treated as a teacher distribution for distillation through $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$3, while an auxiliary alignment loss $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$4 supervises the semantic projectors (He et al., 3 Aug 2025).

This formulation differs from ordinary knowledge distillation in two ways. The teacher is not a single past model, and old knowledge is not averaged uniformly. Instead, SG-AKT constructs an instance-dependent teacher by weighting each historical adapter according to semantic compatibility with the current class label. The paper explicitly contrasts this with Seq., CLIP-KD, Vanilla, and Avg-KD, and positions SG-AKT as a defense against semantic interference from unrelated tasks (He et al., 3 Aug 2025).

The module also includes fixed-capacity adapter-pool management. Each adapter receives a momentum-updated utility score,

$P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$5

and the highest-utility old adapter is removed when a new adapter must be stored. Performance is reported to improve as the pool size grows and to saturate around $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$6 (He et al., 3 Aug 2025).

SECA combines SG-AKT with Semantic-Enhanced Visual Prototype Refinement (SE-VPR). In ablations, SG-AKT alone improves the hybrid PEFT baseline on 10S-ImageNetA from $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$7 to $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$8, on 10S-CIFAR100 from $P_{i,j} = \beta \cdot I_i \cdot T_j^\top.$9 to $t$0, and on 10S-ImageNetR from $t$1 to $t$2. With SE-VPR, the full SECA system reaches $t$3, $t$4, and $t$5 on those benchmarks, respectively (He et al., 3 Aug 2025).

4. Semantic-guided adaptive transfer beyond continual learning

Semantic-guided adaptive transfer has also been instantiated in cross-disciplinary and cross-course knowledge tracing, although the papers use task-specific nomenclature rather than the exact label SG-AKT. In Adaptive Knowledge Transfer for Cross-Disciplinary Cold-start Knowledge Tracing, the problem is CDCKT: predicting student performance in a target discipline when target-discipline interaction data are extremely sparse (Deng et al., 25 Nov 2025). The framework, called ACKT, pretrains a source-discipline KT model, clusters student knowledge states into $t$6 categories with MiniBatchKMeans, and uses the Silhouette coefficient to choose the optimal $t$7. Each student is assigned a category

$t$8

These categories are then used as semantic guidance for a Category-enhanced Mixture of Experts (CMOE). For student $t$9, the fused representation is

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$0

the gate computes expert weights

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$1

and the mapped target representation is

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$2

A discriminator then imposes intra-cluster cohesion and inter-cluster separation on mapped features, using same-category and different-category pairs. The total loss is

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$3

with reported $C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$4. The method is evaluated on 20 extreme cross-disciplinary cold-start scenarios on PTADisc, and the paper reports that ACKT achieves the best results in all reported scenarios. Ablations show the largest drops for ACKT-moe and ACKT-gate, while ACKT-gan and ACKT-prefer also degrade (Deng et al., 25 Nov 2025).

TransKT formulates a related semantic-guided transfer problem for cross-course knowledge tracing (Han et al., 14 May 2025). It constructs a cross-course concept graph $C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$5 with explicit question–concept edges and implicit concept–concept edges discovered by zero-shot LLM prompts. The candidate relation types are Prerequisite_of, Used_for, Hyponym_of, and Part_of, and the appendix states that five independent zero-shot queries with majority vote are used for edge acceptance (Han et al., 14 May 2025). Node text is summarized by an LLM and encoded by a smaller LM,

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$6

after which GraphSAGE-style propagation updates node features: $C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$7 Single-course and cross-course knowledge states are then aligned through a mutual-information-style contrastive objective. The final representation mixes local and transferred information via

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$8

TransKT reports gains over the best baseline of +1.51% ACC and +3.17% AUC on Java–Python, C–DS, and CS–MA datasets (Han et al., 14 May 2025).

A plausible implication is that SG-AKT-like reasoning extends naturally to settings where semantic structure is not text-label similarity but latent student categories or LLM-mediated concept relations.

5. Semantic priors in HDR reconstruction and semantic IDs in recommendation

In HDR reconstruction, SG-AKT is used to improve baseline multi-exposure HDR models when LDR inputs are heavily degraded, underexposed, overexposed, or missing content (Yan et al., 19 Mar 2025). The framework is composed of an Original Reconstruction Model (ORM), a Semantic Priors Guided Reconstruction Model (SPGRM), and a Semantic Knowledge Transfer Scheme (SKTS). The baseline HDR output is first transformed by a differentiable domain transfer operator,

$C^t = \{c_1^t, c_2^t, \dots, c_K^t\}$9

and FastSAM serves as the frozen Semantic Knowledge Bank, producing segmentation maps and multi-scale semantic features. SPGRM then uses these priors to refine the SDR-like reconstruction.

Transfer back to the HDR model occurs through output-level and feature-level self-distillation. The content loss is

$\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$0

with $\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$1, and color alignment is imposed through a Semantic-Guided Histogram Loss applied within semantic regions. Feature transfer is performed by the Semantic Knowledge Alignment Module (SKAM), where student and teacher features are encoded, mixed by complementary masks, decoded, and aligned through

$\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$2

The ORM objective becomes

$\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$3

with $\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$4 and $\mathbf{T}^{t} = \{T_1, T_2, \dots, T_K\}$5 (Yan et al., 19 Mar 2025).

The paper reports improvements for several baselines, including AHDR, HDR-Trans, and TMRNet. On Tel’s dataset and Kalantari’s dataset, the reported average gains are about 0.54 dB and 0.91 dB PSNR, respectively; on Zhang’s BracketIRE and BracketIRE+ datasets, the reported average PSNR improvements are about 2.14 dB and 1.98 dB (Yan et al., 19 Mar 2025).

A related but differently named formulation appears in long-tail recommendation as AKT-Rec (Yan et al., 22 May 2026). That framework uses GME-Qwen2-VL-7B and Qwen3-30B-A3B to build multimodal semantic representations, discretizes them into semantic IDs via RQ-VAE with three codebooks of size 128/256, and performs head-to-tail transfer through Cluster-Guided Adaptive Embedding and Hierarchical Feature Aggregation. The adaptive transfer is asymmetric: head signals guide tail IDs while stop-gradient and activity-aware gates protect head representations from tail noise. The reported gains are +0.35% in AUC and +1.53% in GAUC offline, and +2.76% in CTR plus +3.47% in GMV in online A/B testing on Tmall (Yan et al., 22 May 2026). This is not labeled SG-AKT, but it shares the central logic of semantically structured, activity-aware transfer.

6. Empirical profile, limitations, and recurrent misconceptions

The empirical profile of SG-AKT is domain-dependent, but several patterns recur. First, semantics are most valuable when naive transfer is indiscriminate. In continual learning, SECA argues that unrelated historical knowledge creates semantic interference, and SG-AKT improves over Vanilla and Avg-KD by making transfer instance-adaptive (He et al., 3 Aug 2025). In class-incremental CLIP adaptation, semantically-guided soft labels outperform one-hot supervision, and SG-KD improves over naive KD by biasing old-class predictions toward semantically related old labels (Yu et al., 2024). In HDR reconstruction, the gains are largest when the problem is ill-posed because of degradation or missing content; the paper notes that improvements on Kalantari’s dataset are smaller because those scenes are mainly daytime and less degraded (Yan et al., 19 Mar 2025).

Second, SG-AKT does not imply a uniform mechanism. One common misconception is to treat it as synonymous with generic knowledge distillation. The literature does not support that reading. In one line of work, SG-AKT means semantic soft supervision and semantic priors over old classes (Yu et al., 2024). In another, it means relevance-weighted aggregation over a pool of historical adapters (He et al., 3 Aug 2025). In HDR, it means semantic self-distillation across a domain-transfer boundary plus masked feature reconstruction (Yan et al., 19 Mar 2025).

Third, semantic guidance does not eliminate all structural constraints. ACKT explicitly notes that it still depends on some overlapping learners and does not fully solve the no-overlap setting, even though its adversarial alignment leverages non-overlapping students through shared category structure (Deng et al., 25 Nov 2025). In SECA, textual priors improve transfer selectivity, but the framework still requires a companion module, SE-VPR, because text-based classifiers alone suffer from limited plasticity due to the modality gap in CLIP (He et al., 3 Aug 2025).

A final misconception is that semantic guidance necessarily means explicit natural-language semantics. The broader literature suggests a wider interpretation. In TransKT, the semantic guide is a graph of LLM-predicted concept relations plus LLM-summarized node text (Han et al., 14 May 2025). In ACKT, it is cluster-derived student categories that act as semantic groups of mastery, preference, or interaction patterns (Deng et al., 25 Nov 2025). In AKT-Rec, it is semantic IDs created by discretizing multimodal representations (Yan et al., 22 May 2026). This suggests that “semantic” in SG-AKT often denotes structured latent relatedness rather than only raw text embeddings.

Taken together, the literature presents SG-AKT as a transfer paradigm for settings where direct supervision is incomplete, old knowledge is heterogeneous, or the relevant transfer path is not obvious from low-level signals alone. Its defining move is to replace uniform transfer with semantically conditioned transfer.