Semantic-Enhanced Visual Prototype Refinement

Updated 7 July 2026

The paper introduces a two-stage pipeline where coarse visual prototypes are refined using semantic information to enhance classification and retrieval tasks.
It leverages inter-class semantic relations from textual embeddings and channel-wise gating to reduce noise and enforce semantically structured representations.
Empirical results across continual, few-shot, generative, and place recognition tasks demonstrate improved prototype stability, compactness, and inter-class separation.

Searching arXiv for the primary and closely related papers on SE-VPR and adjacent prototype-refinement methods. arXiv search query: (He et al., 3 Aug 2025) Semantic-Enriched Continual Adaptation SE-VPR Semantic-Enhanced Visual Prototype Refinement (SE-VPR) denotes the refinement of visually derived prototypes by semantic information before classification, retrieval, or matching. In its explicit formulation within Semantic-Enriched Continual Adaptation, SE-VPR refines class prototypes using inter-class semantic relations captured in class-wise textual embeddings (He et al., 3 Aug 2025). Closely related mechanisms appear in few-shot learning, generative zero-shot learning, few-shot action recognition, scene graph generation, and visual place recognition, where semantics act as channel-wise gates, updated semantic conditions, multimodal anchors, prototype regularizers, or semantic layout constraints (Yang et al., 2021, Hou et al., 2024, Ni et al., 2022, Zheng et al., 2023, Peng et al., 2021, Jiwei et al., 2022). This suggests that SE-VPR is both a named module and a broader prototype-refinement pattern.

1. Definition and scope

The common substrate of SE-VPR is a two-stage representation pipeline. First, a prototype is obtained from visual evidence: support images in few-shot learning, averaged class features in continual learning, relation embeddings in scene graph generation, or global place descriptors in visual place recognition. Second, that prototype is refined by semantics so that nuisance variation is suppressed and semantically relevant structure is preserved.

Work	Prototype being refined	Semantic mechanism
SECA	Class-wise visual prototypes $\mathbf c_{V,k}$	Inter-class affinity from CLIP text embeddings (He et al., 3 Aug 2025)
SEGA	Few-shot visual prototypes $\mathbf p^c$	Semantic-to-attention MLP and Hadamard gating (Yang et al., 2021)
VADS	Dynamic semantic prototype for generation	VDKL visual prior and VOSU semantic updation (Hou et al., 2024)
MORN	TRX visual prototypes $P^V$	SE text module and multimodal fusion (Ni et al., 2022)
PE-Net	Predicate prototypes $\mathbf c_c$	GloVe prototypes, PL, and PR (Zheng et al., 2023)
SRALNet / SSR-VLAD	Place descriptors	Semantic-constrained centroids or semantic skeleton aggregation (Peng et al., 2021, Jiwei et al., 2022)

The explicit term SE-VPR is used in SECA, where the objective is to bridge the modality gap between CLIP text-based classifiers and visual-side classifiers in rehearsal-free class-incremental learning (He et al., 3 Aug 2025). In the other works, the term functions as an analytical lens: semantic knowledge is used to reshape a prototype-like visual representation rather than merely appended as auxiliary metadata.

A canonical SE-VPR pipeline begins with a coarse visual prototype. In SECA, for class $k$ up to task $s$ , the coarse prototype is the mean visual feature

$\mathbf c_{V,k}=\frac{1}{N_k}\sum_{(\mathbf x,y)\in\mathcal D^{1:s}}\mathcal F_V(\mathbf x)\,\mathds 1(y=k).$

Class-wise textual embeddings $\mathbf Z_k=\mathcal F_T(k;\mathbf P^s)$ are projected and compared through a Gaussian kernel to produce a semantic affinity matrix

$\mathbf M_{k,j}=\exp\bigl(-\gamma \,\Vert \phi(\mathbf Z_k)\mathbf H_{\text{proj}}-\phi(\mathbf Z_j)\mathbf H_{\text{proj}}\Vert_2^2\bigr),$

and refinement is then performed by semantic smoothing,

$\hat{\mathbf c}_{V,k}=\sum_{j\in\mathcal Y^{1:s}}\frac{\mathbf M_{k,j}}{\sum_{i\in\mathcal Y^{1:s}}\mathbf M_{k,i}}\,\mathbf c_{V,j}.$

The refined prototypes define a cosine-based visual classifier (He et al., 3 Aug 2025).

Other instantiations realize the same step with different operators. SEGA computes a visually derived few-shot prototype $\mathbf p^c$ 0 and maps a class semantic vector $\mathbf p^c$ 1 to a channel-wise attention vector $\mathbf p^c$ 2, yielding the refined prototype

$\mathbf p^c$ 3

so semantics act as feature selection rather than prototype generation (Yang et al., 2021). MORN constructs multimodal prototypes by fusing query-specific visual prototypes and text prototypes,

$\mathbf p^c$ 4

with the text side produced by a semantic-enhanced module over frozen CLIP text embeddings (Ni et al., 2022). VADS forms a dynamic semantic prototype for a generator by concatenating visual prior noise and updated semantics,

$\mathbf p^c$ 5

where $\mathbf p^c$ 6 is built from local bias, global prior, and Gaussian noise, and $\mathbf p^c$ 7 is produced by semantic updation (Hou et al., 2024).

These operators differ algebraically, but they share a common role: semantics modify the geometry of the visual prototype space before the downstream similarity computation. A plausible implication is that SE-VPR is less a single architecture than a family of refinement operators constrained by semantic structure.

3. SE-VPR in continual learning: the SECA formulation

SECA studies rehearsal-free class-incremental learning with CLIP ViT-B/16, lightweight visual adapters, and task-specific soft prompts on the frozen text encoder (He et al., 3 Aug 2025). Its baseline hybrid classifier combines text-based classification with a visual-side classifier; SE-VPR is the component that strengthens the visual classifier by enforcing that visual prototypes inherit inter-class structure from text space.

Beyond the prototype smoothing equations, SECA adds two optimization terms: a visual cross-entropy

$\mathbf p^c$ 8

and a prototype consistency regularizer for old classes,

$\mathbf p^c$ 9

The regularizer is designed to prevent drift of refined prototypes across tasks, while the visual classifier provides plasticity that a text-only classifier lacks.

Setting	H-PEFT	H-PEFT + SE-VPR	SECA
10S-ImageNetA (Last)	55.78%	62.62%	65.09%
10S-CIFAR100 (Last)	73.97%	77.13%	79.79%
10S-ImageNetR (Last)	80.57%	81.30%	83.18%

Classifier ablations further isolate the effect of semantic refinement. On 10S-ImageNetA (Last), Only Text gives 57.91%, Centroid (CLIP) 58.55%, Centroid (Adapted) 63.16%, Linear 51.07%, and SE-VPR 65.09%. On 10S-CIFAR100 (Last), the corresponding values are 75.93%, 75.77%, 78.30%, 74.20%, and 79.79% (He et al., 3 Aug 2025). The pattern is consistent: adapted visual prototypes already help, but semantically refined prototypes help more.

The conceptual significance of SECA is precise. Textual priors are treated as stable class-relationship structure rather than as final classifier weights only. Visual prototypes remain in the visual branch, but their neighborhood relations are inherited from text space. This directly targets the stability-plasticity dilemma and the CLIP modality gap.

4. Few-shot and multimodal realizations

SEGA is an almost direct realization of SE-VPR for few-shot learning (Yang et al., 2021). Its pipeline first forms a visual prototype

$P^V$ 0

where $P^V$ 1 is the class mean and $P^V$ 2 transfers visual prior from base-class weights. A semantic vector from GloVe, processed via WordNet hypernyms, is mapped by an MLP into a channel-wise gate $P^V$ 3, and refinement is then

$P^V$ 4

On miniImageNet, the Dynamic-FSL baseline under the same ResNet-12 setup gives 62.81% in 1-shot and 78.97% in 5-shot, while SEGA reaches 69.04% and 79.03%. On tieredImageNet, the corresponding numbers are 68.55% and 83.95% for the baseline versus 72.18% and 84.28% for SEGA. On CIFAR-FS, 67.78% becomes 76.24% in 1-shot, and fake semantics drop 3–5% below the no-semantics baseline. A common misconception is that semantics in this model generate prototypes; they do not. The paper states that semantic information acts as a feature selector on the prototype (Yang et al., 2021).

MORN extends the same logic to few-shot action recognition with a multimodal prototype-enhanced network (Ni et al., 2022). Visual prototypes are produced by TRX, label texts are encoded by a frozen CLIP text encoder, and a semantic-enhanced module based on multi-head attention refines the text embeddings before inflation to the visual-prototype shape. The default multimodal prototype is

$P^V$ 5

MORN also defines PRIDE,

$P^V$ 6

to quantify prototype compactness and separability. The TRX baseline improves from 75.6 to 86.3 on HMDB51, from 96.1 to 96.9 on UCF101, from 85.9 to 91.6 on Kinetics, and from 64.6 to 71.1 on SSv2 when moving to TRX $P^V$ 7. On SSv2, PRIDE rises from 10.4 for TRX $P^V$ 8 to 47.3 for TRX $P^V$ 9 (Ni et al., 2022). This indicates that prototype refinement can be evaluated not only by accuracy but also by direct prototype-quality diagnostics.

5. Generative and relational extensions

VADS places SE-VPR inside generative zero-shot learning by replacing the static conditioning pair $\mathbf c_c$ 0 with a visual-augmented dynamic semantic prototype (Hou et al., 2024). The Visual-aware Domain Knowledge Learning module produces a visual prior

$\mathbf c_c$ 1

while the Vision-Oriented Semantic Updation module maps a predefined semantic prototype $\mathbf c_c$ 2 to an updated semantic vector

$\mathbf c_c$ 3

The generator is then conditioned on

$\mathbf c_c$ 4

The method reports superior CZSL and GZSL performances on SUN, CUB, and AWA2, with averaging increases by 6.4\%, 5.9\% and 4.2\% on SUN, CUB and AWA2, respectively. As a plug-in to CLSWGAN, AWA2 improves from Acc/H 71.3/69.3 to 82.5/79.3, SUN from 66.0/44.6 to 76.3/55.7, and CUB from 80.1/65.2 to 86.8/74.3. The same paper also shows that a wrong-domain global prior hurts: “Other domain global prior (SUN → CUB)” gives 84.0 / 69.3 versus 86.8 / 74.3 for the matched VDKL prior (Hou et al., 2024).

PE-Net addresses scene graph generation by treating predicate label semantics as class-wise prototypes in a common embedding space (Zheng et al., 2023). Predicate prototypes are defined from GloVe embeddings as

$\mathbf c_c$ 5

and relation prediction is reformulated as matching a relation embedding $\mathbf c_c$ 6 to these prototypes. Prototype-guided Learning combines a cosine-softmax loss and a margin-based Euclidean triplet loss, while Prototype Regularization discourages semantic overlap among prototypes. Relative to VCTree, PE-Net with PL only raises PredCls mR@100 from 17.9 to 25.4, SGCls mR@100 from 8.3 to 14.8, and SGDet mR@100 from 7.3 to 11.0; the full PE-Net further raises PredCls mR@100 to 33.8 (Zheng et al., 2023). Here refinement is not an explicit smoothing step over class means, but the same principle holds: visually diverse relation instances are pulled toward semantically anchored prototypes, and the prototype manifold itself is regularized.

6. Place recognition variants

In visual place recognition, the refined object is often a place descriptor rather than a class centroid. SSR-VLAD builds such a descriptor from semantic skeleton representation (Jiwei et al., 2022). Each semantic category yields local Shape Context descriptors over skeleton keypoints, and the category descriptor is

$\mathbf c_c$ 7

Descriptors are concatenated across categories and, on the reference side, temporally aggregated as

$\mathbf c_c$ 8

with $\mathbf c_c$ 9. On CMU Season, SSR-VLAD reaches Recall@100%Precision 51.8 and AUC 0.961, while “SSR-VLAD (no preprocess)” drops to 6.9 and 0.696. On SYNTHIA, with perfect segmentation, SSR-VLAD reaches AUC 0.999 (Jiwei et al., 2022). This suggests a place-level SE-VPR interpretation in which semantics refine a global prototype by stabilizing its geometry under appearance variation.

SRALNet uses a more explicit attention-based refinement of VLAD-style place prototypes (Peng et al., 2021). Local features are softly assigned to representative centroids $k$ 0 and then reweighted by an interpretable intra-cluster saliency

$k$ 1

producing weighted residual aggregation

$k$ 2

Semantic-constrained initialization uses DeepLabV3 predictions: representative centroids are initialized from static classes such as building, road, traffic signs, and vegetation, while shadow centroids are initialized from sky, person, and vehicle. On Pitts250k-test, SRALNet-SC reaches Recall@1 85.8, and on Tokyo24/7 it reaches 68.6 at original dimensionality and 72.1 after PCA-whitening to 4096-D (Peng et al., 2021). The evidence also shows that semantic priors alone are not sufficient; the gain comes from combining semantic-constrained initialization with triplet-based fine-tuning.

7. Empirical regularities, misconceptions, and future directions

A consistent empirical regularity is that semantics help when they are structurally aligned with the visual task, and can hurt when they are not. In SEGA, fake semantics perform 3–5% below the no-semantics baseline (Yang et al., 2021). In VADS, a prior learned on another dataset degrades CUB performance (Hou et al., 2024). In SSR-VLAD and SRALNet, segmentation quality is a hard bottleneck: performance drops sharply without morphological preprocessing or under poor semantic maps (Jiwei et al., 2022, Peng et al., 2021). These results rule out the misconception that any semantic auxiliary signal is beneficial by default.

A second regularity is that many successful SE-VPR systems use semantics conservatively. SEGA performs channel-wise gating rather than semantic prototype generation (Yang et al., 2021). SECA smooths visual prototypes according to text-space affinities but still classifies with visual-side prototypes (He et al., 3 Aug 2025). In MORN, the default weighted average fusion is retained because heavier attention-based fusion is not uniformly best, especially on fine-grained SSv2 labels (Ni et al., 2022). This suggests that semantics often function best as structural priors over visual geometry rather than as full replacements for visual evidence.

Current future directions are already visible in the cited literature. SEGA explicitly proposes richer semantics, iterative refinement, and spatial plus channel-wise attention (Yang et al., 2021). VADS points to end-to-end image-level optimization and stronger semantic sources (Hou et al., 2024). A distinct but relevant development is training-free semantic prototype optimization by iterative definition refinement, where textual definitions themselves are treated as prototypes and improved through example-guided, confusion-aware, and history-aware LLM loops; all 13 embedding models improve on both B2MWT-10C and N24News under this strategy (Rehmat et al., 30 Apr 2026). A plausible implication is that future SE-VPR systems may jointly optimize visual prototypes and the semantic descriptions that anchor them, while keeping backbone encoders fixed.

Across these lines of work, SE-VPR has come to denote a specific technical proposition: visually estimated prototypes are often too noisy, too unstable, or too weakly separated to serve as final decision anchors, and semantic structure can be used to regularize, gate, smooth, update, or reinterpret them. The exact operator varies by task, but the recurrent objective remains prototype compactness, inter-class separation, and semantically meaningful geometry.