Adaptive & Efficient Vision-Language Pseudo-Labeling

• The paper introduces AEVLP as a design pattern that leverages vision-language priors to adaptively generate, verify, and integrate pseudo-labels for tasks like object detection and adaptation.
• Adaptive and Efficient Vision-Language Pseudo-Labeling is a framework that combines dynamic reliability mechanisms with lightweight computational strategies to enhance pseudo-label quality.
• The framework demonstrates practical improvements across regimes such as test-time adaptation, unsupervised and semi-supervised learning, achieving notable performance gains on standard benchmarks.

Searching arXiv for the cited work and closely related papers to ground the article. Adaptive and Efficient Vision-Language Pseudo-Labeling (AEVLP) denotes a family of methods in which a vision-LLM, or vision-language priors derived from it, are used to generate, verify, rank, propagate, or reweight pseudo-labels under constraints of robustness and computational economy. In the cited literature, AEVLP appears as instance-level VLM verification for class incremental object detection, cache-based pseudo-label refinement for test-time adaptation, dual-prototype fusion for unsupervised adaptation, multi-focus pseudo-labeling plus robust loss for single-positive multi-label learning, graph-based label propagation for zero-/few-shot adaptation, detector-guided denoising for open-vocabulary detection, and text-informed Optimal Transport pseudo-labeling for semi-supervised few-shot adaptation (Kim et al., 2024, Karmanov et al., 2024, Ali et al., 2024, Tran et al., 28 Aug 2025, Li et al., 2024, Wang et al., 2024, Silva-Rodríguez et al., 3 Mar 2026). This suggests that AEVLP is best understood as a design pattern rather than a single canonical algorithm.

1. Conceptual scope and defining properties

Across the literature, the term “adaptive” refers to pseudo-labeling mechanisms that respond to instance context, uncertainty, neighborhood structure, class imbalance, or training dynamics. Examples include instance-level, custom prompts per pseudo ground-truth in VLM-PL; entropy-prioritized positive and negative caches in TDA; convex fusion of textual and image prototypes plus confidence reweighting in DPA; pseudo-label refresh “every epoch and across multiple local focuses of the image” in the SPML AEVLP framework; dynamic graph expansion with context-aware feature re-weighting in ECALP; detector-guided online mining in MarvelOVD; and class-balanced OT assignments with adaptive class-specific regularization in SS-Text-U (Kim et al., 2024, Karmanov et al., 2024, Ali et al., 2024, Tran et al., 28 Aug 2025, Li et al., 2024, Wang et al., 2024, Silva-Rodríguez et al., 3 Mar 2026).

The term “efficient” is used more specifically. In VLM-PL, verification is performed with a pre-trained VLM “without training an auxiliary detector or classifier” and “no extra detector heads, teacher models, or replay buffers are required” (Kim et al., 2024). TDA is “training-free” and “entirely avoids backpropagation at test time,” relying on matrix multiplications and cache retrieval (Karmanov et al., 2024). ECALP is likewise training-free, uses sparse modality-aware k-NN graphs, avoids matrix inversion, and supports incremental graph expansion (Li et al., 2024). In the SPML AEVLP framework, CLIP encoders are used off-the-shelf, the label graph is frozen, and DAMP relies on “simple patch extraction and weak augmentation” (Tran et al., 28 Aug 2025). SS-Text-U keeps the encoders frozen and updates only the linear probe and pseudo-labels through Sinkhorn scaling and closed-form prototype updates (Silva-Rodríguez et al., 3 Mar 2026).

Method	Setting	Adaptive / efficient mechanism
VLM-PL	CIOD	instance-level prompt verification; training-free Ferret filtering
TDA	test-time adaptation	dynamic positive/negative caches; no backpropagation
DPA	unsupervised adaptation	dual prototypes, convex fusion, alignment
AEVLP (DAMP + GPR)	SPML	epoch-wise multi-focus pseudo-label refresh; frozen CLIP
ECALP	zero-/few-shot adaptation	dynamic graph expansion; training-free propagation
MarvelOVD	open-vocabulary detection	online mining, adaptive reweighting, stratified assignment
SS-Text-U	semi-supervised few-shot adaptation	OT pseudo-labels; closed-form linear-probe updates

A recurring misconception is that AEVLP always means thresholding zero-shot CLIP scores. The cited work shows a broader space: binary VLM gating, cache-based refinement, graph diffusion, OT-constrained soft assignments, detector-guided reweighting, and robust loss design all fall under the same umbrella when they use vision-language structure to obtain pseudo-supervision adaptively and efficiently.

2. Shared algorithmic structure

Most AEVLP instantiations begin with an initial pseudo-label source derived from vision-language similarity or a detector. In VLM-PL, a pre-trained detector $M_{\text{old}}$ first produces candidate pseudo ground-truths, with query-level acceptance defined by

$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$

and a candidate is accepted if $s(q) \ge \tau$ with $\tau = 0.3$ (Kim et al., 2024). In TDA, the base pseudo-label is the one-hot argmax of the CLIP softmax over text embeddings, while uncertain samples may also receive negative pseudo-labels through an entropy-gated criterion (Karmanov et al., 2024). In DPA, pseudo-labels are not taken from a single classifier: the textual classifier $p_t$ and the image-prototype classifier $p_v$ are fused by

$$\hat{p} = \beta \cdot DA(p_t) + (1-\beta)\cdot p_v, \qquad \hat{y} = \arg\max_c \hat{p}_c,$$

where $DA(\cdot)$ denotes distribution alignment (Ali et al., 2024).

The second stage is typically a reliability mechanism. VLM-PL queries Ferret-13B with a region-grounded prompt and sets a binary gate $w_i \in \{0,1\}$ according to whether the answer is “yes” or “no,” yielding

$$\hat{y}_{\text{refined}} = \{(\hat{c}_i,\hat{B}_i)\mid w_i=1\}.$$

The prompt explicitly combines global image features, spatial coordinates, region-grounded embeddings, and the predicted class name (Kim et al., 2024). TDA uses entropy-prioritized queues and a negative cache; its adapted logits are

$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$0

This makes pseudo-label refinement progressive rather than one-shot (Karmanov et al., 2024). DPA ranks pseudo-labels by the “dual-consensus” score

$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$1

so that self-training emphasizes samples on which both classifiers agree (Ali et al., 2024).

A third stage integrates the refined pseudo-labels into downstream supervision. In VLM-PL, refined pseudo GTs for old classes are merged with real GTs for new classes and trained with the same DETR-style objective as the baseline, with no additional distillation term (Kim et al., 2024). In the SPML AEVLP framework, DAMP converts global and local CLIP scores into pseudo-labels $$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$2, and GPR Loss applies different case-wise losses and weights depending on whether a class is the observed positive, pseudo-negative, pseudo-positive, or undefined (Tran et al., 28 Aug 2025). In ECALP, pseudo-labeling itself is formulated as graph diffusion:

$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$3

with label resets for prompt and few-shot nodes at every iteration to prevent drift (Li et al., 2024).

This repeated pattern—initial proposal, reliability control, and structured integration—explains why AEVLP appears in otherwise different regimes. The vision-language signal is rarely treated as ground truth; it is filtered, fused, propagated, or regularized before entering the learning loop.

3. Detection-oriented AEVLP

The most explicit detection-oriented AEVLP formulation is VLM-PL for Class Incremental Object Detection. Its baseline is Deformable DETR with ResNet-50 and $$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$4 object queries. For each candidate pseudo GT, CLIP ViT-L/14 provides global image embeddings, a binary mask extracts region-specific features, and Ferret-13B answers the prompt: “<image feature> Considering the region <location> <region feature> of the image, would you classify it as a <object name> category without any doubt? Respond with only ‘yes’ or ‘no’.” Accepted pseudo GTs are then mixed with real GTs for the current task (Kim et al., 2024).

The method targets a specific failure mode of conventional CIOD pseudo-labeling: in multi-scenario settings such as 5+5+5+5 or 10+5+5, the old detector’s pseudo GTs become increasingly noisy, which degrades retention of old classes. VLM-PL addresses this with region-grounded verification rather than replay or distillation. On Pascal VOC, it reports AP50 = 65.5% in the 5+5+5+5 setting, “+6.78 over previous SoTA DMD+IFD,” and AP50 = 67.9% in 10+5+5, “matching ABR without replay.” In dual-scenario VOC it reports overall AP50 = 73.6% for 19+1, 72.4% for 15+5, 78.3% for 10+10, and 81.0% for 5+15. On COCO 70+10 it reports AP = 39.8, AP50 = 58.2, AP75 = 43.3, APS = 23.3, APM = 43.5, and APL = 51.6 (Kim et al., 2024).

MarvelOVD addresses a related but distinct problem in open-vocabulary detection: VLM-generated pseudo-labels for novel objects are biased because CLIP lacks a background concept and often lacks sufficient proposal context. MarvelOVD therefore combines offline CLIP proposal scoring with detector-aware Online Mining, Adaptive Reweighting, and Stratified Label Assignments. Its combined pseudo-label score is

$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$5

and novel training boxes are weighted by

$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$6

where $$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$7 is the detector’s background probability (Wang et al., 2024).

The empirical role of these mechanisms is explicit. On COCO 2017, MarvelOVD improves AP50$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$8/AP50$$\hat{c}(q) = \arg\max_c \hat{O}(q,c), \qquad s(q) = \max_c \hat{O}(q,c),$$9/AP50$s(q) \ge \tau$0 from 32.3/54.0/48.3 for the VL-PLM baseline to 38.9/56.4/51.8. Its ablation sequence shows 32.7/54.0/48.5 for the baseline with standard settings, 34.2/53.9/49.1 after weak–strong augmentation, 34.4/56.4/50.5 after stratified assignment, 37.8/56.5/51.3 after online mining, and 38.9/56.6/51.8 after adaptive reweighting (Wang et al., 2024). A notable controversy addressed by this paper is the “base-novel-conflict” problem: pseudo-labels for novel categories can harm base-category performance unless label assignment is stratified.

Taken together, VLM-PL and MarvelOVD show that in detection, AEVLP often means box-level verification or weighting rather than direct pseudo-label trust. Region grounding, detector background estimates, and assignment rules are as important as text-image similarity.

4. Training-free and lightweight adaptation

TDA is a prototypical AEVLP method for test-time adaptation of CLIP-like models. It uses a positive cache for confident pseudo-labels and a negative cache for “absence-of-class” pseudo-labels when the entropy lies in the uncertainty interval $s(q) \ge \tau$1. The default hyperparameters are $s(q) \ge \tau$2, $s(q) \ge \tau$3, $s(q) \ge \tau$4, $s(q) \ge \tau$5, and $s(q) \ge \tau$6 (Karmanov et al., 2024). The positive and negative caches are updated online by replacing high-entropy pairs with lower-entropy ones, so pseudo-label quality improves gradually as more target samples arrive.

Its efficiency claim is unusually concrete. On ImageNet with a single NVIDIA Quadro RTX 6000 GPU, CLIP-ResNet-50 requires 12 minutes, TDA 16 minutes, TPT 12 hours 50 minutes, and DiffTPT 34 hours 45 minutes. In the same setting, ImageNet accuracy is 59.81% for CLIP, 60.74% for TPT, 60.80% for DiffTPT, and 61.35% for TDA (Karmanov et al., 2024). On the broader OOD benchmark, TDA reaches average 49.58 and OOD average 46.63 with ResNet-50, and average 65.01 and OOD average 63.89 with ViT-B/16 (Karmanov et al., 2024). The negative cache is therefore not an incidental addition; it is one of the mechanisms through which AEVLP can mitigate pseudo-label noise rather than simply accepting confident positives.

ECALP occupies a neighboring but graph-theoretic position. It constructs a graph over text prototype nodes, labeled few-shot nodes, and unlabeled test nodes; applies modality-aware sparse k-NN, power sparsification with $s(q) \ge \tau$7, and iterative label propagation with $s(q) \ge \tau$8; and reuses attenuated pseudo-labels with $s(q) \ge \tau$9 (Li et al., 2024). Its adaptation remains training-free because it uses only propagation on pre-trained embeddings. On ImageNet RN50, ECALP reports 28.0 ms test-time versus 898.7 ms for TPT and 2472.5 ms for DiffTPT; dynamic expansion yields “~6× speedup vs static construction on ImageNet” (Li et al., 2024).

DPA is not training-free, but it is parameter-efficient. It updates textual prototypes $\tau = 0.3$0 and the layer-normalization weights of the visual encoder while keeping the text encoder unused after prototype initialization. The image prototypes $\tau = 0.3$1 are recomputed from a memory bank at the end of each epoch, and at inference the image prototypes are discarded so prediction uses textual prototypes only (Ali et al., 2024). On 13 downstream vision tasks with ViT-B/32, DPA achieves 72.29% average versus 64.46% for zero-shot CLIP, an improvement of +7.83%; on ViT-B/16 it reports 74.26% average across 11 tasks (Ali et al., 2024).

SS-Text-U extends the same efficiency logic to semi-supervised few-shot medical adaptation. It computes pseudo-labels for unlabeled samples by Sinkhorn-scaled OT under a class-marginal constraint, and updates the linear probe by closed form:

$\tau = 0.3$2

It reports average ACA improvements over SS-Text+ of +10.9%, +7.1%, +2.7%, +1.3%, and +0.3% for $\tau = 0.3$3, respectively, and an end-to-end runtime of approximately 25 ms on a commodity laptop (Silva-Rodríguez et al., 3 Mar 2026).

These papers collectively show that “efficient” in AEVLP is not limited to one operational form. It may mean no backpropagation at test time, sparse graph updates, prototype-only inference, or closed-form probe adaptation, provided that the vision-language pseudo-labeling mechanism remains lightweight relative to the downstream task.

5. Single-positive multi-label learning and the formal AEVLP framework

In Single Positive Multi-Label Learning, the training signal is especially sparse: each image has only one confirmed positive label and all other labels are unannotated. VLPL established a direct vision-language pseudo-labeling approach for this setting by using a fixed CLIP branch in parallel with a trainable image encoder. With prompt template “A photo of X,” CLIP probabilities are computed by a temperature-softmax over image-text dot products, and labels above threshold $\tau = 0.3$4 become pseudo-positives (Xing et al., 2023). Although the initial formulation allowed pseudo-negatives, the paper reports that pseudo-negatives hurt performance on Pascal VOC: mAP is 89.10 with positives-only, versus 88.51, 88.43, 88.33, and 88.13 when pseudo-negatives are added at $\tau = 0.3$5 (Xing et al., 2023).

The later paper “More Reliable Pseudo-labels, Better Performance: A Generalized Approach to Single Positive Multi-label Learning” explicitly states that DAMP and GPR Loss “together, these contributions form the Adaptive and Efficient Vision-Language Pseudo-Labeling (AEVLP) framework” (Tran et al., 28 Aug 2025). DAMP generates a global view and multiple local patch views, computes CLIP similarities for all views, and aggregates them into positive and negative pseudo-labels. Its local aggregation is

$\tau = 0.3$6

where $\tau = 0.3$7 and $\tau = 0.3$8 are the max and min local scores, respectively (Tran et al., 28 Aug 2025).

GPR Loss then treats pseudo-labels as structured rather than uniform supervision:

$\tau = 0.3$9

The case-wise terms differ for observed positives, undefined labels, pseudo-negatives, and pseudo-positives; the weights $p_t$0 include Gaussian-shaped attenuation for ambiguous cases and clamped weights for pseudo-positives (Tran et al., 28 Aug 2025). This is a substantive departure from VLPL’s fixed positives-only regime and illustrates how AEVLP in SPML moved from pseudo-label generation alone to joint pseudo-label generation and robust optimization.

The empirical comparison is correspondingly direct. With ResNet-50 as the classifier backbone and CLIP ViT-B/16 for pseudo-labeling, AEVLP reports mAP 90.46 on VOC, 73.54 on COCO, 50.70 on NUS-WIDE, and 24.89 on CUB. The corresponding baselines include GR Loss at 89.83/73.17/49.08/21.64 and VLPL at 89.10/71.45/49.55/24.02 (Tran et al., 28 Aug 2025). On pseudo-label quality, DAMP reports, for missing positives, VOC average precision 65.13 and accumulated precision 60.33, with average recall 48.29 and accumulated recall 51.10; accumulated recall exceeding average recall is interpreted in the paper as evidence that DAMP expands positive coverage across epochs while maintaining stable precision (Tran et al., 28 Aug 2025).

This SPML line also clarifies that AEVLP does not require huge VLMs. The AEVLP paper emphasizes that it uses CLIP ViT-B/16, approximately 149.6M parameters, rather than larger CLIP ViT-L variants of approximately 428M parameters used by VLPL, while surpassing VLPL on all four datasets (Tran et al., 28 Aug 2025).

6. Limitations, tensions, and likely extensions

The literature is consistent that AEVLP improves pseudo-label quality, but it is equally clear that no single mechanism is universally reliable. VLM-PL notes that if new-task images contain few instances of old classes, pseudo-labeling “has limited coverage,” and that VLM judgments may fail for “small, heavily occluded, or stylized/domain-shifted objects”; prompt phrasing and class aliases can also reduce reliability (Kim et al., 2024). MarvelOVD acknowledges that it “does not improve pseudo-label localization quality itself,” because candidate boxes still come from a base-trained proposal generator (Wang et al., 2024). ECALP lists mixed or multi-modal classes, sparse manifolds, and noisy few-shot exemplars as failure modes, while SS-Text-U reports that performance depends strongly on embedding clusterability and can be sensitive to misestimated class marginals under severe imbalance or domain shift (Li et al., 2024, Silva-Rodríguez et al., 3 Mar 2026).

A second tension concerns negative pseudo-labels. TDA explicitly benefits from a negative cache, arguing that absence-of-class evidence reduces spurious reinforcement under uncertainty (Karmanov et al., 2024). VLPL, by contrast, reports that pseudo-negative labels degrade mAP and therefore uses only pseudo-positives in its best configuration (Xing et al., 2023). The SPML AEVLP framework occupies an intermediate position: it includes negative pseudo-label mining, but the contribution is made robust through case-wise weighting and calibration in GPR Loss rather than through direct trust in negative assignments (Tran et al., 28 Aug 2025). A plausible implication is that the utility of negative pseudo-labeling is regime-dependent: it can help when uncertainty gating or robust weighting is strong, and harm when false negatives are prevalent and unmitigated.

A third misconception is that efficiency always implies a fully frozen pipeline. The record is mixed by design. TDA and ECALP are fully training-free at adaptation time (Karmanov et al., 2024, Li et al., 2024). VLM-PL uses a training-free VLM verification stage but still trains a new detector on mixed supervision (Kim et al., 2024). DPA updates textual prototypes and layer-normalization weights (Ali et al., 2024). The SPML AEVLP framework trains the classifier backbone while keeping CLIP and the label graph frozen (Tran et al., 28 Aug 2025). SS-Text-U freezes the encoders but updates the linear probe in closed form (Silva-Rodríguez et al., 3 Mar 2026). “Efficient,” in this literature, therefore denotes constrained adaptation cost, not necessarily zero learning.

The extension paths proposed by the papers are also diverse. VLM-PL suggests segmentation, keypoint detection, open-world settings, richer attribute prompts, batched VLM queries, lighter VLMs with LoRA adapters, and soft weights $p_t$1 instead of binary gating (Kim et al., 2024). The SPML AEVLP framework suggests better prompts and prompt ensembling, calibrated temperatures, co-occurrence statistics, label hierarchies, and open-vocabulary extensions (Tran et al., 28 Aug 2025). MarvelOVD suggests detector-in-the-loop refinement of pseudo-box coordinates, improved context modeling for CLIP crops, prompt engineering, and uncertainty-aware schedules for $p_t$2, $p_t$3, and $p_t$4 (Wang et al., 2024). ECALP’s formulation implies further work on dynamic graph construction without external support sets, while SS-Text-U points toward broader use of OT-constrained pseudo-labeling in low-shot, imbalanced domains (Li et al., 2024, Silva-Rodríguez et al., 3 Mar 2026).

Taken as a whole, AEVLP names a technical convergence: pseudo-labels generated from vision-language structure are no longer treated as static surrogates, but as objects to be verified, ranked, propagated, calibrated, or balanced before they become supervision. That convergence is visible across CIOD, OVD, SPML, test-time adaptation, unsupervised domain adaptation, zero-/few-shot inference, and semi-supervised few-shot medical learning (Kim et al., 2024, Karmanov et al., 2024, Ali et al., 2024, Tran et al., 28 Aug 2025, Li et al., 2024, Wang et al., 2024, Silva-Rodríguez et al., 3 Mar 2026).