Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labels

Abstract: Prompt learning has gained significant attention as a parameter-efficient approach for adapting large pre-trained vision-LLMs to downstream tasks. However, when only partial labels are available, its performance is often limited by label ambiguity and insufficient supervisory information. To address this issue, we propose Holistic Optimal Label Selection (HopS), leveraging the generalization ability of pre-trained feature encoders through two complementary strategies. First, we design a local density-based filter that selects the top frequent labels from the nearest neighbors' candidate sets and uses the softmax scores to identify the most plausible label, capturing structural regularities in the feature space. Second, we introduce a global selection objective based on optimal transport that maps the uniform sampling distribution to the candidate label distributions across a batch. By minimizing the expected transport cost, it can determine the most likely label assignments. These two strategies work together to provide robust label selection from both local and global perspectives. Extensive experiments on eight benchmark datasets show that HopS consistently improves performance under partial supervision and outperforms all baselines. Those results highlight the merit of holistic label selection and offer a practical solution for prompt learning in weakly supervised settings.

• The paper introduces HopS, which integrates local density filtering and global optimal transport to mitigate label ambiguity in prompt learning.
• It employs a kNN-based local filter (LDF) and entropy-regularized optimal transport (GOP) via Sinkhorn iterations to refine pseudo-labels.
• Empirical results demonstrate that HopS outperforms traditional PLL methods in few-shot and instance-dependent noisy settings.

Holistic Optimal Label Selection for Robust Prompt Learning Under Partial Labels

Problem Motivation and Background

Prompt learning serves as an efficient alternative to full-scale fine-tuning for pre-trained vision-LLMs (VLMs) such as CLIP, leveraging learnable prompts while keeping massive learned encoders frozen. Its utility is maximized in few-shot settings, retaining zero-shot generalization and reducing training cost. However, practical deployment suffers under partial-label learning (PLL): each instance is associated with a set of candidate labels, of which only one is correct, but the identity of the ground truth is unknown. This label ambiguity induces severe supervision noise, particularly acute in low-data regimes, and current PLL approaches typically break down in such scenarios, especially for prompt-tuning frameworks that rely on fixed representations rather than deeply fine-tuned networks.

Holistic Optimal Label Selection: Methodological Contributions

Local Density-Based Filter (LDF)

The LDF component identifies locally consistent candidate labels through a $k$-nearest-neighbors (kNN) analysis in frozen image feature space induced by the pre-trained vision backbone. For each image, the candidate label sets of itself and its $k$-nearest neighbors are aggregated and label frequencies are computed. Labels with frequency above a threshold $\tau$ form a consensus-aware refined candidate set. Final label assignment within this filtered set uses softmax-based selection over the prompt-conditioned class probability. This exploits local semantic structure and regularizes against spurious candidates, with minimal computational overhead due to prior affinity matrix computation.

Global Optimal Transport Planner (GOP)

GOP formulates the label assignment problem globally as an entropy-regularized optimal transport (OT) task over each mini-batch. The instance-to-label allocation seeks to minimize the expected transport cost, where the cost is constructed from model-predicted image–label similarities, but with infeasible assignments (non-candidate labels) prohibited. OT marginal constraints enforce alignment between the prior over instances and the batch-level candidate label distribution, yielding a soft, globally harmonized pseudo-label scheme. The transport plan is computed efficiently via Sinkhorn iterations.

Figure 1: The construction of the cost matrix and result of the OT-based transport plan between uniform source (instances) and target (candidate label distributions).

Integration and Objective

LDF and GOP are orthogonally combined in HopS. For each instance, both modules generate their highest-probability pseudo-label. The prompt parameters are then optimized by a weighted sum of cross-entropy losses with respect to these two labels:

$$\min_{\mathbf{t}} \mathcal{L}_\text{CE}(p(y \mid \mathbf{x}; \mathbf{t}), y^\mathrm{local}) + \lambda \mathcal{L}_\text{CE}(p(y \mid \mathbf{x}; \mathbf{t}), y^\mathrm{global})$$

where $\lambda$ balances the local and global components.

Empirical Evaluation

Experimental Setup

Eight datasets, including both generic (Caltech, EuroSAT) and fine-grained (FGVCAircraft, Food, Flowers, etc.) collections, are used under various partial-label regimes. Two confusion types are synthesized: random-uniform (rand) and instance-dependent (insd), the latter reflecting real-world annotator or data-dependent ambiguity via feature-similarity-driven label selection. A wide range of candidate set sizes and confusion rates $\gamma_c$ are evaluated, with backbone features frozen and learnable prompts optimized via CoOp-style framework in both unified and class-specific modes. Baselines include state-of-the-art loss-based PLL/robust learning algorithms and recent specialized methods (Papi, CroSel, SoLar).

Main Results and Quantitative Analysis

HopS consistently outperforms all baselines across confusion regimes and datasets, including difficult instance-dependent and long-tailed settings. High performance holds even as $\gamma_c \to 0.9$, well beyond typical robustness breakdown thresholds for previous approaches.

Figure 2: HopS achieves superior test accuracy under both uni-prompt and cls-prompts settings across a spectrum of confusion rates (random confusion)

Figure 3: Averaged test accuracy across confusion rates for insd-confusion (left), and demonstration of LDF’s positive influence as a guide for GOP optimization (right).

Notably, HopS with only 16-shots competes robustly against methods trained on the full data. Even when ground-truth labels are systematically missing from candidate sets (reflecting real-world annotator error), HopS maintains performance advantages, especially for instance-dependent noise.

Figure 4: HopS produces strong accuracy under severe label noise conditions and missing ground-truth labels in candidate sets.

Complementarity between LDF and GOP is empirically confirmed: early optimization is guided by LDF's exploitation of local structure, while GOP’s global harmonization achieves stable, higher plateaus of accuracy as training advances.

Figure 5: The overlap of ground-truth label identification by LDF and GOP demonstrates strong complementary effect.

Extensive ablations highlight that local neighbor-based refinement is more effective than confidence- or graph-based strategies in the few-shot regime, and that batch sizes close to the number of classes are optimal for OT/GOP performance.

Figure 6: Testing accuracy as a function of batch size $B$; accuracy is optimized when $B$ is close to the number of classes.

HopS’s computational overhead is minimal compared to standard PLL losses, owing to the locality of LDF computations (precomputed affinity) and efficient batch-wise Sinkhorn iterations for GOP.

Additional Analyses: Complementarity, Generalization, and Scalability

Across modalities, HopS demonstrates stable hyperparameter sensitivity, scalability to large class sizes, and robust memory usage characteristics. Its effectiveness is validated under reduced supervision settings (4-shot, 8-shot), long-tail class imbalance, and on both image-text and generic vision tasks.

Figure 7: Performance of HopS across batch sizes and confusion rates confirms the recommended configuration of $B \approx C$.

(further dataset-wise ablations shown in Figures 8–15)

Implications and Future Directions

The synthesis of local structural analysis and global batch-level transport optimization establishes a flexible, extensible framework for prompt learning under partial or ambiguous supervision. By explicitly leveraging the generalization capacity of frozen VLM encoders, HopS circumvents the need for encoder fine-tuning, thus preserving zero-shot capabilities and memory efficiency.

Potential extensions include:

• Incorporating continual learning to further leverage the sparsity and transferability of VLM features;
• Dynamic adjustment of local/global weighting coefficients over training & dataset-specific adaptation;
• Integration with semi-supervised learning schemes for additional positive transfer in scenarios with a mix of labeled, partially-labeled, and unlabeled data;
• Extension to multimodal (image/text/audio) ambiguous supervision contexts.

Conclusion

HopS provides a robust, efficient, and practical solution for prompt tuning under partial-label supervision in vision-LLMs. Through the combined strengths of local density-based filtering and global optimal transport planning, it achieves state-of-the-art accuracy and label disambiguation across diverse datasets and challenging label ambiguity regimes, with negligible computational overhead. The method establishes a strong foundation for future advancements in weakly supervised, prompt-based adaptation of large-scale pre-trained models.

(2604.06614)