Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels

Abstract: Understanding multi-label images remains a challenging task in computer vision. With the rapid progress of vision-language multimodal learning, vision-LLMs (VLMs) enable zero-shot recognition without labeled data. However, due to their intrinsic design, these models often prioritize the most iconic object and omit other contextual positives. This intrinsic bias conflicts with the nature of multi-label learning, thereby limiting their applicability. In this work, we propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding, enabling label-free multi-label image recognition. Our approach consists of two key stages, cutting'' andsewing'': In the cutting stage, we present the multi-sampling response estimator to prevent the model from concentrating only on one single object. In the second sewing stage, the multi-object blend adaptation is introduced to adjust the labels to better conform to the multi-label distribution while preserving the intrinsic characteristics of the original model within only one epoch. Extensive experiments show that our framework significantly outperforms existing unsupervised approaches on four public datasets, even surpassing several representative weakly supervised baselines. These results demonstrate the potential of adapting pre-trained VLMs for more comprehensive visual understanding without manual annotations. Our code is publicly available at https://github.com/iCVTEAM/TailorCLIP.

• The paper presents a two-stage framework where the cutting phase diversifies local responses and the sewing phase rebalances predictions for true multi-label distributions.
• It achieves significant performance improvements, raising mAP on benchmarks like MS-COCO from 36.9% to nearly 79.8% without human annotations.
• The study demonstrates that unsupervised adaptation can unlock latent multi-label knowledge in VLMs, overcoming the inherent one-positive bias.

Adapting Vision-LLMs for Unsupervised Multi-Label Recognition

Introduction

Multi-label image recognition is central to a range of vision tasks, from detection and segmentation to retrieval. Recent advances in Vision-LLMs (VLMs), especially CLIP and its derivatives, have enabled strong zero-shot transfer for single-label tasks by leveraging natural language supervision. However, VLMs trained with image-caption pairs are fundamentally iconically biased: they tend to highlight the most salient object per image and suppress other co-occurring, contextually significant classes. This hampers their applicability in multi-label scenarios—the norm in realistic outlets like MS-COCO—where multiple relevant objects per image are routine.

The paper "Adapting Vision-LLMs from Iconic to Inclusive for Multi-Label Recognition Without Labels" (2606.11626) addresses this core misalignment. It introduces an unsupervised framework that reforms (adapts) VLMs into inclusive, multi-label predictors, requiring zero human annotations. The methodology hinges on two synergistic stages: a "cutting" phase to diversify object responses and a "sewing" phase to rebalance and adapt model predictions to true multi-label distributions. This essay provides a rigorous technical synopsis and critical analysis of the model, focusing on its design, empirical performance, implications for vision-language learning, and future directions.

Motivation: Intrinsic One-Positive Bias in VLMs

The pivotal observation is that conventional VLMs, by virtue of their contrastive pretraining, are innately optimized for iconic, single-object recognition—implicitly enforcing a one-positive regime. Empirical analysis on MS-COCO reveals that the mean average precision (mAP) of zero-shot CLIP plummets to 36.9%, compared to 78.5% for a multi-label-optimized baseline. The expected mAP, were CLIP capable of true multi-label reasoning given its top-1 accuracy, is 65.8%, highlighting a marked gap due solely to architectural and data biases. Additionally, response distributions show that, on average, only one class per image is activated, regardless of the true positive count.

Figure 1: CLIP's predictions are heavily skewed toward a single "iconic" object per image, underrepresenting multi-label ground truth and exhibiting intrinsic prediction bias.

This bias undermines both the recall of secondary/contextual objects and the utility of VLMs for in-the-wild, multi-object visual understanding tasks. Prior methods (e.g., CDUL, TagCLIP) partially mitigate this via pseudo-labeling and auxiliary refinements but do not fundamentally extract or rectify the underlying multi-label semantics learned by the VLMs.

Methodology: Cutting and Sewing Framework

The proposed unsupervised adaptation framework features two distinct but mutually reinforcing components:

Cutting: Multi-Sampling Response Estimation

This stage addresses the one-positive bias by forcibly diversifying the regions presented to the VLM. Instead of global evaluation, the image is "cut" into multiple random crops, each covering a $\rho$-proportion of the image. For every sampled crop, CLIP's response vector is extracted. Non-salient, context-rich local patches—otherwise suppressed in global views—surface as top responses for secondary classes. These local logits are then fused via a max-suppression scheme with hyperparameter $\alpha$, yielding pseudo-label confidence vectors that robustly reflect multiple objects, not just the most salient one.

Figure 2: The cutting operation enables previously suppressed non-salient objects to be recovered and correctly scored by the VLM, mitigating one-positive bias.

This data-driven, unsupervised estimator thus facilitates more complete multi-label ground truths without requiring any image-level annotations.

Sewing: Multi-Object Blend Adaptation

Although the cutting stage enriches object responses, it does not yield a multi-label predictor with tuned parameters. The sewing stage reconstructs pseudo-multi-label training data by compositing (sewing) object-centric image crops—sampled via high-confidence local responses—over base images at random positions and scales. The corresponding pseudo-labels are amalgamated as the class-wise max over the constituent cropped images. An order-persistent label correction function $g(\cdot)$ further regularizes confidences, selectively boosting top-class logits (e.g., via a square-root rescaling in the sigmoid domain) while preserving the intra-image confidence order to facilitate knowledge transfer.

A lightweight adapter (transformer-based decoder atop the frozen CLIP backbone) is then trained using binary cross-entropy with these blended pseudo-labels. The process is iteratively refined via an Expectation-Maximization scheme: in each M-step, the model is updated with current pseudo labels; in the E-step, pseudo labels are refined based on model predictions and blended images.

Figure 3: The pipeline integrates (1) cutting for local, multi-object discovery and (2) sewing for data recomposition and model adaptation, with interactive EM optimization.

Empirical Results

The framework is rigorously validated on four benchmarks: PASCAL VOC 2007/2012, MS-COCO, and NUS-WIDE. Noteworthy empirical results include:

• On VOC2012, this fully unsupervised method attains an mAP of 91.4%, outperforming all prior unsupervised approaches and even surpassing several weakly supervised baselines that leverage 10% labels.
• On MS-COCO, using ViT-L/14, the framework achieves an mAP of 79.8%, nearly closing the gap with fully supervised methods.
• The mean number of objects predicted per image closely matches ground truth after sewing, validating that bias correction is effective at the distributional level.

  Figure 4: Sewing elevates the mean number of detected objects per image to match true multi-label distributions, without distorting CLIP's output logit scale.

Ablation studies demonstrate that each component (response estimator, order-preserving correction, blend adaption) incrementally improves both pseudo-label quality and final classifier accuracy. Notably, hard thresholding or label binarization increases the number of pseudo-label order inversions and degrades performance, underscoring the necessity of order-preserving corrections.

Figure 5: Ablations show sensitivity and robustness with respect to pseudo-labeling and blending hyperparameters.

Qualitative Analyses

Class Activation Map (CAM) analysis reveals that, after adaptation, the model is able to localize and distinguish multiple objects per image, rectifying canonical CLIP's tendency to embed non-salient objects as mere context to the predominant object.

Figure 6: Class activation maps show that the adapted model (a) focuses on true object regions, while vanilla CLIP (b) confounds secondary objects as contextual noise.

Feature embedding visualizations (via t-SNE) show that classwise features are more distinctly separated after adaptation, a reflection of improved multi-label discriminability.

Figure 7: Embedding visualizations illustrate improved class separability as a result of the proposed adaptation process.

Theoretical and Practical Implications

This work establishes that the bottleneck for multi-label recognition in VLMs is not a lack of representational power, but a product of single-positive pairing in contrastive pretraining. It demonstrates that careful unsupervised re-balancing—without additional positive labels—can unlock latent multi-label knowledge.

Practical implications are significant: the method operates without any human label supervision and is computationally efficient, requiring only an adapter over a frozen backbone. It is demonstrably robust to backbone architecture (ResNet, ViT) and hyperparameter settings, making it attractive for deployment in annotation-scarce domains.

Future Directions

The methodology foregrounds several exciting directions:

• Exploring joint adaptation schemes that operate during VLM pretraining (rather than post-hoc) to natively support multi-label distributions.
• Scaling adaptation to larger or more semantically rich multi-modal models, possibly integrating generative or retrieval-augmented feedback.
• Leveraging cut-and-sew mechanisms in other domains: object detection, dense segmentation, or temporal sequence modeling (e.g., video).
• Incorporating stronger constraints from language priors to guide object co-occurrence or mutual exclusion.

Conclusion

The framework in "Adapting Vision-LLMs from Iconic to Inclusive for Multi-Label Recognition Without Labels" (2606.11626) delivers a principled and effective unsupervised solution to the iconicity bias of VLMs in multi-label tasks. Through systematic local response estimation and distribution-accurate data recomposition, it enables label-free adaptation of powerful pre-trained models. The empirical results set new benchmarks for zero-label multi-label recognition and reveal unexploited capacities of vision-language pretraining paradigms. This work is poised to inform future research on unsupervised adaptation, multi-label reasoning, and the broader utilization of VLMs for comprehensive scene understanding.