[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation

Abstract: Vision-LLMs such as CLIP exhibit strong zero-shot recognition capability by aligning images with textual concepts, yet they often underperform on multi-label recognition where multiple objects co-exist. A key bottleneck is that the [CLS] token, as a single global visual representation, is insufficient to faithfully encode diverse targets with varying scales, contexts, and co-occurrence patterns. To address this limitation, we present a new multi-label image recognition framework, termed PIAA, which formulates prediction as Patch-level Inference followed by Adaptive Aggregation. Specifically, we first enhance patch-wise predictions from two complementary perspectives: (i) mitigating semantic entanglement in the visual encoder to obtain more discriminative patch representations, and (ii) learning an unsupervised visual classifier to narrow the vision-language modality gap. We then introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction. Notably, the entire pipeline is fully training-free, requiring no gradient updates or parameter fine-tuning. Experiments show that our method achieves strong improvements with minimal extra computation, exceeding a 6% mAP gain on the challenging NUS-WIDE benchmark over representative baselines. Code is available at https://github.com/akang-wang/PIAA.

• The paper proposes a novel training-free framework (PIAA) that uses patch-level inference with adaptive aggregation to overcome the limitations of a single [CLS] token.
• It introduces an unsupervised PVCL method leveraging Gaussian Discriminant Analysis to calibrate patch classifiers and reduce modality misalignment.
• PIAA achieves significant gains in mAP and efficiency, outperforming state-of-the-art baselines on benchmarks like COCO and NUS-WIDE.

Patch-Level Inference and Adaptive Aggregation for Training-Free Multi-Label Recognition

Motivation and Problem Statement

Zero-shot vision-LLMs (VLMs) such as CLIP have established effective global image-text alignment, enabling transfer learning and open-vocabulary classification. However, their reliance on a single [CLS] token limits expressivity in multi-label scenarios, especially when multiple objects of differing scales or co-occurring patterns are present. Conventional adaptations, including heuristic region cropping and iterative optimization, incur significant computational costs and sensitivity to region quality. The paper proposes that patch-level inference, inspired by segmentation-style reasoning, provides a complementary and principled solution for multi-label recognition that is both training-free and computationally efficient.

PIAA Framework: Architecture and Algorithmic Innovation

Patch-based Visual Classifier Learning (PVCL)

A principal challenge for patch-level inference in VLMs is the vision-language modality gap: patch features are not well aligned with text prototypes, resulting in unreliable patch-wise classification. PVCL addresses this by learning unsupervised visual classifiers directly from patch embeddings, exploiting statistical properties of the visual manifold without backpropagation. The approach leverages Gaussian Discriminant Analysis (GDA):

• Stage I: Entropy-guided bootstrapping identifies low-entropy, highly confident patch candidates for class memory banks using zero-shot text alignment.
• Stage II: Vision-driven purification uses the preliminary GDA classifier to select bank elements with high vision-driven confidence, reducing cross-modal noise.
• Stage III: Robust shrinkage induction computes confidence-weighted class prototypes and applies regularized covariance estimation for stable classifier weights.

Statistically modeling the patch distribution ensures calibrated intra-modal decision boundaries and circumvents modality misalignment.

Prediction Adaptive Aggregation (PAA)

Patch-level predictions are susceptible to spatial noise and may be unstable in the presence of small or occluded objects. Aggregating patch scores via naive pooling dilutes discriminative cues. PAA addresses this by:

• Normalizing patch-wise GDA logits to probabilities.
• Max-pooling patch probabilities category-wise and applying a secondary softmax to form valid distributions.
• Fusing patch-level and global [CLS] predictions through convex combination, with empirical analysis favoring a heavy patch-level bias (α = 0.9) to optimally balance local sensitivity and global semantic stability.

This fusion mechanism exploits scale-based complementarity: patch inference excels for small objects, while [CLS] provides scene-level regularization.

Experimental Evaluation and Results

Benchmarking and Baselines

PIAA is benchmarked on four multi-label datasets—VOC07, VOC12, MS COCO, NUS-WIDE—against fully supervised, weakly supervised, training-based unsupervised, and training-free methods including TagCLIP and SPARC. All experiments use frozen CLIP backbones without gradient updates or manual annotation exposure during classifier learning.

Numerical Results

PIAA achieves strong numerical gains:

• NUS-WIDE: Exceeds TagCLIP by +11.9% mAP and CCD by +6.1% mAP.
• COCO: Matches or surpasses fully supervised baselines in zero-shot mode.
• Vanilla CLIP ViT-B/16 baseline mAP rises from 49.2% to 68.8% on COCO with PIAA, a +19.6% improvement.
• The best-performing variant (EVA-02-CLIP ViT-L/14 + PIAA) achieves up to 79.6% mAP on COCO in training-free mode.

Ablation studies demonstrate that both PVCL and PAA provide additive and synergistic gains, with optimal performance achieved through their combination.

Efficiency Analysis

PIAA outpaces prior methods in computational efficiency:

• Learning speedup: 362x faster than CCD due to single-pass closed-form classifier computation.
• Inference throughput: 50x faster than TagCLIP owing to fully parallelized scoring; complexity decoupled from vocabulary size and image scene complexity.

Scalability

PIAA generalizes across VFMs, including larger-scale architectures and advanced pre-training paradigms (EVA-02). Performance consistently improves, with no diminishing returns as model scale increases.

Theoretical and Practical Implications

PIAA demonstrates that patch-level inference combined with unsupervised visual classification and adaptive aggregation can fundamentally overcome [CLS]-based bottlenecks in multi-label zero-shot recognition. The modularity of PIAA allows integration with any attention-disentangled segmentation-style front-end, providing robustness across architectures.

Practically, PIAA establishes a new state-of-the-art for training-free multi-label recognition, making it appropriate for deployment in resource-constrained or real-time settings. The approach reduces annotation and optimization overhead and is resilient to scene complexity.

Theoretically, the results challenge the necessity of gradient-based tuning and iterative optimization in multi-label paradigms, suggesting that closed-form, statistical patch-based modeling suffices for robust performance.

Limitations and Future Directions

Despite strong results, PIAA is sensitive to initial patch embedding quality and bank purity: noisy or biased patches undermine classifier calibration. Future directions include reliability-aware calibration, more robust patch acquisition, and finer-grained resolution schemes to address small object activation diffusion and semantic co-occurrence ambiguity.

Conclusion

The paper introduces PIAA, a training-free multi-label recognition framework for VLMs via patch-level inference and adaptive aggregation. By reducing the vision-language modality gap and consolidating spatial evidence, PIAA provides robust, scalable, and efficient recognition across benchmarks. The approach expands the theoretical understanding of multi-label learning and offers practical utility for zero-shot vision systems (2605.25821).