CUE: Concept-Aware Multi-Label Expansion to Mitigate Concept Confusion in Long-Tailed Learning

Abstract: Long-tailed distributions are common in real-world recognition tasks, where a few head classes have many samples while most tail classes have very few. Recently, fine-tuning foundation models for long-tailed learning has gained attention due to their excellent performance. However, most existing methods focus solely on mitigating long-tailed distribution bias while overlooking concept confusion caused by the long-tailed distribution. In this paper, we study this problem and attribute it to the mutual exclusivity of single-label supervision under long-tailed distributions, which suppresses feature sharing among related classes and amplifies the dominance of head classes, leading to disrupted inter-class discriminability. To address this, we propose CUE, Concept-aware mUlti-label Expansion, which introduces multi-label concept signals to preserve disrupted inter-class relationships. Specifically, CUE constructs concept sets by (i) extracting instance-level visual cues from zero-shot CLIP and (ii) generating class-level semantic cues with LLM; the two cues are incorporated via separately weighted Binary Logit-Adjustment (BLA) auxiliary losses and jointly optimized with the baseline Logit-Adjustment (LA) loss. Experiments on several long-tailed benchmarks, CUE achieves balanced and strong performance, surpassing recent state-of-the-art methods. Code is available at: https://github.com/zhangruichi/CUE.

• The paper introduces a novel CUE method that integrates VLM- and LLM-derived cues to mitigate concept confusion in long-tailed learning.
• It employs instance-level cues via CLIP and class-level cues via LLM to maintain semantic consistency and boost few-shot accuracy.
• Experimental results show significant improvements on CIFAR100-LT, ImageNet-LT, and iNaturalist2018, notably enhancing tail class performance.

CUE: Concept-Aware Multi-Label Expansion to Mitigate Concept Confusion in Long-Tailed Learning

Introduction and Motivation

Long-tailed learning (LTL) confronts the severe class imbalance ubiquitous in real-world datasets, where head classes are overrepresented and tail classes suffer from scarcity. While numerous approaches address long-tailed bias through balanced sampling and logit adjustment, they frequently overlook concept confusion—the phenomenon where tail samples are misclassified into semantically similar categories, disrupting the underlying inter-class relationships. This issue is particularly amplified when fine-tuning large-scale foundation models, as dominant head classes further suppress feature sharing among related categories due to the mutual exclusivity constraint in single-label supervision.

The paper "CUE: Concept-Aware Multi-Label Expansion to Mitigate Concept Confusion in Long-Tailed Learning" (2605.01309) identifies concept confusion as a distinct source of error, separate from classical class imbalance, and attributes its rise to the disintegration of semantically informed relationships during fine-tuning. The proposed approach, CUE, augments typical LTL pipelines by injecting semantically-aware multi-label supervision sourced from Vision-LLMs (VLMs) and LLMs. This expands the supervision space beyond rigid single-label assignments, enforcing multi-faceted cues to mitigate label exclusivity, and effectively alleviating concept confusion—especially for tail classes.

Figure 1: The CUE framework contrasts prior LTL methods, introducing VLM-backed instance-level and LLM-backed class-level cues to enhance semantic transfer and alleviate concept confusion.

Methodology: Concept-Aware Multi-Label Expansion

CUE is realized as a plug-and-play module, integrating seamlessly into both fine-tuning and from-scratch training strategies. The pipeline incorporates two complementary cue systems:

1. VLM-Based Instance-Level Cues: Exploiting zero-shot predictions from CLIP, instance-level cues are generated via a top-$k$ retrieval of non-ground truth classes exhibiting the highest similarity in CLIP's embedding space. These classes, semantically correlated but not annotated as ground truth, are assigned as additional positive labels per instance, producing a binary multi-label target. This preserves local inter-class structure and discourages representation collapse among visually similar categories.
2. LLM-Based Class-Level Cues: To infuse high-level semantic relations, LLMs are prompted with the label vocabulary to construct neighborhood graphs, identifying for each class a set of semantically proximate counterparts. Batched prompting and careful post-filtering ensure coverage and relevance. These neighbor sets become additional positive signals on the class-level, pushing the model towards globally consistent semantic associations.

   Figure 2: LLM-based class-level cues are derived through batched prompting with full taxonomy, ensuring robust neighbor graph construction despite vocabulary size constraints.

Joint optimization is established as a composite loss. The main single-label objective uses Logit Adjustment (LA) to counteract frequency bias. Instance- and class-level cues are incorporated via separate Binary Logit Adjustment (BLA) terms, with tunable coefficients. This multi-label supervision regularizes the learning trajectory, maintaining semantic coherence even under heavy tail conditions.

Experimental Results

Synthetic and Real-World Benchmarks

CUE is exhaustively evaluated on CIFAR100-LT, ImageNet-LT, Places-LT, and iNaturalist2018, encompassing both synthetic and real-world long-tailed distributions. Across all settings, CUE outperforms strong CLIP-based LTL baselines (LIFT, LiVT, LPT, BALLAD, etc.) in overall and tail-class accuracy, with the most substantial improvements observed for few-shot categories.

• CIFAR100-LT (IR=100): CUE achieves an overall accuracy improvement of +2.5%, and +7.7% for the Few-shot group versus LIFT.
• ImageNet-LT: CUE offers a +1.5% increase on Few-shot, reinforcing its efficacy in heavily imbalanced regimes.
• iNaturalist2018: Gains of +0.6% on Few and +0.5% overall indicate robustness on naturally occurring class imbalance.

Balancedness and Feature Localization

CUE achieves greater representation uniformity as measured by class-wise balancedness scores, promoting consistent separability across class granularity. The method markedly decreases misclassification rates for tail classes, confirming its impact on rectifying semantic confusion.

Grad-CAM visualizations illustrate that CUE yields attention maps more focused and coherent on object-centric regions, outperforming both naive adaptation (LIFT) and zero-shot CLIP in capturing discriminative signals for rare categories.

Figure 3: CUE attention maps display improved object localization versus LIFT and CLIP, indicating more effective semantic transfer and less confusion between interrelated classes.

Ablation Studies and Sensitivity Analysis

Component ablations confirm that both VLM- and LLM-derived cues individually elevate performance over the baseline, with the most salient uplift contributed by VLM instance-level guidance. Combining both cues yields the strongest, most balanced gains. Sensitivity analysis on cue weighting demonstrates strong robustness, with optimal results typically emerging when contributions from both cues are nonzero but not dominant.

Figure 4: The interplay between $\lambda_{\text{VLM}}$ and $\lambda_{\text{LLM}}$ across datasets verifies the stability of CUE under a range of hyperparameter configurations.

Extension and Generalizability

CUE is demonstrated to be general-purpose: it integrates seamlessly with multiple parameter-efficient fine-tuning (PEFT) mechanisms (LoRA, VPT, AdaptFormer, etc.), enhancing all backbones, especially in tail classes. Additionally, CUE can be incorporated into from-scratch LTL methods (e.g., DODA, LOS, LA, ResLT) without architectural changes, again yielding consistent accuracy improvements.

Implications and Future Directions

The advancement provided by CUE underscores the significance of explicitly modeling semantic associations to counteract concept confusion introduced by single-label exclusivity. This paradigm not only reestablishes robust feature sharing across the class hierarchy but also paves the way for foundation models to generalize more equitably under real-world imbalanced conditions.

Pragmatically, CUE’s plug-and-play character allows for wide applicability—it can augment fine-tuning protocols or be deployed in fully supervised, from-scratch settings. The reliance on VLM and LLM priors, however, places some dependency on the external quality and coverage of these models. Future research may explore adaptive or dynamic cue generation, hierarchically structured multi-label expansion, and efficient integration for large-scale scenarios with minimal computational overhead.

Conclusion

CUE proposes a systematic solution to concept confusion in long-tailed learning by augmenting supervision with semantically structured multi-label cues derived from pre-trained vision-language and LLMs. The framework establishes a robust means to preserve inter-class relationships during fine-tuning, leading to more balanced recognition across the class spectrum. Its modular design and proven gains across diverse backbones and training protocols highlight its strong practical and theoretical relevance for future directions in robust foundation model adaptation.