Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation

Abstract: Recent advances in pre-trained Vision LLMs (VLM) have shown promising potential for effectively adapting to downstream tasks through prompt learning, without the need for additional annotated paired datasets. To supplement the text information in VLM trained on correlations with vision data, new approaches leveraging LLMs (LLM) in prompts have been proposed, enhancing robustness to unseen and diverse data. Existing methods typically extract text-based responses (i.e., descriptions) from LLM to incorporate into prompts; however, this approach suffers from high variability and low reliability. In this work, we propose Description-free Multi-prompt Learning(DeMul), a novel method that eliminates the process of extracting descriptions and instead directly distills knowledge from LLM into prompts. By adopting a description-free approach, prompts can encapsulate richer semantics while still being represented as continuous vectors for optimization, thereby eliminating the need for discrete pre-defined templates. Additionally, in a multi-prompt setting, we empirically demonstrate the potential of prompt weighting in reflecting the importance of different prompts during training. Experimental results show that our approach achieves superior performance across 11 recognition datasets.

• The paper introduces a novel method (DeMul) that distills LLM knowledge directly into vector prompts, bypassing text description limitations.
• It employs a dynamic weighting mechanism to prioritize semantically rich prompts in multi-prompt learning for improved classification accuracy.
• Experiments on 11 diverse image datasets show that DeMul outperforms previous zero-shot and few-shot methods, especially in few-shot settings.

Weighted Multi-Prompt Learning with Description-Free LLM Distillation

Introduction

This paper introduces "Weighted Multi-Prompt Learning with Description-Free LLM Distillation," proposing a novel method, DeMul, that enhances the adaptability of Vision LLMs (VLM) by leveraging LLMs without relying on text-based descriptions. Recent advancements in VLMs, like CLIP, utilize prompts derived from LLMs to integrate textual insights with visual data. However, the paper critiques existing methodologies for generating prompts from LLM responses, which suffer from inconsistencies and ambiguity in text descriptions, as illustrated in Figure 1.

Figure 1: High variability and low reliability of GPT-based descriptions impacting prompt consistency.

Methodology

Description-Free Distillation

DeMul addresses the limitations inherent in text-based LLM prompting by directly distilling LLM knowledge into learnable prompts in vector form, bypassing text generation. This approach reduces variability and enhances semantic encapsulation. The methodology involves mapping learnable prompts into an LLM embedding space (e.g., GPT), allowing for optimization and richer semantic representation than traditional template-based prompts.

Prompt Weighting in Multi-Prompt Learning

The paper introduces a weighting mechanism that dynamically adjusts the importance of different prompts during training. This mechanism is crucial in a multi-prompt setting, where various prompts might encapsulate distinct semantic aspects of a class. The prompt weights are optimized using few-shot learning, which ensures that important prompts have more influence on the model's decision-making process.

Architecture and Training

The architecture extends CLIP by integrating a pre-trained GPT embedding model, using it for semantic alignment, as depicted in Figure 2.

Figure 2: An overall framework of DeMul showing the integration and contextual learning process involving multiple trained components.

The training objective combines distillation and classification loss, controlled by a weighting parameter to balance these aspects for different downstream tasks.

Evaluation

Experimental Setup

DeMul's performance was validated on 11 diverse image recognition datasets, ranging from generic categories like ImageNet to specific tasks like EuroSAT. The model's effectiveness was benchmarked against zero-shot and few-shot baseline methods including CLIP, dCLIP, WaffleCLIP, and learnable prompt methods like CoOp, MaPLe, and GalLoP.

Performance Results

As demonstrated in Figure 3, DeMul consistently outperformed both zero-shot description-based methods and learnable prompt methods across datasets. The improvements were particularly significant in few-shot settings, supporting the hypothesis that bypassing text descriptions enhances accuracy and robustness.

Figure 3: Comparison of few-shot learning accuracy showcasing DeMul's superiority.

Analysis and Discussion

Mapping and Prompt Dynamics

The study of mappings between CLIP and GPT embedding spaces revealed the preservation of crucial semantic information through conformal mappings. Figure 4 illustrates the U-Map visualization, confirming that fine-tuning maintains semantic clusters in GPT space.

Figure 4: The U-map visualization of CLIP and GPT embedding spaces indicates robust mapping and semantic retention.

Prompt Weight Analysis

The evaluation of prompt weight dynamics showed that weights correlate with the semantic alignment of prompts to class names. Figure 5 highlights these variations, supporting the prompt weighting framework's validity in enhancing classification accuracy by dynamically emphasizing semantically relevant prompts.

Figure 5: Variations of the weights of prompts during training illustrating effective semantic alignment.

Conclusion

The proposed DeMul method fundamentally improves the integration of LLM-driven insights into VLMs by eliminating reliance on text descriptions, thereby enhancing prompt semantic richness and adaptability in multi-class recognition scenarios. By leveraging description-free distillation and introducing prompt weighting, DeMul achieves superior performance compared to traditional methods. Future work may explore handling diverse unseen distributions while capturing class-specific prompt details efficiently.