Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Prompt with Text Only Supervision for Vision-Language Models (2401.02418v1)

Published 4 Jan 2024 in cs.CV
Learning to Prompt with Text Only Supervision for Vision-Language Models

Abstract: Foundational vision-LLMs such as CLIP are becoming a new paradigm in vision, due to their excellent generalization abilities. However, adapting these models for downstream tasks while maintaining their generalization remains a challenge. In literature, one branch of methods adapts CLIP by learning prompts using visual information. While effective, most of these works require labeled data which is not practical, and often struggle to generalize towards new datasets due to over-fitting on the source data. An alternative approach resorts to training-free methods by generating class descriptions from LLMs and perform prompt ensembling. However, these methods often generate class specific prompts that cannot be transferred to other classes, which incur higher costs by generating LLM descriptions for each class separately. In this work, we propose to combine the strengths of these both streams of methods by learning prompts using only text data derived from LLMs. As supervised training of prompts is not trivial due to absence of images, we develop a training approach that allows prompts to extract rich contextual knowledge from LLM data. Moreover, with LLM contextual data mapped within the learned prompts, it enables zero-shot transfer of prompts to new classes and datasets potentially cutting the LLM prompt engineering cost. To the best of our knowledge, this is the first work that learns generalized prompts using text only data. We perform extensive evaluations on 4 benchmarks where our method improves over prior ensembling works while being competitive to those utilizing labeled images. Our code and pre-trained models are available at https://github.com/muzairkhattak/ProText.

PDF HTML Abstract

Learning to Prompt with Text-Only Supervision for Vision-LLMs: A Professional Overview

The paper "Learning to Prompt with Text-Only Supervision for Vision-LLMs" addresses a significant challenge in adapting vision-LLMs like CLIP for downstream tasks without sacrificing generalization. The authors propose a novel approach that combines the strengths of existing image-supervised prompt learning techniques and training-free prompt ensembling methods using LLMs.

Core Contributions

The paper introduces ProText, a method that leverages text-only supervision to facilitate prompt learning in vision-LLMs. The core contribution of this work is the development of a training framework that enables prompts to learn rich contextual features using only text data obtained from LLMs. This innovative approach bypasses the need for visual sample labels, which are often impractical or expensive to obtain, especially in domains like medical imaging or remote sensing.

Key Methodological Insights

  1. Text-Only Data Utilization: The authors exploit the capabilities of LLMs to curate detailed class-specific descriptions that serve as the basis for prompt learning. By mapping class names to these descriptions, ProText effectively learns to translate the contextual richness of LLM-generated text into a form usable by vision-LLMs like CLIP.
  2. Contextual Mapping Loss: The process employs a contextual mapping objective, allowing the learnable prompts to associate standard class-name templates with enriched class-specific textual features from LLMs. This enables the prompts to encapsulate versatile and transferable contextual information, fostering zero-shot learning across new classes and datasets.
  3. Transferability Across Datasets: ProText’s training does not require visual data, thus preserving VLMs’ ability to adapt to unseen datasets without incurring additional LLM serving or prompt engineering costs. This aspect significantly reduces computational and economic barriers associated with traditional model training approaches.

Methodological Implications and Performance

ProText demonstrates its efficacy through extensive evaluations on four benchmarks, revealing substantial improvements over other prompt ensembling and image-supervised methods. For instance, in the cross-dataset transfer setting, ProText achieves an average accuracy gain of +2.08% over baseline CLIP, outperforming even the best image-supervised methods like MaPLe.

The approach holds promise for enhancing generalization capabilities without the risk of overfitting inherent in image-supervised learning. By tapping into the extensive knowledge embedded within LLMs, ProText equips vision-LLMs with a robust contextual understanding that extends beyond the limitations of training datasets.

Future Prospects

The introduction of text-only supervised prompt learning opens several research avenues. Future work can explore the integration of advanced LLM models and fine-tuning strategies to further enhance the efficiency of ProText. Moreover, expanding the method's applicability to more diverse and complex datasets could provide deeper insights into its scalability and adaptability.

In summary, this paper presents a compelling argument for the use of text-only supervision in enhancing the generalization and transferability of vision-LLMs, emphasizing the importance of leveraging LLMs not just for generating richer features but as a fundamental component in model adaptation strategies. As we move forward, such innovations are poised to redefine how we approach model tuning in the landscape of artificial intelligence.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. More context, less distraction: Improving zero-shot inference of clip by inferring and describing spurious features. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
  2. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arXiv:2203.17274, 2022.
  3. Bridging the gap between object and image-level representations for open-vocabulary detection. NeurIPS, 35:33781–33794, 2022.
  4. Food-101–mining discriminative components with random forests. In ECCV, pages 446–461. Springer, 2014.
  5. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020.
  6. Plot: Prompt learning with optimal transport for vision-language models. In ICLR, 2022.
  7. Describing textures in the wild. In CVPR, pages 3606–3613, 2014.
  8. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009.
  9. Bayesian prompt learning for image-language model generalization. In CVPR, pages 15237–15246, 2023.
  10. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, pages 14084–14093, 2022.
  11. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, pages 178–178. IEEE, 2004.
  12. Clip-adapter: Better vision-language models with feature adapters. IJCV, pages 1–15, 2023.
  13. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, pages 540–557. Springer, 2022.
  14. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. J-STARS, 12(7):2217–2226, 2019.
  15. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340–8349, 2021a.
  16. Natural adversarial examples. In CVPR, pages 15262–15271, 2021b.
  17. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022.
  18. Scaling up visual and vision-language representation learning with noisy text supervision. In ICLR, pages 4904–4916. PMLR, 2021.
  19. A good prompt is worth millions of parameters? low-resource prompt-based learning for vision-language models. arXiv preprint arXiv:2110.08484, 2021.
  20. Maple: Multi-modal prompt learning. In CVPR, pages 19113–19122, 2023a.
  21. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, pages 15190–15200, 2023b.
  22. 3d object representations for fine-grained categorization. In ICCV, pages 554–561, 2013.
  23. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692, 2023.
  24. Language-driven semantic segmentation, 2022.
  25. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061–7070, 2023.
  26. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
  27. Prompt distribution learning. In CVPR, pages 5206–5215, 2022.
  28. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  29. Visual classification via description from large language models. In ICLR, 2023.
  30. Simple open-vocabulary object detection. In ECCV, pages 728–755. Springer, 2022.
  31. I2dformer: Learning image to document attention for zero-shot image classification. NeurIPS, 2022.
  32. I2mvformer: Large language model generated multi-view document supervision for zero-shot image classification. In CVPR, 2023a.
  33. Silc: Improving vision language pretraining with self-distillation. arXiv preprint arXiv:2310.13355, 2023b.
  34. Automated flower classification over a large number of classes. In ICVGIP, pages 722–729. IEEE, 2008.
  35. Cats and dogs. In CVPR, pages 3498–3505. IEEE, 2012.
  36. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pages 15691–15701, 2023.
  37. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  38. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389–5400. PMLR, 2019.
  39. Waffling around for performance: Visual classification with random words and broad concepts. 2023.
  40. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  41. Test-time prompt tuning for zero-shot generalization in vision-language models. NeurIPS, 35:14274–14289, 2022.
  42. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
  43. Stanford alpaca: An instruction-following llama model, 2023.
  44. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019.
  45. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pages 3485–3492. IEEE, 2010.
  46. Filip: Fine-grained interactive language-image pre-training. In ICLR, 2021.
  47. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
  48. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
  49. Conditional prompt learning for vision-language models. In CVPR, pages 16816–16825, 2022a.
  50. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348, 2022b.
  51. Detecting twenty-thousand classes using image-level supervision. In ECCV, pages 350–368. Springer, 2022c.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Muhammad Uzair Khattak (10 papers)
  2. Muhammad Ferjad Naeem (21 papers)
  3. Muzammal Naseer (67 papers)
  4. Luc Van Gool (569 papers)
  5. Federico Tombari (214 papers)
Citations (12)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube