Low-Rank Few-Shot Adaptation of Vision-Language Models

Abstract: Recent progress in the few-shot adaptation of Vision-LLMs (VLMs) has further pushed their generalization capabilities, at the expense of just a few labeled samples within the target downstream task. However, this promising, already quite abundant few-shot literature has focused principally on prompt learning and, to a lesser extent, on adapters, overlooking the recent advances in Parameter-Efficient Fine-Tuning (PEFT). Furthermore, existing few-shot learning methods for VLMs often rely on heavy training procedures and/or carefully chosen, task-specific hyper-parameters, which might impede their applicability. In response, we introduce Low-Rank Adaptation (LoRA) in few-shot learning for VLMs, and show its potential on 11 datasets, in comparison to current state-of-the-art prompt- and adapter-based approaches. Surprisingly, our simple CLIP-LoRA method exhibits substantial improvements, while reducing the training times and keeping the same hyper-parameters in all the target tasks, i.e., across all the datasets and numbers of shots. Certainly, our surprising results do not dismiss the potential of prompt-learning and adapter-based research. However, we believe that our strong baseline could be used to evaluate progress in these emergent subjects in few-shot VLMs.

• The paper introduces a novel low-rank adaptation technique that boosts few-shot performance in vision-language models.
• It shows that adapting both vision and text encoders simplifies training by reducing parameters and avoiding extra inference overhead.
• Evaluation on 11 datasets, including ImageNet, demonstrates that CLIP-LoRA outperforms traditional prompt and adapter methods in accuracy.

Low-Rank Few-Shot Adaptation of Vision-LLMs

Introduction

The paper "Low-Rank Few-Shot Adaptation of Vision-LLMs" (2405.18541) introduces a novel approach to enhancing few-shot learning capabilities in Vision-LLMs (VLMs) using Low-Rank Adaptation (LoRA). Traditional few-shot learning methodologies have predominantly focused on prompt learning and, to a lesser extent, adapters, often requiring extensive computational resources and task-specific hyper-parameter tuning. In contrast, LoRA represents a more efficient alternative, offering significant performance improvements across diverse datasets while maintaining a consistent set of hyper-parameters. This simplification facilitates broader applicability of VLMs without extensive customization, positioning LoRA as a robust baseline for measuring advancements in few-shot learning strategies.

Parameter Efficient Fine-Tuning Strategies

The paper discusses the landscape of Parameter-Efficient Fine-Tuning (PEFT) methods, which aim to alleviate the computational burden associated with tuning large-scale models by optimizing a small subset of model parameters. PEFT encompasses several categories, including selective tuning methods, adapters, prompt tuning, and the novel approach of LoRA.

Selective methods fine-tune existing model weights, while adapter strategies introduce trainable modules, often increasing inference latency. Prompt tuning optimizes input token representations, albeit incurring substantial computational overhead. LoRA, however, adapts by introducing low-rank matrices that augment pre-trained weights. This approach not only reduces training complexity but also minimizes inference time by merging adapted matrices with the original weights, thus eliminating additional parameters post-adaptation.

Figure 1: Prompt, Adapter and Low-rank techniques introduce extra parameters for training, which may potentially extend training duration and/or memory footprint in comparison to selective methods. However, they have the advantage of being more flexible and are often easier to use.

Few-Shot Learning for VLMs

The authors conducted an extensive evaluation of few-shot learning techniques across 11 datasets, including ImageNet and various fine-grained classification sets. The study compared LoRA-based adaptation to prompt and adapter-based methods like CoOp, CoCoOp, and TIP-Adapter-F. CLIP-LoRA demonstrated superior performance by significant margins on key datasets, achieving higher average accuracy across different visual backbones (ViT-B/16, ViT-B/32, ViT-L/14).

The results underscore LoRA’s potential to streamline few-shot training processes while simultaneously achieving state-of-the-art performance benchmarks, particularly noteworthy on datasets like ImageNet and UCF101, which require effective cross-modal feature alignment.

Figure 2: Detailed few-shot learning results on the 10 fine-grained datasets and ImageNet with the ViT-B/16 visual backbone. Average performance for the ViT-B/16, ViT-B/32 and ViT-L/14 on the same 11 datasets is reported in the last three plots, respectively.

Design Considerations for LoRA

Critical design decisions guide the deployment of LoRA modules within VLMs, specifically regarding encoder selection, matrix adaptation, and rank assignment of low-rank matrices. Empirical results suggest that adapting both vision and text encoders, focusing particularly on value or output matrices, provides the most substantial performance gains. The strategic placement and parametrization of LoRA components are pivotal, suggesting further research opportunities in dynamically adjusting ranks and optimizing encoder interactions for specific tasks.

Conclusion

The paper presents CLIP-LoRA as a strong baseline in few-shot adaptation for VLMs, outperforming traditional methods while simplifying model training through fixed hyper-parameters. This approach offers practical implications for deploying VLMs in diverse application domains, facilitating efficient adaptation without compromising accuracy or requiring intricate manual configuration.

Future research should explore the dimensional adaptability of LoRA matrices and investigate further cross-modal tuning techniques, as these areas promise enhancements in model robustness and generalization capabilities in few-shot learning scenarios.