Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (2412.02220v1)

Abstract: LLMs such as ChatGPT demonstrate strong few-shot adaptability without requiring fine-tuning, positioning them ideal for data-limited and real-time applications. However, this adaptability has not yet been replicated in current Visual Foundation Models (VFMs), which require explicit fine-tuning with sufficient tuning data. Besides, the pretraining-finetuning paradigm has led to the surge of numerous task-specific modular components, such as Low-Rank Adaptation (LoRA). For the first time, we explore the potential of reusing diverse pre-tuned LoRAs without accessing their original training data, to achieve tuning-free few-shot adaptation in VFMs. Our framework, LoRA Recycle, distills a meta-LoRA from diverse pre-tuned LoRAs with a meta-learning objective, using surrogate data generated inversely from pre-tuned LoRAs themselves. The VFM, once equipped with the meta-LoRA, is empowered to solve new few-shot tasks in a single forward pass, akin to the in-context learning of LLMs. Additionally, we incorporate a double-efficient mechanism tailored to our framework, significantly accelerating the meta-training process while maintaining or even improving performance. Extensive experiments across various few-shot classification benchmarks across both in- and cross-domain scenarios demonstrate the superiority of our framework.

• The paper introduces LoRA Recycle, a framework enabling tuning-free few-shot adaptation in Visual Foundation Models (VFMs) by recycling pre-tuned LoRAs via meta-learning.
• The framework employs a double-efficient mechanism using token pruning and sparse tokens, alongside a meta-learning objective that explicitly teaches adaptation without fine-tuning.
• Experimental validation shows LoRA Recycle significantly improves performance (up to 6.27% avg. in-domain) and demonstrates strong cross-domain generalization, offering efficient, data-private VFM adaptation.

Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs

The paper "Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs" introduces a pioneering framework, LoRA Recycle, aimed at achieving tuning-free few-shot adaptation in Visual Foundation Models (VFMs). This approach capitalizes on the potential of reusing diverse pre-tuned Low-Rank Adaptations (LoRAs) without necessitating access to their original training data. The method strives to parallel the adaptability seen in LLMs like ChatGPT, which exhibit inherent few-shot capabilities without fine-tuning—a feature that VFMs have yet to replicate effectively.

Key Contributions

1. LoRA Recycle Framework: The framework enables VFMs to perform tuning-free few-shot adaptations by recycling pre-tuned LoRAs using a meta-learning strategy. This is accomplished by distilling a meta-LoRA from various pre-tuned LoRAs, utilizing surrogate data generated via LoRA Inversion, and subsequently enabling the VFM to solve new tasks in a single inference pass.
2. Double-Efficient Mechanism: Enhancements in efficiency are introduced through a double-efficient mechanism. This involves token pruning during the inversion stage to enhance data generation speed and selectively using sparse tokens during meta-training to further accelerate the process. This not only reduces computational complexity but also enhances performance by reducing noise from generated data.
3. Meta-Learning Objective: The proposed meta-learning objective is designed to explicitly teach the meta-LoRA how to adapt to new tasks without fine-tuning. The framework relies on a distribution of expected tasks represented by the diverse LoRAs to reshape the VFM's prior, thereby facilitating rapid adaptation to similarly distributed new tasks.
4. Cross-Task Interpolation: To intensify the task distribution for meta-training, cross-task interpolation is introduced. This strategy creates new tasks by combining classes from different LoRAs, thus broadening the training spectrum and enhancing the generalization capability of the meta-LoRA.

Experimental Validation

The framework was tested across various few-shot classification benchmarks, both within the same domain as the meta-training and across different domains. Notably, in the in-domain scenario, LoRA Recycle demonstrated a significant performance enhancement, achieving average improvements as high as 6.27% over baseline models. The results underscore the framework's robustness and efficacy in providing enhanced adaptability without the need for resource-intensive fine-tuning. Furthermore, the cross-domain experiments validated LoRA Recycle's superior generalization capabilities even when faced with substantial distributional shifts.

Implications and Future Directions

The research affirms the feasibility of using VFMs for adaptable and rapid solutions in environments characterized by limited data availability. By leveraging the accessibility and diversity of pre-tuned LoRAs, LoRA Recycle circumvents issues related to data privacy and computational expense typically associated with traditional fine-tuning approaches.

Future research may explore the application of LoRA Recycle in other domains beyond visual tasks, potentially investigating interactions between VFMs and LLMs. Additionally, expanding the scope of cross-task interpolation may further bolster the framework's adaptability and robustness, hence providing a broader toolkit for deploying adaptable foundation models in real-time and data-constrained applications. This work sets a precedent for developing parameter-efficient and scalability-enhanced adaptation frameworks that can more closely emulate the in-context learning capabilities of LLMs in VFMs.