Sparse High Rank Adapters (2406.13175v2)

Abstract: Low Rank Adaptation (LoRA) has gained massive attention in the recent generative AI research. One of the main advantages of LoRA is its ability to be fused with pretrained models, adding no overhead during inference. However, from a mobile deployment standpoint, we can either avoid inference overhead in the fused mode but lose the ability to switch adapters rapidly, or suffer significant (up to 30% higher) inference latency while enabling rapid switching in the unfused mode. LoRA also exhibits concept-loss when multiple adapters are used concurrently. In this paper, we propose Sparse High Rank Adapters (SHiRA), a new paradigm which incurs no inference overhead, enables rapid switching, and significantly reduces concept-loss. Specifically, SHiRA can be trained by directly tuning only 1-2% of the base model weights while leaving others unchanged. This results in a highly sparse adapter which can be switched directly in the fused mode. We further provide theoretical and empirical insights on how high sparsity in SHiRA can aid multi-adapter fusion by reducing concept loss. Our extensive experiments on LVMs and LLMs demonstrate that finetuning only a small fraction of the parameters in the base model significantly outperforms LoRA while enabling both rapid switching and multi-adapter fusion. Finally, we provide a latency- and memory-efficient SHiRA implementation based on Parameter-Efficient Finetuning (PEFT) Library which trains at nearly the same speed as LoRA while consuming up to 16% lower peak GPU memory, thus making SHiRA easy to adopt for practical use cases. To demonstrate rapid switching benefits during inference, we show that loading SHiRA on a base model can be 5x-16x faster than LoRA fusion on a CPU.

• The paper introduces SHiRA, which modifies only 1-2% of base model weights to ensure rapid adapter switching and minimal inference overhead.
• The paper demonstrates that SHiRA achieves up to 2.7% better accuracy on benchmarks while reducing GPU memory by 16%, outperforming traditional LoRA methods.
• The paper provides rigorous theoretical insights, showing that nearly orthogonal adapters minimize concept loss during multi-adapter fusion on edge devices.

Sparse High Rank Adapters: A New Direction in Efficient Model Adaptation

The paper "Sparse High Rank Adapters" presents a novel approach to overcoming some of the significant limitations associated with Low Rank Adaptation (LoRA) methods in the context of mobile and edge deployment of large generative models. The authors introduce Sparse High Rank Adapters (SHiRA), which promise no inference overhead, rapid adapter switching, and reduced concept loss. This essay explores the core contributions of the paper, scrutinizes the theoretical underpinnings, and explores the broader implications of SHiRA for AI research and deployment.

Core Contributions

The primary motivation for SHiRA stems from critical challenges in deploying LoRA-based models on resource-constrained devices. While LoRA allows for efficient parameter finetuning with negligible overhead during inference when fused, its deployment on mobile devices is fraught with latency and memory issues. LoRA modifies the entire weight tensor, which impacts rapid adapter switching and results in up to 30% higher inference latency in unfused mode.

SHiRA ingeniously sidesteps these issues by training a highly sparse subset (1-2%) of the base model weights. This approach allows the model to retain most of the weights unchanged, facilitating rapid switching and minimizing concept loss during multi-adapter fusion. The core contributions of the paper can be summarized as follows:

1. Introduction of SHiRA: A framework that alters only 1-2% of the base model parameters, yielding a highly sparse adapter. This minimal change facilitates rapid switching and maintains high performance across various tasks.
2. Empirical Validation: Extensive experiments on large vision models (LVMs) and LLMs show that SHiRA not only maintains but often exceeds the performance of traditional LoRA methods. For instance, SHiRA achieves up to 2.7% better accuracy on commonsense reasoning benchmarks compared to LoRA.
3. Latency and Memory Efficiency: The implementation of SHiRA based on the Parameter-Efficient Finetuning (PEFT) Library shows that SHiRA consumes 16% lower peak GPU memory and trains at nearly the same speed as LoRA, emphasizing its practicality.

Theoretical Underpinnings

The authors provide strong theoretical support for SHiRA's advantages. They demonstrate that despite high sparsity, SHiRA maintains high rank, which is crucial for preserving the expressive power of adapters. The theoretical analysis includes several key insights:

1. Parameter and Learning Complexity: The complexity of SHiRA is equated to the number of non-zero elements in the adapter. Hence, the learning complexity and the number of parameters remain manageable.
2. Relation to LoRA: SHiRA can be seen as a higher-rank approximation of LoRA. By tuning only a sparse subset of parameters, SHiRA retains high performance while ensuring the changes remain minimal.
3. Orthogonality in Multi-Adapter Fusion: The paper reveals that non-overlapping SHiRA adapters tend to be nearly orthogonal, thereby reducing concept loss when multiple adapters are fused. Metrics such as Adapter Weight Orthogonality Magnitude (AWOM) and Adapter Weight Orthogonality Ratio (AWOR) provide empirical support for these claims.

Practical and Theoretical Implications

The practical implications of SHiRA are significant. Its ability to facilitate rapid switching of adapters without incurring the latency costs associated with traditional methods presents a substantial advancement for deploying advanced AI models on edge devices. This ensures that users can benefit from the powerful capabilities of LLMs and LVMs in resource-limited environments, enhancing privacy and reducing dependency on centralized servers.

Theoretically, SHiRA opens up new avenues for research. Its unique blend of sparsity and high rank challenges conventional wisdom on parameter-efficient finetuning. The notion that finetuning a mere 1-2% of parameters can suffice for high performance mandates a reevaluation of sparsity and adaptation paradigms in machine learning.

Future Developments

The paper sets a foundation for diverse potential future developments in AI research. Notably:

1. Optimal Mask Creation: Identifying the optimal subset of parameters (masks) for various tasks remains an open question. Future research could explore learning-based approaches to generate these masks dynamically.
2. Hardware-Software Co-Design: Implementing SHiRA efficiently on mobile devices might necessitate specialized hardware accelerators capable of handling scattered weight updates, ensuring that the latency benefits observed in theory materialize in practice.
3. Generalizing to Other Domains: While SHiRA has shown promise in language and vision tasks, its applicability to other domains such as speech and multimodal tasks could be a rich field of exploration.

Conclusion

"Sparse High Rank Adapters" represents a significant advancement in the field of efficient model adaptation. By coupling sparsity with high rank, SHiRA addresses critical deployment challenges associated with LoRA, proving effective on both theoretical and empirical fronts. This work not only enhances the feasibility of deploying AI on resource-constrained devices but also paves the way for future research into more efficient, adaptable, and powerful AI systems.