Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (2409.16167v3)

Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning LLMs to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

• The paper introduces the LoRA-LEGO framework to modularly merge LoRAs via rank-wise clustering.
• It employs minimal semantic units and a dual reweighting strategy to optimize parameter scaling.
• Experimental results demonstrate superior performance over traditional merging techniques on various LLM benchmarks.

The paper "Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering" explores an innovative approach to enhancing the adaptability and performance of Low-Rank Adaptation (LoRA) techniques for fine-tuning LLMs. LoRA is valued for its modularity, allowing modifications and enhancements without retraining entire models.

Key Concepts and Innovations

1. Modularity in LoRA: The paper highlights the inherent modularity of LoRA, which has prompted interest in combining different LoRAs to boost the capabilities of LLMs. Traditional methods often handle these combinations in a manner that leads to interference between parameters and reduced performance.
2. Minimal Semantic Units (MSUs): The authors introduce MSUs, which treat each rank within LoRA as an independent and modular unit. These MSUs possess properties like permutation invariance and concatenation-summation equivalence, making them flexible for recombination. This perspective is akin to using LEGO pieces that can be assembled in various configurations.
3. LoRA-LEGO Framework: To optimize LoRA merging, the paper proposes the LoRA-LEGO framework. The framework utilizes a novel rank-wise clustering approach to categorize MSUs from different LoRAs into $k$ clusters. The centroid of each cluster acts as a representative MSU, allowing for the assembly of new LoRAs with a rank equivalent to $k$.
4. Dual Reweighting Strategy: To further refine the merged LoRA, a dual reweighting strategy is implemented. This method optimizes the scaling of the combined parameters, enhancing the overall effectiveness of the merged model.

Experimental Results

The experiments conducted indicate that the LoRA-LEGO framework provides superior performance across various benchmarks when compared to existing LoRA merging techniques. The method's ability to precisely and flexibly combine multiple LoRAs without substantial training requirements offers significant practical advantages.

Implications

This research underscores the potential for more advanced modular techniques in the adaptation of large models, offering greater flexibility and efficiency. The concepts of MSUs and the LoRA-LEGO framework may encourage further innovations in fine-tuning and model merging practices within the field of LLMs.