PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA

Abstract: With the rapid scaling of LLMs, serving numerous low-rank adaptations (LoRAs) concurrently has become increasingly impractical, leading to unaffordable costs and necessitating more parameter-efficient finetuning methods. In this work, we introduce Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA), an intra-layer sharing mechanism comprising four essential components: broadcast reduction, rotation enhancement, partially-sharing refinement, and rectified initialization strategy. As a superset of LoRA, PRoLoRA retains its advantages, and effectively circumvent the drawbacks of peer parameter-sharing methods with superior model capacity, practical feasibility, and broad applicability. Empirical experiments demonstrate the remarkably higher parameter efficiency of PRoLoRA in both specific parameter budget and performance target scenarios, and its scalability to larger LLMs. Notably, with one time less trainable parameters, PRoLoRA still outperforms LoRA on multiple instruction tuning datasets. Subsequently, an ablation study is conducted to validate the necessity of individual components and highlight the superiority of PRoLoRA over three potential variants. Hopefully, the conspicuously higher parameter efficiency can establish PRoLoRA as a resource-friendly alternative to LoRA.

• The paper introduces PRoLoRA, demonstrating double the parameter efficiency over traditional LoRA methods while maintaining superior performance.
• The method employs innovative techniques such as broadcast reduction, rotation enhancement, and partially-sharing refinement to optimize low-rank adaptations.
• Empirical results on LLaMA2 models validate PRoLoRA's scalability and resource efficiency, making it ideal for personalized and multitask model customization.

PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA

Introduction

The paper introduces Partially Rotation-enhanced Low-Rank Adaptation (PRoLoRA), an advanced method for parameter-efficient finetuning of LLMs like LLaMA2 and GPT-3.5 Turbo. Finetuning LLMs across multiple domains incurs substantial resource costs, especially when employing multiple LoRA instances for personalized or multitasking purposes. The proposed PRoLoRA method addresses these challenges by incorporating intra-layer sharing mechanisms that include broadcast reduction, rotation enhancement, partially-sharing refinement, and a rectified initialization strategy.

PRoLoRA builds upon the Low-Rank Adaptation (LoRA) approach in several ways, enhancing both capacity and practicality while maintaining broad applicability. The empirical results demonstrate superior performance with significantly fewer trainable parameters, making PRoLoRA an ideal candidate for resource-efficient model customization.

Method

PRoLoRA enhances parameter efficiency through careful reparameterization of low-rank matrices and leveraging intra-layer sharing mechanisms. The main components are detailed as follows:

Broadcast Reduction

This process involves partitioning low-rank matrices into smaller chunks and broadcasting the initial chunk across multiple segments. As illustrated in (Figure 1), this approach effectively reduces the number of trainable parameters by sharing them across the expanded matrices, resulting in higher parameter efficiency.

Figure 1: Illustration of the original LoRA, our proposed PRoLoRA, and their intermediate states (i.e., CLoRA and RoLoRA). Here we set the rank $r$, unshared rank $u$, sharing rates $m$ and $n$ of the $\mathbf{A}$ and $\mathbf{B}$ matrices to be 4, 1, 2, and 3, respectively.

Rotation Enhancement

To address issues related to reduced expressiveness due to simple replication, PRoLoRA introduces rotation enhancement by applying a cost-free rotation to differentiate broadcast chunks. This increases the representational capacity without adding extra parameters and refines the weight difference matrix.

Partially-Sharing Refinement

Incorporating unshared parameters into the low-rank matrices allows for more refined matrix expressiveness and improved parameter efficiency. This mechanism differentiates PRoLoRA from CLoRA and RoLoRA methods by providing enhanced adaptability and reduced block-wise symmetry in matrices.

Rectified Initialization Strategy

PRoLoRA adopts a rectified Kaiming uniform initialization for shared chunks, ensuring unified bounds and facilitating effective optimization during training. This careful initialization aids in maintaining superior performance when deploying the proposed sharing mechanisms.

Results and Analysis

Experiments using the LLaMA2-7B and 13B models demonstrate PRoLoRA's scalable efficiency. The model consistently outperformed LoRA-constrained parameter budgets, achieving double the parameter efficiency in practical settings. With identical budget constraints, PRoLoRA excelled in average performance metrics when compared to LoRA, VeRA, and Tied LoRA baselines.

Unshared Rank and Learning Rate

PRoLoRA performance analysis under varied unshared ranks and learning rates showed consistent superiority over LoRA and other alternatives as depicted in (Figure 2).

Figure 2: Performance of PRoLoRA with the rank of 32 with respect to unshared ranks and learning rates given a specific parameter budget on the LLaMA2-7B model and BBH benchmark.

Scalability to Larger Models

PRoLoRA maintains efficient performance with larger models such as LLaMA2-13B, highlighting its adaptability for scaling without compromising resource efficiency or performance.

Conclusion

PRoLoRA offers a comprehensive framework for parameter-efficient adaptation of LLMs through innovative intra-layer sharing mechanisms. Its application significantly alleviates memory burdens and enhances scalability while providing improved performance. PRoLoRA stands as a resource-friendly alternative to existing methods and possesses potential for further integration with other sharing frameworks. The paper suggests promising directions for future developments in efficient model customization, particularly in personalized and multitask scenarios.