Parameter-Efficient Fine-Tuning via Circular Convolution (2407.19342v2)

Abstract: Low-Rank Adaptation (LoRA) has gained popularity for fine-tuning large foundation models, leveraging low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ to represent weight changes (i.e., $\Delta \mathbf{W} = \mathbf{B} \mathbf{A}$). This method reduces trainable parameters and mitigates heavy memory consumption associated with full delta matrices by sequentially multiplying $\mathbf{A}$ and $\mathbf{B}$ with the activation. Despite its success, the intrinsic low-rank characteristic may limit its performance. Although several variants have been proposed to address this issue, they often overlook the crucial computational and memory efficiency brought by LoRA. In this paper, we propose Circular Convolution Adaptation (C$^3$A), which not only achieves high-rank adaptation with enhanced performance but also excels in both computational power and memory utilization. Extensive experiments demonstrate that C$^3$A consistently outperforms LoRA and its variants across various fine-tuning tasks.

Parameter-Efficient Fine-Tuning via Circular Convolution

This paper introduces Circular Convolution Adaptation (CCA), a novel methodology for parameter-efficient fine-tuning (PEFT) in deep learning models. The approach capitalizes on the properties of circular convolution to achieve high-rank adaptation while maintaining efficient computational and memory footprints. The authors position CCA as a more adaptable and efficient alternative to established methods like Low-Rank Adaptation (LoRA) and Vector Random Matrix Adaptation (VeRA).

The backdrop for this research is the challenge posed by fine-tuning large foundational models (LFMs) which come with substantial parameters and computational requirements. Traditional PEFT methods, such as LoRA, mitigate the parameter overhead by using low-rank matrices to represent changes, leading to more feasible fine-tuning. However, these methods are limited in adaptability due to their inherent low-rank structures.

CCA introduces circular convolution as an alternative, leveraging its ability to be represented through circulant matrices. This representation allows for adaptable rank configurations without being linearly constrained by the parameter count. By incorporating Fast Fourier Transform (FFT), CCA ensures computational and memory efficiency. The paper thoroughly compares CCA against LoRA and VeRA, demonstrating that CCA achieves superior performance and efficiency across a range of tasks and models.

Key results from the experiments underscore CCA's advantages:

• Synthetic Data Experiments: CCA exhibited superior expressiveness compared to LoRA under the same parameter constraints, demonstrating its potential for more accurate modeling.
• Natural Language Understanding (GLUE benchmark): CCA consistently demonstrated comparable or superior performance to state-of-the-art PEFT approaches with fewer parameters and reduced memory usage, affirming its efficacy in natural language processing tasks.
• Instruction Tuning: When applied to instruction tuning tasks using LLaMA models, CCA surpassed LoRA, achieving higher accuracy with less than half the parameter count required by LoRA.
• Image Classification: Using Vision Transformers (ViT), CCA showed substantial improvements in classification tasks, maintaining competitive accuracy while halving the parameter count compared to LoRA.

In terms of theoretical implications, CCA's use of circular convolution and its rank flexibility affords it unique advantages. The disentanglement of rank from parameter constraints enables superior adaptation, particularly in data-limited contexts where inductive biases can guide optimization effectively. This aspect makes it especially promising for fine-tuning large foundational models, where traditional methods struggle with scalability without compromising on performance.

The practical implications of CCA are significant. Its efficiency in both computation and memory usage makes it suitable for deployment in resource-constrained environments common in real-world applications. Additionally, CCA presents a compelling approach for future developments in AI, particularly as the community seeks more scalable and efficient methods for adapting large models to specific tasks.

In conclusion, the adoption of circular convolution within PEFT frameworks presents a promising route for overcoming the limitations of existing methods like LoRA. By efficiently managing parameter count and maintaining high adaptability, CCA stands out as a robust method for fine-tuning LFMs, paving the way for advancements in model scalability and application diversity. Future research could explore further optimizations within the Fourier domain and potential applications of CCA in other machine learning paradigms beyond fine-tuning.