On Fairness of Low-Rank Adaptation of Large Models (2405.17512v2)

Abstract: Low-rank adaptation of large models, particularly LoRA, has gained traction due to its computational efficiency. This efficiency, contrasted with the prohibitive costs of full-model fine-tuning, means that practitioners often turn to LoRA and sometimes without a complete understanding of its ramifications. In this study, we focus on fairness and ask whether LoRA has an unexamined impact on utility, calibration, and resistance to membership inference across different subgroups (e.g., genders, races, religions) compared to a full-model fine-tuning baseline. We present extensive experiments across vision and language domains and across classification and generation tasks using ViT-Base, Swin-v2-Large, Llama-2 7B, and Mistral 7B. Intriguingly, experiments suggest that while one can isolate cases where LoRA exacerbates model bias across subgroups, the pattern is inconsistent -- in many cases, LoRA has equivalent or even improved fairness compared to the base model or its full fine-tuning baseline. We also examine the complications of evaluating fine-tuning fairness relating to task design and model token bias, calling for more careful fairness evaluations in future work.

• The paper demonstrates that LoRA does not consistently worsen subgroup fairness compared to full fine-tuning, with fairness largely dependent on the base model's quality.
• The study finds that LoRA achieves comparable calibration and provides resistance to membership inference attacks while offering computational efficiency.
• Experiments across vision and language tasks reveal that LoRA's parameter efficiency does not compromise fairness, paving the way for more equitable adaptations.

On Fairness of Low-Rank Adaptation of Large Models

Abstract Overview

This paper investigates the fairness implications of low-rank adaptation (LoRA) in large models, compared to full-model fine-tuning, across various domains and tasks. LoRA has gained popularity due to its computational efficiency over full-model fine-tuning. However, its effects on fairness metrics such as utility, calibration, and resistance to membership inference attacks (MIA) are not well-understood. The authors conduct extensive empirical analyses using models like ViT-Base, Swin-v2-Large, Llama-2 7B, and Mistral 7B on tasks spanning vision and language domains. They analyze LoRA's fairness performance and its impact on different subgroups defined by attributes such as gender, race, and religion.

Introduction to LoRA and Fairness

Parameter-efficient fine-tuning methods like LoRA have become essential due to the prohibitive costs associated with full-model fine-tuning of large models. LoRA achieves fine-tuning through low-rank adaptations of weight matrices, preserving the pre-trained weights while only updating a small set of parameters. Despite its efficiency and effectiveness, this technique's implications on fairness and robustness remain underexplored. This paper aims to fill this gap by systematically evaluating the effects of LoRA on subgroup fairness across distinct tasks and models.

Experimental Setup and Results

The experiments involve fine-tuning pre-trained models on classification and generative tasks. These are evaluated on metrics of accuracy, calibration, resistance to MIA, and gender bias. Key models used include ViT-Base and Swin-v2-Large for vision tasks, and Llama-2 7B and Mistral 7B for language tasks. The datasets span hatespeech detection, face image classification, machine translation, and text generation tasks.

Key Findings:

1. Subgroup Fairness:
   • The analysis shows no consistent evidence that LoRA exacerbates subgroup fairness compared to full fine-tuning. While isolated cases of LoRA worsening fairness exist, these are sporadic and often depend on the quality of the underlying pre-trained model.
   • Fairness depends on the choice of the base model, with more powerful models generally yielding better fairness results when fine-tuned with LoRA.
2. Calibration:
   • LoRA and full fine-tuning exhibit comparable levels of calibration, though LoRA tends to produce slightly overconfident models.
   • The expected calibration error (ECE) remains low across both methods, suggesting reliable probability estimates.
3. Resistance to Membership Inference Attacks:
   • LoRA generally provides resistance to MIAs comparable to full fine-tuning. In some cases, such as Swin-v2-Large on UTK-Face data, LoRA even outperforms full fine-tuning in terms of privacy preservation.
4. Gender Bias in Generative Tasks:
   • Evaluations of gender bias in LLMs using LoRA show no definitive pattern of exacerbating biases compared to full fine-tuning.
   • Both LoRA and fully fine-tuned models can reflect and sometimes reduce the inherent biases present in generative tasks.
5. Effect of LoRA Rank:
   • The rank of the low-rank adaptation in LoRA does not significantly impact subgroup fairness. Both utility and fairness metrics remain stable across different rank configurations.

Discussion and Limitations

The paper concludes that LoRA does not inherently induce unfair outcomes, thus suggesting that its parameter efficiency may be seen as largely beneficial from a fairness perspective. Nonetheless, this does not imply that LoRA is a universally fair method. Token bias in LLMs, for instance, continues to complicate fairness evaluations. This work also acknowledges the complexities of evaluating fairness in generative models, where token biases can obscure genuine model preferences.

Future Directions:

1. Improving Generative Model Fairness Evaluations:
   • Future efforts should focus on developing evaluation methods that minimize token bias and explore biases through semantics and discourse structures.
2. Extending Evaluations:
   • More comprehensive evaluations that account for intersectional subgroup definitions and their fairness implications.
3. Comparative Analysis of Parameter-Efficient Methods:
   • Investigating other parameter-efficient fine-tuning methods to determine if the fairness properties observed with LoRA are consistent across different approaches.

Conclusion

This paper provides an extensive empirical investigation into the fairness properties of LoRA, demonstrating that it does not consistently worsen subgroup fairness. By carefully assessing LoRA across a variety of models, datasets, and fairness metrics, the paper establishes a foundational understanding of the fairness implications of parameter-efficient fine-tuning methods in large models. This contributes to building more equitable machine learning systems while leveraging the computational benefits of techniques like LoRA.