Rethinking the Rank Threshold for LoRA Fine-Tuning

Abstract: A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition $r(r+1)/2 > KN$ on the LoRA rank $r$ for the absence of spurious local minima under squared-error loss, prescribing $r \geq 12$ on canonical few-shot RoBERTa setups. The condition is stated for general output dimension $K$, so its sharpness in any particular regime, and its practical implication for the cross-entropy loss actually used in fine-tuning, are open. We give three results that together reduce the prescribed rank to $r = 1$ for binary classification in this regime. First, replacing the symmetric Sard-form count with the non-symmetric LoRA manifold dimension yields a strictly weaker capacity requirement, $r(m+n) - r^2 > C^* \cdot KN$ with $C^* \approx 1.35$ under Gaussian-iid features, satisfied at $r = 1$ on canonical setups. Second, in the cross-entropy setting the Polyak--Łojasiewicz inequality removes the rank threshold entirely. Third, a Rademacher-complexity bound predicts rank-one variance optimality precisely when the bias term is saturated, which is the case for binary classification but not for $K > 2$. Empirically, across four GLUE-style binary tasks, three encoder architectures, and at scale on RoBERTa-large, rank one is competitive with the existing prescription $r = 12$; on multi-class MNLI the optimal rank shifts above one, also as predicted. The binary-regime guarantees are conditional on standard NTK assumptions; the multi-class extension is left to future work.

• The paper refines the rank threshold for LoRA fine-tuning by replacing symmetric counting with a non-symmetric condition, showing rank-one suffices under NTK for binary tasks.
• It demonstrates that for cross-entropy loss, the Polyak–Łojasiewicz inequality holds empirically, ensuring global convergence at any rank in binary classification.
• For multi-class settings, empirical results reveal that higher ranks help balance the bias–variance tradeoff, confirming the need for task-specific tuning.

Sharpened Rank Thresholds for LoRA Fine-Tuning in the NTK Regime

Background and Motivation

Low-Rank Adaptation (LoRA) is widely adopted for parameter-efficient fine-tuning of transformer models. The LoRA protocol enforces weight updates of the form $\Delta W = uv^\top$, with matrices $u \in \mathbb{R}^{m \times r}$ and $v \in \mathbb{R}^{n \times r}$, reducing the number of tunable parameters from $mn$ to $r(m+n)$. The prescribed choice of rank $r$ has practical implications for both expressivity and computational efficiency. Existing theoretical analyses, notably in the neural tangent kernel (NTK) regime, set a conservative sufficient condition on $r$ to guarantee spurious-free optimization: for output dimension $K$ and $N$ training samples, $r(r+1)/2 > KN$ is required under squared-error loss, translating to $u \in \mathbb{R}^{m \times r}$0 in canonical RoBERTa-base binary fine-tuning ($u \in \mathbb{R}^{m \times r}$1, $u \in \mathbb{R}^{m \times r}$2) [jang2024lora]. However, the necessity and optimality of this threshold, especially under the cross-entropy loss commonly used in practice, remained unresolved.

Theoretical Advances: Loss-Dependent Rank Thresholds

This paper provides a rigorous refinement of the rank threshold for LoRA fine-tuning in the NTK regime, challenging the conservative $u \in \mathbb{R}^{m \times r}$3 prescription and aligning theoretical guarantees with practical loss functions.

Non-Symmetric Sard Counting

The first theoretical contribution replaces the symmetric dimension count used previously with the correct non-symmetric dimension of the LoRA parameter manifold, yielding $u \in \mathbb{R}^{m \times r}$4, where $u \in \mathbb{R}^{m \times r}$5 under Gaussian-iid features. For typical transformer settings (e.g., RoBERTa-base, $u \in \mathbb{R}^{m \times r}$6), this condition is already satisfied at $u \in \mathbb{R}^{m \times r}$7.

No Rank Threshold for Cross-Entropy

For practical cross-entropy loss, the Polyak–Łojasiewicz (PL) inequality holds empirically and, when satisfied, removes the rank threshold entirely: any rank $u \in \mathbb{R}^{m \times r}$8 suffices to guarantee global convergence in binary classification. This reveals a strong loss-dependence in the behavior of LoRA fine-tuning.

Rank–Generalization Tradeoff and Multi-Class Case

A Rademacher complexity analysis makes explicit the bias–variance tradeoff. For binary classification, the bias for $u \in \mathbb{R}^{m \times r}$9 saturates, and the variance term increases with $v \in \mathbb{R}^{n \times r}$0, making rank-one adaptation variance-optimal. For multi-class tasks ($v \in \mathbb{R}^{n \times r}$1), the bias gap grows, and the optimal rank shifts above one, as confirmed empirically.

Empirical Evidence: Robustness and Performance Across Tasks

Extensive empirical evaluations were conducted across GLUE-style binary tasks (SST-2, QNLI, MR, QQP), encoder architectures (BERT-base, DistilBERT, RoBERTa-base, RoBERTa-large), and at various scales. Results demonstrate that:

• On binary tasks, rank-one ($v \in \mathbb{R}^{n \times r}$2) matches or outperforms the existing recommended $v \in \mathbb{R}^{n \times r}$3, with negligible differences in test accuracy and generalization.
• Higher ranks do not provide generalization advantages and often induce mild overfitting signatures.
• Robustness sweeps over training size, LoRA layer index, and encoder architecture confirm the persistence of rank-one sufficiency.
• In multi-class settings (MNLI, $v \in \mathbb{R}^{n \times r}$4), rank-one adaptation underperforms higher ranks by 3-5 percentage points, consistent with the theoretical prediction that bias dominates for larger $v \in \mathbb{R}^{n \times r}$5.

  Figure 1: Test accuracy as a function of LoRA rank for RoBERTa-base on SST-2 and other GLUE-style binary tasks, showing rank-one is competitive with the existing threshold.

  

  Figure 2: Robustness of test accuracy across train sizes and LoRA layer indices, demonstrating the preservation of the rank-one sufficiency across task and architectural axes.

  

  Figure 3: SST-2 training and test accuracy in a constrained NTK regime; higher ranks clearly improve both fit and generalization in low-capacity settings.

Practical and Theoretical Implications

Practical

• Fine-tuning transformer models for binary classification tasks can achieve optimal or near-optimal performance with rank-one LoRA adaptation, with substantial memory and compute savings vs. recommended defaults.
• In multi-class regimes, the optimal rank is task-dependent and can be empirically tuned above one.

Theoretical

• The refinement from $v \in \mathbb{R}^{n \times r}$6 to $v \in \mathbb{R}^{n \times r}$7 (with $v \in \mathbb{R}^{n \times r}$8) makes the sufficient condition for spurious-free LoRA optimization both tighter and more relevant.
• The PL property in cross-entropy loss under NTK dynamics implies global convergence at all ranks, conditional on network-wide overparameterization.

Future Directions

• Formal PL proofs for LoRA in the NTK regime and theoretical treatment for real (non-Gaussian) NTK Jacobian distributions are required for complete generality.
• Sharpening the rank threshold for multi-class tasks as a function of $v \in \mathbb{R}^{n \times r}$9 remains open.
• Extending the analysis to settings with intrinsic low-rank structure in data and to other parameter-efficient adaptation regimes could further minimize required ranks.

Conclusion

The paper establishes a loss-dependent, tightly quantified rank threshold for LoRA fine-tuning in the NTK regime, demonstrating that past conservative prescriptions significantly overestimate necessary rank in canonical transformer setups. For binary classification, rank-one adaptation suffices, freeing practitioners to prioritize parameter efficiency. For multi-class classification, the bias–variance tradeoff shifts optimal rank higher. The results both revise theoretical understanding and offer direct practical guidance for the design of parameter-efficient fine-tuning routines in LLMs.

---

References: See arXiv paper "Rethinking the Rank Threshold for LoRA Fine-Tuning" (2605.03724) for full bibliographic details.