LoRA Gradients: Efficient Fine-Tuning

• LoRA gradients are low-rank adaptive updates that freeze base model weights and introduce trainable matrices for efficient fine-tuning.
• They compute gradients in a restricted subspace, reducing computational cost and memory requirements while retaining fine-tuning effectiveness.
• Enhanced variants like LoRA-Pro, AltLoRA, and ALLoRA optimize convergence and expressivity, sometimes outperforming full-model training under fixed budgets.

Low-Rank Adaptation (LoRA) gradients form the mathematical and algorithmic backbone of parameter-efficient fine-tuning for large pre-trained neural networks. LoRA freezes the main weights and introduces a low-rank, trainable update; gradient computation and optimization methods for these low-rank adapters have become a focal point for both theoretical analysis and practical innovation. Recent work has systematized and generalized LoRA gradient updates, clarified their relation to full-model fine-tuning, introduced enhancements for expressivity and convergence, and illuminated their computational limits and optimization landscape.

1. Standard LoRA Gradient Formulation

Given a frozen, pre-trained weight matrix $W_0\in\mathbb{R}^{d\times k}$, LoRA introduces an additive low-rank update parameterized as

$W = W_0 + \frac{\alpha}{r}\, B A,$

where $A\in\mathbb{R}^{r\times k}$, $B\in\mathbb{R}^{d\times r}$ are trainable, $r\ll \min(d, k)$ is the rank, and $\alpha>0$ is a scaling factor. For input $x$, the adapted layer computes:

$h = W x = W_0 x + \frac{\alpha}{r} B (A x).$

Gradient computation proceeds as follows, using the chain rule:

• $\Delta W = (\alpha/r) B A$
• For a scalar loss $\mathcal{L}$,

$$\frac{\partial \mathcal{L}}{\partial A} = B^\top \frac{\partial \mathcal{L}}{\partial \Delta W}, \quad \frac{\partial \mathcal{L}}{\partial B} = \frac{\partial \mathcal{L}}{\partial \Delta W} A^\top.$$

Depending on implementation, $A$ and $B$ are updated via SGD or AdamW. This formulation underpins the majority of existing LoRA-variants and their theoretical analyses (Xu et al., 3 Dec 2025, Huang et al., 13 Oct 2024, Wang et al., 25 Jul 2024, Zhang et al., 4 Feb 2024).

2. Theoretical Structure: Gradient Subspace and Expressivity

The LoRA adapter restricts updates to the tensor subspace spanned by products of the column space of $B$ and row space of $A$. Specifically, LoRA fine-tuning is exactly equivalent to full parameter fine-tuning, but with the weight update projected into a low-rank subspace:

$\tilde{g} = s B\, (\nabla_A) + s (\nabla_B) A,$

where $s = \alpha/r$, and $\nabla_A, \nabla_B$ are the gradients w.r.t. $A$ and $B$. For the canonical backward pass:

$$\nabla_A = s B^\top g, \quad \nabla_B = s g A^\top, \quad g = \frac{\partial \mathcal{L}}{\partial W}.$$

This subspace restriction is central to both LoRA's efficiency and its main limitation: the inability to follow components of the full gradient that lie outside the current low-rank approximation. This constraint motivates advanced gradient manipulation techniques to bridge the gap to full fine-tuning (Wang et al., 25 Jul 2024, Yu et al., 18 May 2025).

3. Gradient Enhancement and Approximation Schemes

Several lines of research have formalized and enhanced LoRA gradient updates:

• LoRA-Pro equivalent-gradient matching: Adjusts $\nabla_A,\nabla_B$ at each step to optimally approximate the full gradient in the Frobenius norm, using closed-form solutions derived from the least-squares problem

$$\min_{\nabla_A, \nabla_B} \| s B \nabla_A + s \nabla_B A - g \|_F^2.$$

The solution involves projection operators and resolves non-uniqueness via the Sylvester equation, ensuring both descent and optimal subspace alignment. This approach narrows the empirical performance gap to full fine-tuning (Wang et al., 25 Jul 2024).

• AltLoRA alternating projections: Instead of a single joint solution, alternately aligns A and B to the current full-model gradient by:
  1. Fixing $B$, updating $A$ via

  $$\widetilde{\nabla}_A = \frac{1}{s} (B^\top B)^{-1} B^\top \nabla_{W} \mathcal{L}, \quad A \leftarrow A - \eta \widetilde{\nabla}_A,$$

2. Then fixing $A$ and updating $B$ similarly. This yields iterative best projections onto the low-rank subspace and integrates momentum using only $O((d+k)r)$ memory (Yu et al., 18 May 2025).

• Riemannian preconditioning: By endowing $(A,B)$ with a Riemannian metric, the gradients are rescaled as

$$\mathrm{grad}_A\, \mathcal{L} = \nabla_A \mathcal{L} (B^\top B)^{-1}, \quad \mathrm{grad}_B\, \mathcal{L} = \nabla_B \mathcal{L} (A^\top A)^{-1},$$

leading to better-conditioned steps and increased robustness to hyperparameters, especially for small LoRA ranks (Zhang et al., 4 Feb 2024).

• Adaptive learning rates (ALLoRA): Removes the dependency on scaling factors and dropout by introducing a per-row coefficient

$$c_i = \frac{1}{\sqrt{ \| (A B)_{i, \cdot} \|_2 + 1 / \eta^2 }},$$

which is applied to all gradient computations for $A,B$. This mechanism provably accelerates LoRA's escape from poor initializations and acts as an implicit regularizer (Huang et al., 13 Oct 2024).

• Perturbation and stability (LoRA-GGPO): Introduces targeted, gradient-informed perturbations to the optimizer, with per-row covariance scaling, providing the regularizing effect of sharpness-aware minimization (SAM) in a single step. Empirically, this strategy suppresses gradient spikes and double-descent phenomena (Chang et al., 20 Feb 2025).

4. Specialized Gradient Designs: Magnitude-Direction Factorization and Conditional Updates

Recent work has introduced more expressive adapter updates by decomposing the LoRA perturbation into magnitude and direction components. Notably, Dual LoRA parameterizes the update as

$$\Delta W = \frac{\alpha}{\sqrt{r_1 r_2}} \left[ \mathrm{ReLU}(B A) \odot \mathrm{Sign}(D C) \right],$$

with two low-rank groups: $(A,B)$ for magnitudes (non-negative, gate-able by ReLU) and $(C,D)$ for directions (binary, via Sign and straight-through estimator). Gradient back-propagation through these nonlinearities is handled via careful application of indicator masks and gradient clipping:

$\frac{\partial \mathcal{L}}{\partial A} = \frac{\alpha}{\sqrt{r_1 r_2}} B^\top \left ( \frac{\partial \mathcal{L}}{\partial \Delta W} \odot \mathrm{Sign}(DC) \odot \mathbbm{1}_{BA > 0} \right ),$

with analogous rules for $B, C, D$. This design increases the effective expressivity (often empirically yielding full-rank updates) and allows for conditional freezing of irrelevant parameters (Xu et al., 3 Dec 2025).

5. Gradient Initialization Strategies and Data-Awareness

The choice of adapter initialization strongly affects gradient flow and optimization trajectories:

• Gradient-aligned initialization (LoRA-GA): Sets $A_{init}$ and $B_{init}$ so that the initial low-rank update aligns with the full fine-tuning gradient:

$$A_{init},B_{init} \text{ chosen to minimize } \| \eta^2 (\partial_W A_{init}^\top A_{init} + B_{init} B_{init}^\top \partial_W) - \zeta \partial_W \|_F$$

using leading singular vectors of the full gradient. This substantially accelerates convergence and improves final performance (Wang et al., 6 Jul 2024).

• Data-aware initialization (LoRA-DA): Uses Fisher information and gradient statistics from a small target-domain dataset to optimally pick the initial adapter subspace $A_0$ that minimizes a quadratic error form involving the Fisher and the projected full-model parameter shift, leading to superior performance especially at small ranks (Zhang et al., 28 Oct 2025).
• Gradient-driven rank and initialization selection (GoRA): Accumulates gradients over a calibration set to determine per-layer importance, allocates the total parameter budget proportionally, and initializes $B$ in the subspace of $A$ to compress the accumulated gradient, with scale correction to match expected SGD behavior. This leads to prompt learning of salient features and superior downstream accuracy (He et al., 13 Feb 2025).

6. Computational Structure and Complexity of LoRA Gradients

LoRA's gradient computation admits several algorithmic accelerations:

• Exact computation for single layers is $O(d k r)$ per update, but may dominate when $d,k$ are large.
• Hierarchical low-rank approximations and sparsified matrix products can reduce complexity to nearly linear time in transformer sequence length, under certain norm bounds (Hu et al., 5 Jun 2024).
• Approximated matrix multiplication and double-layer decomposition (e.g., CE-LoRA) significantly reduce backward FLOPs by focusing computation only on critical rows/columns (2502.01378).

The existence of such efficient algorithms is conditioned on matrix product norms; above a threshold (e.g., $\Gamma \gtrsim \sqrt{\log L}$ for sequence length $L$), no subquadratic algorithm is believed to exist under SETH (Hu et al., 5 Jun 2024).

7. Empirical Impact and Theoretical Guarantees

Empirical studies consistently show that advanced LoRA gradient designs—whether via improved projection/approximation, adaptive learning rates, magnitude-direction disentanglement, or gradient-driven initialization—yield faster convergence, robustness to hyperparameters, enhanced expressivity, and at times even outpace full fine-tuning under fixed parameter budgets.

Theoretical analyses provide convergence guarantees for low-rank SGD dynamics (e.g., in the student-teacher problem, LoRA fine-tuning matches the ground truth in $dk^{O(1)}$ iterations, independently of kernel spectral properties) (Dayi et al., 23 Nov 2024). Projected or alternating-projection schemes deliver optimal or provably stable low-rank approximations of full gradients (Wang et al., 25 Jul 2024, Yu et al., 18 May 2025).

Table: Selected Gradient Update Recipes in LoRA and Variants

Variant	Gradient Formula for $A$	Special Techniques
Standard LoRA	$$B^\top \frac{\partial \mathcal{L}}{\partial \Delta W}$$	None
LoRA-Pro	Closed-form solution to $\min \| sB \nabla_A + s \nabla_B A - g \|_F^2$	Optimal low-rank projection
AltLoRA	$(B^\top B)^{-1} B^\top \nabla_{W} \mathcal{L}$	Alternating projection, momentum
Riemannian LoRA	$\nabla_A \mathcal{L} (B^\top B)^{-1}$	Preconditioned step, Riemannian metric
Dual LoRA	$B^\top(\nabla \mathcal{L} \odot \mathrm{Sign}(DC) \odot \mathbbm{1}_{BA > 0})$	ReLU/sign factorization, STE
ALLoRA	Row-wise scaling: $$c_i B^\top \frac{\partial \mathcal{L}}{\partial \Delta W}$$	Adaptive learning rate, regularization

All formulas are as detailed in the corresponding cited works.

---

A plausible implication is that future LoRA-type PEFT methods will likely continue to build on these variants of gradient optimization—particularly exploiting data-driven subspaces, non-Euclidean preconditioning, robust initialization, and adaptive gradient control—to further close the expressivity and convergence gap with full-model fine-tuning while maintaining strict parameter and computational budgets.