Curvature-Guided LoRA: Steering in the pretrained NTK subspace

Abstract: Parameter-efficient fine-tuning methods such as LoRA enable efficient adaptation of large pretrained models but often fall short of full fine-tuning performance. Existing approaches focus on aligning parameter updates, which only indirectly control model predictions. In this work, we introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs. We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient. Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information. Our method is computationally efficient and avoids explicit second-order matrix construction. Preliminary experiments on standard natural language understanding benchmarks demonstrate improved performance and faster convergence compared to existing LoRA variants.

• The paper introduces a curvature-guided LoRA method that leverages NTK-based curvature estimation to align low-rank updates directly with prediction dynamics.
• It employs a novel second-order initialization using whitened gradients and SVD approximations to closely approximate full fine-tuning performance.
• Empirical results on models such as RoBERTa and T5 demonstrate faster convergence, robustness to hyperparameters, and efficient adaptation in low-data regimes.

Curvature-Guided LoRA: Parameter-Efficient Fine-Tuning via NTK Subspace Steering

Overview

The paper "Curvature-Guided LoRA: Steering in the pretrained NTK subspace" (2603.29824) introduces Curvature-Guided LoRA (CG-LoRA), a novel parameter-efficient fine-tuning (PEFT) scheme for large pretrained models. The approach addresses shortcomings of existing LoRA variants by focusing on prediction alignment in the function space rather than parameter space, employing a second-order, curvature-aware methodology grounded in the neural tangent kernel (NTK) regime. CG-LoRA leverages local loss landscape curvature to select and scale low-rank adaptation directions, enabling more effective and computationally efficient fine-tuning that closely approximates full fine-tuning performance.

Prediction Alignment: Function-Space Objective

Traditional LoRA methods restrict parameter updates to low-rank subspaces, drastically reducing compute and memory demands. However, they typically optimize for alignment in parameter space, which is not always predictive of output alignment due to nonlinearity and the lack of explicit function-space control. This paper formalizes the prediction alignment problem, seeking to minimize the distance between the predictors obtained via LoRA-based PEFT and full fine-tuning at the output level. In the NTK regime, this translates to minimizing projection differences onto the Jacobian row space, ensuring only components that affect model predictions are considered. As a result, the CG-LoRA initialization aligns low-rank updates directly with the directions most influential for prediction dynamics.

Curvature-Aware Initialization

The core innovation is leveraging local curvature—captured through the NTK linearization of the model and approximated via Kronecker-factored curvature (K-FAC)—to inform the choice of adaptation directions. The algorithm computes a whitened loss gradient, projecting it onto the dominant singular subspaces of the curvature matrix. The optimal low-rank updates correspond to a Newton-like step: scaling the gradient by the inverse square root of the curvature (whitening), and taking the best rank-$r$ approximation via singular value decomposition. The subspaces are parameterized by the spectrum and singular vectors of the whitened gradient, exploiting second-order structure to maximize expressive power within the low-rank constraint.

(Figure 1)

Figure 1: Evaluation accuracy of finetuned RoBERTa-base on CoLA w.r.t learning rate.

CG-LoRA avoids explicit construction of large curvature matrices and instead uses randomized algorithms and sketching techniques to efficiently estimate the SVDs and whitening transformations. The method also accommodates balanced initialization and scaling, as shown in the algorithmic details and theoretical derivations.

Algorithmic Implementation and Complexity

CG-LoRA consists of:

• Estimating input/output curvature subspaces per layer using sketched power iterations and QR decompositions.
• Computing the whitened gradient and its rank-$r$ SVD.
• Formulating balanced, scaled low-rank initializations for $A_0$ and $B_0$ (adapter parameters).
• Optionally, optimizing scaling and initial offset for improved robustness.

The complexity analysis demonstrates minimal overhead relative to standard LoRA variants, with negligible additional computational or memory costs when sketching and randomization are used. Practical implementations require only a few backward passes and avoid storing full gradients or curvature matrices.

Numerical Results and Empirical Validation

Experiments on RoBERTa-base and T5-base models, fine-tuning on GLUE benchmarks, reveal that CG-LoRA consistently closes the gap with full fine-tuning, outperforming LoRA-GA, LoRA-One, and rsLoRA on classification and language understanding tasks. CG-LoRA delivers:

• Higher mean accuracy and lower variance across seeds and learning rates, particularly in low-data regimes (CoLA, MRPC).
• Faster convergence during training, especially with noisy gradient initialization.
• Robustness to hyperparameter selection, benefiting both shifted and non-shifted weight initializations.

(Figure 2)

Figure 2: Performance of finetuned T5-base on CoLA.

These results confirm the efficacy of curvature-guided prediction alignment and the theoretical claim that second-order, function-space informed initialization yields superior outcomes.

Practical and Theoretical Implications

Curvature-guided prediction alignment represents a shift in PEFT methodology, emphasizing model outputs rather than parameter changes. The theoretical links between NTK spectrum, loss landscape geometry, and expressive power of low-rank updates provide foundational insights for:

• Layer/adapter selection strategies based on NTK minimum eigenvalues.
• Design of rank-selection and scaling rules grounded in curvature statistics.
• Extensions to architectures beyond the NTK regime, potentially incorporating empirical Fisher or other curvature estimators.

CG-LoRA's computational and memory efficiency renders it suitable for large-scale models deployed in resource-constrained settings, facilitating rapid adaptation to new tasks without significant retraining overhead.

Outlook: Future Directions

Potential developments include:

• Extensive benchmarking across broader model families and tasks (including generative LLMs).
• Refined curvature estimation and adaptive approximation strategies for improved generality.
• Exploration of hybrid schemes combining function-space guidance with adaptive spectral or dynamic rank selection.

The approach may also inform theoretical studies on generalization guarantees and optimization landscape characterization for PEFT under strong nonlinearity.

Conclusion

CG-LoRA advances parameter-efficient fine-tuning by introducing a prediction alignment framework rooted in NTK-based curvature guidance. Empirical results validate its superior performance and robustness over baseline LoRA variants, with minimal computational cost. The method underscores the critical role of function-space alignment and second-order information in bridging the gap to full fine-tuning, and opens avenues for principled algorithmic design in PEFT and model adaptation.