LoReFT: Low-Rank Linear Subspace Methods

• LoReFT is a framework that uses low-rank subspaces to decouple matrix structure and constraints, enabling parameter-efficient modeling and fine-tuning.
• It employs Riemannian optimization techniques on spectrahedron manifolds for rapid convergence and robust global optimality in high-dimensional settings.
• LoReFT supports fast signal subspace estimation and Bayesian neural adaptation, achieving improved RMSE, OOD detection, and reduced computational cost.

Low-rank Linear Subspace (LoReFT) methods constitute a class of parameter-efficient modeling and fine-tuning techniques utilizing the geometry and algebra of low-rank matrix manifolds and subspaces. These frameworks exploit the observation that, in high-dimensional models and signal/data representations, much of the essential information and generalization capability is concentrated in low-dimensional affine subspaces, often span-efficiently by low-rank factors. LoReFT methods offer a formalism for (a) learning matrices subject to structural subspace constraints; (b) parameterizing and estimating structured low-rank signal subspaces; and (c) embedding neural network adaptations and their uncertainty within compact projected subspaces. The LoReFT approach unifies advances in structured matrix learning, parameter-efficient adaptation, and Bayesian uncertainty quantification via spectral and geometric analysis.

1. Structured Low-rank Matrix Learning and Subspace Decoupling

Structured low-rank matrix learning formalizes the problem of estimating a matrix $X\in\mathbb R^{d\times T}$ subject to both a rank constraint and additional linear constraints $A(X)=b$, often encoding structural priors or signal models. The key innovation, as described by Jawanpuria & Mishra, is a decoupled factorization that separates the low-rank constraint from other structural properties. The canonical problem is

$$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$

where $L$ is a convex loss, $\|\cdot\|_*$ is the nuclear norm, and $A$ is a linear map. Via duality, the optimizer satisfies a representer theorem: $X = \Theta(Z + A^*(s))$, with $\Theta$ a PSD, unit-trace, and typically low-rank matrix. Imposing $\mathrm{rank}(\Theta)\leq r$ leads to the parametrization $\Theta=UU^T$, $A(X)=b$0, and introduces an auxiliary variable $A(X)=b$1. Thus, all feasible $A(X)=b$2 are exactly parameterized as $A(X)=b$3 with $A(X)=b$4, fully decoupling the low-rank representation (via $A(X)=b$5) from loss and constraints (via $A(X)=b$6, $A(X)=b$7) (Jawanpuria et al., 2017).

2. Optimization on Riemannian Spectrahedron Manifolds

The decoupled LoReFT factorization enables efficient nonlinear optimization on matrix manifolds. The problem reduces to the minimization over the spectrahedron manifold

$A(X)=b$8

with the cost function

$A(X)=b$9

Optimization over $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$0 employs the Riemannian conjugate gradient and trust-region methods, with explicit formulas for the Euclidean and Riemannian gradients, tangent and retraction operations. Convergence guarantees stem from manifold optimization theory: CG with Armijo line-search converges to critical points, while trust-region converges globally with possible superlinear rates (Jawanpuria et al., 2017).

A fundamental duality result provides the globality certificate: the primal-dual gap is

$$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$1

where $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$2 is the largest singular value of $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$3. At rank-deficient $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$4, $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$5, certifying global optimality at the attained rank (Jawanpuria et al., 2017).

3. Signal Subspace Parameterization via GLRRs

LoReFT frameworks admit explicit parameterizations for structured low-rank signals, as in the case of Hankel-structured low-rank approximation. For a time series $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$6 and window $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$7, its Hankel trajectory map $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$8 has rank $$\min_{X\in\mathbb R^{d\times T}}\; C L(X) + \frac{1}{2}\|X\|_*^2\quad\text{s.t.}\; A(X)=b$$9 exactly when $L$0 satisfies a generalized linear recurrence relation (GLRR) of order $L$1. Any such $L$2 belongs to an $L$3-dimensional subspace $L$4, where $L$5 encodes convolution by the GLRR coefficients $L$6. One obtains a smooth local parameterization of the variety of all rank-$L$7 trajectories using "boundary" samples of $L$8 and $L$9 free GLRR coefficients (Theorem 2.1, Proposition 2.2 in (Zvonarev et al., 2021)). The tangent space at $\|\cdot\|_*$0 is determined by the kernel of a convolved GLRR, enabling precise geometric reasoning and first-order optimality conditions for estimation (Zvonarev et al., 2021).

4. Fast Subspace Projections and Stable Algorithms

Efficient LoReFT implementations critically depend on fast and stable projection onto low-rank structured subspaces. The projection onto $\|\cdot\|_*$1, subject to a weighted metric $\|\cdot\|_*$2 (with $\|\cdot\|_*$3 possibly banded), reduces to an explicit oblique projector: $\|\cdot\|_*$4 This permits computation of orthonormal bases using circulant embeddings and FFT, achieving complexity $\|\cdot\|_*$5 and greatly improved stability versus classical approaches. The cost and numerical conditioning scale mildly with $\|\cdot\|_*$6 and the AR polynomial root multiplicity. In application, these projectors and parameterizations anchor variable projection Gauss–Newton algorithms, yielding fast, robust solutions for low-rank signal estimation problems (Zvonarev et al., 2021).

5. Parameter-efficient Subspace Adaptation and Bayesian LoReFT

Bayesian Fine-tuning in Projected Subspaces advances LoReFT by embedding Low-Rank Adaptation (LoRA) updates for neural network weights directly into a prescribed low-dimensional affine subspace. After decomposing a pretrained weight $\|\cdot\|_*$7 by

$\|\cdot\|_*$8

all trainable variation is captured by small core matrices $\|\cdot\|_*$9 per layer, with $A$0 fixed (Dubovik et al., 8 May 2026). Vectorizing and stacking across layers, adapted weights are constrained to the affine space

$A$1

where $A$2 is block-diagonal in Kronecker structure, and $A$3 is the number of layers.

A Bayesian posterior is then placed on the collection of $A$4 (parameter vector $A$5), using either diagonal or Kronecker-factored (KFAC) covariances, or low-rank plus diagonal SWAG approximations. The empirical evidence demonstrates that most posterior mass lies in a very low-dimensional subspace: calibration (measured by ECE) and NLL remain stable even for SWAG rank as low as $A$6 on benchmark models, implying that LoReFT can achieve strong uncertainty quantification and OOD detection with fewer trainable parameters than full-model Bayesianization. Table 2 (Dubovik et al., 8 May 2026) shows improved predictive entropy and AUROC for OOD detection compared to standard LoRA-SWAG.

6. Major Applications and Empirical Results

LoReFT methodologies are validated on a spectrum of matrix and signal learning tasks:

• Standard matrix completion: RSLM-TR achieves the lowest RMSE on Netflix, ML10m/ML20m, outperforming APGL, R3MC, RTRMC and others. RSLM-CG is among the fastest first-order solvers (Jawanpuria et al., 2017).
• Robust matrix completion: With $A$7 or $A$8-SVR loss, RSLM outperforms RMC baselines, notably improving OOD sample handling (Jawanpuria et al., 2017).
• Non-negative matrix completion: With non-negativity constraints, RSLM exceeds the test RMSE performance of BMC, BMA across ranks (Jawanpuria et al., 2017).
• Low-rank Hankel learning: RSLM achieves the lowest RMSE and correctly identifies the true minimal order across settings, outperforming SLRA, DADM, and GCG (Jawanpuria et al., 2017); GLRR-based parameterizations underpin efficient estimation (Zvonarev et al., 2021).
• Multi-task feature learning: RSLM matches or improves NMSE over standard MTFL baselines, confirming global optimality at lower ranks as verified via duality gap (Jawanpuria et al., 2017).
• Bayesian neural adaptation: Bayesian LoReFT yields ECE and NLL comparable to or better than standard LoRA, with 5–15× fewer parameters required for posterior models, and sharp improvements in OOD detection via predictive entropy (Dubovik et al., 8 May 2026).

7. Strengths, Limitations, and Extensions

LoReFT frameworks offer several demonstrated advantages:

• Decoupling of low-rank representation, structural constraints, and variable loss forms enables unified, flexible modeling.
• Avoidance of expensive SVDs or large-matrix eigendecompositions; many inner subproblems admit efficient closed-form or per-column solutions.
• Riemannian optimization ensures rapid local convergence and global stationarity, supporting large-scale high-dimensional data.
• Stability and computational efficiency are maintained even as problem sizes grow (e.g., Netflix 100M entries, signal length $A$9).

Limitations include:

• The required low-rank $X = \Theta(Z + A^*(s))$0 must be specified a priori, although adaptive schemes can be incorporated.
• For very large $X = \Theta(Z + A^*(s))$1, per-column or per-task inner solves may become costly.
• Only linear subspace constraints are supported; nonlinear structures would require new dual formulations or parameterizations.

Potential extensions include automatic rank adaptation (e.g., Riemannian pursuit), stochastic or online variants, distributed computation for extreme-scale settings, generalization to tensor-structured low-rank-plus-structure compositions, and incorporation of additional convex regularizers (e.g., group lasso on $X = \Theta(Z + A^*(s))$2) (Jawanpuria et al., 2017).

In summary, LoReFT provides a unified and theoretically grounded approach to low-rank matrix and signal learning, signal subspace projections, parameter-efficient neural adaptation, and Bayesian uncertainty quantification, supported by provable guarantees, efficient algorithms, and broad empirical validation (Jawanpuria et al., 2017, Zvonarev et al., 2021, Dubovik et al., 8 May 2026).