---
title: Hybrid LS/Gradient Descent Methods
url: https://www.emergentmind.com/topics/hybrid-least-squares-gradient-descent-method
type: topic
---

# Hybrid LS/Gradient Descent Methods

Searching arXiv for the cited papers to ground the article in current literature.
Searching arXiv: 2602.09057
A hybrid least squares/gradient descent method is an optimization scheme that combines least-squares structure—residual maps, normal equations, Jacobians, Gauss–Newton matrices, recursive least-squares recursions, or exact least-squares subsolves—with gradient-based iteration, momentum, adaptive steps, or stochastic-gradient updates. In recent arXiv work, the term covers several distinct but mathematically related designs: SVD-preconditioned descent for nonlinear least squares, one-pass algorithms bridging recursive least-squares and orthogonal gradient descent, randomized conjugate-gradient least-squares, Krylov–regularization hybrids, and alternating least-squares/gradient training for operator networks [2602.09057], [2207.13853], [2605.25034], [2409.09104], [2508.15394], [2607.06976].

## 1. Structural principle

The common template starts from a least-squares objective and then modifies a gradient-based update by injecting problem-specific quadratic structure. In nonlinear least squares one writes
\[
\min_{\theta\in\mathbb{R}^m} f(\theta):=\frac12\|F(\theta)\|_2^2,
\]
with gradient and Hessian
\[
\nabla f(\theta)=J_F(\theta)^\top F(\theta),\qquad
\nabla^2 f(\theta)=J_F(\theta)^\top J_F(\theta)+\sum_{i=1}^n F_i(\theta)\nabla^2F_i(\theta).
\]
Classical Gauss–Newton and Levenberg–Marquardt exploit \(J_F^\top J_F\), whereas plain gradient descent uses only \(J_F^\top F\). Hybrid methods typically keep an update of the form
\[
\theta_{t+1}=\theta_t-\alpha_t P_t \nabla f(\theta_t),
\]
but choose \(P_t\) from least-squares structure rather than from a generic first-order rule [2602.09057].

The same pattern appears outside standard nonlinear least squares. In stochastic least-squares regression, accelerated schemes retain stochastic-gradient updates but tune them to the covariance structure of the quadratic loss, improving the bias decay while preserving the minimax-optimal variance term \(O(d\sigma^2/n)\) [1704.08227]. In distributed linear least squares, iterative pre-conditioning learns an approximate inverse of \(A^\top A\) and then uses it inside a gradient-descent outer loop [2008.02856]. In operator learning, the last layer of DeepONets and MIONets is linear in its parameters, so those parameters can be updated by exact or alternating least-squares solves while the hidden parameters are updated by Adam or gradient descent [2508.15394], [2607.06976].

A plausible unifying description is that hybrid methods partition the optimization variables or directions into components that admit closed-form least-squares treatment and components that remain better handled by iterative descent.

## 2. Principal algorithmic families

The recent literature organizes naturally into several families. They differ in where the least-squares structure enters—preconditioning, projection, variance reduction, or alternating exact solves—but all preserve an iterative descent backbone.

| Family | Least-squares ingredient | Gradient ingredient |
|---|---|---|
| Spectral/preconditioned descent | \(J_F^\top J_F\), Fisher-type matrices, or \(A^\top A\) define preconditioners | Preconditioned GD, Adam, or distributed GD |
| Orthogonality/RLS hybrids | Recursive LS or minimum-norm interpolation constraints | Orthogonal gradient projection or stochastic-gradient updates |
| Krylov/projection hybrids | CGLS/LSMR projections, sketched normal equations, inner LSQR solves | Conjugate-gradient-style or projected descent |
| Alternating LS/GD in operator nets | Exact or alternating LS solves for last-layer parameters | Adam/GD for hidden branch and trunk parameters |
| Reduced-objective LS/GD methods | Linear parameters eliminated by LS, leaving nonlinear variables | Gradient descent on reduced objective |

Representative instances are "SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems" [2602.09057], "One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares" [2207.13853], "Randomized conjugate gradient least squares" [2605.25034], "Hybrid LSMR algorithms for large-scale general-form regularization" [2409.09104], "A Novel Gradient Descent Least Squares (GDLS) Algorithm for Efficient SMV Gridless Line Spectrum Estimation with Applications in Tomographic SAR Imaging" [2203.08574], "Hybrid Least Squares/Gradient Descent Methods for DeepONets" [2508.15394], and "Hybrid Least Squares/Gradient Descent Methods for MIONets" [2607.06976].

## 3. Canonical mechanisms

A first mechanism is **spectral preconditioning of the gradient**. In SPGD, the Jacobian SVD \(J_F(\theta_t)=U_t\Sigma_tV_t^\top\) is used to define
\[
B_t=\bigl(J_F(\theta_t)^\top J_F(\theta_t)\bigr)^{\dagger\,\frac12},
\]
yielding
\[
\theta_{t+1}=\theta_t-\alpha\,B_t\nabla f(\theta_t)
=\theta_t-\alpha\,V_tU_t^\top F(\theta_t).
\]
The Adam variant treats \(\lambda_t=B_t\nabla f(\theta_t)\) as the effective gradient and then applies first- and second-moment adaptation; for cross-entropy the paper replaces \(J_F^\top J_F\) by \(J_F^\top C J_F+\delta I\), where \(C\) is built from softmax Fisher blocks [2602.09057].

A second mechanism is **recursive least-squares geometry enforced through gradient projections**. ORFit projects the current loss gradient onto the orthogonal complement of the span of past sensitive directions, then chooses the step size so that the new point is interpolated in one step. For overparameterized linear models it solves
\[
w_i=\arg\min_w \|w-w_0\|_2^2
\quad\text{s.t.}\quad y_k=w^\top x_k,\;k=1,\dots,i,
\]
and is equivalent to EW-RLS with \(\lambda=0\), \(\Pi=I\). This connects minimum-norm interpolation, recursive least squares, and orthogonal gradient descent in a single update rule [2207.13853].

A third mechanism is **constraint-corrected gradient construction**. RCGLS starts from the least-squares problem
\[
\min_{x\in\mathbb{R}^d} \frac12\|Ax-b\|_2^2
\]
and interprets classical CGLS as the solution of a constrained correction problem that keeps the next search direction close to the negative gradient while enforcing \(A^\top A\)-conjugacy. The randomized version replaces the full gradient by a sketched gradient \(S_kS_k^\top A^\top r^k\), then adds the conjugate correction term \(\tau_k p^k\). The result is gradient-related, but its geometry is explicitly inherited from the least-squares normal equations [2605.25034].

A fourth mechanism is **exact least-squares elimination of linear parameters**. GDLS for gridless line spectrum estimation writes
\[
\mathbf{y}=\mathbf{A}(\mathbf{f})\mathbf{c}+\mathbf{w},
\]
eliminates amplitudes by
\[
\hat{\mathbf{c}}=\mathbf{A}^+\mathbf{y},
\]
and then minimizes the reduced least-squares objective
\[
T(\mathbf{f})=\|\mathbf{y}-\mathbf{A}(\mathbf{f})\mathbf{A}(\mathbf{f})^+\mathbf{y}\|_2^2
\]
by gradient descent over the continuous frequencies [2203.08574].

A fifth mechanism is **alternating least-squares inside neural-operator training**. DeepONets are linear in the last-layer parameters of the branch network, so those parameters solve a ridge-type LS problem while the hidden branch and trunk parameters are updated by Adam. MIONets generalize this by multilinearity: each branch last layer is optimized in turn by ALS, with the other branch last layers frozen, and hidden/trunk parameters are optimized by gradient descent between ALS sweeps [2508.15394], [2607.06976].

## 4. Convergence and acceleration properties

The strongest analytical results arise when the least-squares structure regularizes the difficult directions of the gradient flow. For SPGD, local linear convergence is proved under standard regularity assumptions. The comparison with plain GD is explicit: near a zero-residual solution, GD has a factor governed by
\[
1-\frac{\sigma_{\min}^2}{\sigma_{\max}^2},
\]
whereas SPGD has a factor of the form
\[
1-\frac{\sigma_{\min}}{2\sigma_{\max}},
\]
so the degradation depends linearly rather than quadratically on the Jacobian condition number. A modified AMSGrad-style SPGD–Adam with
\[
B_t=(J_t^\top J_t+\mu I)^{-p}
\]
and monotone second moments is shown to have vanishing average gradient norm under boundedness and smoothness assumptions [2602.09057].

RCGLS converges linearly in expectation, with a contraction bound that is always at least as good as the corresponding randomized coordinate-descent bound because the factor \(\gamma_k\) satisfies \(\gamma_k\ge 1\). The method also inherits a natural variance-reduction property from the sketched gradient \(g_k=S_kS_k^\top\nabla f(x^k)\): as \(x^k\to x^*\), the estimator variance automatically vanishes [2605.25034].

In stochastic least-squares regression, the accelerated stochastic-gradient method of "Accelerating Stochastic Gradient Descent For Least Squares Regression" achieves the minimax-optimal variance term \(O(d\sigma^2/n)\) while improving the bias contraction rate from \(O(1/\kappa)\) to \(O(1/\sqrt{\widetilde{\kappa}})\). This refutes the blanket claim that acceleration is unusable in noisy stochastic approximation for least squares [1704.08227].

In distributed linear least squares, iterative pre-conditioning converges linearly in general and achieves superlinear convergence when the least-squares problem has a unique solution. The analysis compares favorably with traditional GD and with the accelerated methods discussed in that paper [2008.02856].

ORFit supplies a different convergence statement: in overparameterized linear models, its iterate is exactly the point to which sufficiently small-step-size multi-pass SGD would converge, and the nonlinear NTK analysis extends the same minimum-norm interpretation to highly overparameterized networks [2207.13853].

## 5. Computational realizations and scalability

A recurrent theme is that hybridization is only useful if the least-squares substructure can be exploited without destroying scalability. SPGD addresses the cost of \(J_F^\top J_F\) by approximating \(B_tg\) with Lanczos iteration in Krylov subspaces
\[
\mathcal{K}_k(g)=\operatorname{span}\{g,Ag,A^2g,\dots,A^{k-1}g\},
\]
using only Jacobian-vector and vector-Jacobian products. This avoids explicit formation of \(J_F\) and makes the method viable for models with hundreds of thousands of parameters [2602.09057].

ORFit uses incremental PCA to compress the basis of past sensitive directions. The memory footprint drops from \(O(ip)\) to \(O(mp)\), with per-step overhead \(O(m^2(p+m))\), while still approximating the orthogonality constraints required for one-pass learning [2207.13853].

RCGLS develops an equivalent sparse implementation that avoids full-dimensional vector operations by maintaining implicit representations of the iterates and applying \(A\) or \(A^\top\) only to sketched pieces such as \(S_k^\top A^\top\) and \(AS_k\). This is central for sparse matrices, where vanilla RCGLS would otherwise be impractical [2605.25034].

Hybrid LSMR introduces a two-level architecture. The outer Golub–Kahan/LSMR step constructs a rank-\(k\) projected normal-equation problem; the inner correction solves
\[
\min_z \|L(I-Q_kQ_k^\top)z-Lx_k\|
\]
with LSQR. The paper proves that the condition number of \(L(I-Q_kQ_k^\top)\) decreases with \(k\), and states that it is generally enough to set \(tol=10^{-6}\) in LSQR [2409.09104].

DeepONet and MIONet hybrids turn an infeasible gigantic least-squares system into small structured problems. DeepONet uses Kronecker products and a generalized Sylvester equation for the last-layer solve; MIONet generalizes this with Khatri–Rao products and tensor permutation matrices, then solves one branch last layer at a time by ALS. The practical consequence is that LS updates can be inserted into training at modest cost relative to Adam epochs [2508.15394], [2607.06976].

## 6. Applications, limitations, and interpretation

Hybrid least squares/gradient descent methods have been deployed across markedly different problem classes. SPGD is tested on function approximation, PDE solving, and image classification on the CIFAR-10 dataset [2602.09057]. ORFit is evaluated on a Rotated MNIST streaming-regression task and on overparameterized linear and NTK-linearized settings [2207.13853]. RCGLS extends naturally to ridge regression and sparse large-scale linear problems [2605.25034]. Hybrid LSMR targets ill-posed inverse problems with general-form regularization [2409.09104]. GDLS is motivated by gridless line spectrum estimation and TomoSAR imaging [2203.08574]. DeepONet and MIONet LSGD methods target supervised and physics-informed operator learning for advection, diffusion–reaction, and Poisson-type equations [2508.15394], [2607.06976]. The Riemannian LS inverse eigenvalue method interprets lift-and-projection as gradient descent in an induced metric and includes an application to a \(32400\times32400\) sparse Mn\(_6\) Hamiltonian [2504.07809].

Several misconceptions are corrected by this literature. First, a hybrid least-squares/gradient method is not synonymous with “run Gauss–Newton instead of gradient descent.” Some hybrids use a square-root inverse of \(J_F^\top J_F\) rather than a full inverse, some insert least-squares solves only on a subset of parameters, and some use least-squares only to build control variates or memory summaries [2602.09057], [2507.20981]. Second, the least-squares component need not appear as explicit normal equations; it may appear as a projection constraint, a recursive filter, a Sylvester equation, or a reduced objective after eliminating linear variables [2207.13853], [2508.15394], [2203.08574]. Third, the acceleration is not unconditional. In ill-conditioned Bratu regimes, the extrapolation-accelerated preconditioned gradient methods studied in "A New Combination of Preconditioned Gradient Descent Methods and Vector Extrapolation Techniques for Nonlinear Least-Squares Problems" did not consistently improve convergence, whereas restarted GNKS remained robust [2602.00897].

The main limitations are structural. SPGD requires Jacobian-based spectral approximations and uses regularization and AMSGrad-style modifications to avoid Adam-type non-convergence issues [2602.09057]. ORFit’s IPCA compression inevitably loses some information and can induce forgetting [2207.13853]. Hybrid LSMR reduces to plain LSMR when \(L=I\), so its distinctive regularization role appears only for \(L\neq I\) [2409.09104]. DeepONet LSGD depends on linearity of the last branch layer and on operator factorizations that preserve the LS structure; MIONet LSGD further assumes product-form data layouts and tensor factorizability [2508.15394], [2607.06976].

Taken together, these methods show that least-squares structure can be exploited at many granularities: as curvature information, as exact subproblem solvability, as recursive memory, as variance-reduction regression, or as tensorized linear algebra inside neural operators. The recurring mathematical idea is to extract the parts of the model for which least-squares geometry is informative or tractable, and to embed that information inside a gradient-based outer iteration rather than replacing descent altogether.

Source: https://www.emergentmind.com/topics/hybrid-least-squares-gradient-descent-method