---
title: Sparse Gauss-Newton (SGN) Methods
url: https://www.emergentmind.com/topics/sparse-gauss-newton-sgn
type: topic
---

# Sparse Gauss-Newton (SGN) Methods

Searching arXiv for the cited SGN-related papers to ground the article in recent literature.
Search query: 2107.03285 Sparse Gauss-Newton accelerated sensitivity analysis
Search query: 1610.03095 Greedy Gauss-Newton sparse nonlinear underdetermined systems
Search query: 2507.07557 Sparse Signal Recovery From Quadratic Systems with Full-Rank Matrices
Sparse Gauss-Newton (SGN) denotes a family of Gauss-Newton methods in which sparsity enters through the optimization variables, the linear algebra, or the residual coupling structure rather than through a single canonical algorithm. In current arXiv usage, the term covers at least three distinct lines: an exact sparse saddle-point reformulation of reduced Gauss-Newton for equilibrium-constrained inverse problems, sparse support-restricted Gauss-Newton methods for nonlinear systems and sparse quadratic recovery, and block-sparse Gauss-Newton systems induced by localized architectures such as domain-decomposed PINNs [2107.03285][1610.03095][2507.07557][2510.27018]. A plausible unifying description is that SGN keeps the Gauss-Newton least-squares geometry while avoiding dense reduced Hessians or dense full-space updates.

## 1. Terminology and scope

The acronym SGN is overloaded, and disambiguation is essential. In "SGN: Sparse Gauss-Newton for Accelerated Sensitivity Analysis" [2107.03285], SGN explicitly denotes **Sparse Gauss-Newton**. By contrast, "Non-Asymptotic Optimization and Generalization Bounds for Stochastic Gauss-Newton in Overparameterized Models" states that its subject is **stochastic Gauss-Newton (SGN)** and is not “sparse Gauss–Newton” in the usual sense [2511.03972]. "Sketch-and-Project Meets Newton Method: Global $\mathcal O(k^{-2})$ Convergence with Low-Rank Updates" uses SGN for **Sketchy Global Newton**, not Gauss-Newton [2305.13082]. "A Randomised Subspace Gauss-Newton Method for Nonlinear Least-Squares" studies a variable-domain sketched or block-coordinate Gauss-Newton method rather than classical sparse Gauss-Newton based on sparse Jacobians or sparse normal equations [2211.05727].

Within the sparse-Gauss-Newton literature itself, sparsity can mean different things. In equilibrium-constrained inverse problems, sparsity refers to replacing a dense reduced Hessian by a larger sparse saddle-point system while preserving the same reduced search direction exactly [2107.03285]. In sparse recovery, sparsity refers to explicit support constraints such as $\|x\|_0\le s$, combined with support-restricted Gauss-Newton updates [2507.07557]. In nonlinear underdetermined systems, sparsity is enforced by greedy support growth, so that each Gauss-Newton step is computed on a small active set of variables [1610.03095]. In localized PINNs, sparsity is induced structurally: non-overlapping subdomains do not couple in the Gramian, so the Gauss-Newton matrix becomes block sparse [2510.27018].

## 2. Canonical problem classes

The main SGN formulations in the literature can be organized by the source of sparsity.

| Setting | Canonical formulation | Sparsity mechanism |
|---|---|---|
| Equilibrium-constrained inverse problems | $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$ | Sparse saddle-point reformulation [2107.03285] |
| Nonlinear underdetermined systems | $f(x)=0$, with $m<N$ | Greedy support-restricted GN [1610.03095] |
| Sparse quadratic recovery | $\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$ | Hard-thresholding and restricted GN [2507.07557] |
| Domain-decomposed PINNs | $\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$ | Block-sparse $J^\top J$ from localization [2510.27018] |

For equilibrium-constrained inverse problems, the starting point is
\[
\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,
\]
with $x \in \mathbb R^{n_x}$, $p \in \mathbb R^{n_p}$, and $c(x,p)\in \mathbb R^{n_c}$. Under the common case $n_c=n_x$ and full rank of $\frac{\partial c}{\partial x}$, the implicit function theorem yields a reduced problem $\min_p f(x(p),p)$, but the reduced Gauss-Newton Hessian is dense in general even when the simulation Jacobians are sparse [2107.03285].

For nonlinear underdetermined systems, the objective is to find a sparse solution of
\[
f(x)=0,\qquad x\in\mathbb R^N,\quad m<N,
\]
by minimizing the Gauss-Newton merit function
\[
F(x)=\frac12\|f(x)\|_2^2
\]
while restricting updates to a small active support [1610.03095].

For sparse quadratic systems, the measurements are
\[
y_i = x^\ast A_i x,\qquad i=1,\dots,m,
\]
and the recovery problem is posed as a sparse nonlinear least-squares objective,
\[
f(z) := \frac{1}{2m}\sum_{i=1}^m (z^\top A_i z - y_i)^2,
\]
with an explicit sparsity constraint $\|z\|_0\le s$ [2507.07557].

For localized PINNs, the least-squares structure comes from residual minimization,
\[
\mathcal L(\theta)=\frac{1}{N}\sum_{i=1}^N r_i(\theta)^2,
\]
while the parameter vector is partitioned by subdomain, $\theta=(\theta_1,\dots,\theta_K)$, so that non-overlapping subdomains produce zero off-diagonal Gramian blocks [2510.27018].

## 3. Principal algorithmic constructions

A central SGN construction is the exact sparse reformulation of reduced Gauss-Newton for equilibrium-constrained inverse problems. With
\[
A=\sum_i w_i \frac{\partial r_i}{\partial x}^T \frac{\partial r_i}{\partial x}, \qquad
B=\sum_i w_i \frac{\partial r_i}{\partial p}^T \frac{\partial r_i}{\partial x}, \qquad
C=\sum_i w_i \frac{\partial r_i}{\partial p}^T \frac{\partial r_i}{\partial p},
\]
the dense reduced Gauss-Newton step solves
\[
H_{GN}\,\delta p = -\frac{df}{dp}^T.
\]
SGN introduces auxiliary variables $\delta x$ and $\delta\lambda$ and instead solves the sparse saddle-point system
\[
\begin{bmatrix}
A& B^T & \frac{\partial c}{\partial x}^T\\
B & C & \frac{\partial c}{\partial p} ^T\\
\frac{\partial c}{\partial x} & \frac{\partial c}{\partial p} & 0
\end{bmatrix}
\begin{bmatrix}
\delta x\\
\delta p\\
\delta\lambda
\end{bmatrix}
=
\begin{bmatrix}
0\\
- \frac{d f}{d p}^T\\
0
\end{bmatrix}.
\]
The sparse reformulation is exact in the sense that the computed $\delta p$ is exactly the same as the $\delta p$ obtained from the dense reduced Gauss-Newton system [2107.03285].

A second SGN pattern is greedy support expansion for underdetermined nonlinear systems. At iteration $k$, with current support $\Omega_k$ and restricted Jacobian $L_k=J_k(:,\Omega_k)$, each candidate new index $t\in\overline{\Omega}_k$ defines a restricted Gauss-Newton step
\[
q_t = - (L_k,J_k(:,t))^+ f_k.
\]
The paper proposes two greedy rules. The **Maximum Descent** criterion is
\[
t_{MD} = \argmax_{t\in\overline{\Omega}_k}
\frac{\left|f_k^T(I-L_kL_k^+)J_k(:,t)\right|}
{\|(I-L_kL_k^+)J_k(:,t)\|},
\]
while the **Orthogonal Matching** criterion is
\[
t_{OM} = \argmax_{t\in\overline{\Omega}_k}
\frac{\left|f_k^T(I-L_kL_k^+)J_k(:,t)\right|}
{\|J_k(:,t)\|}.
\]
After selecting $t_*$, the method performs a line-search update
\[
x_{k+1}=x_k+\alpha_k p_k,
\]
with support growth by at most one coordinate per iteration [1610.03095].

A third pattern is hard-thresholding Gauss-Newton for sparse quadratic recovery. Writing
\[
F_i(z)=\frac{1}{\sqrt m}(z^\top A_i z-y_i),
\qquad
J(z)=\frac1{\sqrt m}
\begin{bmatrix}
(A_1+A_1^\top)z,\dots,(A_m+A_m^\top)z
\end{bmatrix}^{\!\top},
\]
the method alternates between a sparse support proposal
\[
u^k = \mathcal H_s\big(x^k-\mu^k \nabla f(x^k)\big),\qquad
S_{k+1}=\operatorname{supp}(u^k),
\]
and a restricted Gauss-Newton solve on $S_{k+1}$,
\[
x^{k+1}_{S_{k+1}} = x^k_{S_{k+1}} - p^k_{S_{k+1}}, \qquad
x^{k+1}_{S_{k+1}^c}=0.
\]
The restricted direction is
\[
p^k_{S_{k+1}} =
\Big[J_{S_{k+1}}(x^k)^\top J_{S_{k+1}}(x^k)\Big]^{-1}
\Big(
\nabla_{S_{k+1}} f(x^k)
-
J_{S_{k+1}}(x^k)^\top J_{S_{k+1}^c}(x^k)x^k_{S_{k+1}^c}
\Big),
\]
so the linear algebra is confined to an $s\times s$ system [2507.07557].

A fourth pattern arises in domain-decomposed PINNs. With $\theta=(\theta_1,\dots,\theta_K)$ and Jacobian blocks $J=[J_1\;J_2\;\cdots\;J_K]$, the Gauss-Newton matrix is
\[
G=J^\top J=
\begin{bmatrix}
J_1^\top J_1 & J_1^\top J_2 & \cdots & J_1^\top J_K\\
J_2^\top J_1 & J_2^\top J_2 & \cdots & J_2^\top J_K\\
\vdots & \vdots & \ddots & \vdots\\
J_K^\top J_1 & J_K^\top J_2 & \cdots & J_K^\top J_K
\end{bmatrix}.
\]
If subdomains $i$ and $j$ do not overlap, then
\[
J_i^\top J_j = 0,
\]
so localization turns the dense Gauss-Newton system into a block-sparse one [2510.27018].

## 4. Mathematical guarantees

The strongest exact-equivalence result appears in equilibrium-constrained inverse problems. Theorem 1 states that solving the larger sparse SGN saddle-point system is equivalent to solving the dense reduced system
\[
H\delta p = -\frac{df}{dp}^T,
\]
with the same reduced search direction $\delta p$ [2107.03285]. The same paper also proves a nonlinear-programming connection: if the Lagrange multipliers are chosen as the adjoint variables,
\[
\lambda = -\left[\frac{\partial c}{\partial x}\right]^{-T} \frac{\partial f}{\partial x}^T,
\]
then the KKT system and the second-order sensitivity system
\[
\frac{d^2 f}{dp^2}\delta p = -\frac{df}{dp}^T
\]
give the same search direction $\delta p$. This establishes a precise reduced-Hessian / projected-SQP equivalence for the stated problem class [2107.03285].

For sparse quadratic recovery, the theory separates identifiability, initialization, and local refinement. The algebraic-geometry results state that generic measurements suffice for uniqueness on sparse signal classes: if
\[
m\ge 2s,
\]
the real measurement map is injective on $\Sigma_s(\mathbb R^n)$ up to global sign, and if
\[
m\ge 4s-2,
\]
the complex Hermitian case is injective on $\Sigma_s(\mathbb C^n)$ up to global phase [2507.07557]. Algorithmically, support-restricted spectral initialization satisfies
\[
\operatorname{dist}(x^0,x)\le \delta_0\|x\|_2
\]
with high probability provided
\[
m > C(\delta_0)\left(1+\frac{\sigma}{\|x\|_2^2}\right)^2 s^2 \log(mn),
\]
and the refinement stage requires only
\[
m > C' s\log n.
\]
In the noiseless case, refinement is first linear and then quadratic: after at most
\[
k \ge c_1\log\!\left(\frac{\|x\|_2}{x_{\min}}\right)+c_2,
\]
the iterates satisfy
\[
\operatorname{dist}(x^{k+1},x)\le \nu' \operatorname{dist}(x^k,x)^2.
\]
The paper emphasizes that this quadratic convergence is obtained **without requiring resampling** [2507.07557].

For greedy support-expansion Gauss-Newton, the guarantees are classical descent-method results rather than sparse identifiability theorems. Under Lipschitz continuity of $\nabla F$, compactness of the level set, descent directions, and Armijo-Goldstein line search, the iterates satisfy
\[
\lim_{k\to\infty}\frac{p_k^T\nabla F(x_k)}{\|p_k\|}=0.
\]
If, additionally, $\alpha_k=1$, $x_k\to \hat x$, $f(\hat x)=0$, and
\[
\|p_k\|\le C\|f_k\|,
\]
then the local convergence is quadratic [1610.03095]. A further structural result shows that, after finitely many iterations under constant Jacobian rank on the relevant level set, the method attains the same descent measure as standard Gauss-Newton on the discovered effective subspace [1610.03095].

The block-sparse PINN literature is more qualitative on theory in the provided text. It establishes the exact block-sparsity mechanism
\[
G_{ij}=J_i^\top J_j=0
\quad\text{for non-overlapping subdomains } \Omega_i,\Omega_j,
\]
but does not provide a timing-based sparse linear-solver theory in the same level of detail [2510.27018].

## 5. Applications and empirical behavior

In equilibrium-constrained optimization, SGN is evaluated on inverse elastic design, shell form finding, rod dome design, car control, and cloth control. The paper emphasizes that SGN and dense GN produce the same search directions, so the comparison is about runtime and memory. In inverse elastic design, the block-solve variant BGN gives about **30%** speedup over SGN at low resolution and **more than 50%** at higher resolution. In the shell roof problem, SGN clearly outperforms all other methods, while dense GN runs **out of memory** on the large 15,443-vertex case; the sparse matrix has
\[
1.77\cdot 10^7
\]
nonzeros versus a dense Hessian with
\[
2.13\cdot 10^9
\]
entries, and PARDISO reports
\[
2.18\cdot 10^8
\]
nonzeros in the factorization. In the rod dome example, break-even is around **200 parameters**. In cloth control, dense GN runs **out of memory** for
\[
n_x > 1.8\cdot 10^5.
\]
These experiments support the paper’s claim that SGN preserves the reduced Gauss-Newton step while removing the dense reduced-Hessian bottleneck [2107.03285].

In sparse quadratic recovery, the proposed support-restricted spectral initialization gives lower average relative error than thresholded spectral initialization, about **20% lower** than best TSI when \(m/n\le 0.3\). Phase-transition experiments show that, for relatively low sparsity \((s/n\le 0.4)\), both TWF and SGN can exceed 50% success when \(m/n>0.5\), while for higher sparsity SGN requires fewer measurements than TWF for the same success rate. A convergence experiment reports that SGN needs only about **one tenth as many iterations** as the compared existing algorithm to achieve successful recovery. In noisy experiments, relative error decreases steadily as SNR increases, and even at relatively low SNR the method attains log relative error below about \(-3.5\) [2507.07557].

For localized PINNs, the main evidence is structural plus optimization behavior. In the 1D high-frequency ODE example, the reported relative $\ell_2$ test errors are \(7.8\times 10^{-3}\) for FBPINN trained with Adam and \(8.0\times 10^{-4}\) for FBPINN trained with Gauss-Newton. In the 2D Helmholtz example, the corresponding errors are \(2.0\times 10^{-4}\) and \(1.5\times 10^{-4}\). The paper also shows an exemplary sparsity pattern of the Gramian \(G\) for the 1D problem with 8 subdomains and argues that domain decomposition creates pronounced sparsity; however, it explicitly states, “While we do not report computing times…”, so the efficiency claim is structural rather than benchmarked in wall-clock terms [2510.27018].

## 6. Limitations, misconceptions, and related directions

The most common misconception is terminological: not every paper using “SGN” studies Sparse Gauss-Newton. Recent literature also uses SGN for stochastic Gauss-Newton [2511.03972], Sketchy Global Newton [2305.13082], and inexact stochastic generalized Gauss-Newton baselines in deep-learning optimization [2405.14402]. Randomised Subspace Gauss-Newton is also related but distinct: it restricts the step to a random low-dimensional subspace of the variable domain rather than exploiting sparse Jacobians or sparse saddle-point systems [2211.05727].

Sparse Gauss-Newton methods themselves have regime-dependent limitations. The equilibrium-constrained formulation assumes \(n_c=n_x\) and full-rank \(\frac{\partial c}{\partial x}\), depends on sparse direct solvers, and must handle an indefinite saddle-point system; sparse direct solver dependence and robust stabilization are explicit limitations in that work [2107.03285]. The sparse quadratic recovery method attains \(O(s\log n)\) refinement complexity only after initialization has already entered the local basin; end-to-end sample complexity is still dominated by the initialization stage at \(O(s^2\log n)\), the algorithmic proofs assume i.i.d. Gaussian measurement matrices, and the main convergence analysis is presented only for the real setting [2507.07557]. In localized PINNs, the Gramian may be “(nearly) singular” and “generally ill-conditioned,” so the method requires regularization \(G+\mu I\), and the paper does not provide a detailed sparse linear-solver study or timing study [2510.27018].

A broader conceptual limitation is that “sparsity” is not uniform across the literature. In some papers it is an exact sparse reformulation that preserves the full reduced Gauss-Newton direction [2107.03285]. In others it is a support constraint or support-pursuit mechanism [1610.03095][2507.07557]. In domain-decomposed PINNs it is block sparsity induced by local support [2510.27018]. This suggests that Sparse Gauss-Newton is best understood not as a single algorithmic template, but as a family of Gauss-Newton methods that exploit sparse structure at different levels: variables, residual couplings, or linear systems.

Source: https://www.emergentmind.com/topics/sparse-gauss-newton-sgn