Sparse Gauss-Newton (SGN) Methods

• SGN is a family of Gauss-Newton methods that exploit sparsity in optimization variables, linear algebra, or residual couplings to bypass the dense Hessian bottleneck.
• It applies to diverse problem classes such as equilibrium-constrained inverse problems, nonlinear underdetermined systems, sparse quadratic recovery, and domain-decomposed PINNs.
• SGN delivers significant computational efficiency and maintains the exact Gauss-Newton step with strong theoretical guarantees and robust empirical performance.

Searching arXiv for the cited SGN-related papers to ground the article in recent literature. Search query: (Zehnder et al., 2021) Sparse Gauss-Newton accelerated sensitivity analysis Search query: (Gulliksson et al., 2016) Greedy Gauss-Newton sparse nonlinear underdetermined systems Search query: (Wen et al., 10 Jul 2025) Sparse Signal Recovery From Quadratic Systems with Full-Rank Matrices Sparse Gauss-Newton (SGN) denotes a family of Gauss-Newton methods in which sparsity enters through the optimization variables, the linear algebra, or the residual coupling structure rather than through a single canonical algorithm. In current arXiv usage, the term covers at least three distinct lines: an exact sparse saddle-point reformulation of reduced Gauss-Newton for equilibrium-constrained inverse problems, sparse support-restricted Gauss-Newton methods for nonlinear systems and sparse quadratic recovery, and block-sparse Gauss-Newton systems induced by localized architectures such as domain-decomposed PINNs (Zehnder et al., 2021, Gulliksson et al., 2016, Wen et al., 10 Jul 2025, Heinlein et al., 30 Oct 2025). A plausible unifying description is that SGN keeps the Gauss-Newton least-squares geometry while avoiding dense reduced Hessians or dense full-space updates.

1. Terminology and scope

The acronym SGN is overloaded, and disambiguation is essential. In "SGN: Sparse Gauss-Newton for Accelerated Sensitivity Analysis" (Zehnder et al., 2021), SGN explicitly denotes Sparse Gauss-Newton. By contrast, "Non-Asymptotic Optimization and Generalization Bounds for Stochastic Gauss-Newton in Overparameterized Models" states that its subject is stochastic Gauss-Newton (SGN) and is not “sparse Gauss–Newton” in the usual sense (Cayci, 6 Nov 2025). "Sketch-and-Project Meets Newton Method: Global $\mathcal O(k^{-2})$ Convergence with Low-Rank Updates" uses SGN for Sketchy Global Newton, not Gauss-Newton (Hanzely, 2023). "A Randomised Subspace Gauss-Newton Method for Nonlinear Least-Squares" studies a variable-domain sketched or block-coordinate Gauss-Newton method rather than classical sparse Gauss-Newton based on sparse Jacobians or sparse normal equations (Cartis et al., 2022).

Within the sparse-Gauss-Newton literature itself, sparsity can mean different things. In equilibrium-constrained inverse problems, sparsity refers to replacing a dense reduced Hessian by a larger sparse saddle-point system while preserving the same reduced search direction exactly (Zehnder et al., 2021). In sparse recovery, sparsity refers to explicit support constraints such as $\|x\|_0\le s$, combined with support-restricted Gauss-Newton updates (Wen et al., 10 Jul 2025). In nonlinear underdetermined systems, sparsity is enforced by greedy support growth, so that each Gauss-Newton step is computed on a small active set of variables (Gulliksson et al., 2016). In localized PINNs, sparsity is induced structurally: non-overlapping subdomains do not couple in the Gramian, so the Gauss-Newton matrix becomes block sparse (Heinlein et al., 30 Oct 2025).

2. Canonical problem classes

The main SGN formulations in the literature can be organized by the source of sparsity.

Setting	Canonical formulation	Sparsity mechanism
Equilibrium-constrained inverse problems	$\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$	Sparse saddle-point reformulation (Zehnder et al., 2021)
Nonlinear underdetermined systems	$f(x)=0$, with $m<N$	Greedy support-restricted GN (Gulliksson et al., 2016)
Sparse quadratic recovery	$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$	Hard-thresholding and restricted GN (Wen et al., 10 Jul 2025)
Domain-decomposed PINNs	$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$	Block-sparse $J^\top J$ from localization (Heinlein et al., 30 Oct 2025)

For equilibrium-constrained inverse problems, the starting point is

$$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$

with $x \in \mathbb R^{n_x}$, $\|x\|_0\le s$0, and $\|x\|_0\le s$1. Under the common case $\|x\|_0\le s$2 and full rank of $\|x\|_0\le s$3, the implicit function theorem yields a reduced problem $\|x\|_0\le s$4, but the reduced Gauss-Newton Hessian is dense in general even when the simulation Jacobians are sparse (Zehnder et al., 2021).

For nonlinear underdetermined systems, the objective is to find a sparse solution of

$\|x\|_0\le s$5

by minimizing the Gauss-Newton merit function

$\|x\|_0\le s$6

while restricting updates to a small active support (Gulliksson et al., 2016).

For sparse quadratic systems, the measurements are

$\|x\|_0\le s$7

and the recovery problem is posed as a sparse nonlinear least-squares objective,

$\|x\|_0\le s$8

with an explicit sparsity constraint $\|x\|_0\le s$9 (Wen et al., 10 Jul 2025).

For localized PINNs, the least-squares structure comes from residual minimization,

$\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$0

while the parameter vector is partitioned by subdomain, $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$1, so that non-overlapping subdomains produce zero off-diagonal Gramian blocks (Heinlein et al., 30 Oct 2025).

3. Principal algorithmic constructions

A central SGN construction is the exact sparse reformulation of reduced Gauss-Newton for equilibrium-constrained inverse problems. With

$\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$2

the dense reduced Gauss-Newton step solves

$\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$3

SGN introduces auxiliary variables $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$4 and $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$5 and instead solves the sparse saddle-point system

$\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$6

The sparse reformulation is exact in the sense that the computed $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$7 is exactly the same as the $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$8 obtained from the dense reduced Gauss-Newton system (Zehnder et al., 2021).

A second SGN pattern is greedy support expansion for underdetermined nonlinear systems. At iteration $\min_{x,p} f(x,p)\ \text{s.t.}\ c(x,p)=\mathbf 0$9, with current support $f(x)=0$0 and restricted Jacobian $f(x)=0$1, each candidate new index $f(x)=0$2 defines a restricted Gauss-Newton step

$f(x)=0$3

The paper proposes two greedy rules. The Maximum Descent criterion is

$f(x)=0$4

while the Orthogonal Matching criterion is

$f(x)=0$5

After selecting $f(x)=0$6, the method performs a line-search update

$f(x)=0$7

with support growth by at most one coordinate per iteration (Gulliksson et al., 2016).

A third pattern is hard-thresholding Gauss-Newton for sparse quadratic recovery. Writing

$f(x)=0$8

the method alternates between a sparse support proposal

$f(x)=0$9

and a restricted Gauss-Newton solve on $m<N$0,

$m<N$1

The restricted direction is

$m<N$2

so the linear algebra is confined to an $m<N$3 system (Wen et al., 10 Jul 2025).

A fourth pattern arises in domain-decomposed PINNs. With $m<N$4 and Jacobian blocks $m<N$5, the Gauss-Newton matrix is

$m<N$6

If subdomains $m<N$7 and $m<N$8 do not overlap, then

$m<N$9

so localization turns the dense Gauss-Newton system into a block-sparse one (Heinlein et al., 30 Oct 2025).

4. Mathematical guarantees

The strongest exact-equivalence result appears in equilibrium-constrained inverse problems. Theorem 1 states that solving the larger sparse SGN saddle-point system is equivalent to solving the dense reduced system

$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$0

with the same reduced search direction $$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$1 (Zehnder et al., 2021). The same paper also proves a nonlinear-programming connection: if the Lagrange multipliers are chosen as the adjoint variables,

$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$2

then the KKT system and the second-order sensitivity system

$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$3

give the same search direction $$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$4. This establishes a precise reduced-Hessian / projected-SQP equivalence for the stated problem class (Zehnder et al., 2021).

For sparse quadratic recovery, the theory separates identifiability, initialization, and local refinement. The algebraic-geometry results state that generic measurements suffice for uniqueness on sparse signal classes: if

$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$5

the real measurement map is injective on $$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$6 up to global sign, and if

$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$7

the complex Hermitian case is injective on $$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$8 up to global phase (Wen et al., 10 Jul 2025). Algorithmically, support-restricted spectral initialization satisfies

$$\min_{z,\ \|z\|_0\le s}\frac{1}{2m}\sum_{i=1}^m (z^\top A_i z-y_i)^2$$9

with high probability provided

$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$0

and the refinement stage requires only

$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$1

In the noiseless case, refinement is first linear and then quadratic: after at most

$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$2

the iterates satisfy

$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$3

The paper emphasizes that this quadratic convergence is obtained without requiring resampling (Wen et al., 10 Jul 2025).

For greedy support-expansion Gauss-Newton, the guarantees are classical descent-method results rather than sparse identifiability theorems. Under Lipschitz continuity of $\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$4, compactness of the level set, descent directions, and Armijo-Goldstein line search, the iterates satisfy

$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$5

If, additionally, $\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$6, $\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$7, $\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$8, and

$\min_\theta \frac1N\sum_{i=1}^N r_i(\theta)^2$9

then the local convergence is quadratic (Gulliksson et al., 2016). A further structural result shows that, after finitely many iterations under constant Jacobian rank on the relevant level set, the method attains the same descent measure as standard Gauss-Newton on the discovered effective subspace (Gulliksson et al., 2016).

The block-sparse PINN literature is more qualitative on theory in the provided text. It establishes the exact block-sparsity mechanism

$J^\top J$0

but does not provide a timing-based sparse linear-solver theory in the same level of detail (Heinlein et al., 30 Oct 2025).

5. Applications and empirical behavior

In equilibrium-constrained optimization, SGN is evaluated on inverse elastic design, shell form finding, rod dome design, car control, and cloth control. The paper emphasizes that SGN and dense GN produce the same search directions, so the comparison is about runtime and memory. In inverse elastic design, the block-solve variant BGN gives about 30% speedup over SGN at low resolution and more than 50% at higher resolution. In the shell roof problem, SGN clearly outperforms all other methods, while dense GN runs out of memory on the large 15,443-vertex case; the sparse matrix has

$J^\top J$1

nonzeros versus a dense Hessian with

$J^\top J$2

entries, and PARDISO reports

$J^\top J$3

nonzeros in the factorization. In the rod dome example, break-even is around 200 parameters. In cloth control, dense GN runs out of memory for

$J^\top J$4

These experiments support the paper’s claim that SGN preserves the reduced Gauss-Newton step while removing the dense reduced-Hessian bottleneck (Zehnder et al., 2021).

In sparse quadratic recovery, the proposed support-restricted spectral initialization gives lower average relative error than thresholded spectral initialization, about 20% lower than best TSI when $J^\top J$5. Phase-transition experiments show that, for relatively low sparsity $J^\top J$6, both TWF and SGN can exceed 50% success when $J^\top J$7, while for higher sparsity SGN requires fewer measurements than TWF for the same success rate. A convergence experiment reports that SGN needs only about one tenth as many iterations as the compared existing algorithm to achieve successful recovery. In noisy experiments, relative error decreases steadily as SNR increases, and even at relatively low SNR the method attains log relative error below about $J^\top J$8 (Wen et al., 10 Jul 2025).

For localized PINNs, the main evidence is structural plus optimization behavior. In the 1D high-frequency ODE example, the reported relative $J^\top J$9 test errors are $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$0 for FBPINN trained with Adam and $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$1 for FBPINN trained with Gauss-Newton. In the 2D Helmholtz example, the corresponding errors are $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$2 and $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$3. The paper also shows an exemplary sparsity pattern of the Gramian $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$4 for the 1D problem with 8 subdomains and argues that domain decomposition creates pronounced sparsity; however, it explicitly states, “While we do not report computing times…”, so the efficiency claim is structural rather than benchmarked in wall-clock terms (Heinlein et al., 30 Oct 2025).

6. Limitations, misconceptions, and related directions

The most common misconception is terminological: not every paper using “SGN” studies Sparse Gauss-Newton. Recent literature also uses SGN for stochastic Gauss-Newton (Cayci, 6 Nov 2025), Sketchy Global Newton (Hanzely, 2023), and inexact stochastic generalized Gauss-Newton baselines in deep-learning optimization (Korbit et al., 2024). Randomised Subspace Gauss-Newton is also related but distinct: it restricts the step to a random low-dimensional subspace of the variable domain rather than exploiting sparse Jacobians or sparse saddle-point systems (Cartis et al., 2022).

Sparse Gauss-Newton methods themselves have regime-dependent limitations. The equilibrium-constrained formulation assumes $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$5 and full-rank $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$6, depends on sparse direct solvers, and must handle an indefinite saddle-point system; sparse direct solver dependence and robust stabilization are explicit limitations in that work (Zehnder et al., 2021). The sparse quadratic recovery method attains $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$7 refinement complexity only after initialization has already entered the local basin; end-to-end sample complexity is still dominated by the initialization stage at $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$8, the algorithmic proofs assume i.i.d. Gaussian measurement matrices, and the main convergence analysis is presented only for the real setting (Wen et al., 10 Jul 2025). In localized PINNs, the Gramian may be “(nearly) singular” and “generally ill-conditioned,” so the method requires regularization $$\min_{x,p} f(x,p) \quad \text{s.t.} \quad c(x,p)=\mathbf 0,$$9, and the paper does not provide a detailed sparse linear-solver study or timing study (Heinlein et al., 30 Oct 2025).

A broader conceptual limitation is that “sparsity” is not uniform across the literature. In some papers it is an exact sparse reformulation that preserves the full reduced Gauss-Newton direction (Zehnder et al., 2021). In others it is a support constraint or support-pursuit mechanism (Gulliksson et al., 2016, Wen et al., 10 Jul 2025). In domain-decomposed PINNs it is block sparsity induced by local support (Heinlein et al., 30 Oct 2025). This suggests that Sparse Gauss-Newton is best understood not as a single algorithmic template, but as a family of Gauss-Newton methods that exploit sparse structure at different levels: variables, residual couplings, or linear systems.