Gauss–Newton Influence in Optimization

• Gauss–Newton influence is a method that approximates the Hessian by using Jacobian-induced metrics to define search directions and control error scaling in optimization.
• It replaces complex second-order derivative terms with a structured, separable curvature form, enabling efficient updates in least-squares, neural networks, and PDE-constrained problems.
• Structured variants and relaxations of this approach improve stability and computational performance across domains like full waveform inversion, motion optimization, and reinforcement learning.

Gauss–Newton influence denotes the family of effects induced by replacing an exact Hessian with a Gauss–Newton or generalized Gauss–Newton curvature model, and by using that model either to define search directions, to characterize optimization geometry, or to approximate perturbations of fitted parameters. In least-squares settings this curvature takes the classical form $J^\top J$; in more general settings it appears as $J^\top H_\ell J$, data-space factorizations, or sample-wise rank-one quantities. Across inverse problems, neural-network training, PDE-constrained optimization, motion optimization, reinforcement learning, and post-hoc uncertainty quantification, the common theme is that Gauss–Newton structure changes how error signals are scaled, projected, or propagated, often by suppressing troublesome second-order terms, exposing separable curvature, or yielding symmetric positive definite subproblems (Gholami, 2023).

1. Classical meaning and scope

In nonlinear least squares with residual vector $r(x)$, the exact Hessian decomposes into a positive semidefinite Jacobian term and a residual-weighted second-derivative term, while the Gauss–Newton approximation keeps only the Jacobian part. In the weighted least-squares formulation used for recursive filtering, this yields the normal equations

$$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$

with the Hessian approximation $J_r(x_k)^T W J_r(x_k)$ and update $x_{k+1}=x_k+\Delta x_k$ (Nadjiasngar et al., 2011). In full waveform inversion (FWI), the corresponding reduced objective

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$

leads to the Gauss–Newton system

$$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$

where $J_s$ is the Fréchet derivative of the forward map for source $s$ (Gholami, 2023).

The term is also used in a more specific influence-function sense. In neural network regression for full conformal prediction, Gauss–Newton influence refers to a local retraining approximation in which the effect of adding a candidate test label is modeled by a single Newton-style perturbation using a Gauss–Newton matrix rather than the full Hessian. The resulting parameter perturbation is

$J^\top H_\ell J$0

with $J^\top H_\ell J$1 (Tailor et al., 27 Jul 2025).

This dual usage suggests two closely related meanings. One is curvature-based: Gauss–Newton influence is the way $J^\top H_\ell J$2 or $J^\top H_\ell J$3 shapes local optimization. The other is perturbational: it is the way the same curvature object controls first-order approximations of parameter re-optimization under data perturbations. In both senses, the central object is the Jacobian-induced metric rather than the full second derivative tensor.

2. Core algebraic mechanism

For least-squares neural-network or inverse-problem objectives, the standard rationale for Gauss–Newton is that the exact Hessian can be written as a sum of a Jacobian Gram term and a residual-dependent remainder. In the variational PDE setting,

$J^\top H_\ell J$4

with

$J^\top H_\ell J$5

and $J^\top H_\ell J$6 collecting second derivatives of the network output with respect to parameters. Near a good minimizer, the paper shows that $J^\top H_\ell J$7 can be made arbitrarily small in a neighborhood of $J^\top H_\ell J$8, so the Hessian is well approximated by $J^\top H_\ell J$9 (Hao et al., 2023).

In shallow ReLU approximation, the same decomposition appears at the layer level. The Hessian with respect to nonlinear hidden-layer parameters splits into a principal Gauss–Newton piece and a residual term involving Dirac distributions on breaking hyperplanes. The Gauss–Newton matrix is

$r(x)$0

and the full Hessian keeps an additional residual term $r(x)$1 that is discarded in the Gauss–Newton approximation (Cai et al., 2024).

In phase retrieval, the mechanism is modified by symmetry. The lifted Gauss–Newton matrix $r(x)$2 is rank-deficient because the infinitesimal global phase direction lies in its null space: $r(x)$3 The Gauss–Newton step is therefore defined as the minimal-norm solution orthogonal to the trivial phase direction, equivalently via the pseudoinverse $r(x)$4 (Huang, 2024).

A broad misconception is that Gauss–Newton is merely a crude Hessian surrogate. Several of the cited works show a more specific structure. In motion optimization, for objective terms built from finite-differenced task-space derivatives,

$r(x)$5

the exact Hessian converges to the Gauss–Newton Hessian as $r(x)$6 at rate $r(x)$7, where $r(x)$8. For squared task-space velocity and acceleration, the rates are $r(x)$9 and $$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$0, respectively (Ratliff et al., 2016). This suggests that, in such discretized trajectory objectives, the neglected tensor term is asymptotically negligible rather than intrinsically dominant.

3. Geometry, conditioning, and projection effects

One recurrent interpretation of Gauss–Newton influence is geometric: it replaces parameter-space curvature dominated by second derivatives of the model with function-space or output-space curvature induced by the Jacobian. In the underparameterized smooth-network regime, Gauss–Newton flow induces a Riemannian gradient flow on the embedded manifold

$$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$1

and the output-space dynamics are

$$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$2

where

$$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$3

is the orthogonal projector onto the tangent space $$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$4 (Cayci, 2024). In this formulation, Gauss–Newton is literally the Riemannian gradient flow of the loss restricted to realizable outputs.

A function-space analysis goes further by contrasting Gauss–Newton and Newton. For empirical risk $$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$5, the generalized Gauss–Newton matrix is

$$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$6

while the least-squares Gauss–Newton matrix is $$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$7. The paper “Error whitening: Why Gauss-Newton outperforms Newton” argues that the generalized Gauss–Newton projects the Newton direction in function space onto the model’s tangent space, while the Jacobian-only variant projects the function-space loss gradient onto the same tangent space. In both cases, the dynamics of the prediction-target mismatch no longer depend on the parameterization through $$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$8; this replacement of $$\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),$$9 by the identity on the reachable subspace is կոչված “error whitening” (McKay et al., 11 May 2026).

Conditioning results make this geometry quantitative. For deep linear networks trained with mean-squared error, the Gauss–Newton matrix

$J_r(x_k)^T W J_r(x_k)$0

has a pseudo-condition number $J_r(x_k)^T W J_r(x_k)$1. Tight bounds show that $J_r(x_k)^T W J_r(x_k)$2 is controlled by the input covariance $J_r(x_k)^T W J_r(x_k)$3 and products of layer condition numbers, with residual connections strictly improving these bounds by shifting singular values through $J_r(x_k)^T W J_r(x_k)$4. The same analysis extends to Toeplitz representations of convolutional layers and to a one-hidden-layer Leaky-ReLU setting (Zhao et al., 2024). This suggests that Gauss–Newton influence is not only algorithmic; it is also architectural, because depth, width, residual pathways, convolutional structure, and data whitening all change the curvature seen by Gauss–Newton.

4. Structured variants and explicit relaxations

Many modern uses of Gauss–Newton influence arise not from the plain $J_r(x_k)^T W J_r(x_k)$5 system but from structured relaxations that expose additional algebraic regularity.

In FWI, the standard Gauss–Newton Hessian can be rewritten as

$J_r(x_k)^T W J_r(x_k)$6

and the usual vector update can be represented as the diagonal of a matrix equation

$J_r(x_k)^T W J_r(x_k)$7

Relaxing the diagonality constraint yields the Extended Gauss–Newton system

$J_r(x_k)^T W J_r(x_k)$8

whose regularized solution is

$J_r(x_k)^T W J_r(x_k)$9

Equivalently,

$x_{k+1}=x_k+\Delta x_k$0

with small receiver- and source-side Hessians $x_{k+1}=x_k+\Delta x_k$1 and $x_{k+1}=x_k+\Delta x_k$2 (Gholami, 2023). Here Gauss–Newton influence becomes a two-dimensional deblurring in the source–receiver plane rather than a direct model-space solve.

In shallow ReLU learning, the structure-guided Gauss–Newton method alternates between an exact linear solve for output-layer parameters and a damped Gauss–Newton step for hidden-layer parameters. The mass matrix $x_{k+1}=x_k+\Delta x_k$3 and the layer Gauss–Newton matrix $x_{k+1}=x_k+\Delta x_k$4 are shown to be symmetric and positive definite under distinct-hyperplane assumptions, so the method naturally produces an effective search direction without the shift required by Levenberg–Marquardt (Cai et al., 2024).

In nonsmooth inverse problems, Gauss–Newton influence is stabilized through proximal regularization and relaxation. For

$x_{k+1}=x_k+\Delta x_k$5

the relaxed inexact proximal Gauss–Newton method solves at iteration $x_{k+1}=x_k+\Delta x_k$6

$x_{k+1}=x_k+\Delta x_k$7

then updates

$x_{k+1}=x_k+\Delta x_k$8

The convergence proof hinges on a quadratic linearization error bound

$x_{k+1}=x_k+\Delta x_k$9

and shows convergence to a disjoint component of Clarke-critical points (Jauhiainen et al., 2020).

These variants support a general pattern: Gauss–Newton influence is often amplified when the original problem is rewritten so that the Jacobian-induced term becomes separable, positive definite on the relevant subspace, or cheaply invertible.

5. Domain-specific manifestations

The concrete role of Gauss–Newton influence differs by domain, but a consistent pattern emerges.

Domain	Role of Gauss–Newton influence	Representative paper
Full waveform inversion	Data-space deblurring, source/receiver-side curvature separation, extended-model robustness	(Gholami, 2023)
Shallow ReLU approximation	SPD layer curvature for moving breaking hyperplanes	(Cai et al., 2024)
Variational neural PDEs	Local Newton-like acceleration near semiregular zeros	(Hao et al., 2023)
Conformal prediction	Add-one-in retraining approximation without full retraining	(Tailor et al., 27 Jul 2025)
Recursive filtering	Information-form recursive update with LM damping	(Nadjiasngar et al., 2011)
Phase retrieval	Minimal-norm Gauss–Newton step orthogonal to phase ambiguity	(Huang, 2024)
Reinforcement learning with MPC	Superlinear policy optimization without second-order policy sensitivities	(Brandner et al., 4 Aug 2025)

In variational PDE solvers with neural-network discretizations, the Gauss–Newton iteration

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$0

is analyzed near semiregular zeros of the vanishing gradient. The theory yields local fast convergence, with quadratic behavior in the early phase and very fast linear convergence asymptotically, and numerically it attains much lower $E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$1 and $E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$2 errors than SGD, Adam, or L-BFGS in one- and two-dimensional elliptic problems (Hao et al., 2023).

In approximate full conformal prediction, Gauss–Newton influence and network linearization make each residual an affine function of the candidate label,

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$3

which recovers the piecewise linear structure of conformalized ridge regression and avoids retraining the model for every candidate label (Tailor et al., 27 Jul 2025).

In recursive state estimation, the batch Gauss–Newton filter yields an information matrix

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$4

and information vector

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$5

The recursive formulation updates these as

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$6

$E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$7

which is a Gauss–Newton information filter equipped with Levenberg–Marquardt damping via $E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$8 (Nadjiasngar et al., 2011).

In complex phase retrieval, the modified Gauss–Newton step achieves asymptotic quadratic convergence without sample splitting under $E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2$9 complex Gaussian measurements by staying in a region of incoherence and contraction and using the minimal-norm pseudoinverse solution of the lifted Gauss–Newton system (Huang, 2024).

In reinforcement learning with model predictive control, the deterministic policy Hessian approximation of Kordabad et al. splits into $$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$0, and the paper proposes the Gauss–Newton approximation

$$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$1

This removes the need for second-order policy derivatives, preserves superlinear convergence near $$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$2, and is combined with momentum-based Hessian averaging for robustness under noisy estimates (Brandner et al., 4 Aug 2025).

6. Limitations, misconceptions, and open questions

A common misconception is that Gauss–Newton uniformly dominates Newton because it is “closer to the Hessian.” The cited literature is more precise. In several settings Gauss–Newton succeeds because the omitted term is small near the solution, because symmetry makes a pseudoinverse step more meaningful than a raw Newton step, or because the Jacobian-induced projection improves function-space dynamics. In the function-space analysis of neural networks, the distinction is rigorous: Newton contains second derivatives of the network that can distort prediction dynamics, whereas Gauss–Newton follows projected function-space directions with parameterization effects removed by error whitening (McKay et al., 11 May 2026).

Another misconception is that Gauss–Newton is always positive definite. It is only positive semidefinite in general and may be rank-deficient because of symmetries, overparameterization, or sample-wise rank collapse. Complex phase retrieval provides a direct example through the global phase null direction (Huang, 2024). Incremental Gauss–Newton descent in per-sample least squares faces the same issue, because the sample Gauss–Newton matrix is rank one; the paper resolves this by regularizing the null space and obtains the scalar preconditioner

$$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$3

which converts the Gauss–Newton step into a scaled SGD update (Korbit et al., 2024).

Open questions also recur. In FWI, the optimal subsurface-offset range and weighting $$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$4 remain open, as does time-domain extension with large $$\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,$$5 factors (Gholami, 2023). In conformal prediction, approximation error from linearization and Gauss–Newton curvature means coverage is no longer exact, and analogous bounds to Newton-step influence are not fully developed (Tailor et al., 27 Jul 2025). In deep-learning geometry, extensions from shallow smooth networks or deep linear models to deep nonsmooth architectures, stochastic settings, and beyond-near-initialization regimes remain unresolved (Cayci, 2024). In motion optimization, the strong asymptotic theorem covers a broad but still specific class of derivative-based task-space objectives, while the Cholesky-based workaround for unknown task maps remains approximate (Ratliff et al., 2016).

Taken together, these results suggest that Gauss–Newton influence is best understood not as a single algorithmic trick but as a general principle: when the dominant curvature of a problem is induced by how outputs depend on parameters, Jacobian-based curvature can govern optimization, sensitivity analysis, and even uncertainty quantification more effectively than raw second derivatives. The precise benefit, however, depends on structure—task maps, symmetries, separability, data geometry, and the extent to which the neglected terms vanish, average out, or can be regularized away.