---
title: Gauss–Newton Influence in Optimization
url: https://www.emergentmind.com/topics/gauss-newton-influence
type: topic
---

# Gauss–Newton Influence in Optimization

Gauss–Newton influence denotes the family of effects induced by replacing an exact Hessian with a Gauss–Newton or generalized Gauss–Newton curvature model, and by using that model either to define search directions, to characterize optimization geometry, or to approximate perturbations of fitted parameters. In least-squares settings this curvature takes the classical form \(J^\top J\); in more general settings it appears as \(J^\top H_\ell J\), data-space factorizations, or sample-wise rank-one quantities. Across inverse problems, neural-network training, PDE-constrained optimization, motion optimization, reinforcement learning, and post-hoc uncertainty quantification, the common theme is that Gauss–Newton structure changes how error signals are scaled, projected, or propagated, often by suppressing troublesome second-order terms, exposing separable curvature, or yielding symmetric positive definite subproblems [2302.04124].

## 1. Classical meaning and scope

In nonlinear least squares with residual vector \(r(x)\), the exact Hessian decomposes into a positive semidefinite Jacobian term and a residual-weighted second-derivative term, while the Gauss–Newton approximation keeps only the Jacobian part. In the weighted least-squares formulation used for recursive filtering, this yields the normal equations
\[
\big(J_r(x_k)^T W J_r(x_k)\big)\, \Delta x_k = - J_r(x_k)^T W\, r(x_k),
\]
with the Hessian approximation \(J_r(x_k)^T W J_r(x_k)\) and update \(x_{k+1}=x_k+\Delta x_k\) [1110.5212]. In full waveform inversion (FWI), the corresponding reduced objective
\[
E(m)=\frac12\sum_{s=1}^{N_s}\|F_s(m)-d_s\|_2^2
\]
leads to the Gauss–Newton system
\[
\Big[\sum_{s=1}^{N_s} J_s^T J_s\Big]\delta m = -\sum_{s=1}^{N_s} J_s^T \delta d_s,
\]
where \(J_s\) is the Fréchet derivative of the forward map for source \(s\) [2302.04124].

The term is also used in a more specific influence-function sense. In neural network regression for full conformal prediction, Gauss–Newton influence refers to a local retraining approximation in which the effect of adding a candidate test label is modeled by a single Newton-style perturbation using a Gauss–Newton matrix rather than the full Hessian. The resulting parameter perturbation is
\[
\hat{\theta}_*^+(y) = \theta_* +  \frac{\hat{e}_{N+1}(y)}{1+\hat{h}_{N+1}}\, H_{\mathrm{GN}}^{-1} J_{N+1},
\]
with \(\hat{h}_{N+1}=J_{N+1}^\top H_{\mathrm{GN}}^{-1}J_{N+1}\) [2507.20272].

This dual usage suggests two closely related meanings. One is curvature-based: Gauss–Newton influence is the way \(J^\top J\) or \(J^\top H_\ell J\) shapes local optimization. The other is perturbational: it is the way the same curvature object controls first-order approximations of parameter re-optimization under data perturbations. In both senses, the central object is the Jacobian-induced metric rather than the full second derivative tensor.

## 2. Core algebraic mechanism

For least-squares neural-network or inverse-problem objectives, the standard rationale for Gauss–Newton is that the exact Hessian can be written as a sum of a Jacobian Gram term and a residual-dependent remainder. In the variational PDE setting,
\[
H(\theta)=J(\theta)+Q(\theta),
\]
with
\[
J(\theta)=\int_{\Omega} \big(\nabla_\theta\nabla u\big)\big(\nabla_\theta\nabla u\big)^T + \big(\nabla_\theta u\big)\big(\nabla_\theta u\big)^T\,dx,
\]
and \(Q(\theta)\) collecting second derivatives of the network output with respect to parameters. Near a good minimizer, the paper shows that \(Q(\theta)\) can be made arbitrarily small in a neighborhood of \(\theta^*\), so the Hessian is well approximated by \(J(\theta)\) [2306.08727].

In shallow ReLU approximation, the same decomposition appears at the layer level. The Hessian with respect to nonlinear hidden-layer parameters splits into a principal Gauss–Newton piece and a residual term involving Dirac distributions on breaking hyperplanes. The Gauss–Newton matrix is
\[
\mathcal{H}(z)=\int_\Omega \mu(x)\,[H(x)H(x)^T]\otimes[xx^T]\,dx,
\]
and the full Hessian keeps an additional residual term \(\widehat{\mathcal{H}}\) that is discarded in the Gauss–Newton approximation [2404.05064].

In phase retrieval, the mechanism is modified by symmetry. The lifted Gauss–Newton matrix \(A(z)^*A(z)\) is rank-deficient because the infinitesimal global phase direction lies in its null space:
\[
A(z)\begin{bmatrix}\mathbf i z\\ -\mathbf i \bar z\end{bmatrix}=0.
\]
The Gauss–Newton step is therefore defined as the minimal-norm solution orthogonal to the trivial phase direction, equivalently via the pseudoinverse \(A(z)^\dagger\) [2406.09903].

A broad misconception is that Gauss–Newton is merely a crude Hessian surrogate. Several of the cited works show a more specific structure. In motion optimization, for objective terms built from finite-differenced task-space derivatives,
\[
F(\xi)=\sum_{t=1}^T f(D^{k_1}z_t^c,\dots,D^{k_l}z_t^c)\,\Delta t,
\]
the exact Hessian converges to the Gauss–Newton Hessian as \(\Delta t\to 0\) at rate \(O(\Delta t^{2k})\), where \(k=\min_i k_i\). For squared task-space velocity and acceleration, the rates are \(O(\Delta t^2)\) and \(O(\Delta t^4)\), respectively [1605.09296]. This suggests that, in such discretized trajectory objectives, the neglected tensor term is asymptotically negligible rather than intrinsically dominant.

## 3. Geometry, conditioning, and projection effects

One recurrent interpretation of Gauss–Newton influence is geometric: it replaces parameter-space curvature dominated by second derivatives of the model with function-space or output-space curvature induced by the Jacobian. In the underparameterized smooth-network regime, Gauss–Newton flow induces a Riemannian gradient flow on the embedded manifold
\[
\mathcal{M}=\alpha f(B)\subset\mathbb R^n,
\]
and the output-space dynamics are
\[
\dot{\alpha f(w_t)}=-P(\alpha f(w_t))\,\nabla g(\alpha f(w_t)),
\]
where
\[
P(\alpha f(w))=Df(w)\big(D^\top f(w)Df(w)\big)^{-1}D^\top f(w)
\]
is the orthogonal projector onto the tangent space \(T_{\alpha f(w)}\mathcal M\) [2412.14031]. In this formulation, Gauss–Newton is literally the Riemannian gradient flow of the loss restricted to realizable outputs.

A function-space analysis goes further by contrasting Gauss–Newton and Newton. For empirical risk \(L(\theta)=\frac1d\sum_i \ell(f_\theta(x_i),y_i)\), the generalized Gauss–Newton matrix is
\[
G=J^\top H_\ell J,
\]
while the least-squares Gauss–Newton matrix is \(G_J=J^\top J\). The paper “Error whitening: Why Gauss-Newton outperforms Newton” argues that the generalized Gauss–Newton projects the Newton direction in function space onto the model’s tangent space, while the Jacobian-only variant projects the function-space loss gradient onto the same tangent space. In both cases, the dynamics of the prediction-target mismatch no longer depend on the parameterization through \(JJ^\top\); this replacement of \(JJ^\top\) by the identity on the reachable subspace is կոչված “error whitening” [2605.11316].

Conditioning results make this geometry quantitative. For deep linear networks trained with mean-squared error, the Gauss–Newton matrix
\[
G=\frac1n\sum_{i=1}^n \nabla_\theta F_\theta(x_i)\,\nabla_\theta F_\theta(x_i)^\top
\]
has a pseudo-condition number \(\kappa(G)=\lambda_{\max}(G)/\lambda_{\min}^+(G)\). Tight bounds show that \(\kappa(G)\) is controlled by the input covariance \(\Sigma_x\) and products of layer condition numbers, with residual connections strictly improving these bounds by shifting singular values through \(W^{(\ell)}+\beta I\). The same analysis extends to Toeplitz representations of convolutional layers and to a one-hidden-layer Leaky-ReLU setting [2411.02139]. This suggests that Gauss–Newton influence is not only algorithmic; it is also architectural, because depth, width, residual pathways, convolutional structure, and data whitening all change the curvature seen by Gauss–Newton.

## 4. Structured variants and explicit relaxations

Many modern uses of Gauss–Newton influence arise not from the plain \(J^\top J\) system but from structured relaxations that expose additional algebraic regularity.

In FWI, the standard Gauss–Newton Hessian can be rewritten as
\[
\sum_{s=1}^{N_s}J_s^T J_s = (S^T S)\circ(U^T U),
\]
and the usual vector update can be represented as the diagonal of a matrix equation
\[
\big[S^TS\,\operatorname{diag}(\delta m)\,U^TU\big]_{ii}=\big[S^T\Delta d\,U\big]_{ii}.
\]
Relaxing the diagonality constraint yields the Extended Gauss–Newton system
\[
S^TS\,\Delta m\,U^TU = S^T\Delta d\,U,
\]
whose regularized solution is
\[
\Delta m=(S^TS+\mu_S I)^{-1}S^T\Delta d\,U\,(U^TU+\mu_U I)^{-1}.
\]
Equivalently,
\[
\Delta m=S^T\Delta d^e U,\qquad \Delta d^e = H_r^{-1}\Delta d\,H_s^{-1},
\]
with small receiver- and source-side Hessians \(H_r=SS^T+\mu_S I\) and \(H_s=UU^T+\mu_U I\) [2302.04124]. Here Gauss–Newton influence becomes a two-dimensional deblurring in the source–receiver plane rather than a direct model-space solve.

In shallow ReLU learning, the structure-guided Gauss–Newton method alternates between an exact linear solve for output-layer parameters and a damped Gauss–Newton step for hidden-layer parameters. The mass matrix \(\mathcal A(z)\) and the layer Gauss–Newton matrix \(\mathcal H(z)\) are shown to be symmetric and positive definite under distinct-hyperplane assumptions, so the method naturally produces an effective search direction without the shift required by Levenberg–Marquardt [2404.05064].

In nonsmooth inverse problems, Gauss–Newton influence is stabilized through proximal regularization and relaxation. For
\[
J(x)=\tfrac12\|A(x)\|^2+F(x),
\]
the relaxed inexact proximal Gauss–Newton method solves at iteration \(k\)
\[
\min_x \tfrac12\|A_k(x)\|^2+F(x)+\tfrac\beta2\|x-z_k\|^2,
\]
then updates
\[
z_{k+1}=(1-\omega)z_k+\omega x_k.
\]
The convergence proof hinges on a quadratic linearization error bound
\[
\|A(x)-\tilde A_y(x)\|\le C\|x-y\|^2
\]
and shows convergence to a disjoint component of Clarke-critical points [2002.08044].

These variants support a general pattern: Gauss–Newton influence is often amplified when the original problem is rewritten so that the Jacobian-induced term becomes separable, positive definite on the relevant subspace, or cheaply invertible.

## 5. Domain-specific manifestations

The concrete role of Gauss–Newton influence differs by domain, but a consistent pattern emerges.

| Domain | Role of Gauss–Newton influence | Representative paper |
|---|---|---|
| Full waveform inversion | Data-space deblurring, source/receiver-side curvature separation, extended-model robustness | [2302.04124] |
| Shallow ReLU approximation | SPD layer curvature for moving breaking hyperplanes | [2404.05064] |
| Variational neural PDEs | Local Newton-like acceleration near semiregular zeros | [2306.08727] |
| Conformal prediction | Add-one-in retraining approximation without full retraining | [2507.20272] |
| Recursive filtering | Information-form recursive update with LM damping | [1110.5212] |
| Phase retrieval | Minimal-norm Gauss–Newton step orthogonal to phase ambiguity | [2406.09903] |
| Reinforcement learning with MPC | Superlinear policy optimization without second-order policy sensitivities | [2508.02441] |

In variational PDE solvers with neural-network discretizations, the Gauss–Newton iteration
\[
\theta_{k+1}=\theta_k-J(\theta_k)^\dagger \nabla_\theta L(\theta_k)
\]
is analyzed near semiregular zeros of the vanishing gradient. The theory yields local fast convergence, with quadratic behavior in the early phase and very fast linear convergence asymptotically, and numerically it attains much lower \(L^2\) and \(H^1\) errors than SGD, Adam, or L-BFGS in one- and two-dimensional elliptic problems [2306.08727].

In approximate full conformal prediction, Gauss–Newton influence and network linearization make each residual an affine function of the candidate label,
\[
R_i(y)\approx |a_i+b_i y|,
\]
which recovers the piecewise linear structure of conformalized ridge regression and avoids retraining the model for every candidate label [2507.20272].

In recursive state estimation, the batch Gauss–Newton filter yields an information matrix
\[
\mathbf W_n=\mathbf T_n^T\mathbf R_n^{-1}\mathbf T_n
\]
and information vector
\[
\xi_n=\mathbf T_n^T\mathbf R_n^{-1}\delta\mathbf Y_n.
\]
The recursive formulation updates these as
\[
\mathbf W_n=\lambda A(\bar X_{n-1})^{-T}\mathbf W_{n-1}A(\bar X_{n-1})^{-1}+M(\bar X_n)^T R^{-1}M(\bar X_n),
\]
\[
\xi_n=\lambda A(\bar X_{n-1})^{-T}\xi_{n-1}+M(\bar X_n)^T R^{-1}\delta Y_n,
\]
which is a Gauss–Newton information filter equipped with Levenberg–Marquardt damping via \((\mathbf W_n+\mu I)\delta\hat X_n=\xi_n\) [1110.5212].

In complex phase retrieval, the modified Gauss–Newton step achieves asymptotic quadratic convergence without sample splitting under \(m\ge O(n\log^3 n)\) complex Gaussian measurements by staying in a region of incoherence and contraction and using the minimal-norm pseudoinverse solution of the lifted Gauss–Newton system [2406.09903].

In reinforcement learning with model predictive control, the deterministic policy Hessian approximation of Kordabad et al. splits into \(M_1(\theta)+M_2(\theta)\), and the paper proposes the Gauss–Newton approximation
\[
B(\theta)=M_2(\theta)=\mathbb E_s\!\left[\nabla_\theta\pi_\theta^\top(s)\,\nabla_a^2 Q^{\pi_\theta}(s,a)\big|_{a=\pi_\theta(s)}\,\nabla_\theta\pi_\theta(s)\right].
\]
This removes the need for second-order policy derivatives, preserves superlinear convergence near \(\theta^\star\), and is combined with momentum-based Hessian averaging for robustness under noisy estimates [2508.02441].

## 6. Limitations, misconceptions, and open questions

A common misconception is that Gauss–Newton uniformly dominates Newton because it is “closer to the Hessian.” The cited literature is more precise. In several settings Gauss–Newton succeeds because the omitted term is small near the solution, because symmetry makes a pseudoinverse step more meaningful than a raw Newton step, or because the Jacobian-induced projection improves function-space dynamics. In the function-space analysis of neural networks, the distinction is rigorous: Newton contains second derivatives of the network that can distort prediction dynamics, whereas Gauss–Newton follows projected function-space directions with parameterization effects removed by error whitening [2605.11316].

Another misconception is that Gauss–Newton is always positive definite. It is only positive semidefinite in general and may be rank-deficient because of symmetries, overparameterization, or sample-wise rank collapse. Complex phase retrieval provides a direct example through the global phase null direction [2406.09903]. Incremental Gauss–Newton descent in per-sample least squares faces the same issue, because the sample Gauss–Newton matrix is rank one; the paper resolves this by regularizing the null space and obtains the scalar preconditioner
\[
\xi_t=\frac{1}{\|\nabla_{\mathbf w}f_{\mathbf w_t}(x_t)\|_2^2+\epsilon},
\]
which converts the Gauss–Newton step into a scaled SGD update [2408.05560].

Open questions also recur. In FWI, the optimal subsurface-offset range and weighting \(\phi(h)\) remain open, as does time-domain extension with large \(SS^T\) factors [2302.04124]. In conformal prediction, approximation error from linearization and Gauss–Newton curvature means coverage is no longer exact, and analogous bounds to Newton-step influence are not fully developed [2507.20272]. In deep-learning geometry, extensions from shallow smooth networks or deep linear models to deep nonsmooth architectures, stochastic settings, and beyond-near-initialization regimes remain unresolved [2412.14031]. In motion optimization, the strong asymptotic theorem covers a broad but still specific class of derivative-based task-space objectives, while the Cholesky-based workaround for unknown task maps remains approximate [1605.09296].

Taken together, these results suggest that Gauss–Newton influence is best understood not as a single algorithmic trick but as a general principle: when the dominant curvature of a problem is induced by how outputs depend on parameters, Jacobian-based curvature can govern optimization, sensitivity analysis, and even uncertainty quantification more effectively than raw second derivatives. The precise benefit, however, depends on structure—task maps, symmetries, separability, data geometry, and the extent to which the neglected terms vanish, average out, or can be regularized away.

Source: https://www.emergentmind.com/topics/gauss-newton-influence