Inexact Newton Regularization

• Inexact Newton regularization is an approach that computes approximate Newton steps with a fixed multiplicative error criterion to balance computational efficiency and solution accuracy.
• It employs regularized Hessian approximations and adaptive subproblem solvers to ensure global convergence and optimal complexity in both convex and nonconvex settings.
• This framework is widely applied in inverse problems, machine learning, and image processing, where adaptive error control and efficient computations yield significant practical benefits.

Inexact Newton regularization refers to a broad class of algorithms for nonlinear optimization and inverse problems in which Newton-type (second-order) steps are not computed exactly, but are regularized and solved only approximately at each iteration. This inexactness can take the form of approximate solutions to the regularized (local) Newton subproblem, truncated or stabilized linear algebra in the Newton system, and/or the use of approximate derivatives. Such strategies are essential for large-scale, ill-posed, nonsmooth, or otherwise computationally demanding settings, and are theoretically grounded by global convergence and complexity results that account for the measure of inexactness.

1. Mathematical Setting and Fundamental Principles

Inexact Newton regularization methods target composite minimization problems of the form

$\min_{x \in \mathbb{R}^n} \ F(x) = f(x) + \psi(x),$

where $f$ is typically smooth (possibly nonconvex) with Lipschitz gradient, and $\psi$ is closed proper convex, possibly nonsmooth. The general iteration at step $k$ forms a quadratic (second-order) model around $x_k$:

$$m_k(d) = \langle \nabla f(x_k), \, d \rangle + \frac{1}{2} d^T H_k d + \psi(x_k + d),$$

where $H_k$ is a symmetric positive definite approximation of $\nabla^2 f(x_k)$, ranging from the true Hessian (Newton), through quasi-Newton surrogates (e.g., L-BFGS), to diagonal or block-diagonal approximations (Lee et al., 2018).

The algorithm seeks an inexact minimizer $d_k$ of this model according to a multiplicative inexactness criterion:

$m_k(d_k) \leq (1-\eta) m_k^*,$

where $f$0 and $f$1 is fixed (Lee et al., 2018). This form automatically tightens as iterates approach optimality, unlike additive error schemes.

After obtaining $f$2, a globalization step is performed (e.g., backtracking line search or adaptive parameter update) to ensure sufficient decrease in $f$3.

2. Regularization, Inexactness, and Subproblem Solvers

The regularization strategy is indirect but crucial: the local model $f$4 is enforced to be strongly convex (eigenvalues in $f$5), and for nonconvex or ill-posed problems, further regularization is incorporated, e.g., cubic or trust-region terms:

$f$6

or Tikhonov-type constraints in the linearized system for inverse problems (Ghadimi et al., 2017, Jin, 2011).

Inexactness is central:

• Subproblem solutions can be iterative, often carried out to only moderate precision, and the iteration complexity is controlled via $f$7.
• Hessians and gradients may be approximated by subsampling, finite differencing, or limited-memory surrogates, with explicit error bounds entering global rates (Yao et al., 2018, Ghadimi et al., 2017).
• For ill-posed inverse problems, the linearized Newton equations are themselves solved inexactly by inner regularization schemes (Landweber, Tikhonov, etc.) (Jin et al., 2010, Jin, 2011).

The allowed inexactness regime is formalized either by residual (norm of the KKT or stationarity measure), functional optimality gap, or geometric criteria (model decrease relative to Cauchy or negative-curvature direction).

3. Global Convergence and Complexity Theory

The inexact Newton regularization framework admits rigorous global convergence results under mild conditions:

• For convex $f$8 with optimal-set strong convexity, linear convergence is achieved with rate $f$9, where $\psi$0 depends on problem constants and the line search/trust-region parameters (Lee et al., 2018).
• For general convex (but not strongly convex) $\psi$1, convergence is $\psi$2, with an initial linear phase if suboptimality is large.
• For nonconvex problems, the minimal prox-Newton residual over $\psi$3 steps satisfies $\psi$4 (Lee et al., 2018).
• In exact-Newton, ARC, or trust-region frameworks, worst-case global evaluation complexity is $\psi$5 for first-order stationarity and $\psi$6 for second-order stationarity, under optimal subproblem inexactness regimes (Curtis et al., 2017, Yao et al., 2018).

A distinctive insight is that a fixed multiplicative inexactness $\psi$7 is sufficient for global guarantees—no need to drive subproblem solution errors to zero as iterations proceed. This results in a fixed per-iteration cost for the inner solver, with total performance degraded only by a moderate function of $\psi$8 (e.g., $\psi$9 factor in rates) (Lee et al., 2018).

4. Algorithmic Variants and Regularization in Practice

A wide variety of algorithmic instantiations fit into this framework:

Variant	$k$0 Structure	Step Selection	Regularization
Proximal gradient	$k$1 ($k$2 a global Lipschitz)	Unit step	Implicit
Proximal Newton	$k$3	Backtracking	Possibly damped
Proximal quasi-Newton	$k$4 from L-BFGS, spectral bounds	Line/trust reg.	Implicit/dynamic
Inexact trust-region	Problem-adapted $k$5, trust region	Trust-update	Quadratic
ARC (cubic reg.)	$k$6 plus $k$7 term	ARC update	Cubic
Inverse problem	$k$8 from derivative, regularized	Discrepancy	Tikhonov etc.

Efficient subproblem solvers (proximal-gradient, coordinate descent, primal-dual splitting) are employed, with stopping criteria aligned to the required inexactness. Practical implementations exploit adaptive setting of regularization and step-size, manifold identification (for partial smoothness) (Lee, 2020), and dimension-reduced subproblems post active-set stabilization (Wu et al., 2023).

5. Regularization in Inverse Problems and Ill-posed Settings

In the context of nonlinear or ill-posed inverse problems, inexact Newton regularization is a primary methodology:

• Each Newton step linearizes the forward operator $k$9 at $x_k$0 and applies a regularized inversion method (Landweber, implicit iteration, Tikhonov, or filter-based approaches) to compute the step.
• Regularization is achieved via stopping rules, discrepancy principles, or explicit regularization in the linearized system.
• Under source conditions on the true solution and Hilbert-scale smoothness, such methods achieve order-optimal convergence rates (e.g., $x_k$1, where $x_k$2 is the data noise, $x_k$3 the degree of ill-posedness, $x_k$4 the source smoothness) (Jin et al., 2010, Jin, 2011).
• The applicability spans both smooth and nonsmooth regularizers, variational models, and high-dimensional discretizations.

6. Advanced Developments: Manifold Identification and Superlinearity

Recent advances have established that inexact Newton-type regularization, when applied to problems with partly smooth regularizers, enjoys finite iteration manifold identification: after a finite number of iterations (under generic conditions and moderate inexactness), the iterates lock onto the active manifold where the regularizer is smooth. This enables a regime switch to smooth Newton acceleration (locally fast rates and bounded per-iteration time), even in degenerate solution structures or when classical second-order sufficiency is not satisfied (Lee, 2020).

Local convergence can be controlled by tuning inexactness: unit-step acceptance and driving inexactness relative to the KKT residual yield Q-superlinear or quadratic convergence in favorable cases (Byrd et al., 2013).

7. Practical Remarks and Numerical Considerations

• The overall performance of inexact Newton regularization approaches is robust to the level of inner inexactness, provided theoretical conditions on $x_k$5 or equivalent are met.
• Per-iteration computational costs can be minimized by balancing inner (subproblem) solution accuracy with outer (global) rate requirements, often leading to dramatic savings over exact or high-accuracy solvers (Yao et al., 2018, Allaire et al., 16 Dec 2025).
• Inverse problems, machine learning, image processing, and large-scale signal estimation are domains where these frameworks realize competitive or state-of-the-art efficiency.

In summary, inexact Newton regularization unifies and extends second-order optimization methodologies with explicit regularization and error tolerance controls. It is supported by a comprehensive theory guaranteeing global convergence and optimal or near-optimal complexity in convex, nonconvex, and ill-posed settings (Lee et al., 2018, Ghadimi et al., 2017, Yao et al., 2018, Byrd et al., 2013, Jin et al., 2010, Jin, 2011, Wu et al., 2023).