Proximal Regularized Gauss–Newton Framework

• The PRGN framework extends the Gauss–Newton method by incorporating proximal mappings to handle nonsmooth regularization in composite optimization problems.
• It blends quadratic local modeling with adaptive proximal updates, ensuring enhanced convergence properties and stability in solving large-scale inverse problems.
• Practical implementations utilize efficient solvers for constrained, high-dimensional settings, with applications in imaging, structured statistical estimation, and signal processing.

The Proximal Regularized Gauss–Newton (PRGN) framework refers to a family of numerical algorithms for solving nonlinear least-squares and more general composite optimization problems that combine nonlinear data-fitting with convex (often nonsmooth) regularization and/or constraints. These algorithms extend the classical Gauss–Newton method by incorporating proximity operators or proximal mappings to handle nonsmooth regularization, enabling the efficient and stable solution of complex inverse problems, structured regression, and high-dimensional statistical estimation.

1. Mathematical Formulation and Origins

The core problem addressed by the PRGN framework is the minimization of functionals of the form

$$\min_{x \in X} \ \varphi(x) = \|F(x) - y\|^2 + J(x)$$

where $F: X \to Y$ is a nonlinear Fréchet differentiable operator between Hilbert spaces, and $J: X \to [0, +\infty]$ is a proper, convex, lower semicontinuous penalty functional (potentially nonsmooth or an indicator of a convex constraint) (1103.0414). When $J \equiv 0$, this reduces to the standard nonlinear least-squares problem, typically approached by the Gauss–Newton method.

The PRGN approach generalizes the Gauss–Newton update by incorporating the regularization term $J$ via proximal mappings. At iteration $n$, given a current iterate $x_n$, the update is defined as

$$x_{n+1} = \operatorname{prox}_{J}^{H(x_n)}\big(x_n - [F'(x_n)^* F'(x_n)]^{-1} F'(x_n)^* (F(x_n) - y)\big)$$

where $F'(x_n)$ is the Fréchet derivative and $H(x_n) = F'(x_n)^* F'(x_n)$. The proximity operator with respect to the metric $H$ is

$$\operatorname{prox}_{J}^H(z) = \arg\min_x \left\{ J(x) + \tfrac{1}{2} \|x - z\|_H^2 \right\}$$

with $\|v\|_H = \sqrt{\langle v, H v \rangle}$.

This framework enables the solution of penalized or constrained nonlinear least-squares problems and coincides with the classical Gauss–Newton step in the absence of regularization.

2. Algorithmic Structure and Extensions

The PRGN framework unifies and extends both Gauss–Newton and modern proximal (splitting) algorithms by:

• Decoupling Data Fidelity and Regularization: The quadratic model for $F$ is updated at each iteration, and the regularization or constraint $J$ is imposed through the proximal operator. This allows for flexible inclusion of sparsity constraints, group penalties, indicator functions, or structure-promoting regularizers (1103.0414, 1301.1459).
• Metric Adaptation: The use of variable-metric proximal operators (with metric $H(x_n)$) enables adaptation to the local curvature of the data-fidelity term, yielding improved convergence properties and robustness in ill-posed or ill-conditioned settings.
• Generalization to Composite Models: The approach seamlessly generalizes to problems involving a smooth (possibly self-concordant or nonconvex) loss plus a nonsmooth convex regularizer:

$\min_x \{ f(x) + g(x) \}$

where, for example, $f$ is self-concordant and $g$ is ℓ₁-norm or group norm regularization (for sparsity), or is the indicator of a constraint set (1301.1459).

Algorithmic refinements include adaptive step-size selection, local or global convergence controls, and exploiting duality to avoid expensive matrix inversions or decompositions (1301.1459).

3. Convergence Theory

The convergence of PRGN algorithms has been analyzed under a variety of regularity conditions:

• Generalized Lipschitz and Majorant Conditions: Under injectivity and closed range of $F'(x^*)$ at the solution $x^*$, as well as a (possibly generalized) Lipschitz condition on the derivative, local convergence is established. The contraction estimate is

$$\|x_{n+1} - x^*\| \leq q(\|x_n - x^*\|) \|x_n - x^*\|$$

with an explicitly computable function $q$, enabling estimates for the convergence basin (radius) (1103.0414). Error estimates of the form

$$\|x_{n+1} - x^*\| \leq C_2 \|x_n - x^*\|^2 + C_1 \|x_n - x^*\|$$

are derived, and quadratic local convergence can be obtained if the data fit is exact ($F(x^*) = y$).

• Majorant Function Techniques: Replacement of the classical Lipschitz assumption for $F'$ by a majorant condition allows broader classes of data-fidelity terms, including those with analytic structure (Smale’s condition) (1304.6461). This leads to quadratic or superlinear convergence under suitable bounds.
• Composite and Structured Nonconvex Problems: For extensions to nonconvex and composite models, as in the inexact regularized proximal Newton methods, convergence (global and superlinear) is established under Kurdyka–Łojasiewicz properties or local Hölderian error bounds, even in the absence of uniform strong convexity (Liu et al., 2022, Dahl et al., 3 Apr 2024).
• Manifold Identification and Acceleration: In problems where the regularizer induces an active manifold (e.g., sparsity or low-rank structure), PRGN steps can identify this manifold, at which point the algorithm locally reduces to a smooth problem and enables quadratic convergence via Riemannian Newton updates (Bareilles et al., 2020).

4. Practical Implementation Considerations

PRGN algorithms are designed for efficient numerical implementation in large-scale and high-dimensional settings:

• Handling Constraints: When $J$ is the indicator of a convex set, the proximal step becomes a projection with respect to the metric $H$. For box constraints or simplex constraints, closed-form or fast iterative projections are often available (1103.0414, Alberti et al., 22 Jul 2025).
• Efficient Solvers for Proximal Steps: For certain regularizers (e.g., ℓ₁ or group norms), specialized algorithms (e.g., semismooth Newton methods, dual augmented Lagrangian schemes) are employed to solve the inner proximal subproblems efficiently (Liu et al., 2022, Kanzow et al., 2022).
• Avoidance of Expensive Decompositions: Dual formulations, analytic step-size rules, and limited-memory approximations can circumvent the need for Cholesky decompositions or matrix inversions, improving scaling for large problems (graph learning, covariance estimation) (1301.1459, Kanzow et al., 2022).
• Adaptive Regularization and Globalization: Regularization parameters or trust-region radii are adjusted adaptively—often based on achieved reduction (actual vs. predicted) or on stationarity residuals—enabling robust global convergence without classical line search (Kanzow et al., 2022, Dahl et al., 3 Apr 2024).
• Unrolling as Deep Networks: In certain applications, such as multi-frequency EIT, the entire PRGN iteration is unrolled as layers in a deep network (deep unfolding), with proximal steps replaced by learned denoisers (e.g., GNNs), thus merging numerical optimization with data-driven learning (Alberti et al., 22 Jul 2025).

5. Representative Applications

The PRGN methodology has been applied to a range of practical problems:

• Inverse Problems and Imaging: Constrained and regularized nonlinear inverse problems, such as Electrical Impedance Tomography (EIT), where the PRGN framework ensures enforcement of physical constraints (e.g., nonnegativity, simplex structure) and robustness to modeling errors (Alberti et al., 22 Jul 2025).
• Structured Statistical Estimation: Sparse inverse covariance estimation (graphical LASSO, covariance selection) where high-dimensionality, sparsity, and positive-definiteness are handled efficiently (1301.1459).
• Signal Processing and Robust Regression: Group-sparse regression, Student's t-penalized estimation, and nonconvex image restoration, where nonsmooth penalties and heavy-tailed modeling require stable handling by PRGN-type methods (Liu et al., 2022, Kanzow et al., 2022).
• Matrix Manifold and Geometric Optimization: Composite objectives over the Stiefel manifold or similar matrix manifolds, where quadratic models and projection-like retractions produce globally convergent and superlinearly convergent algorithms (Wang et al., 17 Apr 2024).

The introduction of deep unfolding networks for nonlinear inverse problems, such as in multi-frequency EIT, showcases the integration of PRGN steps with learned proximal maps realized as graph neural networks on irregular computational meshes. This approach achieves improved reconstruction accuracy and interpretability by embedding physical models and learning on structured data (Alberti et al., 22 Jul 2025).

6. Variations, Strengths, and Limitations

Key strengths of the PRGN framework include adaptability to nonsmooth and structured regularization, local quadratic convergence under suitable regularity, scalability to large dimensions, and robust enforcement of constraints. The ability to switch between global convergence (using adaptive regularization, majorant or error bound assumptions) and very fast local rates (once in the attraction basin or when the manifold is identified) is highlighted in both theoretical and empirical results (1103.0414, Bareilles et al., 2020).

Typical limitations are:

• Dependence on local regularity for quadratic convergence; global convergence may rely on sufficiently strong error bounds or KL properties.
• Proximal operator computation can be challenging for general or composite regularizers, though dual methods and semismooth Newton strategies alleviate this for many applications (Liu et al., 2022, Kanzow et al., 2022).
• Manifold identification and switching between first-order and Newton-type updates require robust criteria, especially in high-dimensional or nonconvex scenarios (Bareilles et al., 2020).

7. Summary Table: Core Algorithmic Ingredients and Application Contexts

Algorithmic Theme	Description	Representative Application
Gauss–Newton + Proximal	Quadratic local modeling + nonsmooth regularizer	Penalized nonlinear least squares (1103.0414)
Majorant/Lipschitz/Analytic Assumptions	Flexible convergence guarantees via relaxed conditions	Analytic or non-Lipschitz F' (1304.6461)
Self-concordance Exploitation	Analytic step-size, domain preservation	Graph learning (1301.1459)
Dual/Low-Rank Solvers	Matrix inversion/descent avoidance	High-dimensional estimation (1301.1459, Kanzow et al., 2022)
Dynamic Regularization	Adaptive, residual-based metric regularization	Nonconvex regression (Liu et al., 2022)
Deep Unfolding Integration	Unrolled iterative blocks as NN layers	Physics-driven imaging (Alberti et al., 22 Jul 2025)
Manifold Identification	Switching to Riemannian Newton steps	Sparse/low-rank learning (Bareilles et al., 2020)

The Proximal Regularized Gauss–Newton (PRGN) framework thus forms a foundational approach for modern large-scale and structured nonlinear optimization, underpinning both classical regularized inverse problems and emerging hybrid paradigms incorporating data-driven models. Its methodological and theoretical developments continue to inform the design of robust, efficient, and interpretable optimization algorithms in scientific computing, signal processing, statistical learning, and computational imaging.