---
title: Gauss-Newton-Bartlett Estimator
url: https://www.emergentmind.com/topics/gauss-newton-bartlett-gnb-estimator
type: topic
---

# Gauss-Newton-Bartlett Estimator

The Gauss-Newton-Bartlett (GNB) estimator refers to a structured diagonal Hessian approximation used in iterative algorithms for large-scale nonlinear least squares (NLS) optimization. The GNB estimator, as detailed in [2002.01871], exploits the particular form of the Hessian in NLS problems by constructing a diagonal approximation that is updated iteratively via a secant equation and robust safeguarding strategies. This makes it particularly suitable for scalable, matrix-free implementations, preserving descent directions and ensuring positive definiteness of the approximate Hessian.

## 1. Mathematical Foundation and Problem Structure

Nonlinear least squares problems take the standard form
\[
\min_{x \in \mathbb{R}^n} f(x), \qquad f(x) = \frac{1}{2} \| F(x) \|^2,
\]
where \( F : \mathbb{R}^n \to \mathbb{R}^m \). The gradient and Hessian are structured as
\[
g(x) = J(x)^\top F(x),
\]
\[
H(x) = J(x)^\top J(x) + C(x), \qquad C(x) = \sum_{i=1}^{m} F_i(x) \nabla^2 F_i(x),
\]
with \( J(x) \) the Jacobian matrix of \( F \).

The GNB estimator constructs a **diagonal matrix \( D_k \) that approximates the Hessian at iteration \( k \)**, aiming to accurately capture the leading curvature information necessary for Newton-type or quasi-Newton methods, while sidestepping the cost of storing or manipulating the full Hessian. This approach contrasts with traditional Gauss-Newton (GN) methods, which drop the second term and use \( J^\top J \), and with generic diagonal BFGS updates that are agnostic to the Hessian's NLS structure.

## 2. Structured Diagonal Hessian Approximation: GNB Update Rule

At each iteration, the update for the diagonal entries is derived from a componentwise secant condition specific to NLS structure. Define
\[
s_{k-1} = x_k - x_{k-1}, \qquad
y_{k-1} = J_k^\top J_k s_{k-1} + (J_k - J_{k-1})^\top F_k = \hat{y}_{k-1} + \overline{y}_{k-1}.
\]
The GNB estimator seeks \( h_k^i \), the \( i \)-th element of \( D_k \), via
\[
D_k s_{k-1} \approx y_{k-1},
\]
implemented by solving a **constrained least-squares problem** for each component:
\[
h_k^i =
\begin{cases}
\frac{y_{k-1}^i}{s_{k-1}^i}, & s_{k-1}^i \neq 0, \\
1, & s_{k-1}^i = 0.
\end{cases}
\]
Robustness and numerical stability require **safeguarded projection**:
\[
h_k^i =
\begin{cases}
\frac{y_{k-1}^i}{s_{k-1}^i}, & l \le \frac{y_{k-1}^i}{s_{k-1}^i} \le u, \\
l, & \frac{y_{k-1}^i}{s_{k-1}^i} < l, \\
u, & \frac{y_{k-1}^i}{s_{k-1}^i} > u, \\
1, & s_{k-1}^i = 0,
\end{cases}
\]
with bounds \( l = 10^{-30} \), \( u = 10^{30} \), and further adjustments if the numerator and denominator are of opposite sign (to enforce positivity).

This update succinctly incorporates the structure of the NLS Hessian (including changes in \( J \)), making it **distinct from standard iterative diagonal approximation rules** used in generic quasi-Newton frameworks.

## 3. Iterative Algorithm and Workflow

The GNB estimator is embedded in a globally convergent matrix-free optimization framework:

1. **Residual and Jacobian computations:** At each step, compute \( F_k \) and Jacobian–vector products \( J_k v \) (without storing \( J_k \) explicitly, enabling scalability).
2. **Search direction:** Solve for \( d_k = -H_k^{-1} g_k \), where \( H_k = \text{diag}(h_k^i) \).
3. **Non-monotone line search:** Apply the Zhang–Hager rule to select an adaptive step size \( \alpha_k \).
4. **Update iterate:** \( x_{k+1} = x_k + \alpha_k d_k \).
5. **Safeguard and update diagonal:** Use the GNB estimator to update \( D_{k+1} \).
6. **Convergence test:** Stop when \( \|g_k\| \leq \epsilon \).

## 4. Convergence Properties and Robustness

The GNB estimator is equipped with algorithmic guarantees:

- **Descent direction:** The safeguarded diagonal ensures \( D_k \) is positive definite, thus \( d_k \) is always a descent direction.
- **Global convergence:** Under standard NLS assumptions (bounded level sets; Lipschitz continuity of \( F \) and \( J \)), the algorithm satisfies
  \[
  \liminf_{k \to \infty} \|g_k\| = 0.
  \]
  With line search parameters constrained (\( \eta_{\max} < 1 \)), full convergence follows:
  \[
  \lim_{k \to \infty} \|g_k\| = 0.
  \]

## 5. Numerical Performance and Scaling

The GNB estimator demonstrates superior numerical performance on high-dimensional NLS problems:

- On a 30-problem test suite (dimensions up to \( n = m = 10^4 \)), the GNB-based solver (ASDH) solves 80% of problems with minimal overall work (iterations, function evaluations, matrix–vector products), and 70% with minimal CPU time.
- **Robustness:** ASDH solves all instances without failure; a comparable structured diagonal method failed on three problems.
- **Efficiency:** In nearly all cases, ASDH (using GNB) converges with fewer iterations and lower CPU cost than matrix-free alternatives utilizing less structured diagonal approximations.

## 6. Implementation and Practical Considerations

**Implementation highlights:**

- **Matrix-free:** Only Jacobian–vector products are required; full storage or construction of \( J \) or \( H \) is avoided.
- **Safeguarding:** Critical for global convergence and stability—implemented componentwise with explicit bounds and positivity enforcement.
- **Parallelization:** The structure naturally permits parallel application in each diagonal entry update and search direction computation.
- **Resource utilization:** Suited for large-scale settings (potentially \( n, m \gg 10^3 \)), benefitting from the reduced storage and computational complexity compared to full-matrix or even banded methods.

**Limitations:**

- The diagonal structure limits the rate at which the method can exploit strong off-diagonal curvature (non-separability), particularly relevant for extremely ill-conditioned or highly coupled NLS problems.
- The method is tailored for NLS; adaptation to general unconstrained smooth optimization may lose beneficial structure.

**Deployment scenarios:** The method is especially effective for large-scale NLS problems in scientific computing, model fitting, inverse problems, and machine learning applications where residual structure and Jacobian products can be exploited efficiently.

## 7. Comparison to Related Approaches

- **Classic Gauss-Newton:** Ignores the \( C(x) \) term and uses \( J^\top J \); GNB further exploits structure while retaining low computational cost.
- **Quasi-Newton/Diagonal BFGS:** Updates are generic, lack NLS specificity, and typically do not enforce positive definiteness as robustly.
- **Structured Diagonal Heuristics:** Previously proposed methods (e.g., SDHAM [Mohammad & Sandra 2018]) lack the improved secant-based update and safeguarding components; GNB shows improved robustness and efficiency over these.

| Method          | Hessian Approx.        | Safeguarding | Matrix-free | NLS Structure Used | Global Convergence |
|-----------------|-----------------------|--------------|-------------|--------------------|--------------------|
| Classic GN      | $J^\top J$             | None         | Yes         | Yes                | No                 |
| Quasi-Newton Diag | Diag approx., generic  | Optional     | Yes         | No                 | Often weak         |
| SDHAM           | Structured diag.       | Limited      | Yes         | Yes                | Partial            |
| **GNB/ASDH**      | Structured diag. (secant) | Strong      | Yes         | Yes                | Yes                |

## 8. Summary

The Gauss-Newton-Bartlett estimator provides a principled, computationally efficient approach to diagonal Hessian approximation in large-scale nonlinear least squares problems, achieving matrix-free operation, robust safeguarding, and provable global convergence. Its design capitalizes on the segregation of Jacobian and curvature terms unique to NLS, outperforming less structured alternatives, and makes it a preferred methodology for scalable second-order NLS solvers [2002.01871].

Source: https://www.emergentmind.com/topics/gauss-newton-bartlett-gnb-estimator