---
title: Diagonal Hessian Estimation Techniques
url: https://www.emergentmind.com/topics/diagonal-hessian-estimation
type: topic
---

# Diagonal Hessian Estimation Techniques

Diagonal Hessian estimation is the process of approximating or computing the diagonal entries of the Hessian matrix, $\text{diag}[\nabla^2 f(x)]$, of a scalar function $f:\mathbb{R}^n\to\mathbb{R}$—either exactly or approximately—using only gradient, function, or limited Hessian-vector (or matrix-vector) information. The diagonal often serves as a computationally tractable proxy for the full Hessian in optimization, machine learning, numerical PDEs, and statistical estimation, where access to the entire matrix is prohibitive due to dimensionality, cost, or structural constraints.

## 1. Foundational Methods: Finite Differences, Function Evaluation, and Sample Set Design

Diagonal entries can be individually approximated by classical second-order finite difference schemes in the coordinate directions:
\[
\big[\nabla^2 f(x)\big]_{ii} \approx \frac{f(x + h e_i) - 2 f(x) + f(x - h e_i)}{h^2}
\]
where $e_i$ is the standard basis and $h$ a small step. This approach requires $O(n)$ function evaluations and is widely adopted in blackbox/derivative-free optimization and negative curvature detection [2206.05318]. In settings where only function values are available, such as black-box optimization or simulation-based models, the diagonal can be systematically estimated for all $i$ with $2n+1$ evaluations.

More advanced approaches, such as the Generalized Centered Simplex Hessian Diagonal (CSHD) [2104.11821], use a set of sample directions $S=[s^1,\dots,s^k]$ and combine evaluations at $x^0 \pm s^i$ using an explicit formula:
\[
d_{\mathrm{CSHD}} = (W^\top)^\dagger \varepsilon_f(x^0; S)
\]
where $W = [s^1 \odot s^1, ..., s^k \odot s^k]$ (elementwise square), and $\varepsilon_f$ encodes symmetrized quadratic increments. Achieving optimal quadratic accuracy, i.e., $O(\Delta_S^2)$ with $\Delta_S = \max \|s^i\|$, requires the sample set $S$ to be a full row rank "lonely matrix"—each $s^i$ aligned with a single coordinate direction. If the off-diagonal (strictly upper triangular) terms in the error bound are nonzero due to the sampling geometry, accuracy and convergence rates degrade regardless of the number of evaluations or step size [2104.11821, 2304.03222].

| Method         | Complexity | Accuracy                  | Sample Set Requirement           |
|----------------|------------|---------------------------|----------------------------------|
| Centered FD    | $O(n)$     | $O(h^2)$                  | Axis-aligned, equally spaced     |
| CSHD           | $O(n)$     | $O(\Delta_S^2)$           | Lonely matrix, full row rank     |
| GSH/GCSH       | $O(n)$     | $O(\Delta^{1,2})$         | Arbitrary/projected for GSH      |

## 2. Stochastic and Algorithmic Approaches in Machine Learning

In large-scale machine learning models, especially deep neural networks, exact computation or explicit storage of full Hessians is prohibitive. Multiple scalable, approximate methods for diagonal Hessian estimation have been developed.

### Curvature Propagation (CP)
Curvature Propagation [1206.6464] is a stochastic algorithm that backpropagates random probe vectors through the computational graph. Each pass provides an unbiased rank-1 estimator of the Hessian, and the diagonal is estimated as $E[S(V)\odot S(V)]$, where $S(V)$ results from the recursion with random starting points at each node. A single sample costs about two reverse-mode (gradient) passes; variance decays as $O(1/K)$ with $K$ samples. CP achieves significantly lower variance on the diagonal compared to Hessian-vector product-based outer product estimators.

### HesScale and Modern Diagonal Recursions
HesScale [2210.11639, 2406.03276] revisits and refines deterministic diagonal backpropagation methods (e.g., Becker & LeCun, 1989). The key insight is to use the exact Hessian diagonal for the output layer (when available, e.g., in softmax-cross-entropy), which seeds a precise layerwise recursion:
\[
\frac{\partial^2 \mathcal{L}}{\partial a_l^2} \doteq [o'(a_l)]^2 \sum_k W_{l+1, k, i}^2 \frac{\partial^2 \mathcal{L}}{\partial a_{l+1, k}^2} + o''(a_l) \sum_k W_{l+1, k, i} \frac{\partial \mathcal{L}}{\partial a_{l+1, k}}
\]
The method ignores off-diagonal second-derivative terms, yielding linear scaling and high-diacronal accuracy. Empirical studies show that HesScale provides the best $L_1$ approximation to the true Hessian diagonal compared to stochastic estimators (AdaHessian, Hutchinson/MC-GGN, etc.), and its efficiency enables use in large-scale settings.

This family of methods is supported by theoretical and empirical analyses of layerwise diagonal dominance in DNN Hessians, especially for wide/deep architectures [2210.11639, 2406.03276].

| Estimator      | Unbiased | Complexity | Diag. Error (ML) | Comments            |
|----------------|----------|------------|------------------|---------------------|
| CP (stochastic)| Yes      | 2$\times$grad | Low             | Need averaging      |
| HesScale       | No       | ~2$\times$grad | Very Low        | Deterministic, best |
| BL89           | No       | ~2$\times$grad | Medium          | Older approx.       |
| AdaHessian     | Yes      | $k{\times}$grad | High unless $k\gg1$ | Stochastic       |

## 3. Structural and Block-diagonal Hessian Approximations

Recent theoretical work [2505.02809] has illuminated why diagonal and block-diagonal Hessian estimators are particularly effective for modern neural networks, particularly those with many output classes (as in large language models). Both empirical and rigorous analyses show that as the number of output classes $C\to\infty$, off-diagonal block norms decay as $O(1/C)$ in output layer Hessians, and as $O(1/\sqrt{C})$ for hidden-layer blocks. This "static force" arises from architectural design (layer, class decomposition) and is supplemented by further "dynamic" suppression during training. Thus, block-diagonal or even diagonal approximations become asymptotically near-exact for large classification or language models.

Block-diagonal estimators, as instantiated in Block Diagonal Hessian-Free optimization [1712.07296], generalize the diagonal approach by modeling intra-block curvature while ignoring inter-block interactions. This framework yields robustness, improved optimization, and efficient parallelization, particularly for large minibatch training. The diagonal is recovered as the limiting case of $1\times1$ blocks.

| Structure         | Decay of off-diagonal/block      | Implication            |
|-------------------|----------------------------------|------------------------|
| Output (classes)  | $O(1/C)$, $O(1/C^2)$             | Large $C$: nearly exact|
| Hidden layer      | $O(1/\sqrt{C})$                  | Large $C$: block-diag  |

## 4. Derivative-Free, Blackbox, and Quantum Settings

In derivative-free optimization (DFO), or when only blackbox function evaluations are possible, diagonal Hessian estimation must avoid gradients/Hessian-vector products. Matrix algebra [2304.03222] and generalized simplex formulations [2104.11821] enable explicit, order-optimal diagonal estimation using only targeted function values. Accuracy is critically determined by the geometry and rank of the sample set; axis-aligned or "lonely" (one nonzero per column) sample sets achieve quadratic accuracy in the sample set radius.

For high-efficiency detection of negative curvature, algorithms first check Hessian diagonals (via finite differences or exact values) and proceed to build off-diagonal elements only as needed; negative diagonal entries immediately certify negative eigenvalues [2206.05318].

Quantum algorithms have achieved exponential speedup for diagonal and sparse Hessian estimation in settings where function access is via phase oracle [2407.03833]. The quantum spectral method achieves query complexity $\widetilde{O}(1/\varepsilon)$ for diagonal Hessian estimation, exponentially faster than classical $O(d/\varepsilon^2)$ approaches for $d$-dimensional problems, provided the Hessian is diagonal. For generic (dense) Hessians, quantum methods achieve at most quadratic speedup.

| Approach          | Query Complexity (diag)      | Assumptions                        |
|-------------------|-----------------------------|------------------------------------|
| Finite difference | $O(d/\varepsilon^2)$        | Classical, adjust $h$              |
| Quantum spectral  | $\widetilde{O}(1/\varepsilon)$ | Quantum, diagonal/sparse case   |
| Quantum finite diff | $\widetilde{O}(d/\varepsilon)$ | Quantum, dense case           |

## 5. Applications in Optimization, Quantization, and High-dimensional Inference

Diagonal Hessian estimation is broadly used for:

- **Second-order optimization**: Diagonal preconditioners support adaptive step sizes (e.g., Adam, RMSProp, AdaHesScale) and stabilize convergence in deep learning [2210.11639, 2406.03276, 1712.07296].
- **Blackbox optimization/derivative-free algorithms**: Directional curvature estimation (HE-ES) [2003.13256, 2009.02732] uses finite-difference directional probing to tune covariance matrices and adapt search distributions.
- **Quantization and compression**: In data-free post-training quantization (DFQ), diagonal and block-diagonal Hessian approximations inform sensitivity-aware quantizer placement, as in SQuant [2202.07471]. Data-free, architecture-agnostic algorithms leverage a progression of diagonal/kernal/global channel estimates to optimize quantization objectives without fine-tuning or data access.
- **High-dimensional statistics**: Diagonal elements of sparse precision (inverse covariance) matrices are critical for inference and structure learning. Residual variance, maximum likelihood, and symmetry-enforcing estimators provide risk-optimal or robust diagonal estimates, depending on accuracy of regression coefficient estimation [1504.04696].
- **Distributed optimization**: Newton-like methods with diagonal correction balance communication and performance by (block-)diagonal Hessian inversion and efficient local updates [1509.01703].

## 6. Limitations, Error Bounds, and Best Practices

Theoretical analyses [2104.11821, 2304.03222] show that:

- *Accuracy of diagonal estimation depends critically on sample set geometry*: With "lonely" (coordinate) directions, diagonal finite-difference/CSHD achieves $O(\Delta_S^2)$ error, matching optimal order in sample size.
- *Off-diagonal coupling is a major source of error*: If the sampling scheme or approximation discards high-magnitude off-diagonal Hessian terms, error does not vanish with smaller sample radii.
- *Stochastic estimators (CP, AdaHessian, Hutchinson-type)* are unbiased but may require many samples to reach the accuracy of deterministic, structure-aware recursions like HesScale.
- *Block/diagonal approximations become near-exact in large-class, wide, or deep neural network regimes, a direct consequence of architectural decoupling at scale* [2505.02809].

For robust implementation in DFO or automated optimization code:

- Use symmetric, coordinate-based or well-poised sample sets to guarantee quadratic accuracy in finite-difference frameworks [2104.11821, 2304.03222].
- For large-scale deep learning, use HesScale for Hessian diagonals, or block-diagonal approximations if grouped parameter structure is significant [2210.11639, 1712.07296].
- In quantum or blackbox settings, adopt the quantum spectral method for functions where phase oracles are available and sparsity permits, otherwise use classic finite difference diagonals [2407.03833].
- In high-dimensional statistical inference, when regression coefficients are accurately estimated, use symmetry-enforcing maximum likelihood; otherwise use the residual variance estimator for safety [1504.04696].

---

**References**

- [2104.11821] Approximating the diagonal of a Hessian: which sample set of points should be used
- [2210.11639] HesScale: Scalable Computation of Hessian Diagonals
- [2406.03276] Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning
- [1206.6464] Estimating the Hessian by Back-propagating Curvature
- [2304.03222] A matrix algebra approach to approximate Hessians
- [1712.07296] Block-diagonal Hessian-free Optimization for Training Neural Networks
- [2505.02809] Towards Quantifying the Hessian Structure of Neural Networks
- [2206.05318] Detecting negative eigenvalues of exact and approximate Hessian matrices in optimization
- [2407.03833] Quantum spectral method for gradient and Hessian estimation
- [1504.04696] On estimation of the diagonal elements of a sparse precision matrix
- [2003.13256] The Hessian Estimation Evolution Strategy
- [2002.01871] Iterative algorithm with structured diagonal Hessian approximation for solving nonlinear least squares problems
- [1509.01703] Newton-like method with diagonal correction for distributed optimization
- [2202.07471] SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation
- [2509.23587] Sketching Low-Rank Plus Diagonal Matrices

Source: https://www.emergentmind.com/topics/diagonal-hessian-estimation