Structured Diagonal Hessian Approximation

• Structured Diagonal Hessian Approximation is a technique that decomposes the Hessian into diagonal and block-diagonal parts to provide scalable curvature estimates.
• It reduces computational costs in large-scale optimization by capturing dominant per-parameter or grouped curvature information at near-linear complexity.
• It is widely applied in data-free quantization, adaptive second-order methods, and derivative-free solvers to enhance performance in complex deep learning models.

A structured diagonal Hessian approximation is a class of techniques for efficiently approximating the Hessian matrix—specifically, its diagonal or a structured sum of diagonal and block-diagonal components—by exploiting theoretical, statistical, or algorithmic structure in optimization, learning, or inverse problems. These approximations play a central role in large-scale second-order optimization, scalable quantization of deep networks, curvature-based adaptive methods, and derivative-free solvers, by capturing leading per-parameter or per-group curvature information at linear or near-linear computational cost. The structured diagonal Hessian methodology decomposes the global curvature matrix into interpretable units—such as elements, kernels, channels, or groups—and either drops or locally models off-diagonal interactions to yield a highly efficient local surrogate. Below is a detailed treatment of the key principles, methodologies, and applications of structured diagonal Hessian approximations.

1. Model Formulation and Structural Motivations

The canonical context is the approximation of the Hessian $H = \nabla^2 L(w)$ of a loss (or objective) $L(w)$ with respect to network parameters $w\in\mathbb{R}^P$. Exact formation, storage, or inversion of $H$ is infeasible for modern deep models. Structured diagonal Hessian approximations address this by positing that $H$ decomposes as a sum of structured, positive-semidefinite, diagonal or block-diagonal matrices, such that

$H \approx H_e + H_k + H_c,$

where:

• $H_e$ is the element-wise diagonal, i.e., $\operatorname{diag}(H_{11},\ldots,H_{PP})$,
• $H_k$ is block-diagonal at the kernel (e.g., convolutional filter) level,
• $H_c$ is block-diagonal at the output channel level (Guo et al., 2022).

This structure is justified when intra-kernel or intra-channel activation correlations dominate, and cross-kernel/channel couplings are weak or noisy, as often occurs due to the statistical independence induced by deep architectures and local receptive fields. For very high-dimensional problems, further structure—such as block-diagonal Kronecker factorizations or low-rank plus diagonal decompositions (see block-KFAC, SKETCHLORD)—can be employed when off-diagonal blocks retain statistical meaning, or when one aims to capture a minimal completion of the full curvature (Ritter et al., 2018, Fernandez et al., 28 Sep 2025).

2. Algorithmic Schemes and Main Approximation Families

Structured diagonal Hessian approximations are instantiated across several algorithmic paradigms:

a) Progressive Summation (e.g., SQuant)

SQuant decomposes $L(w)$0 into three granularities (element-wise, kernel-wise, channel-wise), forming a structured sum $L(w)$1. Quantization is cast as discrete optimization of a convex surrogate CASE objective:

$L(w)$2

with $L(w)$3 in a quantized grid, minimized via a three-stage flipping algorithm that iteratively satisfies group-wise constraints in linear time without data or backpropagation (Guo et al., 2022).

b) Diagonal-Only Filtering for Robustness (e.g., DASH-Q)

DASH-Q for LLM quantization discards all off-diagonals from the sample-based Hessian estimate derived from calibration data; only $L(w)$4 is kept:

$L(w)$5

Parameter quantization is then phrased as decoupled weighted least squares regressions per quant group, enabling noise-filtered subspace preservation, batch stability, and closed-form or coordinate-descent solutions (Kim et al., 15 Apr 2026).

c) Layerwise Deterministic Backpropagation (e.g., HesScale, BL89)

Approximates the diagonal using deterministic layerwise backpropagation by dropping all cross-neuron/off-diagonal second-derivative terms within backpropagated curvature recursions:

$L(w)$6

where $L(w)$7 and $L(w)$8 are activation derivatives. Enhanced forms (e.g., HesScale) inject exact output-layer curvature (Elsayed et al., 2024, Elsayed et al., 2022).

d) Stochastic and Derivative-Free Approximations

Curvature Propagation and central difference/interpolation schemes construct unbiased or structured diagonal estimates from function and/or gradient evaluations without analytic differentiation, leveraging variance-minimizing random probe constructions or structured sampling sets (regular minimal positive bases, centered simplex directions) (Martens et al., 2012, Coope et al., 2020, Jarry-Bolduc, 2021).

e) Matrix-Free Secant Updates and Quasi-Newton Scaling

For composite objectives or nonlinear least-squares, diagonal approximations are constructed to satisfy structured secant equations, often using blockwise, coordinatewise, or groupwise ratios between changes in gradient and parameters, with safeguarding to enforce positive definiteness (Awwal et al., 2020, Mannel et al., 2024).

3. Computational Complexity and Scalability

Across frameworks, a central objective is to preserve $L(w)$9 or $w\in\mathbb{R}^P$0 complexity, as opposed to $w\in\mathbb{R}^P$1 for dense Hessians. For instance:

• SQuant’s element/kernel/channel-wise passes are $w\in\mathbb{R}^P$2 or $w\in\mathbb{R}^P$3 per kernel, parallelizable and local (Guo et al., 2022).
• Layerwise diagonal Hessian backpropagation as in HesScale remains $w\in\mathbb{R}^P$4, matching standard gradients (Elsayed et al., 2024, Elsayed et al., 2022).
• Blockwise schemes, e.g., Kronecker-factor or block-diagonal approximations, have cost scaling with block sizes ($w\in\mathbb{R}^P$5 per layer) (Ritter et al., 2018).
• Derivative-free finite difference/interpolation yields Hessian diagonals at the same cost as the gradient, $w\in\mathbb{R}^P$6, with $w\in\mathbb{R}^P$7 or $w\in\mathbb{R}^P$8 black-box evaluations, provided the interpolation matrix has the required structure (Coope et al., 2020, Jarry-Bolduc, 2021).

4. Applications and Empirical Impact

Structured diagonal Hessian approximations are foundational in several domains:

• Data-Free/Post-Training Quantization: Enables accurate sub-second quantization, even at 4-bit precisions, with no access to original data, outperforming previous data-free and calibration-based PTQ methods and opening new possibilities for on-device deployment of large models (Guo et al., 2022, Kim et al., 15 Apr 2026).
• Adaptive Second-Order and Quasi-Newton Methods: Used as preconditioners, scaling matrices, or seed matrices in large-scale optimization, boosting both convergence stability and speed in nonconvex settings (vision, translation, language modeling), outperforming first-order (Adam, SGD) and vanilla quasi-Newton counterparts (Ma, 2020, Mannel et al., 2024).
• Derivative-Free and Inverse Problems: Efficiently produces structured Hessian information for preconditioning CG-like solvers or initializing limited-memory BFGS for inverse PDE problems and imaging, leading to improvements in convergence rate and practical runtime over traditional diagonally scaled or unstructured initializations (Watson et al., 2021, Awwal et al., 2020, Mannel et al., 2024).
• Variance Reduction and Control Variates: Enhances the correlation of stochastic gradient estimators in SVRG-type algorithms with low additional per-update cost, yielding condition-number improvements in theory and larger stable stepsizes in practice (Gower et al., 2017).
• Scaling Trust-Region/Step-Size Parameters: Diagonal estimates are used to effectively normalize second-order update steps in trust-region procedures, trust-aware adaptive optimizers, and step-size scaling in stochastic policy optimization, yielding uniform step-size robustness (Elsayed et al., 2024).

5. Theoretical Guarantees, Assumptions, and Variance Properties

The efficacy of structured diagonal Hessian approximations rests on several mathematical underpinnings:

• Variance Reduction: Dropping off-diagonals dramatically reduces the variance of diagonal estimates, especially when cross-feature or cross-channel entries have poor SNR due to limited data or high stochasticity (Kim et al., 15 Apr 2026).
• Error Bounds: For derivative-free schemes with structured sample sets (e.g., regular minimal positive bases or “lonely” direction matrices), $w\in\mathbb{R}^P$9 convergence to the true diagonal is guaranteed under $H$0- or $H$1-smoothness assumptions (Coope et al., 2020, Jarry-Bolduc, 2021).
• Convergence: Structured diagonal preconditioners, when used as seed or initial Hessians in L-BFGS or other quasi-Newton methods, retain global and (under additional conditions) linear convergence rates, with explicit spectral bounds ensuring iteration stability (Mannel et al., 2024, Awwal et al., 2020).
• Batch-Stability and Overfitting: Empirically, diagonal approximations are far more batch-stable than full-matrix estimates in low-sample or low-bit regimes. Incorporating off-diagonal terms can lead to overfitting calibration noise, exploding perplexities, and erratic downstream scaling, as evidenced in ultra low-bit quantization (Kim et al., 15 Apr 2026).

6. Extensions, Limitations, and Future Directions

Structured diagonal Hessian approximations are being extended or combined in several ways:

• Low-Rank Plus Diagonal: SKETCHLORD and related sketching algorithms jointly estimate a diagonal and low-rank component, addressing settings where neither alone is sufficient and dramatically outperforming sequential low-rank or diagonal estimation for large-scale operators (Fernandez et al., 28 Sep 2025).
• Block-Diagonal and Kronecker: Block-diagonal structures (often at the layer or group level) and Kronecker-factored approximations balance structure and tractability for moderate block sizes, effectively capturing intra-layer dependencies (Ritter et al., 2018).
• Preconditioning and Optimization: Diagonal or structured scaling is being embedded as preconditioners into trust-region, natural-gradient, and adaptive learning strategies. Combination with blockwise and curvature-matching updates is actively explored (Elsayed et al., 2022, Elsayed et al., 2024).
• Limitations: The accuracy of strictly diagonal or block-diagonal methods critically depends on the problem’s intrinsic curvature structure, in particular, the dominance of on-block or diagonal terms. In highly coupled, non-diagonally-dominant regimes, more sophisticated structure or low-rank augmentation is necessary (Fernandez et al., 28 Sep 2025).
• Variance-Minimizing Stochastic Estimation: Techniques such as curvature propagation or Hutchinson’s trick are employed when deterministic structure is unavailable, with CP delivering provably minimal variance under randomized probes (Martens et al., 2012).
• Robust Hyperparameter and Structural Adaptation: Real-world deployments tune structural parameters (group size, block scale, projection intervals) and implement runtime statistical checks for diagonal dominance, variance, and stability, dynamically adjusting method choice according to properties observed in practice (Elsayed et al., 2024, Kim et al., 15 Apr 2026).

7. Representative Algorithms and Methodological Summary

Framework	Structural Decomposition	Core Use Case	Reference
SQuant	Diag + Kernel-wise + Channel-wise	Data-free quantization	(Guo et al., 2022)
DASH-Q	Diagonal-only (per group)	LLM ultra-low-bit PTQ	(Kim et al., 15 Apr 2026)
HesScale/BL89	Layerwise deterministic diagonal	Efficient 2nd order	(Elsayed et al., 2024)
Derivative-Free	Structured finite-difference	Black-box opt.	(Coope et al., 2020)
L-BFGS Scaling	Diagonal seed for L-BFGS	Inverse/hybrid opt.	(Mannel et al., 2024)
Curvature Prop.	Unbiased stochastic diagonal	General computation	(Martens et al., 2012)
SKETCHLORD	Joint low-rank + diagonal	Matrix approximation	(Fernandez et al., 28 Sep 2025)

These methods underscore the centrality of structured diagonal Hessian approximations for large-scale, high-dimensional learning and inverse problems, offering an optimal trade-off between computational efficiency, statistical robustness, and fidelity to dominant curvature structure.