Kronecker-Factored Inverse Hessian

Updated 22 January 2026

Kronecker-factored inverse Hessian approximation is a scalable method that decomposes layerwise Hessian blocks into a Kronecker product, drastically reducing storage and computational requirements.
It leverages efficient inversion by computing inverses of two smaller matrices per layer, which underpins second-order optimizers and structured pruning algorithms.
The approach is critical in continual learning, using curvature-aware corrections to preserve past-task knowledge and prevent catastrophic forgetting.

A Kronecker-factored inverse Hessian approximation provides a scalable, computationally efficient surrogate to the true inverse Hessian of a large neural network by assuming a block-diagonal (layerwise) structure and decomposing each block as a Kronecker product of two smaller factors. This methodology underlies several second-order optimization techniques, structured pruning algorithms, and quadratic regularization strategies in continual learning, and is foundational to memory- and computation-efficient algorithms for deep networks.

1. Theoretical Foundation and Motivation

Given a scalar loss $L(\theta)$ over parameters $\theta \in \mathbb{R}^n$ , the Hessian $H \equiv \nabla^2 L(\theta)$ is a symmetric $n \times n$ matrix whose eigenspectrum captures local curvature. Directly storing or inverting $H$ is prohibitive for modern deep networks (memory $O(n^2)$ , inversion $O(n^3)$ ). In scenarios such as continual learning, preserving previous-task knowledge requires regularizing or rescaling updates to ensure movement primarily in directions of low past-task curvature, which naturally leads to an inverse Hessian preconditioner (Eeckt et al., 21 Jan 2026, Martens et al., 2015).

To make this feasible, the Hessian is approximated as block-diagonal across layers. For a weight matrix $W \in \mathbb{R}^{d_o \times d_i}$ in a linear or affine layer, the Hessian block is further approximated by a Kronecker product:

$H_W \approx A \otimes B,$

with $A \in \mathbb{R}^{d_o \times d_o}$ and $B \in \mathbb{R}^{d_i \times d_i}$ . This exploits the statistical structure induced by forward and backward activations, and can be motivated by the empirical Fisher or Gauss–Newton approximation, as well as cumulant expansions (Martens et al., 2015, Goldfarb et al., 2020).

The Kronecker structure enables efficient inversion via

$(A \otimes B)^{-1} = A^{-1} \otimes B^{-1},$

reducing the bottleneck to inverting two small matrices per layer rather than a massive dense block.

2. Construction of Kronecker Factors

For a fully connected layer computing $z = W x$ , with upstream gradient $\delta = \partial L / \partial z$ and input $x \in \mathbb{R}^{d_i}$ , the curvature w.r.t. $W$ is approximated as:

$H_W \approx E[ \mathrm{vec}(\delta x^\top) \; \mathrm{vec}(\delta x^\top)^\top ] = E[\delta\delta^\top] \otimes E[xx^\top],$

so the Kronecker factors are given by empirical (or batch-averaged) second moments: $A = E_t[\delta_t \delta_t^\top] + \lambda_A I_{d_o}, \qquad B = E_t[x_t x_t^\top] + \lambda_B I_{d_i},$ where $\lambda_A,\lambda_B > 0$ are small damping coefficients. For batch- or convolutional layers, analogous expressions apply, possibly involving spatial averages or structured Khatri-Rao products (Martens et al., 2015, Ren et al., 2021).

In networks with batch normalization or inter-example couplings, extensions to the standard Kronecker structure have been developed to incorporate dependencies (Lee et al., 2020).

3. Practical Algorithms and Update Rules

Numerous second-order and continual learning algorithms leverage the Kronecker-factored inverse Hessian, including K-FAC, K-BFGS, structured pruning, and Laplace regularization. The generic update utilizing the Kronecker-factored approximation takes the form

$\Delta W = B^{-1} (\widetilde{W} - W_0) A^{-1},$

where $\widetilde{W}$ is a fine-tuned or proposed update, and $W_0$ is the reference. Inverse factors $A^{-1}, B^{-1}$ are computed via Cholesky or eigendecomposition.

For optimization, stochastic quasi-Newton methods such as K-BFGS or K-BFGS(L) perform two or more per-layer curvature updates (e.g., BFGS or Hessian-action BFGS) and apply double-damping for stability (Goldfarb et al., 2020, Ren et al., 2021). For pruning and Bayesian online learning, quadratic penalties are imposed using the Kronecker-factored inverse as the precision of the approximate posterior (Ritter et al., 2018).

Illustrative high-level pseudocode for an inverse Hessian merging step in continual learning, as implemented in ASR, is:

for each linear layer l:
    # Compute second moment statistics on past task data
    A_l = E[δ δᵀ] + λ I
    B_l = E[x xᵀ] + λ I
    A_l_inv = inverse(A_l)
    B_l_inv = inverse(B_l)
    ΔW_l = W̃_l - W_{l}^{t-1}
    ΔW_l_corr = B_l_inv @ ΔW_l @ A_l_inv
    α = τ * norm(ΔW_l) / norm(ΔW_l_corr)
    W_l^t = W_{l}^{t-1} + α * ΔW_l_corr

(Eeckt et al., 21 Jan 2026)

4. Computational and Memory Efficiency

The Kronecker-factored approximation achieves drastic computational savings over the full Hessian. For $W \in \mathbb{R}^{d_o \times d_i}$ , the full block requires $O(d_o^2 d_i^2)$ storage and $O((d_o d_i)^3)$ inversion, whereas the Kronecker surrogate needs $O(d_o^2 + d_i^2)$ storage and $O(d_o^3 + d_i^3)$ inversion (Eeckt et al., 21 Jan 2026, Martens et al., 2015).

Applying $(A \otimes B)^{-1}$ to a vectorized matrix can be computed as $B^{-1} X A^{-1}$ for $X \in \mathbb{R}^{d_o \times d_i}$ , with cost $O(d_o^2 d_i + d_o d_i^2)$ per layer. This computational pattern is highly parallelizable and suited to GPU kernel optimization.

The Kronecker structure is maintained layerwise; cross-layer curvature is neglected, but the per-layer approximation is empirically effective. Conjugate-gradient and iterative matrix-free algorithms further reduce memory and inversion costs by never forming the full Kronecker factors explicitly (Chen, 2021).

5. Applications in Continual Learning and Quadratic Penalty Methods

Inverse Hessian regularization is central to memory-efficient continual learning. After fine-tuning on a new domain, the adaptation is merged with the pre-existing model by applying a single (blockwise) inverse-Hessian correction. This suppresses forgetting by damping movement along directions of high past-task curvature, as measured by the Kronecker-factored Hessian of the old task (Eeckt et al., 21 Jan 2026, Ritter et al., 2018, Lee et al., 2020).

Empirical results on ASR benchmarks demonstrate that such regularization essentially eliminates catastrophic forgetting, achieving backward transfer near –0.1% versus –0.3% for naive averaging, and improving WER significantly. Performance closely matches stronger replay-based methods without requiring storage of previous-task data (Eeckt et al., 21 Jan 2026).

Bayesian online learning algorithms maintain a Gaussian posterior with a Kronecker-factored precision matrix, recursively updating the quadratic penalty as new tasks arrive while retaining strict scalability (Ritter et al., 2018). Extensions properly handle layers with batch normalization via statistical reparameterization and merged curvature factors (Lee et al., 2020).

6. Extensions: Optimization, Pruning, and Variants

Kronecker-factored inverse Hessian preconditioners underpin several modern optimization and model compression techniques:

Second-order optimizers: K-FAC (Martens et al., 2015), K-BFGS (Goldfarb et al., 2020, Ren et al., 2021), Shampoo (Mei et al., 2023), and KrADagrad (Mei et al., 2023) use Kronecker-product surrogates for curvature to accelerate training. KrADagrad, for example, maintains factors via efficient matrix operations, avoiding numerically unstable inverse roots required by Shampoo and permitting 32-bit precision deployment.
Structured pruning: The EigenDamage algorithm diagonalizes the Kronecker factors to prune weights in the Kronecker-factored eigenbasis, enabling accurate, loss-aware structured compression with minimal accuracy degradation (Wang et al., 2019).
Batch normalization and nonstandard layers: Extended K-FAC variants capture curvature with coupled batch statistics, using extra terms (e.g., Khatri-Rao products) to handle inter-sample dependencies (Lee et al., 2020).

A summary table of core computational properties:

Method	Storage per layer	Inversion cost	Use case
Full Hessian	$O(d_o^2 d_i^2)$	$O((d_o d_i)^3)$	Theoretical/Small nets
Kronecker-factored	$O(d_o^2 + d_i^2)$	$O(d_o^3 + d_i^3)$	DNN, Continual Learning
Diagonal	$O(d_o d_i)$	$O(d_o d_i)$	AdaGrad/EWC

(Eeckt et al., 21 Jan 2026, Martens et al., 2015, Goldfarb et al., 2020)

7. Empirical Performance and Limitations

Empirical benchmarks confirm that Kronecker-factored regularizers (e.g., Inverse-Hessian Regularization) substantially reduce forgetting in continual learning, with backward transfer close to zero and WER improvements highly significant ( $p < 0.001$ ) on major ASR tasks (Eeckt et al., 21 Jan 2026). Kronecker-factored quasi-Newton and natural-gradient updates achieve wall-clock performance comparable or superior to first-order methods with minimal extra cost (Goldfarb et al., 2020, Ren et al., 2021).

The main limitations are:

Layerwise block-diagonality: Ignores cross-layer interactions, which may be significant in some architectures.
Damping and stability: Requires well-chosen damping to avoid numerical instability; ill-conditioned factors may need additional regularization (Mei et al., 2023).
Applicability: Kronecker structure is most natural for fully connected and standard convolutional layers; extensions to complex modules require additional analysis (Lee et al., 2020, Ren et al., 2021).

A plausible implication is that as models grow larger and tasks more diverse, efficient, layer-local, curvature-aware inverses will remain central to scalable, high-fidelity regularization and optimization.

References:

(Eeckt et al., 21 Jan 2026) Inverse-Hessian Regularization for Continual Learning in ASR
(Martens et al., 2015) Optimizing Neural Networks with Kronecker-factored Approximate Curvature
(Goldfarb et al., 2020) Practical Quasi-Newton Methods for Training Deep Neural Networks
(Lee et al., 2020) Continual Learning with Extended Kronecker-factored Approximate Curvature
(Ritter et al., 2018) Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
(Wang et al., 2019) EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
(Mei et al., 2023) KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization
(Ren et al., 2021) Kronecker-factored Quasi-Newton Methods for Deep Learning
(Chen, 2021) An iterative K-FAC algorithm for Deep Learning