Fisher Divergence in Statistical Inference

• Fisher divergence is a functional that quantifies differences between probability densities by comparing their score functions rather than density values.
• Its score-based structure allows evaluation without explicit normalizing constants, making it pivotal for applications like score matching and Bayesian variational inference.
• Variations such as forward and reverse Fisher divergence impact optimization behavior, benefiting fields like reinforcement learning and Monte Carlo methods.

Fisher divergence is a discrepancy functional between smooth probability densities that compares their score functions rather than their density values. For densities $p$ and $q$ on $\mathbb{R}^d$, with scores $s_p(x)=\nabla_x\log p(x)$ and $s_q(x)=\nabla_x\log q(x)$, a standard form is

$$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$

with some authors including a factor $1/2$. Because it depends only on derivatives of log-densities, it can be evaluated without the normalizing constant of $q$, and in several formulations without the normalizing constant of the target distribution as well. This score-based character makes Fisher divergence central to score matching, variational inference, posterior calibration, and several optimization problems in modern statistics and machine learning (Yang et al., 2019).

1. Definition, conventions, and elementary properties

The classical Fisher divergence between two densities $p$ and $q$ is written in the literature as

$q$0

Equivalent notational variants use

$q$1

The difference is a conventional factor $q$2, not a substantive change in the object being minimized [(Bercher, 2013); (Elkhalil et al., 2020)].

A further distinction concerns the measure under which the squared score mismatch is averaged. In recent score-matching work, the forward Fisher divergence is

$q$3

whereas the reverse Fisher divergence is

$q$4

The two functionals use the same integrand but can have markedly different optimization behavior (Tyurin, 18 Jun 2026).

In Bayesian variational inference, one paper defines

$q$5

and then introduces

$q$6

By integration by parts, $q$7 differs from $q$8 by a term independent of $q$9, so minimizing $\mathbb{R}^d$0 is equivalent to minimizing $\mathbb{R}^d$1 in that setting (Yang et al., 2019).

Several basic properties recur across the literature. Fisher divergence is nonnegative, and in the variational formulation above $\mathbb{R}^d$2 with equality iff $\mathbb{R}^d$3 almost everywhere (Yang et al., 2019). Because only score functions appear, no hidden normalizing constant is required (Yang et al., 2019, Kostrikov et al., 2021). This local, derivative-based structure is also why Fisher divergence is often described as a measure of shape or curvature mismatch rather than density-ratio mismatch.

2. Score matching and Hyvärinen-type objectives

Score matching was introduced to fit unnormalized statistical models by matching score functions rather than densities directly. Under mild conditions, minimizing Fisher divergence is equivalent to minimizing an objective involving only derivatives of the model log-density, not the unknown score of the data distribution (Elkhalil et al., 2020).

A standard decomposition writes

$\mathbb{R}^d$4

where

$\mathbb{R}^d$5

is the Hyvärinen score. Since the first term depends only on $\mathbb{R}^d$6, minimizing $\mathbb{R}^d$7 over $\mathbb{R}^d$8 is equivalent to minimizing $\mathbb{R}^d$9 (Elkhalil et al., 2020).

For i.i.d. samples $s_p(x)=\nabla_x\log p(x)$0, integration by parts yields the empirical score-matching loss

$s_p(x)=\nabla_x\log p(x)$1

This objective is data-dependent and avoids any explicit appearance of $s_p(x)=\nabla_x\log p(x)$2 (Paisley et al., 4 Apr 2025).

Noise-conditional variants replace the clean density by a Gaussian-smoothed density. In the GP-tilted density framework, the noise-conditional Fisher objective averages the same score-matching structure over additive Gaussian perturbations and can be further averaged over a discretized set of noise levels (Paisley et al., 4 Apr 2025). In that model, representing the score with random Fourier features turns the optimization into a convex quadratic in the feature coefficients, yielding closed-form minimizers and a non-iterative learning procedure (Paisley et al., 4 Apr 2025).

The score-matching interpretation also clarifies why Fisher divergence is particularly sensitive to local fluctuations. In the Jensen–Fisher setting, this sensitivity is contrasted with Jensen–Shannon divergence: for sinusoidal, generalized gamma-like, and Rakhmanov-Hermite families, Jensen–Fisher exhibits much larger dynamic range and stronger response to nodes and oscillations (Sánchez-Moreno et al., 2010).

3. Variational inference and posterior approximation

In Bayesian inference with intractable posteriors $s_p(x)=\nabla_x\log p(x)$3, one can choose a tractable family $s_p(x)=\nabla_x\log p(x)$4 and solve

$s_p(x)=\nabla_x\log p(x)$5

Because $s_p(x)=\nabla_x\log p(x)$6 measures squared differences of score functions, it is sensitive to curvature of $s_p(x)=\nabla_x\log p(x)$7 in $s_p(x)=\nabla_x\log p(x)$8. The same source states that matching scores encourages $s_p(x)=\nabla_x\log p(x)$9 to capture tail-behavior and higher cumulants better than the usual KL divergence, which only guarantees good mass covering on average (Yang et al., 2019).

For an exponential-family ansatz

$s_q(x)=\nabla_x\log q(x)$0

one obtains

$s_q(x)=\nabla_x\log q(x)$1

where $s_q(x)=\nabla_x\log q(x)$2 and $s_q(x)=\nabla_x\log q(x)$3. This produces an iteratively re-weighted least-squares scheme with

$s_q(x)=\nabla_x\log q(x)$4

followed by the damped update

$s_q(x)=\nabla_x\log q(x)$5

The formulation does not require conjugacy or mean-field factorization, and it avoids unknown normalizing constants because only $s_q(x)=\nabla_x\log q(x)$6 and $s_q(x)=\nabla_x\log q(x)$7 appear (Yang et al., 2019).

In logistic regression with Gaussian prior and a full multivariate Gaussian variational family, the posterior score is

$s_q(x)=\nabla_x\log q(x)$8

The paper approximates required expectations by a second-order Taylor expansion around the current mean and reports equally good posterior-mean estimates relative to the Jaakkola–Jordan bound method and doubly stochastic VI, but substantially more accurate covariance estimates, with Frobenius-norm errors $s_q(x)=\nabla_x\log q(x)$9–$$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$0 versus $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$1–$$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$2 for competitors. Credible-region coverage curves tracked the $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$3 line closely for the Fisher method, and computation time was seconds, comparable to JJ and one to two orders of magnitude faster than MCMC (Yang et al., 2019).

Later work extends this perspective to the weighted Fisher divergence

$$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$4

where $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$5 is positive semi-definite. The ordinary Fisher divergence is recovered at $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$6, while $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$7, the covariance of $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$8, gives the score-based divergence (Chen et al., 6 Mar 2025). In high-dimensional Gaussian VI, unbiased reparameterization-based SGD for Fisher objectives is reported to have high variance, whereas a batch-objective approximation yields stable algorithms that can exploit sparse precision structure in logistic regression, generalized linear mixed models, and stochastic volatility models (Chen et al., 6 Mar 2025).

4. Optimization behavior and representative applications

Recent analysis of Gaussian mixtures shows that the forward and reverse versions of Fisher divergence can induce qualitatively different landscapes. For Gaussian mixture score matching, empirical and theoretical work reports that gradient descent on the forward divergence can get stuck in spurious minima or even drive some student means off to infinity, whereas reverse Fisher optimization penalizes regions where the student distribution places spurious mass (Tyurin, 18 Jun 2026).

In the single-Gaussian teacher case, Theorem 2.1 establishes global convergence of gradient descent on reverse Fisher divergence from any initialization, with step size $$D_F(p\|q)=\mathbb{E}_{x\sim p}\bigl\|\nabla\log p(x)-\nabla\log q(x)\bigr\|^2,$$9 and rate

$1/2$0

For multiple Gaussian targets, under random initialization on a large sphere and a $1/2$1-separation condition stated as $1/2$2, each student component converges near its closest teacher component with high probability (Tyurin, 18 Jun 2026). A plausible implication is that the choice of expectation measure in a score-based discrepancy is not merely a formal variant but an optimization-relevant modeling decision.

In offline reinforcement learning, Fisher divergence appears as a critic regularizer. Fisher-BRC parameterizes the critic as

$1/2$3

where $1/2$4 is the behavior policy. The Fisher-divergence term between the Boltzmann policy induced by $1/2$5 and $1/2$6 reduces exactly to the gradient penalty

$1/2$7

On D4RL MuJoCo benchmarks, the paper reports improved performance and faster convergence over existing methods, with examples including HalfCheetah-random $1/2$8 versus CQL $1/2$9, Hopper-medium $q$0 versus $q$1, and Walker2d-mixed $q$2 versus $q$3 (Kostrikov et al., 2021).

In Hamiltonian Monte Carlo, Fisher divergence is used to learn preconditioners by minimizing the sample Fisher divergence from a linearly transformed target density to a standard normal distribution. For $q$4 models from posteriordb, the diagonal minimizer of Fisher divergence outperformed the diagonal variance-based estimators used by Stan and PyMC by a median factor of $q$5, and the low-rank plus diagonal minimizer outperformed them by a median factor of $q$6 (Seyboldt et al., 19 Mar 2026).

These examples span posterior approximation, generative score matching, reinforcement learning, and MCMC adaptation. They share a common mechanism: Fisher divergence couples model learning to first-order differential structure rather than to density ratios or likelihood values.

5. Generalizations and derived Fisher-type quantities

A broad family of extensions replaces the Euclidean norm by a weighted quadratic form. For a positive semi-definite matrix $q$7,

$q$8

When $q$9, this is the standard Fisher divergence of Hyvärinen (2005) (Onizuka, 25 Jun 2026). In general Bayesian calibration, this weighted divergence is minimized between two Gaussian asymptotic laws, yielding the closed-form learning rate

$p$0

with $p$1 and $p$2 (Onizuka, 25 Jun 2026).

Another extension starts from a modified $p$3-divergence and defines

$p$4

The comparison with the classical case is explicit: classical Fisher divergence uses weight $p$5, power $p$6, and argument $p$7, whereas the generalized version uses weight $p$8, power $p$9, and argument $q$0 (Bercher, 2013). The same framework proves convexity, data-processing inequalities, generalized Cramér–Rao inequalities, and a characterization of generalized $q$1-Gaussians as minimizers of a $q$2-Fisher information at fixed moment (Bercher, 2013).

A separate line defines the $q$3-Fisher divergence matrix by

$q$4

whose trace is the scalar $q$5-Fisher divergence. When $q$6, this reduces to the classical Fisher divergence (Toranzo et al., 2016). In the same framework, generalized de Bruijn identities connect derivatives of $q$7-entropies and $q$8-divergences to $q$9-Fisher information and $q$00-Fisher divergence (Toranzo et al., 2016).

The Jensen–Fisher divergence is built from the ordinary Fisher information functional

$q$01

and is defined for weighted families by

$q$02

It is nonnegative, symmetric, additive over finite families, vanishes iff all densities are equal, and remains well-defined even when the densities have non-common zeros (Sánchez-Moreno et al., 2010). Because it is controlled by a gradient functional, it is reported to be especially informative for oscillatory distributions (Sánchez-Moreno et al., 2010).

6. Relation to Fisher information geometry and terminological issues

Fisher divergence should be distinguished from the Fisher information matrix and the Fisher–Rao metric. For a parametric family $q$03, the Fisher information matrix is

$q$04

and it defines a Riemannian metric on the statistical manifold (Costa et al., 2012). The associated geodesic or Fisher distance is

$q$05

In the univariate normal family, this geometry is hyperbolic, with constant curvature $q$06, and the resulting distance has a closed form in $q$07 coordinates (Costa et al., 2012).

At the same time, some applied work uses the phrase “Fisher information distance” for the score-based divergence

$q$08

(Kostrikov et al., 2021). This suggests a terminological ambiguity: “Fisher distance” in information geometry refers to the geodesic distance induced by the Fisher–Rao metric, whereas “Fisher divergence” denotes a score-mismatch functional between two densities.

The two notions are related only indirectly. The Fisher information matrix and the Fisher–Rao metric are local geometric objects on a parametric family; Fisher divergence is a discrepancy between two distributions. The shared appearance of score functions and Fisher information explains the naming overlap, but the mathematical roles are distinct [(Costa et al., 2012); (Toranzo et al., 2016)].

Across these uses, a consistent theme remains. Fisher divergence measures mismatch in differential structure, not just mismatch in mass allocation. That is why it supports score matching without normalizing constants, Fisher-based variational objectives, weighted calibrations of general Bayes procedures, stability analyses of score-matching dynamics, and several derived divergences that emphasize local oscillation and curvature.