Reverse Fisher Divergence in Gaussian Mixtures

• Reverse Fisher divergence is a score-matching objective that averages the score discrepancy under the student distribution, yielding a modified optimization landscape.
• It enhances convergence behavior in Gaussian mixture models by focusing the training loss on regions where the student density places mass.
• Gradient descent analyses, supported by Lyapunov functions, prove global convergence under proper initialization and separation conditions.

Reverse Fisher divergence is a score-matching objective in which the score discrepancy between a “student” density and a “teacher” density is averaged under the student distribution rather than the teacher distribution. In the Gaussian-mixture setting studied in "Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence" (Tyurin, 18 Jun 2026), this change in measure leads to markedly different optimization behavior from the standard forward Fisher divergence. The paper analyzes plain gradient descent for fitting Gaussian mixture models (GMMs) and proves global convergence for a single-Gaussian teacher from arbitrary initializations, then extends the analysis to mixture teachers under global random initialization and a separation assumption on the target means (Tyurin, 18 Jun 2026).

1. Definition and role in score matching

Let $p^*(x)$ denote the teacher density and $p_\theta(x)$ the student density, both supported on $\mathbb R^d$. Their score functions are

$$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$

The reverse Fisher divergence is defined by

$D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$

The defining feature is that the expectation is taken with respect to the student distribution. In the terminology of the paper, this is the alternative to the standard forward Fisher divergence, where the expectation is taken with respect to the teacher distribution (Tyurin, 18 Jun 2026).

Within the paper’s framing, score matching is described as a central training objective in modern generative modeling, diffusion models, fitting unnormalized statistical models, and inverse problems. The reverse Fisher formulation is introduced because recent results show that, even in simple Gaussian mixture settings, the forward Fisher objective can exhibit undesirable and initialization-dependent convergence behavior. The reverse objective is studied precisely as a modification of the optimization landscape rather than as a change in the score discrepancy itself.

2. Gaussian-mixture formulation

When both teacher and student are Gaussian mixtures with fixed mixing weights $\{w_i\}$ and identity covariance,

$p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$

and similarly for $p^*$. In this case, the reverse Fisher objective can be written as

$F^{\rm rev}(\{\mu_i\}) = \E_{x\sim p_{\theta}} \biggl\| \sum_{i=1}^{n} r_i(x)\,(x-\mu_i) - \sum_{a=1}^{m} t_a(x)\,(x-\mu_a^*) \biggr\|^2,$

where $r_i(x)$ and $p_\theta(x)$0 are the usual responsibilities (Tyurin, 18 Jun 2026).

This representation makes the objective depend on two posterior-averaged fields: the student mixture’s local reconstruction of $p_\theta(x)$1, and the analogous teacher quantity. The paper’s notation later packages these into mean fields, which is useful for analyzing the gradient flow. In this formulation, reverse Fisher divergence is not merely a statistical discrepancy; it is an objective whose geometry is shaped by where the current student already places mass. This is the basis for the paper’s claim that the reverse objective has a more favorable optimization landscape than the forward objective in the GMM setting.

3. Gradient-descent dynamics

The optimization procedure considered is plain gradient descent with step size $p_\theta(x)$2: $p_\theta(x)$3

In the GMM-on-GMM setting with fixed unit covariances and uniform weights, the paper derives the exact per-component gradient

$p_\theta(x)$4

where

$p_\theta(x)$5

$p_\theta(x)$6

Here $p_\theta(x)$7 is the student mean field and $p_\theta(x)$8 is the teacher mean field (Tyurin, 18 Jun 2026).

A particularly important simplification occurs when the teacher is a single Gaussian, $p_\theta(x)$9, with mean $\mathbb R^d$0. Then the student means obey a linear-algebraic iteration

$\mathbb R^d$1

where each $\mathbb R^d$2 is symmetric and positive semidefinite. This matrix form is the starting point for the global-convergence proof in the single-Gaussian case.

4. Global convergence theorems

For a single-Gaussian teacher $\mathbb R^d$3, with student an $\mathbb R^d$4-component GMM with fixed weights and identity covariances, define

$\mathbb R^d$5

If the step size satisfies

$\mathbb R^d$6

then for all $\mathbb R^d$7 and each student mean $\mathbb R^d$8, the paper gives an explicit error bound. It further states that achieving accuracy $\mathbb R^d$9 in mean-squared error requires only

$$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$0

which is described as logarithmic in $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$1 and as having no explicit dependence on $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$2 when $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$3 (Tyurin, 18 Jun 2026).

For a mixture-of-Gaussians teacher, let the teacher have $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$4 components with unknown means $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$5 spaced by at least $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$6, and let the student have $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$7 components. The initialization is global and random: all $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$8 are drawn uniformly on a sphere of radius $$s_\theta(x) := \nabla_x \log p_\theta(x), \qquad s^*(x) := \nabla_x \log p^*(x).$$9, and the step size is chosen as $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$0. Under the conditions

• $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$1 and $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$2,
• $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$3,

the paper proves that, with probability at least $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$4, each student mean converges to within $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$5 of its nearest teacher mean. More precisely, after

$D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$6

iterations, for every $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$7 there is an assignment $D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$8 such that

$D_F^{\rm rev}\bigl(p_{\theta}\,\|\,p^{*}\bigr) \;=\; \E_{x\sim p_{\theta}}\bigl[\|\,s_{\theta}(x)-s^{*}(x)\|_{2}^{2}\bigr].$9

The paper also establishes a total-variation guarantee. If in addition $\{w_i\}$0, then with the same high-probability event the student density satisfies

$\{w_i\}$1

where $\{w_i\}$2 is the probability that a random direction lands teacher-mode $\{w_i\}$3. In the symmetric-mode case, the conclusion becomes $\{w_i\}$4.

These results delimit the scope of the global guarantees. Arbitrary initialization is proved for the single-Gaussian teacher case; for mixture teachers, the guarantees require both random global initialization and a separation condition on the target means.

5. Lyapunov analysis and optimization structure

The core proof mechanism for the single-Gaussian case is a Lyapunov analysis of the matrix iteration

$\{w_i\}$5

The Lyapunov function is

$\{w_i\}$6

The matrices $\{w_i\}$7 satisfy two key properties: $\{w_i\}$8 and

$\{w_i\}$9

From these, a one-step descent estimate yields

$p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$0

and therefore

$p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$1

Choosing $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$2 guarantees strict decrease until $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$3, after which a second-regime argument gives sublinear convergence to zero (Tyurin, 18 Jun 2026).

The paper identifies the structural lemmas as the symmetry and positive semidefiniteness of $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$4, the operator-norm bound, and the uniform lower bound on the quadratic form $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$5. In the paper’s interpretation, these are the ingredients that ensure global monotonicity of the Lyapunov function and produce explicit rates.

6. Relation to forward Fisher divergence

The forward Fisher divergence considered for comparison is

$p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$6

According to the paper’s empirical discussion, gradient descent on this standard forward objective often becomes trapped in wrong modes or produces spurious stationary points, even in the simplest single-Gaussian-versus-two-component GMM setting. By contrast, the reverse objective “focuses” the loss where the student mass lies and yields an almost convex-like, spurious-free landscape, which the paper presents as the explanation for the uniform, initialization-agnostic convergence theorems in the single-Gaussian case and the high-probability convergence theorems in the mixture case (Tyurin, 18 Jun 2026).

A common misunderstanding is to treat the difference between forward and reverse Fisher divergences as a minor change of sampling measure. In the GMM analysis of this paper, the distinction is substantive: the objective with expectation under $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$7 supports a Lyapunov-based proof of global convergence properties that are not available for the forward objective in the same form. At the same time, the guarantees are not universal. The mixture-teacher results explicitly depend on a global random initialization scheme, a $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$8-separation assumption on the target means, and additional conditions for total-variation convergence. Within those assumptions, the paper’s central conclusion is that swapping the expectation from $p_\theta(x)=\sum_{i=1}^{n} w_i \,\phi(x-\mu_i),$9 to $p^*$0 produces a more benign optimization problem for score matching of Gaussian mixtures.