Energy-Distance Training in Generative Models

• Energy-distance training is defined as a distribution-matching objective that uses attraction to data and repulsion among samples to establish a strictly proper scoring rule.
• It employs spectral formulations with multi-scale STFT magnitude spectrograms, ensuring perceptually faithful comparisons in waveform speech synthesis.
• The method utilizes unbiased minibatch estimates and repulsive terms to prevent sample collapse, achieving consistency and improved synthesis metrics.

Energy-distance training denotes a family of training objectives for implicit generative models in which learning is driven by a generalized energy distance between the model distribution and the data distribution, rather than by an analytical likelihood or a purely adversarial game. In speech, the approach was introduced for fully parallel waveform synthesis in "A Spectral Energy Distance for Parallel Speech Synthesis" (Gritsenko et al., 2020), where the distance is defined on multi-scale magnitude spectrograms, and later applied to autoregressive continuous-latent speech language modeling in "Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space" (Ma et al., 19 May 2025). Across these instantiations, the central mechanism is the same: a sample-to-data attraction term is balanced by a sample-to-sample repulsive term, yielding a strictly proper scoring rule under the stated conditions and providing a stable objective for training implicit models.

1. Classical generalized energy distance

Let $P$ and $Q$ be two probability distributions over $\mathbb{R}^d$. The generalized energy distance is defined as

$$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$

Equivalently, if $d(x,y)=\|x-y\|$ is any negative-definite distance, then

$$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$

Minimizing this distance drives $Q\to P$ in expectation (Gritsenko et al., 2020).

The later SLED formulation writes the same construction as a squared generalized energy distance on a metric space $(\mathbb{R}^n,d)$:

$$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$

A common choice is $d(x,y)=\|x-y\|_2^\beta$ with $Q$0; when $Q$1, this becomes $Q$2 (Ma et al., 19 May 2025).

The defining structural feature of energy-distance training is therefore not a particular architecture, but the use of a distribution-matching objective that depends only on samples from the model. This makes the method naturally compatible with implicit generators for which no tractable density is available.

2. Spectral formulation for waveform speech synthesis

In high-dimensional waveform space, Euclidean distances on raw samples poorly capture perceptual similarity. The spectral formulation replaces raw-sample comparison with a distance on multi-scale STFT magnitude spectrograms. If

$Q$3

denotes the $Q$4-th frame of the STFT of $Q$5 with window size $Q$6, the distance between waveforms $Q$7 and $Q$8 is defined as

$Q$9

Typical choices are $\mathbb{R}^d$0 and $\mathbb{R}^d$1 (Gritsenko et al., 2020).

Inserting this $\mathbb{R}^d$2 into the generalized energy distance yields the spectral energy distance. The paper characterizes this objective as a proper scoring rule with respect to the distribution over magnitude-spectrograms of the generated waveform audio (Gritsenko et al., 2020).

The construction is explicitly multi-scale. The magnitude term uses an $\mathbb{R}^d$3 difference on spectrogram frames, while the log-magnitude term uses an $\mathbb{R}^d$4 difference. The weighting $\mathbb{R}^d$5 is used to roughly balance gradient magnitudes across scales. The paper also reports that multi-scale spectrograms with window sizes $\mathbb{R}^d$6 outperform any single scale (Gritsenko et al., 2020).

3. Unbiased minibatch estimation and optimization

For conditional TTS, the training data are pairs $\mathbb{R}^d$7. An unbiased minibatch estimator of the energy score is obtained by drawing, for each $\mathbb{R}^d$8, two independent noise vectors $\mathbb{R}^d$9 and $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$0, then generating

$$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$1

The minibatch loss is

$$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$2

Because $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$3, this estimator is unbiased (Gritsenko et al., 2020).

The training step described in the paper samples a minibatch of $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$4, samples $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$5, computes $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$6 and $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$7, evaluates the multi-scale spectrogram distances, and updates $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$8 with Adam. The reported hyper-parameters are Adam with $$E_D(P,Q) = 2\,\mathbb{E}_{X\sim P,\,Y\sim Q}[\|X-Y\|] - \mathbb{E}_{X,X'\sim P}[\|X-X'\|] - \mathbb{E}_{Y,Y'\sim Q}[\|Y-Y'\|].$$9, learning rate $d(x,y)=\|x-y\|$0 with $d(x,y)=\|x-y\|$1k-step linear warmup, batch size $d(x,y)=\|x-y\|$2 across $d(x,y)=\|x-y\|$3-way TPU shards, and $d(x,y)=\|x-y\|$4 decay $d(x,y)=\|x-y\|$5 (Gritsenko et al., 2020).

The implementation details are integral to the method’s practical stability. STFTs use Hann windows, $d(x,y)=\|x-y\|$6 overlap, and an $d(x,y)=\|x-y\|$7 overcomplete DCT/DST basis for stable gradients. All sums over time-frequency frames are fused into convolution-style tensor operations for GPU/TPU efficiency. Gradients of $d(x,y)=\|x-y\|$8 backpropagate through the STFT operations and the generator network $d(x,y)=\|x-y\|$9 because all components are differentiable (Gritsenko et al., 2020).

4. Properness, consistency, and the role of repulsion

The theoretical basis for energy-distance training is the energy score

$$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$0

By Gneiting and Raftery (2007), Theorem 5.1, this score is a strictly proper scoring rule whenever $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$1 is a negative-definite distance. In the spectral formulation, each term

$$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$2

and

$$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$3

is negative-definite, and a weighted sum of proper scoring rules remains strictly proper. The resulting spectral energy distance is therefore guaranteed to uniquely identify the true conditional spectrogram distribution in the infinite-data limit; under mild model-capacity assumptions, minimizing this loss is statistically consistent (Gritsenko et al., 2020).

The repulsive term is not incidental. In both the waveform and continuous-latent formulations, it is the mechanism that distinguishes energy-distance training from plain regression. SLED states this directly: a plain regression loss such as RMSE only matches means and collapses variance, whereas the repulsive $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$4 term prevents sample collapse (Ma et al., 19 May 2025).

A recurrent misconception is that the objective is merely a sample-to-target reconstruction loss with an auxiliary regularizer. The formulas in both papers do not support that interpretation. The attraction term and the repulsive term are coequal parts of the scoring rule, and removing the latter changes the objective qualitatively rather than marginally. The empirical ablations reported in both papers are consistent with this interpretation (Gritsenko et al., 2020).

5. Empirical behavior in fully parallel speech synthesis

On a single-speaker $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$5 dataset, the waveform study evaluates four model families with conditional Fréchet Deep Speech Distance (cFDSD), unconditional FDSD, and Mean Opinion Score (MOS). The valid-set values reported in Table 1 are as follows (Gritsenko et al., 2020).

Model	MOS	cFDSD
GAN-TTS (published)	–	0.060
GAN-TTS (re-impl.)	4.16±0.06	0.077
GED (same generator)	4.03±0.06	0.038
GED (no repulsive term)	3.00±0.07	0.048
GED + iSTFT generator	4.10±0.06	0.037
GED + unconditional GAN	4.25±0.06	0.040

These results establish several properties of the training method. Adding the repulsive term halves cFDSD versus GAN-TTS alone, indicating a substantially better match to the conditional waveform distribution. Dropping the repulsive term collapses samples and degrades MOS to approximately $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$6. The fully parallel inverse-STFT generator converges fastest, at approximately $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$7k steps, and achieves the best FDSD, although its MOS of $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$8 still lags slightly. Combining GED with an unconditional GAN discriminator yields the best MOS of $$E_D(P,Q)=\mathbb{E}\bigl[2\,d(X,Y)-d(X,X')-d(Y,Y')\bigr].$$9, surpassing prior GAN-TTS (Gritsenko et al., 2020).

The paper’s practical recommendations follow from these findings. The repulsive term is described as essential; without it, the model collapses to the conditional mean or produces spectrally metallic artifacts. The GED loss alone excels at capturing fine-grained local conditional structure but can sound robotic, whereas adding an unconditional GAN loss helps recover naturalness. Training is reported to be rapid, with best cFDSD in $Q\to P$0–$Q\to P$1k steps, but the models can overfit if run to $Q\to P$2M steps, so early stopping or regularization is advisable (Gritsenko et al., 2020).

6. Continuous-latent autoregressive modeling

The SLED model extends energy-distance training from waveform or spectrogram space to autoregressive modeling in continuous latent space. Its architecture has two modules: an autoregressive network $Q\to P$3, implemented as a Transformer, which consumes all past latents $Q\to P$4 and text tokens when conditioning on a prompt, and produces

$Q\to P$5

and a lightweight implicit generator $Q\to P$6, implemented as an MLP plus AdaLN, which maps $Q\to P$7 to $Q\to P$8 (Ma et al., 19 May 2025).

The per-token loss used in SLED is

$Q\to P$9

where $(\mathbb{R}^n,d)$0 is the ground-truth latent vector at step $(\mathbb{R}^n,d)$1, and $(\mathbb{R}^n,d)$2 is the implicit distribution defined by the conditional sampler $(\mathbb{R}^n,d)$3. Operationally, each step runs $(\mathbb{R}^n,d)$4 on the ground-truth prefix, samples two independent Gaussian noise vectors, computes two simulated latents

$(\mathbb{R}^n,d)$5

retrieves the true latent $(\mathbb{R}^n,d)$6 from the encoder, and forms

$(\mathbb{R}^n,d)$7

Back-propagation flows through both $(\mathbb{R}^n,d)$8 and $(\mathbb{R}^n,d)$9 (Ma et al., 19 May 2025).

The reported hyper-parameters are $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$0 for the $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$1 distance, two noise samples per step, batch size $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$2, $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$3K steps, AdamW with learning rate $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$4, weight decay $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$5, warmup $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$6K steps, and classifier-free guidance strength $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$7 at inference. Computational complexity per step is dominated by the Transformer forward on the prefix plus two MLP passes of dimension $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$8 (Ma et al., 19 May 2025).

The empirical ablation against RMSE is sharp. Replacing the GED loss by simple RMSE causes catastrophic collapse: in zero-shot continuation on LibriSpeech, WER-C jumps from $$\mathrm{GED}_{d}^{2}(P,Q) = \mathbb{E}_{x,x'\sim P\atop y,y'\sim Q} \bigl[ 2\,d(x,y)-d(x,x')-d(y,y') \bigr].$$9 under GED to over $d(x,y)=\|x-y\|_2^\beta$0 under RMSE. Relative to discrete language modeling with residual vector quantization and hierarchical architectures, SLED reports WER-C of approximately $d(x,y)=\|x-y\|_2^\beta$1 and SIM of approximately $d(x,y)=\|x-y\|_2^\beta$2, compared with VALL-E at approximately $d(x,y)=\|x-y\|_2^\beta$3 WER-C. Relative to other continuous models, MELLE reaches approximately $d(x,y)=\|x-y\|_2^\beta$4 WER-C but requires a post-AR network, while FELLE incurs heavy multi-step sampling. SLED is described as purely autoregressive and as supporting streaming synthesis; interleaving text and speech at a $d(x,y)=\|x-y\|_2^\beta$5 or $d(x,y)=\|x-y\|_2^\beta$6 ratio yields WER-C of approximately $d(x,y)=\|x-y\|_2^\beta$7–$d(x,y)=\|x-y\|_2^\beta$8 and DNSMOS of approximately $d(x,y)=\|x-y\|_2^\beta$9–$Q$00, close to offline performance at WER-C of approximately $Q$01 and DNSMOS of approximately $Q$02 (Ma et al., 19 May 2025).

7. Limitations, scope, and open questions

The spectral formulation has explicit limitations. The scoring rule is strict on spectrogram slices but does not fully enforce long-range time dependencies in the waveform. Improvements in FDSD do not always translate to higher MOS, which motivates the paper’s statement that better perceptual metrics are needed. Combining spectral GED with more advanced GAN architectures or flow-based models is left to future work (Gritsenko et al., 2020).

The empirical record also indicates that energy-distance training alone and energy-distance training augmented with adversarial losses optimize different aspects of synthesis quality. In the waveform study, GED alone produces strong cFDSD and stable, non-adversarial training, but adding an unconditional GAN improves MOS. This does not contradict the consistency result: the strict properness claim is with respect to the chosen distance and representation, whereas MOS is a perceptual evaluation protocol rather than the optimized scoring rule (Gritsenko et al., 2020).

A broader implication suggested by the two papers together is that the method’s effective domain is determined by the choice of representation space and distance. In one case, the relevant space is multi-scale STFT magnitude spectrograms derived from audio waveforms; in the other, it is a sequence of continuous latent vectors generated autoregressively. What remains invariant is the training principle: distributional matching by attraction to data and repulsion among model samples, implemented without requiring an explicit density and without the min–max optimization of GANs (Ma et al., 19 May 2025).