---
title: Multi-Mel Discriminator in NAR-TTS
url: https://www.emergentmind.com/topics/multi-mel-discriminator-mmd
type: topic
---

# Multi-Mel Discriminator in NAR-TTS

Multi-Mel Discriminator (MMD) denotes, in an inferred but technically precise sense, a discriminator that operates in Mel-spectrogram space and performs multi-scale discrimination for GAN-based non-autoregressive text-to-speech. The design most directly associated with this notion is the multi-scale time-frequency spectrogram discriminator, which treats a Mel-spectrogram as a 2D image over time and frequency and uses a U-Net-based encoder–decoder to produce coarse-grained and fine-grained discriminator outputs for adversarial training of Mel prediction [2203.01080]. The acronym is terminologically overloaded, because “MMD” is also the standard abbreviation for Maximum Mean Discrepancy in kernel GAN literature [1801.01401]; that overlap is relevant rather than incidental, since later MMD-GAN work explicitly discusses extensions to Multi-Mel-style audio discriminators [1812.09916].

## 1. Conceptual role in non-autoregressive TTS

In GAN-based non-autoregressive TTS, the discriminator is introduced to compensate for the weakening of generative modeling that accompanies the removal of autoregression. The specific problem addressed by the spectrogram-domain design is that NAR-TTS can accelerate inference while producing Mel-spectrograms that are less natural and less faithful than desired. Rather than redesigning the generator with flows, VAEs, or diffusion, the discriminator is attached to an existing Mel-spectrogram–predicting NAR-TTS system and trained to distinguish real from generated Mel-spectrograms. Because the discriminator is spectrogram-based, it can be inserted into virtually any such system [2203.01080].

The central modeling decision is to treat the spectrogram as a 2D image in the time–frequency plane. This reflects the observation that Mel-spectrograms encode structure across both axes: spectral shape, harmonics, pronunciation, prosody, and timbre. The stated motivation is that this structure is simultaneously local and global in time and frequency, so a discriminator restricted to 1D temporal convolutions or per-frame processing fails to exploit correlations that are intrinsic to the spectrogram representation. A Multi-Mel discriminator, in this sense, is therefore not merely a classifier on Mel frames; it is a time–frequency critic intended to capture local fine-grained details such as harmonic clarity and local smoothness, together with coarse-grained organization such as phrase-level prosody and long-term spectral balance.

A common misconception is that “multi-scale” here necessarily means multiple Mel front ends or multiple STFT parameterizations. In the design under discussion, multi-scale discrimination is realized internally by a U-Net hierarchy operating on a single Mel-spectrogram. The Mel representation is already downsampled relative to waveform space and is not re-parameterized into separate frequency scales. Instead, the encoder produces coarse features and the decoder reconstructs fine features, yielding two discriminator outputs on the same Mel input. This is already a multi-scale Mel-domain discriminator in the sense relevant to NAR-TTS.

## 2. Spectrogram-domain architecture

The discriminator input is a spectrogram of shape \((1, T, N)\), with one channel, \(T\) time frames, and \(N\) frequency bins. In the reported experiments, the representation is an 80-dimensional log-Mel spectrogram with frameshift \(12.5\) ms and frame length \(50\) ms. These Mel-spectrograms are produced by a simplified ParallelTacotron2-like NAR-TTS system and are used both as real targets \(S_r\) and generated samples \(S_f\) [2203.01080].

The architecture is U-Net-based. The encoder is a stack of 2D convolutional layers with strided down-sampling in both time and frequency. Starting from \((1, T, N)\), the encoder outputs a feature map of shape \((256, T/8, N/8)\). Each convolutional layer is wrapped by weight normalization, and LeakyReLU with \(\alpha = 0.2\) is applied for all but the input layer. A final Conv2D output layer on top of the encoder produces the coarse-grained discriminator output \(C\), with approximate shape \((1, T/8, N/8)\).

The decoder is symmetrical to the encoder and uses transposed convolutions to upsample back to input resolution. At each decoding stage, the input is the concatenation of the previous decoder feature and the encoder feature at the corresponding resolution, following the standard U-Net skip-connection pattern. This combines low-resolution global context with high-resolution local structure. After the final upsampling stage, another Conv2D output layer produces the fine-grained output \(F\) with the same resolution as the input spectrogram, \((1, T, N)\). The discriminator also exposes hidden features \(H\), defined as all intermediate feature maps across encoder and decoder. The forward pass is written as
\[
C_r, F_r, H_r = Discriminator(S_r)
\]
\[
C_f, F_f, H_f = Discriminator(S_f).
\]

This architecture makes the meaning of “multi-scale” explicit. The bottleneck pathway yields a coarse discriminator map, while the decoder pathway yields a fine discriminator map. The coarse map is intended to capture global structure and smoothness; the fine map is intended to capture detailed local spectral patterns. The paper’s own comparison framework situates this model against a single-scale time discriminator and a multi-scale time-only discriminator:

| Variant | Core design | Reported outcome |
|---|---|---|
| S-T | encoder only, 1D convolutions along time axis | baseline |
| M-T | encoder and decoder, multi-scale along time only | preferred over S-T |
| M-TF | 2D U-Net on time-frequency with coarse and fine outputs | preferred over M-T |

A second misconception is that a Multi-Mel discriminator must consume multiple Mel resolutions as separate inputs. That is not required by this formulation. The reported model uses one 80-bin Mel-spectrogram per utterance, and the multi-scale property is induced by the encoder–decoder hierarchy itself.

## 3. Objective functions and training integration

The discriminator and generator are trained with least-squares GAN losses and a feature-matching term. For each batch, a fake spectrogram is generated from text,
\[
S_f = TTS(text),
\]
after which real and fake Mel-spectrograms are passed through the discriminator to obtain \(C_r, F_r, H_r\) and \(C_f, F_f, H_f\). The discriminator loss is
\[
\begin{aligned}
L_d = & \; MSE(1, C_r) + MSE(1, F_r) \\
      & + MSE(0, C_f) + MSE(0, F_f).
\end{aligned}
\]
Both the coarse-grained output \(C\) and the fine-grained output \(F\) are therefore regressed toward \(1\) for real samples and \(0\) for fake samples [2203.01080].

The generator-side adversarial loss is
\[
L_a = MSE(1, C_f) + MSE(1, F_f),
\]
which forces the generator to fool the discriminator at both scales. Feature matching is defined as
\[
L_f = MAE(H_f, H_r),
\]
with MAE computed across corresponding hidden layers and averaged. The full generator loss is
\[
L_g = L_{tts} + \lambda_a L_a + \lambda_f L_f.
\]
For the U-Net-based discriminators, the reported hyperparameters are \(\lambda_a = 0.2\) and \(\lambda_f = 2\); for the S-T and M-T baselines, \(\lambda_a = 1\) and \(\lambda_f = 10\).

The generator is a simplified ParallelTacotron2. It takes a phoneme sequence with punctuation, encodes it, upsamples the encoder states to frame-level representations with a learned up-sampler that uses explicit phoneme-level durations, and decodes to an 80-dimensional log-Mel spectrogram. The TTS loss is
\[
L_{spec} = MSE(S,\hat S) + MAE(S,\hat S),
\]
\[
L_{dur} = MSE(D,\hat D) + MAE(D,\hat D),
\]
\[
L_{tts} = L_{spec} + \lambda_{dur}L_{dur},
\]
with \(\lambda_{dur} = 0.02\). Ground-truth durations are obtained from Montreal Forced Aligner during training, which avoids costly Soft-DTW.

The training loop alternates discriminator and generator optimization. After the discriminator is updated with \(L_d\), the discriminator outputs are recomputed with the updated discriminator before the generator update, and then \(L_a\), \(L_f\), and \(L_{tts}\) are combined into \(L_g\). At inference, the discriminator is discarded; only the TTS model is used to map text to a predicted Mel-spectrogram, which is then converted to waveform by a vocoder or by Griffin-Lim.

## 4. Empirical behavior in NAR-TTS

The reported experiments use LJSpeech, described as a single-speaker English dataset of approximately \(10\) hours, with \(11000\) text–audio pairs at \(16\) kHz for training. Two vocoder conditions are considered: Griffin-Lim and HiFi-GAN trained on the same dataset, with HiFi-GAN evaluated both with and without fine-tuning on TTS predictions. Training uses RAdam with \((\beta_1 = 0.9, \beta_2 = 0.999)\) plus Lookahead with \((k = 5, \alpha = 0.5)\). The learning rate is exponentially decayed from \(1\times 10^{-3}\) to \(1\times 10^{-5}\) after \(20\)k iterations, training lasts \(200\)k iterations, and the batch size is \(64\). Weight normalization is used in every discriminator convolution layer, and LeakyReLU\((0.2)\) is applied to all discriminator layers except the input layer [2203.01080].

Subjective preference tests isolate both the contribution of multi-scale processing and the contribution of explicit time–frequency modeling. In the comparison between S-T and M-T, M-T is preferred in \(49.58\%\) of cases, which is described as significant and interpreted as evidence that multi-scale discrimination improves prosody and smoothness even in a 1D time-only setting. In the comparison between M-T and M-TF, M-TF is preferred in \(61.11\%\) of cases, described as very significant. The paper attributes the gain to better “spectral clarity, smoothness, and continuity” when the spectrogram is treated as a 2D object.

Mean-opinion-score results further separate the effect of GAN training from the effect of vocoder fine-tuning:

| System | MOS |
|---|---|
| No GAN, no fine-tuning | \(2.91 \pm 0.14\) |
| No GAN, vocoder fine-tuned | \(3.43 \pm 0.15\) |
| GAN with proposed discriminator, no fine-tuning | \(\mathbf{3.81 \pm 0.15}\) |
| GAN + vocoder fine-tuning | \(3.71 \pm 0.14\) |
| Analysis–synthesis (HiFi-GAN on real Mels) | \(3.97 \pm 0.17\) |

These results establish two technical points. First, GAN training with the proposed multi-scale spectrogram discriminator improves naturalness more than vocoder fine-tuning alone. Second, combining GAN training and vocoder fine-tuning does not necessarily help and can slightly degrade fidelity; the stated explanation is that the generated spectrograms may become harder for the vocoder to handle. Visualization of the discriminator maps is consistent with the intended decomposition of roles: the coarse-grained map, when upsampled to input resolution, is smooth and averaged, whereas the fine-grained map is sharper and more localized, highlighting details such as formants.

## 5. Relation to Maximum Mean Discrepancy

The label “MMD” is ambiguous because it also names Maximum Mean Discrepancy, a kernel integral probability metric. For probability distributions \(P\) and \(Q\) with kernel \(k\), the squared MMD is
\[
M_k^2(P,Q) = \mathbb E_{x,x'\sim P}[k(x,x')] -2\mathbb E_{x\sim P,y\sim Q}[k(x,y)] +\mathbb E_{y,y'\sim Q}[k(y,y')],
\]
and in MMD-GAN the discriminator is not a scalar classifier but a feature extractor \(D : \mathcal X \to \mathbb R^d\), so the kernel is applied in feature space rather than directly on raw inputs [1812.09916].

This kernel meaning of MMD is relevant to Multi-Mel discriminators for two reasons. First, “Demystifying MMD GANs” formalizes MMD as an IPM over the unit ball of an RKHS, uses learned feature maps \(h_\theta\), and shows that for a fixed critic the sample gradient of the unbiased estimator of \(\mathrm{MMD}^2\) is an unbiased estimator of the population gradient; however, once the critic is itself learned from finite data, generator gradients become biased because the effective objective involves a supremum over critic parameters [1801.01401]. Second, “Improving MMD-GAN Training with Repulsive Loss Function” explicitly states that the same ideas apply to domain-specific discriminators, including audio and speech GANs with Multi-Mel discriminators. In that formulation, one can define per-scale feature extractors \(D_s\), compute MMDs over the resulting Mel-resolution feature distributions, and train the generator with
\[
L_G = \sum_s M_{k_s\circ D_s}^2(P_X^{(s)}, P_G^{(s)}),
\]
while training the discriminator branches with repulsive losses \(\sum_s L_{D,s}^{\text{rep}}\) and bounded kernels for stability [1812.09916].

Theoretical work on what MMD compares further sharpens the connection. “How many moments does MMD compare?” shows that, through a pseudo-differential-operator factorization of a Mercer kernel, MMD can be interpreted as comparing a finite number of local moments determined by the singular-value decay of the associated symbol. In the truncated case, the critic compares approximately \(r^\ast\) local moments, where \(r^\ast\) depends on how quickly singular values decay [2106.14277]. This suggests that a kernelized Multi-Mel discriminator would not compare “all” aspects of the Mel distribution uniformly; rather, it would compare a finite collection of feature-wise, scale-dependent local expectations whose effective number is controlled by the kernel and feature extractor.

A common misunderstanding is therefore to conflate Multi-Mel discrimination with MMD-GAN training. The former is an architectural notion centered on Mel-spectrogram inputs and multi-scale critic structure; the latter is a kernel discrepancy objective defined on feature distributions. The two are compatible, but they are not synonymous.

## 6. Extensions, limitations, and unresolved design choices

The reported multi-scale time-frequency discriminator already functions as a Mel-domain multi-scale discriminator, but the paper also describes an explicit extension to “true Multi-Mel.” The suggested construction is to derive multiple Mel-spectrograms from the same waveform, for example with different Mel filterbank sizes such as \(80\), \(40\), and \(160\) bins or with different time resolutions, and then feed each representation into a separate or shared U-Net-style branch whose losses are aggregated. This is presented as a straightforward extension analogous to multi-resolution waveform discriminators, but it is not implemented in the paper [2203.01080].

Several implementation details remain unspecified in the reported NAR-TTS setup. The paper does not explicitly state any Mel normalization scheme, and it does not state exact STFT FFT size or padding. A plausible implication is that practical implementations must supply these details independently. The paper also does not numerically specify convolution kernel sizes and strides beyond the fact that 2D strided convolutions reduce the representation to \((T/8, N/8)\) and that transposed convolutions restore the original resolution. These omissions do not change the architecture class, but they matter for reproduction.

The broader MMD-GAN literature introduces additional limitations that become salient if a Multi-Mel discriminator is combined with Maximum Mean Discrepancy objectives. Repulsive MMD losses can be more unstable than attractive ones if learning rates are too large, Lipschitz constraints are too weak, or the parameter \(\lambda\) in the generalized discriminator family is too large; bounded-kernel hyperparameters such as \(b_l\), \(b_u\), and \(\sigma\) are fixed heuristically; PICO-based spectral normalization is more expensive; and the feature dimension at the discriminator output must exceed \(1\) for the repulsive formulation to work well [1812.09916]. Theoretical guarantees are also local rather than global: local exponential stability near equilibrium is studied, but global convergence is not guaranteed.

These constraints frame the present status of the Multi-Mel discriminator concept. In its narrowest and most directly evidenced sense, it denotes a U-Net-based, coarse-to-fine, 2D Mel-spectrogram discriminator for GAN-based NAR-TTS. In a broader sense, it marks a design space in which multiple Mel resolutions, feature-distribution matching, repulsive kernel objectives, and multi-branch critics can be combined. The available evidence shows that multi-scale and time–frequency discrimination improve naturalness and fidelity in NAR-TTS, while the kernel MMD literature indicates how a Multi-Mel discriminator could be generalized into a feature-distribution matcher with explicitly controlled sensitivity to local moments and scale-dependent structure.

Source: https://www.emergentmind.com/topics/multi-mel-discriminator-mmd