Repulsive MMD Loss Variants

• Repulsive MMD Loss Variants are objective functions that modify traditional MMD by introducing repulsive interactions among samples, enhancing diversity and discrimination.
• They leverage specific kernel choices, such as the Riesz kernel, to reformulate energy-distance objectives and improve gradient signals in adversarial frameworks.
• Empirical results on datasets like CIFAR-10 and CelebA show that these variants achieve superior sample quality, computational efficiency, and robust convergence.

Repulsive MMD loss variants are objective functions and algorithmic strategies that modify the traditional Maximum Mean Discrepancy (MMD) loss to incorporate or emphasize explicitly repulsive interactions among samples, particularly within empirical measures, facilitating improved generative modeling and statistical matching. These variants arise either through particular kernel choices—such as the Riesz kernel leading to energy-distance objectives—or via rearrangements of within-group terms in adversarial training, as in MMD-GAN frameworks. The resulting losses not only reshape the optimization landscape but also yield computational and representational advantages that address earlier limitations in sample quality and efficiency.

1. Mathematical Structure and Repulsive MMD Objectives

The classic MMD loss measures the discrepancy between two distributions $\mu$ and $\nu$ using a symmetric kernel $K$, expressed as

$$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$

where the first two terms contract the distributions internally, and the cross-term is repulsive between them. When the Riesz kernel $K(x,y) = -\|x - y\|^r$ with $r \in (0,2)$ is used, the resulting interaction is purely repulsive within each sample distribution (negative sign on within-group terms) and attractive in the cross-term, yielding the so-called energy distance:

$$\mathrm{MMD}_K^2(\mu,\nu) = -\tfrac{1}{2} \mathbb{E}_{x,x'\sim\mu}\|x-x'\|^r - \tfrac{1}{2}\mathbb{E}_{y,y'\sim\nu}\|y-y'\|^r + \mathbb{E}_{x\sim\mu,\,y\sim\nu}\|x-y\|^r,$$

which coincides with Székely–Rizzo energy distance. For empirical measures, the discrete objective decomposes as:

• Repulsive energy: $$E(x) = -\frac{1}{2N^2} \sum_{i,j=1}^N \|x_i - x_j\|$$
• Attractive potential: $$V(x|y) = \frac{1}{MN}\sum_{i=1}^N\sum_{j=1}^M \|x_i - y_j\|$$

These terms create a dynamic where samples are encouraged to spread out within their group but contract across groups, with the repulsive term enhancing sample diversity and coverage (Hertrich et al., 2023).

2. Repulsive Loss in MMD-GANs

Standard MMD-GAN discriminator objectives contract each group internally due to negative within-group terms, often suppressing the learning of fine variations in real data. To address this, a modified "repulsive" discriminator loss is formulated by flipping the sign of the real–real term:

$$L_D^{\mathrm{rep}} = \mathbb{E}_{x,x'\sim P_{\mathrm{real}}}[k_D(x,x')] - \mathbb{E}_{y,y'\sim P_G}[k_D(y,y')]$$

Here, $\nu$0 is incentivized to expand real–real distances, promoting dispersion in the discriminator space and compelling the extraction of fine differences among real samples. The generator objective remains unchanged:

$\nu$1

A general one-parameter family,

$\nu$2

interpolates between purely attractive $\nu$3 and fully repulsive $\nu$4 losses. Experimental results confirm that regimes with $\nu$5 outperform those with $\nu$6 (Wang et al., 2018).

3. Computational Strategies and Scalability

Repulsive MMD loss variants, particularly those based on the Riesz kernel with $\nu$7, support efficient computation by virtue of their functional form. For high-dimensional data, the sliced MMD approach matches the full energy distance exactly for Riesz kernels:

$\nu$8

with $\nu$9. The remarkable result is that for these kernels, the sliced and full MMD coincide up to a constant, ensuring no information loss during projection.

For $K$0, all gradient components in one dimension can be computed via a sort-based algorithm in $K$1 time by leveraging closed-form rank-statistics, significantly improving over the $K$2 complexity typical of standard Gaussian-MMD. Approximation of gradients with $K$3 random projections yields an error of $K$4, requiring $K$5 slices for controlled accuracy (Hertrich et al., 2023).

The following table summarizes the key computational distinctions:

Kernel Type	Gradient Complexity	Approximability via Slices
Riesz $K$6	$K$7 via sorting	Exact; $K$8 error by sampling
Gaussian (RBF)	$K$9 (double sums)	No exact slicing; costlier approximations

4. Theoretical Interpretation and Stability

The repulsive MMD variants offer a theoretical alignment with objectives in linear discriminant analysis (LDA), maximizing both between-class and within-class variances for real data. This effect compels the discriminator network in MMD-GANs to become injective over the real sample manifold, enhancing expressivity and enabling the learning of fine-grained structures (Wang et al., 2018).

The stability of these variants is rigorously established: both attractive and repulsive MMD-GAN training dynamics are locally exponentially stable at any equilibrium where the generator matches the real distribution $$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$0. This guarantees robust convergence and justifies the practical adaptation of repulsive losses in adversarial frameworks.

5. Kernel Choices: Saturation and Boundedness

In practice, kernel choice for MMD critically influences sample diversity and training dynamics. Standard RBF kernels saturate for large or small inter-sample distances, which can cause vanishing or excessively flat gradients. The introduction of a "bounded" RBF kernel (RBF-B), inspired by hinge loss clipping,

$$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$1

with practical thresholds $$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$2, $$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$3, and $$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$4, mitigates this problem by preventing kernel values from saturating, improving adversarial interplay stability. Visualizations confirm that the use of repulsive loss and bounded kernels leads to class-level separation and cluster differentiation in feature space (Wang et al., 2018).

6. Empirical Performance and Quantitative Results

Repulsive MMD loss variants demonstrate superior empirical performance on standard image datasets compared to traditional MMD and other GAN losses. On benchmarks such as CIFAR-10, STL-10, CelebA, and LSUN Bedroom, the MMD-rep and MMD-rep-b losses yield significantly lower Fréchet Inception Distances (FID), indicating improved sample quality:

Dataset	MMD-rbf	MMD-rep	MMD-rep-b	Hinge GAN
CIFAR-10	28.38	16.65	16.21	23.46
STL-10	57.52	36.67	37.63	49.44
CelebA	13.03	7.20	6.79	—
LSUN-bedroom	—	16.91	12.52	16.73

Repulsive regimes ($$\mathrm{MMD}^2_K(\mu,\nu) = \mathbb{E}_{x,x'\sim\mu}[K(x,x')] + \mathbb{E}_{y,y'\sim\nu}[K(y,y')] - 2\,\mathbb{E}_{x\sim\mu,\,y\sim\nu}[K(x,y)],$$5) consistently outperform attractive ones, and ablations indicate robust gains across architectures and kernel choices. Increased discriminator output dimensionality (from 1 to 16–64) further improves sample quality, and stability is retained under spectral normalization (PICO method) (Wang et al., 2018).

7. Connections and Significance in Generative Modeling

Repulsive MMD loss variants constitute an influential unifying framework for both kernel-based generative modeling and adversarial training paradigms. The use of repulsion—in both unsupervised flows via energy-distance and in MMD-GAN discriminators—amplifies fine-grained detail capture, disperses samples appropriately, and strengthens gradient signals for effective learning. These approaches scale efficiently to high-dimensional data, reconcile with sliced discrepancy formulations, and provide robust, stable optimization dynamics even in large-scale image generation tasks (Hertrich et al., 2023, Wang et al., 2018).