MIND: Monge Inception Distance for Generative Models Evaluation

Abstract: We propose the Monge Inception Distance (MIND), a metric for evaluating generative models that addresses key limitations of the widely adopted Fréchet Inception Distance (FID). The MIND metric leverages the sliced Wasserstein distance to compare distributions by averaging one-dimensional optimal transport distances, efficiently computed via sorting. This approach circumvents the estimation of high-dimensional means and covariance matrices, which underlie FID's poor sample complexity and vulnerability to adversarial attacks. We empirically demonstrate three primary advantages: (i) it is more sample-efficient by one order of magnitude, (ii) it is faster to compute by two orders of magnitude, (iii) it is more robust to adversarial attacks such as moment-matching. We show that MIND with 5k samples can replace the evaluation performance of FID with 50k samples, providing high correlation with this standard benchmark and superior discriminative performance. We further demonstrate that even smaller sample sizes (e.g., 1k or 2k) remain highly informative for rapid model iteration.

• The paper introduces MIND, a metric that uses sliced Wasserstein distances over random projections to overcome FID's limitations.
• It demonstrates that MIND achieves comparable generative model evaluation performance with as few as 5K samples and is significantly faster.
• The method is adaptable across various embeddings and modalities, offering enhanced robustness against adversarial moment-matching attacks.

Authoritative Summary of "MIND: Monge Inception Distance for Generative Models Evaluation"

Motivation and Context

Evaluation of generative models, with focus on high-dimensional data such as images, has for years relied heavily upon Fréchet Inception Distance (FID). While FID provided a pragmatic approach based on Gaussian approximations in Inception embedding space, its statistical and computational weaknesses have become acute in the modern large-scale diffusion paradigm. It is not a proper metric, is susceptible to adversarial attacks via moment-matching, and demands massive sample sizes (commonly 50k) for reliable evaluation, thereby hindering iterative development.

This work addresses these deficiencies with Monge Inception Distance (MIND), leveraging optimal transport metrics for distributional comparison. MIND operates via sliced Wasserstein distances, averaging over many one-dimensional projections, resulting in improved sample efficiency, computational speed, and robustness.

MIND: Formal Definition and Implementation

MIND is defined as a scaled average of 1D Wasserstein distances computed over random unit directions across embedding space. For given generative model output and dataset, both are embedded via a feature extractor (commonly Inception-v3, but agnostic to embedding type), then:

$\MIND(\hat{p}_{n,\theta}, \hat{p}_{n,\text{data}}) = \frac{\alpha}{nM} \sum_{i=1}^M \sum_{j=1}^n |\text{sort}(u_i^X)_j - \text{sort}(u_i^Y)_j|^2$

where $n$ is sample size, $M$ is number of projections, $u_i$ is a random unit direction, and $\alpha=3d$ is a scaling factor proportional to embedding dimension to align MIND values with FID.

MIND is computed efficiently—each 1D Wasserstein calculation reduces to sorting, trivially parallelizable across projections and samples.

Figure 1: General pipeline for evaluating generative model sampling distance to a dataset.

This approach removes the dependence on high-dimensional moment estimation, bypassing sample-heavy, numerically unstable matrix computations. Implementations in both JAX and PyTorch require only projections and sorting, further facilitating integration into standard deep learning workflows.

Figure 2: JAX (a) and PyTorch (b) implementation of MIND.

Comparative Analysis: Sample Efficiency, Discrimination, Robustness

Empirical evidence demonstrates that $\MIND_{5k}$ matches the evaluation performance of $FID_{50k}$, and maintains high information content even at $n=1\text{k}$ or $2\text{k}$—enabling efficient, rapid model iteration. MIND’s discriminative power is confirmed in several controlled testing regimes:

• Generated vs. real ranking: MIND reliably separates generated samples from true dataset samples with as few as 5k samples, while FID requires significantly higher sample numbers for similar error rates.

  Figure 3: MIND metric during diffusion model training on ImageNet-64 (log scale), showing $\MIND_{5k}$ can replace $n$0 with higher sample efficiency.

• Perturbation detection: MIND exhibits superior sensitivity to subtle image perturbations, outperforming FID across tasks that require precise ordering of perturbed sets.

  Figure 4: Sample complexity measured by the probability of detecting a small perturbation.

• Monotonicity: MIND maintains robust ordering of model checkpoints even at low sample sizes.

Computational Speed and Memory

MIND’s reliance on sorting within 1D projections yields exceptional computational advantages. With $n$1, MIND is two orders of magnitude faster to compute compared to FID and requires over ten times less peak memory. The ability to scale efficiently is critical for real-time evaluation during training and parameter search, especially for large-scale generative models.

Figure 5: Walltime and peak memory comparison for MIND, MMD, and FID, highlighting MIND's efficiency.

Peak memory utilization remains nearly constant and minimal for MIND across increasing input dimensions, contrasting sharply with FID and MMD, which require temporary storage of large matrices and kernel computations.

Figure 6: (a) Variance of MIND with increased number of projections $n$2; (b) Peak memory usage for metric computation.

Robustness to Adversarial Metric Hacking

The non-metric nature of FID allows adversarial attacks via moment matching: synthetic samples are constructed to match mean and covariance, artificially collapsing FID while maintaining perceptually poor generative quality. MIND is a proper metric, sensitive to higher-order distributional features unaffected by moment matching. Experimental evidence shows MIND retains significantly more of its discriminative value under such attacks—31% of baseline versus FID’s 11%.

MIND’s resistance stems from its reliance on distributional structure in projected spaces, not merely the moments.

Figure 7: Moment-matching adversarial attack; optimized images are visually unchanged, but only MIND remains robust.

Modality-Independence and Flexibility

Although results are presented with Inception-v3 embeddings to ease comparison with FID, MIND’s metric is fundamentally agnostic to the choice of embedding. CLIP, DINO, or domain-specific architectures (audio, video) are equally applicable. This generality broadens MIND’s utility for cross-modality generative evaluation, including non-image domains.

Theoretical Implications and Future Directions

The adoption of optimal transport-based metrics for generative evaluation brings rigor, computational tractability, and heightened resistance to adversarial artifacts. MIND’s properties are aligned with the needs of modern deep learning: scalable, reliable, and actionable metrics that reflect true distributional similarity. The embedding-agnostic nature further suggests future harmonization of generative evaluation across modalities, architectures, and unsupervised representation spaces.

Future directions include extension of MIND for conditional generators, incorporation into automated model-selection pipelines, and further theoretical analysis of its sample complexity and statistical convergence, especially as network embedding dimensions and generative fidelity increase.

Conclusion

MIND is an efficient, proper, and robust metric for generative model evaluation. It overcomes the primary limitations of FID by offering improved sample efficiency—stabilizing at $n$3 samples—computational speed, memory savings, and heightened resistance to moment-matching attacks. Its modality-independence and ease of implementation make it a practical choice for modern generative modeling, laying ground for rigorous, scalable, and interpretable evaluation protocols. The work positions MIND as a reference standard for measuring distributional fidelity in generative modeling.