Fréchet Inception Distance Analysis

• Fréchet Inception Distance (FID) is a metric that evaluates generative models by comparing features' means and covariances of real and generated images.
• Finite-sample bias in FID scales as O(1/N), and a linear regression on 1/N allows extrapolation to obtain bias-free estimates for reliable model comparisons.
• Applying Quasi-Monte Carlo sampling reduces estimator variance, leading to improved metric stability and more effective training dynamics in GANs.

The Fréchet Inception Distance (FID) is a dominant metric for evaluating generative models, especially within image synthesis, due to its simplicity, closed-form computability, and empirical alignment (in aggregate) with perceptual similarity. However, careful theoretical analysis and empirical studies reveal that FID, as conventionally computed, is subject to systematic sample-size-dependent bias—introducing serious limitations for comparative evaluation, benchmarking, and model optimization. The bias, its mathematical structure, and practical correction methods are the focus of the foundational analysis in "Effectively Unbiased FID and Inception Score and where to find them" (Chong et al., 2019).

1. Finite-Sample Bias in FID and IS

Both FID and Inception Score (IS) are computed using Monte Carlo (MC) estimates over a finite sample set of $N$ generated images, rather than the population-level statistics of an ideal infinite sample. For FID, the empirical mean $\mathbf{M}_g$ and covariance $\mathbf{C}_g$ for generated images (analogously, $\mathbf{M}_t$, $\mathbf{C}_t$ for the true distribution) are substituted for the true feature distribution moments. The standard FID expression is: $$\mathrm{FID} = \|\mathbf{M}_t - \mathbf{M}_g\|^2 + \operatorname{Tr}(\mathbf{C}_t + \mathbf{C}_g - 2(\mathbf{C}_t\mathbf{C}_g)^{1/2}).$$

However, $\mathbf{M}_g$ and $\mathbf{C}_g$ are random MC estimates with nonzero variance. When plugging these noisy estimates into FID (a nonlinear function), a Taylor-series expansion reveals a bias of order $O(1/N)$, governed by the Hessian of the FID function and the variance structure of the estimator: $$\operatorname{Bias} \approx \frac{G''(I)}{2} \operatorname{Var}(\xi) \sim \frac{K}{N},$$ where $K$ depends on both the functional form and the generator specifics. For IS, the empirical version estimates the entropy of the predicted label distribution, yielding a negative bias due to the convexity of the entropy function—also scaling as $O(1/N)$. Critically, the $O(1/N)$ bias term in both metrics is generator-dependent: models with different output distributions can yield different bias magnitudes, so direct ranking of models by FID/IS at a fixed $N$ is not reliable.

2. Extrapolation to Obtain Bias-Free Estimates

Because the leading bias in FID and IS is approximately linear in $1/N$, the paper proposes a model for the observed score as

$$\mathrm{FID}_N = \mathrm{FID}_\infty + \frac{K}{N} + O(1/N^2),$$

where $\mathrm{FID}_\infty$ is the asymptotic, bias-free metric for infinite $N$.

Implementation:

• Compute $\mathrm{FID}_N$ at various sample sizes $N$.
• Fit a linear regression: plot $\mathrm{FID}_N$ vs. $1/N$.
• The fitted y-intercept yields an extrapolated estimate, denoted $\overline{\mathrm{FID}_\infty}$, which is not confounded by finite-sample bias.

A similar extrapolation yields $\overline{\mathrm{IS}_\infty}$ for IS. The paper provides both pseudocode and code implementing this workflow.

Significance: This correction makes FID/IS values model-comparable regardless of $N$ and eliminates model-dependent finite-sample ranking artifacts.

3. Quasi-Monte Carlo Integration for Variance Reduction

Reliable extrapolation requires low-noise FID/IS estimates at each $N$. The paper advocates replacing standard MC sampling (iid uniform or Gaussian draws) with Quasi-Monte Carlo (QMC) integration using low-discrepancy sequences (e.g., Sobol sequences), which uniformly cover the domain and reduce estimator variance.

For an integral $I(f) = \int_{[0,1]^d} f(u) du$, QMC yields: $|\hat{I} - I| \leq V(f) \cdot D_n^*,$ where $D_n^*$ is the star discrepancy of the sequence, and $V(f)$ is the function variation. Sobol QMC sequences achieve near $O(1/N)$ convergence, as opposed to the $O(1/\sqrt{N})$ rate of IID MC methods. The paper compares two Gaussianization methods (Box–Muller and inverse CDF), finding that Sobol_Inv yields slightly better performance.

Practical Outcome: Reduced estimator variance improves the accuracy and stability of the $1/N$ regression, enabling better bias-free FID/IS computation.

4. Practical Implications for Evaluation

Employing $\overline{\mathrm{FID}_\infty}$ and $\overline{\mathrm{IS}_\infty}$ directly addresses the bias and instability present in the classical finite-sample metrics. It ensures that:

• Comparison between models is fair and independent of specific $N$.
• Estimates are more stable and reproducible.
• Model rankings reflect actual generative performance rather than artifacts of estimator bias or sample size.

Small differences in (biased) FID/IS can lead to different rankings; correcting for bias is thus essential for robust scientific comparison.

5. QMC Sampling in GAN Training Dynamics

Beyond evaluation, QMC sampling (Sobol sequence-based latent draws) during GAN training yields:

• Small but consistent improvements in $\overline{\mathrm{FID}_\infty}$ for trained models.
• Lower variance across training runs.
• Smoother latent space coverage, improving gradient estimation for loss minimization.

Mechanism: GAN training minimizes expectations with respect to latent variables; QMC sampling reduces the stochastic noise in these gradient estimates, leading to subtly more stable training.

6. Implementation Table

Task	Conventional FID/IS	Bias-Free Extrapolation	QMC Integration
Estimator Bias (fixed $N$)	$O(1/N)$, model-specific	Eliminated ($\overline{\mathrm{FID}_\infty}$)	Reduced variance, supports extrapolation
Model Comparability	No	Yes	Yes
Reproducibility	Weak	Strong	Strong
Computational Complexity	Standard	Slight overhead (multiple $N$, regression)	Negligible (Sobol seq. sampling)

7. Limitations and Extensions

• The reliability of the extrapolation depends on accurate regression. Severe undersampling, misspecification, or nonlinearity in $1/N$ can affect estimates.
• QMC effectiveness depends on the smoothness of $f(u)$; pathologically non-smooth generators may see lesser variance reduction.
• The bias term's magnitude is generator-dependent and can vary by orders of magnitude—highlighting the importance of bias removal.
• Extended to other MC-based model evaluation beyond GANs, and may inform hyperparameter search and model selection regimes.

---

In conclusion, finite-sample FID and IS are inherently biased in a model-dependent way. Extrapolation in $1/N$ and QMC sampling are drop-in, rigorously justified techniques that effectively yield unbiased, reproducible, and more meaningful scores for generative model evaluation and training (Chong et al., 2019).