The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

Abstract: The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.

• The paper demonstrates that training variability in FID evaluations is approximately 3.2 times larger than sampling-induced noise.
• It decomposes the FID variance into contributions from random initialization, data ordering, and per-step training noise.
• The analysis reveals that increasing model scale does not reduce variance, challenging single-seed benchmark claims and reproducibility.

Quantifying Randomness in Generative Model Evaluation: The FID Lottery

Introduction

The evaluation of generative image models has been dominated by the Fréchet Inception Distance (FID), a metric that underpins comparative claims across both architectures and datasets. This metric is routinely reported as a single value from a single training run and a single sampling seed, with subsequent papers using marginal FID improvements to justify substantial compute or methodological changes. However, this paradigm ignores the multiplicity of stochastic factors influencing the final FID value—stemming both from training and sampling processes. This paper presents a systematic measurement and analysis of the distributional properties of FID across these axes of randomness, terming the phenomenon the “FID lottery” (2606.20536).

Experimental Protocol

The study primarily focuses on Scalable Interpolant Transformers (SiT) trained via conditional flow matching on class-conditional ImageNet $256 \times 256$, spanning architectures from S/2 through XL/2. The experimental setup exhaustively varies both training seeds ($N$) and sampling seeds ($K$), forming an $N \times K$ evaluation panel where each cell records an FID score for a model trained and sampled with the respective seeds. The panel enables decomposition of FID variance across model retrains and sample draws. Three primary statistics are computed: (1) $\sigma_{\text{between}}$ quantifies FID variation across training seeds; (2) $\sigma_{\text{within}}$ measures FID variation across sampling seeds for a fixed trained model; (3) the coefficient of variation (CoV) contextualizes variance relative to the mean FID.

Main Findings

Training Variability Dominates Sampling Jitter

Empirical results on hundreds of SiT networks reveal that the variance in FID due to retraining with different training seeds ($\sigma_{\text{between}}$) is approximately $3.2\times$ larger than the variation induced by resampling on a trained network ($\sigma_{\text{within}}$). The coefficient of variation for FID across training seeds is consistently about $1$–$N$0, setting a non-negligible noise floor on the metric. In contrast, standard reporting protocols that average over sampling seeds on a fixed model severely underestimate the actual epistemic uncertainty of the reported FID.

Sources of Training-Time Randomness

The analysis decomposes training-induced variance into three independent sources: random parameter initialization, random data-loader ordering, and per-step Gaussian noise induced by the flow-matching loss. The per-step training noise is quantitatively the largest contributor, but initialization and data ordering also impose substantial, non-additive effects. In contrast, non-determinism from multi-GPU numerical imprecision is demonstrated to be negligible in its impact on FID relative to algorithmic sources.

Invariance Under Model Scale and Compute

Contrary to intuitive expectation, increasing model width or compute does not systematically suppress variance in FID. At all scales and levels of convergence, the CoV remains in the $N$1–$N$2 range. This demonstrates that the FID lottery constitutes a robust statistical floor, insensitive to the typical axes of resource scaling followed in recent generative modeling.

Guidance Optimization Compresses Variance but Alters Rankings

Implementing per-(training, sampling) seed golden-section search for classifier-free guidance (CFG) scale, termed GS-FID, achieves a halving of the relative FID variance ($N$3). However, the optimal guidance setting introduces notable rank reshuffling among training seeds (Spearman $N$4 compared to unguided ranking), undermining the stability of “best-seed” selection. Consequently, “best FID” is highly sensitive to both hyperparameter selection and random seed, and not a stable property of the training recipe.

Seed Optimality and Training Budget: The Lucky Draw Effect

A salient consequence is that a “lucky” training seed, by mere stochasticity, can attain a target FID with up to $N$5 less training compute than an “unlucky” one, even with matched configurations and hyperparameters. The variance in convergence speed and asymptotic performance underlines the futility of one-off, single-seed benchmark claims, particularly in over-claiming training speedup or sample efficiency.

Learning Rate Selection and Variance

Through a $N$6P-coordinated learning rate sweep, the study confirms that optimal FID is achieved over a window of learning rates (width $N$7), not a unique value, and that the variance floor does not dip at the optimization optimum. Consequently, recipe comparison studies that report per-recipe best results across a narrow LR grid are inherently out-resolved by the FID lottery’s magnitude.

Theoretical and Practical Implications

Reproducibility and Statistical Reporting

This work implicates most current generative model papers in severe underreporting of epistemic uncertainty, effectively propagating irreproducible rankings and illusory improvements. The $N$8 noise floor empirically observed for FID on ImageNet $N$9 with SiT-family models sets a rigorous calibration boundary: claims of FID improvements below $K$0 this CoV (i.e., $K$1–$K$2 relative gap) are statistically inconclusive unless supported by multi-seed statistics.

Metric-Intrinsic Variance

The intrinsic randomness arises most acutely in metrics defined over dataset-level statistics, such as FID. This variance is intrinsic to the metric’s sensitivity to moment differences in distributions, not just to the weakness of the models or training protocols. The results extend, with minor differences in magnitude, to modern feature spaces such as DINOv2, and persist even with guidance or post hoc parameter selection.

Benchmark and Community Protocols

Given these observations, the paper explicitly recommends that FID reporting in generative model studies include error bars across multiple independently trained models (training seeds), treat FID gains within the empirically established CoV as inconclusive, and, when feasible, report under “per-cell optimal” guidance tuning. Relying on single-number leaderboard positions or single-seed evaluations in this context constitutes unsupported inference.

Directions for Future Research

While the present analysis focuses on a specific model class (SiT), loss (flow-matching), and dataset/metric combination, it establishes methodology and calibration targets that should be applied more broadly. Future work should include cross-architecture and loss function variance auditing, extension to text-to-image and other generative modalities, and systematic analysis under different evaluation metrics (including, e.g., KID, PRDC variants, and feature-based alternatives). Additional interest lies in developing cheap proxies or predictors of the seed-induced variance floor, making robust evaluation feasible for compute-constrained settings.

Conclusion

The FID lottery imposes a persistent, irreducible noise floor beneath which single-number FID comparisons are not robust. The magnitude and invariance of this floor with scale and compute, as well as its dominant training-time sources, necessitate a shift in the community’s empirical evaluation standards for generative models. Adopting the measurement and reporting practices advocated in this work will improve reproducibility, rigor, and practical relevance of generative model research.