Representation Fréchet Loss for Visual Generation

Abstract: We show that Fréchet Distance (FD), long considered impractical as a training objective, can in fact be effectively optimized in the representation space. Our idea is simple: decouple the population size for FD estimation (e.g., 50k) from the batch size for gradient computation (e.g., 1024). We term this approach FD-loss. Optimizing FD-loss reveals several surprising findings. First, post-training a base generator with FD-loss in different representation spaces consistently improves visual quality. Under the Inception feature space, a one-step generator achieves0.72 FID on ImageNet 256x256. Second, the same FD-loss repurposes multi-step generators into strong one-step generators without teacher distillation, adversarial training or per-sample targets. Third, FID can misrank visual quality: modern representations can yield better samples despite worse Inception FID. This motivates FDr$^k$, a multi-representation metric. We hope this work will encourage further exploration of distributional distances in diverse representation spaces as both training objectives and evaluation metrics for generative models.

• The paper introduces a novel technique for directly optimizing Fréchet Distance in frozen representation spaces, leading to improved generative model fidelity.
• The methodology leverages queue- and EMA-based FD estimators to stabilize moment estimation and support multi-representation training without high computational costs.
• Empirical results demonstrate significant enhancements in one-step generation and multi-representation evaluation, outperforming traditional FID-based assessments.

Representation Fréchet Loss for Visual Generation: An Advanced Assessment

Introduction and Motivation

The paper "Representation Fréchet Loss for Visual Generation" (2604.28190) establishes a rigorous methodology for directly optimizing Fréchet Distance (FD) in representation spaces, repositioning FD from an evaluation tool to an effective training objective. Historically, Fréchet Inception Distance (FID) has been the dominant metric in GAN and diffusion model evaluation, but direct optimization was previously deemed impractical due to the requirement for statistically robust sample populations and prohibitive computational cost. The core insight here is the decoupling of FD estimation population size from gradient computation batch size, enabling scalable FD loss optimization in frozen representation feature spaces. This change yields strong empirical improvements in generative model quality and exposes deficiencies in prevailing evaluation metrics.

Methodology

The FD optimization approach relies on two variants:

1. Queue-based FD estimator: Maintains a queue of features from generated images (size $N$ up to $10^5$), updates statistics with each batch, and backpropagates gradients only through current samples. This ensures stable moment estimation without incurring significant computational overhead.
2. EMA-based FD estimator: Tracks exponential moving averages ($\beta$ typically 0.999) for first- and second-order feature moments, minimizing memory requirements and further reducing computational complexity. Gradients are restricted to the current batch, maintaining scalability.

Both variants support multi-representation training objectives by normalizing FD across several frozen backbone models (Inception, ConvNeXt, DINOv2, MAE, SigLIP2, CLIP). Loss terms are scaled to unit magnitude for stable multi-representation combination.

Empirical Characteristics: Population and Representation Effects

Extensive empirical ablations demonstrate:

• Population size is critical: A queue or EMA window exceeding batch size ($N \gg B$) improves stability and consistently yields lower FID and better visual fidelity (see Table 1, Table 2), but excessive staleness degrades results.
• Representation selection is decisive: Post-training models with Inception FD achieve minimal FID (0.81) but are not perceptually optimal; models trained with modern ViT or vision-language representations show improved object structure and lower multi-representation FD ratios, even with higher FID. This highlights the misalignment between Inception-based FID and genuine visual quality.

Figure 1: One-step samples on ImageNet $256\times256$, before and after post-training with representation FD. Post-training upgrades visual sharpness and structure while supporting both pixel-based (pMF-H) and multi-step-to-one-step (JiT-H) generators.

Metric Analysis and the FDr$^{k}$ Paradigm

The paper introduces $\mathrm{FDr}^{k}$, a normalized FD ratio averaged over $k$ representation spaces, mitigating metric overfitting seen in FID-only evaluations. Strong generators surpass real validation images in FID but remain visually inferior under $\mathrm{FDr}^{6}$, indicating substantial quality gaps. The approach demonstrates that even aggressive FD optimization can result in reward hacking and artifacts (see Figure 2), underscoring the need for metric diversity.

Figure 2: Over-optimization of Inception-based metrics (IS/FID) yields models with extreme scores but clear synthetic artifacts and degraded multi-representation FDr, illustrating reward hacking in single-representation evaluation.

Figure 3: Human preference study interface for evaluating visual fidelity. Pairwise comparisons indicate post-trained models are preferred over base models, but remain below real images, validating $\mathrm{FDr}^{k}$ as a superior diagnostic.

Distribution-Level Post-Training and Model Repurposing

Representation FD loss enables practical post-training of existing generators, drastically improving sample quality without real data access or teacher distillation. Furthermore, multi-step generators (e.g., JiT, SD3.5) can be repurposed as strong one-step generators via post-training, achieving competitive or superior FID/$10^5$0 with a single forward pass, no adversarial loss, and negligible computational cost. One-step pMF-H achieves $10^5$1 FID on ImageNet $10^5$2, and repurposed JiT-L reduces FID from $10^5$3 (1-NFE, untrained) to $10^5$4 (post-trained).

Representation-Diverse Evaluation and Practical Implications

The $10^5$5 metric facilitates representation-diverse evaluation, exposing the saturation of FID and revealing remaining gaps in model performance. Post-training with FD-SIM (SigLIP+Inception+MAE) in both pixel and latent space reliably yields improvements across architectures, model sizes, and resolutions. This paradigm shift suggests that future generative model design and evaluation must pivot away from monolithic metrics and integrate representation diversity for distributional closeness.

Figure 4: Full text-to-image qualitative comparison with post-trained SD3.5, illustrating preservation of content and transfer of reference distribution aesthetic despite drastic reduction in network evaluations.

Figure 5: Extended qualitative samples showcasing the aesthetic and semantic adaption of the post-trained text-conditioned models to reference datasets.

Theoretical and Practical Implications

The findings motivate further exploration of distributional losses in frozen representation spaces for generative modeling, offering a unified framework for post-training, evaluation, and even cross-modal transfer. The research questions the utility of FID as a single "north star" and encourages multi-representation or adaptive metric formulation. For practitioners, the method enables model repurposing and substantial inference speed-ups (one-step over many-step) with minimal computational overhead.

Conclusion

The paper establishes that distributional post-training using decoupled FD loss in representation spaces is both practical and powerful for enhancing generative model fidelity. It exposes metric overfitting phenomena and demonstrates the necessity for multi-representation evaluation. Empirical results delineate strong improvements in FID/$10^5$6 and human preference, supporting broad applicability across model scales, architectures, and modalities. Future research should focus on adaptive, representation-aware objectives and robust, multi-faceted evaluation metrics for generative models.