DiffusionBench: On Holistic Evaluation of Diffusion Transformers
Abstract: Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: methods that improve DiffusionBench are more likely to reflect broader progress.
Summary
- The paper introduces NanoGen, a unified framework for training and evaluating diffusion transformers across class-conditional and text-to-image tasks.
- The paper demonstrates that improvements in ImageNet FID do not reliably predict T2I performance, with Pearson correlations ranging from -0.377 to -0.580.
- The paper emphasizes the need for comprehensive, multi-axis benchmarking to drive progress in general-purpose generative modeling.
Holistic Evaluation of Diffusion Transformers: The DiffusionBench Framework
Motivation and Problem Statement
The proliferation of Diffusion Transformers (DiTs) in image generation research has led to rapid architectural and objective innovation. Historically, evaluation protocols have converged on class-conditional ImageNet generation, with FID as the primary metric. This paper identifies a central issue: improvements in ImageNet FID may not translate to broader generative modeling advances, particularly for text-to-image (T2I) tasks. It challenges the assumption that evaluating on ImageNet alone suffices for comprehensive DiT assessment and argues that T2I benchmarking is neither prohibitively costly nor technically complex under modern frameworks.
NanoGen: Unified Training and Evaluation Framework
NanoGen is introduced as a unified training/evaluation framework for DiTs, supporting both class-conditional and T2I generation with a common codebase, backbone, training loop, optimizer, and evaluation suite. Task adaptation involves minimal engineering, specifically swapping the conditioning module and changing the dataset loader. NanoGen is compatible with a wide spectrum of diffusion model variants, including RAE, VAE, pixel-space, and MeanFlow, and accommodates state-of-the-art vision encoders (e.g., DINOv2-B, SigLIP2-B, Perception Encoder) as latent tokenizers.
Backbone modifications include the DDT split (encoder–decoder separation for computational efficiency), removal of AdaLN from the encoder, and in-context conditioning (all conditioning as input tokens). This ensures minimal architectural drift between tasks. Training protocols are standardized across tasks with AdamW, fixed LR schedules, EMA, and -prediction. Evaluation is performed online via HuggingFace-based harnesses, with all relevant metrics (FID, IS, FDr, MIND for ImageNet; GenEval, DPG-Bench, GenAIBench for T2I) accessible in real time.
Empirical Analysis: ImageNet vs. T2I Correlation
The authors conduct a large-scale empirical study, training and evaluating 21 latent diffusion models under both ImageNet and T2I setups using NanoGen. They observe that method ranking on ImageNet is not strongly correlated with performance on T2I tasks: Pearson correlations are negative and range between and depending on the metric. The lack of correlation holds across both VAE and RAE latent spaces, irrespective of classifier-free guidance (CFG) application.
Figure 1: Pearson correlations between class-conditional ImageNet FID and T2I metrics (GenEval, DPG-Bench, GenAIBench) indicate that FID does not reliably predict T2I quality.
This contradicts prevailing assumptions that ImageNet improvements imply downstream task benefits. Even apparently strong methods on ImageNet, such as FLUX.2-VAE and spatial RAE variants, may display mediocre T2I scores.
Benchmarking and Methodological Results
The system systematically benchmarks DiT variants on both axes. On ImageNet, best FID is achieved by FLUX.2-VAE ($1.37$) and REPA-E VAEs ($1.5$–$1.6$ range), while RAEs trail slightly. Pixel-space models exhibit higher FID and slower convergence. MeanFlow models, offering one/few-step generation, have significantly worse FID than iterative baselines.
On T2I tasks, method ordering diverges from ImageNet: some latent-space methods with top ImageNet scores perform suboptimally, while other end-to-end VAE variants excel in specific metrics. Metrics themselves display partial disagreement, deepening the need for multi-axis reporting.
Wall-clock compute analysis using NanoGen shows that T2I training cost is comparable to ImageNet for latent-space methods, invalidating the notion that T2I evaluation is uniquely expensive.
Figure 2: Wall-clock training time for steps reveals similar computational budget for ImageNet and T2I training across DiT variants.
Qualitative evaluation of T2I samples at resolution further corroborates the quantitative findings, demonstrating variance in sample quality not aligned with ImageNet rankings.
Figure 3: Curated qualitative T2I samples from NanoGen-trained models show significant diversity in visual fidelity and prompt adherence.
Practical and Theoretical Implications
The primary practical recommendation is the adoption of DiffusionBench: a holistic benchmark aggregating ImageNet and T2I metrics. Method improvements should be reported across both axes to ensure generalizability; gains exclusive to one task should be classified as task-specific, not general modeling advances.
Theoretical implications include the need for broader cross-task evaluation to avoid benchmark overfitting and misallocation of research effort. The analysis demonstrates metric hacking vulnerability, especially in T2I arenas, highlighting the importance of metric diversity and resistance mechanisms.
Future directions include extending DiffusionBench to world modeling, video, and 3D generation, refining T2I metrics to prevent reward hacking, and developing community-maintained leaderboards.
Conclusion
DiffusionBench establishes that progress in DiT models cannot be reliably inferred from single-axis benchmarks such as ImageNet FID. NanoGen demonstrates that unified training and evaluation across ImageNet and T2I tasks is both engineering- and compute-efficient, removing barriers for comprehensive assessment. The adoption of holistic benchmarks is essential for advancing general-purpose generative modeling, and future research should address metric robustness and multimodal task expansion (2606.24888).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper is about making sure we fairly judge how well new image‑making AI models really work. The authors focus on “Diffusion Transformers” (DiTs), which are the kind of models behind many modern image generators. Right now, most papers test these models on just one task: making images of ImageNet categories (like “banana” or “fire truck”) and then scoring them with a number called FID. The authors show that doing well on this single test doesn’t always mean the model will be good at the task many people actually care about: turning text prompts into pictures (text‑to‑image, or T2I). They release a tool called NanoGen that lets researchers easily train and test DiTs on both tasks, and they propose a new, broader benchmark called DiffusionBench that combines results from ImageNet and T2I.
Key Questions
- If a method improves a DiT’s score on the ImageNet test (especially FID), does it also make text‑to‑image generation better?
- Can we make it easy for researchers to train and evaluate the same model on both ImageNet and text‑to‑image, without building two separate systems?
- What would a fair, “holistic” benchmark look like for judging real progress in DiT research?
How They Did It (with simple explanations)
Think of making an image like unblurring a photo: diffusion models learn to start from random noise and carefully remove the “noise” step by step until a clear picture appears. A transformer is a type of AI that’s good at understanding sequences (like words or image patches). A Diffusion Transformer (DiT) combines these ideas to create images.
To compare methods fairly, the authors built NanoGen, a unified training system:
- One backbone (the main model), one training loop, one optimizer, one evaluation setup. Switching between tasks is like changing the inputs:
- For ImageNet: feed in class labels (e.g., “cat”).
- For text‑to‑image: feed in the text prompt using a text encoder.
- Minimal changes: about 12 lines of configuration to switch from ImageNet to T2I.
They trained many model variants using different “tokenizers”:
- Latent‑space models: instead of working on full‑resolution pixels, they work on compressed representations (like a smaller blueprint of the image). VAEs and RAEs are different ways to compress images:
- VAE: a learned compressor.
- RAE: uses a strong vision encoder as the compressor.
- Pixel‑space models: work directly on the full image pixels (more expensive).
- They also tried fast “few‑step” methods (MeanFlow) that try to make images in one or two steps, trading some quality for speed.
For judging quality:
- ImageNet metrics: FID, IS, and others.
- Text‑to‑image metrics: GenEval, DPG‑Bench, and GenAIBench. These check how well the picture matches the prompt and how good it looks.
Analogy:
- ImageNet testing is like making pictures that match a specific category list and having a single judge (FID) score them.
- T2I is like drawing from a written description; you need both understanding and creativity. Multiple judges (different metrics) check if the drawing matches the description and looks nice.
Main Findings and Why They Matter
- Doing well on ImageNet doesn’t guarantee doing well on text‑to‑image. Across 21 models, the authors found weak or even negative correlation between ImageNet FID and T2I scores. In simple terms: getting a great FID on ImageNet does not reliably predict good T2I results.
- Some model families consistently do better overall:
- Improved latent‑space approaches (like RAE and end‑to‑end tuned VAEs such as FLUX.2‑VAE and REPA‑E) tend to outperform older VAEs and pixel‑space methods on both tasks.
- Pixel‑space models and “one/few‑step” methods are generally worse on quality at the training budgets tested.
- Training text‑to‑image isn’t dramatically more expensive than training on ImageNet with NanoGen. This means there’s no longer a good excuse to skip T2I evaluation.
- Different T2I metrics don’t always agree perfectly. This suggests we should not rely on a single score when judging text‑to‑image quality.
- Sometimes, images look better after more training, but the scores barely move. This hints that current metrics can miss improvements and may be easy to “hack.”
What This Means
- Reporting only ImageNet results can be misleading. A model might look great on that one test but not be helpful for real text‑to‑image use.
- The authors propose DiffusionBench, which combines both ImageNet and T2I results into a single benchmark. If a method improves DiffusionBench, it’s more likely to reflect genuine, broad progress.
- NanoGen makes it simple to test on both tasks. This lowers the barrier for future research to be more complete and honest about model strengths and weaknesses.
Takeaway and Potential Impact
This paper encourages the community to move from “single test, single number” to “multiple tasks, multiple metrics” when judging image generators. With NanoGen and DiffusionBench, researchers can:
- Quickly check whether a new idea helps both ImageNet and text‑to‑image.
- Avoid over‑fitting to one benchmark.
- Work toward models that are truly better for real users, not just better at one narrow score.
In short: Better evaluation leads to better models. The tools released here could help the field focus on real improvements that matter outside of the lab.
Knowledge Gaps
Unresolved Knowledge Gaps, Limitations, and Open Questions
Below is a consolidated list of concrete gaps and open questions that remain unresolved in the paper, grouped by theme to aid follow-up research.
Benchmark scope and metrics
- Lack of higher-resolution evaluation: all results are at 256×256; it is unclear whether ImageNet–T2I correlation patterns hold at 512–1024 resolutions commonly used in practice.
- Limited task coverage: “holistic” evaluation includes only ImageNet class-conditional and T2I; no assessment on editing, inpainting, video, or multi-view/3D generation where DiTs are increasingly applied.
- No human preference evaluation: T2I comparisons rely on GenEval, DPG-Bench, and GenAIBench; no calibration against human judgments (e.g., arena-style ELO) to validate metric relevance.
- Metric hackability not quantified: the paper notes that GenEval can be “hacked” via fine-tuning but provides no systematic experiments measuring hackability, nor mitigations.
- Aggregation definition for DiffusionBench not specified: the benchmark is proposed but lacks a clear, reproducible aggregation scheme (e.g., metric weighting, variance-aware aggregation) and a protocol for updates/refresh.
- Statistical rigor of correlation analysis: Pearson correlations are reported but no confidence intervals, power analysis, or alternative rank-based measures (Spearman/Kendall); causality/confounders are not analyzed.
- T2I metrics may miss perceptual improvements: authors observe better visuals at 200K steps without score gains; open question on which metrics better capture aesthetics, realism, and overall quality.
Methodology and analysis
- Validity under different samplers/NFEs: conclusions are drawn using Euler (50 NFEs); it is unknown whether ImageNet–T2I correlation patterns persist across samplers (DDIM, Heun, DPM-Solver, RF/CFM) and NFE budgets.
- Sensitivity to guidance schedules: ImageNet uses per-method CFG sweeps; T2I fixes a guidance scale (6.0) and constant schedule—leaving open whether per-method or per-step schedules alter rankings and correlations.
- Objective choice effects: most models use v-prediction, with some baselines in x-prediction; no ablation on whether target choice (x/ε/v) changes cross-task conclusions.
- Limited training budgets: ImageNet trained for 80 epochs and T2I for 100K–200K steps; scaling laws and whether rankings stabilize or invert with longer runs or larger models remain untested.
- Confounders in correlation: differences in latent tokenizers, text encoders, and conditioning length could confound ImageNet–T2I correlation analyses; no controlled experiments isolate these factors.
Architecture and training choices
- Generality beyond DDT backbone with encoder AdaLN removal: findings may be backbone-specific; it is unknown whether MMDiT/U-ViT/SiT or U-Net backbones yield similar cross-task patterns.
- In-context conditioning vs. modulation: the paper adopts all-token in-context conditioning; no comparison to modulation-based designs (AdaLN/FiLM) on either performance or correlation outcomes.
- RAE + CFG behavior is unclear: authors note it is “unclear how RAE benefits further from CFG”; no targeted experiments clarify why or propose remedies.
- FLUX.2-VAE mechanisms are opaque: the paper speculates BN-related mechanisms but lacks analyses isolating architectural or training contributors to its strong ImageNet FID.
- Few-step models underrepresented: MeanFlow is slower and underperforms; no comparison with other 1–4 step approaches (consistency models, rectified flows, distilled samplers) to fairly assess the few-step regime.
Data and conditioning
- Single frozen text encoder: all T2I runs use Qwen3-0.6B; it is unknown how text-encoder choice (e.g., CLIP, SigLIP, T5, larger LLMs) or freezing vs. fine-tuning impacts cross-task rankings.
- Prompt/domain generalization: T2I datasets (JourneyDB + BLIP-3o splits) may bias capabilities; no evaluation on out-of-domain prompts, long-form instructions, multilingual prompts, or safety-sensitive inputs.
- Token budget and conditioning length: T2I uses 256 text tokens; there is no ablation on token length, truncation effects, or positional strategies for long prompts.
- Dataset composition effects: no ablation on the contribution of each pretraining source (JourneyDB, BLIP-3o splits), data quality/cleaning, or caption noise on final rankings.
Compute, efficiency, and fairness of comparison
- Compute-normalized fairness: although wall-clock times are reported, the comparisons do not normalize for per-method FLOPs, memory bandwidth, or NFE × cost trade-offs across latent sizes and token lengths.
- Hardware specificity: results are on 32×H200 GPUs; it is unclear how reproducible and cost-effective the framework is on more common hardware (A100, consumer GPUs) or under constrained memory.
- Latent dimensionality and time shifting: dimension-dependent timestep shifting is used; no ablation shows whether this favors certain latent tokenizers or affects cross-task comparability.
Reproducibility and baseline alignment
- Partial divergence from original baselines: unified backbone and training choices (DDT, AdaLN removal in encoder) may deviate from original methods; open question whether conclusions hold when reproducing each baseline exactly.
- Preprocessing sensitivity: FID is known to be sensitive to resizing/cropping; the paper uses center-crop to 256 but does not test alternative pipelines or report sensitivity analyses.
Safety, ethical, and robustness considerations
- Bias and fairness not assessed: no evaluation on demographic, geographic, or content biases introduced by datasets or tokenizers.
- Safety and misuse: no discussion of content moderation, safety filtering, or risk evaluation for generated outputs within DiffusionBench.
- Watermarking and provenance: the framework and benchmark do not consider watermarking or attribution, which may be critical for responsible deployment.
Benchmark design and release details
- DiffusionBench governance: no plan for periodic refresh, test set curation, or adversarial example integration to prevent overfitting and metric gaming over time.
- Result aggregation and reporting protocol: no standardized reporting template (e.g., fixed NFEs, guidance schedules, negative prompts, decoding options) to ensure apples-to-apples comparisons across papers.
- Lack of ablations guiding adoption: the paper recommends DiffusionBench but does not provide sensitivity/robustness studies to help users decide default settings (e.g., guidance scales, sampler, NFE).
These gaps suggest concrete follow-ups: add higher-resolution and additional tasks to DiffusionBench; incorporate human-preference evaluations; define and validate a robust aggregation scheme; perform controlled ablations on samplers/guidance/objectives/encoders/token lengths; test broader datasets and prompt regimes; analyze compute-normalized fairness; and extend safety and bias assessments.
Practical Applications
Immediate Applications
The paper introduces NanoGen (a unified DiT training/evaluation framework) and DiffusionBench (a holistic benchmark combining ImageNet and text-to-image), showing that ImageNet FID does not reliably predict T2I quality. The following applications can be deployed now:
- Holistic model gating and QA in production pipelines (Software, Creative/Marketing)
- Replace single-metric ImageNet FID gates with DiffusionBench scorecards that include both ImageNet and T2I metrics (e.g., FID, MIND/FDr, GenEval, DPG-Bench, GenAIBench).
- Integrate the provided evaluation harness into CI/CD to block regressions that only show up on T2I.
- Assumptions/dependencies: access to GPUs for periodic evaluations; agreement on target metric thresholds; careful selection of CFG scales; awareness that T2I metrics can be gamed and should be triangulated.
- Rapid R&D iteration on DiTs with minimal engineering overhead (Software/Research)
- Use NanoGen’s single backbone and “12-line config switch” to prototype ideas across ImageNet and T2I without maintaining separate codebases.
- Run ablations that are truly cross-task, preventing overfitting to ImageNet FID alone.
- Assumptions/dependencies: Python/HuggingFace environment; compatible tokenizers (RAE/VAEs) and licenses; curated datasets for T2I (e.g., JourneyDB, BLIP-3o splits).
- Reproducible baselines for academic papers and coursework (Academia, Education)
- Adopt NanoGen’s implementations (RAE, VAE, pixel-space, MeanFlow) as strong, consistent baselines for new DiT methods and student labs.
- Provide students hands-on assignments switching ImageNet↔T2I with one config change, plus live metric dashboards.
- Assumptions/dependencies: modest GPU clusters (paper shows 100K-step T2I runs in ~10 hours on 32×H200; scaled-down runs possible on 8 GPUs with longer wall-clock); dataset usage complies with licensing.
- Vendor/model procurement due diligence and benchmarking (Policy/Industry/Enterprise)
- Evaluate candidate generative models on DiffusionBench rather than accepting ImageNet-only claims; require multi-axis reports in RFPs.
- Create internal acceptance thresholds (e.g., minimum GenEval and DPG-Bench scores alongside FID/MIND) for deployment.
- Assumptions/dependencies: standardized prompts and seeds; comparable NFEs, CFG ranges, and resolutions; documented compute budgets.
- Domain adaptation for productized T2I features (E-commerce, Design, Media)
- Fine-tune NanoGen T2I checkpoints on domain data (catalogs, style guides) to improve prompt adherence and brand consistency.
- Use end-to-end VAEs (REPA-E/FLUX.2-VAE) or RAE latents for faster convergence at 256×256; iterate on conditioning modules or prompt templates.
- Assumptions/dependencies: rights-cleared training data; content safety filters; extension to higher resolutions may require additional engineering and compute.
- Multi-metric monitoring to detect “metric hacking” (Software, Governance)
- Track disagreement across GenEval/DPG-Bench/GenAIBench to flag suspicious improvements and schedule human preference checks.
- Assumptions/dependencies: human-in-the-loop review capacity; robust sampling protocols; clear rollback policies.
- Compute planning and budgeting (Operations)
- Treat T2I pretraining as comparable to ImageNet (per paper’s timing study) for scheduling GPUs and budgeting experiments.
- Assumptions/dependencies: hardware availability; cost controls; reproducible recipes (optimizer, LR schedule, EMA, NFE).
- Open-source contributions and community extensions (Software/Research)
- Plug in new tokenizers, samplers, or conditioners to NanoGen and instantly get cross-task evaluations and leaderboards.
- Assumptions/dependencies: maintenance of a shared config convention; CI that verifies both ImageNet and T2I results.
Long-Term Applications
The findings and framework unlock strategic directions that benefit from further research, scaling, or development:
- Field-wide standardization of multi-axis reporting (Policy, Academia, Standards)
- Journals, conferences, and standards bodies require DiffusionBench-like reporting (ImageNet + T2I) for DiT papers and product claims.
- Assumptions/dependencies: community consensus; ongoing benchmark refresh to avoid saturation; transparent compute disclosures.
- Robust, hack-resistant T2I evaluation and governance (Policy, Research)
- Develop and adopt new T2I metrics that correlate better with human preferences, resist reward hacking, and cover safety/fairness.
- Incorporate human-arena ELOs and auditing practices into regular releases.
- Assumptions/dependencies: curated, regularly refreshed prompt suites; budget for human evaluations; openness to adversarial testing.
- Extension of NanoGen beyond images (Video, 3D, World Models, Robotics)
- Generalize the unified backbone and in-context conditioning to video generation, 3D assets, and world modeling; build multi-task DiffusionBench variants.
- Assumptions/dependencies: new datasets and conditioners (temporal/spatial tokens), scalable tokenizers, larger compute footprints.
- Energy- and cost-aware benchmarking (Energy, Sustainability, Policy)
- Standardize reporting of wall-clock, energy use, and NFE efficiency across methods, enabling “green AI” comparisons.
- Assumptions/dependencies: instrumentation for energy monitoring; comparable hardware and batch sizes; community norms for energy disclosures.
- Commercial “Benchmark-as-a-Service” platforms (Software, Enterprise)
- Offer managed DiffusionBench runs with reproducible environments, dashboards, and model cards to support procurement, compliance, and release gates.
- Assumptions/dependencies: secure model upload; dataset licensing; standardized reporting schemas.
- Sector-specific T2I systems validated by multi-axis tests (E-commerce, Fashion, Architecture, Education)
- Build production T2I pipelines that pass domain-aligned DiffusionBench gates and add vertical tests (e.g., product attribute fidelity).
- Assumptions/dependencies: high-resolution scaling (512–1024+), dataset curation, watermarking and content moderation, localization support.
- Safety, fairness, and watermarking integrated into evaluation (Policy, Safety)
- Extend DiffusionBench to include harmful content filters, demographic fairness checks, IP policy compliance, and watermark robustness.
- Assumptions/dependencies: high-quality safety classifiers/filters; diverse evaluation sets; agreed-upon policy thresholds.
- Few-step, real-time generation research (Software, Edge/On-device, Robotics)
- Advance one-/few-step methods (e.g., MeanFlow-like) to approach multi-step quality, enabling interactive or on-device T2I and simulation assets.
- Assumptions/dependencies: algorithmic advances in stability and fidelity; distillation; hardware acceleration.
- Synthetic data pipelines with multi-axis validation (Healthcare, Manufacturing, Finance)
- Use DiffusionBench-style checks to validate generative synthetic datasets before training discriminative models (e.g., defect detection, document layouts).
- Assumptions/dependencies: strict privacy and compliance constraints; domain shift assessments; clear disclaimers for sensitive sectors (e.g., healthcare).
- AutoML and multi-objective model selection (Software/Research)
- Optimize architectures and hyperparameters jointly for ImageNet and T2I objectives, automating trade-offs under compute/latency constraints.
- Assumptions/dependencies: reliable multi-metric optimization; consistent sampling protocols; robust early-stopping signals.
- Regulatory audits and certifications for generative systems (Policy/Enterprise)
- Third-party auditors use DiffusionBench to certify claims and monitor drift post-deployment.
- Assumptions/dependencies: trusted auditors; standardized evidence packages; secure logging and reproducible seeds.
Notes on general feasibility
- Resolution and scale: The paper’s results are primarily at 256×256; higher-resolution deployment will require additional engineering, compute, and possibly different tokenizers/architectures.
- Hardware: The reported T2I training is feasible on 8–32 high-end GPUs; smaller organizations can still benefit with longer training times or by fine-tuning released checkpoints.
- Metrics: T2I metrics can be gamed; combine multiple automated metrics with periodic human evaluation and qualitative review.
- Data governance: Ensure licensing, safety filters, and bias assessments for training/evaluation datasets used in both ImageNet and T2I setups.
Glossary
- AdamW: An optimizer that decouples weight decay from gradient-based parameter updates to improve generalization. "We use the AdamW~\citep{adamw} optimiser with and "
- AdaLN: Adaptive layer normalization used to modulate transformer blocks; here removed from the encoder to simplify conditioning. "No AdaLN in the encoder."
- Classifier-free guidance (CFG): A sampling technique that mixes conditional and unconditional model predictions to steer generations toward the conditioning signal. "On ImageNet we report results both with and without classifier-free guidance (CFG)~\citep{cfg}; on T2I we report only the with-CFG results."
- Decoupled Diffusion Transformer (DDT): A DiT variant that splits the network into an encoder and a shallow, wide decoder to increase effective width at lower cost. "Decoupled Diffusion Transformer (DDT)~\citep{ddt}."
- DiffusionBench: A proposed holistic benchmark aggregating ImageNet and text-to-image results to better reflect broad progress. "we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research."
- Diffusion transformers (DiTs): Transformer-based backbones for diffusion models that operate over image tokens instead of convolutional U-Nets. "Diffusion transformers (DiTs) have become the dominant method for image generation"
- DPG-Bench: A compositional text-to-image evaluation benchmark assessing prompt grounding and object relationships. "We report GenEval~\citep{geneval}, DPG-Bench~\citep{dpgbench}, and GenAIBench~\citep{genaibench}."
- EMA (exponential moving average): A running average of model weights used during training to stabilize evaluation. "maintain an exponential moving average (EMA) of model weights with decay $0.9995$."
- Euler sampler: An ODE-based sampler for diffusion/flow models that integrates the learned dynamics in discrete steps. "we default to an Euler sampler with 50 function evaluations (NFEs)."
- FDr: A feature-space Fréchet-type distance computed using multiple vision encoders to assess fidelity/diversity. "On ImageNet, we support FID~\citep{fid}, IS~\citep{is}, FDr~\citep{fdloss}, and MIND~\citep{mind}."
- FID (Fréchet Inception Distance): A commonly used metric measuring distance between distributions of real and generated images in an Inception feature space. "We report FID~\citep{fid}, IS~\citep{is}, FDr~\citep{fdloss}, and MIND~\citep{mind} metrics."
- Flow matching: A training objective that regresses the velocity field along the diffusion path instead of the noise. "Flow matching~\citep{fmgen, fm} instead regresses the velocity field along the same path."
- Flow-matching loss: The loss used to train models under flow matching, sometimes mixed with other objectives. "For MeanFlow~\citep{meanflow} models, we use 75\% flow-matching loss mixed with 25\% MeanFlow loss."
- GenAIBench: A text-to-image benchmark, often paired with a VQA-based scorer to evaluate prompt alignment and reasoning. "GenAIBench (VQAScore)~\citep{vqascore}"
- GenEval: A compositionality-focused text-to-image benchmark that tests alignment with complex prompts. "We report GenEval~\citep{geneval}, DPG-Bench~\citep{dpgbench}, and GenAIBench~\citep{genaibench}."
- Gradient clipping: A technique to cap the gradient norm to stabilize training and prevent exploding gradients. "We apply gradient clipping at $1.0$"
- In-context conditioning: Feeding conditioning signals (e.g., timestep, class, text) as tokens prepended to the visual tokens rather than via modulators. "In-context conditioning."
- Inception Score (IS): A metric using an Inception classifier to assess both the quality and diversity of generated images. "We report FID~\citep{fid}, IS~\citep{is}, FDr~\citep{fdloss}, and MIND~\citep{mind} metrics."
- Latent-space: The compressed representation space (e.g., VAE or RAE latents) where diffusion models can be trained for efficiency. "we observe that pixel-space FID is typically higher than latent-space FID"
- Logit-normal distribution: A probability distribution used here to sample training timesteps for diffusion objectives. "training timesteps are sampled from a logit-normal distribution with mean $0$ and standard deviation $1$."
- MeanFlow: A one-/few-step generative method/objective for fast sampling in diffusion-like models. "MeanFlow reaches FID $6.60$ and $5.40$ with one or two steps, respectively."
- MIND: A feature-based metric (computed across multiple encoders) used to evaluate generative quality and diversity. "On ImageNet, we support FID~\citep{fid}, IS~\citep{is}, FDr~\citep{fdloss}, and MIND~\citep{mind}."
- NanoGen: A unified training and evaluation framework that supports both ImageNet and text-to-image setups for DiTs. "We introduce NanoGen, a unified DiT training and evaluation framework."
- NFE(s): Number of function evaluations; the count of model calls a sampler uses during generation. "with 50 function evaluations (NFEs)."
- One-/few-step generation: Generating samples in one or a small number of steps using flow-based or MeanFlow objectives. "NanoGen also supports one-/few-step generation."
- Pearson correlation: A statistic measuring linear correlation; used to quantify agreement between ImageNet and T2I rankings. "Pearson correlation is between -0.377 and -0.580 across three metrics."
- Pixel-space: Operating directly on image pixels (without a latent tokenizer) for diffusion training and sampling. "For pixel-space generation, NanoGen operates directly on pixels without any latent tokenizer."
- RAE: Representation autoencoder; uses a frozen vision encoder as a tokenizer to provide both compression and semantic structure. "Representation autoencoders (RAE)~\citep{rae, raev2} merge their ideas and use a frozen vision encoder directly as tokenizer"
- RAEv2: A newer variant of RAE with updated vision encoders/tokenizers supported by NanoGen. "NanoGen supports RAEv2~\citep{raev2} vision encoders and tokenizers."
- REPA: Representation alignment; aligns DiT internal features with a frozen vision encoder to improve training. "representation alignment (REPA)~\citep{repa} aims to align the internal features of DiTs with those of a frozen vision encoder;"
- REPA-E: An extension of REPA that tunes the VAE end to end using the alignment signal. "REPA-E~\citep{repae} uses this alignment signal to additionally tune the VAE end to end."
- T2I (Text-to-image): The task of generating images from textual prompts. "text-to-image (T2I) generation"
- Task-specific conditioning tokens: Dedicated tokens (e.g., class or text tokens) prepended to the visual sequence to condition the model. "The only per-task difference is the number and meaning of conditioning tokens:"
- Timestep shifting: A rescaling of training timesteps based on input dimensionality to stabilize training across tokenizers. "we additionally apply dimension-dependent timestep shifting , where $\alpha = \sqrt{\frac{m}{n}$, , and is the effective input dimension."
- VAE (Variational Autoencoder): A generative model used as a latent tokenizer for diffusion in compressed spaces. "latent diffusion models~\citep{ldm} diffuse in the lower-dimensional latent space of a pretrained VAE~\citep{vae}"
- VQAScore: A VQA-based scoring method used within GenAIBench to assess prompt alignment and reasoning. "GenAIBench (VQAScore)~\citep{vqascore}"
- v-prediction: Training target where the model predicts the velocity along the diffusion path (instead of noise or clean data). "We use -prediction by default."
- x-prediction: Training target where the model predicts the clean signal directly. "For methods whose original recipe specifies -prediction, e.g., JiT~\citep{jit}, PixelGen~\citep{pixelgen}, we preserve -prediction for faithful reproduction."
Collections
Sign up for free to add this paper to one or more collections.