Scaling Laws For Diffusion Transformers

Abstract: Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, e.g., image and video generation. However, scaling laws of DiT are less explored, which usually offer precise predictions regarding optimal model size and data requirements given a specific compute budget. Therefore, experiments across a broad range of compute budgets, from 1e17 to 6e18 FLOPs are conducted to confirm the existence of scaling laws in DiT for the first time. Concretely, the loss of pretraining DiT also follows a power-law relationship with the involved compute. Based on the scaling law, we can not only determine the optimal model size and required data but also accurately predict the text-to-image generation loss given a model with 1B parameters and a compute budget of 1e21 FLOPs. Additionally, we also demonstrate that the trend of pre-training loss matches the generation performances (e.g., FID), even across various datasets, which complements the mapping from compute to synthesis quality and thus provides a predictable benchmark that assesses model performance and data quality at a reduced cost.

• The paper establishes power-law relationships between compute, training loss, and optimal model/data sizes for diffusion transformers.
• The paper demonstrates that optimal parameters scale as C^(0.5681) and data as C^(0.4319), enabling precise performance predictions.
• The paper validates these findings through extrapolation and FID metrics, providing actionable insights for efficient model design.

Scaling Laws for Diffusion Transformers: A Comprehensive Analysis

The paper in question investigates the scaling laws for Diffusion Transformers (DiT), specifically focusing on their applicability to content recreation tasks such as image and video generation. Previous research has extensively explored scaling laws in LLMs; however, the application of similar principles to diffusion models remains less understood. This paper aims to fill this gap by empirically establishing scaling laws for DiT, demonstrating their relevance and predictive power in determining optimal configurations for given computational budgets.

Scaling Laws Exploration

The authors conducted experiments using a varied range of compute budgets, from 1e17 to 6e18 FLOPs, to capture the scaling behavior of DiT. Their findings suggest that the pretraining loss adheres to a power-law relation with the compute budget. This relationship facilitates not only the calculation of optimal model sizes and data requirements but also allows for accurate predictions of specific loss values, such as the text-to-image generation loss for a 1B parameter model with a 1e21 FLOP budget.

Central to this examination is the ability to fit scaling laws that succinctly describe the interplay between compute, model size, data, and resultant performance. The authors derive power-law expressions for optimal model and data sizes as functions of the compute budget, further validating these through extrapolations to higher budgets.

Empirical Results and Predictions

A key empirical result from this study is the confirmation that the optimal model parameter count and data size scale with compute according to $N_{\text{opt} \propto C^{0.5681}$ and $D_{\text{opt} \propto C^{0.4319}$, respectively. Moreover, the study found that the training loss decreases following $L = 2.3943 \cdot C^{-0.0273}$. These scaling laws were put to test through extrapolation to a 1.5e21 FLOP budget, yielding a parameter estimate of approximately 1B, with predictions aligning closely to the observed performance.

Generative Performance and Broad Applicability

Apart from pretraining loss, the study delved into Fréchet Inception Distance (FID) as a measure of generative quality. They discovered that FID also scales predictably with the compute budget, adhering to a power-law trend. This suggests that generation quality is closely tied to the scaling laws of training metrics, allowing researchers to anticipate the qualitative output of models.

Furthermore, the research highlights the scalability of these laws across diverse datasets, evidenced by experiments on the COCO validation set. Despite inherent dataset-induced performance shifts, the consistent scaling behavior was preserved, underscoring the robustness and generality of the established laws.

Implications and Future Directions

This study's findings hold significant implications for the optimization of diffusion models. By leveraging scaling laws, researchers and practitioners can predict model requirements and performance, thus ensuring efficient allocation of computational resources. This can greatly reduce heuristic searches traditionally necessary for model configuration, streamlining the development process.

The potential application of these scaling laws extends beyond current research, suggesting avenues for future inquiry. Understanding the scalability across other modalities such as audio or video could benefit from a similar approach. Further exploration into how these laws interact with hyperparameters like learning rates could also refine predictive accuracy.

Conclusion

In summary, the paper provides a rigorous exploration of the scaling laws for Diffusion Transformers, establishing foundational relationships that predict model behavior across compute budgets. The clarity and predictive power of these laws promise practical benefits in model design and deployment, contributing to the evolving understanding of diffusion models in the AI landscape.