Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Prediction (2405.13218v2)

Abstract: Nearly every recent image synthesis approach, including diffusion, masked-token prediction, and next-token prediction, uses a Transformer network architecture. Despite this common backbone, there has been no direct, compute controlled comparison of how these approaches affect performance and efficiency. We analyze the scalability of each approach through the lens of compute budget measured in FLOPs. We find that token prediction methods, led by next-token prediction, significantly outperform diffusion on prompt following. On image quality, while next-token prediction initially performs better, scaling trends suggest it is eventually matched by diffusion. We compare the inference compute efficiency of each approach and find that next token prediction is by far the most efficient. Based on our findings we recommend diffusion for applications targeting image quality and low latency; and next-token prediction when prompt following or throughput is more important.

• The paper presents a compute-controlled analysis showing that next-token prediction achieves superior CLIP scores and inference efficiency.
• The paper demonstrates that while next-token prediction offers higher initial compute efficiency for FID, diffusion models scale to match its image quality.
• The paper highlights the impact of optimization practices, including EMA benefits in diffusion models, to guide application-specific synthesis choices.

Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Prediction

The paper explores computational tradeoffs in image synthesis, focusing on three transformer-based approaches: diffusion, masked-token prediction, and next-token prediction. This research provides a comprehensive and compute-controlled analysis to elucidate the efficiency and performance across these methodologies.

Key Results

• Performance Metrics: The paper measures image synthesis performance using CLIP scores and Fréchet Inception Distance (FID). Notably, token-based methods, especially next-token prediction, achieve superior CLIP scores, indicating better controllability.
• Image Quality: Initially, next-token prediction shows higher compute efficiency for FID; however, as compute scales, diffusion models match its performance, suggesting scalability in image quality improvement.
• Inference Efficiency: Next-token prediction demonstrates the highest inference compute efficiency. This efficiency is due to its ability to process tokens in parallel, contrasting with the higher demands of iterative denoising in diffusion and masked-token methods.

Methodological Framework

The research builds on transformer architectures, leveraging their scaling capabilities. The paper employs autoencoders for latent space modeling, with both discrete and continuous regularizers. Continuous variants primarily use KL penalties, while discrete ones utilize lookup-free quantization (LFQ), offering insights into how latent space structure affects synthesis outcomes.

Implications

1. Scalability: The findings reinforce the scalability of transformer-based methods, closely following observed trends in neural network scaling research. This adaptability ensures relevance across variable compute budgets and model sizes.
2. Application Tailoring: Depending on the application requirements—whether lower latency or higher throughput—different methods offer distinct advantages. Diffusion is recommended for scenarios prioritizing image quality and low latency. In contrast, next-token prediction is preferable for tasks needing precise prompt-following and throughput.
3. Training Practices: The paper also highlights the impact of optimization strategies. Notably, the use of EMA benefits diffusion models, suggesting that incorporating advanced optimization techniques can further refine synthesis capabilities.

Speculation on Future Developments

Given the distinct scaling dynamics, future developments might explore combinations of these models to harness the unique strengths of each approach. Additionally, advancements in autoencoding, particularly in discrete spaces, could lead to significant enhancements in synthesis efficiency and quality.

Conclusion

This work provides a thorough analysis of computational tradeoffs in image synthesis, offering valuable insights for researchers and practitioners. The exploration of scaling, efficiency, and performance metrics furnishes a robust framework for understanding and optimizing image synthesis approaches, with clear directives for tailoring methodologies to specific computational and application needs.