Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (1902.00275v2)

Abstract: Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models. Our implementation is available at https://github.com/aravindsrinivas/flowpp

• The paper introduces variational dequantization to enable a more natural density allocation and enhance training performance.
• The paper replaces traditional affine coupling layers with logistic mixture CDF flows to boost transformation expressiveness.
• The paper integrates self-attention in conditioning networks to effectively capture long-range dependencies and improve modeling capacity.

Overview of Flow++: Advancements in Flow-Based Generative Models

The paper, "Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design," presents novel improvements to flow-based generative models, aimed at reducing the performance disparity between flow-based models and the state-of-the-art autoregressive models. The authors target three primary limitations in existing flow-based models: suboptimal dequantization techniques, limited expressiveness of affine coupling layers, and the constraints of purely convolutional conditioning networks. These improvements culminate in the development of the Flow++ model, which establishes new standards for non-autoregressive models in unconditional density estimation.

Key Contributions

1. Variational Dequantization: Flow++ introduces variational dequantization to address issues with uniform dequantization. Unlike uniform dequantization, which can force the model to unnaturally allocate density, variational dequantization uses a flow-based approach, allowing the model to distribute density in a more natural manner. This change significantly improves both training and generalization behavior.
2. Expressive Coupling Layers: The paper introduces logistic mixture CDF coupling flows to replace the traditional affine coupling layers. This modification enhances the expressiveness of the transformation on $x_2$, yielding better density modeling performance without increasing the computational complexity significantly.
3. Enhanced Conditioning Networks: By incorporating self-attention mechanisms within the conditioning networks of coupling layers, Flow++ overcomes the limitations of convolution-only architectures. This change draws inspiration from architectures like the Transformer and provides improved modeling capacity, especially in capturing long-range dependencies.

Experimental Results

The enhancements made to the Flow++ model demonstrate substantial improvements in density modeling tasks. The Flow++ achieves state-of-the-art results among non-autoregressive models across several benchmarks, notably surpassing previous models like RealNVP, Glow, and IAF-VAE. It also competes closely with autoregressive models, achieving metrics on par with the initial iterations of PixelCNN models.

Experiments on datasets such as CIFAR10 and various ImageNet resolutions illustrate Flow++'s capability to closely match autoregressive models' sample quality while maintaining efficient sampling speeds. Notably, Flow++ samples are generated more than an order of magnitude faster than PixelCNN++ samples, highlighting its effectiveness in practical scenarios requiring speed and efficiency.

Ablation Studies

The paper includes comprehensive ablation studies, assessing the impact of each proposed component. Results indicate that variational dequantization and expressive coupling layers significantly enhance model performance, with variational dequantization having the most substantial effect on both training loss and generalization capacity. This highlights the importance of improved dequantization techniques in training flow models effectively.

Implications and Future Research

The advancements proposed in Flow++ point toward more efficient and expressive flow-based generative models. By improving the dequantization process and coupling layer expressiveness while integrating attention mechanisms, Flow++ sets a new benchmark for non-autoregressive density estimation. This work suggests promising directions for future research in designing flow models that balance expressiveness, efficiency, and scalability.

Future work could explore further optimizations in coupling layers, more sophisticated attention mechanisms, and applications to other data modalities. Additionally, investigating the integration of these techniques with hybrid models or in conjunction with variational inference may open up new directions in probabilistic modeling and generative tasks.

In conclusion, Flow++ represents a significant step forward in narrowing the gap between different classes of generative models, enhancing the applicability and performance of flow-based approaches in machine learning and AI.