Parallel Multiscale Autoregressive Density Estimation (1703.03664v1)

Abstract: PixelCNN achieves state-of-the-art results in density estimation for natural images. Although training is fast, inference is costly, requiring one network evaluation per pixel; O(N) for N pixels. This can be sped up by caching activations, but still involves generating each pixel sequentially. In this work, we propose a parallelized PixelCNN that allows more efficient inference by modeling certain pixel groups as conditionally independent. Our new PixelCNN model achieves competitive density estimation and orders of magnitude speedup - O(log N) sampling instead of O(N) - enabling the practical generation of 512x512 images. We evaluate the model on class-conditional image generation, text-to-image synthesis, and action-conditional video generation, showing that our model achieves the best results among non-pixel-autoregressive density models that allow efficient sampling.

• The paper introduces a novel coarse-to-fine strategy that reduces pixel dependencies, enabling parallel sampling and significantly faster inference.
• It achieves a reduction in sampling complexity from O(N) to O(log N), facilitating efficient high-resolution image generation up to 512×512 pixels.
• Experimental results on ImageNet, CUB, and MS-COCO showcase superior performance and versatility in both image and video generation tasks.

Parallel Multiscale Autoregressive Density Estimation

This paper presents a novel modification of the PixelCNN framework, focusing on enhancing the efficiency of autoregressive density estimation through parallelism and multiscale image modeling. The standard PixelCNN provides state-of-the-art density estimation for natural images but operates with high computational costs during inference due to its pixel-by-pixel sequential nature. The proposed approach addresses this by introducing a multiscale, parallelized version, termed "Parallel PixelCNN," which achieves significant improvements in sampling speed by structuring certain pixel groups as conditionally independent.

Background and Motivation

PixelCNN and other autoregressive models have traditionally decomposed image distribution into per-pixel factors, using a carefully structured deep convolutional network to maintain causal pixel dependencies. Although effective, the sequential pixel generation limits inference speed, typically scaled at $O(N)$ for $N$ pixels. Furthermore, these models tend to work inefficiently at high resolutions due to the accumulating computational costs.

Model Overview and Contributions

The proposed Parallel PixelCNN leverages coarse-to-fine multiscale image generation, which dramatically reduces dependencies among pixel groups. By doing so, the approach obtains a sampling complexity of $O(\log N)$, allowing the practical generation of high-resolution images, notably up to $512 \times 512$. The authors propose forming successively higher-resolution views of an image where pixel groups within the same scale can be sampled independently, thereby facilitating parallel generation. This framework effectively uses spatial locality to reduce unnecessary dependencies and dependencies among each group, thus maintaining model expressiveness.

Key contributions include:

1. A novel coarse-to-fine strategy cutting spatial dependencies in PixelCNN allows significant speed increases with minimal performance trade-offs.
2. The ability to perform inference in $O(\log N)$ time, a stark improvement over traditional approaches.
3. Successful scaling to higher resolution images and robust performance across multiple generative modeling tasks.

Results and Evaluation

To validate the effectiveness of the model, extensive experiments were performed across various datasets, such as ImageNet, CUB, and MS-COCO, tackling distinct generative tasks, including class-conditional image generation, text-to-image synthesis, and video generation. Results demonstrate that Parallel PixelCNN achieves compelling density estimation results, outperforming other non-pixel-autoregressive density models by significant margins. The paper quantifies these improvements with orders of magnitude faster sampling without compromising the quality of generated outputs.

In video generation tasks, the model demonstrates superior capabilities in action-conditional scenarios, showcasing its versatility beyond static image tasks.

Implications and Future Directions

This work represents a significant step forward in autoregressive density modeling by addressing one of its key limitations: sampling speed. The approach can be seen as a fundamental architectural improvement that can be particularly valuable in real-time image synthesis applications or scenarios where high-resolution content needs to be generated efficiently.

Future research could further explore the integration of this parallelized approach with other generative frameworks or extensions involving hybrid models combining advantages from autoregressive and non-autoregressive paradigms. Additional work could examine its utility in domain adaptation or generalization to unseen configurations, as observed in the video generation section, highlighting avenues for potential improvements with regularization or more diverse training datasets. Furthermore, integration with caching and in-graph computation technologies promises additional efficiency gains, especially in real-time applications.

The methodology outlined in this paper provides a robust framework for efficiently handling complex image generation tasks, underscoring the potential of parallel autoregressive models in advancing the field of deep generative modeling.