Multiscale Parallelized PixelCNN

• The paper introduces a multiscale strategy that reduces sequential sampling steps from O(N) to O(log N), significantly accelerating high-resolution image generation.
• It leverages coarse-to-fine image decomposition and conditional pixel grouping to enable parallel sampling, maintaining competitive density estimation performance.
• Empirical results demonstrate speedups of up to 1000× while preserving image fidelity, making it practical for rapid synthesis of complex images.

Multiscale Parallelized PixelCNN refers to a class of autoregressive generative image models that exploit a coarse-to-fine, multiscale hierarchy to reduce the sequential depth required for high-fidelity image synthesis, enabling substantial parallelization and improved sampling efficiency over classic PixelCNN variants. The key insight is to partition the image into a hierarchy of scales and locally conditionally independent pixel groups, architecting the joint pixel distribution such that blocks of pixels may be generated simultaneously at each resolution. This approach maintains competitive density estimation performance while drastically accelerating synthesis, with theoretical and empirical speedups of up to three orders of magnitude for high-resolution images (Reed et al., 2017, Kolesnikov et al., 2016).

1. Multiscale Image Decomposition and Pixel Grouping

Both auxiliary-variable PixelCNNs and parallelized multiscale PixelCNNs build a hierarchical representation by recursively downsampling the input image. Let $I$ denote the full-resolution image. Construct an $L+1$-level image pyramid $\{I^0, I^1, \ldots, I^L\}$ such that

$$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$

for $s = L, L-1, \dots, 1$, resulting in a sequence from coarse ($I^0$) to fine ($I^L$) resolution (Kolesnikov et al., 2016).

At each scale $s$, the image is partitioned into non-overlapping $2 \times 2$ spatial blocks. Each block is assigned to four groups (upper-left, upper-right, lower-left, lower-right). No two spatially adjacent high-resolution pixels fall within the same group. These groups are generated in a fixed 1→2→3→4 order. At every scale, generation proceeds by producing all pixels in group $g$ in parallel, conditioned on all previous groups and all pixels at coarser scales (Reed et al., 2017).

2. Probabilistic Model Factorization and Dependency Structure

The full joint distribution over an image $L+1$0 is formulated as a product across scales and across groups within each scale:

$L+1$1

where $L+1$2, $L+1$3 denotes the set of pixels in group $L+1$4 at scale $L+1$5, and $L+1$6 is the set of conditioning variables, including all coarser scale pixels ($L+1$7) and all previous groups at the current scale ($L+1$8) (Reed et al., 2017). When external context $L+1$9 is provided (e.g., class label, text), each factor is augmented as $\{I^0, I^1, \ldots, I^L\}$0.

Pixels within the same group are treated as conditionally independent given the context, enabling their simultaneous sampling.

3. Parallel Sampling Algorithm and Sequential Depth

Sampling proceeds in two phases. First, a PixelCNN is used to generate the base- or coarsest-resolution image (e.g., $\{I^0, I^1, \ldots, I^L\}$1). Subsequently, at each finer scale $\{I^0, I^1, \ldots, I^L\}$2:

1. For each group $\{I^0, I^1, \ldots, I^L\}$3, gather the conditioning variables.
2. Pass the context to the group-specific upscaler network to obtain parameters for all pixels in $\{I^0, I^1, \ldots, I^L\}$4 in parallel.
3. Sample all pixels in $\{I^0, I^1, \ldots, I^L\}$5 simultaneously.

The high-level procedure is as follows (Reed et al., 2017):

$$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$9

The number of sequential network-evaluation steps for an $\{I^0, I^1, \ldots, I^L\}$6-pixel image is $\{I^0, I^1, \ldots, I^L\}$7 (since $\{I^0, I^1, \ldots, I^L\}$8 and $\{I^0, I^1, \ldots, I^L\}$9 groups per scale), in contrast to the strictly $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$0 steps needed for conventional PixelCNN, which generates each pixel sequentially (Reed et al., 2017).

4. Upscaler Network Architecture and Conditional Parameterization

Each upscaler network can adopt one of two variants:

• Type-A (ResNet): A deep stack of residual convolutional blocks, inputting the bilinearly upsampled previous scale plus context features, and outputting predictive parameters for the corresponding group of pixels. The merging into the full-resolution image is via straightforward interleaving.
• Type-B (ResNet + shallow PixelCNN): A deep ResNet encoder produces a feature map, which is split into $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$1 patches. Each patch is fed into a shared shallow PixelCNN (e.g., 4 masked convolutional layers) that predicts the local $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$2 group. This ensures local spatial dependencies are modeled without border artifacts, as patches never abut group-$$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$3 pixels directly (Reed et al., 2017).

All context encodings and shared weights can be precomputed and reused across group computations. For color images, intra-pixel (channel-wise) masking, such as the red→green→blue ordering, is enforced in the masked convolutions.

5. Computational Complexity and Empirical Performance

The proposed factorization transforms the sequential generation depth from $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$4 to $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$5, with computation for pixels in the same group performed in a single GPU tensor operation.

Image Size	PixelCNN (depth)	Multiscale PixelCNN (depth)	Wall-clock Speedup
32×32	O(1024)	O(20)	~100× faster on ImageNet
256×256+	O(65536+)	O(36)	Practical subsecond generation

(Reed et al., 2017)

Empirically:

• On 32x32 ImageNet, Multiscale PixelCNN is approximately 100× faster than standard PixelCNN (Reed et al., 2017).
• Sampling speeds for multi-resolution models are an order of magnitude faster than monolithic PixelCNN++ models at comparable or only marginally worse likelihood (≤2.98 vs 2.92 bits-per-dimension on CIFAR-10) (Kolesnikov et al., 2016, Reed et al., 2017).
• Speedup increases with image resolution, enabling synthesis of 256×256 and 512×512 images in practical timeframes.

Quantitative metrics also show that autoregressive likelihoods remain near state-of-the-art. For class-conditional ImageNet at 64×64 resolution, Multiscale PixelCNN achieves 3.67 bits per sub-pixel, versus 3.48 for PixelCNN, with other density models trailing (Reed et al., 2017).

6. Sample Quality, Applications, and Interpretive Notes

Samples synthesized by multiscale and parallel PixelCNN variants exhibit improved global structural coherence, a direct result of capturing long-range dependencies at coarse scales and incrementally refining details at finer resolutions (Kolesnikov et al., 2016). On datasets such as CelebA, face samples at 128×128 are sharp and diverse; CIFAR-10 samples demonstrate global object coherence, mitigating the “texture bubble” artifacts of flat PixelCNNs. In conditional and video-prediction settings, the multiscale model supports high-resolution synthesis for text-to-image, class-conditional, and action-conditional tasks, often achieving competitive or superior performance among non-pixel-autoregressive models (Reed et al., 2017).

A plausible implication is that local conditional independence assumptions—specifically, grouping $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$6 pixels as conditionally independent given the context—strike a balance between computational tractability and the expressiveness needed for high-fidelity image generation.

7. Summary of Advantages and Methodological Trade-Offs

Multiscale Parallelized PixelCNN achieves:

• Reduction of sequential sampling steps from $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$7 to $$I^L = I, \quad I^{s-1} = \mathrm{Downsample}_2(I^s)$$8 per image.
• Large wall-clock speedups (100–1000×) with negligible loss in likelihood accuracy and high sample fidelity.
• Preservation of maximum-likelihood training without variational approximations, as all auxiliaries derive deterministically from the original image (Kolesnikov et al., 2016, Reed et al., 2017).
• A “divide and conquer” approach where coarse scales model long-range structure and fine scales refine details, using only shallow per-scale networks.

The primary methodological trade-off is the merging of per-pixel factors into local groups, which introduces minor loss in dependency structure but yields dramatic computational gains. This architecture underlies practical large-scale autoregressive image and video generation.