Conditional PixelCNN

• Conditional PixelCNN is an autoregressive generative model that conditions pixel predictions on auxiliary inputs like class labels and latent embeddings.
• It employs masked convolutions, gated activations, and structured conditioning mechanisms to enable effective density estimation and high-quality image synthesis.
• It achieves state-of-the-art performance in tasks such as semantic inpainting, colorization, and scalable image generation, as demonstrated by competitive bits-per-dimension scores on benchmarks.

Conditional PixelCNN is an autoregressive generative model architecture that extends the original PixelCNN framework to enable image generation and density estimation conditioned on auxiliary information. In Conditional PixelCNN, the generative process leverages both local context in the form of previously generated pixels and global or structured side-information such as class labels, latent embeddings, or arbitrary spatial features. This conditional modeling framework underpins state-of-the-art results in conditional image generation, semantic inpainting, colorization, scalable autoencoding, and more, offering exact likelihood computation, flexible conditionality, and high-fidelity sample quality (Oord et al., 2016).

1. Conditional Autoregressive Density Modeling

Conditional PixelCNN models the conditional distribution over pixels of an image $x = (x_1, x_2, ..., x_{n^2})$ in raster scan order, given an external conditioning vector $c$. The conditional factorization follows the autoregressive chain rule:

$$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$

where $x_{<i}$ denotes all previously generated pixels in the pre-specified ordering, and $c$ is an arbitrary side-information vector. This $c$ can represent class labels, learned latent codes, spatial maps, or any hybrid thereof. Each individual conditional $p(x_i \mid x_{<i}, c)$ is parameterized by a deep neural network, facilitating dependencies on both context and conditioning information (Oord et al., 2016).

2. Masked Convolutions and Receptive Field Construction

Strict autoregressive order is enforced in PixelCNN via masked convolutions, where convolutional filters are element-wise multiplied with binary masks to prevent access to "future" pixels. Specifically, Type “A” masks in the input layer block the current and all subsequent pixels, while Type “B” masks used in deeper layers allow local recurrence at the current pixel without leaking information forward. Receptive field increases linearly with the network depth: $1 + (k-1)·L$ for $L$ layers of $k \times k$ convolutions.

Gated PixelCNN, introduced to mitigate the "blind spot" present in single-stack masked architectures, uses parallel "vertical" (unmasked in vertical, masked in horizontal) and "horizontal" (masked in both) stacks. Vertical stack activations are injected into the horizontal stack at each layer:

• $c$0
• $c$1

Here, $c$2 denotes convolution, and $c$3 is a learned projection. This architecture results in an effective full (square) receptive field without inaccessible regions (Oord et al., 2016).

3. Gated Activation Units and Expressivity

To enhance representational power and match the likelihood performance of sequential RNNs such as PixelRNN, the Conditional PixelCNN replaces standard ReLU activations with gated nonlinearities. For each layer input $c$4, two convolutions yield preactivation tensors $c$5 and $c$6; activations are computed as:

$c$7

where $c$8 is elementwise multiplication, and $c$9 denotes the sigmoid function. This data-dependent gating mechanism allows for learnable, multiplicative modulation across features, closing the log-likelihood gap with LSTM-based counterparts while retaining high levels of parallelization (Oord et al., 2016).

4. Conditioning Mechanisms

The conditioning vector $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$0 is injected at every layer as an additive bias, either globally or spatially:

1. Global conditioning: For discrete class labels or non-spatial latent codes, $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$1 passes through learned weight matrices $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$2 and $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$3 at each layer, entering as

$$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$4

This biases each feature map per layer according to $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$5.

2. Spatial conditioning: For conditioning with spatial structure (e.g., segmentation masks, keypoint heatmaps), a deconvolutional network maps $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$6 to a spatial tensor $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$7, and 1×1 convolutions inject $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$8 as a location-dependent bias into the network.

These mechanisms support both class-conditioned models—where $$p(x \mid c) = \prod_{i=1}^{n^2} p(x_i \mid x_{<i}, c)$$9 is a one-hot label—and embedding-conditioned models (e.g. face portrait generation using a deep embedding), as well as spatially-aware tasks such as inpainting or patch-based generation (Oord et al., 2016, Akoury et al., 2017, Dupont et al., 2018).

5. Architectural Instantiations and Training Protocols

A canonical Gated Conditional PixelCNN for 32×32 images uses $x_{<i}$0 layers, $x_{<i}$1 hidden channels, $x_{<i}$2 filter width in both vertical and horizontal stacks, with residual connections and output projection into $x_{<i}$3 logits for multichannel discrete color prediction. Conditioning is incorporated in all convolutional layers using the techniques outlined above. Training is performed using large-batch synchronous SGD, sometimes distributed over up to 32 GPUs, with maximum likelihood as the objective and per-pixel cross-entropy losses in the discrete output setting. Optimization commonly includes teacher-forcing, weight initialization schemes, and batch normalization. No explicit regularization such as dropout is employed in the reference implementation (Oord et al., 2016, Guadarrama et al., 2017).

6. Representative Applications

Class-Conditional and Embedding-Conditional Image Synthesis

Conditional PixelCNN achieves diverse class-conditional generation on datasets such as ImageNet, where a single model is conditioned on any of $x_{<i}$4 class labels, yielding samples with substantial intra-class variation in pose, lighting, and content. Embedding-conditioned models, when provided with a deep feature vector encoding of a reference image, can generate consistent variants of a subject, such as varying facial expressions or poses (Oord et al., 2016).

Probabilistic Semantic Inpainting

Pixel Constrained CNNs extend Conditional PixelCNN to the probabilistic completion of arbitrary image regions. Conditioning is provided not as a vector but as a sparse set of visible pixels and mask. The model fuses two streams: a masked, gated autoregressive prior, and an unmasked conditioning branch, whose logits are summed before softmax. This enables faithful inpainting with diverse plausible completions, robust adherence to conditioning pixels, and principled likelihood scoring (Dupont et al., 2018).

Colorization

Conditional PixelCNNs can autoregressively colorize grayscale images. In PixColor, a PixelCNN is conditioned on the full-resolution luminance channel to generate low-resolution chroma maps. The structured output (coarse but globally consistent colorization) is then refined by a feedforward CNN. PixelCNN provides the sample diversity and adherence to grayscale structure that deterministic models lack (Guadarrama et al., 2017).

Patch-Based and Resolution-Agnostic Generation

Spatial PixelCNNs condition on both coordinate grids and global latent codes to generate images one patch at a time. This approach trains on small patches but synthesizes at arbitrary resolutions, up to 50× scaling, by sampling coordinates across large grids. Coordination between local (masked autoregressive) dependencies and global (coordinate, latent code) conditioning yields high-quality, coherent generations and upscaling performance (Akoury et al., 2017).

Conditional Decoding in Autoencoding Frameworks

PixelGAN autoencoders and related models use Conditional PixelCNN as the decoder in autoencoder architectures. The latent code, typically regularized by adversarial or variational losses, is injected as a global or local bias to shape the decoder’s output distribution. This configuration supports disentanglement between style and content or between global and fine-scale details, and allows generative sampling with latent traversals (Makhzani et al., 2017, Oord et al., 2016).

7. Empirical Performance and Evaluations

Conditional PixelCNN architectures achieve state-of-the-art or near-state-of-the-art likelihoods on benchmarks, with competitive sample quality and diversity. For example, on CIFAR-10, Gated PixelCNN achieves bits per dimension (bpd) of $x_{<i}$5 (train: $x_{<i}$6), matching or outperforming PixelRNN and offering $x_{<i}$7 faster training. On ImageNet 32×32, Gated PixelCNN attains $x_{<i}$8 bpd (PixelRNN: $x_{<i}$9), and on ImageNet 64×64 it achieves $c$0 (PixelRNN: $c$1) (Oord et al., 2016). In conditional tasks such as colorization and semantic inpainting, Conditional PixelCNN models demonstrate superior sample diversity, likelihood-based sample ranking consistent with human judgments, and robust conditional fidelity compared with GANs and other baselines (Guadarrama et al., 2017, Dupont et al., 2018).

Quantitative performance and design choices for several representative models are summarized below:

Task / Model	Key Conditioning	Bits/dim / Metrics	Key Architectural Choices
ImageNet class-conditional	one-hot label	3.83 bpd (32×32)	20 layers, Gated PixelCNN, 384 channels
PixColor colorization	grayscale image	Visual Turing Test (user study)	Masked 7×7/5×5, 10 gated blocks
Spatial PixelCNN	coord grid, VAE latent	≤1.39 bpd (MNIST 16×16)	Gated residual, patch training, VAE code
Pixel Constrained CNN	visible pixel set + mask	pSNR: 17.4–19.5 dB (CelebA)	Gated + unmasked merge, 15-17 residual blks

All reported empirical claims are directly documented in the referenced sources (Oord et al., 2016, Guadarrama et al., 2017, Akoury et al., 2017, Dupont et al., 2018, Makhzani et al., 2017).

Conditional PixelCNN thus constitutes a flexible foundation for conditional generative modeling, density estimation, and structured image synthesis, combining pixel-wise autoregressive dependencies with a range of external conditioning signals in a mathematically principled, computationally efficient framework.