Laplacian Pyramid GANs (LAPGAN)

• The paper introduces LAPGAN, a hierarchical GAN that synthesizes images by decomposing them into multi-scale components using Laplacian pyramid techniques.
• It employs conditional adversarial networks at each scale to refine images incrementally from low-frequency structures to high-frequency textures.
• The model demonstrates improved image fidelity and diversity, validated by quantitative metrics on datasets like CIFAR10 and LSUN, influencing subsequent generative methods.

Laplacian Pyramid Generative Adversarial Networks (LAPGAN) model the synthesis of natural images as a hierarchical process operating across multiple scales. Originating from the work of Denton et al. (2015), LAPGAN exploits the Laplacian pyramid decomposition to break down image generation into conditional adversarial refinement stages, where each stage is responsible for synthesizing band-pass image components conditioned on coarser scale approximations. This coarse-to-fine structured approach advances over single-scale GANs by hierarchically capturing global structure, semantic content, and fine-grained texture, producing images with increased fidelity and perceptual realism (Denton et al., 2015).

1. Laplacian Pyramid Representation and Motivation

LAPGAN is premised on the observation that natural images exhibit strong cross-scale correlations: low-frequency (coarse) bands encode global geometric structure, while higher-frequency bands correspond to edges and texture. Formally, let $I_0 = I$ denote a color image of size $J \times J$. Define a downward operator $d(\cdot)$ for smoothing and decimation by two, yielding a Gaussian pyramid $\{I_0, I_1, ..., I_K\}$, where $I_{k+1} = d(I_k)$ recursively until $I_K$ is reduced to a small size (e.g., $8 \times 8$).

The Laplacian pyramid coefficients $\{h_0, ..., h_K\}$ are computed as

$$h_k = I_k - u(I_{k+1}),\quad \text{for}\ k=0, ..., K-1;\qquad h_K = I_K,$$

where $u(\cdot)$ is an upsampling operator that smooths then doubles the spatial size. Each $h_k$ is a band-pass image encoding detail at spatial frequencies between $J/2^{k+1}$ and $J/2^k$. The full image can be recursively reconstructed by

$$I_k = u(I_{k+1}) + h_k,\quad \text{from}\ k=K\ \text{down to}\ 0.$$

This cascaded representation allows LAPGAN to focus the synthesis task at each scale, simplifying learning and improving synthesis quality by capturing detail layer-wise (Denton et al., 2015).

2. Hierarchical GAN Architecture

At each scale $k$ of the Laplacian pyramid, LAPGAN employs a pair of neural networks, $(G_k, D_k)$, constituting a conditional GAN operating at that level. The generator $G_k$ synthesizes the band-pass image $\tilde{h}_k$ given a noise vector $z_k$ and a low-pass conditioning image $l_k \equiv u(I_{k+1})$. The discriminator $D_k$ receives either the true $h_k$ or synthesized $\tilde{h}_k$—each concatenated with the same $l_k$—and outputs a probability that the input is real.

Network architectures are scale-dependent:

• At the coarsest scale ($k = K$), $G_K$ and $D_K$ are fully connected: e.g., $G_K$ maps a latent code $z_K$ through two hidden layers to an output image, and $D_K$ mirrors this structure with hidden units and sigmoid output.
• At finer scales, $G_k$ is a 3-layer convolutional network (e.g., 5×5 filters, increasing channels, batch normalization and ReLU activations). The noise $z_k$ is projected or tiled and concatenated as an additional channel, enabling stochastic detail.
• $D_k$ is a 2-layer convolutional network ending in a sigmoid, with analogous structure but fewer layers than the generators.

For higher-resolution targets (e.g., LSUN scenes at $64\times64$), generators and discriminators are correspondingly deeper convolutional stacks with larger filter sizes and feature maps (Denton et al., 2015).

3. Training Objectives and Procedure

LAPGAN applies the original GAN minimax game at each pyramid level. For $k < K$ (not the coarsest), the conditional adversarial objective is

$$\min_{G_k}\max_{D_k}\ \mathbb{E}_{h_k, l_k \sim p_\text{data}} \left[ \log D_k(h_k, l_k) \right] + \mathbb{E}_{z_k \sim p_z, l_k \sim p_\text{data}} \left[ \log(1 - D_k(G_k(z_k,l_k), l_k)) \right].$$

At $k=K$, the loss is the standard canonical GAN objective applied to the small image $I_K$ without conditioning. Networks at each pyramid level are trained independently using alternating stochastic gradient descent, and their model selection utilizes Parzen-window log-likelihood on validation splits. For data-limited domains (e.g., CIFAR10), data augmentation such as random cropping is employed to mitigate overfitting (Denton et al., 2015).

For class-conditional image synthesis, a class vector $c$ is appended to each $G_k$ and $D_k$ via a linear projection reshaped as a spatial map, enabling control over generated categories.

4. Sampling and Image Synthesis

Novel sample generation in LAPGAN proceeds via a coarse-to-fine reconstruction analogous to the Laplacian pyramid decoding process. For noise vectors $\{z_K, ..., z_0\}$:

• Initialize the coarsest image $\hat{I}_K = G_K(z_K)$.
• For $k = K-1$ downto $0$:
  1. Upsample the current image: $l_k = u(\hat{I}_{k+1})$.
  2. Generate the synthetic band-pass image: $\hat{h}_k = G_k(z_k, l_k)$.
  3. Aggregate to next finer level: $\hat{I}_k = l_k + \hat{h}_k$. The process continues recursively until the finest resolution $\hat{I}_0$ is constructed (Denton et al., 2015).

An equivalent rephrasing in terms of residual (band-pass) synthesis is utilized in subsequent works such as MelanoGANs (Baur et al., 2018), with variants employing only the coarsest-scale latent code or adapting the upsampling operation (e.g., bilinear, deconvolution, or learned upsampling).

5. Quantitative and Qualitative Evaluation

LAPGAN achieves significant improvements in both quantitative log-likelihood and perceptual quality over single-scale GANs. On CIFAR10:

• Parzen-window log-likelihood: standard GAN $\approx -3617 \pm 353$; LAPGAN $\approx -1799 \pm 826$.

• Human “fooling” rate (percentage of times synthetic samples are labeled as real by evaluators): baseline GAN $< 10\%$, LAPGAN (unconditional) $\approx 35\%$, LAPGAN (class-conditional) $\approx 40\%$, real images $> 90\%$. Visual inspection reveals that LAPGAN generates samples with coherent object structure, sharp edges, and detail across scales, far exceeding single-GAN baselines. For higher-resolution datasets (e.g., LSUN 64x64), LAPGAN synthesizes structured scene images (e.g., church fronts, bedrooms) with realistic large-scale and fine-scale features (Denton et al., 2015).

The comparative study in MelanoGANs (Baur et al., 2018) supports these findings at higher resolutions (256×256). Key observations include:

• LAPGAN generates diverse, detailed samples but can exhibit high-frequency residual artifacts.
• Direct comparison of histogram metrics (JS divergence and Earth-Mover's Distance) with DCGAN and DDGAN (a modified LAPGAN): LAPGAN shows greater visual diversity but less accurate color histograms than DCGAN.
• In medical image synthesis (skin lesions), LAPGAN-derived synthetic augmentations can improve classifier validation accuracy over using only real data (LAPGAN samples: val acc 74.0%; baseline: 71.6%).

6. Variants, Extensions, and Practical Considerations

MelanoGANs (Baur et al., 2018) introduce a set of architectural and training modifications to LAPGAN, motivated by application to high-resolution and data-scarce domains. Notable changes include:

• Single-source noise: Only the coarsest scale generator receives explicit noise, while higher-level generators operate deterministically.
• Image-based discrimination: Discriminators at finer levels are tasked with classifying full images, rather than residual bands.
• Residual-deconvolution blocks: Generators for higher pyramid levels are structured as shallow residual networks (ResDeconv) applying learned corrections atop upsampled lower-resolution images.
• Learned upsampling: Replacement of fixed (e.g., bilinear) upsampling by learned deconvolution layers is explored, with measured artifact tradeoffs.
• End-to-end training: Generators and discriminators across all scales are trained jointly rather than independently, facilitating stability during training.

These modifications provide advantages in training stability, speed, and sample quality under application constraints, though may introduce additional artifacts or reduce diversity depending on configuration. Comparative evaluations show that DCGAN best matches color histograms, LAPGAN produces the most diversity and texture, and DDGAN (upsampling variant) yields a favorable balance between artifact suppression and sample variety at high resolution (Baur et al., 2018).

7. Impact and Applications

LAPGAN and its variants have driven advances in image synthesis by structurally aligning generative modeling with the intrinsic multiscale statistics of natural images, yielding substantial improvements in both visual realism and sample diversity relative to monolithic GAN architectures. Its hierarchical residual structure and independent per-scale modeling simplify generation at each stage, enabling synthesis of high-resolution images on limited data.

In practical settings, LAPGAN-generated images have been successfully applied to data augmentation tasks, such as compensating for class imbalance in medical imaging (melanoma lesion datasets), where synthetic samples bolster classifier accuracy in low-data regimes (Baur et al., 2018). The methodology underlies subsequent multi-scale approaches in generative modeling, influencing research in conditional image synthesis, texture transfer, and even more recent hierarchical diffusion models.

A plausible implication is that further research could explore adaptive scale selection, multivariate hierarchical conditioning, or integration with alternate generative frameworks to enhance fidelity or interpretable control across scales.