Pix2PixHD: Baseline for High-Res I2I Translation

• The paper introduces a two-stage generator and triple multi-scale PatchGAN discriminators, scaling conditional GANs to high resolutions with robust performance across diverse domains.
• It integrates adversarial, feature-matching, and perceptual (VGG) losses to enhance both local realism and global structure, validated on applications like satellite flood visualization and SPC-to-RGB translation.
• Its rigorous training protocol and architectural design set a competitive baseline, influencing subsequent advancements such as spatially-adaptive normalization and physics-aware generative models.

Pix2PixHD Baseline refers to the canonical architecture, loss functions, and training protocols described in Wang et al. [CVPR 2018] and widely adopted in subsequent image-to-image (I2I) translation research and challenge benchmarks. The model represents a key advance over the original pix2pix framework (Isola et al., 2016), scaling conditional GAN-based translation to high resolutions and providing robust baseline performance across a spectrum of domains—including satellite flood visualization (Lütjens et al., 2020), remote sensing (Gou et al., 2022), and binary-to-RGB translation from single-photon camera (SPC) data (Sharma et al., 7 Jun 2025).

1. Architecture

Generator

The baseline employs a two-stage, coarse-to-fine ResNet generator structure composed of:

• Global generator (G₁):
  • Initial 7×7 convolution (64 filters, InstanceNorm, ReLU)
  • Two downsampling 3×3 conv layers (strided, 128→256 filters)
  • Nine ResNet blocks at ¼ input resolution
  • Two upsampling layers (nearest-neighbor upsample + 3×3 conv, 128→64 filters)
  • Final 7×7 conv to 3-channel RGB + Tanh nonlinearity
• Local enhancer network (G₂, for 1024×1024+ inputs):
  • 7×7 conv (32 filters), InstanceNorm, ReLU
  • Two downsampling conv layers (64→128 filters)
  • Three ResNet blocks
  • Two upsampling layers (64→32 filters), ending in 7×7 conv to 3-channel RGB + Tanh

For high-res synthesis, G₁’s output is bilinearly upsampled and fused via skip-connections into G₂. All non-output layers use InstanceNorm and ReLU. The model contains approximately 87 million parameters (Lütjens et al., 2020).

Discriminators

Pix2PixHD baseline uses three multi-scale PatchGAN discriminators {D₁, D₂, D₃}, identical in structure but operating at full, ½, and ¼ resolution, respectively:

• Five 4×4 convolutions (stride=2): 64→128→256→512→1 channels
• LeakyReLU (slope=0.2) after all but last layer
• Spectral normalization on all convolutional layers (enforcing Lipschitz continuity, stabilizing GAN training)
• Input consists of the concatenation of the source and target (x, y) images

This multi-scale approach increases both stability and discriminative sensitivity to local and global artifacts.

2. Loss Functions

The Pix2PixHD baseline objective combines adversarial, feature-matching, and perceptual losses:

• Adversarial loss (cGAN):

$$\mathcal{L}_{GAN}(G, D_i) = \mathbb{E}_{(x,y)}[\log D_i(x, y)] + \mathbb{E}_x[\log(1 - D_i(x, G(x)))]$$

The total adversarial loss sums over all discriminators:

$$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$

• Feature matching loss:

$$\mathcal{L}_{FM}(G, D) = \mathbb{E}_{(x, y)} \sum_{i=1}^3 \sum_{j=1}^{L_i} \frac{1}{N_{ij}} \| D_i^{(j)}(x, y) - D_i^{(j)}(x, G(x)) \|_1$$

Here, $D_i^{(j)}(\cdot)$ is the feature map at the $j$-th layer of $D_i$.

• Perceptual (VGG) loss:

$$\mathcal{L}_{VGG}(G) = \mathbb{E}_{(x, y)} \sum_{l=1}^L \frac{1}{M_l} \| \phi_l(y) - \phi_l(G(x)) \|_1$$

$\phi_l(\cdot)$ is the activation of the $l$-th layer in a pretrained VGG-19.

The full generator objective is:

$$G^* = \arg\min_G \left\{ \max_{D_1, D_2, D_3} \mathcal{L}_{GAN}(G, D) + \lambda_{FM} \mathcal{L}_{FM}(G, D) + \lambda_{VGG} \mathcal{L}_{VGG}(G) \right\}$$

with $$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$0, $$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$1 in typical setups (Lütjens et al., 2020). Some applications may also include a pure $$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$2 pixel regression term ($$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$3), or replace the adversarial loss with its hinge variant (Sharma et al., 7 Jun 2025).

3. Training Protocol

Key training parameters adopted across most reported baselines (Lütjens et al., 2020, Gou et al., 2022, Sharma et al., 7 Jun 2025):

Parameter	Typical Value
Batch size	1 (per GPU)
Optimizer	Adam ($$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$4, $$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$5)
Initial learning rate	$$\mathcal{L}_{GAN}(G, D) = \sum_{i=1}^3 \mathcal{L}_{GAN}(G, D_i)$$6
Learning rate schedule	Linear decay to zero after 100 out of 200 epochs
Number of epochs	200
Normalization	Instance normalization (all generator/discriminator layers)
Regularization	Spectral normalization (discriminators, [miyato2018spectral]), no dropout in baseline
Data augmentation	Random flip, rotation, crop, elastic deformation, hue/contrast shifts ([simard2003best], (Lütjens et al., 2020))

Dataset and resolution choices follow task-specific requirements, e.g., 1024×1024 for xBD flood visualization (Lütjens et al., 2020), 256×256 for satellite I2I (Gou et al., 2022), or inputs mirroring binary SPC data for novel-view synthesis (Sharma et al., 7 Jun 2025).

4. Data Augmentation and Domain Adaptation

Standard augmentations—horizontal flipping, rotation, random cropping, elastic deformation—are routinely applied to increase effective sample diversity and support geometric robustness (Lütjens et al., 2020). Input channel adaptation is performed as necessary (e.g., to accept multi-channel SAR bands (Gou et al., 2022) or 3-channel binary SPC frames (Sharma et al., 7 Jun 2025)).

A plausible implication is that restricting augmentation (as in (Gou et al., 2022)) can lower the risk of distributional mismatch for geospatial or scientific data, whereas aggressive augmentation is useful for natural images or disaster scenarios.

No physics-based conditioning or auxiliary physical input channels are used in the unconditional Pix2PixHD “baseline” (Lütjens et al., 2020).

5. Quantitative Baseline Performance

Metrics are tailored to the application and dataset:

• Physical-consistency (flood visualization): Intersection-over-Union (IoU) between segmentation masks predicted from generated images and ground truth (Lütjens et al., 2020).
• Photorealism: Learned Perceptual Image Patch Similarity (LPIPS) [Zhang_2018]; PSNR and SSIM for regression or remote sensing tasks (Gou et al., 2022, Sharma et al., 7 Jun 2025).
• Composite task metrics: Flood Visualization Plausibility Score (FVPS), the harmonic mean of IoU and (1-LPIPS) (Lütjens et al., 2020).

Notable results:

Domain / Metric	Pix2PixHD Baseline Outcome
Flood: IoU / LPIPS / FVPS	0.226 / 0.293 / 0.275
MultiEarth (MAE, PSNR)	0.02258 MAE, 30.98 dB PSNR (Gou et al., 2022)
SPC-to-RGB/PSNR/SSIM/LPIPS	22.70 / 0.6843 / 0.4949 (Sharma et al., 7 Jun 2025)

In all cases, Pix2PixHD sets a rigorous baseline for more specialized or physics-aware generative pipelines.

6. Relationship to Pix2Pix and Architectural Innovations

Pix2PixHD extends the original pix2pix baseline (Isola et al., 2016), which consisted of a U-Net generator and single-scale PatchGAN discriminator with a cGAN + L₁ loss, to larger scale and higher fidelity:

• Key innovations:
  • Multi-scale (coarse-to-fine) generator capable of 1024×1024 or higher output
  • Three parallel, multi-scale PatchGAN discriminators
  • Feature matching loss for enhanced perceptual and statistical alignment
  • Perceptual (VGG) loss to reinforce high-level content similarity
  • InstanceNorm throughout, with removal of dropout

This modular architecture enables adaptation to diverse I2I translation regimes, from cross-modality satellite data (Gou et al., 2022) to pixel-level hallucination from binary input (Sharma et al., 7 Jun 2025), preserving local realism and global structure under adversarial training.

7. Limitations and Comparisons

Across application domains, the Pix2PixHD baseline typically yields competitive realism and accuracy but is often outperformed by architectures leveraging spatially-adaptive normalization (SPADE), explicit physical constraints, or advanced ensemble strategies (Gou et al., 2022, Lütjens et al., 2020). Omitting specific loss terms (e.g., feature-matching) or architectural components systematically worsens performance, evidenced by PSNR drops and poorer perceptual scores (Sharma et al., 7 Jun 2025).

A plausible implication is that while Pix2PixHD remains a robust and widely recognized baseline, domain adaptation, loss engineering, and data conditioning remain essential for pushing task-specific state of the art. Its rigorous structure and open implementation cement its role as the reference foundation for comparative evaluation in contemporary I2I research.