Pyramid Pix2Pix: Multi-Scale GANs

Updated 16 July 2026

Pyramid Pix2Pix is a family of conditional GANs that incorporate pyramid structures within generators, losses, or both to address multi-scale image translation challenges.
It employs strategies like pyramid-embedded encoders, Gaussian pyramid losses, and cascaded GAN stacks to effectively balance global structure and local details.
Experimental results in Chinese font synthesis, pathology translation, and high-resolution inpainting show improved metrics and reduced structural errors over standard Pix2Pix models.

Searching arXiv for the specified papers and closely related Pix2Pix work to ground the article in current arXiv records. Pyramid Pix2Pix can be understood as an umbrella designation for Pix2Pix-style conditional generative adversarial networks in which pyramid structure is introduced into the generator, the supervision, or the full coarse-to-fine synthesis stack. Across the cited literature, this designation covers at least three recurrent constructions: a pyramid-embedded encoder that injects multi-scale copies of the input into a U-Net-like generator for Chinese font synthesis (Sun et al., 2018); a standard Pix2Pix backbone augmented by Gaussian pyramid multi-scale losses for structurally aligned but not pixel-perfect pathology translation (Liu et al., 2022); and hierarchical stacks of conditional GANs that restore coarse structure at low resolution and refine detail at higher resolutions for high-resolution inpainting (Cao et al., 2020, Qu et al., 2021). In all of these cases, the underlying problem remains paired image-to-image translation, but the role of “pyramid” differs substantially from one formulation to another.

1. Definition within the Pix2Pix lineage

In the Pix2Pix formulation, a generator learns a conditional mapping from an input image to a paired target image, and a discriminator enforces realism in the target domain. The baseline components emphasized across the cited works are a conditional adversarial objective, a pixel-wise reconstruction term such as $L_1$ , and an image-to-image backbone such as U-Net or ResNet. A plain Pix2Pix study on cartoon-to-real translation used a U-Net 256 generator, a PatchGAN discriminator, and $\lambda = 100.0$ for the $L_1$ term, illustrating the standard recipe from which pyramid variants depart (Rajput et al., 2021).

What distinguishes Pyramid Pix2Pix is not a single canonical architecture but a family resemblance. In PEGAN, the pyramid is embedded in the generator’s encoder through cascaded refinement connections that concatenate downsampled copies of the original input to intermediate encoder features, while mirror skip connections preserve the U-Net correspondence between encoder and decoder stages (Sun et al., 2018). In the BCI model, by contrast, the generator and discriminator remain pix2pix-style, and the novelty lies in a Gaussian pyramid applied to both prediction and target so that similarity is enforced at progressively coarser scales rather than only at the original resolution (Liu et al., 2022). In high-resolution inpainting, PyramidFill and the “structure first detail next” model replace the single generator with a pyramid of scale-specific GANs, so that content completion and texture synthesis are separated across resolutions (Cao et al., 2020, Qu et al., 2021).

This suggests that “pyramid” in this context denotes a design principle rather than a fixed module. The principle is to redistribute structure preservation, style transfer, and realism constraints across multiple scales instead of concentrating them at one full-resolution mapping.

2. Principal architectural forms

The main architectural forms can be organized by where the pyramid enters the model.

Form	Mechanism	Representative work
Pyramid-embedded generator	Downsampled inputs are concatenated into encoder layers	PEGAN (Sun et al., 2018)
Pyramid-supervised loss	Gaussian pyramid losses are added to $L_1$ and cGAN losses	BCI “Pyramid Pix2Pix” (Liu et al., 2022)
Pyramid of generators	Separate GANs operate from coarse to fine resolutions	PyramidFill (Cao et al., 2020), “Structure First Detail Next” (Qu et al., 2021)

In PEGAN, the source glyph image $z$ is resized into a pyramid $z_i$ , and encoder layer $e_i$ receives the concatenation of $F_{i-1}$ and the corresponding downsampled image. The resulting mapping is

$F_i = e_i\big([F_{i-1}; z_i]\big),$

with encoder features also connected to decoder features through mirror skip connections (Sun et al., 2018). The explicit intention is to keep exact stroke layout and global glyph shape available at every encoder depth, rather than relying solely on a bottleneck and skips.

In the BCI model, there is no pyramid inside the backbone. The generator is ResNet-9blocks from the pix2pix codebase, the discriminator is a single PatchGAN, and the pyramid exists entirely as a fixed scale-space operator $F_i$ applied to the generated and real images (Liu et al., 2022). Each octave has 5 layers and performs 4 Gaussian blurrings; the first layer of each octave is treated as a scale. The architecture is therefore unchanged at the network level and modified only at the supervision level.

In PyramidFill, the pyramid is a stack of fully convolutional GANs. The lowest-resolution content GAN predicts coarse semantics, and each higher-resolution texture GAN receives the masked image at its scale together with the previous level’s completion, then synthesizes finer textures (Cao et al., 2020). The “structure first detail next” model uses an analogous three-level pyramid, $\lambda = 100.0$ 0, $\lambda = 100.0$ 1, and $\lambda = 100.0$ 2, at $\lambda = 100.0$ 3, $\lambda = 100.0$ 4, and $\lambda = 100.0$ 5, respectively, with each sub-generator itself being a two-stage DeepFillv2-style network (Qu et al., 2021). In that formulation, higher levels refine the sum of the current coarse prediction and the upsampled previous output: $\lambda = 100.0$ 6

A related but distinct direction is the Fractal Pyramid Network, which is not itself a Pix2Pix model but replaces the encoder-decoder bottleneck with multiple information processing pathways and multiple separate small-channel features across scales (Deng et al., 2021). A plausible implication is that PFN-style backbones provide a structural template for future Pyramid Pix2Pix generators that seek stronger global-context propagation without a single dominant bottleneck.

3. Objectives and multi-scale supervision

The loss design of Pyramid Pix2Pix variants retains the Pix2Pix core while redistributing constraints across scales. In the BCI model, the conditional adversarial loss and original $\lambda = 100.0$ 7 loss are preserved,

$\lambda = 100.0$ 8

$\lambda = 100.0$ 9

and a Gaussian pyramid loss is added: $L_1$ 0 The resulting objective is

$L_1$ 1

The explicit rationale is that strict pixel-wise $L_1$ 2 is too rigid when paired images are only structurally aligned, since small registration errors otherwise bias the generator toward blurred averages (Liu et al., 2022).

PEGAN extends Pix2Pix-style adversarial plus $L_1$ 3 supervision in a different direction. Its final objective combines adversarial loss, pixel-wise $L_1$ 4, category loss, and perceptual loss,

$L_1$ 5

with $L_1$ 6, $L_1$ 7, $L_1$ 8, and $L_1$ 9 (Sun et al., 2018). The perceptual term uses VGG-19 features from conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2, with $L_1$ 0 and $L_1$ 1. The category loss is tied to multi-style pretraining and a discriminator head that predicts font style, bringing the design closer to AC-GAN-like conditioning.

The hierarchical inpainting variants distribute adversarial and reconstruction terms across scales rather than merely adding scale-transformed losses. PyramidFill uses a hinge adversarial loss at every level, plus reconstruction loss for all levels, and perceptual and style losses for the texture generators only. The style weights increase with scale, from $L_1$ 2 to $L_1$ 3, so the highest-resolution models are most strongly penalized for texture statistics (Cao et al., 2020). The “structure first detail next” model assigns each sub-generator its own discriminator and loss,

$L_1$ 4

and aggregates them as

$L_1$ 5

with $L_1$ 6, $L_1$ 7, and $L_1$ 8 (Qu et al., 2021). The larger weight on the bottom layer reflects the model’s emphasis on coarse structural correctness.

A recurrent theme is that pyramid supervision is not restricted to adversarial scaling. Some variants pyramidize the representation, some pyramidize the loss, and some pyramidize the entire optimization problem.

4. Application domains and reported performance

The literature uses Pyramid Pix2Pix designs in domains where structure preservation is difficult under a plain single-scale conditional GAN.

For Chinese font synthesis, PEGAN frames the task as aligned image-to-image translation from a source font image $L_1$ 9 to a target font image $z$ 0, with all images resized to $z$ 1. On “Microsoft HeiTi → HuaKang POP3 Std W12,” PEGAN was compared with zi2zi. The reported quantitative results were PSNR 17.52 versus 17.45, SSIM 0.84 versus 0.83, UQI 0.27 versus 0.26, and recognition accuracy 96% versus 95%. In a visual Turing test, human classification accuracy was 65% for PEGAN and 73% for zi2zi, where lower is better because humans are more often fooled (Sun et al., 2018). The paper also states qualitatively that zi2zi often has incomplete strokes and PEGAN reduces these stroke breaks.

For pathology, the BCI benchmark contains 4870 registered HE–IHC patch pairs of size $z$ 2, derived from 51 usable HE–IHC whole-slide pairs, and covers HER2 expression levels 0, 1+, 2+, and 3+ (Liu et al., 2022). On BCI, reported results were PSNR/SSIM of 16.203/0.373 for CycleGAN, 18.654/0.419 for Pix2pix with a U-Net generator, 19.328/0.440 for Pix2pix with a ResNet generator, 19.634/0.471 for Pix2pixHD, and 21.160/0.477 for Pyramid Pix2Pix. On LLVIP, the same study reported 12.191/0.278 for Pyramid Pix2Pix, compared with 12.082/0.207 for Pix2pix with a ResNet generator and 11.156/0.228 for Pix2pixHD. The ablation further showed that adding a single extra scale loss, “pix2pix + S1,” already captured most of the gains on BCI.

For high-resolution inpainting, PyramidFill reported strong gains on CelebA-HQ and Places2. On CelebA-HQ $z$ 3 with center holes, PyramidFill achieved $z$ 4, PSNR $z$ 5, and SSIM $z$ 6, outperforming Global&Local, DeepFill v1, PConv, PEN-Net, and DeepFill v2. On CelebA-HQ $z$ 7, it achieved $z$ 8, PSNR $z$ 9, and SSIM $z_i$ 0 (Cao et al., 2020). The “structure first detail next” pyramid generator likewise reported improvements on Places2 and DIV2K at $z_i$ 1, with 0.777/20.09/0.050 on Places2 square masks and 0.840/26.74/0.034 on DIV2K square masks in SSIM/PSNR/ $z_i$ 2, and it retained an advantage on $z_i$ 3 DIV2K crops (Qu et al., 2021).

These results indicate that pyramidized conditional GANs are especially attractive when the failure mode of a plain model is structural: broken strokes, misregistered staining patterns, or high-resolution holes whose global layout and local texture cannot be reconstructed simultaneously by one generator.

A common misconception is that Pyramid Pix2Pix is synonymous with pix2pixHD. The BCI study explicitly distinguishes the two. Pix2pixHD uses multi-scale discriminators and a coarse-to-fine generator for high-resolution synthesis, whereas Pyramid Pix2Pix in that work keeps one generator and one PatchGAN discriminator and adds a Gaussian pyramid loss instead (Liu et al., 2022). The distinction is substantive: pix2pixHD pyramidizes adversarial architecture, while BCI pyramidizes supervision.

A second misconception is that every pyramid variant relies on multi-scale discriminators. PEGAN confines the pyramid mechanism to the generator, using cascaded refinement connections in the encoder and a discriminator described as Conv → LReLU → (Conv–BN–LReLU) × 3 → FC → Sigmoid, with real/fake and category outputs (Sun et al., 2018). PyramidFill adopts a PatchGAN pair $z_i$ 4 at each level, but its distinctive contribution is the decomposition of content completion and texture synthesis across resolutions rather than simply adding more discriminators (Cao et al., 2020).

A third misconception is that pyramidization presupposes a U-Net backbone. PEGAN indeed uses an encoder-decoder with mirror skip connections, but the BCI model uses ResNet-9blocks as its generator, and PFNs were proposed precisely as an alternative to the widely used encoder-decoder structure for pixel-wise prediction tasks (Liu et al., 2022, Deng et al., 2021). PFNs hold multiple information processing pathways and encode information into multiple separate small-channel features rather than a single large-channel bottleneck. This does not make PFN a Pyramid Pix2Pix model in itself, but it shows that pyramid thinking is compatible with non-U-Net backbones.

A fourth misconception is that improved image metrics imply downstream adequacy. The pathology case is a direct counterexample. Although Pyramid Pix2Pix improved PSNR and SSIM on BCI, pathologist scoring on 40 real-generated IHC pairs yielded accuracies of 37.5% and 40.0% for the two pathologists when agreement with the real counterpart’s HER2 level was used as the criterion (Liu et al., 2022). The technical translation improved, but the generated IHC remained insufficient for clinical decision-making.

6. Limitations and future directions

The limitations reported across these studies are domain-specific but structurally related. In PEGAN, the reported improvements over the Pix2Pix-like zi2zi baseline are modest, and the paper’s own discussion implies that very small training sets or extremely complex styles can still produce failures (Sun et al., 2018). In the BCI setting, the central unresolved difficulty is high HER2 expression, especially 3+ cases; the study explicitly states that all methods struggle to sharply delineate regions of strong membrane staining and that it remains very challenging to establish an accurate mapping from HE to IHC (Liu et al., 2022). In hierarchical inpainting, training cost and architectural complexity increase because multiple generators and discriminators must be optimized, sometimes sequentially, and errors from lower levels can propagate upward (Cao et al., 2020, Qu et al., 2021).

The future directions named in the cited works fall into three broad classes. The first is domain extension. PEGAN explicitly identifies edge → photo, semantic map → photo, sketch → color render, and medical imaging such as CT → MRI translation as natural directions for a generalized pyramid-enhanced Pix2Pix (Sun et al., 2018). The second is architectural extension. The same PEGAN discussion proposes combining a pyramid generator with a multi-scale or PatchGAN discriminator, while PFN suggests that alternative multi-path pyramid backbones may improve global structure without a single dominant bottleneck (Sun et al., 2018, Deng et al., 2021). The third is validation-oriented extension. The BCI work states that future pathology studies should explore the difference in HER2 evaluation between synthetic and real IHC images and examine the possibility of formulating accurate clinical treatment plans using synthetic IHC images, which requires substantially stronger validation than current metrics provide (Liu et al., 2022).

Taken together, these papers define Pyramid Pix2Pix not as one standardized model but as a class of conditional GAN designs that move Pix2Pix from single-scale translation toward explicitly multi-scale structure management. The common goal is to preserve global correspondence while maintaining local fidelity, but the means differ: pyramid-embedded encoders, Gaussian pyramid supervision, and coarse-to-fine stacks of conditional GANs represent three distinct, technically consequential realizations of the same multi-scale imperative.