Laplacian Pyramid Translation Network (LPTN)

• LPTN is an image-to-image translation framework that leverages Laplacian pyramid decomposition to separate global attribute edits from fine detail refinement.
• It uses a lightweight low-frequency translator and progressive masking for high-frequency refinement, achieving state-of-the-art photorealism and computational efficiency.
• Evaluations on benchmarks like MIT-Adobe FiveK show real-time 4K translation with improved PSNR and SSIM, demonstrating its practical impact in high-res image processing.

The Laplacian Pyramid Translation Network (LPTN) is an image-to-image translation framework designed for high-resolution, photorealistic transformation with real-time inference, particularly targeting computational efficiency at ultra-high resolutions. LPTN achieves this by decomposing the input image into low-frequency and high-frequency components using a closed-form Laplacian pyramid. The method applies a lightweight translation network to the low-frequency component for global attribute changes (e.g., illumination, color), and employs a progressive masking strategy to selectively refine high-frequency components, thus preserving detailed content. This design avoids the expensive computation associated with directly processing high-resolution feature maps and enables 4K image translation in real-time on a single commodity GPU (Liang et al., 2021).

1. Laplacian Pyramid Decomposition and Reconstruction

LPTN builds a depth-$N$ Laplacian pyramid to hierarchically decompose the input $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$ as follows:

• Low-pass downsampling: For level $l$,

$L^{l+1} = \mathrm{Down}(L^l)$

where $\mathrm{Down}(\cdot)$ is typically Gaussian filtering followed by 2× subsampling.

• Residual (high-frequency) extraction:

$H^l = L^l - \mathrm{Up}(L^{l+1})$

with $\mathrm{Up}(\cdot)$ bilinear or transposed upsampling.

This yields $L^N$ (the coarsest low-frequency image) and $N$ high-frequency bands $H^0, ..., H^{N-1}$ of decreasing resolution. Since the pyramid representation is lossless, reconstruction is given by:

$x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$0

where $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$1 denotes upsampling (and optional convolution) at level $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$2.

2. Network Architecture

The LPTN model explicitly decouples transformation tasks between scales:

(a) Low-Frequency Translator

• Input: $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$3 of shape $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$4.
• Module:
  • Expand: $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$5 convolution, $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$6 (with $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$7 in practice).
  • Five Residual Blocks: Each with
  • $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$8 conv, $x \equiv L^0 \in \mathbb{R}^{H \times W \times C}$9, LeakyReLU
  • $l$0 conv, $l$1, LeakyReLU
  • Instance-norm precedes each block.
  • Project: $l$2 conv, $l$3; Tanh activation.
  • Output: A delta is predicted;

  $l$4

  where $l$5 is the stack described above.

Parameter count for this module is $l$6.

(b) High-Frequency Refinement via Progressive Masking

Refinement is performed for each Laplacian high-frequency band using learned per-pixel masks:

• Stage 1 (band $l$7):
  • Upsample $l$8 and $l$9 to match $L^{l+1} = \mathrm{Down}(L^l)$0's size.
  • Concatenate $L^{l+1} = \mathrm{Down}(L^l)$1.
  • Pass through a small CNN to produce a per-pixel mask $L^{l+1} = \mathrm{Down}(L^l)$2.
  • Refine:

  $L^{l+1} = \mathrm{Down}(L^l)$3

• Subsequent stages (bands $L^{l+1} = \mathrm{Down}(L^l)$4 to $L^{l+1} = \mathrm{Down}(L^l)$5):
  • Bilinearly upsample the previous mask $L^{l+1} = \mathrm{Down}(L^l)$6 by a factor of 2 to initialize $L^{l+1} = \mathrm{Down}(L^l)$7.
  • Refine $L^{l+1} = \mathrm{Down}(L^l)$8 with a two-layer $L^{l+1} = \mathrm{Down}(L^l)$9 conv block (LeakyReLU).
  • Compute

  $\mathrm{Down}(\cdot)$0

After all bands are refined, reconstruct $\mathrm{Down}(\cdot)$1 using $\mathrm{Down}(\cdot)$2 and $\mathrm{Down}(\cdot)$3 analogously to the original Laplacian reconstruction.

3. Training Objectives and Optimization

LPTN utilizes both pixel-level and adversarial objectives:

• Reconstruction loss:

$\mathrm{Down}(\cdot)$4

• Adversarial loss: Least-Squares GAN (LSGAN)-style with three-scale Patch discriminator $\mathrm{Down}(\cdot)$5.
  • Generator objective:

  $\mathrm{Down}(\cdot)$6 - Discriminator objective:

  $\mathrm{Down}(\cdot)$7

• Joint objective:

$\mathrm{Down}(\cdot)$8

with $\mathrm{Down}(\cdot)$9.

No perceptual loss was used in the published results.

4. Computational Efficiency and Inference Performance

LPTN's architecture strategically exploits the sparsity of the Laplacian representation for computational savings:

• Low-frequency translator operates on $H^l = L^l - \mathrm{Up}(L^{l+1})$0 with cost reduced by a factor $H^l = L^l - \mathrm{Up}(L^{l+1})$1 compared to full-resolution.
• High-frequency refinement only involves small-resolution maps, with further reduction per band.
• Runtime benchmarks on an 11 GB GPU (RTX 2080Ti):

Resolution	N	Time (ms)	FPS
480p	3	3	333
1080p	4	7	143
2K	4	15	67
4K	5	16	62

Decomposition and reconstruction together require less than 2 ms per 4K image. LPTN thus sustains real-time performance even at 4K.

5. Quantitative and Qualitative Evaluation

LPTN was evaluated on image translation and photo-retouching benchmarks, notably the MIT-Adobe FiveK task. Table results for 1080p resolution:

Method	PSNR	SSIM
CycleGAN	20.86	0.846
UNIT	19.32	0.802
MUNIT	20.28	0.815
White-Box	21.26	0.872
DPE	21.94	0.885
LPTN (N=3)	22.09	0.883

LPTN matches or outperforms prior image-to-image translation (I2I) baselines and is comparable to state-of-the-art specifically designed photo retouching methods. Qualitatively, LPTN achieves faithful global edits (illumination, color) while preserving edge-level detail. In user studies (20 participants, 20 images), LPTN achieved highest photorealism over 75% of the time and style faithfulness approximately 50% of the time for day-to-night translation.

6. Implementation and Reproducibility

LPTN is available as open-source software (PyTorch 1.x, Python 3.7) with pre-trained models and scripts for replication and benchmarking. The key hyperparameters are Adam ($H^l = L^l - \mathrm{Up}(L^{l+1})$2), learning rate $H^l = L^l - \mathrm{Up}(L^{l+1})$3, batch size 1, and adversarial weight $H^l = L^l - \mathrm{Up}(L^{l+1})$4. All results and performance metrics were obtained on a single 11 GB GPU. Source and documentation are provided at https://github.com/csjliang/LPTN.

---

LPTN leverages closed-form Laplacian pyramid decomposition to isolate global attribute transformations in the low-frequency domain and detail preservation in the high-frequency domain, applying specialized sub-networks for each. This dual-path strategy enables photorealistic translation of ultra-high-resolution images with a modest computational footprint, providing real-time performance and state-of-the-art translation fidelity on demanding tasks (Liang et al., 2021).