IPT: Image Processing Transformer for Restoration

• IPT is a transformer model pre-trained on synthetic degradations to restore images by addressing tasks like super-resolution, denoising, deraining, and deblurring.
• The architecture features a shared encoder-decoder with task-specific heads and tails, enabling efficient multi-task processing for dense, pixel-accurate reconstruction.
• Pre-training on ImageNet with contrastive learning and synthetic corruptions allows IPT to deliver state-of-the-art performance on diverse low-level vision benchmarks.

Searching arXiv for papers on Image Processing Transformer (IPT) and closely related follow-ups. Image Processing Transformer (IPT) is a large, pre-trained transformer model for low-level vision tasks such as image super-resolution, denoising, deraining, and deblurring. It explicitly imports the “BERT/GPT-style” pre-train–then–fine-tune paradigm into image restoration, using a shared transformer encoder–decoder together with task-specific heads, tails, and task embeddings. In the vision-transformer literature, IPT is presented as the first “large-scale, pre-trained” transformer specifically targeting low-level image restoration and enhancement, rather than classification or detection (Chen et al., 2020, Han et al., 2020).

1. Origins and problem setting

IPT was proposed for low-level image processing problems in which both the input and output are images of the same scene, but the input is corrupted by noise, downsampling, rain streaks, blur, or related degradations. This setting differs from high-level vision, where the model predicts labels or boxes, because restoration requires dense, pixel-accurate reconstruction. The original IPT formulation emphasizes two practical constraints: per-task datasets are often limited, and training and storing a dedicated deep model for every degradation is not realistic (Chen et al., 2020).

Within the transformer landscape, IPT occupies a specific position. It is not an encoder-only classification backbone in the style of ViT, and it is not an image generator in the style of early generative transformers. Instead, it is a pure transformer encoder–decoder for dense image prediction, pre-trained at scale on synthetic restoration tasks derived from ImageNet and then fine-tuned for downstream low-level tasks. A survey on visual transformers places IPT under Low-level vision → Image enhancement and contrasts it with both classification-oriented backbones such as ViT/DeiT and single-task restoration transformers such as TTSR (Han et al., 2020).

A central design premise is that low-level degradations share structure across tasks. Super-resolution, denoising, and deraining all require recovery of edges, textures, and long-range consistency, even though their corruption processes differ. IPT therefore uses one shared transformer body and isolates task differences in the task-specific input and output modules plus decoder conditioning. This shared-backbone view later became a recurring template in multi-task image restoration research (Chen et al., 2020).

2. Architectural organization

IPT consists of a set of task-specific heads $\{H^i\}_{i=1}^{N_t}$, a shared transformer encoder–decoder body, a set of task-specific tails $\{T^i\}_{i=1}^{N_t}$, and learned task embeddings injected into the decoder (Chen et al., 2020).

Each head adapts a corrupted RGB input

$x \in \mathbb{R}^{3 \times H \times W}$

into a feature map

$$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$

In the implementation summary, the head uses one $3\times 3$ convolution from $3 \to 64$ followed by two ResBlocks, each ResBlock containing two $5\times 5$ convolutions from $64 \to 64$ (Chen et al., 2020).

The transformer encoder operates on non-overlapping patches treated as tokens. If the feature map is partitioned into patches $f_{p_i} \in \mathbb{R}^{P^2 \times C}$ with learned positional embeddings $E_{p_i}$, the token sequence is initialized as

$\{T^i\}_{i=1}^{N_t}$0

The encoder then applies standard transformer layers composed of LayerNorm, multi-head self-attention, feed-forward networks, and residual connections, producing encoded patch features $\{T^i\}_{i=1}^{N_t}$1 (Chen et al., 2020).

The decoder mirrors the encoder but is explicitly task-aware. For a given task, a learned task embedding $\{T^i\}_{i=1}^{N_t}$2 is added to the normalized decoder states when forming queries and keys. The decoder first performs task-aware self-attention over decoder states and then a cross-like attention between decoder states and the encoder output. This conditioning mechanism lets a shared body switch behavior across super-resolution, denoising, and deraining while keeping one common representation space (Chen et al., 2020).

Task-specific tails reconstruct the restored image from decoded features. For denoising and deraining, the tail is a convolutional output head that preserves spatial resolution. For super-resolution, the tail includes PixelShuffle upsampling: one PixelShuffle $\{T^i\}_{i=1}^{N_t}$3 for $\{T^i\}_{i=1}^{N_t}$4 SR, one PixelShuffle $\{T^i\}_{i=1}^{N_t}$5 for $\{T^i\}_{i=1}^{N_t}$6 SR, and two cascaded PixelShuffle $\{T^i\}_{i=1}^{N_t}$7 stages for $\{T^i\}_{i=1}^{N_t}$8 SR. The reported overall model size is about 114M parameters with about 33G FLOPs (Chen et al., 2020).

3. Pre-training, synthetic supervision, and contrastive learning

IPT is pre-trained on ImageNet with semantic labels discarded. The original training pipeline crops images into $\{T^i\}_{i=1}^{N_t}$9 RGB patches, yielding >10M training patches from ∼1.2M images, and synthetically corrupts each clean patch to create supervised restoration pairs (Chen et al., 2020).

The pre-training degradations cover six supervised tasks. Three are super-resolution tasks using bicubic downsampling at scales $x \in \mathbb{R}^{3 \times H \times W}$0, $x \in \mathbb{R}^{3 \times H \times W}$1, and $x \in \mathbb{R}^{3 \times H \times W}$2. Two are Gaussian denoising tasks with $x \in \mathbb{R}^{3 \times H \times W}$3 and $x \in \mathbb{R}^{3 \times H \times W}$4. One is a synthetic deraining task produced by adding rain streaks. The supervised loss is an $x \in \mathbb{R}^{3 \times H \times W}$5 reconstruction objective,

$x \in \mathbb{R}^{3 \times H \times W}$6

During pre-training, one task is randomly chosen per iteration because of memory constraints, while the shared body is updated across all tasks over time (Chen et al., 2020).

A second component is patch-level contrastive learning. For decoder patch features from the same image, IPT encourages higher cosine similarity than for patches from different images in the minibatch. The contrastive objective is aggregated over all patch pairs and combined with the supervised reconstruction term: $x \in \mathbb{R}^{3 \times H \times W}$7 This addition is intended to make patch representations more task-agnostic and to improve transfer to unseen degradations. In the reported Set5 $x \in \mathbb{R}^{3 \times H \times W}$8 SR ablation, $x \in \mathbb{R}^{3 \times H \times W}$9 gives 38.27 dB, while $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$0 gives 38.37 dB; both smaller and larger weights are slightly worse (Chen et al., 2020).

The optimization recipe is resource-intensive. The reported pre-training uses 32× Tesla V100 GPUs, 300 epochs, Adam with $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$1, $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$2, an initial learning rate of $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$3 decayed to $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$4 after 200 epochs, and a batch size of 256 patches. Fine-tuning keeps the relevant head, tail, and shared body, drops unused heads and tails, and runs for 30 epochs with learning rate $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$5 (Chen et al., 2020).

The embedding design is not incidental. Learned positional embeddings outperform fixed or removed positional embeddings by about 0.2–0.3 dB, and removing task embeddings drops performance by about 0.1–0.5 dB. Multi-task pre-training performs similarly to single-task pre-training on tasks seen during pre-training, but on new tasks or unseen degradations it is better by about 0.3 dB, indicating more universal representations (Chen et al., 2020).

4. Reported empirical behavior

The original IPT study reports strong results across super-resolution, denoising, deraining, and deblurring benchmarks, with particularly large gains on structured urban scenes and on degradations not explicitly seen during pre-training (Chen et al., 2020).

For super-resolution, IPT is evaluated on Set5, Set14, B100, and Urban100 at scales $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$6, $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$7, and $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$8. At $$f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.$$9, IPT reports 38.37 / 34.43 / 32.48 / 33.76 dB on Set5, Set14, B100, and Urban100, respectively. At $3\times 3$0, the corresponding numbers are 34.81 / 30.85 / 29.38 / 29.49 dB. At $3\times 3$1, IPT reports 32.64 / 29.01 / 27.82 / 27.26 dB. The largest gains are on Urban100, where the improvement over prior methods is about 0.4 dB at several scales, consistent with the claim that global patch interactions help recover repetitive structures and fine line patterns (Chen et al., 2020).

For color denoising at noise levels seen in pre-training, IPT reports 30.75 dB on BSD68 and 32.00 dB on Urban100 at $3\times 3$2, and 28.39 dB on BSD68 and 29.71 dB on Urban100 at $3\times 3$3. The model is also evaluated at unseen noise levels without explicit fine-tuning for those levels. At $3\times 3$4, it reports 36.53 dB on BSD68 and 36.99 dB on Urban100; at $3\times 3$5, it reports 26.92 dB on BSD68 and 27.90 dB on Urban100. This generalization result is presented as evidence that transformer pre-training plus contrastive patch learning improves robustness beyond the exact degradations used during pre-training (Chen et al., 2020).

For deraining on Rain100L, IPT reports 41.62 dB PSNR and 0.9880 SSIM, compared with 40.00 dB and 0.9860 for RCDNet. For deblurring on GoPro, IPT reports 32.58 dB, and IPT+ with self-ensemble reports 32.91 dB (Chen et al., 2020).

The ablation on pre-training scale is equally consequential. When the pre-training data is absent or small, specifically <60%, CNNs outperform transformers. When the pre-training scale reaches ≥80–100%, IPT surpasses CNNs clearly. This reported crossover is one of the clearest empirical statements about IPT’s operating regime: the model is highly effective, but its advantages depend on large-scale pre-training (Chen et al., 2020).

5. Successors, variants, and related adaptations

Several later works treat IPT as a reference architecture and modify different parts of its design space. “IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions” explicitly presents itself as a successor to IPT and argues that full global self-attention in IPT has two major limitations: quadratic complexity in resolution and lack of explicit efficient local modeling. IPT-V2 replaces uniform global attention with Focal Context Self-Attention (FCSA) for local context and Global Grid Self-Attention (GGSA) for global dependencies, and adds a structurally re-parameterized feed-forward module. Reported restoration results include 40.05 dB / 0.961 SSIM on SIDD, 40.09 dB / 0.957 SSIM on DND, 33.92 dB / 0.968 SSIM on GoPro, and 31.73 dB / 0.911 SSIM on Rain100H. The same architecture is also used as a latent diffusion backbone and at CFG = 1.5 and 800k iterations reports FID = 7.71, compared with 33.56 for DiT-S/2 at similar computational scale (Tu et al., 2024).

“LIPT: Latency-aware Image Processing Transformer” addresses a different weakness: practical inference speed. Rather than optimizing only FLOPs or parameter count, it redesigns the backbone around a 1:3 ratio of self-attention to convolution blocks, introduces Non-Volatile Sparse Masking Self-Attention (NVSM-SA), and replaces heavy MLPs with a High-frequency Reparameterization Module (HRM). In the reported super-resolution setting on 720p images, LIPT-Tiny reaches 29 ms GPU latency at $3\times 3$6, 12 ms at $3\times 3$7, and 8 ms at $3\times 3$8, and is described as the first real-time transformer on GPU for these SR settings. LIPT-Small reports, for example, 32.87 dB on Urban100 $3\times 3$9 at 99 ms GPU latency, compared with 32.76 dB for both ELAN-Light and SwinIR-Light at substantially higher latency (Qiao et al., 2024).

“Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation” broadens the task range. Instead of feature adaptation, it modulates selected task-sensitive weights of an IPT-V2 backbone by adding low-rank task-specific biases and training the shared backbone and biases synchronously. The target task set includes Gaussian denoising at $3 \to 64$0, motion deblurring, deraining, dehazing, and desnowing. The paper reports that Instruct-IPT achieves the highest PSNR in all tasks among the compared all-in-one methods. It also reports two model sizes: Instruct-IPT-Tiny with 2.60M params and 0.95G FLOPs, and Instruct-IPT with 26.49M params and 10.01G FLOPs. An additional extension lets the model receive human prompts through a text-encoder interface (Tian et al., 2024).

IPT has also been adapted outside its original benchmark suite. In underwater image enhancement, a pre-trained denoising IPT model is fine-tuned on the UFO-120 dataset using the denoising head, shared encoder–decoder, and denoising tail. On the reported test split, the IPT-based method achieves 23.14 dB PSNR and 0.90 SSIM. The paper emphasizes the best SSIM among compared methods but also notes a smoothing tendency and a runtime of about 12–13 seconds for a 320×240 image on a Quadro P5000, making it unsuitable for real-time onboard robotic use (Boudiaf et al., 2022).

A separate neighboring line appears in learned image compression. “Transformer-based Image Compression” places IPT in the broader family of transformer-based low-level vision methods but distinguishes compression from restoration. IPT minimizes reconstruction error without explicit rate modeling, whereas TIC is built in a VAE compression framework with main and hyper encoder–decoder pairs, a rate–distortion objective

$3 \to 64$1

and a causal attention module for entropy modeling. This comparison is useful because it marks the boundary between an image restoration transformer and a rate-controlled codec (Lu et al., 2021).

6. Limitations, terminology, and historical significance

IPT’s strengths are paired with clear assumptions. The pre-training degradations are synthetic: bicubic downsampling, Gaussian noise, and synthetic rain. The original discussion states that real-world degradations can deviate significantly and that performance on truly real, unknown degradations is not fully characterized. The model is also data-hungry and compute-intensive: about 114M parameters, about 33G FLOPs, pre-training on 32 V100 GPUs for 300 epochs, and patch-based training on 48×48 crops with sliding-window testing using 10-pixel overlap (Chen et al., 2020).

These limitations align with broader transformer assessments in low-level vision. The survey literature emphasizes that transformers rely heavily on massive datasets because of weaker inductive bias than CNNs and that patch-sequence processing is computationally expensive, especially for high-resolution restoration. IPT is therefore both a proof of transformer capability in restoration and a demonstration of the cost required to unlock that capability (Han et al., 2020).

The acronym itself also requires clarification. In the restoration literature, IPT refers to Image Processing Transformer. However, in “Determining Sequence of Image Processing Technique (IPT) to Detect Adversarial Attacks,” IPT and IPTS refer instead to classical Image Processing Technique(s) and their genetic-algorithm-searched sequences for adversarial-input detection. That work is not about transformer architectures (Gupta et al., 2020).

Historically, IPT is best understood as a transfer of large-scale transformer pre-training from NLP and high-level vision into image restoration. The survey literature treats it as evidence that a unified, multi-task transformer for low-level vision is realistic, while later architectures such as IPT-V2, LIPT, and Instruct-IPT can be read as successive attempts to relax IPT’s original constraints: quadratic attention cost, latency, and limited task heterogeneity (Han et al., 2020).