---
title: 'IPT: Image Processing Transformer for Restoration'
url: https://www.emergentmind.com/topics/image-processing-transformer-ipt
type: topic
---

# IPT: Image Processing Transformer for Restoration

Searching arXiv for papers on Image Processing Transformer (IPT) and closely related follow-ups.
Image Processing Transformer (IPT) is a large, pre-trained transformer model for low-level vision tasks such as image super-resolution, denoising, deraining, and deblurring. It explicitly imports the “BERT/GPT-style” pre-train–then–fine-tune paradigm into image restoration, using a shared transformer encoder–decoder together with task-specific heads, tails, and task embeddings. In the vision-transformer literature, IPT is presented as the first “large-scale, pre-trained” transformer specifically targeting low-level image restoration and enhancement, rather than classification or detection [2012.00364][2012.12556].

## 1. Origins and problem setting

IPT was proposed for low-level image processing problems in which both the input and output are images of the same scene, but the input is corrupted by noise, downsampling, rain streaks, blur, or related degradations. This setting differs from high-level vision, where the model predicts labels or boxes, because restoration requires dense, pixel-accurate reconstruction. The original IPT formulation emphasizes two practical constraints: per-task datasets are often limited, and training and storing a dedicated deep model for every degradation is not realistic [2012.00364].

Within the transformer landscape, IPT occupies a specific position. It is not an encoder-only classification backbone in the style of ViT, and it is not an image generator in the style of early generative transformers. Instead, it is a pure transformer encoder–decoder for dense image prediction, pre-trained at scale on synthetic restoration tasks derived from ImageNet and then fine-tuned for downstream low-level tasks. A survey on visual transformers places IPT under **Low-level vision → Image enhancement** and contrasts it with both classification-oriented backbones such as ViT/DeiT and single-task restoration transformers such as TTSR [2012.12556].

A central design premise is that low-level degradations share structure across tasks. Super-resolution, denoising, and deraining all require recovery of edges, textures, and long-range consistency, even though their corruption processes differ. IPT therefore uses one shared transformer body and isolates task differences in the task-specific input and output modules plus decoder conditioning. This shared-backbone view later became a recurring template in multi-task image restoration research [2012.00364].

## 2. Architectural organization

IPT consists of a set of task-specific heads $\{H^i\}_{i=1}^{N_t}$, a shared transformer encoder–decoder body, a set of task-specific tails $\{T^i\}_{i=1}^{N_t}$, and learned task embeddings injected into the decoder [2012.00364].

Each head adapts a corrupted RGB input
\[
x \in \mathbb{R}^{3 \times H \times W}
\]
into a feature map
\[
f_H = H^i(x) \in \mathbb{R}^{C \times H \times W}, \quad C=64.
\]
In the implementation summary, the head uses one $3\times 3$ convolution from $3 \to 64$ followed by two ResBlocks, each ResBlock containing two $5\times 5$ convolutions from $64 \to 64$ [2012.00364].

The transformer encoder operates on non-overlapping patches treated as tokens. If the feature map is partitioned into patches $f_{p_i} \in \mathbb{R}^{P^2 \times C}$ with learned positional embeddings $E_{p_i}$, the token sequence is initialized as
\[
y_0 = [E_{p_1} + f_{p_1},\ E_{p_2} + f_{p_2},\ \dots,\ E_{p_N} + f_{p_N}].
\]
The encoder then applies standard transformer layers composed of LayerNorm, multi-head self-attention, feed-forward networks, and residual connections, producing encoded patch features $f_{E_i}$ [2012.00364].

The decoder mirrors the encoder but is explicitly task-aware. For a given task, a learned task embedding $E_t$ is added to the normalized decoder states when forming queries and keys. The decoder first performs task-aware self-attention over decoder states and then a cross-like attention between decoder states and the encoder output. This conditioning mechanism lets a shared body switch behavior across super-resolution, denoising, and deraining while keeping one common representation space [2012.00364].

Task-specific tails reconstruct the restored image from decoded features. For denoising and deraining, the tail is a convolutional output head that preserves spatial resolution. For super-resolution, the tail includes PixelShuffle upsampling: one PixelShuffle $\times 2$ for $\times 2$ SR, one PixelShuffle $\times 3$ for $\times 3$ SR, and two cascaded PixelShuffle $\times 2$ stages for $\times 4$ SR. The reported overall model size is about **114M parameters** with about **33G FLOPs** [2012.00364].

## 3. Pre-training, synthetic supervision, and contrastive learning

IPT is pre-trained on ImageNet with semantic labels discarded. The original training pipeline crops images into $48 \times 48$ RGB patches, yielding **>10M training patches** from **∼1.2M images**, and synthetically corrupts each clean patch to create supervised restoration pairs [2012.00364].

The pre-training degradations cover six supervised tasks. Three are super-resolution tasks using bicubic downsampling at scales $\times 2$, $\times 3$, and $\times 4$. Two are Gaussian denoising tasks with $\sigma=30$ and $\sigma=50$. One is a synthetic deraining task produced by adding rain streaks. The supervised loss is an $L_1$ reconstruction objective,
\[
\mathcal{L}_{\text{supervised}}
= \sum_{i=1}^{N_t} L_1\big(\text{IPT}(I^i_{\text{corrupted}}),\ I_{\text{clean}}\big).
\]
During pre-training, one task is randomly chosen per iteration because of memory constraints, while the shared body is updated across all tasks over time [2012.00364].

A second component is patch-level contrastive learning. For decoder patch features from the same image, IPT encourages higher cosine similarity than for patches from different images in the minibatch. The contrastive objective is aggregated over all patch pairs and combined with the supervised reconstruction term:
\[
\mathcal{L}_{\text{IPT}} = \lambda \cdot \mathcal{L}_{\text{contrastive}} + \mathcal{L}_{\text{supervised}}.
\]
This addition is intended to make patch representations more task-agnostic and to improve transfer to unseen degradations. In the reported Set5 $\times 2$ SR ablation, $\lambda=0$ gives **38.27 dB**, while $\lambda=0.1$ gives **38.37 dB**; both smaller and larger weights are slightly worse [2012.00364].

The optimization recipe is resource-intensive. The reported pre-training uses **32× Tesla V100 GPUs**, **300 epochs**, Adam with $\beta_1=0.9$, $\beta_2=0.999$, an initial learning rate of $5 \times 10^{-5}$ decayed to $2 \times 10^{-5}$ after 200 epochs, and a batch size of **256** patches. Fine-tuning keeps the relevant head, tail, and shared body, drops unused heads and tails, and runs for **30 epochs** with learning rate $2 \times 10^{-5}$ [2012.00364].

The embedding design is not incidental. Learned positional embeddings outperform fixed or removed positional embeddings by about **0.2–0.3 dB**, and removing task embeddings drops performance by about **0.1–0.5 dB**. Multi-task pre-training performs similarly to single-task pre-training on tasks seen during pre-training, but on new tasks or unseen degradations it is better by about **0.3 dB**, indicating more universal representations [2012.00364].

## 4. Reported empirical behavior

The original IPT study reports strong results across super-resolution, denoising, deraining, and deblurring benchmarks, with particularly large gains on structured urban scenes and on degradations not explicitly seen during pre-training [2012.00364].

For super-resolution, IPT is evaluated on Set5, Set14, B100, and Urban100 at scales $\times 2$, $\times 3$, and $\times 4$. At $\times 2$, IPT reports **38.37 / 34.43 / 32.48 / 33.76 dB** on Set5, Set14, B100, and Urban100, respectively. At $\times 3$, the corresponding numbers are **34.81 / 30.85 / 29.38 / 29.49 dB**. At $\times 4$, IPT reports **32.64 / 29.01 / 27.82 / 27.26 dB**. The largest gains are on Urban100, where the improvement over prior methods is about **0.4 dB** at several scales, consistent with the claim that global patch interactions help recover repetitive structures and fine line patterns [2012.00364].

For color denoising at noise levels seen in pre-training, IPT reports **30.75 dB** on BSD68 and **32.00 dB** on Urban100 at $\sigma=30$, and **28.39 dB** on BSD68 and **29.71 dB** on Urban100 at $\sigma=50$. The model is also evaluated at unseen noise levels without explicit fine-tuning for those levels. At $\sigma=10$, it reports **36.53 dB** on BSD68 and **36.99 dB** on Urban100; at $\sigma=70$, it reports **26.92 dB** on BSD68 and **27.90 dB** on Urban100. This generalization result is presented as evidence that transformer pre-training plus contrastive patch learning improves robustness beyond the exact degradations used during pre-training [2012.00364].

For deraining on Rain100L, IPT reports **41.62 dB** PSNR and **0.9880** SSIM, compared with **40.00 dB** and **0.9860** for RCDNet. For deblurring on GoPro, IPT reports **32.58 dB**, and **IPT+** with self-ensemble reports **32.91 dB** [2012.00364].

The ablation on pre-training scale is equally consequential. When the pre-training data is absent or small, specifically **<60%**, CNNs outperform transformers. When the pre-training scale reaches **≥80–100%**, IPT surpasses CNNs clearly. This reported crossover is one of the clearest empirical statements about IPT’s operating regime: the model is highly effective, but its advantages depend on large-scale pre-training [2012.00364].

## 5. Successors, variants, and related adaptations

Several later works treat IPT as a reference architecture and modify different parts of its design space. “IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions” explicitly presents itself as a successor to IPT and argues that full global self-attention in IPT has two major limitations: quadratic complexity in resolution and lack of explicit efficient local modeling. IPT-V2 replaces uniform global attention with **Focal Context Self-Attention (FCSA)** for local context and **Global Grid Self-Attention (GGSA)** for global dependencies, and adds a structurally re-parameterized feed-forward module. Reported restoration results include **40.05 dB / 0.961 SSIM** on SIDD, **40.09 dB / 0.957 SSIM** on DND, **33.92 dB / 0.968 SSIM** on GoPro, and **31.73 dB / 0.911 SSIM** on Rain100H. The same architecture is also used as a latent diffusion backbone and at **CFG = 1.5** and **800k** iterations reports **FID = 7.71**, compared with **33.56** for DiT-S/2 at similar computational scale [2404.00633].

“LIPT: Latency-aware Image Processing Transformer” addresses a different weakness: practical inference speed. Rather than optimizing only FLOPs or parameter count, it redesigns the backbone around a **1:3** ratio of self-attention to convolution blocks, introduces **Non-Volatile Sparse Masking Self-Attention (NVSM-SA)**, and replaces heavy MLPs with a **High-frequency Reparameterization Module (HRM)**. In the reported super-resolution setting on **720p** images, **LIPT-Tiny** reaches **29 ms** GPU latency at $\times 2$, **12 ms** at $\times 3$, and **8 ms** at $\times 4$, and is described as the first real-time transformer on GPU for these SR settings. **LIPT-Small** reports, for example, **32.87 dB** on Urban100 $\times 2$ at **99 ms** GPU latency, compared with **32.76 dB** for both ELAN-Light and SwinIR-Light at substantially higher latency [2404.06075].

“Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation” broadens the task range. Instead of feature adaptation, it modulates selected task-sensitive weights of an IPT-V2 backbone by adding low-rank task-specific biases and training the shared backbone and biases synchronously. The target task set includes Gaussian denoising at $\sigma \in \{15,25,50\}$, motion deblurring, deraining, dehazing, and desnowing. The paper reports that **Instruct-IPT achieves the highest PSNR in all tasks** among the compared all-in-one methods. It also reports two model sizes: **Instruct-IPT-Tiny** with **2.60M params** and **0.95G FLOPs**, and **Instruct-IPT** with **26.49M params** and **10.01G FLOPs**. An additional extension lets the model receive human prompts through a text-encoder interface [2407.00676].

IPT has also been adapted outside its original benchmark suite. In underwater image enhancement, a pre-trained denoising IPT model is fine-tuned on the **UFO-120** dataset using the denoising head, shared encoder–decoder, and denoising tail. On the reported test split, the IPT-based method achieves **23.14 dB** PSNR and **0.90** SSIM. The paper emphasizes the best SSIM among compared methods but also notes a smoothing tendency and a runtime of about **12–13 seconds** for a **320×240** image on a Quadro P5000, making it unsuitable for real-time onboard robotic use [2204.04199].

A separate neighboring line appears in learned image compression. “Transformer-based Image Compression” places IPT in the broader family of transformer-based low-level vision methods but distinguishes compression from restoration. IPT minimizes reconstruction error without explicit rate modeling, whereas TIC is built in a VAE compression framework with main and hyper encoder–decoder pairs, a rate–distortion objective
\[
L = R(\hat{y}) + \lambda D(x,\hat{x}),
\]
and a causal attention module for entropy modeling. This comparison is useful because it marks the boundary between an image restoration transformer and a rate-controlled codec [2111.06707].

## 6. Limitations, terminology, and historical significance

IPT’s strengths are paired with clear assumptions. The pre-training degradations are synthetic: bicubic downsampling, Gaussian noise, and synthetic rain. The original discussion states that real-world degradations can deviate significantly and that performance on truly real, unknown degradations is not fully characterized. The model is also data-hungry and compute-intensive: about **114M parameters**, about **33G FLOPs**, pre-training on **32 V100 GPUs** for **300 epochs**, and patch-based training on **48×48** crops with sliding-window testing using **10-pixel overlap** [2012.00364].

These limitations align with broader transformer assessments in low-level vision. The survey literature emphasizes that transformers rely heavily on massive datasets because of weaker inductive bias than CNNs and that patch-sequence processing is computationally expensive, especially for high-resolution restoration. IPT is therefore both a proof of transformer capability in restoration and a demonstration of the cost required to unlock that capability [2012.12556].

The acronym itself also requires clarification. In the restoration literature, **IPT** refers to **Image Processing Transformer**. However, in “Determining Sequence of Image Processing Technique (IPT) to Detect Adversarial Attacks,” **IPT** and **IPTS** refer instead to classical **Image Processing Technique(s)** and their genetic-algorithm-searched sequences for adversarial-input detection. That work is not about transformer architectures [2007.00337].

Historically, IPT is best understood as a transfer of large-scale transformer pre-training from NLP and high-level vision into image restoration. The survey literature treats it as evidence that a unified, multi-task transformer for low-level vision is realistic, while later architectures such as IPT-V2, LIPT, and Instruct-IPT can be read as successive attempts to relax IPT’s original constraints: quadratic attention cost, latency, and limited task heterogeneity [2012.12556].

Source: https://www.emergentmind.com/topics/image-processing-transformer-ipt