ML-CrAIST: Transformer for Image Super-Resolution

• ML-CrAIST is a transformer-based architecture for single-image super-resolution that integrates multi-scale frequency decomposition with cross-attention mechanisms.
• It employs spatial–channel self-attention and attention-based fusion to effectively enhance high-frequency details like edges and textures.
• Quantitative results show improved PSNR and SSIM compared to state-of-the-art methods, confirming its efficacy in fine detail restoration.

ML-CrAIST (Multi-scale Low-high Frequency Information-based Cross Attention with Image Super-resolving Transformer) is a transformer-based architecture specifically designed for single-image super-resolution (SR). ML-CrAIST addresses the longstanding challenge in SR of effectively reconstructing high-frequency image regions, such as edges and textures, which are typically underrepresented in conventional models and present significant complexity relative to low-frequency, smooth regions. The architecture integrates multi-scale frequency analysis, spatial-channel self-attention, attention-based fusion, and cross-attention mechanisms to explicitly model and fuse information across spatial, channel, and frequency domains, resulting in superior restoration of fine details and improved quantitative metrics compared to prior state-of-the-art approaches (Pramanick et al., 2024).

1. Problem Formulation and Motivation

Single-image super-resolution seeks a mapping from an observed low-resolution (LR) image $I_{LR}$ to its high-resolution (HR) counterpart $I_{HR}$. Given the ill-posedness of SR—where many possible $I_{HR}$ can correspond to the same $I_{LR}$—designing models that recover realistic and detailed HR outputs is a significant challenge. Traditional convolutional architectures (SRCNN, EDSR, RCAN) are typically limited to capturing local context and frequently result in blurred high-frequency regions. Transformer-based models such as SwinIR, Restormer, and OmniSR improved global context aggregation via self-attention but generally do not explicitly treat frequency components differently.

High-frequency (HF) components—edges, textures, and fine structures—are empirically harder to reconstruct than low-frequency (LF) components due to their complex spatial variability and local sparsity. Moreover, image structure manifests across multiple spatial scales, making multi-scale analysis critical for detail preservation. ML-CrAIST therefore integrates multi-scale frequency decomposition using the 2D discrete wavelet transform (2dDWT), spatial–channel self-attention, attention-based nonlinear fusion of frequency bands, and cross-frequency-domain attention to directly address these challenges (Pramanick et al., 2024).

2. Network Architecture and Information Flow

ML-CrAIST employs a multi-stage, multi-branch architecture combining spatial-transformer and frequency-decomposition paths. The core pipeline is summarized as follows:

1. Input Feature Extraction: The LR image $I_{LR} \in \mathbb{R}^{H\times W\times3}$ undergoes initial $3\times3$ convolution to produce shallow features $f_0$.
2. Spatial–Channel Transformer Path: $f_0$ traverses $N=5$ stacked Spatial-Channel Attention Transformer Blocks (SCATBs), yielding deep features $f_d$.
3. Two-Scale Frequency Path:
   • First-level decomposition: $I_{LR}$ is decomposed into $$\{\mathrm{LL}_1,\mathrm{LH}_1,\mathrm{HL}_1,\mathrm{HH}_1\}$$ via 2dDWT.
   • High-frequency fusion: The HF sub-bands $\{\mathrm{LH}_1,\mathrm{HL}_1,\mathrm{HH}_1\}$ are fused with an Attention-Based Fusion Block (AFB) to form $f^1_f$.
   • Low-frequency refinement: The LF sub-band $\mathrm{LL}_1$ undergoes $N$ SCATBs to yield $f^1_s$.
   • Cross-attention fusion: Cross-Attention Block (CAB) fuses $f^1_s$ and $f^1_f$ to obtain $f^1_{sf}$.
   • Second-level decomposition: $\mathrm{LL}_1$ is recursively decomposed and processed analogously, resulting in $f^2_{sf}$.
4. Multi-scale Feature Aggregation: $f^2_{sf}$ is upsampled and cross-attended with $f^1_{sf}$; the result is upsampled and cross-attended with $f_d$ to yield the final feature $f^0_{sf}$.
5. Reconstruction: $f^0_{sf}$ is projected with $3\times3$ convolution, upscaled by PixelShuffle, and combined with a bicubic-upsampled skip of $I_{LR}$ to form the final HR output.

This design enables joint modeling of multi-scale, multi-frequency, and spatial–channel information, significantly enhancing detail fidelity in SR outputs (Pramanick et al., 2024).

3. Attention Mechanisms and Mathematical Structure

ML-CrAIST utilizes advanced attention operations within and between domains:

• Spatial–Channel Self-Attention (SCATB): Each SCATB executes spatial and channel self-attention sequentially:
  • Spatial attention: Features are reshaped into spatial sequences. Attention is applied via the softmax of the query-key dot-product, resulting in spatially-aware aggregation.
  • Channel attention: Features are reshaped along the channel dimension and attended similarly via softmax, aggregating inter-channel relationships.
  • The sum of these attentions is further processed via an Enhanced Spatial Attention (ESA) block for locality enhancement.
• Attention-Based Fusion Block (AFB): Nonlinear fusion of the three HF sub-bands at each wavelet scale, with the attention map computed from LH vs. HL and reweighted HH components.
• Cross-Attention Block (CAB): For features $F'$ (query) and $F''$ (key, value), CAB computes cross-domain associations, merges the attended values via a $1\times1$ convolution, and adds the result to the residual $F'$.

This coordinated framework enables effective interaction between LF and HF representations at each scale, as well as cross-scale aggregation.

4. Training Setup and Implementation Details

ML-CrAIST is trained end-to-end with an $\ell_1$ loss between predicted and ground-truth HR images, exclusively in the Y channel of YCbCr color space:

$$\mathcal{L}_1 = \frac{1}{M}\sum_{n=1}^M \left\| I_{HR}^{(n)} - I_{HR}^{g, (n)} \right\|_1.$$

No additional perceptual or adversarial losses are utilized.

Key training and implementation specifications include:

• SCATB depth: $N=5$ per path.
• Feature dimension: $c=64$ ($c=48$ for "light" variant).
• Attention heads: 4; local window size: 8.
• Training data: DIV2K HR images downsampled with bicubic kernel; augmentation via random flips and rotations.
• Crop size: $64\times64$; batch size: 32; total iterations: 1,000,000.
• Optimizer: Adam with initial learning rate $10^{-4}$, halved every 200k iterations.
• Hardware: NVIDIA V100, PyTorch framework.

5. Quantitative and Qualitative Results

ML-CrAIST demonstrates state-of-the-art performance across standard SR benchmarks. The following results (PSNR, dB) correspond to $4\times$ upscaling, with best results in bold:

Method	Set5	Set14	B100	Urban100	Manga109
SwinIR	32.44	28.77	27.69	26.47	30.92
OmniSR	32.49	28.78	27.71	26.64	31.02
Ours-Li	32.15	28.40	27.73	26.53	31.11
Ours	32.36	28.53	27.78	26.68	31.17

On Manga109 $4\times$, ML-CrAIST improves peak signal-to-noise ratio (PSNR) by +0.15 dB (+0.0029 SSIM) over OmniSR. Qualitative illustrations indicate crisper edges and preservation of complex patterns (road markings, building details) compared to baselines (Pramanick et al., 2024).

6. Ablation and Analysis

Ablation studies demonstrate the contribution of each architectural component:

Variant	PSNR	SSIM	FLOPs (G)
w/o AFB (sum)	32.28	0.8974	42.8
w/o AFB (concat)	32.29	0.8974	42.8
1-level DWT only	32.15	0.8957	41.1
w/o CAB	32.31	0.8977	41.8
w/o LHFIB (no freq)	32.29	0.8975	42.5
Full ML-CrAIST	32.36	0.8984	42.9

• Attention-Based Fusion (AFB) removal reduces PSNR by ≈0.07 dB.
• Using single-level wavelet (no multi-scale) leads to a larger ≈0.21 dB PSNR drop.
• CAB removal reduces PSNR by ≈0.05 dB.
• Removing all frequency processing (i.e., LHFIB) reduces PSNR by ≈0.07 dB.

Additional metrics (LPIPS, BRISQUE, EPI) and edge/key-point evaluation confirm that multi-scale low-high frequency cross-attention markedly enhances detail fidelity without introducing artifacts.

7. Significance and Implications

ML-CrAIST demonstrates that explicit and hierarchical modeling of frequency-domain information, coupled with advanced attention-based fusion strategies across spatial and channel dimensions, leads to measurable improvements in SR, especially for high-frequency structures. This supports prioritizing multi-scale frequency decomposition and cross-domain attention over purely spatial or frequency-agnostic transformer architectures for tasks requiring fine detail recovery (Pramanick et al., 2024). A plausible implication is the applicability of this paradigm to other restoration tasks where frequency-local features are critical.