CLIT: Cascaded Local Implicit Transformer

• The paper introduces a cascaded transformer that achieves arbitrary-scale image super-resolution by integrating local attention with continuous implicit mapping.
• It employs a multi-stage cascade where each stage refines residuals, enabling precise recovery of high-frequency details at varying scales.
• Cumulative curriculum training and ablation analyses demonstrate robust improvements in PSNR and visual quality compared to existing meta-super-resolution methods.

The Cascaded Local Implicit Transformer (CLIT) is a neural architecture designed for arbitrary-scale image super-resolution by combining local implicit neural representations with transformer-based attention mechanisms and a multi-scale cascade. The architecture introduces innovations in local attention over feature grids, continuous implicit mapping, and cumulative curriculum training to achieve high-fidelity upsampling across integer and non-integer scales. The framework generalizes to various feature encoders and supports flexible scale factors, yielding advances over preceding local-implicit and meta-super-resolution approaches (Chen et al., 2023).

1. Implicit Neural Representation Foundation

CLIT is built upon implicit neural representation, where the goal is to learn a continuous function mapping from 2D spatial coordinates in the high-resolution (HR) image domain to RGB pixel values. Let $I^{LR}\colon[0,H)\times[0,W)\to\mathbb R^3$ denote the input low-resolution (LR) image. The objective is to optimize a continuous mapping

$$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$

where, for an arbitrary HR spatial coordinate $\mathbf p=(x,y)$, $f(\mathbf p;\theta)$ predicts its RGB value. In practice, $f$ is implemented as an MLP decoder conditioned on features gathered via local attention: $$\phi(\mathbf p):\text{frequency-encoded positional embedding},\quad \widetilde{\mathbf z}:\text{aggregated local features via CSLAB},\quad c: \text{cell (HR pixel width, height)},$$

$$f(\mathbf p;\theta) = \mathrm{MLP}_\theta\bigl([\phi(\mathbf p),\,\widetilde{\mathbf z},\,c]\bigr)$$

Frequency encoding is performed using $L$ bands (paper sets $L=10$), providing a rich spatial basis. This implicit approach circumvents explicit upsampling and is agnostic to output grid resolution.

2. Local Implicit Transformer Structure

The Local Implicit Transformer (LIT) core comprises four main modules:

• Encoder ($E_\varphi$): Extracts a feature map $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$0 from the LR image using a pretrained backbone (e.g., EDSR, RDN, SwinIR).
• 1×1 Convolutions: Project encoder features into query, key, and value spaces.
• Cross-Scale Local Attention Block (CSLAB): For each HR query position, aggregates local context:
  1. Locates the nearest LR pixel, defines a $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$1 grid ($$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$2 recommended).
  2. Computes relative offsets $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$3 and applies frequency encoding $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$4.
  3. Produces $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$5, $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$6, $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$7 via interpolation/sampling from the encoder’s map.
  4. Generates learned positional bias $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$8.
  5. Computes per-head scaled dot-product attention with positional bias:

  $$f\bigl(\mathbf p;\,\theta\bigr)\;:\;\mathbb R^2\to\mathbb R^3$$9

1. Concatenates multi-head outputs to produce aggregated local feature $\mathbf p=(x,y)$0.

• MLP Decoder: A five-layer network with GELU activations, maps the concatenated feature to RGB.

By restricting attention to local regions, LIT captures high-frequency details and is efficient for high-resolution output, as the architecture is fully convolution-free during decoding.

3. Multi-Scale Cascade Design and Residual Aggregation

For large upsampling factors $\mathbf p=(x,y)$1, a single-stage mapping may lose image detail due to limited context aggregation. CLIT addresses this via a cascade of $\mathbf p=(x,y)$2 LIT modules, each operating at intermediate scales. Let $\mathbf p=(x,y)$3 be positive scale factors such that $\mathbf p=(x,y)$4, with $\mathbf p=(x,y)$5. At each stage $\mathbf p=(x,y)$6:

1. Upsample the encoder feature map to scale $\mathbf p=(x,y)$7: $\mathbf p=(x,y)$8.
2. Apply LIT module $\mathbf p=(x,y)$9 to $f(\mathbf p;\theta)$0 to predict a residual $f(\mathbf p;\theta)$1 at scale $f(\mathbf p;\theta)$2.
3. Aggregate residuals with decay $f(\mathbf p;\theta)$3 (default $f(\mathbf p;\theta)$4) onto a baseline upsampled LR image: $f(\mathbf p;\theta)$5

This cascaded design enables progressive refinement, allowing CLIT to recover fine structures lost with direct large-factor upsampling.

4. Cumulative Curriculum Training Strategy

Simultaneously training on a wide range of scale factors degrades super-resolution accuracy, particularly out-of-distribution. CLIT adopts a two-phase cumulative schedule:

• Phase 1: Train the first LIT on scales $f(\mathbf p;\theta)$6.
• Phase 2: Alternate between small ($f(\mathbf p;\theta)$7) and large ($f(\mathbf p;\theta)$8) scales during fine-tuning; incorporate deeper cascade stages LIT$f(\mathbf p;\theta)$9, LIT$f$0..., each retrained using a mix of previously seen and new scales.

The multi-stage loss function is

$f$1

where $f$2 is the ground-truth image downsampled to the intermediate stage scale.

This cumulative curriculum yields notable improvement on out-of-distribution and high-magnification scales, indicating that progressive curriculum over scales is critical for robustness.

5. Training, Inference, and Implementation Details

CLIT is configurable with respect to encoder choice, local attention window, and other hyper-parameters:

• Encoder: EDSR-baseline, RDN, or SwinIR, with up to 64 feature channels.
• CSLAB: Four attention heads ($f$3); per-head dimension $f$4; local neighborhood $f$5.
• Frequency Encoding: 10 bands ($f$6).
• MLP Decoder: 5 layers with hidden size 256.
• Scales Supported: $f$7, including non-integers.
• Pseudo-code: The training loop samples cascades of scales per batch, computes intermediate residuals, aggregates outputs, and accumulates the loss progressively across all stages. Inference proceeds identically, evaluating the encoder and each cascade stage on the target scale.

The architecture efficiently supports patch-based or full-image inference owing to its local attention and implicit coordinate handling.

6. Empirical Results and Ablation Analyses

Quantitative results on DIV2K (validation, EDSR-baseline encoder) show that CLIT achieves favorable or superior PSNR compared to Meta-SR, LIIF, and LTE. For instance:

Scale & Method	×2	×3	×4	×12
Meta-SR	34.64	30.93	28.92	23.55
LIIF	34.67	30.96	29.00	23.71
LTE	34.72	31.02	29.04	23.78
EDSR-CLIT (ours)	34.82	31.14	29.17	23.85

Ablation findings:

• Removing CSLAB (–a): –0.05 dB, confirming the importance of local attention.
• Omitting cell input (–c): –0.1 dB for small scales.
• Enlarging the local grid to $f$8 yields small gains (~0.01–0.02 dB).
• Switching from uniform to cumulative sampling yields a +0.07 dB gain on out-of-distribution (×12) scale.

Across multiple datasets (Set5/Set14/B100/Urban100) and with advanced encoders, CLIT consistently restores sharper edges and high-frequency textures, excelling at both integer and non-integer super-resolution factors (Chen et al., 2023).

7. Comparative Positioning and Significance

CLIT differentiates itself from earlier parameter-sharing and meta-learning based super-resolution models (e.g., Meta-SR, LIIF) by:

• Integrating cross-scale local attention within the implicit mapping framework.
• Employing a multi-scale residual cascade, enabling quality preservation even for extreme or unusual scale factors.
• Combining curriculum-based cumulative training, which mitigates performance drop for large upsampling ratios.

This synthesis enables a single architecture to achieve state-of-the-art performance in arbitrary-scale super-resolution, demonstrating improvements in both quantitative metrics (PSNR, SSIM) and visual quality, particularly in edge sharpness and texture fidelity (Chen et al., 2023).