ESC-MISR: Enhanced Multi-Image SR

• ESC-MISR is an end-to-end deep learning framework for multi-image super-resolution in remote sensing that enhances spatial correlations via advanced attention mechanisms.
• The architecture integrates a CMT encoder, a novel Multi-Image Spatial Transformer (MIST) fusion module, and an FFC decoder to achieve state-of-the-art cPSNR and cSSIM on PROBA-V.
• A random shuffle training protocol mitigates temporal bias in weakly correlated satellite images, leading to improved HR reconstruction and generalization.

ESC-MISR (Enhancing Spatial Correlations for Multi-Image Super-Resolution) is an end-to-end deep learning framework developed for Multi-Image Super-Resolution in Remote Sensing (MISR-RS) applications. The method is specifically designed to generate high-resolution (HR) images by jointly leveraging multiple low-resolution (LR) satellite observations of the same scene, effectively fusing their spatial and weakly correlated temporal information. ESC-MISR introduces a novel Multi-Image Spatial Transformer (MIST) fusion module, a random shuffle training protocol to attenuate temporal bias, and employs a composite architecture that advances the state of the art on challenging MISR benchmarks such as PROBA-V (Zhang et al., 2024).

1. Motivation and Problem Setting

MISR in remote sensing aims to recover an HR image from several sub-pixel shifted LR satellite captures. The temporal sequence of these observations is weakly correlated due to variable sensor revisit intervals and atmospheric conditions. Traditional MISR methods often treat the input series as strongly temporally correlated, thus failing to fully exploit inherent spatial correlations and instead introducing undesirable temporal dependencies. ESC-MISR addresses this by directly enhancing spatial correlation modeling across LR inputs and explicitly suppressing artificial temporal ordering biases.

Given inputs $$\mathcal{I}_{\text{LR}} = \{I_{\text{LR}}^i\}_{i=1\dots K}$$ with each $$I_{\text{LR}}^i \in \mathbb{R}^{H\times W\times C_{in}}$$, the core MISR aim is to generate a super-resolved $I_{\text{SR}} \approx I_{\text{HR}}$, where $I_{\text{HR}}$ is a reference HR satellite image (e.g., $384\times384$ in PROBA-V) (Zhang et al., 2024).

2. ESC-MISR Architecture Overview

ESC-MISR is composed of three main subnetworks: a lightweight CNNs-Meet-Transformers (CMT) encoder, the Multi-Image Spatial Transformer (MIST) fusion module, and a decoder based on Fast Fourier Convolution (FFC).

Encoder (CMT)

The encoder processes each LR input independently:

• Initial Feature Extraction: $$X_C = \text{BN} (\text{GeLU}(\text{Conv}_{3\times 3}(I_{\text{LR}})))$$
• Residual Branch: $X_C' = X_C + \text{Conv}_{3\times 3}(X_C)$
• Self-Attention: $X_A = X_C' + \text{MHSA}(\text{LN}(X_C'))$
• Feedforward: $X_M = X_A + \text{MLP}(\text{LN}(X_A))$
• Encoding: $F_E = \text{Conv}_{3\times 3}(X_M)$

This yields per-image features $F_E^i \in \mathbb{R}^{H\times W\times C}$ for all $K$ images ($C=16$ in implementation).

Fusion: Multi-Image Spatial Transformer (MIST)

MIST is central to the architecture. It performs:

• Per-image message extraction: Partition each $F_E^i$ into $N$ patches; project to tokens ${m_p^j}$.
• Per-image attention: Message tokens are updated via MHSA and concatenated back with $F_E^i$.
• Multi-image spatial attention: All updated per-image features are stacked, flattened, and passed through a cross-image MHSA and MLP. The fused representation $F_{\text{SE}}$ is averaged over $K$ to produce $F_{\text{SE,out}}$.
• MHSA Definition: $Q = XW_q,\ K = XW_k,\ V = XW_v$ and $$\mathrm{Attention}(Q,K,V) = \mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + B\right)V$$.

Decoder (FFC)

• Branch-wise Processing: Fused feature is split into local $f_l$ and global $f_g$ branches.
• Local branch: $$X_l = \text{ReLU}(\text{BN}(\text{Conv}(f_l) + \text{Conv}(f_g)))$$
• Global (Fourier) branch: Global mix via FFT/IFFT operations: $$f_{g\to g} = f_g + \text{IFFT}(\text{Conv}(\text{FFT}(f_g)))$$
• Feature Concatenation: $X_{\text{FFC}} = [X_l, X_g]$; final upsampling via pixel-shuffle to output high-resolution imagery.

3. Multi-Image Spatial Transformer (MIST): Mechanism and Impact

MIST is architected as a stack of Multi-Image Spatial Attention Blocks (MISABs):

• Message Token Extraction: Each per-image feature map is patchified (patch size $n=1$ used), projected, and attended over spatial tokens.
• Global Feature Augmentation: The token set is upsampled and concatenated with the original feature map.
• Cross-Image Attention: The K augmented feature maps are assembled, facilitating inter-image self-attention across spatial and inter-image domains.
• Averaging: The K outputs are averaged to yield the fused spatial encoding.

Ablation studies report on the PROBA-V NIR band: GAP (47.24 dB), 3D-Conv (47.40 dB), generic self-attention (48.31 dB), and MISAB (48.60 dB), evidencing a 0.29 dB improvement attributable to this specific spatial attention mechanism.

4. Random Shuffle Training Strategy

The random shuffle protocol mitigates spurious frame-order bias and enhances generalization for weakly correlated multispectral time series:

• Protocol: For each epoch and scene batch, the K input images are randomly permuted $T$ times (with $T \approx 6$) per scene per epoch. This randomization is only applied during training to disrupt any learned strict temporal dependencies.
• Implementation: For each shuffle, the network processes the permuted stack, computes the $\ell_2$ reconstruction loss, and backpropagates.
• Impact: Ablations (NIR cPSNR) show that increasing $T$ from 0 (no shuffle) to 6 yields consistent but diminishing improvements, saturating at $T=6$ ($49.42$ dB vs. $49.25$ dB with no shuffling).

5. Training Procedure, Loss, and Dataset

Dataset (PROBA-V)

• Composition: 1,450 scenes (1,160 train/val, 290 test); each with 9–35 LR observations (128 × 128), one HR reference (384 × 384), and per-pixel quality masks.
• Bands: RED and NIR.
• Padding: If $N < 24$ observations per scene, pad with the clearest frames to $K=24$ inputs.

Loss Function

ESC-MISR is trained with masked $\ell_2$ loss, focusing on cloud-free pixels only:

$$\mathcal{L}_{\text{rec}} = \|M \cdot (I_{\text{SR}} - I_{\text{HR}})\|_2^2$$

where $M$ is the pixel-quality mask (1 for clear, 0 otherwise).

Hyper-parameters

• Batch size: 4 scenes
• Epochs: 400
• Optimizer: Adam$(\beta_1=0.9,\,\beta_2=0.999)$; learning rate $10^{-4}$, halved every 100 epochs
• Architecture: CMT embedding dim: 16; MIST: 6 MISABs; FFC channels: 32 → 16 → 32.

6. Evaluation Metrics

Metrics are defined over cloud-masked, clear pixels:

• Cloud-masked PSNR (cPSNR):

$$\text{cPSNR} = 10\cdot \log_{10} \left( \frac{\text{MAX}^2 \cdot \sum_i w_i}{\sum_i w_i \cdot (SR_i - HR_i)^2} \right)$$

with $\text{MAX}=1$, $w_i$ as the binary mask.

• Cloud-masked SSIM (cSSIM): SSIM evaluated only on pixels with $w_i=1$.

7. Quantitative Performance and Component Analysis

ESC-MISR achieves state-of-the-art results on PROBA-V:

Band	Best Prior (cPSNR dB/cSSIM)	ESC-MISR (cPSNR dB/cSSIM)	Absolute Gain
NIR	48.72 / 0.9883 (PIU)	49.42 / 0.9896	+0.70 dB / +0.0013
RED	50.67 / 0.9932 (TR-MISR)	51.38 / 0.9932	+0.71 dB / +0.0011

Component ablation demonstrates the integrative utility of each architectural element. For instance, the combination of CMT+MIST yields 49.12/51.07 dB (NIR/RED), which increases to 49.25/51.21 dB with full CMT+MIST+FFC pipeline. Input count $K$ ablation indicates performance increases with more images ($K=4 \to 45.8$ dB, $K=24 \to 49.42$ dB), with additional gains minimal due to overfitting tendency in RED.

Summary

ESC-MISR establishes a new benchmark in MISR-RS by focusing on spatial relationship enhancement, robust inter-image fusion, and frame-order invariant learning. The framework's confluence of CMT encoding, advanced multi-image attention (MIST), global-local decoding (FFC), and principled random shuffle training collectively yield significant PSNR improvements over previous MISR baselines. The method provides a foundation for further advances in weakly temporally correlated remote sensing super-resolution (Zhang et al., 2024).