xLSTM Architecture for Robust RefSR

• xLSTM is a transformer-based architecture that leverages Adaptive Implicit Correlation Gating to control and fuse features from both low-quality and reference images.
• It integrates self-attention on LQ inputs with reference-aware cross-attention, dynamically regulating feature contributions even under misalignment.
• Empirical results show xLSTM improves metrics like PSNR and SSIM while offering efficient, scalable performance in reference-based super-resolution tasks.

The xLSTM architecture, as described in the reference-based super-resolution (RefSR) literature, refers to transformer-based attention backbones incorporating specialized cross-attention modules for adaptive fusion of features from both low-quality (LQ) and high-quality reference (Ref) images. Unlike canonical transformer designs that apply standard cross-attention, xLSTM architectures introduce modifications—such as Adaptive Implicit Correlation Gating (AICG)—to control the flow of reference information dynamically and robustly, particularly under misalignment or uncertain correspondences (Wang et al., 2 Feb 2026).

1. Architectural Foundations and Motivation

xLSTM architecture emerged to address limitations in explicit LQ–Ref feature matching and static attention mechanisms, which can lead to either over-reliance on incorrect reference patches or underutilization of genuinely helpful information. The need for dynamic, context-driven regulation of reference feature integration is especially acute in real-world settings, where degradations and misalignments disrupt reliable correspondences. xLSTM models are constructed to leverage both self-attention (on LQ features) and reference-aware cross-attention, with additional gating or summary-token mechanisms to modulate the contribution of the Ref pathway (Wang et al., 2 Feb 2026).

2. Attention Backbone Integration and AICG Module

xLSTM replaces the vanilla cross-attention in standard transformer/U-Net layers with a sequence comprising:

• Self-attention on LQ input features: $H_{\text{src}}^l$.
• Reference attention computed as:

$$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$

$$\mathrm{RA}(H_{\text{src}}^l, H_{\text{ref}}^l) = \mathrm{ZeroLinear}\left(\mathrm{Softmax}\left(\frac{QK^T}{\sqrt d}\right)V\right)$$

• Adaptive Implicit Correlation Gating (AICG): introduces $M$ learnable summary tokens $T_S \in \mathbb{R}^{M \times d}$, projected into key space and used to softly cluster and summarize reference features. Source token–to–summary token correlations produce gating weights $G \in (0,1)^{L_q \times 1}$, which modulate the cross-attention output by:

$$H_{\text{fused}} = G \odot \mathrm{RA}(H_{\text{src}}, H_{\text{ref}})$$

followed by a ZeroLinear projection and residual addition, yielding the final output for the block:

$$H_{\text{src}}^{l+1} = H_{\text{src}}^l + \mathrm{ZeroLinear}(G^l \odot \mathrm{RA}^l(H_{\text{src}}^l, H_{\text{ref}}^l)) + \text{MLP residual}$$

This mechanism ensures that reference feature flow is adaptively gated per-token, with near-negligible computational overhead relative to explicit LQ–Ref gating (Wang et al., 2 Feb 2026).

3. Training Objectives and Reference Conditioning

The xLSTM-based Ada-RefSR model employs a “Trust but Verify” principle, operationalized through its gating strategy:

• During training, 20% of reference pairs are deliberately replaced with irrelevant sources.
• The network is trained using:
  • Reconstruction loss ($L_2$)
  • Perceptual loss (VGG)
  • Adversarial loss (GAN)

$$\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\text{rec}} + \lambda_2 \mathcal{L}_{\text{per}} + \lambda_3 \mathcal{L}_{\text{adv}}$$

• Only the reference-attention (RA) and AICG modules are trainable; the core diffusion backbone is frozen.

This protocol explicitly forces the gating module to learn to detect and suppress unreliable reference information through backpropagation, improving robustness under reference misalignment or corruption (Wang et al., 2 Feb 2026).

4. Empirical Performance and Efficiency

Empirical evaluation of the xLSTM-based Ada-RefSR demonstrates:

• Top performance on PSNR, SSIM, and LPIPS across multiple evaluation datasets (CUFED5, WRSR, birds, faces).
• Maintained perceptual quality on naturalness metrics (NIQE, MUSIQ, CLIP-IQA), matching leading diffusion baselines.
• Substantial efficiency: Ada-RefSR achieves $$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$030$$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$1 speedup (one step vs. 50 steps in multi-step ReFIR-based methods), owing to the single-step diffusion and lightweight AICG.
• Parameter profile: of 2,678.9M total model parameters, only 61.98M (RA layers) + 0.20M (summary tokens) are trainable, making adaptation memory-efficient.
• FLOPs overhead from AICG is only 0.13% compared to 16% for explicit matrix gating in ReFIR (Wang et al., 2 Feb 2026).

At higher resolutions (e.g., 1024$$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$21024) the runtime advantage over traditional approaches widens, reinforcing the utility of xLSTM-style adaptive attention for scalable RefSR.

5. Context within Reference-Based Super-Resolution

xLSTM and AICG modules are foundational in closing the gap between single-image super-resolution and reference-based approaches. They facilitate robust, context-sensitive integration of unaligned reference cues, contrasting with earlier models based on explicit patch or token-wise matching. The architectural leveraging of summary tokens and gating reflects a trend in vision transformers toward more flexible, data-driven modulation of external context, analogous in spirit to memory and hypernetwork-augmented transformers.

Furthermore, ablation studies and cross-method retraining (on methods such as MASA, C²-Matching) indicate the generality of xLSTM-style gating: dynamic reference utility assessment yields measurable improvements in fidelity and robustness to reference noise across a range of SR backbones (Wang et al., 2 Feb 2026, Mei et al., 2023). This suggests broad applicability in noisy, real-world RefSR workflows.

6. Implementation and Hyperparameter Details

Optimal operation involves:

• Summary token count $$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$3.
• Learning rate $$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$4, batch size 16, 11K iterations.
• Loss weightings $$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$5, $$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$6, $$Q = H_{\text{src}}^l W_Q, \quad K = H_{\text{ref}}^l W_K, \quad V = H_{\text{ref}}^l W_V$$7 matching S3Diff defaults.
• Hardware: two A40 GPUs (for reference implementation).
• Key architectural modules (AICG, RA layers) are slotted into transformer/U-Net backbones; the remainder of the block (including feed-forward MLPs) is left unmodified (Wang et al., 2 Feb 2026).

A plausible implication is that xLSTM elements may be incorporated into other vision tasks beyond RefSR wherever external context reliability is variable, and adaptive feature gating is required.

---

xLSTM, as instantiated through AICG and adaptive reference-attention, represents an efficient, scalable, and context-aware transformer adaptation for robust reference-conditioned image restoration and super-resolution tasks (Wang et al., 2 Feb 2026).