Gather-Scatter Mamba for VSR

• Gather-Scatter Mamba (GSM) is a hybrid framework for video super-resolution that fuses shifted-window self-attention with a selective state-space model for efficient, alignment-aware temporal propagation.
• It employs a two-stage paradigm where spatial refinement via shifted-window attention is followed by temporal propagation using an occlusion-aware gather-scatter mechanism.
• GSM achieves competitive PSNR/SSIM performance on benchmarks, offering improved efficiency and robustness compared to traditional RNN and Transformer-based approaches.

Gather-Scatter Mamba (GSM) is a hybrid framework for video super-resolution (VSR) that integrates shifted-window self-attention for local spatial modeling with an alignment-aware, windowed state-space model (Mamba) for efficient and robust temporal propagation. GSM explicitly addresses the limitations of recurrent neural networks (RNNs) and Transformer-based attention mechanisms in VSR by combining the parallelism and spatial context modeling strengths of local self-attention with the linear scalability and input-dependent state transitions of selective state-space models while mitigating alignment and occlusion issues through dedicated gather and scatter procedures (Ko et al., 1 Oct 2025).

1. Architectural Overview

GSM adopts a two-stage processing paradigm tailored for VSR:

• Spatial Refinement: Each low-resolution frame is independently processed through a stack of Swin-like shifted-window self-attention (SWSA) blocks. Each block comprises two 3D self-attention layers: one using non-shifted windows and one with temporal and spatial window shifts (e.g., size (2, 8, 8)). This structure enables dense extraction of local spatial features and short-range temporal context within frames.
• Temporal Propagation: Temporal information exchange between frames is achieved via a windowed propagation module centered at each frame (the anchor). For each anchor position, spatially aligned features from temporally neighboring frames are gathered (via optical-flow-based warping), concatenated, and then processed with Mamba's selective scan, which models long-range dependencies efficiently in flattened spatiotemporal domains. The updated residuals are subsequently scattered back to their corresponding frames.

The input is a sequence of low-resolution frames $I^{LR} \in \mathbb{R}^{T\times H\times W\times C}$, and the output is high-resolution frames $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$, where $s$ is the scaling factor.

2. Mathematical Foundations and Propagation Mechanisms

2.1 State-Space Model and Mamba Selective Scan

GSM's temporal module builds upon Mamba, a selective scan SSM implementation. Starting from a continuous LTI system

$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$

discretization (step $\Delta$) produces

$$h_k = \bar{A} h_{k-1} + \bar{B} x_k, \quad y_k = C h_k,$$

where $\bar{A} = \exp(\Delta A)$ and $$\bar{B} = (\Delta A)^{-1} (\exp(\Delta A) - I)\Delta B$$.

In Mamba, parameters $B_k$, $C_k$, $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$0 become input-dependent per timestep via learned selectors $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$1:

$$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$2

yielding the recurrent update:

$$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$3

The sequence $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$4 is processed with a parallel selective scan in $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$5 time, where $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$6 is effective sequence length and $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$7 feature dimension.

2.2 Gather-Scatter Paradigm

For windowed temporal propagation centered at anchor $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$8, GSM introduces:

• Gather: For each neighbor frame $$I^{SR} \in \mathbb{R}^{T\times sH\times sW\times C}$$9 in window $s$0, feature maps $s$1 are warped toward anchor $s$2 using optical flow $s$3:

$s$4

Stacking these yields $s$5, which is then temporally flattened.

• Mamba Processing: The flattened stack $s$6 is processed by a Mamba selective scan $s$7.
• Scatter: Output residuals $s$8 are reshaped, then scattered back by warping to their original temporal positions:

$s$9

Bidirectional context flow is ensured by performing both forward and backward scans over the sequence.

3. Algorithmic Summary and Pseudocode

The GSM processing pipeline is accurately captured by the following stages:

1. Feature Extraction and Spatial Refinement:
   • Each frame $$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$0 undergoes patch embedding and passes through the stack of SWSA blocks.
2. Temporal Propagation Across Windows:
   • For each anchor, gather aligned neighborhood features via warping.
   • Apply Mamba's selective scan on the concatenated, temporally-flattened tensor.
   • Scatter Mamba's update residuals back to all frames in the window using the inverse optical flows, updating all features in parallel.
   • Both forward and backward temporal passes are conducted.
3. Final Reconstruction:
   • Each refined feature tensor is processed by a reconstruction head to synthesize the super-resolved output.

$$h_k = \bar{A} h_{k-1} + \bar{B} x_k, \quad y_k = C h_k,$$0

4. Computational Complexity Analysis

The computational complexity of GSM is primarily dictated by the per-window propagation cost and local spatial operations. For $$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$1 spatial tokens, feature dimension $$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$2, and constant window size $$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$3:

Operation	Time Complexity	Memory Complexity
Full spatiotemporal SA	$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$4	$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$5
Shifted-window SA (SWSA)	$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$6	$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$7
Standard Mamba (all N)	$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$8	$$\dfrac{d h(t)}{dt} = A h(t) + B x(t), \quad y(t) = C h(t),$$9
GSM (per stage)	$\Delta$0	$\Delta$1

The alignment via optical flow in GSM incurs $\Delta$2 per window for warping, and $\Delta$3 for the selective scan. Amortizing the SWSA cost, total per-stage complexity remains $\Delta$4, achieving linear scaling in both time and memory relative to sequence length. This enables VSR on long video sequences, with efficient context modeling and substantially reduced misalignment.

5. Occlusion Handling and Alignment Robustness

GSM introduces a center-anchoring strategy, aligning neighboring frames directly to the window center instead of propagating alignment across multiple hops. This minimizes motion path lengths, reducing the potential for occluded or out-of-frame pixels contaminating the propagation process. The optical-flow-based gather ensures that the sequence input to Mamba represents a temporally consistent view, effectively addressing positional distortions that can arise when processing a misaligned flattened sequence. The scatter procedure redistributes the aggregated context to all supporting frames, improving context utilization and temporal consistency. Empirical evidence (e.g., Figure 1) demonstrates that this approach reduces occluded regions and temporal artifacts.

6. Experimental Evaluation

GSM demonstrates competitive efficacy and efficiency on standard VSR benchmarks:

• Benchmarks: REDS4, Vimeo-90K-T, and Vid4 for $\Delta$5 VSR.
• Quantitative Results: On Vimeo-90K-T, GSM achieves a PSNR/SSIM of 38.25 dB/0.9534 using 10.5M parameters and 1.52T FLOPs, outperforming larger models like VRT and IART. Results on REDS4 (32.69 dB/0.9105) and Vid4 (28.03 dB/0.8462) are also competitive.
• Efficiency: GSM runs in 1.07 s per $\Delta$6 clip on an RTX 3090, faster than RVRT and IART with lower FLOPs.
• Ablation Studies: Temporal-first scanning with alignment is superior to spatial or Hilbert scanning. Center-anchored gather outperforms forward-anchored; scatter updates add $\Delta$70.1 dB PSNR and 0.0007 SSIM.
• Qualitative Properties: Reconstructed results exhibit sharper textures and fewer artifacts compared to baselines (BasicVSR++, VRT, RVRT, IART).

7. Limitations and Prospects for Future Research

Several limitations are identified:

• Dependence on precomputed optical flow introduces sensitivity to flow quality, especially for large motions or texture-deprived regions.
• Fixed window length $\Delta$8 constrains long-range context propagation; future work may explore adaptive or hierarchical window strategies.
• Current state size for Mamba is set to $\Delta$9; larger or dynamically learned states could offer potential improvement.
• GSM's alignment-aware propagation suggests extensibility to non-super-resolution video tasks, such as deblurring and frame interpolation, given the generality of its design (Ko et al., 1 Oct 2025).

GSM bridges structured spatiotemporal modeling (selective scan SSMs) and fine-grained spatial context aggregation (shifted-window attention) via an occlusion-aware gather-scatter mechanism. This unification yields a VSR solution that achieves linear computational scaling, competitive perceptual accuracy, and resilience to alignment artifacts.