Burst Image Super-Resolution (CFA)

• Burst image super-resolution with CFA is the process of reconstructing high-resolution RGB images from multiple noisy, shifted RAW frames captured by Bayer-mosaic sensors.
• Recent methods leverage variational inversion, deep learning for precise alignment, and attention-based fusion to effectively handle CFA artifacts and noise.
• Advanced pipelines perform joint demosaicing and super-resolution in an end-to-end framework, improving image detail, color fidelity, and reducing degradation.

Burst image super-resolution with color filter array (CFA) data refers to the problem of reconstructing a clean, high-resolution (HR) RGB image from a sequence (burst) of low-resolution, noisy RAW frames captured with a typical Bayer-mosaic CFA sensor. Unlike single-image super-resolution, burst approaches leverage complementary spatial, spectral, and aliased information distributed across multiple shifted frames, often acquired with natural hand tremor or slight viewpoint variations. Recent advances span direct variational inversion methods, classical empirical merging, and modern deep learning pipelines, with special handling for the unique degradations caused by CFA mosaicking, sensor noise, and geometric misalignment.

1. Imaging Model and Problem Definition

Burst CFA super-resolution operates on $N$ RAW frames $\{y_i\}_{i=1}^N$, each modeled as a downsampled, blurred, warped, mosaicked, and noisy observation of an underlying unknown HR image $x$: $y_i = M \circ D_s \circ B \circ W_i(x) + n_i$ where:

• $W_i$: geometric warp for frame $i$ (subpixel shift, affine, or homography)
• $B$: optical point-spread function (convolutional blur)
• $D_s$: downsampling by scale $s$ (e.g., $s=4$)
• $\{y_i\}_{i=1}^N$0: CFA operator (Bayer RGGB, per-pixel color masking)
• $\{y_i\}_{i=1}^N$1: signal-dependent sensor noise, often modeled as Poisson–Gaussian

The goal is to estimate $\{y_i\}_{i=1}^N$2 (typically in linear RGB space) such that, when subjected to the forward model above, the set of simulated frames closely reproduces the observed RAW burst, with fidelity measured by (weighted) pixel losses, perceptual distances, or downstream color accuracy (Lian et al., 2021, Wei et al., 2023, Bhat et al., 2021, Lecouat et al., 2022, Lecouat et al., 2021, Wronski et al., 2019).

2. Registration and Motion Alignment

Accurate alignment is essential to aggregate information across frames for both shift-variant aliasing and noise statistics.

• Classical kernels: Block-matching and Lucas–Kanade refinement provide subpixel registration under affine or translational models, often accelerated via pyramidal search and tile-wise motion modeling (Wronski et al., 2019, Lecouat et al., 2021). Enhanced robustness is achieved by per-tile weights and structure-tensor–guided anisotropy, to prevent merging across occlusion or strong edges.
• Learned alignment: Modern deep networks use feature-space registration with optical flow or homography estimation. For example, FA-based feature maps or PWC-Net–derived flows warp all non-reference frames to the base frame's grid (Bhat et al., 2021, Wu et al., 2023, Luo et al., 2022). Some Transformer-based designs estimate global planar motion (homography) with ECC maximization in RAW space, which is then refined by affinity-based deep fusion (Wei et al., 2023).
• Deformable and kernel-aware approaches: Several networks employ deformable convolutions, conditioned on scene structure or blur priors, to align features at multi-scale or kernel-adaptive contexts (Lian et al., 2021, Luo et al., 2022, Mehta et al., 2023).

3. Fusion, Aggregation, and Burst Information Integration

• Empirical kernel-regression: Classical methods perform robust pixel-wise merging over all frames using locally adaptive Gaussian kernels and per-pixel outlier rejection, directly accumulating color samples at output HR positions without explicit demosaicing (Wronski et al., 2019).
• Attention and affinity: Deep models frequently employ attention-based fusion, predicting per-frame or per-pixel (and channel) weights for merging aligned feature tensors, enabling data-driven selection of complementary information while suppressing misaligned or low-SNR areas (Bhat et al., 2021, Wei et al., 2023, Wu et al., 2023). Affinity is computed via learned inner-product or transformer-style softmax-normalized weights.
• Recurrent strategies: Recurrent fusion architectures sequentially aggregate features across the burst sequence, with explicit base-frame prompting to anchor the fusion process and handle varying burst lengths, offering improved denoising and flexibility (Wu et al., 2023).
• Global transformer and state-space models: Multi-layer transformer backbones or state-space networks (e.g., Mamba S6), process either the whole burst stack or the key-frame with injected temporal priors; Transformer decoders capture long-range dependencies and global scene context (Luo et al., 2022, Wei et al., 2023, Unal et al., 25 Mar 2025).

4. Joint Super-Resolution and Demosaicing in CFA Domain

Operating directly on RAW CFA inputs preserves subpixel aliasing cues and avoids demosaicing artifacts (Lecouat et al., 2021, Umer et al., 2021, Lecouat et al., 2022, Wronski et al., 2019). Approaches include:

• End-to-end learning from CFA to HR RGB: Network pipelines pack $\{y_i\}_{i=1}^N$3 Bayer blocks into multi-channel inputs for deep feature extraction, with upsampling and color reconstruction steps jointly supervised to deliver full-resolution demosaicked RGB (Bhat et al., 2021, Wu et al., 2023).
• No explicit demosaicing: Some classical and learned methods recover HR color at every pixel by aggregating aliased CFA samples across frames and spatial neighborhoods, using kernel interpolation or deep decoders (Wronski et al., 2019, Umer et al., 2021).
• Handling color regularity: Regularization terms or loss components explicitly maintain cross-channel color consistency, and some models introduce color regularization losses in the RAW and RGB spaces (Wei et al., 2023).

5. Optimization, Training Protocols, and Loss Functions

Optimization strategies are tailored to the burst SR setting with CFA RAW data:

• Variational formulations and unrolled optimization: Several models pose the problem as regularized least-squares (with data-fidelity and image-prior terms), then solve via HQS or majorization-minimization. The proximal step is replaced by deep CNNs or U-Nets acting as learned image priors (Umer et al., 2021, Lecouat et al., 2021, Lecouat et al., 2022).
• End-to-end deep networks: Direct training of deep architectures—CNNs, Transformers, state-space models—supervised on synthetic and real bursts, often with ℓ1 or aligned ℓ1 loss, complemented by perceptual (VGG) and CFA-aware color terms (Wei et al., 2023, Unal et al., 25 Mar 2025).
• Data augmentation and noise modeling: Simulated bursts are generated from large ground-truth RGB datasets by un-processing to RAW, applying random geometric shifts, blur, Bayer mosaicking, and signal-dependent Poisson–Gaussian noise calibrated to real sensor characteristics. Real bursts with DSLR ground-truth enable fine-tuning for domain adaptation (Bhat et al., 2021, Lian et al., 2021, Wei et al., 2023).

6. Performance Benchmarks and Results

Recent methods show substantial improvements in both synthetic and real-world settings:

Method	Dataset	PSNR (dB)	SSIM	LPIPS	Notable Features
KBNet (Lian et al., 2021)	BurstSR (real)	48.27	0.9856	0.0248	Kernel-aware, CFA input, pyramid KAD
FBAnet (Wei et al., 2023)	RealBSR-RAW	49.57	0.990	–	Homography + FAF + Transformer decoder
BSRT-Large (Luo et al., 2022)	BurstSR (real)	48.57	0.986	0.021	Swin Transformer, flow-guided deformable alignment
RBSR (Wu et al., 2023)	BurstSR (real)	48.80	0.987	–	Recurrent fusion, CFA-packing, base-guided
GMTNet (Mehta et al., 2023)	BurstSR (real)	48.95	0.986	–	MBFA+TAFM+RTFU, CFA raw, multi-resolution
BurstMamba (Unal et al., 25 Mar 2025)	RealBSR-RAW	28.03	0.832	0.064	Mamba S6 SSM, OFS, ψ-reparam, CFA/RGB duality
Lucas–Kanade Reloaded (Lecouat et al., 2021)	Synthetic & Real	–	–	End-to-end warping + Prox-CNN prior

Visual qualitative gains include improved edge sharpness, reduced moiré and zipper artifacts, and retention of fine textural details under real-world noise and misregistrations.

7. Algorithmic Challenges and Perspectives

• Degradation-awareness: Methods such as KBNet and others explicitly estimate per-frame motion blur kernels, yielding robustness to unknown or inconsistent degradation across a burst (Lian et al., 2021).
• Alignment under strong aliasing and noise: Bayesian and deep models use subpixel registration and noise-weighted residuals to stably align heavily aliased and low-SNR CFA frames (Lecouat et al., 2021, Wronski et al., 2019).
• Local motion, occlusion, and exposure variation: Advanced approaches leverage learned pixelwise weights, motion confidence maps, and tilewise affine warps to suppress errors from occlusion, object motion, or bracketing (Lecouat et al., 2022, Wronski et al., 2019).
• Computational and memory efficiency: With the shift toward state-space and attention-efficient architectures, e.g., Mamba S6 and transformers with linear complexity, inference times and hardware requirements decrease without compromising accuracy (Unal et al., 25 Mar 2025, Mehta et al., 2023).

Future research opportunities include extending to HDR, multi-exposure, non-Bayer CFA burst super-resolution, lightweight or mobile-friendly deployments, and further unsupervised or self-supervised learning paradigms.