Burstormer: Burst Image Restoration and Enhancement Transformer (2304.01194v1)

Abstract: On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. However, individual frames in a burst are misaligned due to inevitable motions and contain multiple degradations. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. Towards this direction, we propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement. In comparison to existing works, our approach exploits multi-scale local and non-local features to achieve improved alignment and feature fusion. Our key idea is to enable inter-frame communication in the burst neighborhoods for information aggregation and progressive fusion while modeling the burst-wide context. However, the input burst frames need to be properly aligned before fusing their information. Therefore, we propose an enhanced deformable alignment module for aligning burst features with regards to the reference frame. Unlike existing methods, the proposed alignment module not only aligns burst features but also exchanges feature information and maintains focused communication with the reference frame through the proposed reference-based feature enrichment mechanism, which facilitates handling complex motions. After multi-level alignment and enrichment, we re-emphasize on inter-frame communication within burst using a cyclic burst sampling module. Finally, the inter-frame information is aggregated using the proposed burst feature fusion module followed by progressive upsampling. Our Burstormer outperforms state-of-the-art methods on burst super-resolution, burst denoising and burst low-light enhancement. Our codes and pretrained models are available at https:// github.com/akshaydudhane16/Burstormer

• The paper presents an enhanced deformable alignment module that achieves multi-scale alignment of burst features with inter-frame communication.
• It employs cyclic burst sampling and progressive feature fusion, yielding superior PSNR performance in burst super-resolution and effective noise reduction.
• The study demonstrates improved low-light enhancement using perceptual and L1 losses, setting new benchmarks in burst image processing.

Burstormer: Burst Image Restoration and Enhancement Transformer

In the field of burst image processing, where high-quality composite images are generated from multiple degraded frames, the need for an effective mechanism to address alignment challenges and feature fusion is pivotal. The paper presents Burstormer, a novel algorithm based on the transformer architecture, tailored for the tasks of burst image restoration and enhancement.

Methodological Contributions

Burstormer introduces a sophisticated approach leveraging the inherent capabilities of transformers to improve upon existing methodologies in burst image processing. The primary innovations include:

1. Enhanced Deformable Alignment (EDA): This module emphasizes multi-scale hierarchical alignment of burst features. Unlike conventional methods that handle alignment at a single level or with explicit pre-trained networks, the EDA handles complex motions effectively. By utilizing inter-frame communication and a reference-based feature enrichment mechanism, it aligns burst features with respect to a reference frame, facilitating robust management of misalignments arising from challenging motion scenarios.
2. Cyclic Burst Sampling and Burst Feature Fusion: In the image reconstruction phase, the Burstormer incorporates cyclic burst sampling to aggregate information efficiently, thereby enabling progressive fusion. The burst feature fusion module takes advantage of burst-wide context modeling and adaptive upsampling, significantly outperforming rigid late fusion strategies.

Evaluation and Results

The evaluation spans over tasks such as burst super-resolution, denoising, and low-light enhancement, underscoring its versatility. On synthetic and real-world datasets, Burstormer demonstrates measurable improvements:

• Burst Super-Resolution: On the SyntheticBurst dataset, the model achieves a PSNR improvement over previous best methods like BIPNet, indicating its superior feature alignment and fusion capabilities.
• Burst Denoising: Experiments conducted on both grayscale and color datasets reflect Burstormer's efficacy in noise reduction without sacrificing detail, reinforcing the robustness of the EDA module in handling misalignments and noise simultaneously.
• Low-Light Enhancement: By leveraging perceptual loss alongside the conventional L1 loss, Burstormer achieves enhanced outputs closer to ground truth, a testament to its refined feature transformation process.

Theoretical and Practical Implications

Burstormer contributes theoretically by advancing the understanding of transformer capabilities in multi-image processing and practically by setting new benchmarks across multiple image restoration tasks. The novel introduction of reference-based feature enrichment and cyclic sampling offers a broader implication for future work, suggesting pathways for more efficient yet powerful architectures in computational photography.

Prospective Research Directions

Future research could explore the integration of more sophisticated transformers that extend beyond simple self-attention mechanisms, potentially incorporating hybrid architectures to balance computational cost and model performance. Additionally, expanding this framework to multi-frame video processing could prove beneficial, paving the way for its adoption in real-time applications where efficiency is as crucial as accuracy.

In summary, Burstormer stands as a significant contribution to the field of burst image processing, both in theoretical insights and empirical performance, highlighting the transformative potential of leveraging transformer architectures in addressing complex challenges within this domain.