Flickerformer: Transformer for Flicker Removal

• The paper introduces Flickerformer, integrating phase-based fusion, autocorrelation feed-forward, and wavelet-based attention modules to significantly boost flicker suppression.
• Flickerformer employs a U-shaped encoder–decoder with transformer blocks that leverage periodicity and directional priors to robustly eliminate flicker artifacts without ghosting.
• Flickerformer achieves state-of-the-art performance on the BurstDeflicker benchmark, improving PSNR by up to 0.58 dB over traditional restoration methods.

Flickerformer is a transformer-based neural architecture specifically designed for burst flicker removal in short-exposure photography. Flicker artifacts, typically arising from unstable illumination and row-wise exposure inconsistencies (such as those caused by rolling-shutter sensors and artificial lighting), present structured, spatial-temporal degradation patterns distinct from generic image noise or low-light artifacts. Standard restoration frameworks insufficiently suppress flicker, frequently introducing ghosting. Flickerformer addresses this by embedding both periodicity and directionality priors in its architecture—via a phase-based fusion module (PFM), an autocorrelation feed-forward network (AFFN), and a wavelet-based directional attention module (WDAM)—thereby achieving state-of-the-art performance in quantitative and perceptual metrics (Qu et al., 24 Mar 2026).

1. Architectural Overview

Flickerformer employs a U-shaped encoder–decoder backbone tailored to exploit the explicit characteristics of flicker. The input is a burst of three RGB frames $$\{I_0, I_1, I_2\} \in \mathbb{R}^{H \times W \times 3}$$, with $I_1$ designated as the base frame. All frames are concatenated along the channel axis and transformed by a grouped convolution, producing three separate feature maps $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$.

The PFM fuses these low-level features into a composite $F_0 \in \mathbb{R}^{H \times W \times C}$ derived from phase consistency across the burst. The U-shaped encoder–decoder operates in three hierarchical levels (denoted $l=1,2,3$), employing transformer blocks at each scale. Each encoder and decoder level consists of attention mechanisms and specialized feed-forward submodules aligned with flicker prior modeling (AFFN in the encoder and WDAM+AFFFN in the decoder). Skip connections and coarse-to-fine upsampling allow high-frequency details to be preserved and flicker artifacts to be robustly suppressed. The final output is a residual map $R \in \mathbb{R}^{H \times W \times 3}$, added to $I_1$ to generate the restored, flicker-free image $\hat{I}_1$.

2. Phase-Based Fusion Module (PFM)

The PFM exploits the empirical property that flicker is phase-encoded in the frequency domain. For each input feature $X_t$, a 2D FFT yields complex coefficients: $\mathcal{F}(X_t)[k] = A_t(k) e^{i \phi_t(k)}$, where $I_1$0 indexes frequency.

Inter-frame phase correlation is measured relative to the base frame $I_1$1:

$I_1$2

Here, $I_1$3 indicates strong consistency; discrepancies highlight frame-wise phase-shifting flicker.

A frequency-domain weight map $I_1$4 is computed by applying a $I_1$5 convolution and sigmoid activation to the real part of $I_1$6. Each transformed feature is modulated in the frequency domain:

$I_1$7

An inverse FFT produces spatially enhanced features $I_1$8. The set $I_1$9 is concatenated along the channel axis, followed by a $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$0 convolution and ReLU to yield the fused $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$1.

This explicit phase-based aggregation enables the network to adaptively leverage flicker-consistent features across the burst, substantially improving flicker suppression while preserving detail.

3. Autocorrelation Feed-Forward Network (AFFN)

AFFN infuses intra-frame periodicity into each transformer block, modeling flicker as spatially recurring patterns.

Given an input feature $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$2, its spatial autocorrelation $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$3 is calculated leveraging the Wiener–Khinchin theorem:

$$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$4

Dual-domain modulation is performed, introducing learnable coefficients $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$5:

$$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$6

$$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$7

This amplifies periodic frequency components and reinforces repeating spatial structures.

The processed feature $$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$8 is then split channel-wise, gated via depth-wise convolution:

$$\{X_0, X_1, X_2\} \in \mathbb{R}^{H \times W \times C}$$9

AFFN replaces standard MLPs in transformers, resulting in measurable boosts in PSNR (0.2–0.3 dB) over feed-forward alternatives.

4. Wavelet-Based Directional Attention Module (WDAM)

WDAM encodes directionality priors, recognizing that flicker frequently manifests with specific orientation (e.g., horizontal/vertical stripes from rolling-shutter artifacts or lighting hum).

A discrete Haar wavelet transform (DWT) decomposes input feature $F_0 \in \mathbb{R}^{H \times W \times C}$0 into four subbands: $F_0 \in \mathbb{R}^{H \times W \times C}$1 (low-low), $F_0 \in \mathbb{R}^{H \times W \times C}$2 (low-high: horizontal edges), $F_0 \in \mathbb{R}^{H \times W \times C}$3 (high-low: vertical edges), $F_0 \in \mathbb{R}^{H \times W \times C}$4 (high-high: diagonal).

A directional mask $F_0 \in \mathbb{R}^{H \times W \times C}$5 is generated by concatenating $F_0 \in \mathbb{R}^{H \times W \times C}$6 and $F_0 \in \mathbb{R}^{H \times W \times C}$7, passing through a $F_0 \in \mathbb{R}^{H \times W \times C}$8 convolution and sigmoid. $F_0 \in \mathbb{R}^{H \times W \times C}$9 enhances attention along the dominant flicker orientation.

Low-frequency content ($l=1,2,3$0) undergoes windowed multi-head attention, modulated by the directional prior:

$l=1,2,3$1

where $l=1,2,3$2 is a relative positional bias.

High-frequency bands are recombined and projected; all subbands are reconstructed by the inverse DWT to produce $l=1,2,3$3.

WDAM replaces the standard attention mechanism and MLP in the decoder, yielding improvements of approximately 0.2 dB PSNR versus vanilla Swin or Top-k attention.

5. Training Objectives and Optimization

Training employs a composite loss:

• $l=1,2,3$4 loss $l=1,2,3$5 encourages fidelity at the pixel level.
• Perceptual loss $l=1,2,3$6 using VGG-19 features ensures structural consistency.
• The total loss is $l=1,2,3$7 (equal weighting).

Optimization utilizes the Adam algorithm with a constant learning rate of $l=1,2,3$8. Batch size and epoch details are not specified in the source.

6. Quantitative and Qualitative Evaluation

On the BurstDeflicker benchmark, Flickerformer achieves 31.226 dB PSNR, 0.580 dB above the next-best model (AST: 30.646 dB); SSIM is 0.920 (vs 0.918 AST), LPIPS 0.045 (vs 0.050 AST). Replacing any core module—PFM, AFFN, or WDAM—with generic alternatives leads to PSNR drops of 0.28–0.38 dB. Restormer (30.630 dB) is also outperformed (Qu et al., 24 Mar 2026).

Qualitatively, Flickerformer eliminates both subtle and severe flicker patterns without visible ghosting or color shift. The main failure mode arises when no “clean” reference exists in the burst (e.g., persistent darkness across frames), resulting in partial recovery only.

7. Broader Significance and Future Directions

Flickerformer demonstrates that explicit modeling of phase-based periodicity and orientation-specific structure, implemented via hybrid frequency–wavelet domain transformer modules, outperforms prior approaches relying on generic attention or convolution-based restoration. This paradigm—domain-informed transformer architecture design—suggests a promising trajectory for future restoration models structured around explicit artifact priors, especially for complex, highly structured degradations not amenable to generic denoising methods (Qu et al., 24 Mar 2026).