EmambaIR: Event-Guided Image Reconstruction

• EmambaIR is an event-guided image reconstruction network that restores degraded frames using sparse cross-modal fusion and gated state-space modeling.
• It integrates a Top-k Sparse Attention Module (TSAM) and a Gated State-Space Module (GSSM) to capture spatial sparsity and temporal continuity effectively.
• It targets motion deblurring, image deraining, and HDR enhancement, demonstrating competitive efficiency and superior performance metrics compared to state-of-the-art methods.

Searching arXiv for EmambaIR and closely related Mamba-based vision restoration and multispectral fusion papers. EmambaIR is an event-guided image reconstruction network built around an efficient visual state-space model for restoring degraded frame images from a paired degraded image and continuous event stream. The method is presented as a unified framework for three restoration tasks—motion deblurring, image deraining, and HDR enhancement—and its central design claim is that event data should be processed in a way that reflects two of their defining properties: spatial sparsity and temporal continuity. To that end, EmambaIR combines a cross-modal Top-k Sparse Attention Module (TSAM) for sparse image–event fusion with a Gated State-Space Module (GSSM) for global contextual modeling using linear-complexity state-space processing augmented by nonlinear gating (Yu et al., 8 May 2026).

1. Definition and scope

EmambaIR is a UNet-based hierarchical encoder–decoder for event-guided image reconstruction. Its input consists of a degraded image and a corresponding continuous event stream; its output is a restored image, denoted $I_{restored}$. The framework is task-agnostic at the architectural level and is retrained for motion deblurring, deraining, and HDR enhancement rather than redesigned separately for each task (Yu et al., 8 May 2026).

The method is positioned against two common backbone families in event-guided restoration. CNN-based approaches are characterized as effective for local detail extraction but limited in long-range dependency modeling, while Vision Transformers provide global modeling at quadratic complexity, $\mathcal{O}(n^2)$, which is unfavorable at high resolution. EmambaIR instead adopts a visual SSM design with linear complexity, $\mathcal{O}(n)$, and supplements it with explicit sparse cross-modal fusion to exploit the structure of event signals (Yu et al., 8 May 2026).

A common source of confusion is nomenclature. Despite its name, EmambaIR is not an RGB-infrared fusion detector and is not an infrared-only super-resolution method. Its domain is event-guided image reconstruction, where frame imagery is restored with event guidance rather than fused with thermal imagery.

2. Problem formulation and motivation

The problem setting assumes a degraded frame image whose degradation depends on the target task: a blurry frame for deblurring, a rainy image for deraining, or a low/poor dynamic-range image for HDR enhancement. This frame is paired with an event stream captured around the same time. The image modality supplies dense appearance, texture, color, and structure, whereas the event modality supplies sparse but informative motion, edge, and intensity-change cues with temporal continuity (Yu et al., 8 May 2026).

The motivation for EmambaIR is tied directly to event-camera statistics. The paper argues that events are spatially sparse, so a fusion mechanism should not assign equal compute to all spatial locations, and that events are temporally continuous, so the model should capture long-range dependencies efficiently. TSAM is introduced to address the former by retaining only the most informative cross-modal correspondences, and GSSM is introduced to address the latter by combining visual SSMs with a nonlinear gate (Yu et al., 8 May 2026).

The paper also emphasizes application regimes in which frame-only restoration is difficult: high-speed motion, insufficient exposure, large illumination variation, and adverse weather effects such as rain streak motion. In those settings, event streams provide complementary information unavailable from a degraded frame alone. This suggests that EmambaIR is designed less as a generic backbone substitution and more as a modality-aware reconstruction architecture whose inductive biases are aligned with event sensing.

3. Architectural organization

The overall pipeline begins with separate convolutional feature extraction for the degraded image and the event stream. These features are then fused by TSAM, processed globally by GSSM, refined locally by a Residual Local Feature Block (RLFB), and propagated through a hierarchical encoder–decoder with skip connections before reconstruction of $I_{restored}$ (Yu et al., 8 May 2026).

The information flow is directional at the fusion stage. In TSAM, queries are derived from image features, while keys and values are derived from event features. The fused output then enters GSSM, which performs long-range contextual modeling. RLFBs, described as three stacked convolution blocks followed by ReLU layers, supply local detail extraction before decoder-side reconstruction (Yu et al., 8 May 2026).

This ordering is significant. EmambaIR does not treat global modeling and modality fusion as interchangeable operations. Cross-modal alignment is performed first, with sparsification imposed at the fusion stage, and only then are the sparse fused features passed into the visual SSM block for global reasoning. Architecturally, this separates correspondence selection from contextual aggregation.

The paper does not specify exact stage depths, channel widths, encoder/decoder counts, tokenization formulas, or patch-embedding formulas in the provided text. The exact event representation is likewise not formalized, although the method section states that event features are extracted with standard convolutions and the experimental section refers to event voxels and ESIM-generated event streams (Yu et al., 8 May 2026).

4. Cross-modal Top-k Sparse Attention Module

TSAM is the cross-modal fusion mechanism of EmambaIR. Its purpose is to learn image–event correspondences, retain only the most informative event guidance through top-$k$ sparsification, reduce contamination from irrelevant event responses, and lower the practical cost of cross-modal interaction relative to dense attention (Yu et al., 8 May 2026).

The module forms image queries $\mathbf{Q}_I$ and event keys and values $(\mathbf{K}_E,\mathbf{V}_E)$ using $3\times 3$ depth-wise convolutions and normalized $1\times 1$ convolutions. Attention scores are computed via cosine similarity over a transposed spatial attention matrix

$M \in \mathbb{R}^{HW \times HW},$

so each image location can, in principle, attend to all event locations. The key operation is sparsification: after score computation, only the top $\mathcal{O}(n^2)$0 values are kept for each query, the remaining entries are zeroed, and only the retained entries are normalized for softmax computation (Yu et al., 8 May 2026).

The paper gives the top-$\mathcal{O}(n^2)$1 masking rule as

$\mathcal{O}(n^2)$2

Operationally, TSAM therefore performs sparse spatial attention rather than dense channel mixing. Multi-head outputs with small $\mathcal{O}(n^2)$3 are concatenated, a linear projection is applied, and value aggregation is implemented with sparse matrix multiplication (Yu et al., 8 May 2026).

The default configuration uses

$\mathcal{O}(n^2)$4

chosen because it balances accuracy and efficiency well, although the paper reports that $\mathcal{O}(n^2)$5 yields the highest absolute PSNR. This establishes a controlled accuracy–efficiency trade-off rather than a fixed design optimum. In the ablations, TSAM also outperforms dense-attention substitutes inside the same reconstruction framework, including Restormer attention and SwinIR-based variants (Yu et al., 8 May 2026).

Conceptually, TSAM is the component that makes EmambaIR specifically event-aware rather than merely Mamba-based. The module exploits the claim that many event locations are uninformative for a given image query, so forcing full-resolution dense fusion would waste computation and may inject noise.

5. Gated State-Space Module

GSSM is the global modeling block applied after TSAM has produced sparse fused image–event features. Its role is to model long-range contextual dependencies efficiently while compensating for the limited expressivity of purely linear SSM mappings when event dynamics are irregular or scene-dependent (Yu et al., 8 May 2026).

For an input feature

$\mathcal{O}(n^2)$6

GSSM has three stages. First, a multi-scale enhancement block with varying convolution sizes learns correlations under different event densities. Second, the enhanced features are processed by a deep $\mathcal{O}(n^2)$7 convolution, Layer Normalization, and a Visual SSM layer. Third, a nonlinear gated unit modulates channels to increase adaptability to irregular temporal patterns (Yu et al., 8 May 2026).

The paper states the standard continuous-time state-space equations as

$\mathcal{O}(n^2)$8

with $\mathcal{O}(n^2)$9, $\mathcal{O}(n)$0, $\mathcal{O}(n)$1, and $\mathcal{O}(n)$2. These equations are used as the formal basis for the visual SSM layer, although the provided text does not spell out explicit discrete selective-scan formulas or 2D traversal equations for EmambaIR’s implementation (Yu et al., 8 May 2026).

The principal novelty inside GSSM is the Nonlinear Gated Unit (NGU). The paper first introduces a generic gated form,

$\mathcal{O}(n)$3

and then defines the proposed nonlinear version as

$\mathcal{O}(n)$4

Here $\mathcal{O}(n)$5 is specified as GeLU, $\mathcal{O}(n)$6 is global average normalization, and the gating acts in the channel dimension after channel projection into two branches (Yu et al., 8 May 2026).

The stated rationale is that event streams exhibit irregular timing due to varying exposure and asynchronous triggering, so a purely linear SSM may be insufficient. GSSM preserves the linear-complexity advantage of visual SSMs, $\mathcal{O}(n)$7, while adding nonlinear channel weighting to enhance adaptability. In the ablations, replacing GSSM with alternative Mamba-style choices such as MambaIR, Freqmamba, or Wave-Mamba yields lower PSNR and SSIM than the full EmambaIR design (Yu et al., 8 May 2026).

6. Optimization, computational profile, and empirical results

EmambaIR is trained with a simple $\mathcal{O}(n)$8 reconstruction objective,

$\mathcal{O}(n)$9

with no auxiliary perceptual, edge, frequency, or adversarial losses reported in the provided text. The implementation uses PyTorch, a single NVIDIA RTX 3090, Adam optimization, an initial learning rate of $I_{restored}$0, cosine annealing to $I_{restored}$1, 200,000 training iterations, a crop size of $I_{restored}$2, and full-resolution testing. Data augmentation includes horizontal flipping, vertical flipping, random noise, and hot-pixel simulation in event voxels; compared methods are retrained with the same data and training strategy for fairness (Yu et al., 8 May 2026).

The evaluation spans six datasets across three tasks. The named principal datasets in the provided text are GoPro and H2D for deblurring, Adobe240 for deraining, and SDSD for HDR enhancement. GoPro contains 3214 blurry/sharp pairs at $I_{restored}$3, with 2103 used for training and 1111 for testing. H2D provides 603 real-world event-guided deblurring pairs for testing. Adobe240 consists of 120 sequences at 240 fps and $I_{restored}$4, with 50 sequences for training and 10 for testing. SDSD comprises paired real-world low/high dynamic sequences at $I_{restored}$5, with 125 training sequence pairs and 25 test pairs; its event streams are generated using ESIM with the default noise model (Yu et al., 8 May 2026).

On HDR enhancement over SDSD, EmambaIR reports $I_{restored}$6 PSNR and $I_{restored}$7 SSIM, exceeding esL-Net ($I_{restored}$8), RetinexFormer ($I_{restored}$9), Uformer ($k$0), and Evlight ($k$1). On GoPro deblurring, it reports $k$2 PSNR and $k$3 SSIM, exceeding MPRNet ($k$4), Restormer ($k$5), NAFNet ($k$6), and EFNet ($k$7). On Adobe240 deraining, it reports $k$8 PSNR and $k$9 SSIM, exceeding WGWS-Net ($\mathbf{Q}_I$0), Histoformer ($\mathbf{Q}_I$1), FADformer ($\mathbf{Q}_I$2), and MPRNet ($\mathbf{Q}_I$3) (Yu et al., 8 May 2026).

The paper further notes that EmambaIR is, to their knowledge, the first event-guided image deraining algorithm. Qualitative results are described as showing sharper edges and more detailed textures in deblurring, cleaner removal of dense rain streaks with better background preservation in deraining, and improved recovery of structural details in underexposed regions for HDR enhancement (Yu et al., 8 May 2026).

The ablation study isolates the contribution of TSAM and GSSM. A baseline variant without either module reports $\mathbf{Q}_I$4 PSNR and $\mathbf{Q}_I$5 SSIM, while the full model reports $\mathbf{Q}_I$6 PSNR and $\mathbf{Q}_I$7 SSIM. Variants that replace TSAM with Restormer attention or SwinIR components, or replace GSSM with MambaIR, Freqmamba, or Wave-Mamba, all underperform the full configuration (Yu et al., 8 May 2026).

The computational profile is a major part of the method’s empirical identity. On the deblurring comparison reported in Table 2, EmambaIR achieves the best PSNR among the listed methods while also having the lowest parameter count, lowest FLOPs, and lowest runtime: $\mathbf{Q}_I$8 PSNR, $\mathbf{Q}_I$9M parameters, $(\mathbf{K}_E,\mathbf{V}_E)$0G FLOPs, and $(\mathbf{K}_E,\mathbf{V}_E)$1 ms. The corresponding comparison set includes HINet ($(\mathbf{K}_E,\mathbf{V}_E)$2M, $(\mathbf{K}_E,\mathbf{V}_E)$3G, $(\mathbf{K}_E,\mathbf{V}_E)$4 ms), MPRNet ($(\mathbf{K}_E,\mathbf{V}_E)$5M, $(\mathbf{K}_E,\mathbf{V}_E)$6G, $(\mathbf{K}_E,\mathbf{V}_E)$7 ms), Restormer ($(\mathbf{K}_E,\mathbf{V}_E)$8M, $(\mathbf{K}_E,\mathbf{V}_E)$9G, $3\times 3$0 ms), REDNet ($3\times 3$1M, $3\times 3$2G, $3\times 3$3 ms), EFNet ($3\times 3$4M, $3\times 3$5G, $3\times 3$6 ms), and REFID ($3\times 3$7M, $3\times 3$8G, $3\times 3$9 ms). The paper also states that EmambaIR achieves an average training speed 20% faster than compared methods (Yu et al., 8 May 2026).

7. Relation to nearby Mamba-based methods and limitations

EmambaIR belongs to a broader family of Mamba- or VMamba-inspired visual models, but its task and modality assumptions differ sharply from those of similarly named “IR” methods. IRSRMamba, for example, is a single-image infrared super-resolution model rather than an event-guided restoration system. It combines multiscale shallow feature extraction, a wavelet transform feature modulation block, and a VMamba-style backbone with Residual State-Space Groups and Residual State-Space Blocks to reconstruct high-resolution infrared images from low-resolution infrared inputs (Huang et al., 2024). WaveMamba, by contrast, is an RGB-infrared object detector that performs wavelet-domain fusion between visible and infrared features for detection, not image restoration (Zhu et al., 24 Jul 2025).

This distinction matters because EmambaIR’s “IR” should not be read as an infrared label in the sense used by infrared super-resolution or RGB-thermal fusion papers. IRSRMamba addresses infrared image super-resolution, with evaluation on $1\times 1$0 and $1\times 1$1 scaling, while WaveMamba addresses RGB-Infrared object detection with DWT/IDWT-based fusion and a modified YOLO head. EmambaIR instead addresses event-guided reconstruction using frame–event pairs and targets deblurring, deraining, and HDR enhancement (Huang et al., 2024); (Zhu et al., 24 Jul 2025).

The paper’s explicit future direction is extension from image reconstruction to video reconstruction tasks, including video deblurring, video HDR, and video frame interpolation (Yu et al., 8 May 2026). The limitations that are evident from the provided text are primarily specification-related rather than framed as negative empirical findings. The exact event representation is not rigorously defined in the method section; exact VSSM scanning and discretization details are not given; and architectural stage widths and depths are not specified. The paper also does not present a dedicated failure-case analysis. This suggests that the principal novelty lies in sparse cross-modal fusion and gated state-space integration rather than in a new event encoding formalism or a newly derived state-space theory.

Source code and data are reported as publicly available at https://github.com/YunhangWickert/EmambaIR (Yu et al., 8 May 2026).