Event-to-Video Reconstruction using Spatio-Temporal and Frequency-Enhanced Deep Neural Networks

Abstract: Event cameras offer significant advantages over conventional frame-based counterparts, including high temporal resolution, low latency, and energy efficiency. These characteristics make them suitable for high-speed and high-dynamic range scene acquisition scenarios; however, the lack of dense intensity frames limits the direct applicability of conventional computer vision methods for scene understanding. Event-to-video (E2V) reconstruction seeks to bridge this gap by converting asynchronous event streams into a sequence of synchronous video frames. Existing E2V reconstruction methods based on convolutional neural networks and transformers operate primarily in the spatial domain and often struggle to recover fine structural details while suppressing severe reconstruction artifacts. To address these issues, we propose MSFET-E2V, a novel multiscale frequency-enhanced transformer model. At its core lies a cross-domain attention module, which fuses spatio-temporal features with frequency-aware representations derived from the discrete wavelet transform. Unlike prior methods relying solely on spatial attention, our approach effectively captures both local and global structures by taking into account low- and high-frequency components, enhancing detail preservation and robustness across various motion scenarios. Furthermore, we propose a lightweight wavelet-enhanced skip block that serves as a skip connection, facilitating artifact suppression and structural detail refinement through joint spatial-frequency domain processing. Extensive experiments demonstrate that MSFET-E2V achieves superior performance over state-of-the-art methods on multiple real-world event datasets, offering significant gains in reconstruction quality. Moreover, compared to the existing transformer-based method, our proposed model significantly reduces the number of parameters, the GPU memory usage, and inference time.

• The paper introduces MSFET-E2V, a multiscale transformer architecture that fuses spatio-temporal and frequency-domain features for superior event-to-video reconstruction.
• It leverages innovative modules like the cross-domain attention module (CDAM), wavelet-enhanced skip block (WSB), and residual-guided decoder (RGD) to achieve faster inference and reduced GPU memory usage.
• The integration of discrete wavelet transforms and residual-guided decoding enhances structural preservation, minimizes artifacts, and excels under low-light and high-dynamic-range conditions.

Multiscale Spatio-Temporal and Frequency-Enhanced Event-to-Video Reconstruction

Motivation and Context

Event cameras deliver high temporal resolution, low latency, and energy efficiency. However, their asynchronous output—sparse event streams—makes conventional frame-based algorithms for scene understanding inapplicable. The event-to-video (E2V) reconstruction task aims to convert these event streams into dense, synchronous intensity image sequences, facilitating visualization and broad applicability to object tracking and other vision tasks. Existing CNN-based and transformer-based models for E2V reconstruction either fail to capture long-range dependencies (CNNs, due to limited receptive fields) or incur high computational costs and lose fine structural details (transformers relying solely on spatial-domain self-attention). These deficiencies are exacerbated under conditions of low event density, fast motion, and challenging lighting.

Contributions of MSFET-E2V

This work introduces MSFET-E2V—a multiscale, frequency-enhanced transformer architecture for E2V reconstruction. The novel design addresses three principal limitations:

• The cross-domain attention module (CDAM) fuses spatio-temporal representations (via CRNN and ConvLSTM) with frequency-domain features generated by discrete wavelet transform (DWT), leveraging both local (high-frequency) and global (low-frequency) structure.
• The wavelet-enhanced skip block (WSB) operates in spatial and frequency domains, refining shallow features and suppressing artifacts before merging them with deep decoder features.
• The residual-guided decoder (RGD) employs residual blocks for stable gradient propagation and enhanced feature preservation.

Comprehensive experiments demonstrate that MSFET-E2V surpasses state-of-the-art E2V methods in reconstruction quality, with substantial efficiency improvements: >70% faster inference and >50% lower GPU memory usage compared to transformer-based baselines.

Architectural Details

Encoding begins by transforming grouped events (spatial coordinates, timestamp, polarity) into voxel grids discretized into temporal bins (optimal number found to be five). The input passes through DownConv blocks to build a multiscale feature pyramid, subsequently processed by CDAM and WSB modules. The overall architecture is as follows:

Figure 1: Overall architecture of MSFET-E2V, integrating DownConv, CDAM, WSB, and RGD blocks for spatio-temporal and frequency-domain fusion.

Discrete Wavelet Transform

DWT decomposes input features into four subbands (LL, LH, HL, HH), covering localized frequency components. The hierarchical decomposition provides efficient expansion of receptive fields and simultaneous access to fine structural details.

Figure 2: DWT workflow, applying LPF and HPF along rows and columns to extract LF and HF subbands.

Cross-Domain Attention Module (CDAM)

CDAM fuses spatial and frequency information as follows:

• Spatio-temporal branch: residual blocks and ConvLSTM extract spatial and temporal cues.
• Frequency domain branch: Haar DWT decomposes features and subsequent RBs process LF and HF channels.
• Attention heads are cross-domain: $Q$ from spatio-temporal, $K$ and $V$ from frequency domain. Element-wise addition aligns domains before attention fusion.

  Figure 3: CDAM structure combining spatio-temporal and frequency-domain feature streams via cross-domain multi-head self-attention.

Wavelet-Enhanced Skip Block (WSB) and Residual-Guided Decoder (RGD)

WSB processes shallow encoder outputs through DWT, applies targeted noise suppression to the HH subband, and reconstructs features via IWT before merging with spatial branch output. RGD blocks upsample and enhance features using residual blocks.

Figure 4: Internal structure of RGD (a) and WSB (b), detailing skip connection enhancement and decoder feature refinement.

Experimental Results

MSFET-E2V sets a new benchmark across datasets (HQF, MVSEC, ECD, HDR, CED) in PSNR, SSIM, and LPIPS metrics. The model demonstrates minimal ghosting, bleeding, and blur artifacts and superior structural and textural fidelity compared to CNN and transformer-based baselines. In low-light scenarios (MVSEC_night, HDR), MSFET-E2V achieves lowest BRISQUE scores, indicating perceptual quality closest to natural images.

Figure 5: Visual results on HQF, MVSEC, and ECD datasets; MSFET-E2V reconstructs high-quality frames with minimal artifacts.

Figure 6: Qualitative comparison in slow-motion conditions; MSFET-E2V preserves details across all temporal speeds.

Figure 7: Comparison in low-light; significant reduction in BRISQUE score compared to baselines.

Figure 8: Challenging HDR conditions; MSFET-E2V yields best perceptual quality in all cases.

Color reconstruction experiments using the CED dataset confirm MSFET-E2V's ability to assign vibrant, natural colors and maintain structural detail.

Figure 9: Color event reconstruction; MSFET-E2V produces vibrant, natural colors and sharp detail.

Efficiency and Robustness

MSFET-E2V reduces inference time and memory consumption by exploiting frequency-domain operations and compact multi-head self-attention. Ablation studies validate the necessity of combined LF and HF frequency inputs (Table: ablation_cdam), the optimal encoder-decoder depth (three layers), and the optimal voxel bin number (five). Robustness analysis under variable temporal windows and event grouping confirms consistent superiority in LPIPS scores.

Figure 10: LPIPS across different temporal windows and event groupings; MSFET-E2V consistently outperforms SOTA baselines.

Ablation Studies

Ablation demonstrates:

• Loss of HF subbands in CDAM (“CDAM_LL”) causes perceptual and structural degradation.
• Using only HF subbands (“CDAM_HF”) improves texture but impairs global structure; the combination achieves best performance (see Table: ablation_cdam, Figure 11).
• Encoder-decoder depth and bin number both have non-monotonic optimal points; excessive depth or bins leads to diminished returns or sparsity issues.

Figure 12: Visual ablation of model variants; full MSFET-E2V preserves fine detail and achieves lowest perceptual distortion.

Figure 11: CDAM frequency variant results; full LF+HF integration yields highest reconstruction quality.

Implications and Future Directions

The incorporation of frequency domain analysis via DWT in both attention and skip connections represents a robust approach to event-driven vision reconstruction, facilitating significant improvements in both efficiency and quality. The CDAM enables the model to capture long-range dependencies and fine details simultaneously, and the WSB suppresses event-noise effectively, making this methodology suitable for real-time, high-resolution, and high-dynamic-range applications. This architecture demonstrates the practical feasibility of frequency-driven attention for event-based vision, and its efficient design makes deployment on resource-constrained devices viable. Theoretical implications include the potential extension of wavelet-based attention mechanisms to other domains (e.g., spiking neural networks and video enhancement).

Future efforts may focus on:

• Extension to unsupervised and self-supervised training regimes for real-world deployment.
• Further optimization of frequency domain integration with transformers for even lower latency and inference cost.
• Application to other asynchronous sensor modalities (e.g., neuromorphic audio, radar).

Conclusion

MSFET-E2V provides a comprehensive solution to event-to-video reconstruction by integrating spatio-temporal and frequency domains within a multiscale transformer framework. The CDAM and WSB modules together deliver improved reconstruction quality, artifact suppression, and computational efficiency, outperforming existing CNN- and transformer-based models. The success of frequency-enhanced attention architectures positions them as promising candidates for the next generation of event-driven vision and real-time sensing applications (2605.25804).