Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor

Abstract: Motion blur arises when rapid scene changes occur during the exposure period, collapsing rich intra-exposure motion into a single RGB frame. Without explicit structural or temporal cues, RGB-only deblurring is highly ill-posed and often fails under extreme motion. Inspired by the human visual system, brain-inspired vision sensors introduce temporally dense information to alleviate this problem. However, event cameras still suffer from event rate saturation under rapid motion, while the event modality entangles edge features and motion cues, which limits their effectiveness. As a recent breakthrough, the complementary vision sensor (CVS), Tianmouc, captures synchronized RGB frames together with high-frame-rate, multi-bit spatial difference (SD, encoding structural edges) and temporal difference (TD, encoding motion cues) data within a single RGB exposure, offering a promising solution for RGB deblurring under extreme dynamic scenes. To fully leverage these complementary modalities, we propose Spatio-Temporal Difference Guided Deblur Net (STGDNet), which adopts a recurrent multi-branch architecture that iteratively encodes and fuses SD and TD sequences to restore structure and color details lost in blurry RGB inputs. Our method outperforms current RGB or event-based approaches in both synthetic CVS dataset and real-world evaluations. Moreover, STGDNet exhibits strong generalization capability across over 100 extreme real-world scenarios. Project page: https://tmcDeblur.github.io/

• The paper introduces STGDNet, which leverages CVS data to decouple motion and structure for accurate deblurring.
• It employs multi-branch encoders and a temporal recurrent refinement module to achieve state-of-the-art PSNR and SSIM performance.
• The study uses a real-captured SportsSloMo-CVS dataset, demonstrating robust generalization across varied exposure conditions.

Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor: An In-Depth Analysis

Introduction

Motion deblurring remains a critical open problem, especially under high dynamic range scenarios and large, non-linear motion where traditional RGB-based methods are fundamentally ill-posed. The paper "Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor" (2604.10554) introduces a new approach, leveraging the Complementary Vision Sensor (CVS), which synchronously acquires dense spatio-temporal signals along with RGB data. This work addresses fundamental limitations in conventional RGB and brain-inspired sensors, proposing a tailored neural architecture (STGDNet) for multi-modal fusion and deblurring, and systematically evaluates its strengths over alternative sensor paradigms.

Background and Motivation

Most deblurring pipelines—whether prior-driven, kernel-based, or data-driven—lack direct access to intra-exposure motion cues, forcing implicit inference from insufficient inputs. Event cameras and neuromorphic sensors have been deployed to overcome this, but they suffer from event rate saturation, entanglement of motion and edge features, and hardware alignment complexities. The recently developed Tianmouc CVS provides time-aligned multi-bit spatial difference ($\mathcal{SD}$) and temporal difference ($\mathcal{TD}$) streams, physically decoupling structural and motion information, and thus establishing a more tractable foundation for deblurring in real-world high-speed scenes.

The STGDNet Architecture

The proposed Spatio-Temporal Difference Guided Deblur Net (STGDNet) operates on three aligned modalities: a blurry RGB frame, a spatial difference snapshot (midpoint of RGB exposure), and a high-frame-rate sequence of temporal difference signals. The network features:

• Multi-Branch Encoders: Dedicated branches for RGB, $\mathcal{SD}$, and each $\mathcal{TD}_i$, extracting complementary features.
• Temporal Recurrent Refinement Module (TRRM): A recurrent encoder-decoder structure that sequentially incorporates $\mathcal{TD}$ slices, progressively refining the latent representation and, critically, tracking temporal evolution across the exposure.
• Cross-Modal Complementary Fusion (CCF): Hierarchical attention-based fusion mechanisms inject motion (TD) and structure (SD) cues at multiple scales within each TRRM step.
• Supervised Attention Module (SAM): Refines the deblurring residuals through spatial attention, emphasizing correlation with blurred regions.

This design ensures both flexibility to variable numbers of $\mathcal{TD}$ frames (driven by exposure length) and robust alignment between modalities.

Dataset Construction and Experimental Protocol

The authors introduce SportsSloMo-CVS, a large-scale, real-captured and pixel-aligned dataset produced using a DMD-based illumination pipeline, generating realistic motion-blur and corresponding CVS modalities from high-fps RGB sources. This approach circumvents domain gaps plaguing synthetic datasets. Evaluation extends over 100+ complex, real-world scenes, and includes comparison to competitive RGB, event-based, and prior CVS fusion methods, retrained as necessary for compatibility.

Quantitative and Qualitative Results

STGDNet shows consistent superiority in both PSNR and SSIM across all tested exposure times. For instance, at the most challenging blur (N=11, 14,520 µs), STGDNet attains 40.12 dB PSNR and 0.9874 SSIM, surpassing state-of-the-art event-based (EFNet 39.37/0.9847) and RGB-only (Restormer 31.35/0.9186) baselines. Notably, augmentation of RGB cues with $\mathcal{SD}$ and $\mathcal{TD}$ channels in otherwise-competitive backbones (e.g., Restormer*) yields improvement, but does not match the efficacy of disciplined multi-modal fusion. Diffusion-based CVS approaches, while principled, exhibit significant color artifacts and computational expense.

Qualitative comparisons further reveal that event-guided methods are fragile under extreme motion due to event rate loss, producing residual ghosting and color blends, whereas STGDNet delivers color-faithful and structurally sharp reconstructions. Ablation analysis demonstrates that both $\mathcal{SD}$ and $\mathcal{TD}$ are indispensable; their removal severely degrades metrics.

Theoretical and Practical Implications

The architecture exploits the hardware-level disentanglement of motion and structure perceived directly in the CVS. This stands in contrast with event cameras, where such cues are inherently entangled, complicating algorithmic separation and fusion. The explicit use of physical difference data—free from blur and aligned with RGB in time—eliminates much of the ambiguity that plagues RGB- or event-only learning. Furthermore, the recurrent fusion scheme is naturally extensible to arbitrary exposure lengths, and the network generalizes robustly with no fine-tuning even on real-captured scenes far outside the training set.

The CVS pipeline unifies deblurring, temporal upsampling, and exposure-invariant restoration in a single coherent learning framework. The authors explicitly demonstrate the extension of single-image deblurring to video frame recovery within the exposure window: by selecting alternative $\mathcal{TD}$0 frames, STGDNet can predict the scene at multiple, temporally offset moments—essentially converting a single blurry image into a temporally consistent sharp sequence.

Performance Boundaries and Generalization

Systematic boundary analyses using controlled disk-rotation and illumination experiments show that residual blur increases with scene dynamics, but, provided CVS signal is available and exposure conditions are reasonable, restoration is reliable and exposure-invariant. A Mean Relative Blurred Edge Width (BEW) metric quantifies this. Sequence augmentation methods facilitate generalization to arbitrary, non-integer exposure durations.

Limitations and Future Directions

Practical deployment remains tied to the availability and maturity of CVS hardware, which is not yet mainstream. Furthermore, although spatio-temporal modality fusion proves robust, real-world noises, sensor artifacts, and edge-case saturations (e.g., in extremely low light) may require adaptive denoising or domain adaptation. The diffusion of CVS-based designs into mobile, industrial, and scientific imaging will drive further investigation into even more efficient, scalable, and explainable fusion architectures. Potential extensions include event/video super-resolution, high-dynamic-range fusion, and generalized cross-sensor image restoration.

Conclusion

This work establishes a new benchmark in motion deblurring by utilizing the unique complementary pathways of the CVS for robust, exposure-adaptive scene restoration. The STGDNet framework combines theoretical clarity in sensor-cue separation with practical superiority on both synthetic and real scene datasets, evidencing both robust generalization and significant performance margin over established methods. The implications are broad for the future of high-dynamic, machine-perception pipelines wherever dynamic blur is a limiting factor (2604.10554).