SNNSIR: Spike-Driven Stereo Restoration

• SNNSIR is a fully spike-driven spiking neural network designed for stereo image restoration under conditions like rain, raindrops, low light, and low resolution.
• It utilizes a coarse-to-fine two-stage framework with spike-compatible modules (SRBB, SSCM, SSCA, and SSRB) to extract features and refine restoration efficiently.
• The architecture significantly reduces computational and energy cost, achieving competitive restoration quality compared to traditional ANN-based methods.

Searching arXiv for SNNSIR and closely related stereo restoration SNN work. SNNSIR is a fully spike-driven Spiking Neural Network (SNN) for stereo image restoration, introduced as “SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration” (Xu et al., 17 Aug 2025). It is designed to restore degraded stereo image pairs under rain streaks, raindrops, low light, and low resolution while maintaining low computational and energy cost. The method is organized as a coarse-to-fine two-stage framework and is built around spike-compatible modules that avoid operations such as floating-point matrix division, exponentiation, and sigmoid activations, which the authors identify as incompatible with the binary and event-driven nature of SNNs (Xu et al., 17 Aug 2025). Its central claim is that stereo restoration can be performed with competitive restoration quality and substantially reduced energy consumption when stereo interaction, nonlinearity, and refinement are redesigned for the spike-driven setting.

1. Scope and problem formulation

Stereo image restoration takes a degraded left-right image pair as input and reconstructs restored outputs for both views (Xu et al., 17 Aug 2025). Relative to single-image restoration, the stereo setting can exploit cross-view complementary information: if one view is occluded or corrupted, the other may preserve missing detail, and the geometric relation between views can support more reliable recovery of structure and texture. The paper places this setting in the context of downstream tasks such as stereo matching, depth estimation, 3D object detection/tracking, robot vision, and embodied intelligence (Xu et al., 17 Aug 2025).

SNNSIR addresses four stereo restoration tasks: rain streak removal, raindrop removal, low-light enhancement, and super-resolution (Xu et al., 17 Aug 2025). The design objective is not only restoration accuracy but also low-power deployment. This emphasis follows from the observation that stereo restoration is computationally heavy and that conventional ANN-based systems often rely on expensive floating-point operations and complex attention or fusion modules (Xu et al., 17 Aug 2025).

A common misconception is that any SNN-based restoration network is inherently spike-driven. SNNSIR explicitly rejects that equivalence. The paper argues that prior hybrid SNN-ANN approaches may still rely on ANN-style nonlinearities and thus do not fully preserve the binary, sparse, event-driven computational model associated with SNNs (Xu et al., 17 Aug 2025).

2. Spike-driven design rationale

The architectural premise of SNNSIR is that SNNs communicate through binary spikes and use sparse event-driven computation, which can greatly reduce energy consumption and improve compatibility with low-power and neuromorphic hardware (Xu et al., 17 Aug 2025). The paper adopts the spike-driven paradigm in a strict sense: the network should rely on spike-compatible operations and avoid ANN-style nonlinearities and dense floating-point components as much as possible (Xu et al., 17 Aug 2025).

The paper identifies three limitations in prior hybrid SNN-ANN stereo restoration methods. First, they may include non-spike-compatible operations such as floating-point matrix division, exponentiation, sigmoid activations, and related ANN-style nonlinearities. Second, they often provide limited nonlinear representation, which is consequential because binary spikes require architectural compensation to recover expressive power. Third, stereo interaction is often handled by expensive dual-branch encoders, guided alignment, parallax attention, or similarly costly fusion mechanisms (Xu et al., 17 Aug 2025). SNNSIR is presented as a response to this combination of computational and representational constraints.

This suggests that the contribution is as much about operational closure under spike-compatible primitives as about absolute restoration performance. A plausible implication is that SNNSIR should be interpreted not merely as an SNN baseline, but as a design study in how stereo restoration components can be reformulated without reintroducing ANN-specific operators.

3. Network organization and spiking dynamics

SNNSIR uses a coarse-to-fine two-stage framework (Xu et al., 17 Aug 2025). Stage 1 performs degradation removal or coarse restoration with a U-shaped encoder-decoder that carries out multi-scale stereo feature extraction and restoration. Stage 2 is a lightweight full-resolution refinement stage that removes artifacts and recovers fine details (Xu et al., 17 Aug 2025). The main architectural components are the Spike Residual Basic Block (SRBB), Spike Stereo Convolutional Modulation (SSCM), Spike Stereo Cross-Attention (SSCA), and Spike Stereo Refinement Block (SSRB) (Xu et al., 17 Aug 2025).

The network employs a Leaky Integrate-and-Fire neuron, and static images are replicated over time into a sequence of length $T$ (Xu et al., 17 Aug 2025). The paper gives the spiking dynamics as

$\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$

$\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$

$$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$

$\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$

where $\mathbf{X}_i[t]$ is the input at time step $t$, $\mathbf{V}_i[t-1]$ is the previous membrane potential, $\mathbf{U}_i[t]$ is the charged membrane potential, $u_{\text{th}$ is the firing threshold, $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$0 is the binary output spike, $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$1 is the reset potential, and $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$2 is the membrane time constant (Xu et al., 17 Aug 2025).

For left and right images $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$3, temporal replication produces $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$4, followed by a $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$5 convolution to extract shallow features $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$6 (Xu et al., 17 Aug 2025). The resulting features are processed through Feature Extraction Blocks built from SRBBs, SSCM, downsampling encoder stages, SSCA at deeper levels, a decoder with skip connections, and output residual prediction; the coarse restored output is then added to the original input (Xu et al., 17 Aug 2025). The second stage uses four SSRBs with 32 channels at full resolution (Xu et al., 17 Aug 2025).

The paper also defines two spike-compatible convolutional primitives:

$\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$7

and

$\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$8

where SCU denotes Spike Convolution Unit, SDU denotes Spike Depth-wise Convolution Unit, and tdBN denotes temporal-dimension batch normalization (Xu et al., 17 Aug 2025).

4. Core modules

SRBB is the basic local feature extraction unit. It replaces standard convolution layers in a residual block with $\mathbf{U}_{i}[t]=\mathbf{V}_{i}[t-1]+\frac{1}{\tau}(\mathbf{X}_{i}[t]-(\mathbf{V}_{i}[t-1]-u_{\text {rest}))$9 SCUs and uses a Membrane Shortcut to preserve identity mapping in spiking networks (Xu et al., 17 Aug 2025). Two SRBBs are stacked to form a Feature Extraction Block (Xu et al., 17 Aug 2025). The stated purpose is to preserve identity mapping, support deep residual learning in SNNs, and maintain efficient local feature extraction within a spike-compatible design (Xu et al., 17 Aug 2025).

SSCM is introduced to address weak nonlinearity in SNNs and limited stereo-aware interaction (Xu et al., 17 Aug 2025). Instead of sigmoid or related ANN activations, it uses element-wise multiplication as a spike-compatible nonlinear substitute. Given concatenated left-right features $\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$0, the paper defines channel modulation as

$\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$1

spatial modulation as

$\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$2

and residual modulation on each view as

$\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$3

Here, $\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$4 is global average pooling, $\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$5 is global max pooling, $\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$6 is a shared linear transform, and $\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$7 and $\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$8 denote element-wise multiplication (Xu et al., 17 Aug 2025). The functional interpretation given in the paper is that channel modulation selects important feature channels, spatial modulation highlights degraded regions, and the residual form supports stable learning (Xu et al., 17 Aug 2025).

SSCA is the stereo correspondence module. Its premise is that stereo geometry mainly has horizontal disparity rather than vertical disparity, so attention is computed efficiently along the horizontal direction (Xu et al., 17 Aug 2025). The paper gives the update equations

$\mathbf{S}_i[t] = {\theta}(\mathbf{U}_i[t] - u_{\text{th})$9

and

$$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$0

The module uses $$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$1 SCUs for projection and avoids activation functions and softmax-based attention normalization (Xu et al., 17 Aug 2025). Its role is to model stereo correspondence, transfer details between views, and help recover structures missing or corrupted in one image (Xu et al., 17 Aug 2025).

SSRB is the refinement component used in the second stage. It includes Spike Separable Convolution inspired by inverted separable convolution, concatenation of both views, SCU-based fusion, element-wise multiplication for branch modulation, and a Membrane Shortcut for residual refinement (Xu et al., 17 Aug 2025). Its stated purpose is to refine textures, edges, and residual artifacts after coarse restoration (Xu et al., 17 Aug 2025).

5. Optimization, energy model, and implementation setting

SNNSIR uses different losses in its two stages (Xu et al., 17 Aug 2025). Stage 1 uses $$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$2 loss:

$$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$3

to encourage pixel-level accuracy in coarse restoration (Xu et al., 17 Aug 2025). Stage 2 uses perceptual loss:

$$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$4

The paper notes slightly inconsistent subscripts in this formula, but states that the intent is to match perceptual features from multiple layers in order to improve visual quality and structural consistency (Xu et al., 17 Aug 2025). The total loss is

$$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$5

The efficiency argument is formalized through an energy model. The paper distinguishes ANN compute cost in FLOPs, mostly multiply-accumulate operations, from SNN compute cost in synaptic operations, mostly accumulate operations (Xu et al., 17 Aug 2025). Under 45nm hardware assumptions, the costs are given as $$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$6 pJ for MAC and $$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$7 pJ for AC, and the total energy is defined by

$$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$8

where $$\theta(x)= \begin{cases} 1, & x\ge 0\ 0, & x<0 \end{cases}$$9 denotes spike or binary data and $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$0 denotes floating-point data (Xu et al., 17 Aug 2025). This model is used to support the claim that SNNSIR minimizes floating-point computation and keeps most of the pipeline spike-driven.

Implementation details reported in the paper include SpikingJelly as the framework, NVIDIA RTX 3090 for training, AdamW with $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$1, $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$2, weight decay $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$3, learning rate $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$4, and default time steps $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$5 (Xu et al., 17 Aug 2025). The spike threshold is $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$6 for low-light enhancement and $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$7 for the other tasks (Xu et al., 17 Aug 2025). Evaluation uses PSNR, SSIM, MS-SSIM, LPIPS, and efficiency metrics including parameters, FLOPs, SOPs, energy consumption, and runtime (Xu et al., 17 Aug 2025).

6. Experimental results and ablations

The paper evaluates SNNSIR on Stereo Waterdrop for raindrop removal; RainKT12 and RainKT15 for rain streak removal; Middlebury and Holopix50k for low-light enhancement; and Middlebury and Flickr1024 for super-resolution (Xu et al., 17 Aug 2025).

For stereo raindrop removal on Stereo Waterdrop, SNNSIR reports PSNR $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$8, MS-SSIM $\mathbf{V}_i[t] = (1 - \mathbf{S}_i[t])\mathbf{U}_i[t] + \mathbf{S}_i[t]u_{\text{rest}$9, LPIPS $\mathbf{X}_i[t]$0, parameters $\mathbf{X}_i[t]$1M, and energy $\mathbf{X}_i[t]$2 mJ (Xu et al., 17 Aug 2025). Compared with Restormer, the paper reports PSNR $\mathbf{X}_i[t]$3, parameters $\mathbf{X}_i[t]$4M $\mathbf{X}_i[t]$5M, and energy $\mathbf{X}_i[t]$6 mJ $\mathbf{X}_i[t]$7 mJ, corresponding to an $\mathbf{X}_i[t]$8 parameter reduction and a $\mathbf{X}_i[t]$9 energy reduction (Xu et al., 17 Aug 2025). For rain streak removal, SNNSIR reports $t$0 on RainKT12 and $t$1 on RainKT15, in PSNR/SSIM format, with energy $t$2 mJ (Xu et al., 17 Aug 2025). For low-light enhancement, it reports $t$3 on Middlebury and $t$4 on Holopix50k, and the paper states that it uses only about $t$5 of DRBN’s energy (Xu et al., 17 Aug 2025). For super-resolution, it reports $t$6 on Middlebury and $t$7 on Flickr1024, with energy $t$8 mJ (Xu et al., 17 Aug 2025).

The paper compares against ANN-based baselines including Pix2Pix, Qian et al., Liu et al., Quan et al., DRCDNet, Restormer, RAttNet, DID-MDN, DeHRain, ZeroDCE, RetinexNet, and DRBN, as well as SNN-based baselines including Spikformer, Spike-driven Transformer, Meta-SpikeFormer, E-SpikeFormer, and ESDNet (Xu et al., 17 Aug 2025). The reported interpretation is that SNNSIR improves restoration quality, lowers energy, reduces parameters, and outperforms existing SNN baselines across tasks (Xu et al., 17 Aug 2025).

The ablation study isolates the contributions of the main modules. The paper reports the following results on the studied setting:

Configuration	Result
SEW-RBB	23.03 PSNR
SRBB	24.96 PSNR
SRBB + SSCM	25.91 PSNR
SRBB + SSCA	25.10 PSNR
SRBB + SSCA + SSRB	25.78 PSNR
SRBB + SSCM + SSCA + SSRB	26.57 PSNR / 0.903 SSIM

These numbers support several specific conclusions. SRBB improves over SEW-RBB while keeping the same parameter count (Xu et al., 17 Aug 2025). SSCM adds a substantial gain over SRBB alone (Xu et al., 17 Aug 2025). SSCA contributes to stereo interaction, and SSRB further improves fine-grained restoration (Xu et al., 17 Aug 2025). The full configuration is the best-performing variant in the table (Xu et al., 17 Aug 2025).

The paper also compares spike-compatible modulation with a sigmoid-based variant: SRBB + SSCM$t$9 + SSCA + SSRB gives $\mathbf{V}_i[t-1]$0 PSNR, whereas the fully spike-compatible version gives $\mathbf{V}_i[t-1]$1 PSNR (Xu et al., 17 Aug 2025). In addition, replacing spike neurons with ReLU and adding activations to form SNNSIR-ANN yields $\mathbf{V}_i[t-1]$2 PSNR and $\mathbf{V}_i[t-1]$3 mJ, compared with SNNSIR-SNN at $\mathbf{V}_i[t-1]$4 PSNR and $\mathbf{V}_i[t-1]$5 mJ (Xu et al., 17 Aug 2025). The reported interpretation is that the SNN version is slightly better in quality and dramatically more energy efficient (Xu et al., 17 Aug 2025).

Time-step analysis reports $\mathbf{V}_i[t-1]$6: $\mathbf{V}_i[t-1]$7 PSNR, $\mathbf{V}_i[t-1]$8: $\mathbf{V}_i[t-1]$9 PSNR, $\mathbf{U}_i[t]$0: $\mathbf{U}_i[t]$1 PSNR, and $\mathbf{U}_i[t]$2: $\mathbf{U}_i[t]$3 PSNR, leading the paper to identify $\mathbf{U}_i[t]$4 as the best trade-off (Xu et al., 17 Aug 2025). This suggests that temporal dynamics are beneficial but do not improve monotonically with increasing time horizon.

7. Interpretation, significance, and limitations of scope

The paper characterizes SNNSIR as the first dedicated high-performing baseline for stereo image restoration using SNNs and as a fully spike-driven architecture that preserves stereo interaction without resorting to ANN-style attention and fusion mechanisms (Xu et al., 17 Aug 2025). Its reported significance lies in demonstrating that spike-compatible residual learning, spike-compatible modulation, stereo cross-attention without softmax normalization, and lightweight refinement can jointly support high-quality stereo restoration at much lower energy cost (Xu et al., 17 Aug 2025).

The paper’s analysis of spike firing behavior reinforces this interpretation. It reports structured firing maps in which spikes are concentrated in degraded or informative areas, with low and high activation regions alternating cyclically; in deraining, spikes align with raindrop regions, whereas in low-light enhancement firing is weaker overall, which the paper suggests may explain why SNNs are less naturally suited to very dark inputs (Xu et al., 17 Aug 2025). This suggests that the network’s sparsity pattern is task-dependent and semantically organized rather than incidental.

At the same time, the claims should be read within the scope of the reported benchmarks and hardware-energy assumptions. The energy figures rely on the 45nm MAC and AC costs specified in the paper (Xu et al., 17 Aug 2025). Likewise, the significance claims are benchmark-relative: the reported superiority concerns the datasets, baselines, and evaluation configuration included in the study (Xu et al., 17 Aug 2025). Within that scope, SNNSIR is presented as evidence that stereo restoration can be reformulated as a fully spike-driven problem without sacrificing competitive restoration quality.

In summary, SNNSIR denotes a fully spike-driven SNN for stereo image restoration whose defining features are SRBB-based residual extraction, SSCM-based spike-compatible modulation, SSCA-based stereo correspondence, and SSRB-based refinement in a two-stage coarse-to-fine pipeline (Xu et al., 17 Aug 2025). Its main technical message is that binary, event-driven computation can be retained throughout stereo restoration if the core operators are redesigned for spike compatibility rather than inherited from ANN architectures.