Spike Stereo Convolutional Modulation (SSCM)

• SSCM is a spike-compatible stereo-aware modulation module that fuses left/right features using pooling and element-wise operations to improve image restoration.
• It employs sequential channel and spatial modulation to highlight noise-sensitive regions and enable cross-view feature recalibration without relying on ANN functions.
• Its integration within SNNSIR yields measurable PSNR/SSIM improvements with low-energy consumption, demonstrating practical benefits in stereo image restoration.

Searching arXiv for the cited papers to ground the article. Spike Stereo Convolutional Modulation (SSCM) is a module introduced within SNNSIR, a fully spike-driven network for stereo image restoration, to provide stereo-aware nonlinear feature recalibration without relying on ANN-style operations such as sigmoid, softmax, exponentiation, or matrix division/normalization (Xu et al., 17 Aug 2025). It is positioned as a response to a central constraint in spiking restoration architectures: binary spiking neurons are computationally efficient but have limited expressive power, especially for restoration tasks that require subtle degradation modeling, local detail recovery, and cross-view interaction. In SNNSIR, SSCM operates on left-right stereo features after early single-view extraction, fuses them into a stereo-aware representation, and applies channel and spatial modulation through pooling, shared linear transforms, an SCU, element-wise multiplication, and residual refinement (Xu et al., 17 Aug 2025). Its intended role is not explicit disparity estimation, but spike-compatible nonlinear stereo-aware feature selection that highlights noise-sensitive regions and allows each view to benefit from information present in the other.

1. Problem setting and motivation

SSCM was proposed for stereo image restoration, where the input is a degraded left-right image pair and the output is a clean, geometrically consistent stereo pair (Xu et al., 17 Aug 2025). The paper studies four restoration settings: stereo raindrop removal, stereo rain streak removal, stereo low-light enhancement, and stereo super-resolution (Xu et al., 17 Aug 2025). The stereo setting is consequential because one view may preserve scene details that are degraded, occluded, or missing in the other; therefore, a successful model requires cross-view interaction rather than independent left-right processing.

The design motivation is tightly coupled to the fully spike-driven objective of SNNSIR. In the authors’ formulation, after the initial and final standard convolutions, the network should mainly process binary spike sequences produced by spiking neurons using event-driven, low-cost operations (Xu et al., 17 Aug 2025). This requirement is presented as more than a formal architectural preference: prior hybrid SNN-ANN restoration models are criticized for retaining floating-point division, exponentiation, sigmoid, or conventional attention normalization, all of which are described as incompatible with the binary and event-driven nature of SNNs and as undermining their energy and hardware advantages (Xu et al., 17 Aug 2025).

A second motivation is representational. Binary activations improve efficiency but reduce expressiveness relative to continuous ANN activations, which is particularly problematic in image restoration, where fine-grained degradations and view-complementary cues must be encoded (Xu et al., 17 Aug 2025). Within SNNSIR, this limitation is addressed by two main mechanisms: SRBB for spike-compatible residual information flow and SSCM for stronger nonlinear representation and feature selection (Xu et al., 17 Aug 2025). The paper explicitly states that SSCM introduces simplified nonlinearity through element-wise multiplication and highlights noise-sensitive regions via cross-view-aware modulation (Xu et al., 17 Aug 2025). This identifies SSCM as a stereo-aware modulation block rather than merely an attention substitute.

2. Placement within SNNSIR

SNNSIR adopts a coarse-to-fine architecture (Xu et al., 17 Aug 2025). In the first stage, the network uses a 5-layer U-shaped encoder-decoder with channel progression $[32, 64, 96, 128, 160]$ (Xu et al., 17 Aug 2025). Static left and right images are temporally replicated over $T$ time steps, shallow features are extracted, and the network then applies a sequence comprising FEB, SSCM, downsampling and deeper encoding, SSCA, decoding with skip connections, and final residual prediction for coarse restoration (Xu et al., 17 Aug 2025). A second lightweight refinement stage based on SSRBs then improves local details (Xu et al., 17 Aug 2025).

SSCM is placed early in the encoder, immediately after the Feature Extraction Block (FEB) and before deeper encoding and SSCA (Xu et al., 17 Aug 2025). The paper states that features are processed by FEB for single-view feature extraction, followed by SSCM, which introduces nonlinearity and highlights noise-sensitive regions (Xu et al., 17 Aug 2025). This placement is structurally important: SSCM operates before the deeper stereo correspondence machinery, suggesting that its function is early stereo-aware recalibration rather than late-stage explicit matching.

The relation among SRBB, SSCM, and SSCA is differentiated in the text. SRBB and FEB perform local single-view spatiotemporal feature extraction while preserving information flow through residual learning; SSCM performs cross-view-guided modulation, adding spike-compatible nonlinearity and emphasizing important restoration regions; SSCA then performs more explicit long-range cross-view interaction for correspondence reasoning, especially across horizontal disparities (Xu et al., 17 Aug 2025). This division of labor suggests a progression from local feature extraction, to stereo-aware modulation, to deeper bidirectional stereo interaction.

3. Architectural formulation

The module operates on left and right features after FEB, denoted as

$$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$

SSCM first forms a fused stereo representation $\mathbf{F}$ by concatenating the left and right view features (Xu et al., 17 Aug 2025). The paper states that SSCM takes the concatenated left and right view features as input to form $\mathbf{F}$, although the exact post-concatenation fusion operator is not fully specified mathematically (Xu et al., 17 Aug 2025). A faithful reading is therefore that concatenation provides a stereo-aware tensor that is subsequently used for modulation.

The module consists of two sequential submodules: Spike Channel Modulation (SCM) and Spike Spatial Modulation (SSM) (Xu et al., 17 Aug 2025). SCM uses global average pooling and global max pooling on the fused tensor, applies a shared linear transform $\mathbf{W}(\cdot)$ to both pooled descriptors, sums the results, and multiplicatively applies the resulting channel descriptor to the feature tensor (Xu et al., 17 Aug 2025). SSM again uses average and max pooling, but in a spatially arranged form, concatenates the pooled maps, applies an SCU, and multiplies the resulting spatial modulation map with the feature tensor (Xu et al., 17 Aug 2025). Both branches use element-wise multiplication as the core nonlinear operation.

The exact equations given by the paper are

$$\begin{aligned} \text{SCM}(\mathbf{F})&=\mathbf{F}\circledast(\mathbf{W}(\text{GAP}(\mathbf{F})) + \mathbf{W}(\text{GMP}(\mathbf{F}))),\ \text{SSM}(\mathbf{F})&=\mathbf{F}\odot(\text{SCU}([\text{GAP}(\mathbf{F}),\text{GMP}(\mathbf{F})])),\ \mathbf{F}^{'}_{l/r}&=\text{SSM}(\mathbf{F})\odot\mathbf{F}_{l/r} + \mathbf{F}_{l/r}, \end{aligned} \tag{5}$$

where $\text{GAP}$ is global average pooling, $\text{GMP}$ is global max pooling, $\mathbf{W}$ is a shared linear operation, $T$0 denotes channel-wise element-wise multiplication, and $T$1 denotes spatial-wise element-wise multiplication (Xu et al., 17 Aug 2025).

The final equation is especially significant. The stereo modulation tensor derived from the fused left-right representation is not treated as an output in itself; rather, it is used to modulate each original branch separately and then combined with a residual connection (Xu et al., 17 Aug 2025). Thus, SSCM can be understood as a shared stereo-aware guidance branch feeding two view-specific refinement branches. The outputs are refined features $T$2 and $T$3 with the same per-view structure as the original features.

The paper states that SSCM sequentially applies SCM and SSM, although the printed equations write both as functions of $T$4 directly rather than explicitly feeding SCM into SSM (Xu et al., 17 Aug 2025). The textual description therefore supports a sequential interpretation, while the algebraic presentation leaves the implementation path partially open. This is one of the few points where a strict reproduction from text alone requires interpretive caution.

4. Spike compatibility and temporal embedding

SSCM is designed to remain compatible with spike-driven computation. The module relies on pooling, linear or convolutional transforms, SCU, element-wise multiplication, and residual addition, while explicitly avoiding sigmoid, softmax, exponentiation, and matrix division/normalization (Xu et al., 17 Aug 2025). This design principle aligns with the broader SNNSIR objective of minimizing reliance on non-spiking floating-point operators.

The surrounding neuron model is a leaky integrate-and-fire formulation: $T$5

$T$6

$T$7

$T$8

with the usual interpretation of membrane potential, thresholding, and reset variables, despite minor typographic truncation in the manuscript (Xu et al., 17 Aug 2025).

The Spike Convolution Unit is defined as

$T$9

and the Spike Depth-wise Convolution Unit as

$$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$0

(Xu et al., 17 Aug 2025). Because SSCM’s spatial modulation branch explicitly uses an SCU, the module is temporally embedded within the spiking dynamics of the network rather than behaving as a detached floating-point attention block.

Static input images are replicated over time into

$$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$1

and SSCM’s internal feature tensors therefore also evolve over $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$2 time steps (Xu et al., 17 Aug 2025). The paper does not provide a separate per-time-step SSCM equation, but the module operates on time-indexed spike feature tensors, and its SCU is itself time-aware through the LIF-convolution-tdBN composition (Xu et al., 17 Aug 2025). This suggests that SSCM’s modulation is not solely channel-wise or spatial-wise; it is also embedded in the evolving event-driven dynamics of the spiking representation.

5. Functional role in stereo restoration

The principal functional role of SSCM is stereo-aware feature recalibration. By fusing left and right features into a shared representation $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$3, the module derives modulation signals that are informed by both views and then applies them back to each branch individually (Xu et al., 17 Aug 2025). In effect, the left feature is enhanced using guidance from the joint left-right representation, and the same is true for the right feature. This enables one view to assist the other when one branch contains clearer scene structure or less severe degradation.

The paper states that the fused representation enables the network to leverage global context for compensating view-specific degradations caused by noise or occlusion (Xu et al., 17 Aug 2025). The intended application scenarios include cases where a raindrop occludes one view more heavily, streak density differs between views, low-light visibility is asymmetric, or local textures required for super-resolution are better preserved in one branch than the other (Xu et al., 17 Aug 2025). In such settings, stereo fusion functions as a restoration prior rather than as a purely geometric matching process.

SSCM is repeatedly described as highlighting noise-sensitive regions and guiding the network to focus on degraded regions (Xu et al., 17 Aug 2025). Mechanistically, the channel modulation stage emphasizes degradation-relevant feature channels, while the spatial modulation stage emphasizes restoration-critical locations (Xu et al., 17 Aug 2025). The paper further reports, in its SFR analysis, that SNNSIR’s high-activation regions align with raindrop points, which is presented as evidence that the spiking activity is concentrated around degradations (Xu et al., 17 Aug 2025).

At the same time, SSCM is not presented as a disparity estimator or explicit alignment mechanism. It performs implicit cross-view conditioning through fused modulation rather than explicit stereo matching (Xu et al., 17 Aug 2025). This distinguishes it from SSCA, which is described as the deeper-stage module for explicit bidirectional cross-attention and stronger correspondence reasoning across views (Xu et al., 17 Aug 2025). A plausible implication is that SSCM handles early degradation-aware feature enhancement, while SSCA addresses more structured stereo correspondence dependencies.

6. Empirical evidence and efficiency profile

The most direct evidence for SSCM appears in the ablation study labeled “Impact of Different Designs” (Xu et al., 17 Aug 2025). The baseline row “SRBB” reports $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$4M parameters, $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$5 PSNR, and $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$6 SSIM, whereas “SRBB+SSCM” reports $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$7M parameters, $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$8 PSNR, and $$\mathbf{F}_l,\mathbf{F}_r \in \mathbb{R}^{T \times C \times H \times W}.$$9 SSIM (Xu et al., 17 Aug 2025). This corresponds to a gain of $\mathbf{F}$0 dB PSNR and $\mathbf{F}$1 SSIM for an increase of only $\mathbf{F}$2M parameters (Xu et al., 17 Aug 2025). The paper interprets this as evidence that the nonlinearity introduced by SSCM is effective after removing conventional activations from the SNN (Xu et al., 17 Aug 2025).

A second ablation compares a sigmoid-based variant against the spike-compatible SSCM design. “SRBB+SSCM($\mathbf{F}$3)+SSCA+SSRB” reports $\mathbf{F}$4 PSNR and $\mathbf{F}$5 SSIM, while “SRBB+SSCM+SSCA+SSRB” reports $\mathbf{F}$6 PSNR and $\mathbf{F}$7 SSIM (Xu et al., 17 Aug 2025). The paper states that activation functions involving exponential operations are contrary to the characteristics of spike-driven computation (Xu et al., 17 Aug 2025). The comparison therefore indicates that the module’s spike-compatible formulation is not only motivated by hardware efficiency but is also empirically preferable within this architecture.

The paper does not provide a standalone FLOP or SOP breakdown for SSCM, but its lightweight character is inferred from its use of pooling, linear transform, SCU, and element-wise multiplication rather than transformer-style dense attention with softmax (Xu et al., 17 Aug 2025). At the full-model level, SNNSIR is reported on Stereo Waterdrop with $\mathbf{F}$8 G FLOPs, $\mathbf{F}$9 G SOPs, and $\mathbf{F}$0 mJ energy, compared with $\mathbf{F}$1 mJ for Restormer, $\mathbf{F}$2 mJ for RAttNet, and $\mathbf{F}$3 mJ for ESDNet (Xu et al., 17 Aug 2025). The paper further states that, relative to Restormer, SNNSIR reduces energy by $\mathbf{F}$4 while achieving slightly better PSNR, $\mathbf{F}$5 versus $\mathbf{F}$6 dB (Xu et al., 17 Aug 2025). Since SSCM is an internal module of this fully spike-driven system, these system-level results provide indirect support for its compatibility with low-power stereo vision goals.

The energy model used in the paper is

$\mathbf{F}$7

which links reduced reliance on floating-point operators directly to lower energy consumption (Xu et al., 17 Aug 2025). Within that framework, SSCM’s avoidance of ANN-style nonlinear modules is part of the network’s broader efficiency strategy.

7. Interpretation, implementation considerations, and relation to adjacent work

SSCM can be summarized as a stereo-aware, spike-compatible modulation block that bridges single-view spiking feature extraction and deeper stereo interaction (Xu et al., 17 Aug 2025). Its implementation-level reconstruction from the text is straightforward at a high level: input left and right spike feature sequences, concatenate them into a fused stereo tensor, apply a channel modulation branch based on global average and max pooling plus a shared linear transform, apply a spatial modulation branch based on pooled maps plus an SCU, and use the resulting stereo modulation tensor to refine each view through multiplicative residual enhancement (Xu et al., 17 Aug 2025). The default temporal setting in SNNSIR is $\mathbf{F}$8; the neuron model is LIF; training is implemented in SpikingJelly with surrogate gradients; and the U-shaped degradation-removal stage uses channels $\mathbf{F}$9, while the refinement stage uses 4 SSRBs with embedding dimension $\mathbf{W}(\cdot)$0 (Xu et al., 17 Aug 2025). Threshold settings are $\mathbf{W}(\cdot)$1 for low-light enhancement and $\mathbf{W}(\cdot)$2 for other tasks (Xu et al., 17 Aug 2025).

Several details remain only partially specified. The exact fusion operator after concatenation is not fully formalized, and tensor dimensionalities after pooling and modulation are not given in complete algebraic form (Xu et al., 17 Aug 2025). The printed equations also leave ambiguity as to whether SSM should consume $\mathbf{W}(\cdot)$3 directly or the output of SCM, although the text describes a sequential design (Xu et al., 17 Aug 2025). These are limitations of documentation rather than of the conceptual design itself.

In relation to neighboring spike stereo research, it is important to distinguish SSCM from broader “spike stereo” mechanisms. For example, SpikeStereoNet addresses stereo depth estimation from raw spike streams and does not define a module named SSCM (Gao et al., 26 May 2025). The closest functional correspondences in that work are the correlation volume construction module, local correlation and disparity feature processing inside the update block, and the RSNN-based update operator with convolutionally produced adaptive gates $\mathbf{W}(\cdot)$4 (Gao et al., 26 May 2025). That architecture fuses binocular information primarily through all-pairs correlation

$\mathbf{W}(\cdot)$5

then refines disparity iteratively through recurrent spiking dynamics (Gao et al., 26 May 2025). By contrast, SSCM in SNNSIR is a dedicated feedforward stereo modulation block for restoration, not a recurrent post-correlation refinement operator.

This contrast clarifies a potential misconception. “Stereo convolutional modulation” need not denote a generic left-right interaction mechanism in any spike-based stereo model. In the literature provided here, SSCM specifically refers to the SNNSIR module that performs early cross-view-guided channel and spatial recalibration under spike-compatible constraints (Xu et al., 17 Aug 2025). In other spike stereo systems, analogous functionality may instead be distributed across correlation construction, motion encoding, and recurrent gating, without any standalone SSCM block (Gao et al., 26 May 2025).