Spike Stereo Cross-Attention (SSCA) in SNNSIR

• SSCA is a spike-compatible cross-attention mechanism that exchanges encoded left and right features to recover degraded or missing details in stereo image restoration.
• It employs 1×1 SCU projections and matrix-product based horizontal attention to model long-range cross-view dependencies without using softmax or floating-point operations.
• Empirical evidence in SNNSIR shows that incorporating SSCA improves PSNR and supports its role in efficient, low-energy stereo restoration performance.

Spike Stereo Cross-Attention (SSCA) is the spike-compatible mechanism for explicit cross-view feature interaction in SNNSIR, a fully spike-driven spiking neural network for stereo image restoration. Its stated role is to let the left and right branches exchange information after single-view encoding, so that details degraded or missing in one view can be recovered from complementary content in the other while remaining compatible with the binary, event-driven constraints of a fully spike-driven SNN. Within SNNSIR, SSCA is introduced for stereo restoration tasks including rain streak removal, raindrop removal, low-light enhancement, and super-resolution, and is positioned as the module responsible for cross-view long-range dependencies and efficient bidirectional feature interaction across views (Xu et al., 17 Aug 2025).

1. Motivation and functional scope

SNNSIR studies stereo image restoration under the premise that stereo restoration is stronger than monocular restoration because “the model [can] recover scene details that are degraded or missing in one image using cues from its counterpart.” In that setting, cross-view interaction is central rather than auxiliary. SSCA is introduced specifically to model correspondence across left and right views in cases involving missing or heavily degraded regions in one view, geometric consistency, and long-range dependency modeling across views (Xu et al., 17 Aug 2025).

The paper situates SSCA against two constraints. First, recent stereo restoration methods use guided alignment, parallax attention, and semantic fusion, but these mechanisms are described as computationally heavy. Second, prior SNN restoration approaches are described as relying on hybrid SNN-ANN operations such as floating-point matrix division, exponentiation, and sigmoid, which are stated to be incompatible with the binary and event-driven nature of SNNs. SSCA is therefore framed as a lightweight, spike-compatible stereo interaction mechanism that preserves cross-view reasoning without leaving the spike-driven regime (Xu et al., 17 Aug 2025).

Within SNNSIR, SSCA has a distinct role relative to Spike Stereo Convolutional Modulation (SSCM). SSCM is described as introducing spike-compatible non-linear activation, enabling stereo-aware feature refinement through convolutional modulation, and highlighting noise-sensitive region; SSCA, by contrast, is described as further improving stereo correspondence through bidirectional feature interaction and modeling cross-view long-range dependencies (Xu et al., 17 Aug 2025).

Module	Stated role	Stated interaction type
SSCM	Nonlinearity and stereo-aware feature refinement	Convolutional modulation
SSCA	Stereo correspondence and cross-view long-range dependencies	Bidirectional cross-view interaction

2. Architectural placement in SNNSIR

From the method description and Figure 1, SNNSIR uses a 5-layer U-shaped encoder-decoder in its first stage. The processing order is stated as shallow feature extraction by a $3\times 3$ convolution, Feature Extraction Block (FEB) for single-view extraction, SSCM, progressive downsampling and encoding, SSCA, decoding with skip connections, output residual map generation, and a refinement stage with SSRBs. SSCA is therefore not part of the refinement stage; it is placed after encoding at the encoded-feature interaction point of the first-stage U-Net (Xu et al., 17 Aug 2025).

The stereo inputs are temporally replicated as

$$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$

and, after shallow convolution, become

$$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$

For SSCA itself, the paper writes its formulas using $\mathbf{F}_l$ and $\mathbf{F}_r$, and explicitly states that these feature maps are first reshaped into $H\times W\times C$ dimensions. The output features $\mathbf{F}'_l$ and $\mathbf{F}'_r$ are then reshaped back to $C\times H\times W$ after cross-attention computation and residual fusion (Xu et al., 17 Aug 2025).

This placement is consequential. SSCA operates on encoded features rather than raw inputs or late refinement features, so its function is to mediate inter-view correspondence after single-view representation learning has already occurred but before decoding reconstructs the restored outputs. A plausible implication is that SNNSIR separates local or modulation-oriented enhancement from explicit stereo exchange: SSCM enriches and modulates features earlier, whereas SSCA performs the explicit bidirectional transfer at the bottleneck (Xu et al., 17 Aug 2025).

3. Computational mechanism

The SSCA module is defined by the paper through the following equations: $$\begin{aligned} \mathbf{F}^{'}_l &= \mathbf{W}_l^3 \big(\mathbf{W}_l^1 \mathbf{F}_l \times (\mathbf{W}_r^1 \mathbf{F}_r)^T \times \mathbf{W}_r^2 \mathbf{F}_r\big) + \mathbf{F}_l,\ \mathbf{F}^{'}_r &= \mathbf{W}_r^3 \big((\mathbf{W}_l^1 \mathbf{F}_l \times (\mathbf{W}_r^1 \mathbf{F}_r)^T)^T \times \mathbf{W}_l^2 \mathbf{F}_l\big) + \mathbf{F}_r. \end{aligned} \tag{8}$$ Here $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$0 and $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$1 are encoded left and right feature maps, $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$2 and $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$3 are the refined outputs, $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$4 denotes transpose, $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$5 denotes matrix multiplication, and the learned weight matrices $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$6 are implemented by $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$7 SCUs (Xu et al., 17 Aug 2025).

The paper states that attention computation is performed in the horizontal direction because “horizontal disparities are present while there is no significant disparity in the vertical direction,” making horizontal attention more efficient. In operational terms, the left and right encoded features are reshaped from $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$8 into $$\mathbf{X}_l,\mathbf{X}_r \in \mathbb{R}^{T \times 3 \times H \times W},$$9, so that cross-view interaction is organized in a row-wise manner consistent with stereo geometry (Xu et al., 17 Aug 2025).

The module is explicitly bidirectional. The left branch computes an affinity term

$$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$0

and uses it to aggregate right-view content through $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$1, after which $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$2 projects the result before residual addition. The right branch uses the transpose of the same affinity term and symmetrically aggregates left-view content. The structure is therefore symmetric in interaction pattern but not weight-tied in the formula, since the left and right branches use separate learned matrices $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$3 and $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$4 (Xu et al., 17 Aug 2025).

The paper does not label these projections as query, key, and value. Functionally, they are analogous to such roles, but that characterization is interpretive rather than explicit. What is explicit is that SSCA performs projected cross-view affinity computation, opposite-view feature aggregation, output projection, and residual fusion within a stereo-specific, spike-compatible framework (Xu et al., 17 Aug 2025).

4. Spike compatibility and relation to conventional attention

SSCA is described as “a SNN variant of the Stereo Cross-Attention Module [NAFSSR],” with two key modifications: the introduction of SCU and the removal of the activation function, which together make it spike-compatible. This is the central design distinction between SSCA and conventional ANN cross-attention (Xu et al., 17 Aug 2025).

The SCU used inside SSCA is defined by

$$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$5

where $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$6 is the binary spike output of the LIF neuron at layer $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$7, time $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$8. The spike generation in the network follows LIF dynamics, so the SCU projections consume spike activations rather than dense analog activations. The paper does not separately annotate all intermediate SSCA matrix products with temporal indices, so it does not explicitly state that every intermediate tensor inside Equation (8) is binary. The safer statement is that SSCA is made spike-compatible because its learned transforms are implemented with SCUs and because activation is removed (Xu et al., 17 Aug 2025).

A defining property of SSCA is the absence of softmax normalization, exponentiation, and floating-point division in its stated formulation. Equation (8) contains matrix products, transpose operations, learned $$\mathbf{F}_{l},\mathbf{F}_{r} \in \mathbb{R}^{T \times C \times H \times W}.$$9 SCU projections, and residual addition, but no softmax or normalization term. This aligns with the paper’s broader argument that operations such as floating-point matrix division and exponentiation are incompatible with fully spike-driven computation. The ablation discussion further states that “activation functions involving exponential operations are contrary to the characteristics of spike-driven,” which supports the design choice to remove activation from the inherited stereo cross-attention module (Xu et al., 17 Aug 2025).

Relative to standard cross-attention, SSCA retains the high-level pattern of projecting left and right features, computing cross-view affinity, aggregating opposite-view information, and fusing it residually. What changes is the implementation substrate and the numerical form: conventional Q/K/V linear projections are replaced by $\mathbf{F}_l$0 SCUs; softmax-normalized attention is replaced by plain matrix-product interaction without activation; and unrestricted 2D global attention is replaced by stereo-structured horizontal interaction (Xu et al., 17 Aug 2025).

5. Empirical evidence in SNNSIR

The principal SSCA-specific empirical evidence is the ablation study on the proposed components. The paper reports the following relevant rows for Table 6: (b) SRBB with PSNR 24.96 and SSIM 0.862, (d) SRBB+SSCA with PSNR 25.10 and SSIM 0.862, (e) SRBB+SSCA+SSRB with PSNR 25.78 and SSIM 0.891, and (g) SRBB+SSCM+SSCA+SSRB with PSNR 26.57 and SSIM 0.903 (Xu et al., 17 Aug 2025).

Setting	PSNR / SSIM	Params
SRBB	24.96 / 0.862	2.96M
SRBB+SSCA	25.10 / 0.862	3.09M
SRBB+SSCA+SSRB	25.78 / 0.891	—
SRBB+SSCM+SSCA+SSRB	26.57 / 0.903	—

Comparing row (b) with row (d), the parameter count increases from $\mathbf{F}_l$1M to $\mathbf{F}_l$2M, PSNR increases from $\mathbf{F}_l$3 to $\mathbf{F}_l$4, and SSIM remains $\mathbf{F}_l$5. This suggests that adding SSCA alone improves PSNR in that ablation while leaving SSIM unchanged. The full model row shows that SSCA participates in the best-performing complete design, although that comparison does not isolate SSCA because additional modules are also present (Xu et al., 17 Aug 2025).

Because SSCA is embedded in the shared SNNSIR architecture, its effects are tied to the model’s stereo restoration results on Stereo Waterdrop for raindrop removal, RainKT12 and RainKT15 for deraining, Middlebury and Holopix50k for low-light enhancement, and Middlebury and Flickr1024 for super-resolution. No SSCA-only runtime or energy table is given. For the full SNNSIR model on Stereo Waterdrop, the paper reports 3.26M parameters and 29.32 mJ energy. A plausible implication is that SSCA is intended to provide stereo interaction without violating the low-energy design philosophy of the full model, but the paper does not provide a standalone SSCA energy breakdown (Xu et al., 17 Aug 2025).

6. Scope, related mechanisms, and common misconceptions

SSCA is specific to SNNSIR and should not be treated as a generic component of spike-based stereo vision. A common misconception is that spike stereo models in general use SSCA or transformer-style cross-attention as their binocular fusion primitive. SpikeStereoNet provides a counterexample: it does not contain a module literally called Spike Stereo Cross-Attention, nor does it use transformer-style left-right cross-attention. Its stereo interaction is instead described as all-pairs correlation volume construction, correlation pyramid formation, and iterative recurrent refinement through an RSNN update operator (Gao et al., 26 May 2025).

This distinction matters because SSCA and SpikeStereoNet solve different problems with different fusion mechanisms. SSCA is used for stereo image restoration and is designed for restoration-oriented feature exchange between degraded left and right views. By contrast, SpikeStereoNet addresses stereo depth estimation directly from raw spike streams and uses cost-volume-based binocular matching rather than learned cross-attention. Its core matching equation is

$\mathbf{F}_l$6

followed by correlation-pyramid lookup and iterative disparity refinement, not attention-based token interaction (Gao et al., 26 May 2025).

Another misconception is that SSCA should be read as an explicit disparity-estimation block. The data do not support that characterization. SSCA forms horizontal cross-view affinity and transfers information between views, but in SNNSIR it is used for restoration-oriented feature exchange, not explicit disparity prediction. Compared with correlation-based stereo matching, it is closer to a restoration module that exploits stereo correspondence than to a dedicated stereo matching head (Xu et al., 17 Aug 2025).

Taken on its own terms, SSCA is best understood as a stereo-specific, spike-compatible cross-attention block inserted at the encoded feature stage of SNNSIR’s first-stage U-Net. It reshapes encoded left and right features to $\mathbf{F}_l$7, projects them using $\mathbf{F}_l$8 SCUs, computes horizontal cross-view affinity through matrix products, aggregates opposite-view information in both directions, and fuses the result back with residual connections, all while avoiding softmax, exponentiation, and floating-point division in order to remain compatible with a fully spike-driven SNN (Xu et al., 17 Aug 2025).