---
title: 'ISFM: Interactive Spatial-Frequency Fusion Mamba'
url: https://www.emergentmind.com/topics/interactive-spatial-frequency-fusion-mamba-isfm
type: topic
---

# ISFM: Interactive Spatial-Frequency Fusion Mamba

Interactive Spatial-Frequency Fusion Mamba (ISFM) denotes a Mamba-based multimodal image-fusion framework in which spatial-domain and frequency-domain representations are not merely processed serially or in parallel, but interact so that frequency features guide spatial features across modalities; in related Mamba literature, the same term also denotes a broader design pattern in which spatial and frequency representations interact iteratively across scales and along the sequence dimension rather than being fused only once [2602.04405] [2508.08232]. In its explicit MMIF formulation, ISFM was introduced for infrared–visible image fusion and medical image fusion, with a Modality-Specific Extractor, a Multi-scale Frequency Fusion module, and an Interactive Spatial-Frequency Fusion module built on Vision State-Space Modules, with the stated goal of retaining texture details and preserving significant information while maintaining linear-complexity long-range modeling [2602.04405].

## 1. Conceptual basis

ISFM is motivated by a specific critique of earlier multimodal image-fusion pipelines: sequential spatial–frequency fusion first performs spatial-domain fusion and then frequency-domain enhancement, while parallel spatial–frequency fusion processes both branches simultaneously but without an explicit mechanism by which frequency features dynamically guide spatial fusion or vice versa [2602.04405]. Under this formulation, the central distinction of ISFM is not the mere presence of a frequency branch, but the use of an interaction mechanism that allows frequency-domain evidence to modulate spatial-domain fusion across modalities [2602.04405].

This definition aligns with a broader line of Mamba-based research that treats spatial–frequency interaction as a structural design principle rather than a post hoc enhancement. In remote sensing semantic change detection, the same term is used for “a Mamba-based architecture where spatial and frequency representations interact iteratively across scales and along the sequence dimension, rather than being fused only once” [2508.08232]. In image deraining, FreqMamba formulates an analogous principle through complementary triple interaction structures including spatial Mamba, frequency band Mamba, and Fourier global modeling [2404.09476]. A common misconception is therefore to equate ISFM with simple concatenation of spatial and frequency features; the relevant literature instead defines it through interaction, gating, or state-space coupling [2602.04405] [2404.09476].

At the backbone level, ISFM inherits the state-space perspective of visual Mamba. The continuous-time state-space form used across this literature is
$$
\dot{h}(t) = A h(t) + B x(t), \quad y(t) = C h(t) + D x(t),
$$
with visual variants implemented through selective scan and 2D state-space modeling to obtain long-range dependency modeling with linear complexity [2602.04405] [2404.07932] [2406.18950]. This choice places ISFM within the broader family of Visual State Space Models rather than within CNN-only or Transformer-only fusion architectures.

## 2. Architectural organization

In the MMIF formulation, ISFM consists of three principal subsystems: a Modality-Specific Extractor (MSE), a Multi-scale Frequency Fusion module (MFF), and an Interactive Spatial-Frequency Fusion module (ISF), followed by final reconstruction [2602.04405]. For infrared–visible image fusion, the visible image is converted from RGB to YCbCr and only the luminance channel \(I_{vi}^Y\) is fused with the infrared image \(I_{ir}\), resolving the channel mismatch between a 1-channel infrared input and a 3-channel visible input [2602.04405].

The MSE performs shallow convolutional embedding and then applies separate VSSM branches with unshared weights:
$$
\begin{aligned}
F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \\
F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big),
\end{aligned}
$$
followed by
$$
\begin{aligned}
F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \\
F_{vi}^M &= \Psi_{vi}(F_{vi}^S).
\end{aligned}
$$
These modality-specific features are then routed simultaneously into MFF and ISF [2602.04405].

The module layout can be summarized as follows.

| Component | Role | Main operation |
|---|---|---|
| MSE | Modality-specific spatial extraction | Conv \(3\times3\) + VSSM |
| MFF | Frequency-domain fusion | DWT, LFFB, HFFB, inverse DWT |
| ISF | Interactive spatial–frequency fusion | FGM + FGG |
| Reconstruction | Final fused image generation | Concat + Conv |

This organization places the frequency pathway before final reconstruction but also lets it act as an internal control signal. That is the architectural point of ISF: the fused frequency representation is not only reconstructed into \(F_{fre}^{fuse}\), but also compressed into guidance features that directly gate the spatial Mamba fusion process [2602.04405].

## 3. Multi-scale frequency fusion

MFF applies Discrete Wavelet Transform to the modality-specific Mamba features and decomposes them into \(LL\), \(LH\), \(HL\), and \(HH\) subbands, separating low-frequency approximation content from high-frequency detail content [2602.04405]. The low-frequency branch is handled by the Low-Frequency Fusion Block (LFFB), while the high-frequency branch is handled by the High-Frequency Fusion Block (HFFB) [2602.04405].

For low-frequency fusion, the two modality-specific low-frequency components are combined and passed through spatial attention and multi-scale depth-wise convolutions:
$$
\begin{aligned}
X_{lf}^{ir} &= \text{DWT}(F_{ir}^M), \\
X_{lf}^{vi} &= \text{DWT}(F_{vi}^M), \\
X_{lf} &= \text{SiLU}\big(\text{Conv}_{1\times1}(X_{lf}^{ir} + X_{lf}^{vi})\big),
\end{aligned}
$$
with attention
$$
\sigma_{s} = \text{Sigmoid}\left(\text{Conv}_{3\times3}\big([\mathcal{M}(X_{lf}); \mathcal{A}(X_{lf})]\big)\right),
$$
and multi-scale refinement
$$
\begin{aligned}
U' &= \text{SiLU}\big(\text{Conv}_{1\times1}(\text{DWConv}_{3\times3}(X_{lf}))\big),\\
U'' &= \text{SiLU}\big(\text{Conv}_{1\times1}(\text{DWConv}_{5\times5}(X_{lf}))\big),
\end{aligned}
$$
leading to
$$
\begin{aligned}
U &= (U' + U'') \otimes \sigma_{s} + X_{lf}, \\
X_{lf}^{fuse} &= \text{SiLU}\big(\text{Conv}_{1\times1}(U)\big).
\end{aligned}
$$
This block is explicitly intended to adaptively fuse global low-frequency information from both modalities [2602.04405].

For high-frequency fusion, MFF uses an unsharp-mask-like enhancement:
$$
\begin{aligned}
X_{hf}^{ir} &= \text{DWT}(F_{ir}^M), \\
X_{hf}^{vi} &= \text{DWT}(F_{vi}^M), \\
X_{hf} &= \text{SiLU}\big(\text{Conv}_{1\times1}(X_{hf}^{ir} + X_{hf}^{vi})\big),
\end{aligned}
$$
then
$$
\begin{aligned}
X_{hf}' &= \mathcal{A}_{3\times3}(X_{hf}), \\
X_{hf}'' &= \mathcal{A}_{5\times5}(X_{hf}),
\end{aligned}
$$
and
$$
\begin{aligned}
S_1 &= \text{SiLU}\big(\text{Conv}_{1\times1}(X_{hf} - X_{hf}')\big),\\
S_2 &= \text{SiLU}\big(\text{Conv}_{1\times1}(X_{hf} - X_{hf}'')\big),\\
S &= S_1 + S_2,\\
X_{hf}^{fuse} &= \text{SiLU}\big(\text{Conv}_{1\times1}(S + X_{hf})\big).
\end{aligned}
$$
The stated purpose is to strengthen structural high-frequency features such as edges while attenuating noise [2602.04405].

MFF has two outputs. First, inverse DWT reconstructs a fused frequency feature \(F_{fre}^{fuse}\) for final image synthesis. Second, fused subbands are compressed into a compact frequency descriptor for gating:
$$
F_f = [X_{ll}^{fuse}; X_{lh}^{fuse}+X_{hl}^{fuse}+X_{hh}^{fuse}],
$$
followed by
$$
Z_f = \text{GAP}(\text{FC}(F_f)).
$$
This compact descriptor is later injected into the spatial fusion pathway through the Frequency-Guided Gate [2602.04405].

## 4. Interactive spatial–frequency fusion

ISF is the defining mechanism of ISFM. It consists of a Frequency-Guided Mamba (FGM) and a Frequency-Guided Gate (FGG), and the interaction between them is what differentiates the framework from non-interactive serial or parallel spatial–frequency designs [2602.04405].

FGM first computes linear projections for each modality:
$$
\begin{aligned}
X_{ir} &= \text{Linear}(\text{LN}(F_{ir}^M)), & Z_{ir} &= \text{Linear}(\text{LN}(F_{ir}^M)), \\
X_{vi} &= \text{Linear}(\text{LN}(F_{vi}^M)), & Z_{vi} &= \text{Linear}(\text{LN}(F_{vi}^M)).
\end{aligned}
$$
The \(X\)-streams enter a VSSM-style state-space branch:
$$
\begin{aligned}
H_{ir} &= \text{LN}(\text{SSM}(\text{SiLU}(\text{DWConv}_{3 \times 3}(X_{ir})))), \\
H_{vi} &= \text{LN}(\text{SSM}(\text{SiLU}(\text{DWConv}_{3 \times 3}(X_{vi})))).
\end{aligned}
$$
The \(Z\)-streams are routed to FGG, which generates the gates \(G_{ir}\) and \(G_{vi}\) [2602.04405].

FGG forms a joint spatial descriptor from the two modalities,
$$
Z_{global} = \text{LN}(\text{Linear}([Z_{ir}; Z_{vi}])),
$$
and then injects the compact frequency descriptor:
$$
Z_{g} = Z_f \otimes Z_{global} + Z_{global}.
$$
After splitting,
$$
\begin{aligned}
Z_{g}', Z_{g}'' &= \text{Split}(Z_g), \\
G_{ir} &= \text{SiLU}(\text{LN}(\text{Linear}(Z_{g}'))), \\
G_{vi} &= \text{SiLU}(\text{LN}(\text{Linear}(Z_{g}''))).
\end{aligned}
$$
These gates are therefore explicitly frequency-guided but conditioned on joint spatial context [2602.04405].

The final interactive fusion is
$$
\begin{aligned}
F_h &= H_{ir} \otimes G_{ir} + H_{vi} \otimes G_{vi}, \\
F_{spa}^{fuse} &= \text{Linear}(F_h) + s_1 \cdot F_{ir}^M + s_2 \cdot F_{vi}^M.
\end{aligned}
$$
This makes the role of interaction precise: the frequency pathway does not merely provide an auxiliary reconstruction term, but modulates how the spatial Mamba outputs from the two modalities are weighted and combined [2602.04405].

A related misconception is that Mamba alone is sufficient for complete spatial and frequency perception. Adjacent work explicitly argues otherwise. FreqMamba characterizes vanilla spatial Mamba as “frequency-blind” and introduces a Frequency-SSM block with spatial Mamba, frequency band Mamba, and Fourier global modeling [2404.09476]. SFMFusion similarly states that Mamba “lacks full spatial and frequency perceptions” and proposes a Spatial-Frequency Enhanced Mamba Block and a Dynamic Fusion Mamba Block for MMIF [2511.06593]. Within this context, ISFM is best understood as a corrective architecture: it retains Mamba’s linear-complexity long-range modeling while supplying explicit multi-scale frequency interaction [2602.04405] [2511.06593].

## 5. Position within the Mamba-based fusion literature

ISFM belongs to a wider design space in which Mamba is used as the main long-range modeling operator while explicit fusion modules compensate for the absence of native frequency-domain reasoning. Several neighboring formulations are especially relevant.

FreqMamba introduces a Frequency-SSM block with three complementary branches—spatial Mamba, frequency band Mamba, and Fourier global modeling—and fuses them through concatenation and \(1\times1\) convolution; this provides a direct precedent for treating frequency correlation and Fourier global degradation modeling as first-class companions to Mamba [2404.09476]. FusionMamba extends the single-input Mamba block to a dual-input FusionMamba block, using one input to generate \((\mathbf{B}, \mathbf{C}, \Delta)\) while scanning the other input, thereby establishing a plug-and-play template for bidirectional cross-stream fusion that is conceptually close to ISFM’s guided multimodal interaction [2404.07932].

AdaSFFuse introduces Adaptive Approximate Wavelet Transform and Spatial-Frequency Mamba Blocks for task-generalized multimodal image fusion, with adaptive wavelet-like frequency decoupling and a 2D State Space Duality module augmented by spatial-aware and frequency-filtering branches [2508.15505]. MMR-Mamba, in multi-modal MRI reconstruction, combines Target modality-guided Cross Mamba, Selective Frequency Fusion, and Adaptive Spatial-Frequency Fusion, using channel-wise cross-domain enhancement to supplement less informative channels in one domain with corresponding channels from the other [2406.18950]. Mamba-FCS, although formulated for semantic change detection, explicitly generalizes toward ISFM by proposing a Joint Spatio-Frequency Fusion block and suggesting a Mamba-based architecture where spatial and frequency representations interact iteratively across scales and along the sequence dimension [2508.08232].

Other works enlarge the same space from different angles. Spatial-Mamba moves spatial structure directly into state space through a structure-aware state fusion equation implemented by multi-scale dilated depth-wise convolutions, suggesting that explicit state-level spatial connectivity can coexist with Mamba’s sequence recurrence [2410.15091]. S\(^{2}\)Fin, for multimodal remote sensing classification, combines a high-frequency sparse enhancement transformer, an adaptive frequency channel module, a high-frequency resonance mask, and Mamba-based spatial-spectral attention fusion, reinforcing the broader view that effective Mamba systems for multimodal perception frequently require explicit spatial–spectral–frequency interaction rather than pure sequence scanning [2510.04628].

This landscape suggests that ISFM is not an isolated module but a named convergence point of several strands: dual-input Mamba fusion, explicit frequency decomposition, channel- or phase-aware gating, and state-space backbones with linear-complexity global context [2404.07932] [2508.15505] [2410.15091] [2510.04628].

## 6. Empirical behavior, downstream effects, and limitations

On infrared–visible image fusion benchmarks, the explicit ISFM framework reports its strongest aggregate performance on MSRS, FMB, and RoadScene. On MSRS it achieves EN \(= 6.70\), SF \(= 11.42\), AG \(= 3.77\), VIF \(= 1.01\), MI \(= 2.78\), \(Q^{AB/F} = 0.68\), SCD \(= 1.79\), and Avg.Rank \(= 1.14\); on FMB it reports SF \(= 13.65\), AG \(= 4.10\), and Avg.Rank \(= 1.71\); on RoadScene it reports SF \(= 14.87\), AG \(= 5.50\), VIF \(= 0.59\), \(Q^{AB/F} = 0.53\), and Avg.Rank \(= 2.00\) [2602.04405]. On medical image fusion, without fine-tuning on the Harvard subsets, it reports the best Avg.Rank on MRI–PET and MRI–SPECT and the second-best Avg.Rank on MRI–CT [2602.04405].

The same study also reports gains on downstream tasks. With YOLOv5 on MSRS fused images, ISFM achieves Precision \(= 0.984\), Recall \(= 0.962\), and mAP@0.5 \(= 0.985\); with DeepLabV3+ it reaches mIoU \(= 70.53\), exceeding the second-best MMDRFuse result of \(69.02\) [2602.04405]. These results are consistent with the module-level ablations: adding MFF improves all metrics over a baseline that concatenates modality-specific features only, and adding the full ISF with FGM and FGG further improves EN, SF, AG, SCD, VIF, and \(Q^{AB/F}\) [2602.04405].

The reported efficiency profile is mixed rather than uniformly minimal. ISFM has 9.15 M parameters in total, with 0.98 M in MSE, 4.42 M in MFF, 3.16 M in ISF, and 0.59 M in other layers, making it far smaller than some high-parameter baselines such as AdaFuse, DDFM, and FusionMamba [2602.04405]. Its inference time is 0.28 s per image on MSRS, and its complexity is 371 G FLOPs; the same source characterizes this as a good accuracy–efficiency trade-off rather than as the lightest design [2602.04405]. A plausible implication is that the interaction mechanism delivers measurable benefits, but that DWT-based multi-branch frequency processing still imposes a noticeable computational load.

The limitations and forward directions stated in the ISFM context are correspondingly specific. The current formulation uses fixed DWT rather than learnable frequency transforms, is framed for still-image multimodal fusion rather than video fusion, and leaves open extension to additional modalities such as SAR–optical or hyperspectral–RGB [2602.04405]. Across the neighboring literature, analogous future directions include deeper integration of frequency into state-space dynamics, more powerful multi-task interaction between branches, learnable adaptive filters, and lighter variants for real-time or resource-constrained deployment [2508.08232] [2508.15505] [2406.18950]. Taken together, these directions suggest that ISFM is best understood not as a closed architecture, but as an active research program for making Mamba explicitly spatial-frequency aware.

Source: https://www.emergentmind.com/topics/interactive-spatial-frequency-fusion-mamba-isfm