ISFM: Interactive Spatial-Frequency Fusion Mamba

• ISFM is a multimodal image-fusion framework that interactively integrates spatial and frequency representations to guide feature fusion across modalities.
• It employs modality-specific extraction, multi-scale frequency processing via wavelet transforms, and state-space modules to capture both global context and fine details.
• Experimental results demonstrate improved fusion metrics in infrared-visible and medical imaging, while maintaining efficient long-range modeling with linear complexity.

Interactive Spatial-Frequency Fusion Mamba (ISFM) denotes a Mamba-based multimodal image-fusion framework in which spatial-domain and frequency-domain representations are not merely processed serially or in parallel, but interact so that frequency features guide spatial features across modalities; in related Mamba literature, the same term also denotes a broader design pattern in which spatial and frequency representations interact iteratively across scales and along the sequence dimension rather than being fused only once (Zhu et al., 4 Feb 2026, Wijenayake et al., 11 Aug 2025). In its explicit MMIF formulation, ISFM was introduced for infrared–visible image fusion and medical image fusion, with a Modality-Specific Extractor, a Multi-scale Frequency Fusion module, and an Interactive Spatial-Frequency Fusion module built on Vision State-Space Modules, with the stated goal of retaining texture details and preserving significant information while maintaining linear-complexity long-range modeling (Zhu et al., 4 Feb 2026).

1. Conceptual basis

ISFM is motivated by a specific critique of earlier multimodal image-fusion pipelines: sequential spatial–frequency fusion first performs spatial-domain fusion and then frequency-domain enhancement, while parallel spatial–frequency fusion processes both branches simultaneously but without an explicit mechanism by which frequency features dynamically guide spatial fusion or vice versa (Zhu et al., 4 Feb 2026). Under this formulation, the central distinction of ISFM is not the mere presence of a frequency branch, but the use of an interaction mechanism that allows frequency-domain evidence to modulate spatial-domain fusion across modalities (Zhu et al., 4 Feb 2026).

This definition aligns with a broader line of Mamba-based research that treats spatial–frequency interaction as a structural design principle rather than a post hoc enhancement. In remote sensing semantic change detection, the same term is used for “a Mamba-based architecture where spatial and frequency representations interact iteratively across scales and along the sequence dimension, rather than being fused only once” (Wijenayake et al., 11 Aug 2025). In image deraining, FreqMamba formulates an analogous principle through complementary triple interaction structures including spatial Mamba, frequency band Mamba, and Fourier global modeling (Zhen et al., 2024). A common misconception is therefore to equate ISFM with simple concatenation of spatial and frequency features; the relevant literature instead defines it through interaction, gating, or state-space coupling (Zhu et al., 4 Feb 2026, Zhen et al., 2024).

At the backbone level, ISFM inherits the state-space perspective of visual Mamba. The continuous-time state-space form used across this literature is

$$\dot{h}(t) = A h(t) + B x(t), \quad y(t) = C h(t) + D x(t),$$

with visual variants implemented through selective scan and 2D state-space modeling to obtain long-range dependency modeling with linear complexity (Zhu et al., 4 Feb 2026, Peng et al., 2024, Zou et al., 2024). This choice places ISFM within the broader family of Visual State Space Models rather than within CNN-only or Transformer-only fusion architectures.

2. Architectural organization

In the MMIF formulation, ISFM consists of three principal subsystems: a Modality-Specific Extractor (MSE), a Multi-scale Frequency Fusion module (MFF), and an Interactive Spatial-Frequency Fusion module (ISF), followed by final reconstruction (Zhu et al., 4 Feb 2026). For infrared–visible image fusion, the visible image is converted from RGB to YCbCr and only the luminance channel $I_{vi}^Y$ is fused with the infrared image $I_{ir}$, resolving the channel mismatch between a 1-channel infrared input and a 3-channel visible input (Zhu et al., 4 Feb 2026).

The MSE performs shallow convolutional embedding and then applies separate VSSM branches with unshared weights:

$$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$

followed by

$$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$

These modality-specific features are then routed simultaneously into MFF and ISF (Zhu et al., 4 Feb 2026).

The module layout can be summarized as follows.

Component	Role	Main operation
MSE	Modality-specific spatial extraction	Conv $3\times3$ + VSSM
MFF	Frequency-domain fusion	DWT, LFFB, HFFB, inverse DWT
ISF	Interactive spatial–frequency fusion	FGM + FGG
Reconstruction	Final fused image generation	Concat + Conv

This organization places the frequency pathway before final reconstruction but also lets it act as an internal control signal. That is the architectural point of ISF: the fused frequency representation is not only reconstructed into $F_{fre}^{fuse}$, but also compressed into guidance features that directly gate the spatial Mamba fusion process (Zhu et al., 4 Feb 2026).

3. Multi-scale frequency fusion

MFF applies Discrete Wavelet Transform to the modality-specific Mamba features and decomposes them into $LL$, $LH$, $HL$, and $I_{vi}^Y$0 subbands, separating low-frequency approximation content from high-frequency detail content (Zhu et al., 4 Feb 2026). The low-frequency branch is handled by the Low-Frequency Fusion Block (LFFB), while the high-frequency branch is handled by the High-Frequency Fusion Block (HFFB) (Zhu et al., 4 Feb 2026).

For low-frequency fusion, the two modality-specific low-frequency components are combined and passed through spatial attention and multi-scale depth-wise convolutions:

$I_{vi}^Y$1

with attention

$I_{vi}^Y$2

and multi-scale refinement

$I_{vi}^Y$3

leading to

$I_{vi}^Y$4

This block is explicitly intended to adaptively fuse global low-frequency information from both modalities (Zhu et al., 4 Feb 2026).

For high-frequency fusion, MFF uses an unsharp-mask-like enhancement:

$I_{vi}^Y$5

then

$I_{vi}^Y$6

and

$I_{vi}^Y$7

The stated purpose is to strengthen structural high-frequency features such as edges while attenuating noise (Zhu et al., 4 Feb 2026).

MFF has two outputs. First, inverse DWT reconstructs a fused frequency feature $I_{vi}^Y$8 for final image synthesis. Second, fused subbands are compressed into a compact frequency descriptor for gating:

$I_{vi}^Y$9

followed by

$I_{ir}$0

This compact descriptor is later injected into the spatial fusion pathway through the Frequency-Guided Gate (Zhu et al., 4 Feb 2026).

4. Interactive spatial–frequency fusion

ISF is the defining mechanism of ISFM. It consists of a Frequency-Guided Mamba (FGM) and a Frequency-Guided Gate (FGG), and the interaction between them is what differentiates the framework from non-interactive serial or parallel spatial–frequency designs (Zhu et al., 4 Feb 2026).

FGM first computes linear projections for each modality:

$I_{ir}$1

The $I_{ir}$2-streams enter a VSSM-style state-space branch:

$I_{ir}$3

The $I_{ir}$4-streams are routed to FGG, which generates the gates $I_{ir}$5 and $I_{ir}$6 (Zhu et al., 4 Feb 2026).

FGG forms a joint spatial descriptor from the two modalities,

$I_{ir}$7

and then injects the compact frequency descriptor:

$I_{ir}$8

After splitting,

$I_{ir}$9

These gates are therefore explicitly frequency-guided but conditioned on joint spatial context (Zhu et al., 4 Feb 2026).

The final interactive fusion is

$$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$0

This makes the role of interaction precise: the frequency pathway does not merely provide an auxiliary reconstruction term, but modulates how the spatial Mamba outputs from the two modalities are weighted and combined (Zhu et al., 4 Feb 2026).

A related misconception is that Mamba alone is sufficient for complete spatial and frequency perception. Adjacent work explicitly argues otherwise. FreqMamba characterizes vanilla spatial Mamba as “frequency-blind” and introduces a Frequency-SSM block with spatial Mamba, frequency band Mamba, and Fourier global modeling (Zhen et al., 2024). SFMFusion similarly states that Mamba “lacks full spatial and frequency perceptions” and proposes a Spatial-Frequency Enhanced Mamba Block and a Dynamic Fusion Mamba Block for MMIF (Sun et al., 10 Nov 2025). Within this context, ISFM is best understood as a corrective architecture: it retains Mamba’s linear-complexity long-range modeling while supplying explicit multi-scale frequency interaction (Zhu et al., 4 Feb 2026, Sun et al., 10 Nov 2025).

5. Position within the Mamba-based fusion literature

ISFM belongs to a wider design space in which Mamba is used as the main long-range modeling operator while explicit fusion modules compensate for the absence of native frequency-domain reasoning. Several neighboring formulations are especially relevant.

FreqMamba introduces a Frequency-SSM block with three complementary branches—spatial Mamba, frequency band Mamba, and Fourier global modeling—and fuses them through concatenation and $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$1 convolution; this provides a direct precedent for treating frequency correlation and Fourier global degradation modeling as first-class companions to Mamba (Zhen et al., 2024). FusionMamba extends the single-input Mamba block to a dual-input FusionMamba block, using one input to generate $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$2 while scanning the other input, thereby establishing a plug-and-play template for bidirectional cross-stream fusion that is conceptually close to ISFM’s guided multimodal interaction (Peng et al., 2024).

AdaSFFuse introduces Adaptive Approximate Wavelet Transform and Spatial-Frequency Mamba Blocks for task-generalized multimodal image fusion, with adaptive wavelet-like frequency decoupling and a 2D State Space Duality module augmented by spatial-aware and frequency-filtering branches (Wang et al., 21 Aug 2025). MMR-Mamba, in multi-modal MRI reconstruction, combines Target modality-guided Cross Mamba, Selective Frequency Fusion, and Adaptive Spatial-Frequency Fusion, using channel-wise cross-domain enhancement to supplement less informative channels in one domain with corresponding channels from the other (Zou et al., 2024). Mamba-FCS, although formulated for semantic change detection, explicitly generalizes toward ISFM by proposing a Joint Spatio-Frequency Fusion block and suggesting a Mamba-based architecture where spatial and frequency representations interact iteratively across scales and along the sequence dimension (Wijenayake et al., 11 Aug 2025).

Other works enlarge the same space from different angles. Spatial-Mamba moves spatial structure directly into state space through a structure-aware state fusion equation implemented by multi-scale dilated depth-wise convolutions, suggesting that explicit state-level spatial connectivity can coexist with Mamba’s sequence recurrence (Xiao et al., 2024). S$$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$3Fin, for multimodal remote sensing classification, combines a high-frequency sparse enhancement transformer, an adaptive frequency channel module, a high-frequency resonance mask, and Mamba-based spatial-spectral attention fusion, reinforcing the broader view that effective Mamba systems for multimodal perception frequently require explicit spatial–spectral–frequency interaction rather than pure sequence scanning (Liu et al., 6 Oct 2025).

This landscape suggests that ISFM is not an isolated module but a named convergence point of several strands: dual-input Mamba fusion, explicit frequency decomposition, channel- or phase-aware gating, and state-space backbones with linear-complexity global context (Peng et al., 2024, Wang et al., 21 Aug 2025, Xiao et al., 2024, Liu et al., 6 Oct 2025).

6. Empirical behavior, downstream effects, and limitations

On infrared–visible image fusion benchmarks, the explicit ISFM framework reports its strongest aggregate performance on MSRS, FMB, and RoadScene. On MSRS it achieves EN $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$4, SF $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$5, AG $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$6, VIF $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$7, MI $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$8, $$\begin{aligned} F_{ir}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{ir})\big), \ F_{vi}^S &= \text{SiLU}\big(\text{Conv}_{3\times3}(I_{vi}^Y)\big), \end{aligned}$$9, SCD $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$0, and Avg.Rank $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$1; on FMB it reports SF $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$2, AG $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$3, and Avg.Rank $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$4; on RoadScene it reports SF $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$5, AG $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$6, VIF $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$7, $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$8, and Avg.Rank $$\begin{aligned} F_{ir}^M &= \Psi_{ir}(F_{ir}^S), \ F_{vi}^M &= \Psi_{vi}(F_{vi}^S). \end{aligned}$$9 (Zhu et al., 4 Feb 2026). On medical image fusion, without fine-tuning on the Harvard subsets, it reports the best Avg.Rank on MRI–PET and MRI–SPECT and the second-best Avg.Rank on MRI–CT (Zhu et al., 4 Feb 2026).

The same study also reports gains on downstream tasks. With YOLOv5 on MSRS fused images, ISFM achieves Precision $3\times3$0, Recall $3\times3$1, and [email protected] $3\times3$2; with DeepLabV3+ it reaches mIoU $3\times3$3, exceeding the second-best MMDRFuse result of $3\times3$4 (Zhu et al., 4 Feb 2026). These results are consistent with the module-level ablations: adding MFF improves all metrics over a baseline that concatenates modality-specific features only, and adding the full ISF with FGM and FGG further improves EN, SF, AG, SCD, VIF, and $3\times3$5 (Zhu et al., 4 Feb 2026).

The reported efficiency profile is mixed rather than uniformly minimal. ISFM has 9.15 M parameters in total, with 0.98 M in MSE, 4.42 M in MFF, 3.16 M in ISF, and 0.59 M in other layers, making it far smaller than some high-parameter baselines such as AdaFuse, DDFM, and FusionMamba (Zhu et al., 4 Feb 2026). Its inference time is 0.28 s per image on MSRS, and its complexity is 371 G FLOPs; the same source characterizes this as a good accuracy–efficiency trade-off rather than as the lightest design (Zhu et al., 4 Feb 2026). A plausible implication is that the interaction mechanism delivers measurable benefits, but that DWT-based multi-branch frequency processing still imposes a noticeable computational load.

The limitations and forward directions stated in the ISFM context are correspondingly specific. The current formulation uses fixed DWT rather than learnable frequency transforms, is framed for still-image multimodal fusion rather than video fusion, and leaves open extension to additional modalities such as SAR–optical or hyperspectral–RGB (Zhu et al., 4 Feb 2026). Across the neighboring literature, analogous future directions include deeper integration of frequency into state-space dynamics, more powerful multi-task interaction between branches, learnable adaptive filters, and lighter variants for real-time or resource-constrained deployment (Wijenayake et al., 11 Aug 2025, Wang et al., 21 Aug 2025, Zou et al., 2024). Taken together, these directions suggest that ISFM is best understood not as a closed architecture, but as an active research program for making Mamba explicitly spatial-frequency aware.