---
title: Dual-frequency Fusion in Multimodal Research
url: https://www.emergentmind.com/topics/multimodal-dual-frequency-fusion
type: topic
---

# Dual-frequency Fusion in Multimodal Research

Multimodal dual-frequency fusion denotes a class of fusion strategies that does not treat heterogeneous modalities as a single undifferentiated feature source, but instead explicitly separates, approximates, or emphasizes complementary frequency components—typically low-frequency structure and high-frequency detail—and then recombines them through cross-modal interaction. In the recent literature, this formulation appears in infrared-visible image fusion, visible-infrared object detection, RGB-thermal semantic segmentation, medical image fusion, and even text-image rumor detection, with implementations based on FFT/IFFT, discrete wavelet transforms, Gaussian filtering, bilinear decomposition, and discrepancy-guided state-space exchange [2601.05538][2511.10046][2506.03555][2602.04405][2312.11023]. Across these settings, the central objective is stable: preserve global consistency and modality-salient structure while avoiding the loss of fine textures, edges, boundaries, or thermal targets that occurs when one modality dominates the other.

## 1. Core formulation and problem setting

Multimodal fusion is motivated by the fact that different sensors capture complementary yet domain-specific cues. Infrared-visible fusion seeks images with prominent targets and rich texture details; RGB-thermal segmentation addresses adverse lighting conditions where RGB alone is insufficient; medical fusion combines intensity and anatomical detail; and multimodal rumor detection treats text and image as heterogeneous carriers of semantically correlated evidence [2601.05538][2410.22837][2605.26273][2312.11023]. A recurring difficulty is *information imbalance*: one modality can overpower the other, producing fused outputs that either over-prioritize infrared intensity at the cost of visible detail or preserve visible structure while diminishing thermal salience [2601.05538][2511.10046].

Within this setting, “dual-frequency” usually refers to an explicit or implicit partition between low-frequency and high-frequency information. Low-frequency components are used for global brightness, smooth structure, silhouettes, or thermal outlines, while high-frequency components are used for edges, textures, boundaries, and fine-grained detail [2511.10046][2506.03555][2605.26273]. Some methods realize this literally through sub-band decomposition. Others use a proxy. DIFF-MF, for example, constructs a feature discrepancy map
$$
Diff = \tanh \left|F_{vi}^{share} - F_{ir}^{share}\right|
$$
and uses it to recalibrate modality-specific features, describing this discrepancy-driven signal as the basis for balancing thermal targets and visible textures [2601.05538]. This suggests that dual-frequency fusion is not restricted to classical spectral transforms; it can also be instantiated as modality difference modeling when the difference map is used to guide complementary feature extraction.

A second defining property is that dual-frequency fusion is rarely only a decomposition step. The decomposition is typically followed by cross-modal exchange, adaptive gating, or reconstruction constraints designed to prevent simple averaging of heterogeneous cues. This is explicit in works that criticize purely spatial-domain fusion, purely local convolutions, or simple serial/parallel spatial-frequency combination without interaction [2410.22837][2602.04405].

## 2. Frequency representations and decomposition mechanisms

The literature uses several distinct mechanisms to expose low- and high-frequency structure.

| Method | Frequency representation | Characteristic operation |
|---|---|---|
| DIFF-MF [2601.05538] | Discrepancy-guided “dual-frequency” signal | $\tanh |F_{vi}^{share}-F_{ir}^{share}|$ gates private branches |
| FreDFT [2511.10046] | Fourier-domain attention | $\mathcal{F}(Q)\odot\mathcal{F}(K)$ with IFFT reconstruction |
| FSRU [2312.11023] | 1D DFT spectrum for text and image | spectrum compression and cross-modal spectrum co-selection |
| SFDFusion [2410.22837] | FFT amplitude and phase | separate fusion of amplitude and phase, then IFFT |
| WIFE-Fusion [2506.03555] | DWT sub-bands | $LL$, $LH$, $HL$, $HH$ with IFSA and IFI |
| AdaSFFuse [2508.15505] | Adaptive Approximate Wavelet Transform | learnable $K_{LL},K_{LH},K_{HL},K_{HH}$ grouped convolutions |
| ISFM [2602.04405] | DWT low/high multi-scale fusion | LFFB and HFFB over wavelet bands |
| FMRFusion [2606.07985] | Bilinear frequency decomposition | Transformer-style low-frequency and MS-SPM high-frequency branches |
| RGB-T segmentation [2605.26273] | Gaussian low/high split | $T_{low}=G_\sigma\circledast T_i'$, $T_{high}=T_i'-T_{low}$ |

FFT-based systems formulate fusion directly in the Fourier domain. In FreDFT, queries and keys are transformed by 2D FFT, correlated by element-wise multiplication, and returned to the spatial domain by inverse FFT:
$$
A_{RGB} = \hat Q_{RGB}\odot \hat K_{RGB}, \qquad
S_{RGB} = LN(\mathcal{F}^{-1}(A_{RGB}))
$$
after which the resulting weights modulate the other modality’s values [2511.10046]. SFDFusion instead separates amplitude and phase,
$$
AMP_{ir}=|FFT(I_{ir})|,\qquad PHA_{ir}=\angle FFT(I_{ir}),
$$
fuses them independently, and reconstructs a spatial feature by IFFT [2410.22837]. FSRU applies a one-dimensional DFT along the sequence or patch dimension for text and image, compresses the power spectrum, and performs cross-modal spectrum co-selection before inverse transformation [2312.11023].

Wavelet-based systems operate on explicitly localized frequency bands. WIFE-Fusion decomposes each modality into one low-frequency subband and three high-frequency subbands via DWT, treating structure and detail separately [2506.03555]. ISFM also uses DWT and then applies distinct low-frequency and high-frequency fusion blocks [2602.04405]. AdaSFFuse replaces fixed wavelet filters with learnable analysis filters and 2D kernels
$$
K_{LL}=\mathbf{u}_0\otimes\mathbf{u}_0^T,\quad
K_{LH}=\mathbf{u}_0\otimes\mathbf{u}_1^T,\quad
K_{HL}=\mathbf{u}_1\otimes\mathbf{u}_0^T,\quad
K_{HH}=\mathbf{u}_1\otimes\mathbf{u}_1^T
$$
followed by sub-band-specific dilated convolutions [2508.15505].

Other formulations are domain-specific but frequency-aware in the same sense. RGB-thermal segmentation uses a fixed Gaussian filter with $k=7$ and $\sigma=2.0$ to obtain low- and high-frequency thermal features [2605.26273]. FMRFusion uses a bilinear decomposition in which shallow features are split into low-frequency base features through a Transformer-style block and high-frequency detail features through a Multi-Scale Structural Perception Module [2606.07985]. These designs broaden the concept of dual-frequency fusion beyond classical Fourier analysis.

## 3. Cross-modal interaction operators

Once frequency structure has been exposed, fusion quality depends on how the modalities interact. The recent literature proposes several families of interaction operators.

A first family uses cross-modal spectral weighting. FSRU summarizes each modality’s compressed spectrum by
$$
S^t = Conv(Avg(\hat X^t\odot \Theta^t)),\qquad
S^v = Conv(Avg(\hat X^v\odot \Theta^v)),
$$
then broadcasts these summary vectors to re-weight the other modality:
$$
\tilde X^t=\hat X^t\odot S^v,\qquad
\tilde X^v=\hat X^v\odot S^t.
$$
The paper explicitly frames this as dual-frequency fusion because each modality’s spectral bands are modulated by the other modality’s global spectral priorities [2312.11023]. FreDFT adopts a related but detection-oriented formulation: MFDA computes frequency-domain affinities, while FDFFL reassembles mixed-scale low-, mid-, and high-band information from depth-wise convolutions with kernel sizes $\{3,5,7\}$ [2511.10046].

A second family uses cross-modal interaction within and across sub-bands. WIFE-Fusion’s Intra-Frequency Self-Attention performs cross-modal attention inside the same band, for example
$$
X'_{L,1}=\mathrm{MHSA}(Q_{L,2},K_{L,1},V_{L,2}),
$$
while Inter-Frequency Interaction concatenates channel-attended low-frequency features from one modality with spatial-attended high-frequency features from the other [2506.03555]. ISFM similarly separates low-frequency fusion from high-frequency fusion, but then goes further by using the resulting frequency-complementary feature to guide spatial fusion through a Frequency-Guided Gate [2602.04405].

A third family uses discrepancy-driven or exchange-based interaction. DIFF-MF first recalibrates modality-private features using the discrepancy map:
$$
F_{ir}^{self'}=(1-Diff)\odot F_{ir}^{self}+Diff\odot F_{vi}^{self},
$$
$$
F_{vi}^{self'}=(1-Diff)\odot F_{vi}^{self}+Diff\odot F_{ir}^{self'}.
$$
It then applies a Channel-Exchange Module with cross-modal state-space exchange and an adaptive gate, followed by a Spatial-Exchange Module that interleaves infrared and visible features along rows, columns, or concatenation paths [2601.05538]. This design is explicitly intended to prevent either modality from “drowning out” the other.

These mechanisms collectively show that dual-frequency fusion is not merely a decomposition problem. It is a routing and reweighting problem in which low-frequency and high-frequency components are selectively exchanged, aligned, and recombined. This is also why several papers argue against simple serial or parallel spatial-frequency pipelines without interaction [2602.04405].

## 4. State-space and Mamba-based formulations

A major current direction is the replacement of quadratic attention with state-space models, especially Mamba or VSS-based components, while retaining explicit spatial-frequency interaction.

DIFF-MF is a representative state-space design. Its Channel-Exchange Module flattens modality features into token sequences, splits projected tokens into SSM analogues of $Q$, $K$, and $V$, and exchanges channels in both directions before adaptive channel re-weighting. Its Spatial-Exchange Module realigns features through row-wise, column-wise, and concatenation-based patterns, then processes them with VSS blocks whose scans are linear recurrences. The paper states that the overall spatial fusion is $O(HW)$ rather than $O((HW)^2)$, and reports total FLOPs of approximately $393\,G$ versus approximately $337\,T$ for a Restormer-style transformer [2601.05538].

SFMFusion also integrates Mamba with explicit spatial and frequency enhancement. Its three-branch architecture couples an infrared-reconstruction branch, a visible-reconstruction branch, and a multi-modal fusion branch. The core Spatial-Frequency Enhanced Mamba Block is
$$
X_{out}=MMB(\bar X)+CEB(\bar X)+FEB(\bar X),
$$
where MMB introduces mixed-scale spatial context into 2D Selective-Scan Mamba, CEB models cross-channel dependencies with dual-pool gating, and FEB operates in the Fourier domain on amplitude and phase [2511.06593]. The Dynamic Fusion Mamba Block then uses a learned spatial weight map
$$
W=\sigma(DWConv_{3\times3}(Linear(\bar D_v+\bar D_f+\bar D_i)))
$$
to shift emphasis between infrared and visible reconstruction features [2511.06593].

AdaSFFuse embeds spatial-frequency reasoning directly inside a Mamba2-style SSD module. Its Spatial-Frequency Mamba block contains a spatial branch $\mathcal{S}$ based on $3\times3$ convolution and a frequency branch $\mathcal{T}$ based on FFT, learnable thresholding, and IFFT:
$$
\Lambda=\mathrm{FFT}(L'),\qquad
M=|\Lambda|^2>\lambda,\qquad
L_{\mathcal T}=\mathrm{IFFT}(\Lambda\odot M).
$$
The concatenated spatial and frequency outputs are split into $\{X,A,B,C\}$ and processed by a 2D State-Space Duality update [2508.15505].

ISFM places a Modality-Specific Extractor built from Vision State-Space Modules ahead of Multi-scale Frequency Fusion and Interactive Spatial-Frequency Fusion. Its Frequency-Guided Mamba computes modality-specific spatial features through 2D-SSM, then modulates them with frequency-derived gates before residual fusion [2602.04405]. Across these designs, state-space modeling is used to preserve global dependency modeling at linear complexity, while the frequency pathway corrects Mamba’s lack of full spatial and frequency perceptions, a limitation explicitly identified in SFMFusion [2511.06593].

## 5. Optimization objectives and empirical behavior

Dual-frequency fusion models are typically trained with composite objectives that couple structural similarity, intensity preservation, edge or gradient fidelity, and task-specific auxiliary terms. DIFF-MF uses
$$
\mathcal{L}_{total}=\lambda_1\mathcal{L}_{ssim}+\lambda_2\mathcal{L}_{text}+\lambda_3\mathcal{L}_{int},
$$
where $\mathcal{L}_{text}$ matches fused-image gradients to the maximum source-image gradients and $\mathcal{L}_{int}$ matches intensity to a modality combination $M(I_{ir},I_{vi})$ [2601.05538]. WIFE-Fusion uses an analogous weighted sum of intensity, texture, and SSIM losses with $\alpha=2$, $\beta=10$, and $\gamma=1$ [2506.03555]. SFMFusion adds two reconstruction losses to the fusion loss,
$$
\mathcal{L}_{total}=\mathcal{L}_f+\alpha_1\mathcal{L}_v+\alpha_2\mathcal{L}_i,
$$
so that image reconstruction acts as an auxiliary task for multimodal fusion [2511.06593].

Some methods introduce explicitly frequency-aware supervision. SFDFusion adds a frequency-domain fusion loss
$$
\mathcal{L}_{fre}
= CC(M\odot X,M\odot I_{ir}) + CC((1-M)\odot X,(1-M)\odot I_{vis}),
$$
where $M$ is a pre-computed infrared saliency mask and $X$ is the IFFT reconstruction from the fused spectrum [2410.22837]. FSRU is distinctive in combining classification cross-entropy with intra-modal and inter-modal contrastive objectives, thereby enforcing both unimodal spectral discriminability and cross-modal alignment [2312.11023].

Empirically, reported improvements are measured not only by generic fusion scores such as Entropy (EN), Standard Deviation (SD), Spatial Frequency (SF), Average Gradient (AG), Mutual Information (MI), Visual Information Fidelity (VIF), and $Q^{AB/F}$, but also by downstream detection or segmentation performance. DIFF-MF reports, for example, on M³FD: EN $7.19$, SD $55.09$, SF $18.787$, MI $3.04$, VIF $0.825$, and AG $6.33$; on TNO: EN $7.10$, SD $63.03$, SF $14.672$, MI $2.521$, VIF $0.836$, and AG $5.393$; and on DroneVehicle: EN $7.52$, SD $71.69$, SF $22.816$, MI $2.402$, VIF $0.643$, and AG $8.195$ [2601.05538]. FreDFT reaches mAP50 $83.5\%$ on FLIR, $98.2\%$ on LLVIP, and $88.4\%$ on M³FD, with ablations reporting that MFDA outperforms spatial cross-attention by approximately $1.1\%$ mAP and FDFFL beats a standard MLP by $0.4\%$ mAP [2511.10046]. SFMFusion attains the best average rank on six benchmarks and reports, on MSRS, EN $6.72$, SD $45.17$, SF $11.94$, AG $3.90$, MI $3.39$, VIF $1.07$, $Q^{ab/f}$ $0.71$, and Avg.Rank $1.14$ [2511.06593]. WIFE-Fusion reports downstream object detection improvement on MSRS to mAP@[.5:.95] $0.663$, compared with $0.588$ for IR alone and $0.556$ for VI alone [2506.03555]. For RGB-thermal semantic segmentation, the lightest variant of the frequency-guided fusion architecture reports $61.73\%$ mIoU on MFNet and $86.24\%$ mIoU on PST900 with $35.43M$ parameters [2605.26273].

Taken together, these results indicate a consistent empirical pattern: dual-frequency fusion tends to improve both perceptual fusion metrics and downstream task metrics when the model explicitly preserves the trade-off between low-frequency structure and high-frequency detail.

## 6. Scope, misconceptions, limitations, and extensions

The scope of multimodal dual-frequency fusion is broader than infrared-visible image fusion alone. The surveyed systems cover infrared-visible fusion, RGB-NIR fusion, CT-MRI and PET-MRI medical fusion, multi-exposure fusion, multi-focus fusion, RGB-thermal semantic segmentation, visible-infrared object detection, and text-image rumor detection [2506.03555][2508.15505][2605.26273][2511.10046][2312.11023]. Some authors also state that the underlying mechanism can be adapted to audio-visual speech separation, LiDAR-camera fusion, and medical MRI-CT alignment when complementary bandwidths exist [2511.10046].

A common misconception is that dual-frequency fusion is synonymous with wavelet decomposition. The recent literature contradicts this directly: methods use FFT amplitude-phase fusion, Gaussian low/high separation, adaptive learnable wavelets, bilinear base-detail decomposition, and discrepancy maps between modalities [2410.22837][2605.26273][2508.15505][2606.07985][2601.05538]. A second misconception is that adding a frequency branch is sufficient. ISFM explicitly argues that many methods rely on simple serial or parallel spatial-frequency fusion without interaction, and proposes interactive guidance instead [2602.04405]. WIFE-Fusion and FreDFT similarly make interaction—not decomposition alone—the core mechanism [2506.03555][2511.10046].

The principal limitations are also recurrent. AdaSFFuse states that it requires roughly pixel-aligned inputs and that large geometric misalignments still challenge the sub-band summation; it also identifies single-scale wavelet levels as a constraint [2508.15505]. WIFE-Fusion notes that DWT uses a fixed filter bank and that the number of WIFE blocks introduces compute and memory overhead [2506.03555]. ISFM reports moderate GFLOPs of $371\,G$ and $9.15\,M$ parameters, and notes that DWT/IDWT adds implementation complexity and restricts the architecture to power-of-2 spatial dimensions [2602.04405]. These are not incidental implementation details; they define the current trade space between interpretability, efficiency, alignment robustness, and reconstruction fidelity.

The proposed extensions follow directly from these limitations. WIFE-Fusion suggests learnable or multi-level wavelet transforms, extending IFSA/IFI to more than two modalities, and incorporating frequency-aware perceptual losses or adversarial training [2506.03555]. AdaSFFuse proposes multi-level AdaWAT, end-to-end task-aware fusion, upstream geometric alignment modules, and non-linear wavelet bases via small MLPs [2508.15505]. FreDFT suggests replacing conventional cross-attention with Dual-Frequency Fusion Transformer layers in broader multimodal settings [2511.10046]. A plausible implication is that future work will continue to move away from fixed, hand-crafted frequency partitioning toward adaptive decomposition, interactive cross-domain gating, and tighter coupling with downstream objectives.

In contemporary usage, then, multimodal dual-frequency fusion is best understood not as a single architecture, but as a design principle: isolate or approximate structurally distinct frequency content, let modalities exchange complementary evidence under learned gates or state-space dynamics, and reconstruct a fused representation that preserves both global coherence and modality-specific high-frequency detail.

Source: https://www.emergentmind.com/topics/multimodal-dual-frequency-fusion