---
title: Frequency Spectrum Fusion (FSRU)
url: https://www.emergentmind.com/topics/frequency-spectrum-representation-and-fusion-fsru
type: topic
---

# Frequency Spectrum Fusion (FSRU)

Frequency Spectrum Representation and Fusion (FSRU) denotes, in its original usage, the **“Frequency Spectrum Representation and fUsion network”** introduced for multimodal rumor detection, in which text and image features are transformed into the frequency domain, compressed, co-selected across modalities, and then mapped back for classification [2312.11023]. In a broader interpretive sense, later work uses closely related designs in which explicit spectral representations—obtained by DFT/FFT, DWT, STFT, or learned Fourier bases—are separated into components such as amplitude and phase or low- and high-frequency bands, then fused with spatial, temporal, graph, or cross-modal features to improve detail preservation, robustness, or complementary representation learning [2602.04405].

## 1. Origin, scope, and conceptual role

The original FSRU formulation was proposed for multimodal rumor detection over text-image pairs. Its pipeline comprises text and image embedding, Fourier-domain transformation, **unimodal spectrum compression (USC)**, **cross-modal spectrum co-selection (CSC)**, inverse DFT, and final rumor prediction with distribution-similarity fusion and dual contrastive learning [2312.11023]. In that formulation, “representation” refers to learning more discriminative unimodal features in the spectrum domain, while “fusion” refers to performing cross-modal interaction directly in that domain before returning to the original feature space.

Subsequent papers often do not use the acronym FSRU, but they instantiate the same architectural idea: a spectral branch is treated as a first-class representation pathway rather than as auxiliary post-processing. Examples include **Multi-scale Frequency Fusion (MFF)** plus **Interactive Spatial-Frequency Fusion (ISF)** in multi-modal image fusion [2602.04405], **Frequency Domain Fusion Module (FDFM)** in infrared-visible fusion [2410.22837], **Frequency Guided Fusion (FGF)** in underwater enhancement [2504.01243], **Wavelet Transform Feature Decomposer (WTFD)** plus **Multiscale Dual-Representation Alignment Filter (MDAF)** in remote-sensing segmentation [2405.01992], and **Spectrum Decomposition Attention (SDA)** in wavelet-domain segmentation decoding [2510.21079]. This suggests that FSRU is no longer only a model name; a plausible implication is that it has become a useful descriptor for a broader family of spectrum-aware representation-and-fusion architectures.

## 2. Spectrum representation mechanisms

A central variation across FSRU-style systems is the choice of spectral representation. The original rumor-detection FSRU applies **1D DFT** to embedded text sequences and image patch sequences, producing complex spectra \(\mathbf{X}^t\in\mathbb{C}^{m\times d}\) and \(\mathbf{X}^v\in\mathbb{C}^{n\times d}\). Its USC module operates on the **power spectrum** \(|\mathbf{X}^a|^2\), using a learned filter bank and cosine weighting to compact informative energy before cross-modal interaction [2312.11023]. In multimodal recommendation, SMORE similarly projects item modalities into a latent space, applies **DFT/FFT**, and keeps the resulting spectral tensors in \(\mathbb{C}^{n\times d}\), with modality-specific complex filters for denoising and later fusion [2412.14978]. In time-series representation learning, BTSF transforms the whole augmented series with **FFT**, then learns paired temporal and spectral token sequences \(F_t\in\mathbb{R}^{m\times d}\) and \(F_s\in\mathbb{R}^{n\times d}\) for bilinear interaction [2202.04770].

Other systems use explicit image-spectrum decompositions. SFDFusion performs **FFT** on infrared and visible inputs, splits the result into **amplitude** and **phase**, fuses the two separately with small convolutional blocks, reconstructs with **IFFT**, and then projects the result into a learned feature tensor \(F_{fre}\) [2410.22837]. FUSION also uses **2D FFT**, but does so independently for each RGB channel, learns mainly on the **magnitude spectrum**, preserves the original **phase**, and reconstructs channel-wise spatial features through IFFT before fusion [2504.01243]. BASFNet likewise decomposes features into amplitude and phase by FFT, but emphasizes **phase-spectrum-based** structural representation, using amplitude residuals to guide phase refinement before inverse reconstruction [2604.17879].

Wavelet-based systems encode frequency structure differently. ISFM applies **DWT** to modality-specific latent features \(F_{ir}^M\) and \(F_{vi}^M\), producing \(LL\), \(LH\), \(HL\), and \(HH\) subbands in latent feature space rather than on raw pixels [2602.04405]. SFFNet uses a **single-level 2D Haar wavelet decomposition** of intermediate features into \(A\), \(H\), \(V\), and \(D\), which function as low- and high-frequency components for later alignment with spatial branches [2405.01992]. WaveSeg applies **DWT/IDWT** both to resized input images and to decoder feature maps, explicitly routing \(LL\) into low-frequency semantic processing and \(LH/HL/HH\) into high-frequency structural processing [2510.21079]. STFT-based systems occupy a related position: the ECG framework of Zhang et al. converts waveforms into **STFT spectrograms**, so its “frequency representation” is a 2D time-frequency surface rather than a direct Fourier-coefficient tensor [1901.06469].

A further variant is learned Fourier-basis decomposition. SFFR does not compute an explicit FFT; instead, it uses **FourierKAN** to map feature values into learned sine-cosine basis responses, then performs **Frequency Component Exchange KAN (FCEKAN)** across RGB and IR features [2511.06298]. FMRFusion goes in the opposite direction: it uses no explicit transform at all, but learns an **implicit** low-/high-frequency decomposition through separate base and detail branches, regularized so that base features are more correlated across modalities while detail features remain more modality-specific [2606.07985]. This suggests that, in the contemporary literature, “frequency representation” spans both explicit transform-domain modeling and learned architectural surrogates.

## 3. Fusion operators and interaction patterns

The fusion stage in FSRU-style systems is rarely a simple concatenation. In the original FSRU, USC first compresses each modality’s spectrum, and CSC then uses one modality’s compressed spectrum to filter the other. The co-selection step is global rather than token-aligned: average-pooled spectral summaries from one modality generate a \(1\times1\) convolutional selector that multiplicatively modulates the other modality’s spectrum before inverse DFT [2312.11023]. The resulting refined text and image features are then fused by a Jensen–Shannon-divergence-guided detector rather than by direct concatenation alone.

In image fusion, more elaborate interaction appears. ISFM separates low- and high-frequency fusion inside **MFF**: its low-frequency branch uses attention-weighted multi-scale depth-wise convolutions, while its high-frequency branch performs a learned unsharp-masking-style enhancement through residuals against \(3\times3\) and \(5\times5\) average-pooled references [2602.04405]. These fused spectral features then drive **ISF**, where **Frequency-Guided Gate (FGG)** forms modality-specific gates \(G_{ir},G_{vi}\) from both spatial latent features and fused wavelet summaries, and **Frequency-Guided Mamba (FGM)** uses these gates to modulate Mamba-extracted spatial states. The core interaction is therefore multiplicative: frequency summaries do not replace spatial features, but reweight them.

Time-series and recommendation papers emphasize second-order interaction. BTSF forms all temporal-spectral pairs through a bilinear interaction \(F_t(i)^\top F_s(j)\), pools them into a global bilinear feature, and then iteratively updates both domains through **Spectrum-to-Time (S2T)** and **Time-to-Spectrum (T2S)** aggregation [2202.04770]. SMORE performs spectral fusion by **element-wise multiplication** across modality spectra, followed by a fusion-domain transfer function \(\delta_f\); its interpretation is that point-wise product in frequency space captures rich cross-modal correlations while adaptive spectral filters suppress modality-specific contamination [2412.14978].

Wavelet-domain segmentation papers emphasize alignment. SFFNet deliberately pairs **low-frequency wavelet features with global spatial features** and **high-frequency wavelet features with local spatial features**, then aligns them through MDAF, whose defining operator is **dual-cross attention**: frequency queries spatial keys and values, and spatial queries frequency keys and values, before concatenative recombination [2405.01992]. BASFNet operates differently: its **FEEM** first performs intra-frequency fusion by using enhanced amplitude residuals to reweight phase, then its **SFFIM** fuses the frequency-enhanced and spatial branches with parallel **Local Fusion** and **Global Fusion** paths before a final refinement convolution [2604.17879]. Across these systems, the recurring pattern is separation first, interaction second, and reconstruction or refinement third.

## 4. Supervision, reconstruction, and optimization

FSRU-style models differ substantially in how they supervise spectral learning. The original rumor-detection FSRU uses **binary cross-entropy** for classification together with **fully-supervised intra-modal contrastive learning** and **self-supervised inter-modal InfoNCE-style contrastive learning**. Frequency-domain refinement is therefore trained indirectly: spectral compression and co-selection are optimized because they improve separability and cross-modal agreement [2312.11023].

In image fusion, one line of work relies on architecture more than explicit spectral loss. ISFM reconstructs the fused luminance from concatenated spatially fused and frequency-derived features, but all losses remain in the spatial image domain: \(\mathcal{L}_{cont}\), \(\mathcal{L}_{int}\), \(\mathcal{L}_{grad}\), and \(\mathcal{L}_{ssim}\). The paper explicitly states that there is **no direct wavelet-domain reconstruction or consistency loss**, so frequency learning is architecturally enforced rather than spectrally supervised [2602.04405]. By contrast, SFDFusion adds a dedicated **frequency-domain fusion loss** \(\mathcal{L}_{fre}\), computed by reconstructing a spatial image from fused amplitude and phase and then applying saliency-guided correlation-coefficient terms to encourage infrared-target consistency in salient regions and visible-detail consistency elsewhere [2410.22837].

Other systems add task-specific structural supervision. BASFNet supplements segmentation losses with a **boundary-aware training strategy**, deriving dilated edge targets from Canny boundaries and supervising the edge prediction branch with weighted BCE [2604.17879]. FMRFusion uses a three-stage training schedule: first reconstruction and decomposition losses, then fusion learning, and finally **conditional flow matching**. Its decomposition loss,
\[
\mathcal{L}_{decomp}=\frac{(CC(F_I^{D},F_V^{D}))^2}{CC(F_I^{B},F_V^{B})+\epsilon},
\]
encourages high correlation in base features and low correlation in detail features, explicitly regularizing the learned low-/high-frequency split [2606.07985]. In Q-FSRU, the image and question embeddings are transformed by FFT and fused before retrieval-augmented prediction, but exact complex handling, gating, and retrieval merge operators remain underspecified, which limits reproducibility of its spectral mechanism [2508.12036].

## 5. Empirical evidence across application domains

The original FSRU showed that Fourier-domain representation and fusion can materially improve multimodal rumor detection. On **Weibo**, it achieved **Accuracy 0.901** and **F1 0.902**; on **Twitter**, **Accuracy 0.952** and **F1 0.950**. Removing USC reduced performance to **0.866 / 0.865** on Weibo and **0.910 / 0.908** on Twitter, while removing CSC reduced it to **0.883 / 0.882** and **0.924 / 0.922**, respectively, indicating that both spectral compression and cross-modal co-selection are central rather than cosmetic [2312.11023].

In multi-modal image fusion, ISFM provides direct module-level evidence for interactive spatial-frequency fusion. On MSRS, a baseline without the key modules reached **EN 6.46, SF 10.69, AG 3.39, SCD 1.68, VIF 0.87, \(Q^{AB/F}\) 0.59**, whereas the full model reached **6.70, 11.42, 3.77, 1.79, 1.01, 0.68**. Its frequency ablation further showed that using both low- and high-frequency fusion plus interactive gating outperformed variants without frequency-domain information or without FGG [2602.04405]. SFDFusion reports a similar pattern: on MSRS, removing **FDFM** degraded **VIF** from **1.011** to **0.589** and **Qabf** from **0.680** to **0.390**, while removing only the frequency-domain loss also caused substantial declines [2410.22837].

In dense prediction, spectrum-aware decoder design also produces measurable gains. SFFNet achieved **84.80 mIoU** on Vaihingen and **87.73 mIoU** on Potsdam; removing the low-frequency branch reduced these to **83.99** and **86.55**, while removing the high-frequency branch reduced them to **84.11** and **86.93** [2405.01992]. WaveSeg’s decoder ablations are comparably direct: on the SegMAN-T backbone, **with SDA** the model reached **43.8 mIoU**, versus **42.3** without SDA; its **Wavelet Mixer** also outperformed both MHSA and VSSM without wavelet decomposition, indicating that decomposition itself—not only the sequence model—matters [2510.21079].

Outside classical fusion, related evidence appears in other domains. FUSION reported **PSNR 23.717**, **SSIM 0.883**, and **LPIPS 0.112** on UIEB, with only **0.28M** parameters, supporting the utility of per-channel FFT-guided fusion for underwater enhancement [2504.01243]. SMORE improved Recall and NDCG consistently on **Baby**, **Sports**, and **Clothing**, and its ablations showed that removing spectrum modality fusion degraded recommendation quality, while retaining both uni-modal and fused representations outperformed using either alone [2412.14978]. These results suggest that frequency spectrum representation and fusion is not limited to one modality pair or one task family.

## 6. Limitations, ambiguities, and emerging directions

A recurring limitation is under-specification of the actual spectral operator. Several papers omit details that are important for exact reproduction: ISFM does not specify the wavelet basis, decomposition level, padding policy, or detailed subband bookkeeping [2602.04405]; FUSION does not state exact tensor shapes, normalization type, or complex-output handling after IFFT [2504.01243]; BTSF does not specify how complex FFT outputs are converted before spectral convolution [2202.04770]; Q-FSRU leaves FFT type, complex-number handling, and gated fusion equations unspecified [2508.12036]. Even in methods with explicit “frequency” nomenclature, the operational representation can therefore range from rigorous transform-domain modeling to only partially specified spectral surrogates.

Another limitation is computational trade-off. Spectrum-aware designs range from lightweight to heavy: SFDFusion reports **0.14M parameters** and **42.81G FLOPs** [2410.22837], FUSION reports **0.28M parameters** and **36.73G FLOPs** [2504.01243], ISFM reports **9.148M parameters**, **371 GFLOPs**, and **0.28 s** inference time on MSRS [2602.04405], and SFFR increases a YOLOv5-based detector from **110.6M** to **167.1M** parameters while raising computation to **104.8G FLOPs** [2511.06298]. The evidence therefore does not support a single efficiency narrative: spectral fusion can be extremely compact, but it can also impose substantial overhead when combined with multi-branch reconstruction and KAN-based frequency exchange.

The literature also points to several research directions. WaveSeg explicitly identifies the fixed selective-scan order of its wavelet-domain Mamba block as a limitation and suggests **adaptive or learnable scan order** as future work [2510.21079]. SFFR argues for more efficient **frequency band partitioning** and better **geometric alignment** in RGB-IR detection [2511.06298]. More broadly, a plausible implication is that future FSRU-style systems will continue to move away from static “frequency side branches” toward deeper interaction schemes in which spectral components guide spatial processing, spatial context constrains spectral refinement, and reconstruction is supervised not only by end-task loss but also by structural or semantic priors.

Source: https://www.emergentmind.com/topics/frequency-spectrum-representation-and-fusion-fsru