Frequency Spectrum Fusion (FSRU)

• FSRU is a framework that transforms modalities like text and images into frequency domain representations using techniques such as FFT, DWT, and learned Fourier bases.
• It employs spectral compression and cross-modal co-selection to integrate amplitude, phase, and spatial features for robust fusion and improved performance.
• Empirical results show FSRU’s effectiveness in multimodal rumor detection, image fusion, and dense prediction tasks with measurable gains in accuracy and detail preservation.

Frequency Spectrum Representation and Fusion (FSRU) denotes, in its original usage, the “Frequency Spectrum Representation and fUsion network” introduced for multimodal rumor detection, in which text and image features are transformed into the frequency domain, compressed, co-selected across modalities, and then mapped back for classification (Lao et al., 2023). In a broader interpretive sense, later work uses closely related designs in which explicit spectral representations—obtained by DFT/FFT, DWT, STFT, or learned Fourier bases—are separated into components such as amplitude and phase or low- and high-frequency bands, then fused with spatial, temporal, graph, or cross-modal features to improve detail preservation, robustness, or complementary representation learning (Zhu et al., 4 Feb 2026).

1. Origin, scope, and conceptual role

The original FSRU formulation was proposed for multimodal rumor detection over text-image pairs. Its pipeline comprises text and image embedding, Fourier-domain transformation, unimodal spectrum compression (USC), cross-modal spectrum co-selection (CSC), inverse DFT, and final rumor prediction with distribution-similarity fusion and dual contrastive learning (Lao et al., 2023). In that formulation, “representation” refers to learning more discriminative unimodal features in the spectrum domain, while “fusion” refers to performing cross-modal interaction directly in that domain before returning to the original feature space.

Subsequent papers often do not use the acronym FSRU, but they instantiate the same architectural idea: a spectral branch is treated as a first-class representation pathway rather than as auxiliary post-processing. Examples include Multi-scale Frequency Fusion (MFF) plus Interactive Spatial-Frequency Fusion (ISF) in multi-modal image fusion (Zhu et al., 4 Feb 2026), Frequency Domain Fusion Module (FDFM) in infrared-visible fusion (Hu et al., 2024), Frequency Guided Fusion (FGF) in underwater enhancement (Walia et al., 1 Apr 2025), Wavelet Transform Feature Decomposer (WTFD) plus Multiscale Dual-Representation Alignment Filter (MDAF) in remote-sensing segmentation (Yang et al., 2024), and Spectrum Decomposition Attention (SDA) in wavelet-domain segmentation decoding (Xu et al., 24 Oct 2025). This suggests that FSRU is no longer only a model name; a plausible implication is that it has become a useful descriptor for a broader family of spectrum-aware representation-and-fusion architectures.

2. Spectrum representation mechanisms

A central variation across FSRU-style systems is the choice of spectral representation. The original rumor-detection FSRU applies 1D DFT to embedded text sequences and image patch sequences, producing complex spectra $\mathbf{X}^t\in\mathbb{C}^{m\times d}$ and $\mathbf{X}^v\in\mathbb{C}^{n\times d}$. Its USC module operates on the power spectrum $|\mathbf{X}^a|^2$, using a learned filter bank and cosine weighting to compact informative energy before cross-modal interaction (Lao et al., 2023). In multimodal recommendation, SMORE similarly projects item modalities into a latent space, applies DFT/FFT, and keeps the resulting spectral tensors in $\mathbb{C}^{n\times d}$, with modality-specific complex filters for denoising and later fusion (Ong et al., 2024). In time-series representation learning, BTSF transforms the whole augmented series with FFT, then learns paired temporal and spectral token sequences $F_t\in\mathbb{R}^{m\times d}$ and $F_s\in\mathbb{R}^{n\times d}$ for bilinear interaction (Yang et al., 2022).

Other systems use explicit image-spectrum decompositions. SFDFusion performs FFT on infrared and visible inputs, splits the result into amplitude and phase, fuses the two separately with small convolutional blocks, reconstructs with IFFT, and then projects the result into a learned feature tensor $F_{fre}$ (Hu et al., 2024). FUSION also uses 2D FFT, but does so independently for each RGB channel, learns mainly on the magnitude spectrum, preserves the original phase, and reconstructs channel-wise spatial features through IFFT before fusion (Walia et al., 1 Apr 2025). BASFNet likewise decomposes features into amplitude and phase by FFT, but emphasizes phase-spectrum-based structural representation, using amplitude residuals to guide phase refinement before inverse reconstruction (Yu et al., 20 Apr 2026).

Wavelet-based systems encode frequency structure differently. ISFM applies DWT to modality-specific latent features $F_{ir}^M$ and $F_{vi}^M$, producing $LL$, $\mathbf{X}^v\in\mathbb{C}^{n\times d}$0, $\mathbf{X}^v\in\mathbb{C}^{n\times d}$1, and $\mathbf{X}^v\in\mathbb{C}^{n\times d}$2 subbands in latent feature space rather than on raw pixels (Zhu et al., 4 Feb 2026). SFFNet uses a single-level 2D Haar wavelet decomposition of intermediate features into $\mathbf{X}^v\in\mathbb{C}^{n\times d}$3, $\mathbf{X}^v\in\mathbb{C}^{n\times d}$4, $\mathbf{X}^v\in\mathbb{C}^{n\times d}$5, and $\mathbf{X}^v\in\mathbb{C}^{n\times d}$6, which function as low- and high-frequency components for later alignment with spatial branches (Yang et al., 2024). WaveSeg applies DWT/IDWT both to resized input images and to decoder feature maps, explicitly routing $\mathbf{X}^v\in\mathbb{C}^{n\times d}$7 into low-frequency semantic processing and $\mathbf{X}^v\in\mathbb{C}^{n\times d}$8 into high-frequency structural processing (Xu et al., 24 Oct 2025). STFT-based systems occupy a related position: the ECG framework of Zhang et al. converts waveforms into STFT spectrograms, so its “frequency representation” is a 2D time-frequency surface rather than a direct Fourier-coefficient tensor (Zhang et al., 2019).

A further variant is learned Fourier-basis decomposition. SFFR does not compute an explicit FFT; instead, it uses FourierKAN to map feature values into learned sine-cosine basis responses, then performs Frequency Component Exchange KAN (FCEKAN) across RGB and IR features (Zuo et al., 9 Nov 2025). FMRFusion goes in the opposite direction: it uses no explicit transform at all, but learns an implicit low-/high-frequency decomposition through separate base and detail branches, regularized so that base features are more correlated across modalities while detail features remain more modality-specific (Zhoua et al., 6 Jun 2026). This suggests that, in the contemporary literature, “frequency representation” spans both explicit transform-domain modeling and learned architectural surrogates.

3. Fusion operators and interaction patterns

The fusion stage in FSRU-style systems is rarely a simple concatenation. In the original FSRU, USC first compresses each modality’s spectrum, and CSC then uses one modality’s compressed spectrum to filter the other. The co-selection step is global rather than token-aligned: average-pooled spectral summaries from one modality generate a $\mathbf{X}^v\in\mathbb{C}^{n\times d}$9 convolutional selector that multiplicatively modulates the other modality’s spectrum before inverse DFT (Lao et al., 2023). The resulting refined text and image features are then fused by a Jensen–Shannon-divergence-guided detector rather than by direct concatenation alone.

In image fusion, more elaborate interaction appears. ISFM separates low- and high-frequency fusion inside MFF: its low-frequency branch uses attention-weighted multi-scale depth-wise convolutions, while its high-frequency branch performs a learned unsharp-masking-style enhancement through residuals against $|\mathbf{X}^a|^2$0 and $|\mathbf{X}^a|^2$1 average-pooled references (Zhu et al., 4 Feb 2026). These fused spectral features then drive ISF, where Frequency-Guided Gate (FGG) forms modality-specific gates $|\mathbf{X}^a|^2$2 from both spatial latent features and fused wavelet summaries, and Frequency-Guided Mamba (FGM) uses these gates to modulate Mamba-extracted spatial states. The core interaction is therefore multiplicative: frequency summaries do not replace spatial features, but reweight them.

Time-series and recommendation papers emphasize second-order interaction. BTSF forms all temporal-spectral pairs through a bilinear interaction $|\mathbf{X}^a|^2$3, pools them into a global bilinear feature, and then iteratively updates both domains through Spectrum-to-Time (S2T) and Time-to-Spectrum (T2S) aggregation (Yang et al., 2022). SMORE performs spectral fusion by element-wise multiplication across modality spectra, followed by a fusion-domain transfer function $|\mathbf{X}^a|^2$4; its interpretation is that point-wise product in frequency space captures rich cross-modal correlations while adaptive spectral filters suppress modality-specific contamination (Ong et al., 2024).

Wavelet-domain segmentation papers emphasize alignment. SFFNet deliberately pairs low-frequency wavelet features with global spatial features and high-frequency wavelet features with local spatial features, then aligns them through MDAF, whose defining operator is dual-cross attention: frequency queries spatial keys and values, and spatial queries frequency keys and values, before concatenative recombination (Yang et al., 2024). BASFNet operates differently: its FEEM first performs intra-frequency fusion by using enhanced amplitude residuals to reweight phase, then its SFFIM fuses the frequency-enhanced and spatial branches with parallel Local Fusion and Global Fusion paths before a final refinement convolution (Yu et al., 20 Apr 2026). Across these systems, the recurring pattern is separation first, interaction second, and reconstruction or refinement third.

4. Supervision, reconstruction, and optimization

FSRU-style models differ substantially in how they supervise spectral learning. The original rumor-detection FSRU uses binary cross-entropy for classification together with fully-supervised intra-modal contrastive learning and self-supervised inter-modal InfoNCE-style contrastive learning. Frequency-domain refinement is therefore trained indirectly: spectral compression and co-selection are optimized because they improve separability and cross-modal agreement (Lao et al., 2023).

In image fusion, one line of work relies on architecture more than explicit spectral loss. ISFM reconstructs the fused luminance from concatenated spatially fused and frequency-derived features, but all losses remain in the spatial image domain: $|\mathbf{X}^a|^2$5, $|\mathbf{X}^a|^2$6, $|\mathbf{X}^a|^2$7, and $|\mathbf{X}^a|^2$8. The paper explicitly states that there is no direct wavelet-domain reconstruction or consistency loss, so frequency learning is architecturally enforced rather than spectrally supervised (Zhu et al., 4 Feb 2026). By contrast, SFDFusion adds a dedicated frequency-domain fusion loss $|\mathbf{X}^a|^2$9, computed by reconstructing a spatial image from fused amplitude and phase and then applying saliency-guided correlation-coefficient terms to encourage infrared-target consistency in salient regions and visible-detail consistency elsewhere (Hu et al., 2024).

Other systems add task-specific structural supervision. BASFNet supplements segmentation losses with a boundary-aware training strategy, deriving dilated edge targets from Canny boundaries and supervising the edge prediction branch with weighted BCE (Yu et al., 20 Apr 2026). FMRFusion uses a three-stage training schedule: first reconstruction and decomposition losses, then fusion learning, and finally conditional flow matching. Its decomposition loss,

$\mathbb{C}^{n\times d}$0

encourages high correlation in base features and low correlation in detail features, explicitly regularizing the learned low-/high-frequency split (Zhoua et al., 6 Jun 2026). In Q-FSRU, the image and question embeddings are transformed by FFT and fused before retrieval-augmented prediction, but exact complex handling, gating, and retrieval merge operators remain underspecified, which limits reproducibility of its spectral mechanism (Thakur et al., 16 Aug 2025).

5. Empirical evidence across application domains

The original FSRU showed that Fourier-domain representation and fusion can materially improve multimodal rumor detection. On Weibo, it achieved Accuracy 0.901 and F1 0.902; on Twitter, Accuracy 0.952 and F1 0.950. Removing USC reduced performance to 0.866 / 0.865 on Weibo and 0.910 / 0.908 on Twitter, while removing CSC reduced it to 0.883 / 0.882 and 0.924 / 0.922, respectively, indicating that both spectral compression and cross-modal co-selection are central rather than cosmetic (Lao et al., 2023).

In multi-modal image fusion, ISFM provides direct module-level evidence for interactive spatial-frequency fusion. On MSRS, a baseline without the key modules reached EN 6.46, SF 10.69, AG 3.39, SCD 1.68, VIF 0.87, $\mathbb{C}^{n\times d}$1 0.59, whereas the full model reached 6.70, 11.42, 3.77, 1.79, 1.01, 0.68. Its frequency ablation further showed that using both low- and high-frequency fusion plus interactive gating outperformed variants without frequency-domain information or without FGG (Zhu et al., 4 Feb 2026). SFDFusion reports a similar pattern: on MSRS, removing FDFM degraded VIF from 1.011 to 0.589 and Qabf from 0.680 to 0.390, while removing only the frequency-domain loss also caused substantial declines (Hu et al., 2024).

In dense prediction, spectrum-aware decoder design also produces measurable gains. SFFNet achieved 84.80 mIoU on Vaihingen and 87.73 mIoU on Potsdam; removing the low-frequency branch reduced these to 83.99 and 86.55, while removing the high-frequency branch reduced them to 84.11 and 86.93 (Yang et al., 2024). WaveSeg’s decoder ablations are comparably direct: on the SegMAN-T backbone, with SDA the model reached 43.8 mIoU, versus 42.3 without SDA; its Wavelet Mixer also outperformed both MHSA and VSSM without wavelet decomposition, indicating that decomposition itself—not only the sequence model—matters (Xu et al., 24 Oct 2025).

Outside classical fusion, related evidence appears in other domains. FUSION reported PSNR 23.717, SSIM 0.883, and LPIPS 0.112 on UIEB, with only 0.28M parameters, supporting the utility of per-channel FFT-guided fusion for underwater enhancement (Walia et al., 1 Apr 2025). SMORE improved Recall and NDCG consistently on Baby, Sports, and Clothing, and its ablations showed that removing spectrum modality fusion degraded recommendation quality, while retaining both uni-modal and fused representations outperformed using either alone (Ong et al., 2024). These results suggest that frequency spectrum representation and fusion is not limited to one modality pair or one task family.

6. Limitations, ambiguities, and emerging directions

A recurring limitation is under-specification of the actual spectral operator. Several papers omit details that are important for exact reproduction: ISFM does not specify the wavelet basis, decomposition level, padding policy, or detailed subband bookkeeping (Zhu et al., 4 Feb 2026); FUSION does not state exact tensor shapes, normalization type, or complex-output handling after IFFT (Walia et al., 1 Apr 2025); BTSF does not specify how complex FFT outputs are converted before spectral convolution (Yang et al., 2022); Q-FSRU leaves FFT type, complex-number handling, and gated fusion equations unspecified (Thakur et al., 16 Aug 2025). Even in methods with explicit “frequency” nomenclature, the operational representation can therefore range from rigorous transform-domain modeling to only partially specified spectral surrogates.

Another limitation is computational trade-off. Spectrum-aware designs range from lightweight to heavy: SFDFusion reports 0.14M parameters and 42.81G FLOPs (Hu et al., 2024), FUSION reports 0.28M parameters and 36.73G FLOPs (Walia et al., 1 Apr 2025), ISFM reports 9.148M parameters, 371 GFLOPs, and 0.28 s inference time on MSRS (Zhu et al., 4 Feb 2026), and SFFR increases a YOLOv5-based detector from 110.6M to 167.1M parameters while raising computation to 104.8G FLOPs (Zuo et al., 9 Nov 2025). The evidence therefore does not support a single efficiency narrative: spectral fusion can be extremely compact, but it can also impose substantial overhead when combined with multi-branch reconstruction and KAN-based frequency exchange.

The literature also points to several research directions. WaveSeg explicitly identifies the fixed selective-scan order of its wavelet-domain Mamba block as a limitation and suggests adaptive or learnable scan order as future work (Xu et al., 24 Oct 2025). SFFR argues for more efficient frequency band partitioning and better geometric alignment in RGB-IR detection (Zuo et al., 9 Nov 2025). More broadly, a plausible implication is that future FSRU-style systems will continue to move away from static “frequency side branches” toward deeper interaction schemes in which spectral components guide spatial processing, spatial context constrains spectral refinement, and reconstruction is supervised not only by end-task loss but also by structural or semantic priors.