Joint Spatio-Frequency Fusion Block

• The Joint Spatio-Frequency Fusion Block is a module that fuses spatial details with transform-domain features to enhance image representation and fusion.
• It leverages various methods such as DCT, FFT, and wavelet decompositions to capture both local geometry and frequency structure in tasks like change detection and restoration.
• Recent designs incorporate recurrent mechanisms, gating, and adaptive subband weighting to robustly integrate spatial and frequency cues while addressing alignment and noise challenges.

to=arxiv_search.search 彩神争霸 彩神争霸大发快三 0 0 {"query":"\"Joint Spatio-Frequency Fusion\" OR \"Spatial-Frequency\" fusion Mamba wavelet FFT image fusion","max_results":10,"sort_by":"submittedDate"} to=arxiv_search.search 天天中彩票不能 െയാണ് 0 0 {"query":"(Peng et al., 2024) OR (Wijenayake et al., 11 Aug 2025) OR (Gao et al., 20 Feb 2025) OR (Hu et al., 2024) OR (Yang et al., 2024) OR (Zhu et al., 4 Feb 2026) OR (Li et al., 13 May 2026) OR (Zhao et al., 6 Jul 2025)","max_results":20,"sort_by":"relevance"} A Joint Spatio-Frequency Fusion Block is a module that fuses spatial-domain representations with frequency-domain or transform-domain representations inside a single computational unit. In the literature, this idea appears in several distinct forms: blockwise DCT-domain focus selection with spatial consistency verification for multi-focus fusion (S., 2023), wavelet- and FFT-based multimodal fusion modules coupled to Mamba or SSD backbones (Wang et al., 21 Aug 2025, Zhu et al., 4 Feb 2026), log-amplitude spectral injection into change-detection decoders (Wijenayake et al., 11 Aug 2025), frequency-decoupled cross-attention for image-event depth estimation (Sun et al., 25 Mar 2025), FFT-based amplitude-phase fusion in infrared-visible pipelines (Li et al., 13 May 2026), and hybrid spatial-frequency feature fusion for restoration and classification (Gao et al., 20 Feb 2025, Zhao et al., 6 Jul 2025). The term is therefore best understood as a family of operators rather than a single canonical block.

1. Definition, scope, and terminology

In common usage, a Joint Spatio-Frequency Fusion Block combines two representation domains. The spatial branch preserves localization, geometry, boundaries, and local semantics; the frequency branch encodes low-/high-frequency structure, transform coefficients, spectra, or subband energies. A useful abstraction suggested by the cited designs is

$Y = \phi\big(S(X), F(X)\big),$

where $S(\cdot)$ is a spatial encoder, $F(\cdot)$ is a frequency-domain or frequency-derived encoder, and $\phi(\cdot)$ is a fusion operator such as selection, gating, attention, residual mixing, or state-space interaction.

The scope of “frequency” is not uniform across papers. In classical multi-focus fusion, “spatial frequency” is a sharpness criterion defined from pixel differences or DCT AC energy (S., 2023). In FFT-based multimodal fusion, frequency means magnitude and phase spectra, sometimes with log compression (Wijenayake et al., 11 Aug 2025, Li et al., 13 May 2026). In wavelet-based designs, frequency is represented by subbands such as LL, LH, HL, and HH (Wang et al., 21 Aug 2025, Yang et al., 2024, Maqsood et al., 25 May 2026). By contrast, FusionMamba is frequently adjacent to this literature but is not a true spatial-frequency model in the Fourier, DCT, or wavelet sense: in that paper, “spectral” strictly refers to wavelength-wise bands, and “there is no Fourier/DCT/Wavelet transform” (Peng et al., 2024).

This terminological distinction matters because “spectral fidelity” in hyperspectral or pansharpening systems is not the same object as “frequency preservation” in FFT- or wavelet-based fusion. A plausible implication is that papers using “spatial-spectral” and papers using “spatial-frequency” often solve analogous fusion problems with materially different signal models (Peng et al., 2024, Wang et al., 21 Aug 2025).

2. Representation domains and transform choices

The transform layer is the defining design choice of a joint spatio-frequency block. The earliest formulation in the cited set is DCT-based multi-focus fusion, where each aligned image pair is partitioned into non-overlapping blocks, transformed with the 2D DCT, scored by spatial frequency, and selected by a thresholded decision map refined through Consistency Verification over a $3\times 3$ neighborhood (S., 2023). In this formulation, the frequency signal is explicit, localized, and blockwise.

Wavelet-based formulations replace block DCTs with multi-resolution decompositions. AdaSFFuse introduces AdaWAT, which decouples each modality’s feature map into LL, LH, HL, and HH subbands at quarter resolution using adaptive analysis vectors embedded via grouped and dilated convolutions (Wang et al., 21 Aug 2025). SFFNet uses fixed Haar wavelet decomposition in its Wavelet Transform Feature Decomposer, then projects low-frequency and concatenated high-frequency subbands before aligning them with global and local spatial features in MDAF (Yang et al., 2024). MSFET-E2V likewise uses channel-wise Haar DWT, but its skip block selectively refines the $X_{HH}$ subband before inverse reconstruction, while CDAM feeds wavelet-derived features into transformer attention (Maqsood et al., 25 May 2026).

FFT-based blocks use global spectra rather than localized subbands. Mamba-FCS computes an orthonormal 2D FFT per channel, then uses the log-amplitude spectrum

$F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$

as a frequency feature, discarding phase entirely (Wijenayake et al., 11 Aug 2025). DSFF in SFRF decomposes registered infrared and visible inputs into amplitude and phase, processes each with $1\times 1$ convolutions, and fuses them with residual references to visible phase and infrared amplitude (Li et al., 13 May 2026). DFFNet applies channel-wise 2D FFT, generates a dynamic filter kernel in the frequency domain by pooling the input and synthesizing a weighted combination of learnable basis filters, then applies IFFT and complementary spatial/frequency depthwise convolutions (Zhao et al., 6 Jul 2025).

A different route is frequency decoupling without Fourier coefficients. FreDFuse in FUSE uses Gaussian-Laplacian pyramids to separate low-frequency Gaussian components from high-frequency Laplacian details, then performs event-driven high-frequency and image-driven low-frequency cross-attention (Sun et al., 25 Mar 2025). SFAFNet’s FDGM is also FFT-free: it learns a spatially variant low-pass filter via Softmax-normalized kernels and defines the complementary high-pass operator as $F^H = I - F^L$ (Gao et al., 20 Feb 2025).

3. Recurrent fusion mechanisms

Across these papers, several recurrent fusion mechanisms appear.

A first pattern is decision-based fusion with spatial regularization. In the DCT multi-focus method, the block does not learn a latent fusion map. Instead it computes per-block spatial frequency, forms a ternary decision map $W_{i,j}\in\{-1,0,+1\}$, and refines isolated decisions through a local majority-consistency check (S., 2023). This is the most literal interpretation of a joint spatio-frequency block: frequency scores decide, spatial neighborhoods regularize.

A second pattern is parallel spatial and frequency branches followed by attention or residual merging. SFDFusion exemplifies this structure: DMRM extracts complementary spatial features, FDFM fuses amplitude and phase via FFT/IFFT, and a lightweight head concatenates spatial and frequency outputs for reconstruction (Hu et al., 2024). DSFF follows a related dual-branch design, with a Restormer spatial branch and an FFT-based frequency branch whose outputs are refined by residual blocks (Li et al., 13 May 2026).

A third pattern is frequency-guided modulation of spatial hidden states. ISFM is explicit here: its MFF computes fused low- and high-frequency wavelet features, FGG converts them into gating signals, and FGM modulates VMamba/VSSM hidden states element-wise before residual aggregation (Zhu et al., 4 Feb 2026). This is not cross-attention; the paper states that ISF injects frequency-domain cues into the spatial fusion path via gating (Zhu et al., 4 Feb 2026).

A fourth pattern is joint state-space parameterization by spatial and frequency selectors. AdaSFFuse’s Spatial-Frequency Mamba Block uses a spatial-aware branch $S(\cdot)$0 and a frequency-filtering branch $S(\cdot)$1, merges them, splits the result into $S(\cdot)$2, and drives a 2D-SSD update (Wang et al., 21 Aug 2025). The state-space parameters are therefore conditioned by both spatial and frequency signals rather than by attention maps alone.

A fifth pattern is hybrid spatial-frequency restoration blocks. SFAFNet’s GSFFBlock combines a spatial NAFBlock stack, FDGM low-/high-frequency decomposition, channel gates derived from global mean and standard deviation, and cross-attention across spatial, low-frequency, and high-frequency streams (Gao et al., 20 Feb 2025). DFFNet reaches a comparable end through a different decomposition: dynamic FFT filtering in DFB and cross-modal spectral-spatial interaction in SSAFB (Zhao et al., 6 Jul 2025).

4. Representative formulations

Several formulations are especially representative of the concept.

In DCT-domain multi-focus fusion, the core selection variable is

$S(\cdot)$3

followed by local vote accumulation

$S(\cdot)$4

The fused block then inherits the DCT coefficients from $S(\cdot)$5 or $S(\cdot)$6 according to the sign of $S(\cdot)$7, with averaging or tie-breaking when $S(\cdot)$8 (S., 2023). Here, spatial-frequency fusion is literally a fusion of spatial neighborhood logic and frequency-domain block energy.

In Mamba-FCS, the Joint Spatio-Frequency Fusion block at decoder stage $S(\cdot)$9 concatenates spatial features, log-amplitude spectra, and the spatial difference:

$F(\cdot)$0

compresses the $F(\cdot)$1 channels with a $F(\cdot)$2 convolution, and refines the result with channel-first and then spatial CBAM attention (Wijenayake et al., 11 Aug 2025). No inverse FFT is used; the learned projection and attention align spatial tensors with raw log-amplitude spectra inside the decoder.

AdaSFFuse formalizes a stronger state-space interaction. Its frequency branch computes

$F(\cdot)$3

while the spatial branch computes

$F(\cdot)$4

The merged representation is then split into $F(\cdot)$5 and used in the 2D-SSD recurrence

$F(\cdot)$6

This places spatial-frequency fusion inside the state update itself rather than after feature extraction (Wang et al., 21 Aug 2025).

ISFM gives perhaps the cleanest gating formulation. Frequency guidance is summarized as

$F(\cdot)$7

split into two streams, and projected to gates $F(\cdot)$8 and $F(\cdot)$9. The modulated spatial fusion is then

$\phi(\cdot)$0

followed by

$\phi(\cdot)$1

The block is therefore interactive, but the interaction is multiplicative gating rather than attention (Zhu et al., 4 Feb 2026).

In DSFF, the amplitude and phase branches are fused asymmetrically:

$\phi(\cdot)$2

$\phi(\cdot)$3

followed by inverse FFT and a residual spatial-frequency refinement head (Li et al., 13 May 2026). This asymmetry reflects task intent: preserve thermal saliency through infrared amplitude while keeping visible phase as a structural reference.

5. Application areas and empirical behavior

These blocks have been used across fusion, restoration, change detection, depth estimation, segmentation, classification, and event reconstruction.

For multi-focus fusion in JPEG or DCT-domain settings, the DCT + SF + CV method reported the best average RMSE and SSIM among the listed baselines, improving from RMSE $\phi(\cdot)$4 and SSIM $\phi(\cdot)$5 for DCT + SF to RMSE $\phi(\cdot)$6 and SSIM $\phi(\cdot)$7 with consistency verification (S., 2023). This supports the classical view that blockwise spatial-frequency criteria remain effective when transform coefficients are already available.

For semantic change detection, Mamba-FCS reported $\phi(\cdot)$8 Overall Accuracy, $\phi(\cdot)$9 $3\times 3$0, and $3\times 3$1 SeK on SECOND, and $3\times 3$2 Overall Accuracy, $3\times 3$3 $3\times 3$4, and $3\times 3$5 SeK on Landsat-SCD. Removing the FFT2 branch caused drops of $3\times 3$6 OA, $3\times 3$7 $3\times 3$8, $3\times 3$9 mIoU, and $X_{HH}$0 SeK, directly quantifying the contribution of its Joint Spatio-Frequency Fusion unit (Wijenayake et al., 11 Aug 2025).

For task-generalized multimodal image fusion, AdaSFFuse reported approximately $X_{HH}$1M parameters, $X_{HH}$2G FLOPs at $X_{HH}$3, and runtime approximately $X_{HH}$4 ms per image, while its full model reached SSIM $X_{HH}$5 in the reported ablations, compared with $X_{HH}$6 for the baseline and $X_{HH}$7 after adding the shallow SFM stage (Wang et al., 21 Aug 2025). This suggests that spatial-frequency co-fusion is not merely a reconstruction refinement but a core representation-learning component.

For image-event depth estimation, FUSE reported a $X_{HH}$8 Abs.Rel improvement on MVSEC and $X_{HH}$9 on DENSE relative to prior methods, with ablations showing that FreDFuse improved Abs.Rel from $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$0 to $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$1 on MVSEC outdoor_day1 compared with cross-attention-only fusion (Sun et al., 25 Mar 2025). The useful point here is methodological: explicit low-/high-frequency role assignment can outperform modality-agnostic attention.

For deblurring, SFAFNet’s full GSFFBlock improved GoPro PSNR from $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$2 dB for the baseline to $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$3 dB, with intermediate gains from adding FDGM, GATE, and CAM. The larger SFAFNet-B reached $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$4 dB and $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$5 SSIM on GoPro (Gao et al., 20 Feb 2025). This indicates that joint spatial-frequency blocks also function effectively in single-image restoration, not only in multimodal fusion.

For remote sensing segmentation, SFFNet reported mIoU $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$6 on Vaihingen and $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$7 on Potsdam, with ablations showing mIoU drops when low-frequency WTFD-L, high-frequency WTFD-H, or MDAF were removed (Yang et al., 2024). For event-to-video reconstruction, MSFET-E2V reported $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$8M parameters and inference times of $F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)$9 ms at $1\times 1$0, $1\times 1$1 ms at $1\times 1$2, and $1\times 1$3 ms at $1\times 1$4, while its ablations showed that using both LF and HF wavelet paths in CDAM outperformed LF-only and HF-only variants on ECD, HQF, and MVSEC (Maqsood et al., 25 May 2026).

6. Limitations, misconceptions, and extensions

A recurrent limitation is alignment sensitivity. The DCT multi-focus method assumes perfectly registered source images, and even sub-pixel misalignment can cause incorrect block correspondence (S., 2023). FusionMamba explicitly notes that misregistration or parallax is not handled, despite its dual-input state-space fusion (Peng et al., 2024). SFRF was proposed precisely because cumulative registration errors contaminate downstream fusion, and its DSFF depends on MIR to provide aligned infrared inputs (Li et al., 13 May 2026).

A second limitation is incomplete frequency modeling. Mamba-FCS discards phase and uses only log-amplitude spectra (Wijenayake et al., 11 Aug 2025). This improves illumination robustness, but the paper also notes that phase-dependent structures may not be captured well. Conversely, amplitude-phase methods such as DSFF must manage the instability of phase processing and the ambiguity of internal PF/SA implementations, which are only partially specified in the paper (Li et al., 13 May 2026).

A third limitation is artifact trade-offs. Blockwise DCT selection can create blockiness near focus transitions, which CV reduces but does not eliminate (S., 2023). High-frequency branches can amplify noise: HFFB in ISFM is therefore built around subtraction against pooled maps to suppress random high-frequency components while keeping edges (Zhu et al., 4 Feb 2026), and WSB in MSFET-E2V selectively processes the $1\times 1$5 band because shallow high-frequency features often carry noise-like artifacts (Maqsood et al., 25 May 2026).

A common misconception is to conflate spectral with frequency. In remote sensing fusion, spectral fidelity often means preserving wavelength-band consistency, not preserving Fourier or wavelet coefficients. FusionMamba is explicit on this point and even sketches how true spatio-frequency fusion could be added by inserting DCT, FFT, or wavelet stages before its state-space block (Peng et al., 2024). This suggests that the phrase “joint spatio-frequency fusion block” should be reserved for modules whose second branch is genuinely transform-domain, or at least frequency-decoupled, rather than merely multi-band.

The extension direction most repeatedly proposed in the cited literature is toward adaptive, learnable cross-domain coupling: explicit spatio-frequency fusion in Mamba-style recurrent dynamics (Peng et al., 2024, Wang et al., 21 Aug 2025), adaptive wavelets or learnable subband weighting instead of fixed Haar filters (Maqsood et al., 25 May 2026), cross-modality use beyond canonical image pairs (Peng et al., 2024), and more robust handling of noise, parallax, and downstream-task sensitivity (Wang et al., 21 Aug 2025, Zhu et al., 4 Feb 2026). A plausible synthesis is that the field is moving from serial “spatial branch plus frequency branch” designs toward blocks in which frequency signals directly parameterize recurrence, gating, or decoder reasoning.