---
title: Joint Spatio-Frequency Fusion Block
url: https://www.emergentmind.com/topics/joint-spatio-frequency-fusion-block
type: topic
---

# Joint Spatio-Frequency Fusion Block

to=arxiv_search.search  彩神争霸  彩神争霸大发快三 0 0 {"query":"\"Joint Spatio-Frequency Fusion\" OR \"Spatial-Frequency\" fusion Mamba wavelet FFT image fusion","max_results":10,"sort_by":"submittedDate"}
to=arxiv_search.search  天天中彩票不能 െയാണ് 0 0 {"query":"2404.07932 OR 2508.08232 OR 2502.14209 OR 2410.22837 OR 2405.01992 OR 2602.04405 OR 2605.13049 OR 2507.04510","max_results":20,"sort_by":"relevance"}
A Joint Spatio-Frequency Fusion Block is a module that fuses spatial-domain representations with frequency-domain or transform-domain representations inside a single computational unit. In the literature, this idea appears in several distinct forms: blockwise DCT-domain focus selection with spatial consistency verification for multi-focus fusion [2305.11265], wavelet- and FFT-based multimodal fusion modules coupled to Mamba or SSD backbones [2508.15505] [2602.04405], log-amplitude spectral injection into change-detection decoders [2508.08232], frequency-decoupled cross-attention for image-event depth estimation [2503.19739], FFT-based amplitude-phase fusion in infrared-visible pipelines [2605.13049], and hybrid spatial-frequency feature fusion for restoration and classification [2502.14209] [2507.04510]. The term is therefore best understood as a family of operators rather than a single canonical block.

## 1. Definition, scope, and terminology

In common usage, a Joint Spatio-Frequency Fusion Block combines two representation domains. The spatial branch preserves localization, geometry, boundaries, and local semantics; the frequency branch encodes low-/high-frequency structure, transform coefficients, spectra, or subband energies. A useful abstraction suggested by the cited designs is
$$
Y = \phi\big(S(X), F(X)\big),
$$
where $S(\cdot)$ is a spatial encoder, $F(\cdot)$ is a frequency-domain or frequency-derived encoder, and $\phi(\cdot)$ is a fusion operator such as selection, gating, attention, residual mixing, or state-space interaction.

The scope of “frequency” is not uniform across papers. In classical multi-focus fusion, “spatial frequency” is a sharpness criterion defined from pixel differences or DCT AC energy [2305.11265]. In FFT-based multimodal fusion, frequency means magnitude and phase spectra, sometimes with log compression [2508.08232] [2605.13049]. In wavelet-based designs, frequency is represented by subbands such as LL, LH, HL, and HH [2508.15505] [2405.01992] [2605.25804]. By contrast, FusionMamba is frequently adjacent to this literature but is not a true spatial-frequency model in the Fourier, DCT, or wavelet sense: in that paper, “spectral” strictly refers to wavelength-wise bands, and “there is no Fourier/DCT/Wavelet transform” [2404.07932].

This terminological distinction matters because “spectral fidelity” in hyperspectral or pansharpening systems is not the same object as “frequency preservation” in FFT- or wavelet-based fusion. A plausible implication is that papers using “spatial-spectral” and papers using “spatial-frequency” often solve analogous fusion problems with materially different signal models [2404.07932] [2508.15505].

## 2. Representation domains and transform choices

The transform layer is the defining design choice of a joint spatio-frequency block. The earliest formulation in the cited set is DCT-based multi-focus fusion, where each aligned image pair is partitioned into non-overlapping blocks, transformed with the 2D DCT, scored by spatial frequency, and selected by a thresholded decision map refined through Consistency Verification over a $3\times 3$ neighborhood [2305.11265]. In this formulation, the frequency signal is explicit, localized, and blockwise.

Wavelet-based formulations replace block DCTs with multi-resolution decompositions. AdaSFFuse introduces AdaWAT, which decouples each modality’s feature map into LL, LH, HL, and HH subbands at quarter resolution using adaptive analysis vectors embedded via grouped and dilated convolutions [2508.15505]. SFFNet uses fixed Haar wavelet decomposition in its Wavelet Transform Feature Decomposer, then projects low-frequency and concatenated high-frequency subbands before aligning them with global and local spatial features in MDAF [2405.01992]. MSFET-E2V likewise uses channel-wise Haar DWT, but its skip block selectively refines the $X_{HH}$ subband before inverse reconstruction, while CDAM feeds wavelet-derived features into transformer attention [2605.25804].

FFT-based blocks use global spectra rather than localized subbands. Mamba-FCS computes an orthonormal 2D FFT per channel, then uses the log-amplitude spectrum
$$
F_i^{T_j} = \log(1 + |\mathrm{FFT2}(X_i^{T_j})|)
$$
as a frequency feature, discarding phase entirely [2508.08232]. DSFF in SFRF decomposes registered infrared and visible inputs into amplitude and phase, processes each with $1\times 1$ convolutions, and fuses them with residual references to visible phase and infrared amplitude [2605.13049]. DFFNet applies channel-wise 2D FFT, generates a dynamic filter kernel in the frequency domain by pooling the input and synthesizing a weighted combination of learnable basis filters, then applies IFFT and complementary spatial/frequency depthwise convolutions [2507.04510].

A different route is frequency decoupling without Fourier coefficients. FreDFuse in FUSE uses Gaussian-Laplacian pyramids to separate low-frequency Gaussian components from high-frequency Laplacian details, then performs event-driven high-frequency and image-driven low-frequency cross-attention [2503.19739]. SFAFNet’s FDGM is also FFT-free: it learns a spatially variant low-pass filter via Softmax-normalized kernels and defines the complementary high-pass operator as $F^H = I - F^L$ [2502.14209].

## 3. Recurrent fusion mechanisms

Across these papers, several recurrent fusion mechanisms appear.

A first pattern is **decision-based fusion with spatial regularization**. In the DCT multi-focus method, the block does not learn a latent fusion map. Instead it computes per-block spatial frequency, forms a ternary decision map $W_{i,j}\in\{-1,0,+1\}$, and refines isolated decisions through a local majority-consistency check [2305.11265]. This is the most literal interpretation of a joint spatio-frequency block: frequency scores decide, spatial neighborhoods regularize.

A second pattern is **parallel spatial and frequency branches followed by attention or residual merging**. SFDFusion exemplifies this structure: DMRM extracts complementary spatial features, FDFM fuses amplitude and phase via FFT/IFFT, and a lightweight head concatenates spatial and frequency outputs for reconstruction [2410.22837]. DSFF follows a related dual-branch design, with a Restormer spatial branch and an FFT-based frequency branch whose outputs are refined by residual blocks [2605.13049].

A third pattern is **frequency-guided modulation of spatial hidden states**. ISFM is explicit here: its MFF computes fused low- and high-frequency wavelet features, FGG converts them into gating signals, and FGM modulates VMamba/VSSM hidden states element-wise before residual aggregation [2602.04405]. This is not cross-attention; the paper states that ISF injects frequency-domain cues into the spatial fusion path via gating [2602.04405].

A fourth pattern is **joint state-space parameterization by spatial and frequency selectors**. AdaSFFuse’s Spatial-Frequency Mamba Block uses a spatial-aware branch $\mathcal{S}(\cdot)$ and a frequency-filtering branch $\mathcal{T}(\cdot)$, merges them, splits the result into $\{X,B,C,A\}$, and drives a 2D-SSD update [2508.15505]. The state-space parameters are therefore conditioned by both spatial and frequency signals rather than by attention maps alone.

A fifth pattern is **hybrid spatial-frequency restoration blocks**. SFAFNet’s GSFFBlock combines a spatial NAFBlock stack, FDGM low-/high-frequency decomposition, channel gates derived from global mean and standard deviation, and cross-attention across spatial, low-frequency, and high-frequency streams [2502.14209]. DFFNet reaches a comparable end through a different decomposition: dynamic FFT filtering in DFB and cross-modal spectral-spatial interaction in SSAFB [2507.04510].

## 4. Representative formulations

Several formulations are especially representative of the concept.

In DCT-domain multi-focus fusion, the core selection variable is
$$
W_{i,j} =
\begin{cases}
+1, & \text{if } SF_A(i,j)-SF_B(i,j) > T,\\
-1, & \text{if } SF_B(i,j)-SF_A(i,j) > T,\\
0, & \text{otherwise,}
\end{cases}
$$
followed by local vote accumulation
$$
R_{i,j} = \sum_{x=i-1}^{i+1}\sum_{y=j-1}^{j+1} W_{x,y}.
$$
The fused block then inherits the DCT coefficients from $A$ or $B$ according to the sign of $R_{i,j}$, with averaging or tie-breaking when $R_{i,j}=0$ [2305.11265]. Here, spatial-frequency fusion is literally a fusion of spatial neighborhood logic and frequency-domain block energy.

In Mamba-FCS, the Joint Spatio-Frequency Fusion block at decoder stage $i$ concatenates spatial features, log-amplitude spectra, and the spatial difference:
$$
X_i^{cat} = \mathrm{Concat}(X_i^{T_1}, F_i^{T_1}, X_i^{T_2}, F_i^{T_2}, D_i),
$$
compresses the $5C_i$ channels with a $1\times 1$ convolution, and refines the result with channel-first and then spatial CBAM attention [2508.08232]. No inverse FFT is used; the learned projection and attention align spatial tensors with raw log-amplitude spectra inside the decoder.

AdaSFFuse formalizes a stronger state-space interaction. Its frequency branch computes
$$
\Lambda = \mathrm{FFT}(L'), \qquad
L_{\mathcal{T}} = \mathrm{IFFT}\big(\Lambda \odot (\|\Lambda\|^2 > \lambda)\big),
$$
while the spatial branch computes
$$
L_{\mathcal{S}} = \sigma(W_{se}(L')).
$$
The merged representation is then split into $\{X,B,C,A\}$ and used in the 2D-SSD recurrence
$$
H_{ij,t} = A_t H_{ij,t-1} + B_t X_{ij,t}, \qquad
Y_{ij,t} = C_t^\top H_{ij,t}.
$$
This places spatial-frequency fusion inside the state update itself rather than after feature extraction [2508.15505].

ISFM gives perhaps the cleanest gating formulation. Frequency guidance is summarized as
$$
Z_g = Z_f \otimes Z_{global} + Z_{global},
$$
split into two streams, and projected to gates $G_{ir}$ and $G_{vi}$. The modulated spatial fusion is then
$$
F_h = H_{ir}\otimes G_{ir} + H_{vi}\otimes G_{vi},
$$
followed by
$$
F_{spa}^{fuse} = \mathrm{Linear}(F_h) + s_1\cdot F_{ir}^M + s_2\cdot F_{vi}^M.
$$
The block is therefore interactive, but the interaction is multiplicative gating rather than attention [2602.04405].

In DSFF, the amplitude and phase branches are fused asymmetrically:
$$
P_f = a \cdot F_{ir}^P + (1-a)\cdot F_{vis}^P + P_{vis},
$$
$$
A_f = SA(F_{ir}^A)\odot F_{vis}^A + A_{ir},
$$
followed by inverse FFT and a residual spatial-frequency refinement head [2605.13049]. This asymmetry reflects task intent: preserve thermal saliency through infrared amplitude while keeping visible phase as a structural reference.

## 5. Application areas and empirical behavior

These blocks have been used across fusion, restoration, change detection, depth estimation, segmentation, classification, and event reconstruction.

For multi-focus fusion in JPEG or DCT-domain settings, the DCT + SF + CV method reported the best average RMSE and SSIM among the listed baselines, improving from RMSE $4.220$ and SSIM $0.9898$ for DCT + SF to RMSE $4.037$ and SSIM $0.9902$ with consistency verification [2305.11265]. This supports the classical view that blockwise spatial-frequency criteria remain effective when transform coefficients are already available.

For semantic change detection, Mamba-FCS reported $88.62\%$ Overall Accuracy, $65.78\%$ $F_{scd}$, and $25.50\%$ SeK on SECOND, and $96.25\%$ Overall Accuracy, $89.27\%$ $F_{scd}$, and $60.26\%$ SeK on Landsat-SCD. Removing the FFT2 branch caused drops of $0.76\%$ OA, $1.26\%$ $F_{scd}$, $0.91\%$ mIoU, and $1.47\%$ SeK, directly quantifying the contribution of its Joint Spatio-Frequency Fusion unit [2508.08232].

For task-generalized multimodal image fusion, AdaSFFuse reported approximately $0.78$M parameters, $82.81$G FLOPs at $512\times 512$, and runtime approximately $70.13$ ms per image, while its full model reached SSIM $1.51$ in the reported ablations, compared with $0.89$ for the baseline and $1.41$ after adding the shallow SFM stage [2508.15505]. This suggests that spatial-frequency co-fusion is not merely a reconstruction refinement but a core representation-learning component.

For image-event depth estimation, FUSE reported a $14\%$ Abs.Rel improvement on MVSEC and $24.9\%$ on DENSE relative to prior methods, with ablations showing that FreDFuse improved Abs.Rel from $0.366$ to $0.336$ on MVSEC outdoor\_day1 compared with cross-attention-only fusion [2503.19739]. The useful point here is methodological: explicit low-/high-frequency role assignment can outperform modality-agnostic attention.

For deblurring, SFAFNet’s full GSFFBlock improved GoPro PSNR from $32.77$ dB for the baseline to $33.52$ dB, with intermediate gains from adding FDGM, GATE, and CAM. The larger SFAFNet-B reached $34.25$ dB and $0.971$ SSIM on GoPro [2502.14209]. This indicates that joint spatial-frequency blocks also function effectively in single-image restoration, not only in multimodal fusion.

For remote sensing segmentation, SFFNet reported mIoU $84.80\%$ on Vaihingen and $87.73\%$ on Potsdam, with ablations showing mIoU drops when low-frequency WTFD-L, high-frequency WTFD-H, or MDAF were removed [2405.01992]. For event-to-video reconstruction, MSFET-E2V reported $16.71$M parameters and inference times of $16.8$ ms at $240\times 180$, $26.4$ ms at $346\times 260$, and $35.73$ ms at $640\times 480$, while its ablations showed that using both LF and HF wavelet paths in CDAM outperformed LF-only and HF-only variants on ECD, HQF, and MVSEC [2605.25804].

## 6. Limitations, misconceptions, and extensions

A recurrent limitation is **alignment sensitivity**. The DCT multi-focus method assumes perfectly registered source images, and even sub-pixel misalignment can cause incorrect block correspondence [2305.11265]. FusionMamba explicitly notes that misregistration or parallax is not handled, despite its dual-input state-space fusion [2404.07932]. SFRF was proposed precisely because cumulative registration errors contaminate downstream fusion, and its DSFF depends on MIR to provide aligned infrared inputs [2605.13049].

A second limitation is **incomplete frequency modeling**. Mamba-FCS discards phase and uses only log-amplitude spectra [2508.08232]. This improves illumination robustness, but the paper also notes that phase-dependent structures may not be captured well. Conversely, amplitude-phase methods such as DSFF must manage the instability of phase processing and the ambiguity of internal PF/SA implementations, which are only partially specified in the paper [2605.13049].

A third limitation is **artifact trade-offs**. Blockwise DCT selection can create blockiness near focus transitions, which CV reduces but does not eliminate [2305.11265]. High-frequency branches can amplify noise: HFFB in ISFM is therefore built around subtraction against pooled maps to suppress random high-frequency components while keeping edges [2602.04405], and WSB in MSFET-E2V selectively processes the $X_{HH}$ band because shallow high-frequency features often carry noise-like artifacts [2605.25804].

A common misconception is to conflate **spectral** with **frequency**. In remote sensing fusion, spectral fidelity often means preserving wavelength-band consistency, not preserving Fourier or wavelet coefficients. FusionMamba is explicit on this point and even sketches how true spatio-frequency fusion could be added by inserting DCT, FFT, or wavelet stages before its state-space block [2404.07932]. This suggests that the phrase “joint spatio-frequency fusion block” should be reserved for modules whose second branch is genuinely transform-domain, or at least frequency-decoupled, rather than merely multi-band.

The extension direction most repeatedly proposed in the cited literature is toward **adaptive, learnable cross-domain coupling**: explicit spatio-frequency fusion in Mamba-style recurrent dynamics [2404.07932] [2508.15505], adaptive wavelets or learnable subband weighting instead of fixed Haar filters [2605.25804], cross-modality use beyond canonical image pairs [2404.07932], and more robust handling of noise, parallax, and downstream-task sensitivity [2508.15505] [2602.04405]. A plausible synthesis is that the field is moving from serial “spatial branch plus frequency branch” designs toward blocks in which frequency signals directly parameterize recurrence, gating, or decoder reasoning.

Source: https://www.emergentmind.com/topics/joint-spatio-frequency-fusion-block