Frequency-Aware Spatial Attention (FSA)

• Frequency-aware spatial attention is a design pattern that fuses spectral transforms (like DCT/FFT) with spatial attention to model both global structure and local features.
• It employs dual-domain pipelines that separate and enhance low-frequency semantics and high-frequency details, supporting tasks such as segmentation and action recognition.
• Empirical results across various applications demonstrate that integrating FSA can significantly improve performance metrics and computational efficiency.

Searching arXiv for papers and terminology variants relevant to Frequency-aware Spatial Attention. Search query: "Frequency-aware Spatial Attention" arXiv Frequency-aware Spatial Attention (FSA) denotes a family of attention mechanisms that couple spatial-domain feature learning with explicit frequency- or spectral-domain processing so that global structure, local boundaries, and fine-grained detail can be modeled jointly. In the literature represented here, the term is not standardized: closely related mechanisms appear as the Frequency-aware Attention Block (FAB) in FreqMixFormer for skeleton action recognition, the Frequency-Spatial Attention block in SF-UNet, the frequency self-attention of FsaNet, the Spatial-Frequency Attention module in weather forecasting, and the bottleneck-level FA-Attention of MedFlowSeg; by contrast, “FSA” in FSA-YOLOv5 means “full separation attention” and is not a frequency-domain method (Wu et al., 2024, Zhou et al., 2024, Zhang et al., 2022, Wang et al., 2024, Chen et al., 21 Apr 2026, Zhang et al., 2023).

1. Terminological scope and historical usage

Across recent arXiv literature, FSA is better understood as a design pattern than as a single canonical block. The common pattern is to expose a representation to both spatial structure and spectral structure, then use attention, gating, or adaptive fusion to preserve discriminative information that would otherwise be diluted by ordinary spatial-only processing. In segmentation, this often targets edges, textures, and global anatomical priors; in action recognition, it targets subtle rapid motions; in denoising and forecasting, it targets long-range dependency and repeated structure (Zhou et al., 2024, Wu et al., 2024, Guo et al., 2023, Wang et al., 2024).

Paper	Term used	Role
"Frequency Guidance Matters: Skeletal Action Recognition by Frequency-Aware Mixed Transformer" (Wu et al., 2024)	FAB	Frequency-domain counterpart of SAB
"Spatial-Frequency Dual Progressive Attention Network For Medical Image Segmentation" (Zhou et al., 2024)	FSA	Frequency-spatial refinement block
"FsaNet: Frequency Self-attention for Semantic Segmentation" (Zhang et al., 2022)	FSA	Low-frequency DCT self-attention
"Smart Home Device Detection Algorithm Based on FSA-YOLOv5" (Zhang et al., 2023)	FSA	Full Separation Attention, not frequency-aware

This terminological dispersion matters because superficially similar names correspond to different operators. Some methods compute attention directly in the spectral domain, some only derive gating weights from frequency features, some fuse spatial and frequency streams bidirectionally, and some use wavelet or Fourier branches without calling the result FSA. A plausible implication is that “frequency-aware spatial attention” is best treated as an umbrella concept spanning several architectural instantiations rather than a single standardized primitive.

2. Core operational principle

The most consistent operational motif is a dual-domain pipeline: transform features into a frequency representation, compute or modulate an attention-like map there, and then either reconstruct back to the spatial domain or fuse spectral and spatial branches explicitly. In FreqMixFormer, the starting point is standard transformer attention,

$Q = XW_Q,\quad K = XW_K,\quad V = XW_V,$

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$

but the paper argues that vanilla attention “globally normalizes correlations and can dilute subtle but discriminative joint interactions.” Its remedy is to apply the Discrete Cosine Transform (DCT) to each joint trajectory, compute a frequency-domain joint-correlation map, reweight the frequency distribution through a Frequency Operator, and map the result back by IDCT before spatial fusion (Wu et al., 2024).

In SF-UNet, the FSA block uses a two-branch structure. One branch applies a 2D DFT, separates low- and high-frequency components by binary masks, applies a learnable global filter only to the low-frequency part, preserves the high-frequency spectrum directly, and reconstructs by IDFT. The other branch applies a spatial attention module adapted from CBAM. The two outputs are then added:

$f_i' = f_i^{high} \oplus (f_i^{low}\otimes N),$

$\widehat{F_i}=\hat{F_i''}\oplus SA(\hat{F_i'}).$

This realizes frequency-awareness not by discarding detail, but by preserving high-frequency content while adaptively reweighting global low-frequency structure (Zhou et al., 2024).

MedFlowSeg places its FA-Attention at the bottleneck of a conditional flow matching segmentation model. Here the problem is a domain gap between the noisy intermediate flow state $x_t$ and the clean semantic condition features from the image. The module therefore models interactions in both the spatial domain and the frequency domain. In the frequency branch, FFT, FiLM modulation, patch embedding, positional encoding, and cross-attention are used; in the spatial branch, a Token Discrepancy Extractor constructs

$$T_a = T_f \odot T_c, \quad T_d = |T_f - T_c|, \quad T_r = T_f - T_c,$$

and a Neural Modulator applies time-conditioned fusion before IFFT maps the refined representation back to the spatial domain (Chen et al., 21 Apr 2026).

A related cross-domain formulation appears in weather forecasting. SFANet generates a spatial feature $F_s$ and a frequency feature $F_f$, then injects bidirectional relation terms:

$$\hat{F_s} = F_s + \mathcal{R}(F_s, F_f), \qquad \hat{F_f} = F_f + \mathcal{R}(F_f, F_s),$$

followed by concatenation,

$F_i = \operatorname{Concat}(\hat{F_s}, \hat{F_f}).$

In this case, frequency-aware spatial attention is not a spatial mask but a learned relation module between two complementary token spaces (Wang et al., 2024).

3. Major design variants

One major axis of variation is which spectral components are emphasized. FsaNet argues that semantic segmentation mainly needs low-frequency attention because object interiors are mostly low-frequency content and object boundaries are mostly high-frequency content. It therefore applies a 2D DCT per channel, keeps only a low-frequency $$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$0 block, and performs self-attention on those coefficients. With the paper’s linearized structure, the output can be written as

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$1

The paper reports that low frequency self-attention can achieve very close or better performance relative to full frequency even without retraining the network, while reducing memory, FLOPs, and runtime sharply (Zhang et al., 2022).

FreqMixFormer makes the opposite design choice. It explicitly states that unlike many prior DCT methods that discard high-frequency coefficients, it keeps all coefficients and selectively enhances high-frequency ones and suppresses low-frequency ones. In its Frequency-aware Attention Block, the DCT-transformed branch mirrors the spatial attention branch, and the frequency-aware map is

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$2

with $$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$3 controlling the Frequency Operator. High-frequency coefficients are amplified by $$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$4, while low-frequency coefficients are reduced by $$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$5. This design is motivated by confusing actions such as reading, writing, or phone use, which differ primarily in subtle, localized motions that are more visible as high-frequency components than in raw spatial coordinates (Wu et al., 2024).

SF-UNet occupies a third position. It preserves high-frequency content unchanged, applies learnable filtering only to the low-frequency branch, and then fuses the reconstruction with spatial attention. This suggests a division of labor in which global semantics are adjusted through low-frequency filtering while edge and boundary detail are protected from attenuation (Zhou et al., 2024).

A fourth variant replaces direct spectral attention with discrepancy-aware fusion or tri-branch fusion. MedFlowSeg does not merely compute FFT-domain attention; it aligns noisy flow features and clean semantic priors through discrepancy-aware token cues and time-conditioned modulation (Chen et al., 21 Apr 2026). In "Decoding with Structured Awareness: Integrating Directional, Frequency-Spatial, and Structural Attention for Medical Image Segmentation" (Zhang et al., 5 Dec 2025), the closest equivalent to FSA is the Triple Feature Fusion Attention (TFFA) module, which fuses spatial, Fourier, and wavelet branches and then applies adaptive weighting by attention gating. The Fourier branch supplies global structural dependency, the wavelet branch supplies localized edge and texture information, and the spatial branch preserves direct spatial continuity (Zhang et al., 5 Dec 2025).

4. Placement within larger architectures

FSA-like modules are rarely standalone; they are normally embedded at task-specific bottlenecks, heads, shallow stages, or fusion layers. In FreqMixFormer, the architecture is explicitly organized around three components: SAB for spatial joint correlations, FAB for frequency-aware joint correlations, and TAB for temporal correlation across frames. The mixed frequency-spatial representation is formed by

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$6

and the temporal transformer receives

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$7

Here the frequency-aware mechanism is the spatial-frequency front end for later temporal modeling rather than the temporal model itself (Wu et al., 2024).

In medical segmentation, placement varies by objective. MedFlowSeg applies FA-Attention at the bottleneck, where deep condition features and flow features interact; the Neural Modulator uses a default depth $$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$8, and the full FA-Attention module is repeated $$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$9 times (Chen et al., 21 Apr 2026). SF-UNet places FSA after the Multi-scale Progressive Channel Attention block, so MPCA first performs cross-scale semantic enrichment and FSA then refines those fused features with dual-domain processing (Zhou et al., 2024). TFFA in (Zhang et al., 5 Dec 2025) is a decoder module rather than an encoder or bottleneck module.

Other applications illustrate different insertion strategies. In denoising, SFANet uses a U-Net backbone with a Spatial Attention Module and a Frequency Attention Module containing the WFCA block; the frequency branch uses fixed-window FFT to avoid training–inference mismatch in frequency resolution (Guo et al., 2023). In weather forecasting, the Spatial-Frequency Attention module is placed after dual token-mixer encoder branches and before the predictor (Wang et al., 2024). In cross-view geo-localization, MFAF inserts its FSA inside the MFAF module after the EVA02-L backbone, refining original, low-frequency, and high-frequency branches before training-time classification and test-time descriptor concatenation (Liu et al., 16 Sep 2025). In SAR ATR, the relevant mechanism is DSAF, a shallow-stage frequency-spatial enhancement module combining multi-scale convolutions, wavelet convolution, and CBAM rather than a block explicitly named FSA (Lin et al., 23 Mar 2026). In thermal prediction for 2.5D ICs, the frequency-spatial theme is distributed across FSTE, FCIFormer, and FSL rather than a single attention block (Zhang et al., 19 Apr 2025).

5. Empirical findings and ablation evidence

The empirical record in these papers consistently attributes measurable gains to the frequency-aware component, although the precise benefit depends on transform choice, placement, and fusion strategy. In FreqMixFormer, on NTU-60 X-Sub, the baseline spatial MixFormer gets 89.8%, adding FAB improves it to 91.0%, adding FO raises it to 91.2%, and the full model with FAB + FO + TAB reaches 91.5%. The same study reports that the partial DCT version substantially outperforms the full DCT version: 91.5% vs. 87.7%, and the best $f_i' = f_i^{high} \oplus (f_i^{low}\otimes N),$0 is 0.5 (Wu et al., 2024).

SF-UNet reports that adding FSA alone to UNet increases parameters only from 24.89M to 24.94M, while the BUSI ablation changes from DSC = 73.27, IOU = 64.30 to DSC = 74.10, IOU = 65.11. The paper also describes the FSA block as having only 0.05M parameters, while the full SF-UNet reaches DSC = 88.46, IOU = 81.34 on ISIC-2018 and reports up to +9.4% DSC and +10.78% IOU over previous SOTA (Zhou et al., 2024).

MedFlowSeg presents a more explicit comparison against standard spatial attention. Its ablation progression is: baseline 88.3 / 87.6 / 84.2 / 87.1, then + SA gives 88.7 / 88.5 / 85.0 / 86.9, + DB-SA gives 90.4 / 89.0 / 86.4 / 90.3, + DB-SA + FA-Attention (w/o Mod.) gives 91.8 / 90.7 / 87.9 / 91.5, and + DB-SA + FA-Attention + Neural Modulator gives 93.4 / 91.2 / 88.6 / 92.8 on ACDC, Brain Tumor, Optic Cup, and Histology Gland, respectively. The paper therefore associates the best results with the combination of spectral interaction, discrepancy-aware fusion, and dynamic time-aware modulation (Chen et al., 21 Apr 2026).

FsaNet emphasizes efficiency alongside accuracy. Relative to regular self-attention, it reports 87.29% \sim 90.04% less memory, 96.13% \sim 98.07% less FLOPs, and 97.56% \sim 98.18% less runtime. On Cityscapes test, FsaNet-Dot-R2: 83.05 and FsaNet-Lin-R2: 82.81 are reported, with the paper describing this as a new state-of-the-art among ResNet-101-based methods in that comparison (Zhang et al., 2022).

The same pattern appears in other domains. In weather forecasting, adding Transformer-based SFA changes SEVIR ablation results from CSI-M = 0.4433, CSI-219 = 0.1394, MSE = 3.2694 to CSI-M = 0.4692, CSI-219 = 0.1820, MSE = 2.7308 (Wang et al., 2024). In MFAF for cross-view geo-localization, moving from +MFB to +FSA (full model) changes University-1652 results from D2S R@1=94.29, AP=95.24; S2D R@1=95.44, AP=94.64 to D2S R@1=95.06, AP=95.89; S2D R@1=96.01, AP=95.07, and the paper states that the chosen Z-Pooling strategy is best (Liu et al., 16 Sep 2025). In TFFA, the internal ablation progresses from 90.32 DSC with no Fourier or wavelets to 90.71 DSC with Fourier + Mexican Hat + DoG (Zhang et al., 5 Dec 2025).

6. Relation to conventional attention, misconceptions, and open design axes

Frequency-aware spatial attention differs from standard spatial attention in purpose as well as mechanism. Standard spatial attention, as explicitly stated in MedFlowSeg, “works only in the spatial domain,” “learns local reweighting of features,” “does not explicitly account for the mismatch between noisy flow states and clean semantic priors,” and in ablation “brings only marginal improvement.” Frequency-aware designs instead introduce spectral structure, cross-domain relation modeling, or discrepancy-aware refinement so that attention is conditioned not only on spatial saliency but also on how information is distributed across frequency bands (Chen et al., 21 Apr 2026).

A common misconception is that FSA always denotes the same operator. The papers here show otherwise. FsaNet uses low-frequency DCT tokenization and linearized self-attention (Zhang et al., 2022); FreqMixFormer keeps all DCT coefficients and explicitly boosts high-frequency motion cues (Wu et al., 2024); SF-UNet preserves the high-frequency branch and learns on the low-frequency branch (Zhou et al., 2024); MFAF uses average pooling and max pooling plus a two-layer $f_i' = f_i^{high} \oplus (f_i^{low}\otimes N),$1 bottleneck and sigmoid gating on frequency branch features (Liu et al., 16 Sep 2025); and FSA-YOLOv5 uses “full separation attention,” which is a spatial-channel hybrid module rather than a frequency-domain mechanism (Zhang et al., 2023).

This suggests that the main research axes are transform choice, coefficient policy, fusion strategy, and insertion stage. The transforms include DCT, FFT, DFT, and wavelets. The coefficient policies range from low-frequency-only attention, to all-coefficient retention with high-frequency enhancement, to explicit low/high splitting with separate handling. Fusion strategies include addition, concatenation, cross-attention, adaptive gating, discrepancy-aware recalibration, and residual bidirectional relation learning. Insertion stages span shallow feature enhancement, bottleneck semantic alignment, decoder fusion, segmentation heads, and dual-branch pre-predictor fusion. The literature therefore supports a broad but technically coherent interpretation: FSA is a spectral–spatial inductive bias for attention, not a single universally fixed block.