Frequency Adaptive Transformer (FAT)

• FAT is a family of transformer designs that decompose features into low- and high-frequency components for improved global context and fine detail preservation.
• It employs techniques such as latent-space decomposition, Fourier/wavelet-domain processing, and adaptive gating to modulate attention and feed-forward operations.
• Empirical studies show FAT’s effectiveness in underwater image enhancement, hyperspectral reconstruction, and multivariate time-series forecasting, yielding significant performance gains.

Frequency Adaptive Transformer (FAT) denotes a class of transformer designs that modify representation learning, feature fusion, or restoration behavior according to frequency structure. In the narrowest explicit usage, the term refers to the Frequency Adaptive Transformer bottleneck introduced in SFormer for underwater image enhancement, where deep bottleneck features are decomposed into low-frequency/global and high-frequency/detail components and then fused by attention and gating (Tian et al., 26 Aug 2025). In a broader descriptive sense, FAT also serves as an umbrella for related transformer architectures that incorporate Fourier-domain operations, wavelet decomposition, low/high-frequency specialization, or instance-conditioned spectral modulation in vision, time-series forecasting, EEG modeling, and inverse problems. This broader use is conceptually useful but terminologically unstable, because papers use closely related labels such as frequency-aware transformer, frequency-enhanced transformer, or frequency-aware inverted transformer, and some papers use the acronym FAT for unrelated meanings such as Fully Adaptive Transformer (Fan et al., 2023).

1. Terminology and scope

The literature does not present a single canonical meaning of “FAT.” The most explicit use is in SFormer, where FAT is a named bottleneck module. Other papers are clearly adjacent in mechanism or motivation, but use different names and sometimes explicitly distinguish themselves from a strict FAT formulation. This suggests that FAT is best understood as a family resemblance term rather than a universally standardized architectural label.

Paper	Label used in paper	Relation to FAT
SFormer (Tian et al., 26 Aug 2025)	Frequency Adaptive Transformer	Explicit FAT bottleneck
LIC transformer (Li et al., 2023)	Frequency-Aware Transformer	Related, but not called adaptive
FAiT (He et al., 31 May 2026)	Frequency-Aware Inverted Transformer	FAT-like by spectral adaptation
PF-Trans (Gui et al., 9 Jun 2026)	Physics-embedded Frequency-aware Transformer	Conceptual overlap, not canonical FAT
FAT family (Fan et al., 2023)	Fully Adaptive Transformer	Unrelated to frequency

The sharpest terminological distinction is between frequency-adaptive and frequency-aware. In SFormer, FAT is adaptive because it separates low- and high-frequency components and fuses them with gated attention at the bottleneck. In the learned image compression paper, by contrast, the formal name is Frequency-Aware Transformer, with multiscale directional decomposition in attention and FFT-based feed-forward modulation, but not the exact “Frequency Adaptive Transformer” label (Li et al., 2023). PF-Trans is also explicitly described as relevant to FAT only in a broad descriptive sense: it is a dual-domain, physics-guided transformer with an FFT branch, not a transformer whose attention weights are dynamically conditioned on frequency tokens or band-pass routing (Gui et al., 9 Jun 2026).

A second source of ambiguity is acronym collision. The 2023 “FAT family” paper defines FAT as Fully Adaptive Transformer, centered on bidirectional local-global interaction in spatial features rather than frequency modeling, and therefore falls outside the frequency-adaptive meaning despite the identical acronym (Fan et al., 2023).

2. Core architectural motifs

Across the frequency-adaptive and frequency-aware transformer literature, several recurrent mechanisms appear. One recurring premise is that conventional self-attention behaves like a smoothing or low-pass operator, which motivates architectural corrections that preserve or recover high-frequency content. This motivation is explicit in image inpainting, single-image super-resolution, and multivariate time-series forecasting: Dabformer attributes transformer weakness on detail restoration to the low-pass nature of self-attention, CRAFT is built on the hypothesis that transformers are stronger on low-frequency/global content than on high-frequency detail, and FAiT interprets the attention matrix itself as a learnable low-pass operator (He et al., 23 Jun 2025).

A first major motif is explicit decomposition into low- and high-frequency components. In SFormer’s FAT bottleneck, this decomposition is performed in feature space rather than by FFT: $$F_{low} = \text{GAP}(\text{LN}(P'_i)), \qquad F_{high} = \text{LN}(P'_i) - F_{low}.$$ The low-frequency branch is then modulated by channel attention and the high-frequency branch by spatial attention: $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$ This shows that FAT need not require a literal Fourier transform; it may instead use a learned latent-space surrogate for global versus detail content (Tian et al., 26 Aug 2025).

A second motif is explicit Fourier- or wavelet-domain processing inside the transformer pipeline. PF-Trans adds a parallel FFT branch to a spectral transformer backbone, applies learnable complex convolution in Fourier space, and reconstructs by inverse FFT with residual addition: $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$ FEFormer’s FDSA similarly computes attention through FFT-transformed queries and keys, while its WAFF module uses 3D Haar wavelet subbands for encoder–decoder fusion; Dabformer combines wavelet decomposition with Gabor filtering in attention and replaces the conventional FFN with an FFT-based learnable spectral filter (Gui et al., 9 Jun 2026).

A third motif is adaptive spectral gating or modulation. FAiT makes this explicit by constructing a complementary high-pass branch from attention itself: $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$ and then fusing the two with adaptive gates before applying Dynamic Temporal-Frequency Modulation in the Fourier domain. FreEformer instead applies the transformer directly to DFT spectra and augments vanilla attention with an additive learnable matrix followed by row-wise $L_1$ normalization to counter low-rank behavior in sparse frequency-domain attention (He et al., 31 May 2026).

A fourth motif is frequency-aware fusion across branches or scales. CGFformer uses cluster-guided adaptive high/low-frequency separation, dual-stream refinement with cross-attention between frequency streams, and a final spatial-frequency attention module for reconstruction. HistoFusionNet combines a transformer-side adaptive bottleneck based on dynamic-range grouping with a Fourier low/high-frequency refinement stage. These designs indicate that FAT-like systems often extend beyond a single attention operator and instead restructure the entire encoder–decoder or inverse-problem pipeline around frequency-specific branches and fusion rules (Zhou et al., 2 May 2026).

3. The explicit FAT bottleneck in SFormer

In the most literal sense, FAT refers to the bottleneck module of SFormer, a dual-branch U-shaped architecture for underwater image enhancement. One encoder branch processes the RGB image $I_{input} \in \mathbb{R}^{3 \times H \times W}$, while the other processes an SNR map $S \in \mathbb{R}^{1 \times H \times W}$. Both branches are downsampled through four convolutional stages, after which the bottleneck resolution is $\frac{H}{16}\times\frac{W}{16}$. FAT is placed precisely at this encoder–decoder bridge, not in the skip paths and not as a decoder refinement block (Tian et al., 26 Aug 2025).

Its input bottleneck feature is denoted

$$M \in \mathbb{R}^{C \times \frac{H}{16} \times \frac{W}{16}},$$

and both RGB and SNR features are patch-embedded with a $4\times4$ convolution of matching stride. The first explicit operation is SNR-guided cross-attention: $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$0 where $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$1 and $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$2 are RGB and SNR token embeddings. Transformer blocks then follow the standard pre-norm form

$$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$3

$$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$4

The distinctive mechanism lies in the adaptive feed-forward network (AFFN). Instead of a conventional token-wise MLP, AFFN converts tokens back to spatial maps, separates low- and high-frequency components using global average pooling and residual subtraction, and processes them with different attentional biases: $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$5

$$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$6

The fused feature then passes through depth-wise convolution, channel chunking, and gated multiplicative activation: $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$7

$$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$8

This mechanism is adaptive in two senses. First, low-frequency/global context and high-frequency/detail content are treated differently rather than uniformly. Second, their contributions are modulated content-dependently by channel attention, spatial attention, and the gated depth-wise feed-forward path. Importantly, this is not Fourier-domain splitting in the strict signal-processing sense; it is a latent-space decomposition into global and residual components. This distinction is central to the SFormer definition of FAT (Tian et al., 26 Aug 2025).

The empirical evidence in SFormer directly supports the utility of FAT as a bottleneck replacement. On the EUVP ablation study, the paper reports the following progression: Baseline (BL) achieves PSNR 21.145 and SSIM 0.911; BL + ViT reaches 22.759 and 0.914; BL + FAT reaches 24.543 and 0.924; BL + FAST reaches 24.852 and 0.926; and the full model BL + FAT + FAST reaches 27.451 and 0.936. Relative to a plain ViT bottleneck, replacing it with FAT yields a $$F' = \text{CA}(F_{low}) \cdot F_{low} + \text{SA}(F_{high}) \cdot F_{high}.$$9 dB PSNR improvement and improves SSIM by $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$0, while also lowering LPIPS and $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$1. The paper interprets this as evidence that FAT is not merely another transformer block but a more suitable bottleneck for balancing global restoration with detail preservation (Tian et al., 26 Aug 2025).

4. Vision and inverse-problem variants

Outside SFormer, FAT-like design has diversified most strongly in imaging and inverse problems. PF-Trans is a representative example in hyperspectral reconstruction from broadband filter array measurements. It is described as a physics-embedded frequency-aware transformer rather than a generic FAT: the network receives a compressed grayscale measurement and the known BFA mask, reconstructs a hyperspectral cube, and enforces optical consistency through a gray-scale reprojection loss

$$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$2

combined with reconstruction loss using $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$3. Its frequency awareness comes from a parallel FFT branch with learnable complex convolution, not from frequency-conditioned self-attention. The paper therefore presents PF-Trans as conceptually related to FAT but not a one-to-one mechanism match (Gui et al., 9 Jun 2026).

FEFormer extends the idea much further in volumetric segmentation by making frequency adaptation a network-wide principle. Its Frequency-enhanced Dynamic Self-Attention (FDSA) computes attention via FFT-transformed queries and keys, its Frequency-decomposed Gating MLP (FGMLP) adaptively separates low- and high-frequency components inside the feed-forward path, its Wavelet-guided Adaptive Feature Fusion (WAFF) uses 3D Haar DWT to fuse encoder and decoder features subband-wise, and its Frequency-enabled Cross-scale Stem Bridge (FCSB) propagates low-level features through spectral interaction. On four volumetric segmentation tasks, FEFormer reports superior DSC/HD95 trade-offs and substantially lower model complexity than several 3D transformer baselines, with 18.54M parameters and 39.13G FLOPs (Yang et al., 12 May 2026).

The learned image compression paper formalizes another branch of the design space. Its Frequency-Aware Transformer block uses Frequency-Decomposition Window Attention (FDWA) with four window types—$$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$4, $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$5, $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$6, and $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$7—to model low-frequency, high-frequency, and directional content, and replaces the conventional FFN with the Frequency-Modulation Feed-Forward Network (FMFFN): $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$8 The paper explicitly notes that its formal name is Frequency-Aware Transformer, not Frequency Adaptive Transformer, yet the mechanism is clearly FAT-like in its subband-specific attention and spectral modulation (Li et al., 2023).

Other recent restoration models illustrate further variants. HistoFusionNet combines histogram transformer blocks at the bottleneck with a Fourier-domain low/high-frequency refinement module, but does not define a single monolithic FAT block. Dabformer for image inpainting uses wavelet decomposition and Gabor filtering to construct a frequency-aware query in attention and replaces the FFN with a learnable FFT-based filter. CRAFT for super-resolution does not perform explicit spectral attention, but it is organized around a frequency-inspired separation between a convolutional high-frequency prior branch and a transformer global/low-frequency branch. CGFformer for pansharpening uses cluster-guided adaptive frequency separation, dual-stream refinement, and spatial-frequency attention. Taken together, these works indicate that in imaging, FAT-style design often means hybridizing transformer attention with explicit spectral branches, adaptive low/high-frequency fusion, and task-specific priors such as physics models, wavelets, or clustering (Heydari et al., 4 Apr 2026).

5. Temporal, multivariate, and signal-processing variants

In temporal modeling, the strongest explicit analogue to a frequency-adaptive transformer is FAiT. The paper studies multivariate time series forecasting and interprets self-attention itself as a low-pass filter: $$\mathbf{Z} = \mathcal{F}(\mathbf{F}_{in}), \qquad \mathbf{F}_{out} = \mathbf{F}_{in} + \mathrm{LayerNorm}\big(\mathcal{F}^{-1}(\tilde{\mathbf{Z}})\big).$$9 It then defines a complementary high-pass branch

$$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$0

fuses low and high streams with adaptive spectral gates, and applies Dynamic Temporal-Frequency Modulation (DTFM) in the Fourier domain using instance-conditioned prototype recombination. This is a strong example of a FAT-like forecaster because it combines explicit spectral correction of attention bias with instance-conditioned reweighting of spectral sub-bands (He et al., 31 May 2026).

FreEformer takes a different route by transforming each variable into the complex frequency domain via DFT and treating each variable’s entire spectrum as a token. Real and imaginary parts are processed independently, and vanilla attention is modified by adding a learnable matrix followed by row-wise $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$1 normalization: $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$2 The paper argues that vanilla attention in frequency space exhibits a low-rank characteristic because spectra are sparse and softmax is strongly value-focused; the additive matrix is introduced to improve feature diversity and gradient flow. FreEformer therefore represents a FAT-like model in which the transformer operates on spectral summaries directly rather than on time-domain tokens (Yue et al., 23 Jan 2025).

FreqDGT offers yet another pattern. It is not named FAT, but it cleanly separates the “frequency-adaptive” part and the “transformer” part. Frequency-Adaptive Processing (FAP) decomposes EEG features into $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$3 bands, computes band energies,

$$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$4

and forms sample-specific weights through AttentionNet and ImporNet before fusing masked band features: $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$5 A dynamic graph module then models connectivity, and a multi-scale temporal transformer learns temporal structure. In this sense, FreqDGT is a FAT-style system with explicit pre-transformer frequency adaptation and transformer-based temporal reasoning (Li et al., 28 Jun 2025).

These temporal and signal models also clarify an important conceptual distinction. Some FAT-style architectures are frequency adaptive inside attention; others are frequency adaptive before attention or after attention. FAiT changes the attention operator itself through a complementary high-pass branch, FreEformer changes the token domain and attention matrix, and FreqDGT performs sample-adaptive spectral weighting before the transformer stage. This suggests that “frequency-adaptive transformer” is better treated as a design family than as a single mechanism template (He et al., 31 May 2026).

6. Empirical behavior, misconceptions, and limitations

A common misconception is that a FAT must always perform an FFT. The literature does not support that restriction. SFormer’s explicit FAT bottleneck performs low/high decomposition with global average pooling and residual subtraction rather than with a Fourier transform. Conversely, PF-Trans, FEFormer, Dabformer, FAiT, and FreEformer all use explicit spectral transforms, but not all of them are formally named FAT (Tian et al., 26 Aug 2025).

A second misconception is that frequency adaptivity must mean frequency-conditioned self-attention weights. Some papers do adopt that spirit, but several influential designs are adaptive through branch specialization and gated fusion instead. PF-Trans is explicit that its frequency mechanism is a parallel FFT filtering branch rather than an adaptive frequency-conditioned attention operator. CGFformer achieves frequency adaptivity through cluster-guided filter generation before attention. FEFormer distributes frequency adaptation across attention, MLP, fusion, and stem bridging rather than encoding it in one attention formula alone (Gui et al., 9 Jun 2026).

A third misconception is that “FAT” is a settled acronym. It is not. The acronym can refer to Frequency Adaptive Transformer in SFormer, Frequency-Aware Transformer in learned image compression, or Fully Adaptive Transformer in a spatial lightweight vision backbone that is not frequency-based. Taken together, these works indicate that readers must disambiguate the term by paper, domain, and mechanism rather than by acronym alone (Fan et al., 2023).

Empirically, the family is associated with strong task-specific results. PF-Trans reports a PSNR of $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$6 dB, SSIM $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$7, and SAM $$\mathbf{Z}_{\text{low}} = \mathcal{A}\mathbf{V}, \qquad \mathbf{Z}_{\text{high}} = (\mathbf{I}-\mathcal{A})\mathbf{V},$$8 on the GF-5 Shanghai dataset, with an almost 9 dB improvement over MST++ emphasized for the urban scene. FEFormer reports 90.11 mean DSC on AMOS and 95.02 mean DSC on FLARE while remaining substantially smaller than several competing 3D transformer baselines. FAiT reports best or second-best results in 27 of 32 MSE cases and 31 of 32 MAE cases on long-term forecasting benchmarks, and state-of-the-art MAE on all eight long-term datasets. FreEformer reports consistent improvements across eighteen forecasting benchmarks and also improves several external transformer forecasters when its enhanced attention replaces vanilla attention (Yang et al., 12 May 2026).

Limitations are equally consistent across the literature. Many architectures are highly domain-specific: PF-Trans depends on a known BFA mask and forward sensing model; CGFformer is tailored to pansharpening; FEFormer is built for 3D medical segmentation; FreqDGT uses hand-defined canonical EEG bands; SFormer’s FAT is specialized to underwater enhancement. Implementation completeness is often limited. SFormer does not specify the exact CA, SA, or CrossAtt formulas, the number of heads, or the transformer depth inside FAT; PF-Trans leaves many transformer internals undocumented; HistoFusionNet does not specify the exact Fourier mask construction or fusion operator; CGFformer omits a full complexity analysis; and FEFormer does not spell out the exact low/mid/high spectral partition in FDSA (Tian et al., 26 Aug 2025).

The most accurate general characterization is therefore not that FAT is a single architecture, but that it is a family of transformer designs that use frequency structure as an explicit organizing principle for attention, feed-forward transformation, branch specialization, or feature fusion. In the narrowest explicit sense, FAT is SFormer’s bottleneck module. In the broader technical sense used across recent arXiv work, it names a design philosophy: structured degradations, fine details, and multi-scale signals are often handled more effectively when the transformer can decompose, reweight, or fuse information according to frequency content rather than relying on spatial-domain attention alone (He et al., 31 May 2026).