Bilateral Spectral Attention

• Bilateral Spectral Attention is a family of mechanisms that combine bilateral filtering with spectral processing for adaptive, edge-aware information propagation.
• It spans variants such as FFT-based global mixing, graph-spectral filtering, and dual spatial-feature attention used in vision and hyperspectral imaging.
• This approach enhances model interpretability and control in applications like low-dose CT denoising, language modeling, and shape correspondence.

Searching arXiv for the cited papers and related terminology to ground the article in current literature. arXiv search query: "bilateral spectral attention FourierQK Filter2Noise bilateral filter graph spectral interpretation BOAT" Bilateral spectral attention denotes a family of mechanisms in which information propagation is modulated by a bilateral notion of affinity together with an explicitly or implicitly spectral representation. In the strict formulation introduced by FourierQK, it is transformer attention in which query and key sequences are globally mixed in the frequency domain by a symmetric FFT-based kernel before score computation, making the kernel bilateral and structurally non-causal (Zeris, 8 Jul 2026). In a broader graph-spectral interpretation, adaptive bilateral filters define data-dependent graph operators whose local spectral transfer functions are selected by learned parameters, as in the Attention-Guided Bilateral Filter used in Filter2Noise (Sun et al., 18 Apr 2025) and in the graph-spectral interpretation of the bilateral filter (Gadde et al., 2013). Related architectures also realize the idea through joint spatial and feature-space locality, through factorized spatial-and-channel spectral attention, or through descriptor-guided manifold operators, although in several of those cases the spectral reading is interpretive rather than the paper’s original terminology (Yu et al., 2022, Feng et al., 2023, Pai et al., 2019).

1. Terminological scope and conceptual variants

The cited literature does not present a single canonical definition of bilateral spectral attention. Instead, the phrase spans several mathematically related constructions.

In FFT-based sequence models, “bilateral” refers to an even impulse response, $\kappa(-\tau)=\kappa(\tau)$, so each position receives contributions from both past and future tokens after spectral preprocessing. “Spectral” refers to a DFT along the sequence dimension, and attention remains a standard dot-product mechanism applied to spectrally mixed $Q/K$ projections (Zeris, 8 Jul 2026).

In bilateral filtering and graph signal processing, “bilateral” refers to weights that combine spatial proximity and value similarity, while “spectral” refers to the eigenstructure of the graph Laplacian induced by those bilateral weights. The classical bilateral filter becomes a graph spectral filter with response $h_{\text{BF}}(\lambda)=1-\lambda$, and more general $h(L)$ designs can be implemented by iterated bilateral operations (Gadde et al., 2013).

In patch- or feature-based vision architectures, “bilateral” can denote two domains of locality or weighting. BOAT combines image-space local attention with feature-space local attention, where patches are clustered by their features and self-attention is computed within each cluster (Yu et al., 2022). In unsupervised spectral demosaicing, the lightweight spectral attention tensor is factorized into spectral attention matrices in the spatial dimension and a spectral attention vector in the channel dimension, yielding a dual spatial–spectral mechanism (Feng et al., 2023).

A concise comparison is useful.

Regime	“Bilateral” meaning	“Spectral” meaning
FourierQK	Symmetric support over past and future tokens	FFT over sequence positions
Bilateral filtering / F2N	Spatial and range kernels	Graph Laplacian spectrum of the bilateral graph
BOAT	Image-space locality plus feature-space locality	Feature-space grouping can be read as a learned representation-domain analogue
Lightweight spectral demosaicing	Spatial attention matrices plus channel attention vector	Spectral/channel dimension of hyperspectral features

This diversity matters because identical terminology can mask distinct operators. A recurring misconception is that “spectral” necessarily means Fourier analysis; the cited works show Fourier, graph-spectral, representation-space, and channel-spectral usages (Zeris, 8 Jul 2026, Gadde et al., 2013, Feng et al., 2023).

2. Mathematical foundations

The graph-spectral lineage starts from the classical bilateral filter. For an input image $x_{\text{in}}$, bilateral filtering can be written as

$x_{\text{out}} = D^{-1} W x_{\text{in}},$

where $W=[w_{ij}]$ contains bilateral weights depending on both spatial and photometric distances, and $D_{jj}=\sum_i w_{ij}$ is the degree matrix (Gadde et al., 2013). With the normalized Laplacian

$L = I - D^{-1/2} W D^{-1/2},$

the normalized bilateral filter becomes

$$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$

so the spectral response is $Q/K$0 (Gadde et al., 2013). Iterating the filter $Q/K$1 times yields $Q/K$2, which strengthens low-pass behavior (Gadde et al., 2013).

Filter2Noise preserves the explicit bilateral form but replaces global constants $Q/K$3 with spatially varying, stage-dependent maps $Q/K$4 predicted patch-wise by attention (Sun et al., 18 Apr 2025). Its spatial and range kernels are

$Q/K$5

$Q/K$6

with adaptive support size

$Q/K$7

Because $Q/K$8 determine the smoothing bandwidths, the learned parameter maps can be interpreted as selecting local spectral responses on bilateral graphs (Sun et al., 18 Apr 2025, Gadde et al., 2013).

The FFT-based lineage uses a different operator class. FourierQK first projects tokens to $Q/K$9 and $h_{\text{BF}}(\lambda)=1-\lambda$0, applies a DFT along the sequence dimension,

$h_{\text{BF}}(\lambda)=1-\lambda$1

multiplies by a spectral filter $h_{\text{BF}}(\lambda)=1-\lambda$2, reconstructs filtered sequences $h_{\text{BF}}(\lambda)=1-\lambda$3, and then computes

$h_{\text{BF}}(\lambda)=1-\lambda$4

With real-valued FFT symmetry, the induced impulse response is even, $h_{\text{BF}}(\lambda)=1-\lambda$5, so the preprocessing mixes past and future tokens before standard masked attention is applied (Zeris, 8 Jul 2026).

These two lineages are mathematically different but structurally comparable. Both define an affinity operator first and then let a learned module modulate which modes, frequencies, or neighborhoods are emphasized. This suggests a unifying abstraction in which bilateral spectral attention is less a single layer type than a design pattern: bilateral affinities plus content-conditioned spectral shaping.

3. Bilateral filtering as local spectral control

The graph-spectral interpretation of the bilateral filter provides the most direct theoretical basis for bilateral spectral attention. In this view, pixels are nodes of a weighted graph, edge weights are bilateral coefficients, the image is a graph signal, and filtering is a spectral operation on the eigenmodes of the graph Laplacian (Gadde et al., 2013). The paper “Bilateral Filter: Graph Spectral Interpretation and Extensions” shows that arbitrary spectral responses $h_{\text{BF}}(\lambda)=1-\lambda$6 can be designed on this graph and implemented without expensive diagonalization by polynomial approximation and cascaded bilateral operations (Gadde et al., 2013).

Filter2Noise operationalizes this perspective in low-dose CT denoising. Its Attention-Guided Bilateral Filter replaces a black-box CNN denoiser with an explicit, parameterized bilateral filter whose parameters are predicted by a dual-attention module (Sun et al., 18 Apr 2025). The image is partitioned into non-overlapping $h_{\text{BF}}(\lambda)=1-\lambda$7 patches; Feature Attention computes contextualized patch features using scaled dot-product attention, and Sigma Attention maps those features to $h_{\text{BF}}(\lambda)=1-\lambda$8 via attention, projection, and Softplus positivity constraints (Sun et al., 18 Apr 2025). The crucial point is that attention does not directly weight pixels. It predicts parameter maps that govern the subsequent explicit bilateral operator.

The paper itself does not use the term “spectral” or provide an explicit frequency-domain analysis, but it explicitly frames the AGBF as interpretable through the known spectral properties of bilateral filters (Sun et al., 18 Apr 2025). Around each patch, one can define a local bilateral graph with Laplacian $h_{\text{BF}}(\lambda)=1-\lambda$9; the predicted sigmas determine a local transfer function $h(L)$0. Larger $h(L)$1 implies stronger low-pass filtering in that region, whereas smaller $h(L)$2 retains more high-frequency content such as edges and fine detail (Sun et al., 18 Apr 2025). Anisotropy between $h(L)$3 and $h(L)$4 yields directional spectral behavior.

This formulation has two distinctive consequences. First, the denoising mechanism is inspectable: the learned fields $h(L)$5 can be visualized directly (Sun et al., 18 Apr 2025). Second, it is editable after training: a user can multiply $h(L)$6 within a region of interest to strengthen or weaken smoothing there, or enforce upper bounds on sigmas in anatomically sensitive regions (Sun et al., 18 Apr 2025). A plausible implication is that bilateral spectral attention is especially attractive when interpretability, region-specific control, and explicit edge-preservation constraints are required.

Filter2Noise embeds this operator in a self-supervised single-image framework. It introduces Euclidean Local Shuffle on $h(L)$7 blocks to break deterministic alignment of spatially correlated CT noise, and optimizes a three-part multi-scale reconstruction loss together with a Difference-of-Gaussians regularizer that penalizes excessive suppression of edge-like band-pass components (Sun et al., 18 Apr 2025). On Mayo-2016 B30, the two-stage model F2N-S2 achieves PSNR $h(L)$8 dB and SSIM $h(L)$9, compared with ZS-N2N at PSNR $x_{\text{in}}$0 dB and SSIM $x_{\text{in}}$1; on Mayo-2016 D45 it achieves $x_{\text{in}}$2 dB and $x_{\text{in}}$3, compared with $x_{\text{in}}$4 dB and $x_{\text{in}}$5; on Mayo-2020 it achieves $x_{\text{in}}$6 dB and $x_{\text{in}}$7, compared with $x_{\text{in}}$8 dB and $x_{\text{in}}$9 (Sun et al., 18 Apr 2025). The same model uses only $x_{\text{out}} = D^{-1} W x_{\text{in}},$0k parameters, versus approximately $x_{\text{out}} = D^{-1} W x_{\text{in}},$1M for DIP and Noise2Void, $x_{\text{out}} = D^{-1} W x_{\text{in}},$2M for NB2NB, and $x_{\text{out}} = D^{-1} W x_{\text{in}},$3k for ZS-N2N (Sun et al., 18 Apr 2025).

4. FFT-based bilateral spectral attention in transformers

FourierQK provides the most explicit use of the phrase. Standard attention computes

$x_{\text{out}} = D^{-1} W x_{\text{in}},$4

with $x_{\text{out}} = D^{-1} W x_{\text{in}},$5 and $x_{\text{out}} = D^{-1} W x_{\text{in}},$6. FourierQK instead preprocesses $x_{\text{out}} = D^{-1} W x_{\text{in}},$7 in the frequency domain and then computes ordinary dot-product scores from the filtered projections (Zeris, 8 Jul 2026). The generic form is

$x_{\text{out}} = D^{-1} W x_{\text{in}},$8

followed by $x_{\text{out}} = D^{-1} W x_{\text{in}},$9 (Zeris, 8 Jul 2026).

The defining property is the symmetry of the FFT-based kernel. Because the impulse response is even, $W=[w_{ij}]$0, convolution with $W=[w_{ij}]$1 couples each token to both earlier and later tokens. The paper therefore calls the mechanism bilateral and structurally non-causal, and argues that such non-causality is intrinsic to the symmetry of the kernel rather than a boundary-handling artifact (Zeris, 8 Jul 2026). This yields a sharp architectural boundary between bilateral spectral attention and genuinely causal spectral attention.

FourierQK uses a Gaussian frequency selector centered at a learned dominant frequency,

$W=[w_{ij}]$2

and studies fixed random filters, a single learned frequency per head, and four learned frequencies per head (Zeris, 8 Jul 2026). On TinyShakespeare, BASE-DOT obtains validation loss $W=[w_{ij}]$3. Random-QK reaches $W=[w_{ij}]$4 with $W=[w_{ij}]$5; Fourier-QK reaches $W=[w_{ij}]$6 with $W=[w_{ij}]$7; Fourier-QK-Init4 reaches $W=[w_{ij}]$8 with $W=[w_{ij}]$9; and MultiFourier-QK reaches $D_{jj}=\sum_i w_{ij}$0 with $D_{jj}=\sum_i w_{ij}$1, corresponding to an approximately $D_{jj}=\sum_i w_{ij}$2 reduction in validation loss over BASE-DOT (Zeris, 8 Jul 2026). The four learned frequencies converge to a near-geometric multi-scale ordering with periods approximately $D_{jj}=\sum_i w_{ij}$3 tokens per cycle, interpreted as paragraph, sub-paragraph, phrase, and word-group scales (Zeris, 8 Jul 2026).

The paper also clarifies what the gains are not. Random orthogonal projection of $D_{jj}=\sum_i w_{ij}$4 gives validation loss approximately $D_{jj}=\sum_i w_{ij}$5, and random non-orthogonal projection gives approximately $D_{jj}=\sum_i w_{ij}$6, indicating no measurable improvement from generic linear remapping (Zeris, 8 Jul 2026). The authors therefore attribute the benefit specifically to spectral preprocessing and global frequency-domain mixing rather than metric distortion. The paper further distinguishes this architecture from FNet by Lee-Thorp et al., which replaces attention with Fourier mixing of token embeddings; FourierQK preserves learned $D_{jj}=\sum_i w_{ij}$7 projections, the full $D_{jj}=\sum_i w_{ij}$8 score matrix, causal masking, softmax, and value aggregation, and modifies only $D_{jj}=\sum_i w_{ij}$9 before scoring (Zeris, 8 Jul 2026).

A central controversy is leakage. Since FFT preprocessing is bilateral and periodic, spectrally mixed models may exploit future information. FourierQK addresses this with a shuffled-validation diagnostic: token order is randomly permuted once in the validation set, and the shuffled gap is defined as $L = I - D^{-1/2} W D^{-1/2},$0 (Zeris, 8 Jul 2026). BASE-DOT has gap $L = I - D^{-1/2} W D^{-1/2},$1, Random-QK $L = I - D^{-1/2} W D^{-1/2},$2, Fourier-QK $L = I - D^{-1/2} W D^{-1/2},$3, and MultiFourier-QK $L = I - D^{-1/2} W D^{-1/2},$4, all still large and therefore taken as evidence that the models depend on genuine temporal structure rather than only on trivial future leakage (Zeris, 8 Jul 2026). By contrast, when leakage is deliberately maximized in Hilbert-OrthoQK, the gap collapses toward zero while validation loss approaches zero, which the paper treats as clear evidence of non-causal exploitation (Zeris, 8 Jul 2026).

5. Architectural analogues in vision, hyperspectral imaging, and manifolds

Several architectures instantiate closely related ideas without using identical terminology. BOAT, the “Bilateral lOcal Attention vision Transformer,” combines image-space local attention with feature-space local attention (Yu et al., 2022). Image-space local attention uses Swin-style or CSWin-style local windows, while feature-space local attention first applies balanced hierarchical clustering to token features and then runs self-attention within each cluster (Yu et al., 2022). The balanced binary clustering step computes a cosine-similarity ratio

$L = I - D^{-1/2} W D^{-1/2},$5

sorts tokens by $L = I - D^{-1/2} W D^{-1/2},$6, and splits them into equal-size clusters, optionally with overlap at the final level (Yu et al., 2022). This is not an explicit Fourier method, but it realizes a bilateral design in two domains: spatial locality and learned feature-space locality.

Empirically, the feature-space branch contributes nontrivially. On ImageNet-1K, BOAT-Swin-T improves Swin-T from $L = I - D^{-1/2} W D^{-1/2},$7 to $L = I - D^{-1/2} W D^{-1/2},$8, and BOAT-CSWin-T improves CSWin-T from $L = I - D^{-1/2} W D^{-1/2},$9 to $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$0 (Yu et al., 2022). Replacing feature-space local attention with another image-space attention block yields lower accuracy, and overlapping clusters perform slightly better than non-overlapping ones (Yu et al., 2022). This suggests that bilaterality across domains, rather than simply deeper spatial attention, accounts for the gains.

In hyperspectral demosaicing, the paper “Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks” factorizes a full 3D attention tensor into spectral attention matrices in space and a spectral attention vector in channel (Feng et al., 2023). For a feature map $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$1, the refined feature map is

$$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$2

where $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$3 is produced by a shared spatial attention subnetwork applied separately to each channel, and $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$4 is an SE-like channel attention vector (Feng et al., 2023). The effective attention tensor is therefore factorized as $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$5. This is a concrete bilateral spatial–spectral attention mechanism over hyperspectral features.

The factorization is motivated by both compute and training stability. Under the unsupervised framework, the base model without spectral attention has PSNR $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$6 dB, SSIM $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$7, SAM $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$8, ERGAS $$\hat{x}_{\text{out}} = U (I-\Lambda) U^T \hat{x}_{\text{in}},$$9, and $Q/K$00M parameters; the heavyweight spectral attention variant degrades to PSNR $Q/K$01 dB, SSIM $Q/K$02, SAM $Q/K$03, ERGAS $Q/K$04, and $Q/K$05M parameters; the lightweight factorized variant improves to PSNR $Q/K$06 dB, SSIM $Q/K$07, SAM $Q/K$08, ERGAS $Q/K$09, and $Q/K$10M parameters (Feng et al., 2023). The paper states that the lightweight design reduces about $Q/K$11 of the parameters of the heavyweight spectral attention component under its typical settings (Feng et al., 2023).

On manifolds, “Bilateral Operators for Functional Maps” constructs descriptor-guided pairwise kernels

$Q/K$12

and combines them additively with the heat kernel to obtain a bilateral operator

$Q/K$13

In spectral form, this operator becomes

$Q/K$14

Its low-frequency eigenfunctions depict regions of similar descriptor values at various scales of frequency, and correspondence is enforced by the commutativity constraint $Q/K$15 (Pai et al., 2019). This is an instance of bilateral spectral attention on non-Euclidean domains: geometry-conditioned plus descriptor-conditioned smoothing in a spectral basis.

By contrast, some bilateral-attention papers are not spectral in their original formulation. The inpainting model with a Bilateral Attention layer computes a value branch over a $Q/K$16 neighborhood using dot-product similarity and a distance branch over a $Q/K$17 neighborhood using a Gaussian spatial kernel, then fuses them with a $Q/K$18 convolution (Liu et al., 2019). BAFNet uses a dependency path with large kernel attention, a remote-local path combining multi-scale local attention and efficient remote attention, and a feature aggregation module; the paper interprets these components in a low-frequency versus high-frequency manner, but this remains a spectral reading rather than an explicit spectral-domain mechanism (Wang et al., 2024). These works are best understood as templates from which explicit bilateral spectral attention modules could be derived.

6. Empirical patterns, misconceptions, and limitations

Across the cited works, bilateral spectral attention repeatedly appears when there is a need to preserve structure while adapting smoothing or coupling to content. In low-dose CT, the learned sigmas in Filter2Noise make the denoising behavior transparent and user-adjustable, while retaining explicit control over edge preservation (Sun et al., 18 Apr 2025). In character-level language modelling, FFT-based spectral preprocessing of $Q/K$19 reveals strong gains from paragraph-scale and multi-scale phase-coherent structure that ordinary dot-product attention does not exploit as effectively (Zeris, 8 Jul 2026). In hyperspectral demosaicing, factorized spatial–spectral attention improves spectral fidelity and spatial distortion suppression under unsupervised training (Feng et al., 2023). In shape correspondence, bilateral operators encode pairwise descriptor structure that can be preserved under functional maps (Pai et al., 2019).

Several misconceptions recur. First, bilateral spectral attention is not synonymous with Fourier attention. The graph-spectral bilateral filter literature defines spectrality through Laplacian eigenmodes rather than DFT bins, and Filter2Noise explicitly operates on spatial patches in the intensity domain with no Fourier transform (Gadde et al., 2013, Sun et al., 18 Apr 2025). Second, not every bilateral attention module is spectral. The inpainting BA-layer and BAFNet are spatial in implementation; their spectral interpretation is an extension proposed in the analyses summarized here rather than the original claim of those papers (Liu et al., 2019, Wang et al., 2024). Third, generic remapping of query and key vectors is not equivalent to spectral preprocessing: FourierQK shows that random orthogonal and non-orthogonal projections produce no measurable improvement, whereas FFT-based preprocessing does (Zeris, 8 Jul 2026).

The main limitations also differ by regime. In the FFT-based transformer setting, bilateral kernels are structurally non-causal, so strict causal language modelling requires different constructions; the paper explicitly places causal spectral attention in separate time-domain wavelet-based architectures such as MorletQK at word-level tokenization (Zeris, 8 Jul 2026). In graph-spectral bilateral filtering, the clean spectral interpretation assumes a fixed graph during one filtering block; if the graph is updated after each step, the strict linear spectral analysis no longer applies in the same way (Gadde et al., 2013). In factorized spatial–spectral attention, reducing parameters improves robustness in unsupervised settings but also constrains expressivity, which is why the lightweight spectral attention paper emphasizes the trade-off between expressiveness, robustness, and efficiency (Feng et al., 2023).

A plausible synthesis is that bilateral spectral attention is best viewed as a structured alternative to unconstrained attention. Instead of learning arbitrary dense affinities, it starts from an operator with explicit geometry—spatial-plus-range kernels, bilateral graphs, symmetric FFT filters, or dual spatial/feature domains—and then lets learning select the relevant spectral transfer function, frequency band, or factorized modulation. The recurrent value of this template is not that it yields a single universal layer, but that it provides a mathematically grounded route to adaptive, edge-aware, and often interpretable attention across sequences, images, hyperspectral cubes, and manifolds (Zeris, 8 Jul 2026, Sun et al., 18 Apr 2025, Gadde et al., 2013, Pai et al., 2019).