Learnable Frequency Filters in ML

• Learnable frequency filters are adaptive spectral operators that replace fixed passbands with data-optimized masks, enhancing global token mixing.
• They employ static and input-adaptive designs over various spectral bases like FFT, DCT, and graph Fourier methods for different application domains.
• By merging classical DSP with neural methods, these filters offer parameter efficiency, interpretability, and improved performance in vision, speech, and forecasting tasks.

Searching arXiv for recent and foundational papers on learnable frequency filters across vision, audio, time series, and graph learning. arXiv search: "learnable frequency filters" Learnable frequency filters are parameterized spectral operators whose frequency responses are optimized from data rather than fixed a priori. Across modern machine learning, the term encompasses several distinct but related constructions: input-adaptive Fourier masks for global token mixing in vision (Huang et al., 2023); task-adaptive spectral filter banks for speech and speaker verification (Li et al., 2022); DCT-domain masks for presentation attack detection (Fang et al., 2021); dynamic frequency-component weighting for night-time scene parsing (Xie et al., 2022); feature-space Fourier gating for domain generalization (Lin et al., 2022); spectral filtering as a primary temporal modeling primitive in long-horizon forecasting (Yi et al., 2024); and node-variant graph filters that explicitly create graph-frequency content in graph neural networks (Gama et al., 2021). What unifies these formulations is the replacement of hand-designed passbands or implicit spatial filtering by learnable, differentiable mechanisms that control how spectral components are passed, attenuated, or redistributed.

1. Concept and scope

In the broadest sense, a learnable frequency filter is a mapping that modulates a representation after transformation into a frequency basis, or equivalently parameterizes a filter family whose spectral behavior is directly controlled by trainable parameters. The simplest case is a static learned spectral mask shared across inputs; more expressive variants are input-conditioned, channel-specific, and layer-specific. A central distinction in the literature is therefore between static spectral shaping and adaptive spectral filtering.

Several papers make this distinction explicit. In Adaptive Frequency Filtering (AFF), the spectral mask is not a fixed parameter tensor but an input-conditioned function of the current feature map, yielding semantic adaptivity (Huang et al., 2023). In FilterNet for time-series forecasting, the plain shaping filter is static after training, whereas the contextual shaping filter is generated from the current input spectrum and therefore varies across instances (Yi et al., 2024). In Deep Frequency Filtering for domain generalization, the mask is predicted per sample from latent Fourier features and acts as an instance-adaptive soft filter (Lin et al., 2022). By contrast, frequency learning for image classification in FreqNet learns static multiplicative masks over Fourier magnitudes, optimized from labeled data but not recomputed per input (Stuchi et al., 2020).

Another important axis is the choice of spectral basis. The literature spans 2D FFT for images and latent vision tokens (Huang et al., 2023, Lin et al., 2022), DCT for frequency-aware vision modules (Fang et al., 2021, Xie et al., 2022), STFT-based filter banks for speech (Li et al., 2022), sinc-parameterized bandpass filters in the time domain (Wang et al., 28 Aug 2025, Pariente et al., 2019), analytic filter banks for speech separation (Pariente et al., 2019), and graph Fourier bases derived from graph shift eigendecompositions (Gama et al., 2021). This suggests that “frequency” is not tied to a single implementation, but to a spectral decomposition appropriate to the signal domain.

2. Spectral parameterization and mathematical equivalences

A recurring theoretical theme is that spectral multiplication implements a convolution-like operator in the original domain. AFF states this most directly. For an input feature map $\mathbf{X}\in \mathbb{R}^{H\times W\times C}$, AFF computes

$$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$

Using the convolution theorem, the paper derives

$\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$

so the learned spectral mask corresponds to a dynamic depthwise convolution kernel as large as the spatial resolution of the feature map (Huang et al., 2023). This is the main formal justification for viewing learnable frequency filters as efficient global token mixers rather than merely heuristic Fourier reweighting.

An analogous argument appears in FilterNet. For a time series with Fourier transform $\mathcal{X}[k]$ and filter $\mathcal{H}[k]$,

$\mathcal{Y}[k] = \mathcal{X}[k]\mathcal{H}[k]$

is equivalent in time to circular convolution,

$$\mathcal{Y}[k] = \mathcal{H}[k]\mathcal{X}[k] \leftrightarrow y[n] = h[n] \circledast x[n].$$

FilterNet therefore interprets temporal forecasting operators as learnable spectral shaping functions, with a static filter approximating linear temporal mappings and a contextual filter approximating attention-like adaptive mixing (Yi et al., 2024).

In graph learning, the same principle becomes subtler because graph filters need not be diagonal in the graph Fourier basis. For linear shift-invariant graph filters, the graph-Fourier action is diagonal and cannot create new frequency content. Node-variant graph filters instead induce a non-diagonal transfer matrix

$\hat{y}=V^T\big(V\circ(H\Psi^T)\big)\hat{x},$

which allows energy to move across eigenspaces and thus explicitly creates graph-frequency content (Gama et al., 2021). This is a significant generalization: learnable frequency filters need not merely rescale spectral coefficients; they may redistribute them.

A different but related parameterization appears in structured CNN filters based on Gaussian fractional derivatives. There the learnable object is not a discrete mask but the derivative order $\nu$ in a Gaussian derivative basis, interpreted as a direct controller of the filter’s frequency emphasis. The practical filter is

$F(x;\sigma) \approx \alpha G^\nu(x;\sigma),$

or in 2D,

$$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$0

This makes frequency behavior explicit through a continuous spectral parameter rather than through unconstrained kernel weights (Saldanha et al., 2021).

3. Major design patterns

The literature converges on a few recurring implementation patterns.

Static spectral masks

Static filters are trainable parameters optimized end-to-end and fixed at inference. FreqNet learns multiplicative masks $$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$1 over Fourier magnitudes for each image block, with nonnegativity enforced by truncating negative weights to zero (Stuchi et al., 2020). FilterNet’s plain shaping filter learns a complex spectral kernel $$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$2 and applies

$$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$3

where the channel-shared universal filter often performs better than channel-specific alternatives (Yi et al., 2024).

A related class uses parametric analytic filter families rather than arbitrary masks. Learnable Frequency Filters for speaker verification defines triangle-type and bell-type filters over STFT bins:

$$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$4

and

$$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$5

with learnable center frequencies $$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$6 and bandwidths $$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$7 (Li et al., 2022). The output reduces to standard Mel filter banks if these parameters are fixed to Mel-scale values. This is a representative example of task-adaptive generalization of classical DSP front ends.

Input-adaptive spectral masks

Adaptive filters recompute their weights from the input. AFF generates the mask by two group linear layers with ReLU,

$$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$8

and applies it channel-wise in Fourier space (Huang et al., 2023). The channel-wise mask $$\mathbf{\hat{X} = \mathcal{F} ^{-1} [\mathcal{M}(\mathcal{F} (\mathbf{X})) \odot \mathcal{F} (\mathbf{X})]}.$$9 outperforms a channel-shared $\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$0 mask in ablation, with reported values 79.8 versus 79.3 at the same complexity (Huang et al., 2023).

Deep Frequency Filtering uses an instance-adaptive spatial mask over Fourier coordinates of latent feature maps:

$\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$1

This mask is generated per sample and shared across channels, functioning as a soft frequency gate (Lin et al., 2022). The same paper shows that instance-adaptive spatial masking is more effective than task-level or channel-attention alternatives.

Boosting Night-time Scene Parsing with Learnable Frequency uses a DCT-based Learnable Frequency Encoder. After extracting a multi-spectral vector $\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$2, the module predicts normalized component weights via

$\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$3

then reweights each component:

$\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$4

The softmax makes the module a competitive allocation mechanism across frequency components rather than independent unconstrained gains (Xie et al., 2022).

Structured analytic and sinc parameterizations

A large class of learnable frequency filters retains hand-designed spectral structure and learns a small number of physically meaningful parameters. In speech separation, parameterized analytic filter banks define the analysis filter as

$\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$5

with bandwidth $\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$6 and center frequency $\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$7 (Pariente et al., 2019). This moves between fixed STFT-like structure and fully learned front ends.

SincQDR-VAD uses a bandpass sinc extractor whose prototype is

$\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$8

truncated, windowed, and scaled by a learnable gain $\mathbf{\hat{X} = \mathcal{F}^{-1}[\mathcal{M}(\mathcal{F} (\mathbf{X}))] \ast \mathbf{X},$9:

$\mathcal{X}[k]$0

Features are log sub-band energies after time-domain filtering,

$\mathcal{X}[k]$1

with cutoff frequencies and gains trained jointly with the VAD model (Wang et al., 28 Aug 2025).

4. Domain-specific realizations

Vision backbones and token mixing

AFF casts learnable spectral masks as global token mixers that approximate large dynamic depthwise convolutions with FFT cost $\mathcal{X}[k]$2 rather than self-attention’s $\mathcal{X}[k]$3 (Huang et al., 2023). This makes learnable frequency filtering a systems-level replacement for attention, large kernels, or token MLPs in lightweight visual networks.

For domain generalization, DFF inserts learnable Fourier gating into latent feature maps across a network, with FFT over spatial dimensions only and inverse FFT back into feature space (Lin et al., 2022). The motivation is not efficiency but transferability: suppress non-transferable spectral components while preserving domain-generalizable ones. Visualizations in that work show a low-pass-biased tendency, but the masks remain sample-dependent rather than globally fixed (Lin et al., 2022).

For night-time scene parsing, FDLNet uses DCT-based component weighting to model day/night spectral discrepancies and fuses the resulting frequency descriptor with spatial features through a channel-affinity module. Its best setting uses $\mathcal{X}[k]$4 frequency components, indicating that moderately fine spectral partitioning is beneficial (Xie et al., 2022).

For image demoireing, MBCNN employs a learnable bandpass filter inside a DCT-inspired implicit transform path. The core operation is

$\mathcal{X}[k]$5

with nonnegative, all-ones-initialized passband coefficients $\mathcal{X}[k]$6 (Zheng et al., 2020). This design preserves transform structure while allowing selective suppression of moiré frequencies.

Time-series forecasting

FilterNet treats frequency filters themselves as the principal forecasting operator. Its two variants illustrate a clean taxonomy: a static universal or channel-specific complex kernel for linear-like temporal modeling, and a contextual filter

$\mathcal{X}[k]$7

that changes with the current input spectrum (Yi et al., 2024). This reframes forecasting as learnable spectral shaping rather than as attention or time-domain dense mixing.

The later “filter then attend” paradigm places a simple learnable spectral filter before transformer attention, but after embedding. The spectral block applies

$\mathcal{X}[k]$8

where $\mathcal{X}[k]$9 is a learned real-valued kernel transformed by rFFT (Dayag et al., 27 Aug 2025). This suggests that spectral preconditioning can compensate for the low-frequency bias of attention and reduce the embedding dimension required by the downstream transformer.

Speech, speaker verification, and VAD

In speaker verification, LFF learns center frequencies and bandwidths over STFT bins and consistently improves over fixed Mel filter banks, while analysis of learned parameters indicates that narrower bandwidths, rather than radically shifted centers, account for most of the gain (Li et al., 2022). This suggests that task-optimal filtering may preserve conventional frequency allocation while altering resolution.

In noisy VAD, sinc-parameterized filters plus ranking-aware optimization improve AUROC and $\mathcal{H}[k]$0 while remaining highly compact (Wang et al., 28 Aug 2025). The paper reports that removing the Sinc-extractor reduces AVA AUROC from 0.914 to 0.889 and noisy AVA AUROC from 0.815 to 0.784, indicating a substantial front-end contribution (Wang et al., 28 Aug 2025).

In speech enhancement, harmonic enhancement is formulated with a learnable comb filter module. The classical comb structure

$\mathcal{H}[k]$1

is embedded inside a DNN pipeline, where F0 selection and frequency-dependent mixing strengths are learned jointly (Le et al., 2023). The main method keeps the comb kernels fixed and learns when and where to apply them; an ablation makes the comb weights trainable and yields a small further improvement (Le et al., 2023). This illustrates a hybrid notion of learnable frequency filtering: structured harmonic priors with learned control.

Graph neural networks

Node-variant graph filters generalize spectral filtering beyond diagonal graph-Fourier multipliers. A node-variant filter is

$\mathcal{H}[k]$2

with node-dependent taps $\mathcal{H}[k]$3 (Gama et al., 2021). Because its graph-Fourier action is generally non-diagonal, it can create graph-frequency content and thereby separate the role of frequency creation from pointwise nonlinearity. This is conceptually important: learnable frequency filters may act as explicit, interpretable replacements for implicit spectral effects of activation functions.

5. Relationship to attention, convolution, and classical DSP

Learnable frequency filters often sit between classical DSP and modern neural architectures.

From the DSP side, many models preserve interpretable parameters such as center frequency, bandwidth, cutoff frequencies, or gains. Examples include sinc-based bandpass filters (Wang et al., 28 Aug 2025), Mel-generalized spectral projections (Li et al., 2022), and analytic filter banks with learned $\mathcal{H}[k]$4 and $\mathcal{H}[k]$5 (Pariente et al., 2019). These designs benefit from low parameter count and explicit inductive bias.

From the deep-learning side, spectral operators increasingly serve as replacements or approximations for global mixing layers. AFF presents adaptive Fourier masks as efficient substitutes for global dynamic convolutions and self-attention (Huang et al., 2023). FilterNet argues that plain shaping approximates linear mappings while contextual shaping approximates attention-style adaptive dependency learning (Yi et al., 2024). Pre-attention spectral filtering in transformer forecasters is similarly motivated as a way to reshape the representation that attention consumes (Dayag et al., 27 Aug 2025).

This convergence suggests a broader interpretation: learnable frequency filters are not merely preprocessing modules. They can function as first-class neural operators with a clear algebraic relation to global receptive fields.

6. Advantages, limitations, and recurring trade-offs

A consistent advantage is parameter efficiency. Structured parameterizations reduce degrees of freedom relative to unconstrained convolutions. In speaker verification, each filter is controlled by only $\mathcal{H}[k]$6 and $\mathcal{H}[k]$7 (Li et al., 2022). In sinc-based VAD, each bandpass filter uses two cutoffs and one gain (Wang et al., 28 Aug 2025). In structured Gaussian fractional filters, frequency behavior is controlled by a small set of parameters per kernel rather than a full pixelwise convolution kernel (Saldanha et al., 2021).

A second advantage is interpretability. Learned bandwidth narrowing in speaker verification has a direct task-level interpretation (Li et al., 2022). The cutoff ranges and gains in VAD can be visualized as a task-adapted spectral front end (Wang et al., 28 Aug 2025). Learned passbands in demoireing correspond to moiré-sensitive frequencies (Zheng et al., 2020). In graph learning, the transfer matrix reveals when new frequency content is created (Gama et al., 2021).

A third advantage is global context with efficient implementation. FFT-based methods gain global receptive fields at subquadratic or logarithmic transform cost (Huang et al., 2023, Yi et al., 2024). This is particularly attractive in mobile vision and long-horizon forecasting.

The main trade-offs concern expressivity, stability, and shift sensitivity. Simple real-valued masks are stable but less expressive than complex-valued filters (Lin et al., 2022). Static filters are cheap but may underfit varying spectral conditions; adaptive filters improve flexibility but add conditioning networks and may be more sensitive to distribution shift. This is visible in transformer forecasting with spectral prefilters, where performance can degrade under severe train/test spectral mismatch (Dayag et al., 27 Aug 2025). Structured filter families improve interpretability but can be limited by their basis assumptions, as noted for fixed DCT bases in demoireing (Zheng et al., 2020) and night-time parsing (Xie et al., 2022).

Another recurring limitation is that many methods manipulate magnitude or real/imaginary representations without a deeper treatment of phase. FreqNet discards phase entirely (Stuchi et al., 2020). DFF applies the same spatial Fourier mask to both real and imaginary parts at a coordinate (Lin et al., 2022). This suggests an open design space around phase-aware or complex-valued adaptive filtering.

A final trade-off is between fixed prior structure and fully learned flexibility. The strongest empirical designs are often hybrids. Analytic speech filter banks outperform their non-analytic counterparts, showing that structured inductive bias is beneficial (Pariente et al., 2019). Adaptive filter banks followed by time averaging can approximate mel-spectrograms while permitting limited learnability in center frequencies or averaging widths (Doerfler et al., 2017). This suggests that structured adaptivity, rather than unrestricted front-end learning, is often the effective regime.

7. Historical development and outlook

The modern literature on learnable frequency filters reflects a shift from using fixed frequency-domain representations as preprocessing toward making spectral operations trainable, integrated, and sometimes input-adaptive. Earlier work in audio asked whether fixed mel-spectrograms should be replaced or approximated by adaptive filter banks, concluding that structured front-end learning can outperform canonical features while remaining stable (Doerfler et al., 2017). Speech separation further broadened this space by contrasting fixed STFTs, free learned filters, and parameterized analytic filter banks (Pariente et al., 2019).

Later work expanded the idea into latent neural operators and backbone modules. Vision models now use adaptive Fourier masks for token mixing (Huang et al., 2023), latent-space spectral attention for robustness and domain generalization (Lin et al., 2022), and DCT-guided fusion for night-scene understanding (Xie et al., 2022). Time-series forecasting has moved from Transformer-centric modeling to architectures where frequency filtering is either the core temporal operator (Yi et al., 2024) or an explicit preconditioner for attention (Dayag et al., 27 Aug 2025). Graph learning has gone further by making frequency creation itself explicit and learnable (Gama et al., 2021).

Several plausible implications follow from this trajectory. One is that frequency-domain parameterizations increasingly serve as architectural primitives, not just front-end choices. Another is that the most effective designs often combine classical signal-processing constraints with adaptive neural control. A third is that frequency learning is becoming domain-specific: the “right” learnable filter depends strongly on whether the objective is robustness, efficiency, harmonic reconstruction, global token mixing, or discriminative band selection.

Future work is likely to focus on richer complex-valued parameterizations, stronger theory connecting learned spectral responses to generalization and stability, better handling of phase, and more explicit control over interpretable properties such as passband shape, steerability, or cross-frequency coupling. The existing literature already shows that learnable frequency filters are not a single technique but a family of methods for turning spectral structure into trainable inductive bias across signal modalities and model classes.