Wavelet-Enhanced Random Spectral Attention (WERSA)

• The paper introduces WERSA to achieve linear-time attention by combining multi-resolution Haar wavelet filtering with content-adaptive random spectral feature mapping.
• It replaces quadratic softmax attention by filtering and reconstructing scale-selected signals, preserving both local nuances and global context.
• Empirical evaluations on benchmarks like ArXiv and CIFAR-10 demonstrate improved accuracy and reduced computational cost versus standard and alternative methods.

Searching arXiv for WERSA and closely related wavelet/efficient attention papers to ground the article and provide contextual citations. {"^^^^^^^^8query8 "^^^^^^^^8\8 Random Spectral Attention^^^^^^^^8\8 OR WERSA8^^^^^^^^", "^^^^^^^^8max_results8 ^^^^8submittedDate8^^^^, "^^^^^^^^8sort_by8 "^^^^^^^^8submittedDate8 {"^^^^^^^^8query8 "^^^^^^^^8\8 Attention via Wavelet Neural Operators for Vision Transformers^^^^^^^^8\8 "^^^^^^^^8max_results8 ^^^^8submittedDate8^^^^, "^^^^^^^^8sort_by8 "relevance"} {"^^^^^^^^8query8 "wavelet attention linear transformer Waveformer Performer efficient attention", "^^^^^^^^8max_results8 ^^^^8\8query8^^^^, "^^^^^^^^8sort_by8 "relevance"} Wavelet-Enhanced Random Spectral Attention (WERSA) is a Transformer attention mechanism designed for long-sequence processing with linear PRESERVED_PLACEHOLDER_^^^^8query8^^^^ time complexity. It combines content-adaptive random spectral features with multi-resolution Haar wavelets and learnable scale parameters, with the stated aim of preserving linear efficiency while selectively attending to informative scales of data. In contrast to regular softmax attention, which forms an PRESERVED_PLACEHOLDER_^^^^8\8^^^^ interaction matrix and therefore incurs quadratic cost, WERSA replaces full pairwise attention with a pipeline that wavelet-filters queries and keys, reconstructs scale-selected signals, and then applies a random-feature kernel approximation to obtain a linear-time attention computation (&&&^^^^8query8^^^^&&&).

^^^^8\8^^^^. Problem setting and design rationale

The immediate problem addressed by WERSA is the computational bottleneck of standard self-attention on long contexts. In the conventional formulation,

PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8^^^^

and

PRESERVED_PLACEHOLDER_^^^^8max_results8^^^^

Because PRESERVED_PLACEHOLDER_^^^^8sort_by8^^^^ is PRESERVED_PLACEHOLDER_^^^^8submittedDate8^^^^, both compute and memory-related work scale quadratically with sequence length. The paper identifies this as the main reason conventional Transformers become impractical for long documents, long videos, and other high-context inputs (&&&^^^^8query8^^^^&&&).

WERSA is constructed by combining two efficiency ideas that are usually treated separately: random feature linearization of attention and multi-resolution wavelet filtering. The paper’s claim is not merely that this reduces cost, but that it preserves the ability to model both local and global dependencies. The wavelet component provides a multi-resolution prior, while the random spectral component provides the linear-time approximation. The paper explicitly distinguishes WERSA from methods that only optimize the implementation of exact quadratic attention, stating that compared with FlashAttention-^^^^8 OR WERSA8^^^^, WERSA changes the computational form of attention itself rather than serving mainly as a memory optimization of exact attention (&&&^^^^8query8^^^^&&&).

The paper also frames WERSA as selective rather than uniformly frequency-mixing. Low-frequency coefficients are used to represent broad semantic or global structure, while high-frequency detail coefficients capture local variations, abrupt transitions, edges, or compositional substructure. This motivates the use of learned scale weights and content-adaptive gates so that informative scales can be emphasized and noisy or irrelevant scales suppressed (&&&^^^^8query8^^^^&&&).

^^^^8 OR WERSA8^^^^. Mathematical construction

WERSA retains the ^^^^^^^^8query8 structure of a Transformer but alters the attention computation. The mechanism begins with learned projections,

PRESERVED_PLACEHOLDER_^^^^8query8^^^^

with the projected tensors split across PRESERVED_PLACEHOLDER_^^^^8\8^^^^ heads. Queries and keys are then passed through a wavelet transform:

$$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$

In the implementation and pseudocode, the paper uses the Haar transform for simplicity and efficiency, while noting that the framework is generic enough to accommodate alternatives such as Daubechies or symlets (&&&^^^^8query8^^^^&&&).

A defining element of WERSA is its content-adaptive scale filter. For each head, the ^^^^^^^^8query8^^^^^^^^ representation is first averaged,

$$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$

then transformed by a small neural map PRESERVED_PLACEHOLDER_^^^^8\8query8^^^^ followed by a sigmoid,

PRESERVED_PLACEHOLDER_^^^^8\8\8^^^^

The paper further introduces learnable scale-dependent weights,

PRESERVED_PLACEHOLDER_^^^^8\8 OR WERSA8^^^^

where PRESERVED_PLACEHOLDER_^^^^8\8max_results8^^^^ modulates the importance of the PRESERVED_PLACEHOLDER_^^^^8\8sort_by8^^^^-th wavelet scale. These coefficients are applied elementwise in wavelet space, followed by inverse reconstruction:

PRESERVED_PLACEHOLDER_^^^^8\8submittedDate8^^^^

After the wavelet stage, WERSA approximates the softmax kernel by a random feature map. The kernel approximation is written as

PRESERVED_PLACEHOLDER_^^^^8\8query8^^^^

with a ReLU-based map

PRESERVED_PLACEHOLDER_^^^^8\8\8^^^^

and, with a trainable bandwidth parameter PRESERVED_PLACEHOLDER_^^^^8\88^^^^,

PRESERVED_PLACEHOLDER_^^^^8\8^^^^9

where PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8query8^^^^ is a random matrix and PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8\8^^^^ is the number of random features. In the main theorem and experiments, the paper uses PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8 OR WERSA8^^^^ and PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8max_results8^^^^ (&&&^^^^8query8^^^^&&&).

The resulting approximation is

PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8sort_by8^^^^

where PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8submittedDate8^^^^ stabilizes the denominator. In the implementation pseudocode, the same idea appears as efficient tensor contractions:

PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8query8^^^^

PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8\8^^^^

followed by head merging and a final output projection (&&&^^^^8query8^^^^&&&).

^^^^8max_results8^^^^. Complexity, approximation theory, and linear-time claim

The paper provides a componentwise complexity analysis. The stated costs are:

• linear projection: PRESERVED_PLACEHOLDER_^^^^8 OR WERSA88^^^^
• wavelet decomposition/reconstruction: PRESERVED_PLACEHOLDER_^^^^8 OR WERSA8^^^^9
• adaptive filtering: PRESERVED_PLACEHOLDER_^^^^8max_results8query8^^^^ for filter generation and PRESERVED_PLACEHOLDER_^^^^8max_results8\8^^^^ for filter application
• random feature projection: PRESERVED_PLACEHOLDER_^^^^8max_results8 OR WERSA8^^^^
• linear attention computation: PRESERVED_PLACEHOLDER_^^^^8max_results8max_results8^^^^
• final projection: PRESERVED_PLACEHOLDER_^^^^8max_results8sort_by8^^^^

From these terms, the paper derives

PRESERVED_PLACEHOLDER_^^^^8max_results8submittedDate8^^^^

Since PRESERVED_PLACEHOLDER_^^^^8max_results8query8^^^^ and PRESERVED_PLACEHOLDER_^^^^8max_results8\8^^^^ are fixed with respect to sequence length, the sequence-length scaling is PRESERVED_PLACEHOLDER_^^^^8max_results88^^^^. The theorem refines this to

PRESERVED_PLACEHOLDER_^^^^8max_results8^^^^9

with PRESERVED_PLACEHOLDER_^^^^8sort_by8query8^^^^ wavelet levels. If PRESERVED_PLACEHOLDER_^^^^8sort_by8\8^^^^ is fixed, scaling remains linear; if PRESERVED_PLACEHOLDER_^^^^8sort_by8 OR WERSA8^^^^, the complexity becomes PRESERVED_PLACEHOLDER_^^^^8sort_by8max_results8^^^^. In the experiments, PRESERVED_PLACEHOLDER_^^^^8sort_by8sort_by8^^^^, which the paper presents as keeping the method in linear-time territory (&&&^^^^8query8^^^^&&&).

The theoretical approximation guarantee separates random-feature error from wavelet truncation error. The paper states that for any PRESERVED_PLACEHOLDER_^^^^8sort_by8submittedDate8^^^^ and PRESERVED_PLACEHOLDER_^^^^8sort_by8query8^^^^, if

PRESERVED_PLACEHOLDER_^^^^8sort_by8\8^^^^

then with probability at least PRESERVED_PLACEHOLDER_^^^^8sort_by88^^^^,

PRESERVED_PLACEHOLDER_^^^^8sort_by8^^^^9

Here the term PRESERVED_PLACEHOLDER_^^^^8submittedDate8query8^^^^ is attributed to the random-feature approximation, while PRESERVED_PLACEHOLDER_^^^^8submittedDate8\8^^^^ is the wavelet truncation error, with PRESERVED_PLACEHOLDER_^^^^8submittedDate8 OR WERSA8^^^^ depending on the smoothness of the signals. The proof sketch is described as using wavelet approximation theory and Lipschitz continuity of softmax (&&&^^^^8query8^^^^&&&).

A common misconception is that the wavelet stage alone is what makes WERSA linear. The paper explicitly argues otherwise: random features linearize the kernel, whereas wavelets do not by themselves replace the quadratic softmax interaction. The wavelet component supplies explicit scale modeling; the random-feature component supplies the linear-time attention computation (&&&^^^^8query8^^^^&&&).

^^^^8sort_by8^^^^. Empirical evaluation

The experiments were run on a single machine with an AMD Threadripper ^^^^8max_results8max_results8\8query8^^^^X CPU, ^^^^8\8 OR WERSA88^^^^ GB RAM, and one Nvidia A^^^^8query8query8query8query8^^^^ GPU. The benchmark suite includes CIFAR-^^^^8\8query8^^^^, CIFAR-^^^^8\8query8query8^^^^, IMDB, ListOps, ArXiv, and ArXiv-^^^^8\8 OR WERSA88^^^^k. Each model was trained five times, and the paper reports mean and standard deviation for accuracy, precision, recall, F^^^^8\8^^^^, AUC, training time, and FLOPS (&&&^^^^8query8^^^^&&&).

The paper emphasizes broad comparisons against Multiheaded Attention, Flash-Attention-^^^^8 OR WERSA8^^^^, FNet, Linformer, Performer, and Waveformer. Its most detailed efficiency claims concern ArXiv classification and the extreme-length ArXiv-^^^^8\8 OR WERSA88^^^^k setting.

Benchmark	WERSA result	Comparison noted in paper
ArXiv	Accuracy PRESERVED_PLACEHOLDER_^^^^8submittedDate8max_results8^^^^, AUC PRESERVED_PLACEHOLDER_^^^^8submittedDate8sort_by8^^^^, training time PRESERVED_PLACEHOLDER_^^^^8submittedDate8submittedDate8^^^^ s, FLOPS PRESERVED_PLACEHOLDER_^^^^8submittedDate8query8^^^^ G	Standard Transformer: PRESERVED_PLACEHOLDER_^^^^8submittedDate8\8^^^^, PRESERVED_PLACEHOLDER_^^^^8submittedDate88^^^^, PRESERVED_PLACEHOLDER_^^^^8submittedDate8^^^^9 s, PRESERVED_PLACEHOLDER_^^^^8query8query8^^^^ G
ArXiv-^^^^8\8 OR WERSA88^^^^k	Accuracy PRESERVED_PLACEHOLDER_^^^^8query8\8^^^^, AUC PRESERVED_PLACEHOLDER_^^^^8query8 OR WERSA8^^^^, training time PRESERVED_PLACEHOLDER_^^^^8query8max_results8^^^^ s, FLOPS PRESERVED_PLACEHOLDER_^^^^8query8sort_by8^^^^ G	Transformer and FlashAttention-^^^^8 OR WERSA8^^^^ run out of memory
CIFAR-^^^^8\8query8^^^^	Accuracy PRESERVED_PLACEHOLDER_^^^^8query8submittedDate8^^^^, AUC PRESERVED_PLACEHOLDER_^^^^8query8query8^^^^	Best accuracy and best AUC among compared methods
IMDB	Accuracy PRESERVED_PLACEHOLDER_^^^^8query8\8^^^^, AUC PRESERVED_PLACEHOLDER_^^^^8query88^^^^	Slightly ahead of standard Transformer with fewer FLOPS
ListOps	Best accuracy and AUC among alternatives	Used as evidence on hierarchical reasoning

On ArXiv classification, the paper states that WERSA improves accuracy over vanilla attention by about PRESERVED_PLACEHOLDER_^^^^8query8^^^^9 percentage points and reduces training time by about PRESERVED_PLACEHOLDER_^^^^8\8query8^^^^ and FLOPS by about PRESERVED_PLACEHOLDER_^^^^8\8\8^^^^. It also reports that Waveformer attains PRESERVED_PLACEHOLDER_^^^^8\8 OR WERSA8^^^^ accuracy and PRESERVED_PLACEHOLDER_^^^^8\8max_results8^^^^ AUC with PRESERVED_PLACEHOLDER_^^^^8\8sort_by8^^^^ s training time, whereas WERSA is both more accurate and more than twice as fast on that task (&&&^^^^8query8^^^^&&&).

On ArXiv-^^^^8\8 OR WERSA88^^^^k, the paper reports the following viable-method results: WERSA at PRESERVED_PLACEHOLDER_^^^^8\8submittedDate8^^^^ accuracy and PRESERVED_PLACEHOLDER_^^^^8\8query8^^^^ AUC with PRESERVED_PLACEHOLDER_^^^^8\8\8^^^^ s training time and PRESERVED_PLACEHOLDER_^^^^8\88^^^^ G FLOPS; Waveformer at PRESERVED_PLACEHOLDER_^^^^8\8^^^^9 and $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8query8^^^^ with $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8\8^^^^ s and $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8 OR WERSA8^^^^ G FLOPS; Performer at $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8max_results8^^^^ and $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8sort_by8^^^^ with $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8submittedDate8^^^^ s and $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8query8^^^^ G FLOPS; FNet at $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$^^^^8\8^^^^ and $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$8 with $$Q^{WT} = \mathcal{W}(Q'), \quad K^{WT} = \mathcal{W}(K').$$9 s and $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8query8^^^^ G FLOPS; and Linformer at $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8\8^^^^ and $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8 OR WERSA8^^^^ with $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8max_results8^^^^ s and $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8sort_by8^^^^ G FLOPS. The paper’s interpretation is that WERSA is not only efficient enough to fit, but also accurate enough to remain useful on extremely long sequences (&&&^^^^8query8^^^^&&&).

^^^^8submittedDate8^^^^. Relation to other wavelet-enhanced attention models

WERSA belongs to a broader set of models that use wavelet structure to modify attention or token mixing, but it occupies a distinct position within that literature. In computer vision, "Multiscale Attention via Wavelet Neural Operators for Vision Transformers" introduces Multiscale Wavelet Attention (MWA), which replaces self-attention in ViTs with a pipeline consisting of ^^^^8 OR WERSA8^^^^D discrete wavelet transform, convolution in the wavelet domain, GeLU, inverse DWT, and weighted skip branches with $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8submittedDate8^^^^ and $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8query8^^^^ convolutions. That paper argues that wavelets are localized in both space and frequency, better capture coarse-to-fine structure, and yield linear complexity in sequence size; it reports improvements over AFNO and Global Filter Networks on CIFAR and Tiny-ImageNet (&&&^^^^8\8max_results8^^^^&&&).

In graph-based spatio-temporal modeling, "Spatio-Temporal meets Wavelet: Disentangled Traffic Flow Forecasting via Efficient Spectral Graph Attention Network" uses the discrete wavelet transform to obtain low- and high-frequency components of traffic sequences, then combines a dual-channel encoder with a wavelet-based graph positional encoding and a ^^^^^^^^8query8^^^^^^^^ sampling strategy for efficient spectral graph attention. The model’s efficiency comes from sampled queries rather than randomized kernel approximation, and the paper explicitly distinguishes this from a “random spectral attention” formulation (&&&^^^^8\8sort_by8^^^^&&&).

Within efficient attention proper, WERSA is differentiated in the paper as follows. Compared with Multi-Head Attention, it replaces exact quadratic pairwise softmax with a wavelet-filtered random-feature approximation. Compared with FlashAttention-^^^^8 OR WERSA8^^^^, it changes the computational form of attention rather than primarily optimizing exact attention’s memory behavior. Compared with FNet, it uses localized wavelet bases rather than global sinusoidal Fourier bases. Compared with Linformer and Performer, it combines kernel linearization with explicit multi-resolution filtering. Compared with Waveformer, it adds content-adaptive filtering and learnable scale weights, and the paper explicitly contrasts this with Waveformer’s more uniform treatment of wavelet scales (&&&^^^^8query8^^^^&&&).

A plausible implication is that WERSA can be read as an overview of two previously separate research directions: wavelet-enhanced representation learning and linearized kernel attention. That synthesis is explicit in its architecture, where scale-aware filtering precedes random-feature attention rather than replacing it.

^^^^8query8^^^^. Ablations, interpretation, and limitations

The ablation study is used to support the claim that all major components contribute materially to performance. The paper reports that removing the wavelet component reduces accuracy, removing adaptive filters also harms performance, removing scale weights causes the largest drop, and removing random features dramatically increases FLOPS and runtime. The interpretation offered is that wavelets provide scale awareness, adaptive filters provide content selectivity, and random features provide the linear-time approximation (&&&^^^^8query8^^^^&&&).

The paper also positions WERSA as relevant for low-resource hardware and sustainable AI. Because it reduces both FLOPS and runtime substantially, it is presented as making long-context modeling feasible without large clusters. The emphasis on single-GPU experiments is part of that framing. The broader implication stated in the paper is that efficient attention is not only a matter of making models faster; it is also about making long-context Transformers deployable, affordable, and environmentally less costly (&&&^^^^8query8^^^^&&&).

Several interpretive cautions follow directly from the paper’s own comparisons. First, WERSA is not simply a wavelet attention mechanism in the same sense as MWA, because its linearity depends on the random-feature approximation rather than on replacing attention by a wavelet-domain convolutional mixer. Second, it is not equivalent to graph-spectral wavelet attention models such as STWave, whose efficiency comes from ^^^^^^^^8query8^^^^^^^^ sampling on graphs rather than from randomized kernel approximation (&&&^^^^8\8max_results8^^^^&&&, &&&^^^^8\8sort_by8^^^^&&&). Third, the claim of superiority is benchmark-specific: the paper reports best accuracy in all tests within its benchmark suite, but that statement is tied to the reported comparisons and experimental setting on a single GPU (&&&^^^^8query8^^^^&&&).

In summary, WERSA is defined by three coupled components: multi-resolution Haar wavelet decomposition, content-adaptive scale filtering with learnable weights, and a random spectral feature map for linearized attention. Its stated contribution is to preserve $$Q'_{\text{avg}, h} = \frac{1}{n} \sum_{i=1}^{n} Q'_{h, i},$$^^^^8\8^^^^ sequence scaling while improving selectivity over informative scales, and its empirical evidence centers on ArXiv and ArXiv-^^^^8\8 OR WERSA88^^^^k, where it is reported to outperform both exact quadratic baselines and several efficient alternatives under the paper’s experimental conditions (&&&^^^^8query8^^^^&&&).