Attentive Graph Filter (AGF)

• AGF is a method that reconceptualizes Transformer self-attention as a learnable graph filter operating in the singular value domain of a directed graph.
• It applies higher-order spectral filtering using an orthogonal polynomial basis, such as Jacobi polynomials, to capture richer frequency information beyond standard low-pass filtering.
• The AGF architecture maintains linear computational complexity while achieving state-of-the-art performance across language, vision, and time-series tasks.

The Attentive Graph Filter (AGF) is a method that reconceptualizes the Transformer self-attention mechanism as a learnable graph filter in the singular value domain of a directed graph. AGF integrates concepts from graph signal processing (GSP) with efficient linear Transformer architectures, enabling the learning of rich spectral filters beyond the first-order low-pass operation of standard self-attention. AGF achieves state-of-the-art results across language, vision, and time-series tasks, attaining linear computational complexity in the sequence length and broadening the frequency information that can be leveraged in Transformer architectures (Wi et al., 13 May 2025).

1. Reformulation of Self-Attention as a Graph Filter

Transformers process tokens $\{1,\dots,n\}$ as nodes of a fully connected directed graph, where the standard dot-product self-attention matrix

$\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$

serves as a normalized, row-stochastic adjacency or shift operator. The value matrix $V\in\mathbb{R}^{n\times d}$ is interpreted as graph signals on the nodes. GSP defines $K$-hop, shift-invariant filters as polynomials in the graph shift operator $S$:

$H_{\mathrm{GSP}}\,x = \sum_{k=0}^K w_k S^k x.$

Vanilla self-attention corresponds to the case $K=1$, $w_1=1$, $w_0=0$, constituting a first-order (one-hop) low-pass graph filter that attenuates high-frequency variation among token features. The structure, as formalized in Theorem 1 of the referenced work, limits the frequency diversity captured by standard self-attention (Wi et al., 13 May 2025).

2. AGF Architecture: Spectral Filtering in the Singular Value Domain

AGF generalizes self-attention by learning higher-order graph filters in the singular value (spectral) domain of the attention graph. Instead of constructing the usually asymmetric matrix $\bar A$ or its SVD explicitly, AGF simulates SVD decomposition by computing three $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$0 or $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$1 projections: $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$2

$\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$3

Here, $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$4 are learnable parameters. This yields the simulated SVD

$\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$5

A polynomial spectral filter of order $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$6 is learned over the diagonal entries of $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$7 (expanded as $\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$8), using an orthogonal polynomial basis, specifically Jacobi polynomials in practice:

$\bar A = \softmax(QK^\top/\sqrt d)\in\mathbb{R}^{n\times n}$9

where $V\in\mathbb{R}^{n\times d}$0 are learnable coefficients. The full AGF output is given by:

$V\in\mathbb{R}^{n\times d}$1

where $V\in\mathbb{R}^{n\times d}$2 is the value-projection matrix.

3. Flexible Spectral Passbands and Filter Order

The choice of spectral filter coefficients $V\in\mathbb{R}^{n\times d}$3 determines the frequency response:

• If $V\in\mathbb{R}^{n\times d}$4 and $V\in\mathbb{R}^{n\times d}$5, AGF remains low-pass (as in vanilla GSP, Theorem 2).
• Allowing negative $V\in\mathbb{R}^{n\times d}$6 enables emphasis on mid- or high-frequency components. For example, $V\in\mathbb{R}^{n\times d}$7 recovers high-pass behavior.

By adopting polynomials of order $V\in\mathbb{R}^{n\times d}$8, AGF incorporates up to $V\in\mathbb{R}^{n\times d}$9-hop relational information, enabling more expressive token interactions and adaptively controlling the balance across spectral frequencies. This spectral flexibility surpasses the one-hop averaging constraint of standard self-attention. Use of the Jacobi polynomial basis provides improved convergence and stability relative to monomial alternatives.

4. Computational Complexity and Efficiency

AGF achieves $K$0 runtime and $K$1 space, matching the efficiency of many linear Transformer variants. The computational workflow is:

Step	Operation	Complexity
Simulate SVD factors	$K$2, $K$3, $K$4	$K$5
Spectral filter application	$K$6	$K$7
Matrix formation and multiplication	$K$8, $K$9	$S$0
Output computation	Final multiplications	$S$1

The overall complexity remains linear in the input length $S$2, with negligible overhead from spectral polynomial order $S$3 when $S$4.

5. Pseudocode Illustration

A concise pseudocode sketch elucidates the computation:

$S$9

This makes explicit how AGF replaces the directed graph shift operator $S$5 with learnable SVD factors and a spectral polynomial filter.

6. Empirical Validation and Ablation Analysis

AGF demonstrates consistent accuracy improvements across diverse benchmarks:

• On time-series classification (UEA archive, 10 multivariate datasets), AGF achieves 75.1% average accuracy, surpassing vanilla Transformer and linear variants (71–73%).
• For Long Range Arena (5 tasks, lengths 1K–4K), AGF attains an overall 60.1%, outperforming all baselines (60.0 vanilla, 59–59.5 others).
• Incorporating AGF into several layers of DeiT-Small for ImageNet-100/1K yields 0.6–0.5% top-1 accuracy gains without loss in speed.

Ablation studies reveal further insights:

• Fixing $S$6 and learning only $S$7 results in 72.1% average UEA accuracy; adding learnable singular values attains 72.4%; the full AGF with spectral polynomial achieves 75.1%.
• The Jacobi basis for spectral polynomials provides ∼3 points improvement and greater stability over monomials.
• Removing softmax on $S$8 or using alternatives (tanh/sigmoid) degrades performance by 3–5 points, indicating the necessity for probability-like behavior in basis vectors to ensure stability (Wi et al., 13 May 2025).

7. Context and Research Significance

AGF establishes a principled connection between Transformer attention and graph signal processing, demonstrating that self-attention mechanisms can be interpreted as low-pass graph filters. By learning advanced spectral filters in the singular value domain, AGF enhances the expressivity and frequency resolution of sequence modeling architectures, all while maintaining linear complexity. The empirical superiority of AGF across language, vision, and time-series modalities, as well as its stable convergence properties, establishes its role as an important advancement in the field of scalable Transformer models (Wi et al., 13 May 2025).