Fractional Neural Attention

• Fractional Neural Attention is a framework that integrates α-stable Lévy diffusion and the fractional Laplacian to model both short- and long-range dependencies in transformers.
• It replaces local dot-product similarity with a nonlocal, power-law kernel, yielding larger spectral gaps and faster information mixing.
• Empirical results show FNA's competitive performance in text, vision, and machine translation, often reducing the need for deeper layers or higher embedding dimensions.

Fractional Neural Attention (FNA) is a neuroscience-inspired framework for multiscale information processing in neural architectures, most notably Transformers. FNA replaces the local, exponentially decaying structure of conventional self-attention with a principled mechanism based on Lévy diffusion and the fractional Laplacian operator, thereby embedding power-law statistics observed in biological attention directly into the attention mechanism. This approach enables rich, simultaneous modeling of short and long-range dependencies, yielding provable theoretical advantages—including larger spectral gaps, shorter path lengths, and improved information mixing efficiency—while often matching or surpassing conventional baselines in text, vision, and machine translation tasks (Qu et al., 13 Nov 2025).

1. Motivations: Biological Inspiration and Limitations of Classical Attention

Empirical studies in natural vision and neural circuit behavior indicate that biological attention operates through a mixture of small, local shifts and rare, large jumps, a pattern well-modeled by symmetric $\alpha$-stable Lévy processes. These heavy-tailed jump distributions afford efficient coverage and information gathering across complex environments, with step size distributions decaying according to a power law.

In contrast, standard Transformer architectures utilize a dot-product self-attention scheme that, under appropriate normalization and scaling, approximates Brownian diffusion (the heat equation). This mechanism is dominated by local, short-range interactions due to the exponentially decaying Gaussian kernel, leading to slow long-range information propagation. FNA, motivated by these neuroscientific observations and dynamical systems theory, substitutes Brownian with Lévy diffusion, governed by the fractional Laplacian, to facilitate power-law, multiscale interactions.

2. Mathematical Framework

2.1 Fractional Laplacian and Diffusion

The core operator in FNA is the isotropic fractional Laplacian of order $\alpha \in (0,2)$:

$$(−\Delta)^{\alpha/2}u(x) = c_{d,\alpha} \text{P.V.} \int_{\mathbb{R}^d} \frac{u(x) - u(y)}{||x-y||^{d+\alpha}}\,dy,$$

where $c_{d,\alpha}$ is a normalization constant and P.V. denotes the Cauchy principal value. As $\alpha \to 2$, this operator recovers the local Laplacian; for $\alpha < 2$, it encodes nonlocal, long-range interactions.

The corresponding fractional diffusion equation governs the density evolution under symmetric $\alpha$-stable Lévy motion:

$$\partial_t \rho(x,t) = -(−\Delta)^{\alpha/2}\rho(x,t),$$

yielding heat kernels with power-law tails for $\alpha < 2$:

$$k_t(x,y) \asymp t^{-d/\alpha}(1 + ||x-y||/t^{1/\alpha})^{-(d+\alpha)}.$$

2.2 FNA Computation of Attention Weights

Tokens $x_i$ are identified as particles undergoing fractional diffusion. Rather than the dot-product similarity of classic attention ($C_{ij} = q_i \cdot k_j$ followed by softmax normalization), FNA computes a kernelized attention matrix:

$$\tilde{C}_{ij} = \Phi_\alpha \bigg( \frac{||W_Q x_i - W_K x_j||}{\kappa} \bigg),$$

with

• $\Phi_2(r) = \exp(-r^2)$ for $\alpha=2$ (classical Gaussian),
• $\Phi_\alpha(r) \approx (1 + r)^{-(d + \alpha)}$ for $\alpha \in (0,2)$.

Attention weights are then obtained by row normalization:

$$\tilde{A}_{ij} = \frac{\tilde{C}_{ij}}{\sum_k \tilde{C}_{ik}},$$

and the residual update of each token state is

$$x_i \leftarrow x_i + \sum_{j=1}^n \tilde{A}_{ij} V x_j.$$

In the continuum limit, this approach yields the fractional diffusion generator:

$$\mathcal{H}^{\text{FNA}}[f] = \frac{(−\Delta)^{\alpha/2}(f\rho) - f(−\Delta)^{\alpha/2}\rho}{\rho}.$$

3. Theoretical Properties: Spectral Gaps and Information Mixing

The spectral theory of the fractional Laplacian provides a mechanistic explanation for the efficiency of FNA. On a compact domain or graph, its eigenvalues follow the scaling $\lambda_j \propto j^{\alpha/d}$. In the context of the stochastic attention matrix, this induces a spectral gap

$\gamma = 1 - e^{-t\lambda_2},$

which is strictly larger for FNA ($\alpha<2$) than for Brownian/self-attention ($\alpha=2$). A larger spectral gap guarantees faster mixing for the associated Markov chain, i.e., more rapid propagation of information across the token graph.

Moreover, a graph-theoretic analysis reveals that minimal path lengths under FNA are frequently 1 (due to direct, long-range jumps enabled by heavy-tailed kernels), whereas in standard attention they can scale as $O(n)$ for distant pairs in long sequences. This property enables a single FNA layer to capture multiscale dependencies that otherwise would require stacking many conventional layers.

4. Implementation and Efficient Computation

4.1 Kernel and Distance Computation

Computing all pairwise $n^2$ distances between projected queries and keys is $O(n^2 d)$. Practical implementations can leverage:

• Sparse neighborhoods (computing for $k \ll n$ nearest neighbors),
• Kernel approximation methods (Nyström, locality-sensitive hashing),
• Specialized block-matrix or FlashAttention kernels for high throughput.

4.2 Layer Pseudocode

A single FNA layer is computed as follows:

Q = X @ W_Q.T # n×d’ K = X @ W_K.T # n×d’ V = X @ W_V.T # n×d for i in range(n): for j in range(n): D[i,j] = np.linalg.norm(Q[i] - K[j]) if α == 2: C = np.exp(-D**2 / κ**2) else: C = (1 + D / κ)**(- (d' + α)) A = C / C.sum(axis=1, keepdims=True) Y = A @ V # n×d return X + Y # residual connection

A plausible implication is that further engineering improvements can substantially reduce computational overhead in large-scale deployment.

5. Empirical Performance Across Modalities

5.1 Text Classification (IMDb)

For single-layer, single-head models on IMDb, FNA ($\alpha=1.2$, $d=8$) achieves approximately 82% accuracy. In contrast, dot-product and local ($\alpha=2$) attention require embedding dimension $d\geq64$ or $L\geq2$ layers for comparable performance. Edge ablation shows FNA's accuracy rapidly collapses under random masking, indicating that FNA's expressivity is primarily mediated by its attention pathway rather than by feed-forward sublayers.

5.2 Vision (CIFAR-10)

In 4-layer, 6-head Vision Transformers, FNA ($\alpha=1.2$) attains a test accuracy of 76.17% on CIFAR-10, outperforming local attention (75.14%) and matching the baseline Transformer (76.03%). Imposing orthogonality on query/key projections yields small performance drops across methods but preserves FNA's relative superiority.

5.3 Neural Machine Translation (Multi30K En–De)

In a 6-layer, 8-head machine translation setting, FNA with $\alpha=1.2$ achieves BLEU$\approx$34.64, improving over local attention (34.13) and dot-product (34.00) baselines.

6. Diffusion Maps: Visualization and Dimensionality Reduction

FNA's connection to manifold diffusion facilitates the application of the diffusion map algorithm to its learned attention graphs. Given a row-stochastic attention matrix $\tilde{A}$, symmetric normalization yields $\bar{A}$, whose eigenvectors $(\phi_j, \psi_j)$ correspond to intrinsic coordinates on the diffusion geometry.

For an embedding dimension $m \ll n$ and diffusion time $\tau$,

$$\Psi_\tau(x_i) = (\eta_1^\tau\psi_1(i), \eta_2^\tau\psi_2(i), ..., \eta_m^\tau\psi_m(i)) \in \mathbb{R}^m$$

provides a low-dimensional map that preserves diffusion distances, quantifying effective connectivity under multiscale attention. FNA graphs ($\alpha < 2$) exhibit tightly clustered diffusion maps, indicating that all tokens communicate efficiently via long-range connections, in contrast to the more localized structure observed for $\alpha=2$.

7. Summary and Implications

Fractional Neural Attention systematically instantiates $\alpha$-stable Lévy dynamics within transformer attention, yielding a multiscale mechanism that unifies short- and long-range dependencies in a single mathematical operator. The approach connects advances in stochastic processes, spectral graph theory, and geometry to practical neural architectures, with empirical validation across text, vision, and translation tasks. The intrinsic manifold geometry of FNA's attention weights is accessible to established diffusion-based dimensionality reduction techniques, further bridging the gap between theoretical underpinnings and interpretability in deep learning (Qu et al., 13 Nov 2025).