LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

Abstract: The quadratic complexity of softmax attention presents a major obstacle for scaling Transformers to high-resolution vision tasks. Existing linear attention variants often replace the softmax with Gaussian kernels to reduce complexity, but such approximations lack theoretical grounding and tend to oversuppress mid-range token interactions. We propose LaplacianFormer, a Transformer variant that employs a Laplacian kernel as a principled alternative to softmax, motivated by empirical observations and theoretical analysis. To address expressiveness degradation under low-rank approximations, we introduce a provably injective feature map that retains fine-grained token information. For efficient computation, we adopt a Nyström approximation of the kernel matrix and solve the resulting system using Newton--Schulz iteration, avoiding costly matrix inversion and SVD. We further develop custom CUDA implementations for both the kernel and solver, enabling high-throughput forward and backward passes suitable for edge deployment. Experiments on ImageNet show that LaplacianFormer achieves strong performance-efficiency trade-offs while improving attention expressiveness.

• The paper demonstrates that replacing the Gaussian with a Laplacian kernel in linear attention significantly improves expressivity and optimization in dense vision tasks.
• It employs a Nyström-based approximation along with a custom CUDA Newton–Schulz solver to achieve linear time complexity and enhanced numerical stability.
• Empirical results on ImageNet and downstream tasks confirm that LaplacianFormer attains state-of-the-art accuracy and efficiency in both classification and localization.

LaplacianFormer: Rethinking Linear Attention with the Laplacian Kernel

Motivation for Kernel Redesign in Linear Attention

The principal limitation of standard Transformer self-attention in high-resolution vision is its $\mathcal{O}(N^2)$ complexity, which has motivated a proliferation of linear attention mechanisms. The vast majority of these replace softmax attention with kernel-based schemes, often defaulting to Gaussian kernels. The paper "LaplacianFormer: Rethinking Linear Attention with Laplacian Kernel" (2604.20368) advances a critical theoretical and empirical case that this convention lacks justification—particularly for dense vision—where query-key interaction distributions are heavy-tailed and the standard Gaussian kernel imposes unnecessarily rapid decay on mid-range similarities, leading to expressivity loss and optimization issues.

Figure 1: Visualization of $\ell_1$ and $\ell_2^2$ query-key distance distributions in several ViT architectures, showing the heavy-tailed characteristic that undermines squared-distance-based Gaussian kernels.

The authors propose LaplacianFormer, a ViT variant employing the Laplacian kernel as a drop-in replacement for the Gaussian kernel, justified by matching the kernel to the observed empirical distributions and gradient profiles in vision transformer networks. The Laplacian kernel, parameterized as $k(x, y) = \exp(-\|x - y\|_1 / \lambda)$, decays more slowly and affords increased robustness to outliers and a more faithful correspondence with observed token interactions.

Theoretical Analysis: Expressivity and Optimization

A central theoretical contribution is the observation that for vision transformers, squared $\ell_2$ query-key distances exhibit fat tails, which, compounded by the exponential in the Gaussian kernel, result in an oversuppression of mid-range and moderately-relevant context, as well as vanishing gradients that stifle effective learning. In contrast, attention maps and optimization dynamics derived from the Laplacian kernel retain gradient signal for small $\|q - k\|_1$ and support more structured attention patterns, with empirical evidence of improved convergence and higher accuracy in otherwise identical architectures.

Figure 2: (a) SOFT++ and Skyformer networks using Laplacian vs. Gaussian kernels exhibit faster convergence and higher ImageNet top-1 accuracy with Laplacian kernels. (b) Attention maps show improved structure and conditioning under Laplacian kernels, with suppression of spurious sparsity.

Appendix proofs confirm: (1) the Laplacian kernel's injective feature map ensures fine-grained discrimination between tokens, and (2) the change in kernel maintains the desirable full-rank, non-degenerate nature of token matching even in low-rank settings.

Algorithmic Advances: Nyström-Based Approximation and Solver Design

The complexity of computing full pairwise Laplacian similarities scales poorly with sequence length. LaplacianFormer leverages a Nyström approximation: selecting a small set of landmark keys and queries (using average pooling over spatial grids of the input), and constructing a low-rank kernel matrix that enables associativity and linear-time computation.

A primary technical challenge is stable and efficient inversion of the landmark kernel submatrix, required for normalization. The authors introduce a custom CUDA implementation of Newton–Schulz iteration for approximate kernel inversion, which is more robust under adverse condition numbers than conjugate gradient and avoids the inefficiency of SVDs for large matrices.

Figure 3: Comparison between vanilla softmax ($\mathcal{O}(N^2)$) and linear self-attention (with kernel decomposition), indicating the time and space savings conferred by the Laplacian-based method.

Figure 4: Execution time profiles for Laplacian kernel and Newton–Schulz iteration in forward and backward passes, with LaplacianFormer's CUDA kernels demonstrating substantial throughput advantages over PyTorch baselines.

Experimental Results

LaplacianFormer is benchmarked on ImageNet-1K, achieving superior Top-1 accuracy relative to linear and efficient transformer baselines at all FLOP regimes. Notably, LaplacianFormer-Tiny matches the best reported accuracy (81.4%) in the $<$3G FLOP range, and the Large and Huge configurations set new state-of-the-art at their respective scales. GPU memory usage and throughput scale linearly with input size, matching or exceeding the performance of the most efficient known models.

(Figure 2 again, left side)

Figure 2 (left): Discrete accuracy/FLOP tradeoff, illustrating LaplacianFormer’s leading efficiency frontier across transformer backbones.

On downstream tasks (object detection and instance segmentation with Mask R-CNN and RetinaNet), LaplacianFormer also surpasses previous linear attention backbones in both $\mathrm{AP}^b$ and $\mathrm{AP}^m$, confirming generalization beyond classification.

Ablation studies show the significant effect of kernel scale ($\ell_1$0); moderate values (e.g., $\ell_1$1) avoid both excessive suppression and oversmoothing. The Newton–Schulz solver proves essential for stability and high accuracy, consistently outperforming conjugate gradient.

Figure 5: Relative error convergence under varying condition numbers, showing Newton–Schulz’s superior robustness for ill-conditioned kernel inverses versus conjugate gradient.

Figure 6: Qualitative effect of Laplacian kernel scale ($\ell_1$2) on attention map structure, demonstrating optimal trade-off at intermediate kernel scales.

Implications and Future Directions

The findings suggest that the de facto use of the Gaussian kernel in linear self-attention is suboptimal for dense vision problems, where Laplacian or, more generally, heavy-tailed kernels are more consistent with token similarity statistics and optimization dynamics. LaplacianFormer’s injective, well-conditioned kernel mapping supports deeper, richer token interactions while maintaining efficiency for long sequences and high-resolution vision.

Practically, LaplacianFormer is well-suited for deployment on resource-constrained edge devices due to its efficient linear scaling and custom CUDA optimization. Architecturally, the method is modular and compatible as a swap-in for Gaussian or softmax-based mechanisms in standard self-attention blocks.

Theoretically, the LaplacianFormer approach highlights the necessity of grounding kernel choice in empirical distributional analysis and encouraging further investigation into alternative similarity metrics (cosine, polynomial, etc.), as well as in non-vision domains with differing interaction structures.

Conclusion

LaplacianFormer introduces a Laplacian kernel-based injective attention mechanism to linear vision transformers, yielding strong expressivity, efficient scaling, and superior empirical results. Its adoption of the Nyström approximation and a numerically stable CUDA-accelerated Newton–Schulz kernel inverse delivers practical speed and robustness, making it a compelling foundation for future kernelized transformer research in both vision and general sequence domains (2604.20368).