Reducing random-feature variance in high-dimensional kernel attention

Develop methods to eliminate or control the variance of random feature mappings in high-dimensional spaces for Performer-style softmax-attention kernel approximations.

Background

The paper situates MHA-CSP within work that interprets attention as a kernel method. It discusses Performers, which approximate the softmax attention kernel using orthogonal random features and provide convergence guarantees.

Despite those guarantees, the paper identifies the variance of random feature mappings in high-dimensional spaces as an unresolved engineering challenge. This issue is presented as a limitation of pure kernel-based attention approaches and as motivation for considering learned distance metrics instead.

References

Choromanski et al. [8] further introduced Performers, approximating the softmax attention kernel via orthogonal random features with provable convergence—but the variance of random feature mappings in high-dimensional spaces remains an unresolved engineering challenge.

— Mahalanobis-Based Multi-Head Attention for Complex State Propagation  (2608.24462 - Li, 25 Aug 2026) in Section 2.2, “Distance-Based Attention and Kernel Methods,” p. 4