Higher-Order Linear Attention (HLA)

• Higher-order Linear Attention (HLA) is a scalable mechanism that generalizes linear attention beyond first-order approximations, enabling efficient high-order feature interactions.
• It leverages polynomial expansions and tensor algebra to maintain near-linear per-token computational cost while capturing richer interactions compared to standard softmax and linear attention.
• HLA employs streaming prefix statistics and low-rank approximations to support extended context modeling in transformers and recurrent sequence models.

Higher-order Linear Attention (HLA) is a class of scalable attention mechanisms that generalize linear attention beyond first-order (kernelized or feature map) approximations to enable direct, efficient modeling of high-order feature interactions within modern neural architectures, especially transformers and recurrent sequence models. HLA mechanisms combine the statistical richness of higher-order attention (e.g., bilinear, trilinear, or higher tensor contractions) with per-token computational and memory costs that remain linear or near-linear in sequence length, through the analytic exploitation of polynomial expansions, prefix-sufficient statistics, or low-rank tensor algebra. This contrasts with standard softmax attention—whose quadratic (or higher) cost in sequence length prohibits long-context modeling—and with standard linear attention, which is limited to first-order kernels.

1. Motivation and Limitations of Standard and Linear Attention

Classical dot-product attention requires forming an $n\times n$ affinity matrix, yielding $O(n^2)$ time and memory, where $n$ is the sequence length. This scaling is prohibitive in long-context scenarios and increasingly relevant for multimodal or graph-structured data. Linear attention mechanisms address this bottleneck by replacing the softmax kernel with a static feature map $\phi$, such that $exp(q_t^\top k_j)$ is approximated via dot products in a higher-dimensional feature space, reducing the per-token cost to $O(n)$ (Zhang et al., 31 Oct 2025). However, these are strictly first-order; any interaction is linear in $\phi(k_j)$, which fundamentally limits their expressivity. Higher-order and tensorial interactions—proven effective in multimodal modeling and polynomial regression—are not captured within this linear regime (Suzuki, 18 Mar 2025).

State space models (SSMs) offer $O(1)$-per-token recurrence with long-context memory but lack the data-dependent mixing that is defining of attention. HLA mechanisms were developed to address both the scalability and expressivity gaps, enabling explicit, efficient computation of higher-order dependencies or tensor contractions across tokens, keys, queries, or modalities (Zhang et al., 31 Oct 2025, Liang et al., 2024).

2. Methodological Principles and Formulations

2.1 Prefix Sufficient Statistics and Streaming Operators

In the streaming, causal HLA construction (Zhang et al., 31 Oct 2025), the attention head maintains, for each token $t$, all required higher-order prefix moments (outer products, mixed polynomials) of queries, keys, and values. For second-order HLA, per-head statistics include:

$$S^K_t = \sum_{i \leq t} k_i k_i^\top \in \mathbb{R}^{d \times d}, \quad C^{QV}_t = \sum_{i \leq t} q_i v_i^\top \in \mathbb{R}^{d \times d_v}, \quad m^Q_t = \sum_{i \leq t} q_i \in \mathbb{R}^d$$

These summaries allow each per-token HLA output to be written as

$$o_t = \frac{q_t^\top S^K_t C^{QV}_t}{q_t^\top S^K_t m^Q_t + \varepsilon}$$

with $O(d^2+d d_v)$ cost per token. No $n\times n$ matrices are stored or materialized.

2.2 Strict Causality and Masked Statistics

To enforce strict autoregressive masking, additional cross-summaries are required—e.g., $G_t = \sum_{i \leq t} (k_i k_i^\top) C^{QV}_{i-1}$ and $h_t = \sum_{i \leq t} (k_i k_i^\top) m^Q_{i-1}$—subtracted from $S^K_t C^{QV}_t$ and $S^K_t m^Q_t$ to mask future contributions. Online recurrences are maintained in $O(d^2 + d d_v)$ cost. These operations extend to third and higher order with further moment, cross, and mixed statistics—e.g., for order-$r$, prefix moments of the form $$\sum k^{\otimes p} \otimes q^{\otimes q} \otimes v^{\otimes r-p-q}$$ and accompanying causal correction summaries (Zhang et al., 31 Oct 2025).

2.3 Polynomial and Tensor Algebraic Acceleration

Most tensor attention formulations would incur $O(n^r)$ cost for order-$r$ attention. Recent work achieves almost-linear ($O(n^{1+o(1)})$) complexity by applying polynomial kernel approximations and low-rank tensor factorization. For instance, under a bounded-entries assumption ($$\|Q\|_\infty, \|K_i\|_\infty \le B = o((\log n)^{1/3})$$), the softmax row-stochastic matrix is provably $\epsilon$-approximated by a low-rank factorization $S \approx U_1 (V_1 \oslash W_1)^\top$, with all terms constructed via polynomial expansions (Liang et al., 2024).

In practice, the gradients and forward contractions are computed via chains of Kronecker and Khatri-Rao products, and Hadamard–Khatri–Rao algebra, lifting all necessary tensor contractions to operate on small low-dimensional factors, never instantiating the large $n^r$ slices typical of dense higher-order attention (Liang et al., 2024). Both forward and backward passes thus attain $O(n^{1+o(1)})$ time and $O(n\,r)$ memory, with $r=n^{o(1)}$.

2.4 Taylor and Kernel Expansion Approaches

Another instantiation of HLA uses Taylor expansion of the exp kernel, e.g. second-order approximation $exp(z) \approx 1+z+z^2/2$. This motivates the use of a feature map collecting all degree-$\le2$ monomials in the queries and keys (Mercat, 2020). The corresponding contraction enables all necessary terms to be constructed with streaming prefix sums and second-moment statistics—again avoiding explicit $n\times n$ computation.

3. Concrete Instantiations and Algorithms

The table summarizes the main instantiations of HLA mechanisms in the literature.

Paper / Method	Order Supported	Key Statistic / Feature Map	Per-token Cost
(Zhang et al., 31 Oct 2025) (Zhang et al.)	2, 3, $r$	Prefix sums of all outer products, with cross-momentum for causality	$O(d^r + d^{r-1} d_v)$
(Liang et al., 2024) (Gur-Ari et al.)	$m$	Polynomial low-rank and tensor-Kronecker factorizations	$O(n^{1+o(1)})$
(Mercat, 2020) (Mercat)	2	Monomial embedding to quadratic (Taylor-2)	$O(n d^2 d_v)$
(Pan et al., 2020) (X-Linear)	2-$M$	Layer-stacked bilinear (Hadamard/outer-product pooling)	$O(M N D_B)$

HLA heads are typically integrated into transformer layers by replacing the standard QK$^\top$ attention or softmax computation with these higher-order streaming/moment-based or polynomially accelerated mechanisms. For parallel and chunked training, associative scan algorithms (Blelloch-style) enable chunk-wise prefix reductions, maintaining statistical consistency and exact equivalence to true serial recurrence (Zhang et al., 31 Oct 2025).

4. Complexity, Scalability, and Theoretical Guarantees

HLA mechanisms retain the per-token linear scalability of linear attention, but the state size and update cost grow quickly with order. For order-$r$, the parameter and memory cost is $O(d^r + \cdots)$ per head, and online update $O(d^r)$. While second and third orders are computationally feasible for modern accelerators, higher orders rapidly become impractical due to combinatorial growth (Zhang et al., 31 Oct 2025).

The polynomial kernel approximation framework (Liang et al., 2024) is subject to a tight bounded-entries assumption. If query-key-element entries exceed $B = o((\log n)^{1/3})$, a SETH-based hardness result precludes truly subcubic algorithms—even for approximate forward/backward passes. This establishes both the sufficiency and minimality of the boundedness precondition for scalable higher-order attention.

5. Empirical Benefits and Applications

Empirical results in structured vision-language tasks (e.g., image captioning) find that bilinear (second-order) HLA (as in X-Linear) yields measurable improvements in finer-grained grounding, caption richness, and benchmark metrics (e.g., COCO Karpathy test split CIDEr of 132.8 for Transformer+X-Linear) (Pan et al., 2020). The streaming, masked HLA mechanism supports strict autoregressive constraints and chunk-parallel training, aligning with the requirements of large autoregressive LLMs (Zhang et al., 31 Oct 2025).

While the aforementioned polynomial and tensorial HLA variants provide theoretical and algorithmic feasibility, large-scale empirical comparisons to standard softmax attention and first-order linear attention on language (beyond vision/captioning) remain limited in the literature as of 2025. Some variants have only been evaluated on synthetic or random data (Mercat, 2020).

6. Relations and Comparisons with Other High-Order Mechanisms

Bilinear and tensor attention have a long-standing presence in fine-grained vision and multimodal tasks, frequently realized via stacking blocks, channel attention, or direct elementwise pooling (Pan et al., 2020). HLA frameworks generalize and unify these approaches, providing a statistical or polynomial basis for higher-order modeling with explicit efficiency guarantees. The X-Linear block achieves high-order (up to infinite, using ELU) interactions either by stacking or via parameter-free nonlinearities, while remaining compatible with transformer and encoder-decoder frameworks.

Extension to higher orders (beyond third) is possible in principle via prefix-moment maintenance, but rapidly confronts combinatorial explosion in state dimension and per-token update, limiting practical utility unless further algebraic compression or randomized approximation strategies are leveraged (Zhang et al., 31 Oct 2025, Liang et al., 2024, Mercat, 2020).

7. Practical Implementation and Recommendations

Integration of HLA into large-scale transformer models involves substituting the canonical attention head with the higher-order moment or low-rank factor streaming variants. It is recommended to scale and normalize projections tightly ($$\left\|Q\right\|_\infty, \left\|K_i\right\|_\infty = O((\log n)^{1/3})$$) to ensure approximation accuracy and computational feasibility (Liang et al., 2024). Chunk- or scan-based parallel algorithms are mandatory to achieve high throughput on modern hardware.

For higher-order HLA, numerical stability may be improved by adding regularization (ridge terms) to Gram matrices and by controlling polynomial expansion truncation error. Multi-query/key/value sharing, packing symmetric storage, and careful bookkeeping for Kronecker/Hadamard products further reduce memory footprint. Algorithmic details and pseudocode for efficient implementation are provided explicitly in (Liang et al., 2024, Zhang et al., 31 Oct 2025).

Taken together, HLA provides a flexible, scalable framework for leveraging high-order feature and token interactions in deep sequence and multimodal models, closing the expressivity gap previously left by kernel-linear attention, while preserving (quasi-)linear efficiency at both training and inference scales.