---
title: Higher-Order Linear Attention (HLA)
url: https://www.emergentmind.com/topics/higher-order-linear-attention-hla
type: topic
---

# Higher-Order Linear Attention (HLA)

Higher-order Linear Attention (HLA) is a class of scalable attention mechanisms that generalize linear attention beyond first-order (kernelized or feature map) approximations to enable direct, efficient modeling of high-order feature interactions within modern neural architectures, especially transformers and recurrent sequence models. HLA mechanisms combine the statistical richness of higher-order attention (e.g., bilinear, trilinear, or higher tensor contractions) with per-token computational and memory costs that remain linear or near-linear in sequence length, through the analytic exploitation of polynomial expansions, prefix-sufficient statistics, or low-rank tensor algebra. This contrasts with standard softmax attention—whose quadratic (or higher) cost in sequence length prohibits long-context modeling—and with standard linear attention, which is limited to first-order kernels.

## 1. Motivation and Limitations of Standard and Linear Attention

Classical dot-product attention requires forming an $n\times n$ affinity matrix, yielding $O(n^2)$ time and memory, where $n$ is the sequence length. This scaling is prohibitive in long-context scenarios and increasingly relevant for multimodal or graph-structured data. Linear attention mechanisms address this bottleneck by replacing the softmax kernel with a static feature map $\phi$, such that $exp(q_t^\top k_j)$ is approximated via dot products in a higher-dimensional feature space, reducing the per-token cost to $O(n)$ [2510.27258]. However, these are strictly first-order; any interaction is linear in $\phi(k_j)$, which fundamentally limits their expressivity. Higher-order and tensorial interactions—proven effective in multimodal modeling and polynomial regression—are not captured within this linear regime [2503.14080].

State space models (SSMs) offer $O(1)$-per-token recurrence with long-context memory but lack the data-dependent mixing that is defining of attention. HLA mechanisms were developed to address both the scalability and expressivity gaps, enabling explicit, efficient computation of higher-order dependencies or tensor contractions across tokens, keys, queries, or modalities [2510.27258, 2405.16411].

## 2. Methodological Principles and Formulations

### 2.1 Prefix Sufficient Statistics and Streaming Operators

In the streaming, causal HLA construction [2510.27258], the attention head maintains, for each token $t$, all required higher-order prefix moments (outer products, mixed polynomials) of queries, keys, and values. For second-order HLA, per-head statistics include:

$$
S^K_t = \sum_{i \leq t} k_i k_i^\top \in \mathbb{R}^{d \times d}, \quad
C^{QV}_t = \sum_{i \leq t} q_i v_i^\top \in \mathbb{R}^{d \times d_v}, \quad
m^Q_t = \sum_{i \leq t} q_i \in \mathbb{R}^d
$$

These summaries allow each per-token HLA output to be written as

$$
o_t = \frac{q_t^\top S^K_t C^{QV}_t}{q_t^\top S^K_t m^Q_t + \varepsilon}
$$

with $O(d^2+d d_v)$ cost per token. No $n\times n$ matrices are stored or materialized.

### 2.2 Strict Causality and Masked Statistics

To enforce strict autoregressive masking, additional cross-summaries are required—e.g., $G_t = \sum_{i \leq t} (k_i k_i^\top) C^{QV}_{i-1}$ and $h_t = \sum_{i \leq t} (k_i k_i^\top) m^Q_{i-1}$—subtracted from $S^K_t C^{QV}_t$ and $S^K_t m^Q_t$ to mask future contributions. Online recurrences are maintained in $O(d^2 + d d_v)$ cost. These operations extend to third and higher order with further moment, cross, and mixed statistics—e.g., for order-$r$, prefix moments of the form $\sum k^{\otimes p} \otimes q^{\otimes q} \otimes v^{\otimes r-p-q}$ and accompanying causal correction summaries [2510.27258].

### 2.3 Polynomial and Tensor Algebraic Acceleration

Most tensor attention formulations would incur $O(n^r)$ cost for order-$r$ attention. Recent work achieves almost-linear ($O(n^{1+o(1)})$) complexity by applying polynomial kernel approximations and low-rank tensor factorization. For instance, under a bounded-entries assumption ($\|Q\|_\infty, \|K_i\|_\infty \le B = o((\log n)^{1/3})$), the softmax row-stochastic matrix is provably $\epsilon$-approximated by a low-rank factorization $S \approx U_1 (V_1 \oslash W_1)^\top$, with all terms constructed via polynomial expansions [2405.16411].

In practice, the gradients and forward contractions are computed via chains of Kronecker and Khatri-Rao products, and Hadamard–Khatri–Rao algebra, lifting all necessary tensor contractions to operate on small low-dimensional factors, never instantiating the large $n^r$ slices typical of dense higher-order attention [2405.16411]. Both forward and backward passes thus attain $O(n^{1+o(1)})$ time and $O(n\,r)$ memory, with $r=n^{o(1)}$.

### 2.4 Taylor and Kernel Expansion Approaches

Another instantiation of HLA uses Taylor expansion of the exp kernel, e.g. second-order approximation $exp(z) \approx 1+z+z^2/2$. This motivates the use of a feature map collecting all degree-$\le2$ monomials in the queries and keys [2010.14816]. The corresponding contraction enables all necessary terms to be constructed with streaming prefix sums and second-moment statistics—again avoiding explicit $n\times n$ computation.

## 3. Concrete Instantiations and Algorithms

The table summarizes the main instantiations of HLA mechanisms in the literature.

| Paper / Method                         | Order Supported | Key Statistic / Feature Map                                  | Per-token Cost          |
|----------------------------------------|-----------------|--------------------------------------------------------------|------------------------|
| [2510.27258] (Zhang et al.)            | 2, 3, $r$       | Prefix sums of all outer products, with cross-momentum for causality | $O(d^r + d^{r-1} d_v)$ |
| [2405.16411] (Gur-Ari et al.)          | $m$             | Polynomial low-rank and tensor-Kronecker factorizations      | $O(n^{1+o(1)})$        |
| [2010.14816] (Mercat)                  | 2               | Monomial embedding to quadratic (Taylor-2)                   | $O(n d^2 d_v)$         |
| [2003.14080] (X-Linear)                | 2-$M$           | Layer-stacked bilinear    (Hadamard/outer-product pooling)   | $O(M N D_B)$           |

HLA heads are typically integrated into transformer layers by replacing the standard QK$^\top$ attention or softmax computation with these higher-order streaming/moment-based or polynomially accelerated mechanisms. For parallel and chunked training, associative scan algorithms (Blelloch-style) enable chunk-wise prefix reductions, maintaining statistical consistency and exact equivalence to true serial recurrence [2510.27258].

## 4. Complexity, Scalability, and Theoretical Guarantees

HLA mechanisms retain the per-token linear scalability of linear attention, but the state size and update cost grow quickly with order. For order-$r$, the parameter and memory cost is $O(d^r + \cdots)$ per head, and online update $O(d^r)$. While second and third orders are computationally feasible for modern accelerators, higher orders rapidly become impractical due to combinatorial growth [2510.27258].

The polynomial kernel approximation framework [2405.16411] is subject to a tight bounded-entries assumption. If query-key-element entries exceed $B = o((\log n)^{1/3})$, a SETH-based hardness result precludes truly subcubic algorithms—even for approximate forward/backward passes. This establishes both the sufficiency and minimality of the boundedness precondition for scalable higher-order attention.

## 5. Empirical Benefits and Applications

Empirical results in structured vision-language tasks (e.g., image captioning) find that bilinear (second-order) HLA (as in X-Linear) yields measurable improvements in finer-grained grounding, caption richness, and benchmark metrics (e.g., COCO Karpathy test split CIDEr of 132.8 for Transformer+X-Linear) [2003.14080]. The streaming, masked HLA mechanism supports strict autoregressive constraints and chunk-parallel training, aligning with the requirements of large autoregressive language models [2510.27258].

While the aforementioned polynomial and tensorial HLA variants provide theoretical and algorithmic feasibility, large-scale empirical comparisons to standard softmax attention and first-order linear attention on language (beyond vision/captioning) remain limited in the literature as of 2025. Some variants have only been evaluated on synthetic or random data [2010.14816].

## 6. Relations and Comparisons with Other High-Order Mechanisms

Bilinear and tensor attention have a long-standing presence in fine-grained vision and multimodal tasks, frequently realized via stacking blocks, channel attention, or direct elementwise pooling [2003.14080]. HLA frameworks generalize and unify these approaches, providing a statistical or polynomial basis for higher-order modeling with explicit efficiency guarantees. The X-Linear block achieves high-order (up to infinite, using ELU) interactions either by stacking or via parameter-free nonlinearities, while remaining compatible with transformer and encoder-decoder frameworks.

Extension to higher orders (beyond third) is possible in principle via prefix-moment maintenance, but rapidly confronts combinatorial explosion in state dimension and per-token update, limiting practical utility unless further algebraic compression or randomized approximation strategies are leveraged [2510.27258, 2405.16411, 2010.14816].

## 7. Practical Implementation and Recommendations

Integration of HLA into large-scale transformer models involves substituting the canonical attention head with the higher-order moment or low-rank factor streaming variants. It is recommended to scale and normalize projections tightly ($\left\|Q\right\|_\infty, \left\|K_i\right\|_\infty = O((\log n)^{1/3})$) to ensure approximation accuracy and computational feasibility [2405.16411]. Chunk- or scan-based parallel algorithms are mandatory to achieve high throughput on modern hardware.

For higher-order HLA, numerical stability may be improved by adding regularization (ridge terms) to Gram matrices and by controlling polynomial expansion truncation error. Multi-query/key/value sharing, packing symmetric storage, and careful bookkeeping for Kronecker/Hadamard products further reduce memory footprint. Algorithmic details and pseudocode for efficient implementation are provided explicitly in [2405.16411, 2510.27258].

Taken together, HLA provides a flexible, scalable framework for leveraging high-order feature and token interactions in deep sequence and multimodal models, closing the expressivity gap previously left by kernel-linear attention, while preserving (quasi-)linear efficiency at both training and inference scales.

Source: https://www.emergentmind.com/topics/higher-order-linear-attention-hla