Infinite Self-Attention (InfSA)

• Infinite Self-Attention is a family of methods that relax finite attention constraints by aggregating multi-hop token interactions and summing paths of arbitrary lengths.
• Neumann-series formulations and graph diffusion techniques provide tractable approximations for infinite-hops, bridging single-hop attention with global token influence using eigenvector methods.
• Variants like ReAttention and Infini-attention decouple candidate selection from positional encoding and employ compressed memory to efficiently handle unbounded contexts.

Infinite Self-Attention (InfSA) denotes several closely related extensions of self-attention that relax different finite constraints of the standard Transformer. In the cited literature, the label is used for a training-free scheme that supports an infinite context with a finite attention scope in LLMs, a Neumann-series operator that accumulates multi-hop interactions on a token graph, a compressed-memory mechanism studied under the name Infini-attention, and an infinite-width analysis in which multi-head attention converges to NNGP and NTK limits (Liu et al., 2024, Roffo et al., 26 Feb 2026, Huang et al., 29 Dec 2025, Hron et al., 2020, Roffo, 19 Jul 2025). Across these usages, the common theme is that standard one-hop, finite-window attention is replaced by a mechanism that either aggregates over arbitrarily long histories, sums paths of every length, or admits an infinite limit with a tractable kernel description.

1. Standard self-attention and the finite-regime bottleneck

Standard position-aware self-attention for a sequence of length $N$ is written as

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$

In the long-context setting, this requires $\mathcal O(N^2)$ time and memory and forces all key-value pairs to be stored in a single window. As $N\to\infty$, this becomes infeasible both physically, because memory blows up, and statistically, because rotary or absolute position embeddings fall outside the training-time range, causing length-extrapolation failure and entropy explosion in the softmax (Liu et al., 2024).

Length extrapolation is the regime $N>L_{\max}$, where a Transformer trained up to a maximum context $L_{\max}$ degrades at test time. The cited description characterizes this degradation by rising perplexity, vanishing retrieval accuracy, and collapse of generation quality once absolute-position embeddings go out of distribution or attention entropy saturates (Liu et al., 2024). In high-resolution vision, an analogous constraint appears as the quadratic cost of softmax attention, which limits Transformer scalability (Roffo et al., 26 Feb 2026).

A broader affinity-matrix view places standard self-attention inside a larger family of multi-hop propagation operators. Given a learned attention matrix

$W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$

the Transformer output

$Z=W\,V$

is the one-hop case of a more general series over powers of an affinity matrix $A$. In this view, self-attention is a single-hop instance of a construction rooted in Infinite Feature Selection, while Infinite Self-Attention arises when all hop lengths are included rather than only $k=1$ (Roffo, 19 Jul 2025).

2. Neumann-series formulations and infinite-hop propagation

A central mathematical formulation of InfSA is the Neumann series

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$0

where $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$1 is a nonnegative affinity matrix and $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$2 is its spectral radius. Discarding the zero-hop term yields

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$3

The $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$4-th power $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$5 encodes all $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$6-hop walks between pairs of nodes, so the series integrates information from paths of every length (Roffo, 19 Jul 2025).

In the spectral formulation introduced for vision Transformers, the usual attention matrix is

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$7

and a discount factor $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$8 is used to define the infinite-path kernel

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$9

with path-integral operator

$\mathcal O(N^2)$0

For stacked layers with per-layer affinity $\mathcal O(N^2)$1, the accumulated operator is

$\mathcal O(N^2)$2

In the homogeneous case $\mathcal O(N^2)$3, this converges to the Neumann series as $\mathcal O(N^2)$4 (Roffo et al., 26 Feb 2026).

This formulation recasts self-attention as multi-hop propagation on a learned token graph. Standard single-layer attention captures one-hop interactions, while InfSA sums the contributions of all path lengths in closed form. The cited literature explicitly frames this as a generalization of self-attention rather than a separate computational principle (Roffo, 19 Jul 2025).

3. Diffusion, centrality, and absorbing Markov chains

The graph-theoretic interpretation of InfSA treats tokens as graph nodes and the attention-derived affinity matrix as a content-adaptive token graph. One concrete construction uses

$\mathcal O(N^2)$5

A single layer

$\mathcal O(N^2)$6

is then one diffusion step, and stacking layers with discount $\mathcal O(N^2)$7 integrates all paths up to length $\mathcal O(N^2)$8 (Roffo et al., 26 Feb 2026).

Under this view, InfSA is linked to classical graph centrality. Katz centrality is

$\mathcal O(N^2)$9

PageRank arises if $N\to\infty$0 is row-stochastic and $N\to\infty$1 is the damping, and eigenvector centrality corresponds to the principal eigenvector of $N\to\infty$2. The stated consequence is that InfSA weights tokens by their global structural importance, defined by participation in many high-weight walks, rather than only by one-hop affinities (Roffo et al., 26 Feb 2026).

The same paper gives an absorbing Markov chain interpretation. With

$N\to\infty$3

the matrix

$N\to\infty$4

is the fundamental matrix of the absorbing chain. Entry $N\to\infty$5 is the expected number of visits to token $N\to\infty$6 before absorption starting from $N\to\infty$7. Summing columns or rows recovers Katz-style incoming and outgoing centrality,

$N\to\infty$8

This makes the “infinite” operator interpretable as expected random-walk visitation before absorption rather than only as an algebraic matrix inverse (Roffo et al., 26 Feb 2026).

To avoid constructing the full $N\to\infty$9 operator, Linear-InfSA approximates the principal eigenvector directly. For each head it computes

$N>L_{\max}$0

then scores keys by

$N>L_{\max}$1

and pools values as

$N>L_{\max}$2

The vector $N>L_{\max}$3 is broadcast to all positions and heads are concatenated. The stated complexity is $N>L_{\max}$4 time and $N>L_{\max}$5 extra memory per head, and the approximation is reported to match the dominant eigenvector of the quadratic operator with cosine $N>L_{\max}$6 (Roffo et al., 26 Feb 2026).

4. Infinite context with finite attention scope: ReAttention

A distinct usage of “Infinite Self-Attention” is the training-free two-stage scheme ReAttention, which is designed for length extrapolation in LLMs. Its premise is that a model based on the self-attention mechanism can support an infinite context with a finite attention scope under sufficient memory resources by decoupling candidate selection from final position-aware attention (Liu et al., 2024).

At each output step $N>L_{\max}$7, ReAttention partitions the KV cache into global, middle, and local regions. Stage I performs position-agnostic top-$N>L_{\max}$8 selection on the entire middle cache. For each head, scores are computed as

$N>L_{\max}$9

and the top-$L_{\max}$0 indices are retained. Stage II then forms a finite window by concatenating

$L_{\max}$1

applies positional embeddings only after this selection step, and runs ordinary position-aware self-attention on the resulting block of length $L_{\max}$2. In the simplest design, Stage I serves purely to pick a sparse set of candidates and all actual attended values flow through Stage II (Liu et al., 2024).

The reported per-token cost is obtained by streaming the middle cache in chunks of size $L_{\max}$3 and maintaining a running top-$L_{\max}$4 heap. Stage I then costs $L_{\max}$5, Stage II costs $L_{\max}$6, and overall per-token time is

$L_{\max}$7

with memory

$L_{\max}$8

Both are stated to be independent of the unbounded context length $L_{\max}$9 (Liu et al., 2024).

ReAttention is described as a drop-in replacement for standard self-attention. Integration consists of delaying positional embedding until after top-$W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$0 selection, storing keys and values without positional encoding, maintaining three small pointers in the KV cache, and replacing one large $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$1 with the two-stage scheme. No retraining is required. Existing FlashAttention or Triton kernels can be reused for Stage II, while Stage I uses a custom top-$W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$2 kernel plus batched GEMM (Liu et al., 2024).

Empirically, the paper reports LongBench average scores of $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$3 for LLaMA3.1-8B, $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$4 for Mistral-v0.3-7B, and $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$5 for LLaMA3.2-3B-chat, compared with full-attention scores of $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$6, $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$7, and $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$8, respectively. On L-Eval for Mistral-v0.3-7B, ReAttention gives $W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),$9 on closed-ended QA and open-summ. ROUGE, compared with $Z=W\,V$0 for full attention. On InfiniteBench retrieval at $Z=W\,V$1M, $Z=W\,V$2M, and $Z=W\,V$3M tokens, the reported results are $Z=W\,V$4 for LLaMA3.1-8B, $Z=W\,V$5 for Mistral-v0.3-7B, and $Z=W\,V$6 for LLaMA3.2-3B-chat. The same source states that ReAttention matches—or in some cases exceeds—the performance of full attention up to $Z=W\,V$7K tokens, remains robust with retrieval $Z=W\,V$8 out to millions of tokens, and enables context lengths of at least $Z=W\,V$9M, including a $A$0 expansion of LLaMA3.2-3B-chat to $A$1M in Needle-In-A-Haystack tests without further training. A Triton implementation is reported to yield $A$2 speedup and $A$3 memory savings versus naively running full attention on the selected window (Liu et al., 2024).

5. Compressive memory and Infini-attention

A third line of work studies Infini-attention as a recurrent compressed-memory extension of the Transformer. In this architecture, each layer is augmented with a small, fixed-size memory that accumulates compressed summaries of past segments while preserving local causal attention over the current segment. Per layer and per head, the mechanism uses local projections $A$4, a memory store $A$5, and a trainable scalar balance factor $A$6 that interpolates between local and memory attention (Huang et al., 29 Dec 2025).

The data flow is segmental. A full input is split into contiguous segments $A$7 of length $A$8. For segment $A$9, one computes

$k=1$0

Local attention is

$k=1$1

Memory attention retrieves from the compressed store,

$k=1$2

and fusion is performed as

$k=1$3

The current segment is then compressed into the next memory state,

$k=1$4

using a stable nonlinearity summarized as ELU$k=1$5 followed by a fixed-size buffer or truncation step (Huang et al., 29 Dec 2025).

The small-scale pretraining study uses a $k=1$6M-parameter LLaMA decoder with $k=1$7 layers, hidden size $k=1$8, FFN size $k=1$9, $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$00 attention heads, max context $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$01 tokens, vocab $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$02, untied embeddings, RMSNorm $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$03, SiLU activations, and mixed-precision bfloat16. Pretraining uses FineWeb sample-10BT with $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$04M documents and median length $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$05 tokens, sequence length $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$06, segment length $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$07, $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$08 steps, global batch $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$09, $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$10 tokens per step, and approximately $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$11M tokens total. The optimizer is AdamW with $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$12, $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$13, $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$14, weight decay $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$15, gradient clip $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$16, and learning rate $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$17 with $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$18 warmup steps followed by cosine decay to $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$19. The cited comparison notes that, without memory, gradients vanished at $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$20 and doubling to $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$21 stabilized training, whereas Infini-attention required no such adjustment (Huang et al., 29 Dec 2025).

The main empirical claim is improved long-context retrieval relative to a baseline local-attention model. After $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$22-step fine-tuning on a needle-in-a-hay-stack dataset, Infini-attention improves on the baseline by up to $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$23 percentage points at $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$24K tokens, specifically $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$25 versus $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$26 at “needle at start.” The same source states that both models degrade sharply beyond $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$27K because repeated memory compressions are lossy, that accuracy peaks around $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$28K–$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$29K and falls nearly to zero by $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$30K for deep-end targets, and that the balance factor converges to approximately $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$31 on average, with early layers skewing toward memory and later layers relying almost exclusively on local attention (Huang et al., 29 Dec 2025).

The architecture is therefore not an exact infinite-context mechanism in the sense of lossless access to arbitrarily distant tokens. Its stated advantage is that even a small memory can compensate for a $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$32M-parameter model’s limited local window, while its stated limitation is that repeated compression degrades signal over long sequences (Huang et al., 29 Dec 2025).

6. Infinite-width attention and kernel limits

A different use of “infinite attention” studies the limit in which multi-head attention behaves as a Gaussian process or Neural Tangent Kernel. For an input sequence $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$33, head $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$34 computes

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$35

forms logits

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$36

applies an entrywise nonlinearity $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$37 to obtain $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$38, and outputs

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$39

The multi-head result is

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$40

with Gaussian initialization scaled so that pre- and post-activations remain $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$41 as $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$42 (Hron et al., 2020).

The central result is that a single-head layer does not converge to a GP in the $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$43 limit, because the normalized logits remain random and the output is a non-Gaussian scale mixture. By contrast, when

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$44

the multi-head output converges in distribution to a Gaussian array as $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$45. The covariance is determined by the per-head kernel

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$46

and in the joint limit $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$47 this yields the self-attention NNGP kernel (Hron et al., 2020).

The same work derives recursive NNGP and NTK updates through deep stacks of attention layers. For a GP input with covariance $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$48, the per-head NNGP kernel is

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$49

The NTK has additional direct and indirect terms involving $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$50 and its derivative kernel. Positional encodings can be added or concatenated and folded into the kernel transform, while LayerNorm over the head dimension can likewise be incorporated (Hron et al., 2020).

The paper also reports finite-data empirical results. On CIFAR-10, an $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$51-layer Conv+ReLU stack followed by a single attention layer with structured positional encodings and identity $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$52 achieved $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$53 test accuracy for the NNGP and $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$54 for the NTK, compared with $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$55 for the previous Local Average Pooling kernel. The work introduced new features to the Neural Tangents library allowing applications of NNGP and NTK models, with and without attention, to variable-length sequences, including an example on IMDb reviews (Hron et al., 2020).

7. Comparative usage, limitations, and recurrent ambiguities

A recurring source of ambiguity is that “infinite” does not denote a single operational limit across the literature. It may refer to unbounded context length with finite attention scope, to an infinite sum over graph walks, to recurrent memory carried across segments, or to a limit in the number of heads or width (Liu et al., 2024, Roffo et al., 26 Feb 2026, Huang et al., 29 Dec 2025, Hron et al., 2020).

Usage in the cited literature	Core mechanism	Limiting object
ReAttention	Position-agnostic top-$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$56 selection before position-aware self-attention	Infinite context with finite attention scope
Pure InfSA / Linear-InfSA	Discounted Neumann series over attention matrices; principal-eigenvector approximation	Infinite-hop diffusion on a token graph
Infini-attention	Compressed memory plus local causal attention with balance factor $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$57	Recurrent access to past segments
Infinite attention (NNGP/NTK)	Multi-head attention in the $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$58 limit	Gaussian-process and tangent-kernel behavior

Several misconceptions follow from collapsing these meanings into one. ReAttention does not perform full position-aware attention over arbitrarily many tokens; it performs a position-agnostic top-$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$59 scan to select candidates and then applies ordinary position-aware attention on a fixed window (Liu et al., 2024). Pure InfSA in the spectral sense is not automatically linear-time; the linear-time method is the approximation called Linear-InfSA, whereas Pure InfSA remains quadratic per layer (Roffo et al., 26 Feb 2026). Infini-attention does not eliminate long-context degradation; the study explicitly reports that retrieval accuracy drops with repeated memory compressions and that both models degrade sharply beyond $$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.$$60K (Huang et al., 29 Dec 2025). In the kernel-limit literature, “infinite” concerns width or number of heads rather than unbounded sequence length, and the key distinction is that multi-head attention becomes Gaussian whereas single-head attention does not (Hron et al., 2020).

Taken together, these works define Infinite Self-Attention less as a single architecture than as a class of extensions in which the one-hop, finite-window interpretation of self-attention is replaced by multi-hop diffusion, sparse retrieval over unbounded history, recurrent compressed memory, or an infinite-limit kernel description. The cited papers agree on the underlying motivation: standard self-attention is constrained by quadratic scaling and by finite training-time positional structure, and each InfSA variant modifies the attention mechanism to make some aspect of that finite regime non-fundamental (Liu et al., 2024, Roffo, 19 Jul 2025, Roffo et al., 26 Feb 2026).