---
title: Infinite Self-Attention (InfSA)
url: https://www.emergentmind.com/topics/infinite-self-attention-infsa
type: topic
---

# Infinite Self-Attention (InfSA)

Infinite Self-Attention (InfSA) denotes several closely related extensions of self-attention that relax different finite constraints of the standard Transformer. In the cited literature, the label is used for a training-free scheme that supports an infinite context with a finite attention scope in large language models, a Neumann-series operator that accumulates multi-hop interactions on a token graph, a compressed-memory mechanism studied under the name Infini-attention, and an infinite-width analysis in which multi-head attention converges to NNGP and NTK limits [2407.15176], [2603.00175], [2512.23862], [2006.10540], [2507.14560]. Across these usages, the common theme is that standard one-hop, finite-window attention is replaced by a mechanism that either aggregates over arbitrarily long histories, sums paths of every length, or admits an infinite limit with a tractable kernel description.

## 1. Standard self-attention and the finite-regime bottleneck

Standard position-aware self-attention for a sequence of length \(N\) is written as
\[
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d+\mathrm{Bias}\bigr)\,V.
\]
In the long-context setting, this requires \(\mathcal O(N^2)\) time and memory and forces all key-value pairs to be stored in a single window. As \(N\to\infty\), this becomes infeasible both physically, because memory blows up, and statistically, because rotary or absolute position embeddings fall outside the training-time range, causing length-extrapolation failure and entropy explosion in the softmax [2407.15176].

Length extrapolation is the regime \(N>L_{\max}\), where a Transformer trained up to a maximum context \(L_{\max}\) degrades at test time. The cited description characterizes this degradation by rising perplexity, vanishing retrieval accuracy, and collapse of generation quality once absolute-position embeddings go out of distribution or attention entropy saturates [2407.15176]. In high-resolution vision, an analogous constraint appears as the quadratic cost of softmax attention, which limits Transformer scalability [2603.00175].

A broader affinity-matrix view places standard self-attention inside a larger family of multi-hop propagation operators. Given a learned attention matrix
\[
W=\mathrm{softmax}\!\bigl(QK^\top/\sqrt d\bigr),
\]
the Transformer output
\[
Z=W\,V
\]
is the one-hop case of a more general series over powers of an affinity matrix \(A\). In this view, self-attention is a single-hop instance of a construction rooted in Infinite Feature Selection, while Infinite Self-Attention arises when all hop lengths are included rather than only \(k=1\) [2507.14560].

## 2. Neumann-series formulations and infinite-hop propagation

A central mathematical formulation of InfSA is the Neumann series
\[
S(\lambda)=\sum_{k=0}^{\infty}\lambda^kA^k=(I-\lambda A)^{-1},
\qquad \lambda<\frac{1}{\rho(A)},
\]
where \(A\) is a nonnegative affinity matrix and \(\rho(A)\) is its spectral radius. Discarding the zero-hop term yields
\[
S'(\lambda)=\sum_{k=1}^{\infty}\lambda^kA^k=(I-\lambda A)^{-1}-I.
\]
The \(k\)-th power \(A^k\) encodes all \(k\)-hop walks between pairs of nodes, so the series integrates information from paths of every length [2507.14560].

In the spectral formulation introduced for vision Transformers, the usual attention matrix is
\[
A=\mathrm{softmax}\bigl(QK^\top/\sqrt{d_h}\bigr)\in\mathbb R^{N\times N},
\]
and a discount factor \(\gamma\in(0,1/\rho(A))\) is used to define the infinite-path kernel
\[
\sum_{k=0}^\infty \gamma^kA^k=(I-\gamma A)^{-1},
\]
with path-integral operator
\[
S=\sum_{k=1}^{\infty}\gamma^kA^k=(I-\gamma A)^{-1}-I.
\]
For stacked layers with per-layer affinity \(A^{(l)}\), the accumulated operator is
\[
S_L=\sum_{l=1}^L \gamma^l\,\Bigl(A^{(l)}A^{(l-1)}\cdots A^{(1)}\Bigr)\,X^{(0)}.
\]
In the homogeneous case \(A^{(l)}\equiv A\), this converges to the Neumann series as \(L\to\infty\) [2603.00175].

This formulation recasts self-attention as multi-hop propagation on a learned token graph. Standard single-layer attention captures one-hop interactions, while InfSA sums the contributions of all path lengths in closed form. The cited literature explicitly frames this as a generalization of self-attention rather than a separate computational principle [2507.14560].

## 3. Diffusion, centrality, and absorbing Markov chains

The graph-theoretic interpretation of InfSA treats tokens as graph nodes and the attention-derived affinity matrix as a content-adaptive token graph. One concrete construction uses
\[
\hat A=\mathrm{ReLU}(QK^\top)/\|\mathrm{ReLU}(QK^\top)\|_F.
\]
A single layer
\[
Z=\hat A\,V
\]
is then one diffusion step, and stacking layers with discount \(\gamma\) integrates all paths up to length \(L\) [2603.00175].

Under this view, InfSA is linked to classical graph centrality. Katz centrality is
\[
\sum_{k=1}^\infty \gamma^kA^k\,\mathbf 1,
\]
PageRank arises if \(A\) is row-stochastic and \(\gamma\) is the damping, and eigenvector centrality corresponds to the principal eigenvector of \(A\). The stated consequence is that InfSA weights tokens by their global structural importance, defined by participation in many high-weight walks, rather than only by one-hop affinities [2603.00175].

The same paper gives an absorbing Markov chain interpretation. With
\[
M=\gamma\,\hat A,\qquad
R_i=1-\sum_j M_{ij},\qquad
P=\begin{pmatrix}M & R\\ 0 & 1\end{pmatrix},
\]
the matrix
\[
N=(I-M)^{-1}=(I-\gamma\hat A)^{-1}
\]
is the fundamental matrix of the absorbing chain. Entry \(N_{ij}\) is the expected number of visits to token \(j\) before absorption starting from \(i\). Summing columns or rows recovers Katz-style incoming and outgoing centrality,
\[
c^{\mathrm{in}}_j=\sum_i N_{ij},\qquad
c^{\mathrm{out}}_i=\sum_j N_{ij}.
\]
This makes the “infinite” operator interpretable as expected random-walk visitation before absorption rather than only as an algebraic matrix inverse [2603.00175].

To avoid constructing the full \(N\times N\) operator, Linear-InfSA approximates the principal eigenvector directly. For each head it computes
\[
e_i=\|Q_i\|_2,\qquad
\alpha_i=\frac{e_i}{\sum_j e_j+\varepsilon},\qquad
\bar q=\sum_i \alpha_iQ_i,
\]
then scores keys by
\[
s_j=\mathrm{ReLU}(\bar q^\top K_j),\qquad
a_j=\frac{s_j}{\sum_\ell s_\ell+\varepsilon},
\]
and pools values as
\[
h=\gamma\sum_j a_jV_j.
\]
The vector \(h\) is broadcast to all positions and heads are concatenated. The stated complexity is \(\mathcal O(Nd_h)\) time and \(\mathcal O(d_h)\) extra memory per head, and the approximation is reported to match the dominant eigenvector of the quadratic operator with cosine \(0.985\) [2603.00175].

## 4. Infinite context with finite attention scope: ReAttention

A distinct usage of “Infinite Self-Attention” is the training-free two-stage scheme ReAttention, which is designed for length extrapolation in large language models. Its premise is that a model based on the self-attention mechanism can support an infinite context with a finite attention scope under sufficient memory resources by decoupling candidate selection from final position-aware attention [2407.15176].

At each output step \(t\), ReAttention partitions the KV cache into global, middle, and local regions. Stage I performs position-agnostic top-\(k\) selection on the entire middle cache. For each head, scores are computed as
\[
s^{(h)}=\frac{q_t^{(h)}\,\mathcal K_{\mathrm{middle}}^\top}{\sqrt d},
\]
and the top-\(k\) indices are retained. Stage II then forms a finite window by concatenating
\[
[\mathcal K_{\mathrm{global}},\mathcal K_{\mathrm{select}},\mathcal K_{\mathrm{local}}],
\qquad
[\mathcal V_{\mathrm{global}},\mathcal V_{\mathrm{select}},\mathcal V_{\mathrm{local}}],
\]
applies positional embeddings only after this selection step, and runs ordinary position-aware self-attention on the resulting block of length \(w=g+k'+\ell\). In the simplest design, Stage I serves purely to pick a sparse set of candidates and all actual attended values flow through Stage II [2407.15176].

The reported per-token cost is obtained by streaming the middle cache in chunks of size \(c\) and maintaining a running top-\(k\) heap. Stage I then costs \(\mathcal O(d\cdot c+k'\log k')\), Stage II costs \(\mathcal O(d\cdot w)\), and overall per-token time is
\[
\mathcal O(d(c+w)+k'\log k')
\]
with memory
\[
\mathcal O(d(g+\ell+k')).
\]
Both are stated to be independent of the unbounded context length \(N\to\infty\) [2407.15176].

ReAttention is described as a drop-in replacement for standard self-attention. Integration consists of delaying positional embedding until after top-\(k\) selection, storing keys and values without positional encoding, maintaining three small pointers in the KV cache, and replacing one large \(\mathrm{softmax}(QK^\top)V\) with the two-stage scheme. No retraining is required. Existing FlashAttention or Triton kernels can be reused for Stage II, while Stage I uses a custom top-\(k\) kernel plus batched GEMM [2407.15176].

Empirically, the paper reports LongBench average scores of \(35.1\) for LLaMA3.1-8B, \(35.5\) for Mistral-v0.3-7B, and \(30.4\) for LLaMA3.2-3B-chat, compared with full-attention scores of \(35.2\), \(35.6\), and \(30.5\), respectively. On L-Eval for Mistral-v0.3-7B, ReAttention gives \(41.9/4.68\) on closed-ended QA and open-summ. ROUGE, compared with \(41.9/4.98\) for full attention. On InfiniteBench retrieval at \(1\)M, \(2\)M, and \(4\)M tokens, the reported results are \(92\%/89\%/85\%\) for LLaMA3.1-8B, \(94\%/92\%/90\%\) for Mistral-v0.3-7B, and \(90\%/88\%/84\%\) for LLaMA3.2-3B-chat. The same source states that ReAttention matches—or in some cases exceeds—the performance of full attention up to \(400\)K tokens, remains robust with retrieval \(>80\%\) out to millions of tokens, and enables context lengths of at least \(1\)M, including a \(128\times\) expansion of LLaMA3.2-3B-chat to \(4\)M in Needle-In-A-Haystack tests without further training. A Triton implementation is reported to yield \(1.5\times\) speedup and \(2\times\) memory savings versus naively running full attention on the selected window [2407.15176].

## 5. Compressive memory and Infini-attention

A third line of work studies Infini-attention as a recurrent compressed-memory extension of the Transformer. In this architecture, each layer is augmented with a small, fixed-size memory that accumulates compressed summaries of past segments while preserving local causal attention over the current segment. Per layer and per head, the mechanism uses local projections \(Q_\ell,K_\ell,V_\ell\), a memory store \(M_{t-1}\), and a trainable scalar balance factor \(\alpha\in[0,1]\) that interpolates between local and memory attention [2512.23862].

The data flow is segmental. A full input is split into contiguous segments \(X_1,\dots,X_n\) of length \(S\). For segment \(X_t\), one computes
\[
Q=X_tW_Q,\qquad K=X_tW_K,\qquad V=X_tW_V.
\]
Local attention is
\[
A_{\mathrm{local}}=\mathrm{softmax}(QK^\top/\sqrt{d_k})\,V.
\]
Memory attention retrieves from the compressed store,
\[
A_{\mathrm{mem}}=\mathrm{softmax}(QK'^\top/\sqrt{d_k}+\mathrm{bias})\,N(V'),
\]
and fusion is performed as
\[
A=\alpha\cdot A_{\mathrm{mem}}+(1-\alpha)\cdot A_{\mathrm{local}},
\qquad
\alpha=\mathrm{hard\_sigmoid}(w_\alpha).
\]
The current segment is then compressed into the next memory state,
\[
M_t=\mathrm{Compress}(M_{t-1},[K_t,V_t]),
\]
using a stable nonlinearity summarized as ELU\(+1\) followed by a fixed-size buffer or truncation step [2512.23862].

The small-scale pretraining study uses a \(300\)M-parameter LLaMA decoder with \(12\) layers, hidden size \(1{,}024\), FFN size \(4{,}096\), \(8\) attention heads, max context \(8{,}192\) tokens, vocab \(49{,}152\), untied embeddings, RMSNorm \(\varepsilon=1\mathrm e{-5}\), SiLU activations, and mixed-precision bfloat16. Pretraining uses FineWeb sample-10BT with \(14.9\)M documents and median length \(418\) tokens, sequence length \(8{,}192\), segment length \(1{,}024\), \(30{,}000\) steps, global batch \(4\), \(32{,}768\) tokens per step, and approximately \(983\)M tokens total. The optimizer is AdamW with \(\beta_1=0.9\), \(\beta_2=0.95\), \(\varepsilon=1\mathrm e{-8}\), weight decay \(0.1\), gradient clip \(1.0\), and learning rate \(6\mathrm e{-5}\) with \(500\) warmup steps followed by cosine decay to \(6\mathrm e{-6}\). The cited comparison notes that, without memory, gradients vanished at \(6\mathrm e{-5}\) and doubling to \(1.2\mathrm e{-4}\) stabilized training, whereas Infini-attention required no such adjustment [2512.23862].

The main empirical claim is improved long-context retrieval relative to a baseline local-attention model. After \(500\)-step fine-tuning on a needle-in-a-hay-stack dataset, Infini-attention improves on the baseline by up to \(+31\) percentage points at \(16\)K tokens, specifically \(45\%\) versus \(14\%\) at “needle at start.” The same source states that both models degrade sharply beyond \(8\)K because repeated memory compressions are lossy, that accuracy peaks around \(4\)K–\(8\)K and falls nearly to zero by \(16\)K for deep-end targets, and that the balance factor converges to approximately \(0.30\) on average, with early layers skewing toward memory and later layers relying almost exclusively on local attention [2512.23862].

The architecture is therefore not an exact infinite-context mechanism in the sense of lossless access to arbitrarily distant tokens. Its stated advantage is that even a small memory can compensate for a \(300\)M-parameter model’s limited local window, while its stated limitation is that repeated compression degrades signal over long sequences [2512.23862].

## 6. Infinite-width attention and kernel limits

A different use of “infinite attention” studies the limit in which multi-head attention behaves as a Gaussian process or Neural Tangent Kernel. For an input sequence \(x\in\mathbb R^{S\times d'}\), head \(h\) computes
\[
Q^h(x)=xW^{Q,h},\qquad K^h(x)=xW^{K,h},\qquad V^h(x)=xW^{V,h},
\]
forms logits
\[
L^h(x)=\frac{1}{\sqrt d}Q^h(x)K^h(x)^\top,
\]
applies an entrywise nonlinearity \(\zeta\) to obtain \(G^h(x)=\zeta(L^h(x))\), and outputs
\[
f^h(x)=G^h(x)V^h(x).
\]
The multi-head result is
\[
f(x)=\bigl[f^1(x),f^2(x),\dots,f^H(x)\bigr]W^O,
\]
with Gaussian initialization scaled so that pre- and post-activations remain \(O(1)\) as \(d\to\infty\) [2006.10540].

The central result is that a single-head layer does not converge to a GP in the \(d\to\infty\) limit, because the normalized logits remain random and the output is a non-Gaussian scale mixture. By contrast, when
\[
f(x)=\frac1{\sqrt H}\sum_{h=1}^H f^h(x),
\]
the multi-head output converges in distribution to a Gaussian array as \(H\to\infty\). The covariance is determined by the per-head kernel
\[
K^{\rm head}_{ab}(x,x')=\mathbb E\bigl[f^1_{a1}(x)f^1_{b1}(x')\bigr],
\]
and in the joint limit \(\min\{d,H\}\to\infty\) this yields the self-attention NNGP kernel [2006.10540].

The same work derives recursive NNGP and NTK updates through deep stacks of attention layers. For a GP input with covariance \(\Sigma\), the per-head NNGP kernel is
\[
K^{\rm head}_{ab}(x,x')
=\sum_{i,j=1}^S \Sigma_{ij}^{xx'}\,
\mathbb E\bigl[(G^h)_{ai}(x)(G^h)_{bj}(x')\bigr].
\]
The NTK has additional direct and indirect terms involving \(\Sigma\) and its derivative kernel. Positional encodings can be added or concatenated and folded into the kernel transform, while LayerNorm over the head dimension can likewise be incorporated [2006.10540].

The paper also reports finite-data empirical results. On CIFAR-10, an \(8\)-layer Conv+ReLU stack followed by a single attention layer with structured positional encodings and identity \(\zeta\) achieved \(80.7\%\) test accuracy for the NNGP and \(80.1\%\) for the NTK, compared with \(80.4\%\) for the previous Local Average Pooling kernel. The work introduced new features to the Neural Tangents library allowing applications of NNGP and NTK models, with and without attention, to variable-length sequences, including an example on IMDb reviews [2006.10540].

## 7. Comparative usage, limitations, and recurrent ambiguities

A recurring source of ambiguity is that “infinite” does not denote a single operational limit across the literature. It may refer to unbounded context length with finite attention scope, to an infinite sum over graph walks, to recurrent memory carried across segments, or to a limit in the number of heads or width [2407.15176], [2603.00175], [2512.23862], [2006.10540].

| Usage in the cited literature | Core mechanism | Limiting object |
|---|---|---|
| ReAttention | Position-agnostic top-\(k\) selection before position-aware self-attention | Infinite context with finite attention scope |
| Pure InfSA / Linear-InfSA | Discounted Neumann series over attention matrices; principal-eigenvector approximation | Infinite-hop diffusion on a token graph |
| Infini-attention | Compressed memory plus local causal attention with balance factor \(\alpha\) | Recurrent access to past segments |
| Infinite attention (NNGP/NTK) | Multi-head attention in the \(H\to\infty\) limit | Gaussian-process and tangent-kernel behavior |

Several misconceptions follow from collapsing these meanings into one. ReAttention does not perform full position-aware attention over arbitrarily many tokens; it performs a position-agnostic top-\(k\) scan to select candidates and then applies ordinary position-aware attention on a fixed window [2407.15176]. Pure InfSA in the spectral sense is not automatically linear-time; the linear-time method is the approximation called Linear-InfSA, whereas Pure InfSA remains quadratic per layer [2603.00175]. Infini-attention does not eliminate long-context degradation; the study explicitly reports that retrieval accuracy drops with repeated memory compressions and that both models degrade sharply beyond \(8\)K [2512.23862]. In the kernel-limit literature, “infinite” concerns width or number of heads rather than unbounded sequence length, and the key distinction is that multi-head attention becomes Gaussian whereas single-head attention does not [2006.10540].

Taken together, these works define Infinite Self-Attention less as a single architecture than as a class of extensions in which the one-hop, finite-window interpretation of self-attention is replaced by multi-hop diffusion, sparse retrieval over unbounded history, recurrent compressed memory, or an infinite-limit kernel description. The cited papers agree on the underlying motivation: standard self-attention is constrained by quadratic scaling and by finite training-time positional structure, and each InfSA variant modifies the attention mechanism to make some aspect of that finite regime non-fundamental [2407.15176], [2507.14560], [2603.00175].

Source: https://www.emergentmind.com/topics/infinite-self-attention-infsa