Attention-based Neural Tangent Kernel

• Attention-based NTK is a kernel framework that encapsulates the gradient inner product dynamics of attention networks, converging to a deterministic limit in infinite-width or infinite-head regimes.
• It employs attention mechanisms—via queries, keys, values, and positional encodings—to inherit structural and spectral properties, distinguishing Gaussian from non-Gaussian behaviors.
• Practical insights include contrasting infinite-width approximations with finite-width feature learning and revealing how empirical NTK structures can diverge from the lazy-training regime.

An attention-based Neural Tangent Kernel (NTK) is the NTK associated with a network whose representations are produced by attention mechanisms. In its empirical form, it is the gradient inner product

$$\Theta_\theta(x,x')=\nabla_\theta f_\theta(x)\cdot \nabla_\theta f_\theta(x'),$$

and in the infinite-width or infinite-head regime it may converge to a deterministic kernel that governs gradient-descent dynamics in function space. In attention architectures, the definition is formally unchanged from other deep networks, but the gradients pass through queries, keys, values, output projections, positional encodings, and normalization layers, so the kernel inherits the structural and spectral signatures of attention itself (Hron et al., 2020, Golikov et al., 2022).

1. Definition and architectural setting

In the attention literature, the term refers most directly to the NTK of an attention network trained by gradient descent, typically in an infinite-width limit. For a layer receiving a representation $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$, single-head attention is specified by

$$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$

with logits

$G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$

attention weights $A(x)=\zeta(G(x))$, and output $f(x)=A(x)V(x)$. Multi-head attention concatenates $H$ independent heads and applies an output projection,

$$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$

The paper "Infinite attention: NNGP and NTK for deep attention networks" formalizes this setting for attention layers inserted into otherwise standard dense or convolutional stacks, with i.i.d. Gaussian initialization for all $W^Q,W^K,W^V,W^O$ (Hron et al., 2020).

The same NTK formalism also applies to finite networks. The survey "Neural Tangent Kernel: A Survey" presents the empirical NTK as the object governing gradient-flow dynamics,

$$\dot f_t(x)=\hat\Theta_t(x,\vec x)\,(\vec y-f_t(\vec x)),$$

and emphasizes that for any architecture, including Transformers and self-attention, the central question is whether $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$0 remains approximately constant and converges to a deterministic limit under a suitable parameterization (Golikov et al., 2022).

2. Infinite-head and infinite-width limits

A central theoretical distinction is between single-head and multi-head attention. Under standard $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$1 scaling, single-head attention does not have an asymptotically Gaussian distribution; the limiting distribution is a scale mixture of Gaussians. This obstructs the usual Gaussian-process and NTK derivations. By contrast, multi-head attention restores Gaussianity because the independent sum over heads supplies the averaging needed for a CLT. The core result is that, as $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$2, the attention output converges to a GP,

$g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$3

with the logit field $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$4 also converging in distribution to a GP with explicitly stated covariance structure (Hron et al., 2020).

This result yields an attention-based NNGP and, with further recursion, an attention-based NTK. The same paper stresses that the infinite-head regime is the theoretically clean setting for standard $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$5 attention, while the survey places this result within the broader tensor-program program for NTK convergence in general architectures, including Transformers (Golikov et al., 2022).

A second scaling, $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$6, also yields Gaussian limits, but with very different behavior. With default initialization and independent $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$7, the dot products $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$8 collapse to zero as $g^{\ell-1}(x)\in\mathbb{R}^{S\times d_{\text{in}}}$9; softmax of zero logits is uniform, and the layer degenerates to global average pooling. Even the modification $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$0 does not recover full expressivity: Proposition 3.2 in (Hron et al., 2020) states that one cannot match convolution kernels with any fixed attention coefficients under this scaling.

The theoretical consequence is that not every large-width attention network admits a useful NTK limit. A common misconception is that width alone suffices. The attention results instead distinguish sharply between regimes: single-head $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$1 is non-Gaussian, multi-head $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$2 yields an NNGP/NTK limit, and $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$3 gives a Gaussian but often degenerate limit (Hron et al., 2020).

3. Layerwise kernel transformations, positional structure, and normalization

Attention-based NTKs can be constructed compositionally, by propagating layerwise NNGP and NTK updates. In the analytically tractable case $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$4 with $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$5, the attention layer becomes

$$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$6

and the corresponding NNGP update is

$$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$7

For the same configuration, the NTK update is

$$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$8

These formulas are obtained by splitting the NTK into direct terms from $$Q(x)=g^{\ell-1}(x)W^Q,\qquad K(x)=g^{\ell-1}(x)W^K,\qquad V(x)=g^{\ell-1}(x)W^V,$$9 and indirect terms propagated from earlier layers (Hron et al., 2020).

Positional encodings enter the kernel explicitly. For additive or appended positional embeddings, the paper introduces the interpolation operator

$G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$0

and for $G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$1, $G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$2, with positional encodings affecting $G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$3, the NNGP becomes

$G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$4

The same work proposes structured positional kernels that decay with spatial or sequence distance, such as

$G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$5

for sequences, and gives residual attention kernels that add a skip connection around the attention map (Hron et al., 2020).

Layer normalization also acts as a kernel transform. Over channels,

$G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$6

This makes the attention-based NTK a genuinely compositional object: attention, feedforward layers, residual pathways, positional encodings, and normalization each contribute explicit kernel maps that can be chained to obtain the kernel of a full stack (Hron et al., 2020).

4. Structured empirical NTKs and collapse phenomena

A distinct line of work studies not only infinite-width initialization kernels but also the structured empirical NTKs that develop during training. "Neural (Tangent Kernel) Collapse" assumes that empirical NTKs become block-structured and label-aligned: samples within the same class have stronger correlations than samples from different classes. The assumption is formalized by requiring, for a kernel $G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$7,

$G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$8

with $G(x)=\frac{Q(x)K(x)^\top}{\sqrt{d_G}},$9 (Seleznova et al., 2023).

The paper does not explicitly study Transformers, but it states that all NTK formulas hold unchanged for attention-based networks; only the structure of $A(x)=\zeta(G(x))$0 and $A(x)=\zeta(G(x))$1 changes with architecture. In that setting, $A(x)=\zeta(G(x))$2 and $A(x)=\zeta(G(x))$3 are produced by self-attention plus feedforward blocks, and gradients $A(x)=\zeta(G(x))$4 include attention parameters. This makes the block-structured assumption a tractable model of attention-aligned function-space interactions (Seleznova et al., 2023).

Under gradient flow with MSE loss, the block structure yields a three-way eigendecomposition of residual dynamics: a global mode, a class-mean mode, and a within-class mode, with corresponding eigenvalues

$A(x)=\zeta(G(x))$5

The residual evolution decouples into exponential decays at these three rates. In the terminology of the paper, this connects to local elasticity: if $A(x)=\zeta(G(x))$6 and $A(x)=\zeta(G(x))$7, cross-class influence is much smaller than within-class influence. The text explicitly notes that this is conceptually very close to attention, because the kernel acts as a fixed “attention score” in function space (Seleznova et al., 2023).

The same analysis derives decomposed feature dynamics and an invariant

$A(x)=\zeta(G(x))$8

and proves that under block-structured NTK, zero global feature mean, and $A(x)=\zeta(G(x))$9, gradient flow exhibits NC1–NC4: within-class collapse, simplex ETF geometry of class means, self-duality between weights and means, and nearest-class-center classification. For attention-based networks, the paper presents this as an analogue: once the empirical NTK stabilizes into a structured, attention-aligned form, the last phase of training behaves like a factorized dynamics in the space of last-layer representations and classifier weights (Seleznova et al., 2023).

5. Finite-width non-convergence, spectral amplification, and influence malleability

A third perspective shows that attention-based NTKs can depart sharply from the classical lazy-training picture even when an infinite-width kernel exists. "Influence Malleability in Linearized Attention: Dual Implications of Non-Convergent NTK Dynamics" studies a parameter-free, single-head linearized attention map

$f(x)=A(x)V(x)$0

which corresponds to scaled dot-product attention with identity $f(x)=A(x)V(x)$1 projections and a first-order Taylor approximation to softmax. This induces the exact data-dependent kernel

$f(x)=A(x)V(x)$2

so the attention transformation cubes the Gram spectrum (Miñoza et al., 13 Mar 2026).

If $f(x)=A(x)V(x)$3 are the eigenvalues of $f(x)=A(x)V(x)$4, then the transformed Gram matrix has eigenvalues $f(x)=A(x)V(x)$5, and

$f(x)=A(x)V(x)$6

For the associated two-layer ReLU model on top of these attention features, convergence of the finite-width NTK $f(x)=A(x)V(x)$7 to its infinite-width limit requires

$f(x)=A(x)V(x)$8

up to $f(x)=A(x)V(x)$9 factors. On real image datasets, the paper reports $H$0 for MNIST and $H$1 for CIFAR-10, implying $H$2 and $H$3, respectively. This places practical widths far outside the kernel-convergent regime (Miñoza et al., 13 Mar 2026).

The empirical evidence in that paper directly contradicts the assumption that wider attention models necessarily become more NTK-like. For 2L-ReLU, $H$4 decreases monotonically with width. For MLP-Attn, the distance increases with width: on MNIST it rises from $H$5 at $H$6 to $H$7 at $H$8, and on CIFAR-10 it rises from $H$9 to $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$0. The interpretation given is that attention remains in the feature-learning regime rather than entering the lazy or frozen-kernel regime (Miñoza et al., 13 Mar 2026).

The same paper introduces influence malleability, defined through leave-one-out influence and the sign-flip rate under adversarial perturbation of highly influential training points. In the 10-class setting, PGD yields a 3.3% flip rate for 2L-ReLU and 28.9% for MLP-Attn on MNIST, and 3.1% versus 19.1% on CIFAR-10, corresponding to approximately $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$1 and $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$2 higher malleability. The paper interprets this as a dual implication of attention’s data-dependent kernel: improved alignment with dominant data modes can reduce approximation bias, but the same sensitivity increases susceptibility to adversarial manipulation of training data (Miñoza et al., 13 Mar 2026).

6. Applications, software instantiations, and algorithmic use

Attention-based NNGP and NTK models have been instantiated directly as kernels, rather than only as asymptotic descriptions. "Infinite attention" implements these kernels in the Neural Tangents library, including support for variable-length sequences, and evaluates them by exact GP regression and infinite-width NTK regression (Hron et al., 2020).

On CIFAR-10, with convolutional stacks followed by one attention layer, the reported full-dataset accuracies are: Flatten, NNGP 65.5% and NTK 66.3%; GAP, NNGP 77.9% and NTK 77.4%; LAP, NNGP 80.4% and NTK 79.7%; Struct, NNGP 80.6% and NTK 79.9%; Residual, NNGP 80.7% and NTK 80.1%. The paper states that attention-based kernels with positional encodings achieve slightly better than prior best untrained GP kernels without data augmentation (Hron et al., 2020).

On IMDb sentiment classification with variable-length sequences and GloVe embeddings, the same work reports 85.0% for GAP-only, 85.8% for GAP-FCN, and 86.1% for Struct on the full $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$3 split with GloVe-840B 300d embeddings. It also notes that on lower-quality 50d GloVe embeddings the gap between kernels becomes much larger, with CNN-GAP and Struct substantially outperforming GAP-only, indicating that the inductive bias of attention and convolution helps more when embeddings are weak (Hron et al., 2020).

A broader algorithmic use of NTKs appears in "Meta-Learning with Neural Tangent Kernels", which defines meta-learning in the RKHS induced by a model’s NTK and gives two adaptation mechanisms: a fast-adaptive regularizer in RKHS and a closed-form kernel adaptation. The core formula for analytic adaptation is

$$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$4

with the $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$5 limit giving kernel regression. That paper analyzes fully connected and convolutional models rather than attention explicitly, but the formulation is architecture-agnostic. This suggests that, once an attention-based NTK is available, the same RKHS machinery can be used for transformer-style meta-learning without differentiating through an inner-loop optimizer (Zhou et al., 2021).

7. Limitations, misconceptions, and open problems

The main limitation of attention-based NTK analysis is that the existence of an infinite-width kernel does not imply that practical attention models are well approximated by fixed-kernel dynamics. The survey emphasizes that NTK theory describes the lazy regime, whereas real transformers undergo substantial feature learning; finite heads, finite widths, and depth can produce significant deviations from the constant-kernel approximation (Golikov et al., 2022).

For attention specifically, several unresolved issues are explicit in the literature. First, softmax attention under $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$6 leads to intractable Gaussian integrals in the analytic kernel formulas, so the tractable theory in (Hron et al., 2020) often uses $$f^\ell_n(x)=\bigl[f^1_n(x),\dots,f^H_n(x)\bigr]W^O.$$7 rather than softmax. Second, the interplay between finite-head, finite-width transformers and their infinite-head GP/NTK limits is not fully understood. Third, only shallow attention stacks are explored empirically in that work, leaving deeper and more realistic transformer configurations open (Hron et al., 2020).

A second misconception is that attention-based NTKs are necessarily label-agnostic objects with no bearing on learned geometry. The block-structured empirical-kernel analysis in (Seleznova et al., 2023) shows the opposite at the level of training dynamics: once the kernel aligns with class structure, its eigenmodes and invariants can explain the emergence of Neural Collapse. Conversely, (Miñoza et al., 13 Mar 2026) shows that data-dependent attention kernels may move so far from the lazy regime that the infinite-width NTK ceases to predict finite-width behavior. Taken together, these results indicate that attention-based NTK analysis is most informative when used comparatively: as a description of asymptotic function-space dynamics, as a diagnostic of empirical kernel structure, and as a reference point for understanding when attention is behaving like a kernel method and when it is not (Seleznova et al., 2023, Miñoza et al., 13 Mar 2026).