Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

Abstract: Linear attention promises constant-time recurrent inference but degrades sharply on associative recall. We formulate attention recall as a spherical-packing problem and introduce Kernelized Linear Attention Activations (KATA), a framework whose feature maps are derived from first principles by certifying nonnegative attention weights through a self-dual homogeneous cone. Building on this observation, we show that rank-one positive semi-definite (PSD) features offer a favorable capacity--interference tradeoff. KATA recovers a parameter-free convex output gate and characterizes associative capacity through the Welch interference floor. For tolerances above this floor, KATA enlarges the state without adding parameters and admits spherical codes with exponentially many keys in the projection dimension. We implement KATA as fused Triton kernels at two operating points: a flash-attention-style forward up to ${\sim}1.6\times$ FlashAttention-2 throughput, and an exact $O(T)$ chunked-state form that reaches ${\sim}11\times$ FlashAttention-2 forward throughput at $131$k tokens. An associative scan of the first-order feature lowers the inter-chunk recurrence depth to $O(\log(T/C))$ for chunk size $C$ and averages ${\sim}2.4\times$ the throughput of a matched sequential linear-attention baseline. On long-range MQAR and repeated-key overwrite, several KATA variants outperform Gated DeltaNet, with parameter counts and state sizes reported alongside accuracy. Induction preserves near-perfect recall, while kernel benchmarks show that the maps can be implemented efficiently. KATA retains $0.985$ MQAR at a $16\times$ out-of-distribution length, approaching the softmax with roughly one quarter of the KV-cache entries. Experiments on 340M-parameter LLMs reveal a feature-dependent fluency trade-off and clarify how positional embeddings, delta rules, and decay gates interact with feature geometry.

• The paper introduces KATA, a kernelized linear-attention framework that uses rank-one positive-semidefinite feature maps and symmetric-cone geometry to reduce retrieval interference and overcome the linear capacity limits of orthant and Lorentz features.
• The paper shows that PSD features can provide exponentially larger near-orthogonal key dictionaries in theory, while KATA-M1 achieves 0.985 MQAR accuracy and DeltaKATA-M1 reaches 0.999 at 16× context extrapolation.
• The paper delivers practical fused Triton implementations, with the exact PSD linear-state kernel reaching roughly 11× FlashAttention-2 forward throughput at 131K tokens, but reports trade-offs in perplexity, learnability, and out-of-distribution length generalization.

Linear attention replaces the exponential kernel of softmax attention with a finite inner product of feature maps, collapsing quadratic token mixing into a fixed-size recurrent state. The cost of this substitution is memory collision: many key–value bindings must share one state, and retrieval degrades on associative recall at long contexts. The paper under review, "Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones" (2607.17419), develops a principled response. It formulates associative recall as a spherical-packing problem over the feature geometry, uses the Koecher–Vinberg classification of symmetric cones to organize admissible nonnegative feature maps, and shows that rank-one positive-semidefinite (PSD) features offer a favorable capacity–interference tradeoff. The resulting framework, KATA (Kernelized Linear Attention Activations), is implemented as fused Triton kernels and evaluated on synthetic recall tasks and 340M-parameter language-model pretraining.

Cone classification of admissible feature maps

The paper works with normalized linear attention, in which softmax's kernel $\exp(\langle q,k\rangle)$ is replaced by $\langle\psi(q),\psi(k)\rangle$ for a finite feature map $\psi$, yielding an additive recurrence on a state pair of size $n_\psi(d_v+1)$. Three assumptions structure the design space: the feature image lies in a self-dual cone (guaranteeing nonnegative kernels), the map is equivariant under input isometries, and the cone is homogeneous. Under these assumptions the Koecher–Vinberg theorem applies, and the admissible real geometries reduce to products of three irreducible symmetric cones: the positive orthant $\mathbb{R}^d_+$, the Lorentz cone $\mathcal{Q}^m_+$, and the PSD cone $\mathbb{S}^m_+$. The paper restricts itself to these ordinary real factors, explicitly leaving complex Hermitian, quaternionic, and exceptional factors open.

A structural byproduct of this formulation is that the denominator of normalized linear attention is itself a token-conditioned convex output gate: the readout at time $t$ is a convex interpolation between the previous readout and the current value, with weights $D_{t-1}/D_t$ and $c_t/D_t$ that require no learned parameters. The paper contrasts this readout-level gate with the recurrence-level gates of Gated Linear Attention and Gated DeltaNet, which modify the stored state through noncommutative rank-one transitions. The two control points are distinct, and the experiments exploit both.

Capacity as spherical packing

The central theoretical move is to define the $\langle\psi(q),\psi(k)\rangle$0-capacity of a cone: the largest number $\langle\psi(q),\psi(k)\rangle$1 of unit feature vectors in the cone whose maximum pairwise inner product (the interference $\langle\psi(q),\psi(k)\rangle$2) is at most $\langle\psi(q),\psi(k)\rangle$3. This is exactly the mutual coherence of the feature dictionary. The paper then derives per-cone ceilings. For the orthant, the $\langle\psi(q),\psi(k)\rangle$4 coordinate rays are exactly orthogonal, but beyond $\langle\psi(q),\psi(k)\rangle$5 the Welch bound applies, so capacity is linear in dimension. For the Lorentz cone, the lift $\langle\psi(q),\psi(k)\rangle$6 maps raw inner product $\langle\psi(q),\psi(k)\rangle$7 to $\langle\psi(q),\psi(k)\rangle$8; requiring interference below $\langle\psi(q),\psi(k)\rangle$9 forces the raw keys to be mutually obtuse, and Rankin's bound yields the "Lorentz Rankin wall" $\psi$0 — a hard linear ceiling. This is a strong negative result: the Lorentz lift of the raw-key substrate used by DeltaNet and Gated DeltaNet cannot exceed $\psi$1 distinguishable addresses.

The PSD cone is where the capacity wall breaks. The rank-one lift $\psi$2 squares the raw inner product, so PSD interference at most $\psi$3 is equivalent to raw correlations bounded by $\psi$4. A greedy spherical-cap argument then shows that once $\psi$5, the PSD cone admits $\psi$6 near-orthogonal keys in $\psi$7 feature coordinates — exponentially many in $\psi$8. The paper complements this worst-case bound with deterministic constructions: realified mutually unbiased bases give $\psi$9 vectors in $n_\psi(d_v+1)$0 with interference $n_\psi(d_v+1)$1, and DeVore polynomial frames give sparse dictionaries of size $n_\psi(d_v+1)$2 in $n_\psi(d_v+1)$3 with interference $n_\psi(d_v+1)$4. Random-projection (Johnson–Lindenstrauss) guarantees are noted but observed to be loose at per-head dimensions.

A power signal-to-noise (pSNR) analysis connects packing to retrieval under isotropic values. At the Welch-scale interference, linear features give $n_\psi(d_v+1)$5, ideal retrieval only for $n_\psi(d_v+1)$6; the PSD map squares distractor interference and extends the threshold to $n_\psi(d_v+1)$7. Applying the same criterion to temperature-$n_\psi(d_v+1)$8 softmax yields an exponential capacity $n_\psi(d_v+1)$9, and the required inverse temperature scales as $\mathbb{R}^d_+$0 — recovering, from a retrieval perspective, the logarithmic attention-scaling law identified for long-context softmax dynamics. The paper is careful to state that these are idealized criteria under an isotropic-value model, and that exact recall also depends on realized values.

Hardware-aligned kernels

Theory is matched to an implementation story. The additive feature-state recurrence admits a chunkwise forward in which every inter-chunk operation is a GEMM, elementwise op, or warp reduction — no triangular solve, unlike Gated DeltaNet's WY representation. Moreover, the inter-chunk prefix sum is associative, so the paper replaces the standard sequential scan (depth $\mathbb{R}^d_+$1) with a parallel tree scan of depth $\mathbb{R}^d_+$2 across three Triton kernels. For the linear feature, this delivers 1.9–2.9× the throughput of a matched sequential chunk kernel from 2k to 131k tokens, and up to 7.5× Gated DeltaNet at $\mathbb{R}^d_+$3, with the advantage concentrated at small batch where a sequential scan cannot fill the H100's SMs.

The PSD recurrence cannot be tree-scanned directly: the $\mathbb{R}^d_+$4 local state tile exceeds shared-memory budgets, so the paper provides two operating points. The quadratic flash-attention-style forward reaches 1.3–1.6× FlashAttention-2 throughput (KATA-M1) without ever forming the state; the exact linear-state form reaches parity with its quadratic counterpart near 16k tokens and achieves roughly 11× FlashAttention-2 forward throughput at 131k tokens for KATA-M2. To fit hardware, two reduced PSD families are introduced: KATA-$\mathbb{R}^d_+$5 sums $\mathbb{R}^d_+$6 block outer products into one PSD factor, while KATA-M$\mathbb{R}^d_+$7 retains one PSD factor per block, with state sizes scaling accordingly.

Synthetic recall results

On the Zoology MQAR benchmark with length extrapolation to 16× the training context, the orthant and Lorentz baselines collapse just past $\mathbb{R}^d_+$8, exactly where the linear Welch and Rankin ceilings predict, while the PSD variants extend recall deep into the extrapolation regime. KATA-M1 retains 0.985 accuracy at the largest slice with a 266,240-entry fixed state — roughly one quarter of softmax's KV-cache entries at that length — and DeltaKATA-M1 reaches 0.999. These are the paper's strongest empirical claims, and they directly instantiate the packing theory.

On repeated-key overwrite, the additive state superimposes both values at the same address, and the results cleanly disentangle the three recurrence mechanisms. RoPE resolves overwrite in-distribution but extrapolates poorly. A learned multiplicative scalar gate raises KATA-M2's overwrite accuracy from 0.521 to 0.991 at $\mathbb{R}^d_+$9, supporting a positional interpretation of the gate (it assigns each write an age-dependent weight) with state-norm control secondary. The delta rule, which erases the value at the current content address, is complementary: DeltaKATA-M1 is nearly perfect across the overwrite sweep. The trade-off is explicit — GatedDeltaKATA-M2 reaches 0.960 average overwrite accuracy but its MQAR average falls to 0.423.

Pretraining and needle recall

The paper pretrains matched 340M-parameter models for 15B tokens of SlimPajama, swapping only the token-mixing layer. Zero-shot accuracies are broadly comparable, but perplexity shows a meaningful spread: KATA-M1 trails softmax, Gated DeltaNet, and DeltaKATA-M1 (43.5 vs. 38.6 LAMBADA perplexity), and KATA-M2 is higher still (51.8). The paper concedes this fluency trade-off rather than burying it.

On RULER single-needle tasks at in-window lengths, the entropy ladder is diagnostic. Softmax sits at the ceiling (≥0.93 everywhere). Gated DeltaNet degrades sharply with needle entropy — word, number, and UUID recall fall 0.996 → 0.384 → 0.004 at 1K — consistent with its Lorentz Rankin wall on the raw-key substrate. The PSD variants retain much more high-entropy signal: KATA-M2 reaches 0.910 and 0.616 UUID recall at 1K and 2K with half of KATA-M1's state. Notably, the paper reports a reversal of its own geometric prediction: KATA-M1 has the larger packing budget, yet KATA-M2 learns better. The authors attribute this to learnability — two 32-dimensional factors train more easily than one 64-dimensional factor — an honest concession that realized recall depends on optimization, not geometry alone. The delta erase again helps: DeltaKATA-M1 raises UUID recall from 0.758 to 0.908 at 1K.

On the Based cloze suite the picture is more qualified. The paper itself flags that normalized substring matching rewards verbosity: average generation length on SQuAD spans an 18.6× range across models, so the SQuAD/FDA ordering reflects generation style as much as retrieval. The authors accordingly treat NIAH as the primary capacity result. Length extrapolation shows yet another ordering: Gated DeltaNet's multiplicative gate generalizes best beyond the 2048-token training horizon, while RoPE-based softmax and KATA models collapse — strong OOD length generalization and hard-needle recall are shown to be complementary properties.

Limitations and open questions

The paper states its scope and gaps plainly. The capacity theory covers only the real symmetric cones; complex extensions, where equiangular tight frames allow $\mathcal{Q}^m_+$0, are deferred. The pSNR criterion is idealized, assuming isotropic values and a fixed interference scale. The tree scan is not implemented for PSD features, and realizing the parallel-prefix speedup there would require feature blocking within the scan. GatedKATA and GatedDeltaKATA are not systematically pretrained, DeltaKATA is evaluated only at $\mathcal{Q}^m_+$1, and the learnability of the full KATA-M1 geometry beyond $\mathcal{Q}^m_+$2 is untested. Whether the geometric gains persist at higher feature order, larger scale, and heavier contextual demand remains open; tensor-train factorizations are proposed as a route to higher even-order features without materializing the full tensor state, but enforcing nonnegativity under such truncations is unresolved.

Conclusion

The paper contributes a coherent arc from first principles to hardware: nonnegative attention weights certified by a self-dual homogeneous cone reduce the feature-design space to the classical symmetric cones; casting recall as spherical packing yields per-cone capacity laws, including a hard Rankin wall for the Lorentz substrate underlying current delta-rule architectures and exponential capacity for rank-one PSD features at fixed interference; and fused Triton kernels make the resulting maps practical, with the linear-state PSD kernel reaching approximately 11× FlashAttention-2 forward throughput at 131k tokens. The empirical results validate the capacity predictions on MQAR and needle recall while honestly documenting where learnability, fluency, and length extrapolation diverge from pure geometry. The main open question the work leaves is whether the packing-theoretic advantage of PSD features survives at scale once optimization and contextual language demands are fully accounted for.