Kimi Delta Attention: Delta‐Rule Linear Mechanism

• Kimi Delta Attention is a delta-rule based linear attention mechanism that refines decay using channel-wise forgetting, enabling fine-grained memory updates.
• It integrates into hybrid models like Kimi Linear by leveraging a specialized DPLR recurrence and efficient chunkwise WY algorithms to reduce KV cache usage.
• Empirical studies demonstrate KDA’s competitive long-context performance and scalability, establishing it as a key baseline for delta-rule linear attention research.

Searching arXiv for papers on Kimi Delta Attention and related delta-rule linear attention. Kimi Delta Attention (KDA) is a delta-rule-based linear attention mechanism introduced as the core recurrent component of Kimi Linear, a hybrid linear attention architecture that combines KDA with Multi-Head Latent Attention (MLA) (Team et al., 30 Oct 2025). In its defining form, KDA maintains a matrix-valued associative memory and updates it by a delta rule with a channel-wise forget gate, replacing the scalar forgetting used in Gated DeltaNet with a diagonal gate over key dimensions (Team et al., 30 Oct 2025). In the Kimi Linear formulation, the single-head recurrence is

$$\mathbf{S}_t = \left(\mathbf{I}-\beta_t\bm{k}_{t}\bm{k}_{t}^{\top}\right) \operatorname{Diag}\left(\bm{\alpha}_t \right)\mathbf{S}_{t-1} + \beta_t\bm{k}_{t}\bm{v}_{t}^{\top}, \qquad \bm{o}_t = \mathbf{S}^\top_t \bm{q}_t,$$

with $\beta_t\in[0,1]$ a scalar learning-rate gate and $\bm{\alpha}_t\in[0,1]^{d_k}$ a channel-wise forget vector (Team et al., 30 Oct 2025). Subsequent work has treated KDA as a canonical member of the delta-rule linear attention family, alongside DeltaNet and Gated DeltaNet, and as a reference point for extensions that add finer-grained writing control, query-aware updates, online preconditioning, or decoupled erase and write addresses (Sun et al., 21 Apr 2026, Park et al., 7 Jun 2026, Zhou et al., 13 May 2026, Li et al., 25 Jun 2026).

1. Historical positioning and formal identity

KDA was introduced in "Kimi Linear: An Expressive, Efficient Attention Architecture" as an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism (Team et al., 30 Oct 2025). Its stated architectural role is to provide a finite-state recurrent alternative to full attention while remaining compatible with a hardware-efficient chunkwise algorithm based on a specialized Diagonal-Plus-Low-Rank (DPLR) transition (Team et al., 30 Oct 2025). In that paper, Kimi Linear uses a layerwise hybrid of KDA and MLA, and the reported system uses 3B activated parameters and 48B total parameters (Team et al., 30 Oct 2025).

In the broader literature, KDA is consistently described as a delta-rule-based linear attention mechanism. "FG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control" places KDA in the family of delta-rule-based linear attention models whose state acts as an associative memory updated online by a gradient-descent-like delta rule (Sun et al., 21 Apr 2026). "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing" likewise presents KDA as one of four recurrent linear-attention architectures studied in a common recurrent-memory notation, together with DeltaNet, Gated DeltaNet, and Gated DeltaNet-2 (Cerruti et al., 8 Jul 2026).

The core distinction from earlier delta-rule variants is the shift from scalar forgetting to channel-wise forgetting. Starting from Gated DeltaNet,

$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$

KDA replaces scalar $\alpha_t$ by a vector $\alpha_t \in \mathbb{R}^{d_k}$, yielding

$$S_t = (I - \beta_t k_t k_t^\top)\,\text{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^\top,$$

while keeping $\beta_t$ scalar (Sun et al., 21 Apr 2026). This gives per-channel memory decay without changing the basic delta-rule write.

A complementary presentation appears in "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," which uses a feature-mapped fast-weight state $W^{(i)}\in\mathbb{R}^{d_v\times d_\phi}$, transformed keys $\beta_t\in[0,1]$0, and a vector forget gate $\beta_t\in[0,1]$1. There KDA is written as

$\beta_t\in[0,1]$2

with the decayed prediction

$\beta_t\in[0,1]$3

and residual

$\beta_t\in[0,1]$4

This formulation makes explicit that KDA remains a delta-rule mechanism: it decays memory channel-wise and then writes the residual error for the current transformed key (Cerruti et al., 8 Jul 2026).

2. State dynamics, delta rule, and DPLR structure

KDA is commonly interpreted as an online least-squares or online gradient-descent update on an associative memory. In the delta-rule view, the per-token regression objective is

$\beta_t\in[0,1]$5

and the ungated delta update is

$\beta_t\in[0,1]$6

which can be read as erasing conflicting content along the current key direction and then writing the new association (Team et al., 30 Oct 2025, Sun et al., 21 Apr 2026). KDA applies that delta step to a decayed state

$\beta_t\in[0,1]$7

so that forgetting is applied before the corrective write (Team et al., 30 Oct 2025).

This structure is significant because KDA is a specialized DPLR recurrence. In the Kimi Linear exposition,

$\beta_t\in[0,1]$8

which instantiates the general DPLR form

$\beta_t\in[0,1]$9

with

$\bm{\alpha}_t\in[0,1]^{d_k}$0

(Team et al., 30 Oct 2025). The same identification is made in FG$\bm{\alpha}_t\in[0,1]^{d_k}$1-GDN, which writes KDA’s transition as a DPLR matrix and notes that this preserves compatibility with Householder/WY-based chunkwise parallelization (Sun et al., 21 Apr 2026).

A recurrent-memory interpretation is also emphasized in later comparative work. In "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," all variants share the read rule

$\bm{\alpha}_t\in[0,1]^{d_k}$2

so the model class is determined entirely by how the fast-weight memory is updated from $\bm{\alpha}_t\in[0,1]^{d_k}$3 (Cerruti et al., 8 Jul 2026). In that framework, KDA sits strictly between Gated DeltaNet and Gated DeltaNet-2 with respect to memory-control granularity: more expressive than scalar forgetting, but still using a single scalar write-strength gate $\bm{\alpha}_t\in[0,1]^{d_k}$4 (Cerruti et al., 8 Jul 2026).

A plausible implication is that KDA’s main representational advantage lies not in changing the readout equation, which remains standard for delta-rule linear attention, but in enriching the state-transition operator so that different key-feature directions can have different retention times.

3. Parameterization, per-head computation, and hybrid integration in Kimi Linear

In Kimi Linear, KDA is not a standalone model but the recurrent token-mixing component of a hybrid stack. Per head $\bm{\alpha}_t\in[0,1]^{d_k}$5, the paper defines

$\bm{\alpha}_t\in[0,1]^{d_k}$6

$\bm{\alpha}_t\in[0,1]^{d_k}$7

$\bm{\alpha}_t\in[0,1]^{d_k}$8

$\bm{\alpha}_t\in[0,1]^{d_k}$9

(Team et al., 30 Oct 2025). The reported experiments use $^2$0 in all experiments (Team et al., 30 Oct 2025).

The head outputs are combined with a head-wise RMSNorm and output gate: $^2$1 (Team et al., 30 Oct 2025). The same paper reports that the output gate is crucial empirically, with ablations in Table 5 indicating that removing KDA’s output gate hurts performance (Team et al., 30 Oct 2025).

Kimi Linear arranges KDA and MLA in a uniform 3:1 ratio, with three KDA blocks followed by one MLA block (Team et al., 30 Oct 2025). This design underlies the claim that Kimi Linear reduces KV cache usage by up to 75% and achieves up to 6 times decoding throughput for a 1M context while outperforming full MLA under the same training recipe (Team et al., 30 Oct 2025). Because KDA layers maintain a constant-size state per head, they do not need token-wise KV cache; only MLA layers do, at 1/4 of the layer count (Team et al., 30 Oct 2025).

Later comparative work studied KDA in smaller-scale controlled sweeps. In the 350M-parameter, 15B-token setting of "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," KDA was evaluated in both pure and hybrid stacks with AdamW and Muon (Cerruti et al., 8 Jul 2026). That study reports that within the matched sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, specifically 2.273 for KDA+Muon in a hybrid stack (Cerruti et al., 8 Jul 2026). It also reports that pure Gated DeltaNet with AdamW has the highest normalized training throughput, so KDA occupies the more accuracy-oriented side of the throughput–loss trade-off (Cerruti et al., 8 Jul 2026).

4. Parallelization, kernels, and numerical considerations

KDA’s practical viability depends on its chunkwise implementation. The Kimi Linear paper derives a chunkwise WY/UT algorithm that compresses many rank-1 updates into dense per-chunk matrices and avoids the secondary chunking required by more general DPLR kernels (Team et al., 30 Oct 2025). For chunk size $^2$2 and head dimension $^2$3, the reported per-sequence FLOPs for one head are

$^2$4

whereas full attention per head is $^2$5 (Team et al., 30 Oct 2025).

The paper attributes KDA’s computational advantage over general DPLR to the fact that its transition is a specialized DPLR with tied low-rank structure. In the appendix pseudocode comparison, KDA eliminates two intra-chunk secondary chunk matmuls and about three matmuls across inter-chunk and output paths relative to generic DPLR (Team et al., 30 Oct 2025). The reported kernel benchmarks show about 2× kernel speedup over general DPLR at sequence lengths up to 64k (Team et al., 30 Oct 2025).

The chunkwise algorithm depends on lower-triangular systems. "Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers" identifies triangular inversion as a core sub-routine for Delta-rule linear transformer variants including KDA and states that linear attention is integrated into state-of-the-art open-source models including Qwen3.5/3.6, Kimi Linear, and RWKV-7 (Sobczyk et al., 20 May 2026). That work analyzes multiple inversion schemes and reports up to $^2$6 speed-up against the state-of-the-art implementations of SGLang for triangular matrix inversion, leading to significant performance improvements on the entire layer level while maintaining full end-to-end model accuracy (Sobczyk et al., 20 May 2026). This suggests that systems-level optimization of triangular solves is directly relevant to high-throughput KDA deployments.

Kimi Linear also specifies several numerical choices. The chunk math is done in float32 for stability, with inputs and outputs in bf16/fp16; cumulative decays are represented in log space to avoid underflow; L2 normalization is applied to $^2$7 and $^2$8 for spectral stability; and the typical chunk size in the complexity discussion and kernels is $^2$9 (Team et al., 30 Oct 2025). A plausible implication is that KDA’s numerical stability depends not only on its recurrence but also on careful implementation of decay products and triangular transforms.

5. Empirical behavior: performance, scaling, and trade-offs

KDA’s empirical profile is described across several papers at different scales and in different comparisons. The Kimi Linear paper reports that Kimi Linear outperforms full MLA with a sizeable margin across short-context, long-context, and reinforcement learning scaling regimes, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context (Team et al., 30 Oct 2025). At 1.4T tokens pretraining, Kimi Linear is reported to outperform MLA and a GDN hybrid on short-context and long-context benchmarks, with a reported long-context average of 54.5 versus 52.2 for MLA and 51.2 for GDN-H (Team et al., 30 Oct 2025). At 5.7T tokens, Kimi Linear-Base is reported to surpass Moonlight-Base on almost all metrics, including RULER@1M ≈94.8 (Team et al., 30 Oct 2025).

The more controlled 350M sweep in (Cerruti et al., 8 Jul 2026) gives a complementary picture. There, KDA+Muon in a hybrid stack achieves the lowest final validation loss, but pure Gated DeltaNet with AdamW is fastest, and hybrid stacks generally improve loss at a throughput cost (Cerruti et al., 8 Jul 2026). The same study reports that Muon consistently lowers final validation loss relative to AdamW in matched architecture settings and that KDA hybrids are slower than the pure Gated DeltaNet baseline because of both extra linear-attention complexity and the presence of softmax layers (Cerruti et al., 8 Jul 2026).

Later works used KDA as a baseline for improvements in long-context associative recall and language modeling. FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$0-GDN reports that at 1.3B parameters KDA obtains average LM accuracy 50.33, while FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$1-GDN reaches 53.95 and FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$2-GDN+ 53.40 (Sun et al., 21 Apr 2026). On RULER at 1.3B and 4k sequence length, the same paper reports average scores of 61.5 for KDA, 64.8 for FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$3-GDN, and 67.2 for FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$4-GDN+; at 16k sequence length, KDA scores 44.9 while FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$5-GDN+ reaches 48.9 (Sun et al., 21 Apr 2026). On LongBench at 1.3B, KDA scores 16.4, FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$6-GDN 16.0, and FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$7-GDN+ 18.3 (Sun et al., 21 Apr 2026). Those results are presented as evidence that KDA’s channel-wise forgetting improves long-context modeling over GDN, but that KDA’s scalar $$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$8 constrains dimension-specific writing.

Preconditioned DeltaNet reports consistent gains from adding curvature-aware diagonal preconditioning to KDA. At 340M scale, KDA achieves commonsense average 49.12, whereas PKDA reaches 49.89; at 1B, KDA reaches 55.43 and PKDA 56.64 (Tumma et al., 22 Apr 2026). The same study reports LAMBADA perplexity improvements from 31.37 to 25.33 at 340M and from 14.52 to 11.86 at 1B, and modest gains on in-context retrieval averages (Tumma et al., 22 Apr 2026). This suggests that KDA’s baseline update rule benefits from curvature-aware write-key scaling without changing the overall recurrent-memory design.

6. Extensions, critiques, and related mechanisms

A recurring critique of KDA is that it provides fine-grained forgetting but coarse writing. FG$$S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,$$9-GDN states this explicitly: KDA refines GDN’s decay gate from scalar to channel-wise while keeping the learning rate $\alpha_t$0 scalar, which limits dimension-specific adaptation (Sun et al., 21 Apr 2026). FG$\alpha_t$1-GDN addresses this by replacing scalar $\alpha_t$2 with a channel-wise vector absorbed symmetrically into keys and values, preserving a symmetric rank-1 low-rank term and thus the DPLR structure (Sun et al., 21 Apr 2026). FG$\alpha_t$3-GDN+ further decouples key and value scaling, allowing separate control of erasure strength and write strength (Sun et al., 21 Apr 2026).

A second line of critique concerns online curvature. OSDN characterizes DeltaNet, Gated DeltaNet, and KDA as scalar-step delta-rule models analogous to vanilla SGD, and proposes online diagonal preconditioning as an optimization-style improvement (Zhou et al., 13 May 2026). Preconditioned DeltaNet makes a closely related point within the test-time regression framework, deriving preconditioned variants of DeltaNet, GDN, and KDA and arguing that exact preconditioning would make delta-rule updates equivalent to least-squares solutions, while practical diagonal preconditioning improves conditioning in key space (Tumma et al., 22 Apr 2026).

A third critique targets the role of the query. Q-Delta argues that in standard delta-rule attention, queries only appear in the readout $\alpha_t$4, whereas the state evolution is driven only by keys and values (Park et al., 7 Jun 2026). It introduces a mixed key–query prediction error

$\alpha_t$5

and uses it in the state update while preserving delta-rule efficiency (Park et al., 7 Jun 2026). In the taxonomy of that paper, if KDA is key-based delta attention, Q-Delta is a query-aware delta attention (Park et al., 7 Jun 2026).

A fourth extension concerns memory management. Erase-then-Delta Attention argues that standard delta-rule updates, including KDA,

$\alpha_t$6

couple where to erase and where to write: both are anchored to the current key $\alpha_t$7 (Li et al., 25 Jun 2026). EDA adds an independent erase direction $\alpha_t$8 with gate $\alpha_t$9,

$\alpha_t \in \mathbb{R}^{d_k}$0

and reports improved downstream averages at both 2.5B and 25B-A2.8B scale, as well as the highest average RULER score after long-context midtraining (Li et al., 25 Jun 2026). In this framing, KDA remains the backbone but lacks independent address-level cleanup.

A related but distinct development is ATMA, which combines Polar Attention with a gated-delta compression memory. Although it does not present KDA itself, it describes its recurrent memory as conceptually close to Kimi-style incremental KV updates and uses a gated-delta fast-weights rule to maintain a constant-size memory state per head (Akbar, 23 Jun 2026). This suggests a broader convergence between KDA-like recurrent memory and architectures explicitly designed for length-invariant long-context behavior.

7. Conceptual significance and open directions

KDA occupies a specific point in the design space of linear attention architectures. Relative to softmax attention, it replaces explicit $\alpha_t \in \mathbb{R}^{d_k}$1 attention matrices with a compressed recurrent memory and delta-rule updates, yielding linear-time scaling in sequence length (Team et al., 30 Oct 2025, Cerruti et al., 8 Jul 2026). Relative to earlier linear recurrent models, it combines fine-grained per-channel decay with corrective writing, which is why multiple later papers use it as a strong baseline rather than as a minimal one (Sun et al., 21 Apr 2026, Tumma et al., 22 Apr 2026).

Within the delta-rule family, KDA is often treated as the first architecture to make channel-wise forgetting central. In the hierarchy described in (Cerruti et al., 8 Jul 2026),

$\alpha_t \in \mathbb{R}^{d_k}$2

with respect to granularity of memory control. That hierarchy should not be read as a strict empirical ranking across all tasks; the same paper reports that KDA+Muon attains the best validation loss in its sweep, whereas Gated DeltaNet pure stacks maximize normalized training throughput (Cerruti et al., 8 Jul 2026).

Several open directions emerge repeatedly. One is richer write control: vector $\alpha_t \in \mathbb{R}^{d_k}$3, separate $\alpha_t \in \mathbb{R}^{d_k}$4 and $\alpha_t \in \mathbb{R}^{d_k}$5, or more general erase–write decoupling (Sun et al., 21 Apr 2026, Li et al., 25 Jun 2026). Another is better online curvature modeling via diagonal or more expressive preconditioners (Zhou et al., 13 May 2026, Tumma et al., 22 Apr 2026). A third is query-aware memory evolution (Park et al., 7 Jun 2026). A fourth is systems engineering: stable triangular inversion, efficient chunkwise kernels, and hybrid stacks that preserve KDA’s linear-time advantages while retaining some full-attention capacity (Sobczyk et al., 20 May 2026, Team et al., 30 Oct 2025).

Taken together, these developments indicate that KDA is best understood neither as a generic synonym for linear attention nor as a complete endpoint. It is a particular recurrent-memory mechanism defined by a delta-rule update with channel-wise forgetting, a specialized DPLR transition, and efficient chunkwise execution. Its significance lies in establishing that this combination can be competitive enough to serve as the core of a large hybrid model that outperforms full attention under matched recipes (Team et al., 30 Oct 2025), while also providing a well-defined baseline from which later work can isolate the effects of finer-grained writing, query-aware correction, online preconditioning, and independent erase control.