---
title: 'Kimi Delta Attention: Delta‐Rule Linear Mechanism'
url: https://www.emergentmind.com/topics/kimi-delta-attention
type: topic
---

# Kimi Delta Attention: Delta‐Rule Linear Mechanism

Searching arXiv for recent papers on Kimi Delta Attention and related delta-rule linear attention.
Kimi Delta Attention (KDA) is a delta-rule-based linear attention mechanism introduced as the core recurrent component of Kimi Linear, a hybrid linear attention architecture that combines KDA with Multi-Head Latent Attention (MLA) [2510.26692]. In its defining form, KDA maintains a matrix-valued associative memory and updates it by a delta rule with a channel-wise forget gate, replacing the scalar forgetting used in Gated DeltaNet with a diagonal gate over key dimensions [2510.26692]. In the Kimi Linear formulation, the single-head recurrence is
\[
\mathbf{S}_t = \left(\mathbf{I}-\beta_t\bm{k}_{t}\bm{k}_{t}^{\top}\right) \operatorname{Diag}\left(\bm{\alpha}_t \right)\mathbf{S}_{t-1} + \beta_t\bm{k}_{t}\bm{v}_{t}^{\top}, \qquad \bm{o}_t = \mathbf{S}^\top_t \bm{q}_t,
\]
with \(\beta_t\in[0,1]\) a scalar learning-rate gate and \(\bm{\alpha}_t\in[0,1]^{d_k}\) a channel-wise forget vector [2510.26692]. Subsequent work has treated KDA as a canonical member of the delta-rule linear attention family, alongside DeltaNet and Gated DeltaNet, and as a reference point for extensions that add finer-grained writing control, query-aware updates, online preconditioning, or decoupled erase and write addresses [2604.19021] [2606.08804] [2605.13473] [2606.26560].

## 1. Historical positioning and formal identity

KDA was introduced in "Kimi Linear: An Expressive, Efficient Attention Architecture" as an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism [2510.26692]. Its stated architectural role is to provide a finite-state recurrent alternative to full attention while remaining compatible with a hardware-efficient chunkwise algorithm based on a specialized Diagonal-Plus-Low-Rank (DPLR) transition [2510.26692]. In that paper, Kimi Linear uses a layerwise hybrid of KDA and MLA, and the reported system uses 3B activated parameters and 48B total parameters [2510.26692].

In the broader literature, KDA is consistently described as a delta-rule-based linear attention mechanism. "FG\(^2\)-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control" places KDA in the family of delta-rule-based linear attention models whose state acts as an associative memory updated online by a gradient-descent-like delta rule [2604.19021]. "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing" likewise presents KDA as one of four recurrent linear-attention architectures studied in a common recurrent-memory notation, together with DeltaNet, Gated DeltaNet, and Gated DeltaNet-2 [2607.07953].

The core distinction from earlier delta-rule variants is the shift from scalar forgetting to channel-wise forgetting. Starting from Gated DeltaNet,
\[
S_t = \alpha_t (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,
\]
KDA replaces scalar \(\alpha_t\) by a vector \(\alpha_t \in \mathbb{R}^{d_k}\), yielding
\[
S_t = (I - \beta_t k_t k_t^\top)\,\text{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^\top,
\]
while keeping \(\beta_t\) scalar [2604.19021]. This gives per-channel memory decay without changing the basic delta-rule write.

A complementary presentation appears in "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," which uses a feature-mapped fast-weight state \(W^{(i)}\in\mathbb{R}^{d_v\times d_\phi}\), transformed keys \(\kappa^{(i)}\), and a vector forget gate \(\bm{\alpha}^{(i)}\). There KDA is written as
\[
W^{(i)} = W^{(i-1)} D_{\alpha}^{(i)} + \beta^{(i)}\,r_{\bm{\alpha}}^{(i)} \otimes \kappa^{(i)},
\]
with the decayed prediction
\[
\bar v_{\bm{\alpha}}^{(i)} = W^{(i-1)} D_{\alpha}^{(i)} \kappa^{(i)},
\]
and residual
\[
r_{\bm{\alpha}}^{(i)} = v^{(i)} - \bar v_{\bm{\alpha}}^{(i)}.
\]
This formulation makes explicit that KDA remains a delta-rule mechanism: it decays memory channel-wise and then writes the residual error for the current transformed key [2607.07953].

## 2. State dynamics, delta rule, and DPLR structure

KDA is commonly interpreted as an online least-squares or online gradient-descent update on an associative memory. In the delta-rule view, the per-token regression objective is
\[
\mathcal{L}_t(S) = \frac{1}{2} \|S^\top k_t - v_t\|^2,
\]
and the ungated delta update is
\[
S_t = (I - \beta_t k_t k_t^\top) S_{t-1} + \beta_t k_t v_t^\top,
\]
which can be read as erasing conflicting content along the current key direction and then writing the new association [2510.26692] [2604.19021]. KDA applies that delta step to a decayed state
\[
\tilde{\mathbf{S}}_{t-1}=\operatorname{Diag}(\bm{\alpha}_t)\mathbf{S}_{t-1},
\]
so that forgetting is applied before the corrective write [2510.26692].

This structure is significant because KDA is a specialized DPLR recurrence. In the Kimi Linear exposition,
\[
\mathbf{S}_t = \Bigl( \operatorname{Diag}(\bm{\alpha}_t) - \beta_t\bm{k}_t(\bm{k}_t\odot\bm{\alpha}_t)^\top \Bigr)\mathbf{S}_{t-1} + \beta_t\bm{k}_t\bm{v}_t^\top,
\]
which instantiates the general DPLR form
\[
\mathbf{S}_t = (\mathbf{D} - \bm{a}_t \bm{b}_t^{\top}) \mathbf{S}_{t-1} + \bm{k}_t \bm{v}_t^{\top}
\]
with
\[
\mathbf{D} = \operatorname{Diag}(\bm{\alpha}_t),\qquad \bm{a}_t = \beta_t\bm{k}_t,\qquad \bm{b}_t = \bm{k}_t\odot\bm{\alpha}_t
\]
[2510.26692]. The same identification is made in FG\(^2\)-GDN, which writes KDA’s transition as a DPLR matrix and notes that this preserves compatibility with Householder/WY-based chunkwise parallelization [2604.19021].

A recurrent-memory interpretation is also emphasized in later comparative work. In "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," all variants share the read rule
\[
y^{(i)} = W^{(i)}\,\phi\!\left(q^{(i)}\right),
\]
so the model class is determined entirely by how the fast-weight memory is updated from \((\kappa^{(i)}, v^{(i)})\) [2607.07953]. In that framework, KDA sits strictly between Gated DeltaNet and Gated DeltaNet-2 with respect to memory-control granularity: more expressive than scalar forgetting, but still using a single scalar write-strength gate \(\beta\) [2607.07953].

A plausible implication is that KDA’s main representational advantage lies not in changing the readout equation, which remains standard for delta-rule linear attention, but in enriching the state-transition operator so that different key-feature directions can have different retention times.

## 3. Parameterization, per-head computation, and hybrid integration in Kimi Linear

In Kimi Linear, KDA is not a standalone model but the recurrent token-mixing component of a hybrid stack. Per head \(h\), the paper defines
\[
\bm{q}^h_t,\bm{k}^h_t = \operatorname{L2Norm}(\operatorname{Swish}(\operatorname{ShortConv}(\mathbf{W}^h_{q/k}\bm{x}_t))) \in \mathbb{R}^{d_k},
\]
\[
\bm{v}^h_t = \operatorname{Swish}(\operatorname{ShortConv}(\mathbf{W}^h_v\bm{x}_t)) \in \mathbb{R}^{d_v},
\]
\[
\bm{\alpha}^h_t = f(\mathbf{W}_{\alpha}^{\uparrow}\mathbf{W}_{\alpha}^{\downarrow}\bm{x}_t) \in [0,1]^{d_k},
\]
\[
\beta^h_t = \operatorname{Sigmoid}(\mathbf{W}_{\beta}^h\bm{x}_t) \in [0,1]
\]
[2510.26692]. The reported experiments use \(d_k=d_v=128\) in all experiments [2510.26692].

The head outputs are combined with a head-wise RMSNorm and output gate:
\[
\bm{o}_t = \mathbf{W}_o\left( \operatorname{Sigmoid}\left(\mathbf{W}_g^{\uparrow}\mathbf{W}_g^{\downarrow} \bm{x}_t\right) \odot \operatorname{RMSNorm}\left(\operatorname{KDA}\left( \bm{q}_t,\bm{k}_t,\bm{v}_t,\bm{\alpha}_t,\beta_t \right) \right)\right)
\]
[2510.26692]. The same paper reports that the output gate is crucial empirically, with ablations in Table 5 indicating that removing KDA’s output gate hurts performance [2510.26692].

Kimi Linear arranges KDA and MLA in a uniform 3:1 ratio, with three KDA blocks followed by one MLA block [2510.26692]. This design underlies the claim that Kimi Linear reduces KV cache usage by up to 75% and achieves up to 6 times decoding throughput for a 1M context while outperforming full MLA under the same training recipe [2510.26692]. Because KDA layers maintain a constant-size state per head, they do not need token-wise KV cache; only MLA layers do, at 1/4 of the layer count [2510.26692].

Later comparative work studied KDA in smaller-scale controlled sweeps. In the 350M-parameter, 15B-token setting of "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing," KDA was evaluated in both pure and hybrid stacks with AdamW and Muon [2607.07953]. That study reports that within the matched sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, specifically 2.273 for KDA+Muon in a hybrid stack [2607.07953]. It also reports that pure Gated DeltaNet with AdamW has the highest normalized training throughput, so KDA occupies the more accuracy-oriented side of the throughput–loss trade-off [2607.07953].

## 4. Parallelization, kernels, and numerical considerations

KDA’s practical viability depends on its chunkwise implementation. The Kimi Linear paper derives a chunkwise WY/UT algorithm that compresses many rank-1 updates into dense per-chunk matrices and avoids the secondary chunking required by more general DPLR kernels [2510.26692]. For chunk size \(C\) and head dimension \(d_h\), the reported per-sequence FLOPs for one head are
\[
\mathrm{FLOPs}_{\text{KDA}(T; C, d_h)} = 6 T d_h^2 + 3 T C d_h + T C^2,
\]
whereas full attention per head is \(2 T^2 d_h\) [2510.26692].

The paper attributes KDA’s computational advantage over general DPLR to the fact that its transition is a specialized DPLR with tied low-rank structure. In the appendix pseudocode comparison, KDA eliminates two intra-chunk secondary chunk matmuls and about three matmuls across inter-chunk and output paths relative to generic DPLR [2510.26692]. The reported kernel benchmarks show about 2× kernel speedup over general DPLR at sequence lengths up to 64k [2510.26692].

The chunkwise algorithm depends on lower-triangular systems. "Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers" identifies triangular inversion as a core sub-routine for Delta-rule linear transformer variants including KDA and states that linear attention is integrated into state-of-the-art open-source models including Qwen3.5/3.6, Kimi Linear, and RWKV-7 [2605.21325]. That work analyzes multiple inversion schemes and reports up to \(4.3\times\) speed-up against the state-of-the-art implementations of SGLang for triangular matrix inversion, leading to significant performance improvements on the entire layer level while maintaining full end-to-end model accuracy [2605.21325]. This suggests that systems-level optimization of triangular solves is directly relevant to high-throughput KDA deployments.

Kimi Linear also specifies several numerical choices. The chunk math is done in float32 for stability, with inputs and outputs in bf16/fp16; cumulative decays are represented in log space to avoid underflow; L2 normalization is applied to \(q\) and \(k\) for spectral stability; and the typical chunk size in the complexity discussion and kernels is \(C=64\) [2510.26692]. A plausible implication is that KDA’s numerical stability depends not only on its recurrence but also on careful implementation of decay products and triangular transforms.

## 5. Empirical behavior: performance, scaling, and trade-offs

KDA’s empirical profile is described across several papers at different scales and in different comparisons. The Kimi Linear paper reports that Kimi Linear outperforms full MLA with a sizeable margin across short-context, long-context, and reinforcement learning scaling regimes, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context [2510.26692]. At 1.4T tokens pretraining, Kimi Linear is reported to outperform MLA and a GDN hybrid on short-context and long-context benchmarks, with a reported long-context average of 54.5 versus 52.2 for MLA and 51.2 for GDN-H [2510.26692]. At 5.7T tokens, Kimi Linear-Base is reported to surpass Moonlight-Base on almost all metrics, including RULER@1M ≈94.8 [2510.26692].

The more controlled 350M sweep in [2607.07953] gives a complementary picture. There, KDA+Muon in a hybrid stack achieves the lowest final validation loss, but pure Gated DeltaNet with AdamW is fastest, and hybrid stacks generally improve loss at a throughput cost [2607.07953]. The same study reports that Muon consistently lowers final validation loss relative to AdamW in matched architecture settings and that KDA hybrids are slower than the pure Gated DeltaNet baseline because of both extra linear-attention complexity and the presence of softmax layers [2607.07953].

Later works used KDA as a baseline for improvements in long-context associative recall and language modeling. FG\(^2\)-GDN reports that at 1.3B parameters KDA obtains average LM accuracy 50.33, while FG\(^2\)-GDN reaches 53.95 and FG\(^2\)-GDN+ 53.40 [2604.19021]. On RULER at 1.3B and 4k sequence length, the same paper reports average scores of 61.5 for KDA, 64.8 for FG\(^2\)-GDN, and 67.2 for FG\(^2\)-GDN+; at 16k sequence length, KDA scores 44.9 while FG\(^2\)-GDN+ reaches 48.9 [2604.19021]. On LongBench at 1.3B, KDA scores 16.4, FG\(^2\)-GDN 16.0, and FG\(^2\)-GDN+ 18.3 [2604.19021]. Those results are presented as evidence that KDA’s channel-wise forgetting improves long-context modeling over GDN, but that KDA’s scalar \(\beta_t\) constrains dimension-specific writing.

Preconditioned DeltaNet reports consistent gains from adding curvature-aware diagonal preconditioning to KDA. At 340M scale, KDA achieves commonsense average 49.12, whereas PKDA reaches 49.89; at 1B, KDA reaches 55.43 and PKDA 56.64 [2604.21100]. The same study reports LAMBADA perplexity improvements from 31.37 to 25.33 at 340M and from 14.52 to 11.86 at 1B, and modest gains on in-context retrieval averages [2604.21100]. This suggests that KDA’s baseline update rule benefits from curvature-aware write-key scaling without changing the overall recurrent-memory design.

## 6. Extensions, critiques, and related mechanisms

A recurring critique of KDA is that it provides fine-grained forgetting but coarse writing. FG\(^2\)-GDN states this explicitly: KDA refines GDN’s decay gate from scalar to channel-wise while keeping the learning rate \(\beta_t\) scalar, which limits dimension-specific adaptation [2604.19021]. FG\(^2\)-GDN addresses this by replacing scalar \(\beta_t\) with a channel-wise vector absorbed symmetrically into keys and values, preserving a symmetric rank-1 low-rank term and thus the DPLR structure [2604.19021]. FG\(^2\)-GDN+ further decouples key and value scaling, allowing separate control of erasure strength and write strength [2604.19021].

A second line of critique concerns online curvature. OSDN characterizes DeltaNet, Gated DeltaNet, and KDA as scalar-step delta-rule models analogous to vanilla SGD, and proposes online diagonal preconditioning as an optimization-style improvement [2605.13473]. Preconditioned DeltaNet makes a closely related point within the test-time regression framework, deriving preconditioned variants of DeltaNet, GDN, and KDA and arguing that exact preconditioning would make delta-rule updates equivalent to least-squares solutions, while practical diagonal preconditioning improves conditioning in key space [2604.21100].

A third critique targets the role of the query. Q-Delta argues that in standard delta-rule attention, queries only appear in the readout \(o_t = S_t q_t\), whereas the state evolution is driven only by keys and values [2606.08804]. It introduces a mixed key–query prediction error
\[
e_t^{(\text{mix})} = v_t - S_{t-1}k_t - \lambda_t S_{t-1}q_t
\]
and uses it in the state update while preserving delta-rule efficiency [2606.08804]. In the taxonomy of that paper, if KDA is key-based delta attention, Q-Delta is a query-aware delta attention [2606.08804].

A fourth extension concerns memory management. Erase-then-Delta Attention argues that standard delta-rule updates, including KDA,
\[
S_t = (I - \beta_t k_t k_t^\top) D_t S_{t-1} + \beta_t k_t v_t^\top,
\]
couple where to erase and where to write: both are anchored to the current key \(k_t\) [2606.26560]. EDA adds an independent erase direction \(e_t\) with gate \(\gamma_t\),
\[
S_t = (I - \beta_t k_t k_t^\top)(I - \gamma_t e_t e_t^\top)D_t S_{t-1} + \beta_t k_t v_t^\top,
\]
and reports improved downstream averages at both 2.5B and 25B-A2.8B scale, as well as the highest average RULER score after long-context midtraining [2606.26560]. In this framing, KDA remains the backbone but lacks independent address-level cleanup.

A related but distinct development is ATMA, which combines Polar Attention with a gated-delta compression memory. Although it does not present KDA itself, it describes its recurrent memory as conceptually close to Kimi-style incremental KV updates and uses a gated-delta fast-weights rule to maintain a constant-size memory state per head [2606.25156]. This suggests a broader convergence between KDA-like recurrent memory and architectures explicitly designed for length-invariant long-context behavior.

## 7. Conceptual significance and open directions

KDA occupies a specific point in the design space of linear attention architectures. Relative to softmax attention, it replaces explicit \(T\times T\) attention matrices with a compressed recurrent memory and delta-rule updates, yielding linear-time scaling in sequence length [2510.26692] [2607.07953]. Relative to earlier linear recurrent models, it combines fine-grained per-channel decay with corrective writing, which is why multiple later papers use it as a strong baseline rather than as a minimal one [2604.19021] [2604.21100].

Within the delta-rule family, KDA is often treated as the first architecture to make channel-wise forgetting central. In the hierarchy described in [2607.07953],
\[
\text{DeltaNet} \subset \text{Gated DeltaNet} \subset \text{Kimi Delta Attention} \subset \text{Gated DeltaNet-2},
\]
with respect to granularity of memory control. That hierarchy should not be read as a strict empirical ranking across all tasks; the same paper reports that KDA+Muon attains the best validation loss in its sweep, whereas Gated DeltaNet pure stacks maximize normalized training throughput [2607.07953].

Several open directions emerge repeatedly. One is richer write control: vector \(\beta_t\), separate \(\beta_t^k\) and \(\beta_t^v\), or more general erase–write decoupling [2604.19021] [2606.26560]. Another is better online curvature modeling via diagonal or more expressive preconditioners [2605.13473] [2604.21100]. A third is query-aware memory evolution [2606.08804]. A fourth is systems engineering: stable triangular inversion, efficient chunkwise kernels, and hybrid stacks that preserve KDA’s linear-time advantages while retaining some full-attention capacity [2605.21325] [2510.26692].

Taken together, these developments indicate that KDA is best understood neither as a generic synonym for linear attention nor as a complete endpoint. It is a particular recurrent-memory mechanism defined by a delta-rule update with channel-wise forgetting, a specialized DPLR transition, and efficient chunkwise execution. Its significance lies in establishing that this combination can be competitive enough to serve as the core of a large hybrid model that outperforms full attention under matched recipes [2510.26692], while also providing a well-defined baseline from which later work can isolate the effects of finer-grained writing, query-aware correction, online preconditioning, and independent erase control.

Source: https://www.emergentmind.com/topics/kimi-delta-attention