--- title: 'Gated DeltaNet-2: Efficient Linear Attention' url: https://www.emergentmind.com/topics/gated-deltanet-2 type: topic --- # Gated DeltaNet-2: Efficient Linear Attention Gated DeltaNet-2 is a linear attention architecture that extends the delta-rule fast-weight approach by introducing channel-wise decoupled erase and write mechanisms in the memory update, achieving state-of-the-art performance among linear recurrent models, particularly on long-context and retrieval-heavy tasks. It generalizes previous models such as Gated DeltaNet and Kimi Delta Attention (KDA) by inheriting their adaptive, channel-wise forgetting but addresses the key deficiency of those approaches: the use of a single scalar gate to jointly control memory erasure and writing. Instead, Gated DeltaNet-2 deploys independent erase gates for each key channel and write gates for each value channel, thus enabling more flexible and expressive memory dynamics [2605.22791]. ## 1. Motivation and Relation to Previous Methods Linear attention models replace traditional softmax attention's unbounded state with a fixed-size, recurrent associative memory ($M_t \in \mathbb{R}^{d_k \times d_v}$), leading to linear computation time and fixed memory during inference. In standard delta-rule models, memory updates actively overwrite the association addressed by the current key, e.g., in DeltaNet and Gated DeltaNet, while KDA sharpened adaptive forgetting by introducing a channel-wise decay vector $\alpha_t \in (0,1]^{d_k}$. However, these methods tie two distinct operations—erasure on the key axis and writing on the value axis—into a single scalar gate $\beta_t$. This design limits control, particularly as the context grows long and memory interference increases. Gated DeltaNet-2 disentangles these processes by introducing separate, channel-wise gates: - Erase gate $b_t \in [0,1]^{d_k}$ for key-wise erasure - Write gate $w_t \in [0,1]^{d_v}$ for value-wise writing - Retaining channel-wise decay $\alpha_t$ for adaptive forgetting. Special cases of this formulation recover KDA when $b_t = w_t = \beta_t\mathbf{1}$ (with $\alpha_t$ channel-wise) and Gated DeltaNet when $\alpha_t = \alpha_t\mathbf{1}$ is also scalar [2605.22791]. ## 2. Mathematical Formulation Let time step $t$ have key $k_t \in \mathbb{R}^{d_k}$ and value $v_t \in \mathbb{R}^{d_v}$, with input features $x_t$. The gates are computed as: - $b_t = \sigma(W_b x_t) \in [0,1]^{d_k}$ - $w_t = \sigma(W_w x_t) \in [0,1]^{d_v}$ - $\alpha_t = \exp(-\exp(a)\odot \mathrm{softplus}(W_f x_t+\delta)) \in (0,1]^{d_k}$ The recurrent memory state is decayed as: - $\bar{M}_t = \mathrm{diag}(\alpha_t) M_{t-1}$ Read and write vectors, gated channel-wise: - $\tilde{k}_t = b_t \odot k_t \in \mathbb{R}^{d_k}$ - $\tilde{v}_t = w_t \odot v_t \in \mathbb{R}^{d_v}$ The full Gated Delta Rule-2 update is: \[ M_t = \bar{M}_t + k_t(\tilde{v}_t - \bar{M}_t^\top \tilde{k}_t)^\top = (I - \tilde{k}_t \tilde{k}_t^\top)\bar{M}_t + k_t \tilde{v}_t^\top \] This can be equivalently written as an explicit "erase + write" decomposition: \[ M_t = (-\tilde{k}_t \tilde{k}_t^\top)\bar{M}_t + k_t \tilde{v}_t^\top \] When both gates reduce to the same scalar, the update recovers KDA; further restricting decay to a scalar recovers Gated DeltaNet [2605.22791]. ## 3. Fast-Weight Perspective and Chunkwise WY Algorithm Gated DeltaNet-2 implements a fast-weight memory update as the solution to a local quadratic objective at each step: \[ \mathcal{L}_t(M) = \|M - \bar{M}_t\|_F^2 - 2\langle \tilde{v}_t, M^\top \tilde{k}_t - \bar{M}_t^\top \tilde{k}_t \rangle \] The optimizer is: \[ M_t = \bar{M}_t + k_t(\tilde{v}_t - \bar{M}_t^\top \tilde{k}_t)^\top \] For long sequences, chunkwise WY (Woodbury) updates allow for parallel computation by splitting a sequence of length $L$ into chunks of size $C$ and applying cumulative, component-wise decay: - $\Gamma_r = \prod_{i=1}^r \alpha_i$, $\widehat{M}_r = \mathrm{diag}(\Gamma_r)^{-1} M_r$ - Updates combine as a series of rank-one corrections enabling efficient factorization and chunk outputs via triangular forms, which are hardware-friendly [2605.22791]. Computational complexity remains $O(L d_k d_v)$ for memory updates and $O(L d_k^2)$ for chunk solves, with $O(d_k d_v)$ recurrent state memory and $O(C d_k^2)$ per chunk. ## 4. Gate-Aware Backward Pass Gated DeltaNet-2 requires a backward pass that explicitly accumulates gradients with respect to the separate gates $b_t$ and $w_t$ inside the WY factors. For memory- and throughput-efficient parallel training, the gradient computation is adapted so that \[ \mathrm{dW} \mathrel{+}= (\mathrm{dU})(w_t \odot v_t)^\top,\qquad \mathrm{dV} \mathrel{+}= (\mathrm{dU})^\top(w_t \odot U) \] and similar operations for $b_t$ and decays. Gradients with respect to logarithmic decay factors are computed via a reverse cumulative sum. The triangular solves and output computations share the efficient vector-Jacobian structure of KDA [2605.22791]. ## 5. Empirical Performance Gated DeltaNet-2 was evaluated at 1.3B parameters trained on 100B FineWeb-Edu tokens. Results demonstrate dominance over Gated DeltaNet, KDA, Mamba-2, and Mamba-3 variants in both recurrent and hybrid (2K sliding-window attention) configurations: | Model | WikiText PPL | LAMBADA PPL / Acc | Commonsense Acc | RULER S-NIAH-2@4K | RULER MK-NIAH-1@4K | Retrieval Avg | |--------------------|--------------|-------------------|-----------------|-------------------|-------------------|---------------| | Gated DeltaNet-2 | 15.90 | 11.41 / 48.09% | 53.11% | **93.0%** | **37.8%** | **29.88%** | | Gated DeltaNet | 16.40 | 11.88 / 47.13% | 52.85% | 87.2% | 27.8% | 28.09% | | KDA | 16.81 | 12.22 / 47.27% | 51.98% | 89.0% | 28.0% | 28.67% | On synthetic multi-key retrieval benchmarks (RULER MK-NIAH-1 @ 4K), Gated DeltaNet-2 achieves 37.8% (vs. 27.8% for Gated DeltaNet and 28.0% for KDA), highlighting improved resistance to interference in long contexts. Throughput (H100, hybrid model) reaches 38.0 Kt/s for 2K×8 batches, with near-flat scaling at long sequence lengths (dropping only to 36.1 Kt/s at 16K×1), approximately 7% below KDA but vastly outperforming full softmax attention under long sequence loads [2605.22791]. ## 6. Ablation Studies and Analysis Ablation results support the importance of full gate decoupling: - Using a scalar $b$ (erase gate) and channel-wise $w$ (write gate) yields degraded performance (Wiki ppl 16.55, S-NIAH-2@4K=90.6%). - Channel-wise $b$ with scalar $w$ recovers most gains (Wiki ppl 16.12, S-NIAH-2@4K=92.1%). - Full channel-wise decoupling is optimal (Wiki ppl 15.90, S-NIAH-2@4K=93.0%). The memory edit is therefore primarily mediated by key-side (erase) gating, while value-side (write) gating contributes additional improvement. Allowing $b_t \in [0,2]^{d_k}$ produced no consistent gains at this model size. This suggests the principal expressivity arises from the key-gated erasure pathway [2605.22791]. ## 7. Architectural Impact and Connections Gated DeltaNet-2 generalizes and subsumes both KDA and Gated DeltaNet, reduces to FG$^2$-GDN$^+$ in the limit where erasure and write gates control keys and values independently, and fits within the broader trend of per-channel control in sequence models. Whereas FG$^2$-GDN and FG$^2$-GDN$^+$ focus on per-channel step sizes (drawing analogies to AdaGrad/Adam in adaptive optimization), Gated DeltaNet-2 implements direct channel-wise gating, yielding similar fine-grained adaptation in memory dynamics at a modest incremental runtime cost [2604.19021]. Hybrid variants incorporating sliding-window attention retain the benefits of Gated DeltaNet-2 on long-range tasks and further boost local context modeling. This architectural flexibility, combined with efficient chunkwise implementation and state-of-the-art accuracy on retrieval and long-context understanding, position Gated DeltaNet-2 as a leading approach for scalable linear recurrent attention models [2605.22791, 2412.06464, 2604.19021]. Source: https://www.emergentmind.com/topics/gated-deltanet-2