Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Abstract: Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.

• The paper's main contribution is decoupling erase and write operations using independent channel-wise gates, enhancing memory editing in recurrent attention models.
• It employs a fast-weight update and WY matrix transformation to maintain efficient chunkwise training and robust retrieval in long contexts.
• Empirical results demonstrate improved language modeling, reduced interference, and state-of-the-art retrieval across benchmarks using the proposed architecture.

Decoupling Erase and Write in Linear Attention: The Gated DeltaNet-2 Architecture

Background and Motivation

Linear recurrent attention mechanisms have been developed to address the quadratic complexity of conventional self-attention in sequence models. By substituting the standard attention matrix with a fixed-size recurrent state, these approaches maintain constant memory usage and achieve linear computational cost in sequence length. However, fixed-state models face inherent limitations, notably the challenge of interference and information loss when compressing long contexts into bounded memory. In prior work, memory management hinged on applying scalar gates for global decay (forgetting) and targeted overwriting (editing) of state content. State-of-the-art recurrent models such as Gated DeltaNet and Kimi Delta Attention (KDA) incorporate decay and delta-rule residual editing, but these models fundamentally conflate the mechanisms for erasing obsolete information and committing new content to memory, tying both operations to a single scalar parameter per head.

Contributions of Gated DeltaNet-2

Gated DeltaNet-2 introduces a principled decoupling of the erase and write operations in linear recurrent attention, implementing channel-wise gates for each. The architecture maintains KDA's channel-wise decay but replaces its tied scalar delta gate with two independent, channel-wise gates: a key-side erase gate ($b_t \in [0,1]^{d_k}$) and a value-side write gate ($W_t \in [0,1]^{d_v}$). This separation enables selective, channel-specific removal of outdated associations and insertion of new values, providing finer-grained memory editing. Gated DeltaNet-2 generalizes existing delta-rule recurrent attention models: it reduces to KDA when both gates collapse to a scalar and to Gated DeltaNet when the decay is scalar as well.

The authors also develop a fast-weight update perspective that preserves efficient chunkwise training through a WY matrix formulation, where channel-specific decays are absorbed into asymmetric rank-one factors for erase and write, ensuring compatibility with efficient vectorized kernels and backward passes.

Methodology

Detailed Architectural Advances

1. Channel-wise Erase and Write Gates: Instead of using a single scalar to control how much to erase and write, Gated DeltaNet-2 learns independent channel-wise gates for each. The key-side erase gate determines which components of the memory state—projected onto the key dimension—should be overwritten, while the value-side write gate independently selects which value components are allowed to update the state. This breaks the restrictive coupling in earlier models, which required erasing and writing to occur along the same scalar degree.
2. Mathematical Formulation: The recurrence is as follows:

$S_t = (I - k_t e_t^{\top}) D_t S_{t-1} + k_t z_t,$

where $e_t = b_t \odot k_t$, $z_t = W_t \odot U_t$, and $D_t$ is diagonal channel-wise decay. This structure accommodates targeted editing along different coordinate axes for reading out and writing in.

1. Efficient Parallel Training: Through the WY transformation, the authors demonstrate that the recurrence—incorporating channel-wise erase and write—admits an efficient matrix product and triangular solve structure, retaining high hardware and training efficiency in both forward and backward passes.
2. Flexible Reductions: Gated DeltaNet-2 can recover KDA and Gated DeltaNet by appropriately tying its gates and decay, ensuring strict generality over prior channel- and scalar-gated variants.

Hybrid Model Integration

Gated DeltaNet-2 is deployed both as a pure recurrent stack and as the recurrent backbone in hybrid architectures with Sliding-Window Attention (SWA). The hybrid block sequence alternates the recurrent mixer with SWA and MLP layers, ensuring local evidence integration while the recurrent state manages globally persistent memory.

Empirical Results and Analysis

Language Modeling and General Reasoning

On multiple language modeling and commonsense reasoning benchmarks, Gated DeltaNet-2 demonstrates the strongest aggregate performance under controlled parameter and state-size settings versus Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants. In both recurrent-only and hybrid contexts, Gated DeltaNet-2 outperforms others on perplexity and accuracy metrics, confirming that gains arise from improved memory updates rather than increased memory capacity.

Long-Context Retrieval and Interference

The most significant improvements occur on long-context in-context retrieval tasks, specifically on RULER synthetic benchmarks. Gated DeltaNet-2 maintains high recall under severe association interference scenarios—especially in the multi-key setting, where competing associations are encoded in a fixed-state. This attests to the practical utility of decoupled erase and write gates for minimizing destructive interference and sustaining reliable content retrieval across extended sequences.

Real-World Benchmark Evaluation

Across several real-world extraction and question-answering datasets, Gated DeltaNet-2 achieves the best or near-best retrieval accuracy in both recurrent and hybrid model forms. The largest relative improvements are observed in tasks emphasizing robust memory recovery amid noise and distracting evidence. The hybrid variant maintains state-of-the-art recall as sequence lengths scale, confirming that SWA integration does not compromise the underlying model's long-context capabilities.

Ablation and Efficiency Studies

Ablation experiments show that both channel-wise erase and write gates contribute to the observed gains, though the erase gate carries more weight in the context of memory preservation and interference reduction. Prohibiting channel-wise structure in either gate (i.e., reverting to scalar gating) causes consistent degradation on both modeling and retrieval tasks. Throughput analysis confirms that Gated DeltaNet-2 retains near-linear scaling in hardware efficiency, with a negligible constant overhead attributable to its finer-grained gating mechanisms.

Implications and Future Directions

Gated DeltaNet-2 addresses a critical bottleneck in fixed-size recurrent sequence models: the destructive coupling between forgetting (erasing) and learning (writing) in compressed memory states. By enabling axis-aligned, channel-wise control over the content lifecycle, it minimizes association interference and supports more robust retrieval over long contexts. The architecture preserves the efficiency necessary for scalable training and dense deployment. Future work may explore dynamic adaptivity in gating (beyond channel-wise structure), gating policies informed by external memory or hierarchical cues, and broader applications beyond language tasks, such as vision transformers or time-series modeling where long-term memory fidelity under fixed-capacity constraints is essential. Integrating such decoupled gating with further advances in state-space models or leveraging hybrid recurrent-attention backbones may yield further improvements in efficiency, generalization, and scalability.

Conclusion

Gated DeltaNet-2 generalizes and unifies prior delta-rule linear attention models by decoupling the erase and write operations with independent channel-wise gating, significantly advancing the capacity of linear recurrent architectures to efficiently store, retrieve, and update associations in long sequences. It achieves superior empirical results in language modeling and retrieval under fixed memory, and establishes a new standard for efficient, interference-resistant recurrent attention layers in large-scale sequence modeling tasks (2605.22791).