---
title: Channel-wise Erase and Write Gates
url: https://www.emergentmind.com/topics/channel-wise-erase-and-write-gates
type: topic
---

# Channel-wise Erase and Write Gates

Channel-wise erase and write gates are gating mechanisms that enable fine-grained, per-channel modulation of memory updates in recurrent linear attention architectures. Originally introduced in the Gated DeltaNet-2 framework, these gates separately control how existing memory content is erased (on the key side) and how new information is written (on the value side), addressing limitations of previous scalar-gated models. By decoupling erase and write operations and promoting channel-level specificity, channel-wise gates provide enhanced interference robustness and memory fidelity, particularly for long-context sequence processing [2605.22791].

## 1. Mathematical Formulation

At each timestep \(t\), Gated DeltaNet-2 computes two channel-wise gates from the token representation \(x_t\):
- **Erase gate:** \(b_t = \sigma(W_b x_t) \in (0,1)^{d_k}\), with \(W_b \in \mathbb{R}^{d_k \times d_{\rm model}}\), applies a sigmoid activation on a learned projection.
- **Write gate:** \(w_t = \sigma(W_w x_t) \in (0,1)^{d_v}\), with \(W_w \in \mathbb{R}^{d_v \times d_{\rm model}}\), similarly projected and passed through sigmoid.

These control, respectively, which key-side channels are erased and which value-side channels are written. Additionally, a channel-wise decay vector \(a_t = \exp(g_t) \in (0,1]^{d_k}\) modulates decay, where \(g_t\) is from a separate projection and activation.

## 2. Fast-Weight Memory Update and Dynamics

Gated DeltaNet-2 maintains a fast-weight state \(S_t \in \mathbb{R}^{d_k \times d_v}\). Its update comprises:

- **Decay:** \(\bar{S}_t = \mathrm{Diag}(a_t) S_{t-1}\)
- **Edit:** \(S_t = \bar{S}_t + k_t \left(z_t - \bar{S}_t e_t\right)^T\), where
    - \(e_t = b_t \odot k_t\) (erase vector),
    - \(z_t = w_t \odot v_t\) (write vector), 
    - \(k_t \in \mathbb{R}^{d_k}\) and \(v_t \in \mathbb{R}^{d_v}\) are normalized key and value.

This can be organized as:
\[
S_t = (I - k_t e_t^T) \mathrm{Diag}(a_t) S_{t-1} + k_t z_t^T
\]
This structure enables rank-one erasure and writing, with per-channel modulation.

Equivalently, this update optimizes the local least-squares objective:
\[
L_t(S) = \|S - \mathrm{Diag}(a_t) S_{t-1}\|_F^2 - 2\langle S k_t,\ z_t - S e_t \rangle
\]
with stationary point where \(\nabla_S L_t = 0\).

## 3. Relation to Prior Delta-Rule Models

Channel-wise gating generalizes prior delta-rule attention variants:
- If both gates reduce to the same scalar \(\beta_t\), \(b_t = \beta_t \mathbf{1}_{d_k}\), \(w_t = \beta_t \mathbf{1}_{d_v}\), the update becomes Kimi Delta Attention (KDA):
  \[
  S_t = (I - \beta_t k_t k_t^T) \mathrm{Diag}(a_t) S_{t-1} + \beta_t k_t v_t^T
  \]
- If the decay also reduces to a scalar \(a_t = \alpha_t \mathbf{1}_{d_k}\), the formulation matches Gated DeltaNet:
  \[
  S_t = \alpha_t (I - \beta_t k_t k_t^T) S_{t-1} + \beta_t k_t v_t^T
  \]
Thus, channel-wise erase and write gates subsume earlier scalar-gated approaches, enabling strictly more flexible memory control by allowing independent, channel-wise gating.

## 4. Efficient Chunkwise WY Algorithm

For efficient GPU training, updates are performed over chunks of length \(C\):
- Log-decay terms \(G_r\) and exponentials \(Y_r\) are accumulated.
- Key and erase vectors are normalized by decay: \(k_r \leftarrow Y_{r-1} \odot k_r\), \(e_r \leftarrow Y_{r-1} \odot (b_r \odot k_r)\), \(z_r \leftarrow w_r \odot v_r\).
- These yield block matrices \(K, E, Z\) for all \(r\).
- The lower-triangular score matrix, \(T = \mathrm{tril}(E K^T, -1)\), forms a system \(A = (I + T)^{-1}\).
- WY auxiliaries \(Y_{\rm aux}, U_{\rm aux}\) are computed, and final chunk updates are performed with small dense multiplications and diagonal scaling:
  \[
  S_{\rm end} = \mathrm{Diag}(Y_C) S_{\rm start} + K_{\rm tail}(U_{\rm aux} - Y_{\rm aux} S_{\rm start})
  \]
This algorithm, with fixed \(C\), maintains \(O(C d_k d_v)\) complexity and is efficiently mapped to tensor computation.

## 5. Gate-Aware Backward Propagation

Backpropagation must track per-channel gradients through the WY products. Gradients of erase and write gates must be integrated inside the accumulation of \(A\) rather than via post-scaling with a single scalar. For upstream \(\delta U_{\rm aux}, \delta Y_{\rm aux}, \delta A\) the process involves:
- Updating \(dA\) with contributions from \(\delta U_{\rm aux} Z^T\) and \(\delta Y_{\rm aux} E^T\)
- Propagating \(dZ = A^T \delta U_{\rm aux}\), \(dE = A^T \delta Y_{\rm aux}\)
- Adjusting \(dT = -\mathrm{tril}(A^T dA A^T)\)
- Computing per-channel gradients for \(W, V, B, K\) as required.

Absence of per-channel gradients during WY accumulation results in incorrect parameter updates. Thus, channel-wise gate information must be preserved throughout the backward pass.

## 6. Empirical Results and Significance

Gated DeltaNet-2 models featuring channel-wise erase and write gates, at the 1.3B parameter scale with 100B FineWeb-Edu tokens, exceed the performance of Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across several tasks:
- WikiText language modeling perplexity improved from approximately 16.8 to 15.9.
- Zero-shot reasoning accuracy increased from ∼52.3% to 53.1%.
- Synthetic retrieval (S-NIAH-2 @4K) increased from 89.8% to 93.0%; multi-key (MK-NIAH-1 @4K) from 31.8% to 37.8%.
- Real-world average recall rose by ∼1.2–1.5 points.

Ablation studies reveal that collapsing either gate to a scalar consistently degrades performance, with the erase gate yielding the largest recall improvements on long-context benchmarks. Separate channel-wise control of erasure and writing is consequently essential for robust, interference-resistant, long-range fast-weight memory [2605.22791].

Source: https://www.emergentmind.com/topics/channel-wise-erase-and-write-gates