Sparse Delta Memory (SDM)

• Sparse Delta Memory (SDM) is a sparse, explicit-memory extension of Gated DeltaNet that replaces dense state updates with selective reads and writes to boost hidden-state capacity.
• It leverages Product Key Memory style addressing and gated delta updates to efficiently decouple memory size from per-token computational cost.
• Empirical results demonstrate that SDM improves long-context retrieval and in-context learning, outperforming Gated DeltaNet and Full Attention on various language modeling tasks.

Sparse Delta Memory (SDM) is a sparse, explicit-memory extension of Gated DeltaNet designed to increase the hidden-state capacity of gated linear RNNs by orders of magnitude without increasing per-token FLOPs. In the formulation introduced in "Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity" (Cabannes et al., 8 Jul 2026), SDM replaces the dense key-value outer product of Gated DeltaNet with sparse reads and writes to a large memory table, using sparse addressing to decouple state size from compute. The reported motivation is that long-context recall in linear attention models is bounded by hidden-state size; under isoFLOP and identical-parameter comparisons, enlarging the effective state through sparse memory improves in-context learning and long-context retrieval, and learning the initial memory state further turns the memory into a parametric store (Cabannes et al., 8 Jul 2026).

1. Problem formulation and motivation

The starting point for SDM is the associative-memory interpretation of linear attention. In that view, the recurrent state is

$$\mathbf{M}_t = \sum_{i=1}^{t} \phi(\mathbf{k}_i)\,\mathbf{v}_i^\top \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}, \qquad \mathbf{z}_t = \sum_{i=1}^{t} \phi(\mathbf{k}_i),$$

with readout

$$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$

This yields fixed compute per token because the state size is fixed. The reported limitation is that long-context recall is fundamentally constrained by that fixed state size: if the state is small, the model cannot retain many distinct associations (Cabannes et al., 8 Jul 2026).

Gated DeltaNet addresses part of this problem through a delta-rule update with decay,

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$

where $\alpha_t \in (0,1)$ is a forget gate and $\beta_t \in [0,1]$ is an input gate. The subtraction of the currently retrieved value before writing is described as reducing interference and bounding memory growth. However, the state remains dense, $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$, so enlarging the state increases per-token cost as $O(d_{\text{qk}} \times d_{\text{v}})$ (Cabannes et al., 8 Jul 2026).

SDM is introduced precisely to remove that coupling. Its key insight is that the Gated DeltaNet update can be sparsified: rather than updating every entry of a dense matrix, the model reads and writes only a small number of memory slots. This suggests that the central obstacle for fixed-state linear models is not recurrence alone, but dense state updates.

2. Architecture and memory operations

SDM replaces the dense recurrent matrix with an explicit slot table,

$$\mathbf{M}_t \in \mathbb{R}^{N \times d_{\text{v}}},$$

where $N$ is the number of memory slots and can be much larger than $d_{\text{qk}}$. The slot table is accessed through Product Key Memory style addressing, with $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$0 write slots and $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$1 read slots selected per token (Cabannes et al., 8 Jul 2026).

For each head and token, the layer executes four operations. First, it forms projected read and write keys from $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$2 using

$$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$3

These are split into two halves, producing write-key halves $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$4 and read-query halves $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$5. The outer sum

$$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$6

scores the full slot space, and the same construction is used for queries. The selected indices are $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$7 for writes and $$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$8 for reads. The efficiency claim relies on the identity

$$\mathbf{y}_t = \frac{\mathbf{M}_t^\top \phi(\mathbf{q}_t)} {\mathbf{z}_t^\top \phi(\mathbf{q}_t)} = \frac{\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle\, \mathbf{v}_i} {\sum_{i\le t} \langle \phi(\mathbf{q}_t),\phi(\mathbf{k}_i)\rangle}.$$9

so the full $$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$0-way score matrix need not be materialized (Cabannes et al., 8 Jul 2026).

Second, SDM applies a gated delta write to the selected write slots. For each $$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$1,

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$2

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$3

Unselected slots remain unchanged:

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$4

The gates and values are defined by

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$5

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$6

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$7

Third, the layer performs a sparse read from the selected read slots:

$$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$8

Fourth, the retrieved state is normalized with RMS-Norm, gated elementwise by $$\mathbf{M}_t \leftarrow \alpha_t \mathbf{M}_{t-1} + \beta_t \mathbf{k}_t \left(\mathbf{v}_t - \alpha_t \mathbf{M}_{t-1}^\top \mathbf{k}_t\right)^\top,$$9, and projected with $\alpha_t \in (0,1)$0 to produce the layer output $\alpha_t \in (0,1)$1 (Cabannes et al., 8 Jul 2026).

Operationally, SDM is therefore a sparse slot-memory layer with local recurrence, gated overwrite, and explicit separation between addressing and stored value vectors.

3. Relation to Gated DeltaNet and scaling properties

The relation between SDM and Gated DeltaNet is exact in a particular limit. The reported construction recovers Gated DeltaNet when $\alpha_t \in (0,1)$2, $\alpha_t \in (0,1)$3, and the sparse key values $\alpha_t \in (0,1)$4 form a dense vector; the stated exception is that Gated DeltaNet includes 1D convolutions on $\alpha_t \in (0,1)$5, which SDM does not (Cabannes et al., 8 Jul 2026).

This limiting argument is central to the model’s interpretation. Gated DeltaNet updates the entire dense state for every token; SDM stores the same kind of delta-rule dynamics in an explicit table and updates only a sparse subset. The model’s scaling behavior then follows from two cost expressions. For Gated DeltaNet, per-token cost is

$\alpha_t \in (0,1)$6

equivalently

$\alpha_t \in (0,1)$7

For SDM, the cost is

$\alpha_t \in (0,1)$8

equivalently

$\alpha_t \in (0,1)$9

which is independent of $\beta_t \in [0,1]$0 (Cabannes et al., 8 Jul 2026).

The reported isoFLOP comparison fixes

• $\beta_t \in [0,1]$1,
• $\beta_t \in [0,1]$2,
• $\beta_t \in [0,1]$3,
• $\beta_t \in [0,1]$4,

and notes that the top-$\beta_t \in [0,1]$5 Product Key Memory addressing contributes less than 1% of layer FLOPs. Under a parameter constraint, fewer heads do not increase FLOPs but do increase memory size, so the reported design prefers fewer heads to maximize state capacity (Cabannes et al., 8 Jul 2026).

The total memory size per layer is given as

$\beta_t \in [0,1]$6

With $\beta_t \in [0,1]$7, the state size can scale roughly like $\beta_t \in [0,1]$8. A plausible implication is that SDM converts width into memory capacity more aggressively than dense recurrent-state designs.

4. Training configuration and evaluation protocol

The reported training setup uses a hybrid architecture in which short-range layers are Multi-Head Attention with Sliding Window Attention, while long-range or global layers are one of Full Attention, Gated DeltaNet, Mamba2, or SDM. The ratio is 3:1 short:long, with Sliding Window Attention window size 128, GQA with group size 2, gated attention output, gated MLP with SiLU, and RoPE with $\beta_t \in [0,1]$9 (Cabannes et al., 8 Jul 2026).

Within SDM itself, the stated default settings are $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$0 and $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$1, together with softmax-normalized read and write activations. Initialization uses

• $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$2 for the forget gate, and
• $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$3.

Pretraining uses 8192-token sequences, AdamW with $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$4 and $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$5, a warmup-stable-decay schedule, gradient clipping at 1.0, and the same learning rates across architectures at a given scale. Long-context fine-tuning for the 1.4B and 8B models uses 128k-token sequences. The scaling ladder is trained at a compute budget of 160 tokens per parameter, explicitly described as inference-oriented rather than Chinchilla-optimal (Cabannes et al., 8 Jul 2026).

Methodologically, the comparison is arranged so that FullAttn, GDN, and SDM share hyperparameters except for the number of heads $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$6. This is important because the reported claims are framed around isoFLOP and identical-parameter comparisons rather than unrestricted tuning.

5. Empirical performance

Across the reported scaling ladder, SDM outperforms Gated DeltaNet at every scale and shows highly predictable scaling with $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$7. At 8B scale, it is reported to beat both Gated DeltaNet and even Full Attention on training loss. The article’s interpretation is that SDM is not restricted to retrieval benchmarks: it also improves the base language-modeling objective (Cabannes et al., 8 Jul 2026).

For in-context learning and common-knowledge or reasoning evaluation, the 1.4B and 8B models improve average accuracy and validation NLL relative to Gated DeltaNet. At 1.4B, SDM is reported to beat GDN on DCLM NLL, average accuracy, and many reasoning tasks including HellaSwag, WinoGrande, ARC, PIQA, CommonsenseQA, TriviaQA, MMLU, and HumanEval+. At 8B, the reported table gives SDM lower NLL than both GDN and FullAttn, with stronger gains on knowledge-heavy tasks including CommonsenseQA, TriviaQA, NaturalQuestions, HellaSwag, and WinoGrande (Cabannes et al., 8 Jul 2026).

The clearest quantitative advantage appears on RULER long-context retrieval, where the paper reports the best fixed-state performance:

Model size	SDM RULER average	GDN RULER average
1.4B	31.2	20.0
8B	50.2	34.2

SDM improves on 6/6 evaluated RULER subsets. The reported nuance is that Full Attention still wins on some of the hardest retrieval cases, especially multi-key retrieval, because its KV cache is unbounded (Cabannes et al., 8 Jul 2026).

On long code documents, SDM is reported to achieve lower perplexity than GDN and Mamba2, with the advantage growing with context length. The long-context-layer analysis is stated to continue improving out to 1M tokens. This suggests that the sparse explicit memory is beneficial in both retrieval-style and extended autoregressive settings.

6. Learned initial memory, implementation strategy, and limitations

A distinctive feature of the reported SDM formulation is the treatment of the initial memory state $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$8 as a learned parameter rather than a null tensor. In that setting, the initial state becomes a parametric memory: it adds no inference FLOPs, can store useful pretraining knowledge, and improves code NLL, accuracy, and RULER recall. The reported ablations state that SDM with null $$\mathbf{M}_t \in \mathbb{R}^{d_{\text{qk}} \times d_{\text{v}}}$$9 still beats GDN, but learned $O(d_{\text{qk}} \times d_{\text{v}})$0 provides an additional boost; the primary driver of the long-context advantage remains larger memory size rather than initialization alone (Cabannes et al., 8 Jul 2026).

The implementation uses a chunked algorithm based on the WY representation, combining intra-chunk parallel computation with chunkwise recurrent computation across chunks. For sparse slot overlap, the model uses a sparse inner-product kernel based on a two-pointer merge, exploiting the fact that slot indices are sorted after Product Key Memory top-$O(d_{\text{qk}} \times d_{\text{v}})$1 selection. The reported sparse interaction matrix is

$O(d_{\text{qk}} \times d_{\text{v}})$2

which is stated to scale like $O(d_{\text{qk}} \times d_{\text{v}})$3 rather than dense $O(d_{\text{qk}} \times d_{\text{v}})$4 (Cabannes et al., 8 Jul 2026).

The principal limitation identified in the reported implementation is hardware efficiency. Although FLOPs are similar to GDN, the explicit memory lives in HBM rather than fast SRAM, so the current kernel is slower than highly optimized dense GDN kernels. At 8B scale, SDM training throughput is reported as about 1.49× slower than GDN even though the memory state is about 4000× larger; for 1.4B inference decode, SDM is about 10% slower than GDN but much faster than FullAttn (Cabannes et al., 8 Jul 2026).

In that sense, Sparse Delta Memory is best understood as a capacity-scaling architecture for fixed-compute recurrent memory. Its defining claim is not merely that sparsity reduces cost, but that sparsity allows explicit memory state to become large enough to matter for long-context recall, in-context learning, and learned parametric storage under a constant-compute regime.