Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Abstract: Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

• The paper introduces SDM, a novel architecture that scales linear RNN state size by using sparse memory addressing via Product Key Memory.
• Methodologically, SDM maintains constant FLOPs per token while dramatically increasing capacity, leading to superior in-context learning and retrieval performance.
• Empirical evaluations demonstrate that SDM outperforms dense RNNs and transformers on language modeling benchmarks, achieving lower loss and higher accuracy.

Sparse Delta Memory: A Paradigm for Scaling the State of Linear RNNs Through Sparsity

Introduction

Sparse Delta Memory (SDM) introduces an architecture designed to overcome the state size bottleneck that has historically limited the recall and context capabilities of linear RNNs and linear attention models. The approach leverages sparse addressing through Product Key Memory (PKM) to expand the state size by orders of magnitude without a corresponding increase in computational cost, directly addressing the memory-state tradeoff in long-context sequence modeling. By design, SDM can maintain constant FLOPs per token while dramatically increasing the effective capacity of the model's recurrent state, yielding strong improvements in both in-context learning and long-context retrieval compared to prior architectures such as softmax-transformers and dense linear RNNs (e.g., GDN, Mamba2).

Architectural Contributions

SDM builds upon the Gated DeltaNet (GDN) linear attention mechanism, which uses a gated, delta-rule-based update to its recurrent state. The key architectural advances in SDM are:

• Sparse Keyed Memory Table: Replacing the dense key-value outer product with sparse reads and writes to a large explicit memory, indexed via PKM. Each access involves only a small set of memory slots, chosen via a top-$k$ selection among $N$ possible locations.
• IsoFLOP Constraint: SDM is explicitly designed to match the parameter count and FLOPs of dense GDN baselines, ensuring improvements derive solely from scale efficiencies in memory.
• Learned Initial State $\mathbf{M}_0$: The initial state of the SDM memory is learnable, enabling the storage of common knowledge and persistent facts, further reducing loss and improving performance across general reasoning and code tasks.

This design allows for an unprecedented increase in state size (e.g., from hundreds of kB in GDN to several GB in SDM), with memory accesses remaining sublinear in $N$ owing to the PKM indexing strategy.

Empirical Results and Scaling Laws

The empirical section demonstrates that SDM delivers consistently superior performance on language modeling and in-context tasks as model scale increases.

• Scaling Law Superiority: Across model sizes (from 257M to 8.1B parameters, compute budgets of 110B–1.1T tokens), SDM yields lower training loss and validation NLL than GDN with identical FLOPs and parameters. At the 8B parameter scale, SDM not only outperforms GDN but also surpasses a full-attention transformer baseline in loss-per-FLOP efficiency.

  Figure 2: SDM achieves lower training loss than GDN at all scales and outperforms full attention at the largest scale within a constrained FLOP budget.

• Short-context and Reasoning Tasks: SDM outperforms GDN and, in many cases, even full attention on short-context benchmarks, due to its large learnable memory state, as evidenced by DCLM NLL and accuracy improvements on diverse reasoning benchmarks.
• Long-context Retrieval: On the RULER benchmark, SDM achieves the highest recall among fixed-state architectures at both 1.4B and 8B scale, markedly outperforming GDN on all evaluated tasks. The performance gap widens with increasing sequence length, underscoring SDM's scalability and effective context retention.

  Figure 1: On RULER tasks, SDM maintains near-perfect recall even as sequence length grows, where GDN and full attention degrade rapidly beyond their training context windows.

  

  Figure 5: At 8B scale, SDM with post-training maintains strong recall, sometimes surpassing full attention on complex long-context retrieval tasks.

• Code Perplexity: SDM demonstrates lower perplexity across all token positions, including ultra-long code documents (up to 1M tokens), outperforming Mamba2 and GDN by a substantial margin.

Ablation Analysis

Ablation studies dissect the drivers of SDM efficacy:

• Role of Memory Size: State capacity is the main factor; increasing the number of PKM slots consistently improves both language modeling loss and contextual recall, confirming the theoretical linkage between state size and long-context modeling capability.
• Learned Initial State: Learning $\mathbf{M}_0$ offers further gains, but the primary improvements stem from memory size itself, distinguishing SDM from dense RNNs where learned initialization is ineffectual given small state dimensions.

Figure 6: Ablation on learned initial state shows substantial benefits for SDM (not GDN), attributing gains primarily to memory scale.

• Adaptive Access Patterns: Analysis of write/read key distributions shows that access patterns adapt to available capacity, with writes being more peaked and localized, and reads more uniform and diffuse. The architecture exhibits compositional access behavior, further leveraging memory sparsity.

Figure 7: Distribution of write keys indicates moderate peakedness, while read key distributions are more uniform, showing adaptive utilization of the memory capacity.

Memory Utilization and Efficiency

SDM achieves high utilization of its memory space during training, with all memory slots used for reads and a growing proportion used for writes as training progresses.

Figure 8: During training, SDM maintains highly dynamic memory access, with a wide range in forget/input gate values, and effective utilization of unique memory slots.

Despite the large state size, SDM's performance degrades gracefully as the number of slots is reduced, and memory requirements can be mitigated via quantization techniques in extreme-scale or resource-constrained scenarios.

The main practical limitation remains hardware efficiency: While SDM matches GDN in theoretical FLOPs, the required bandwidth for large memory access is not yet fully hardware-optimized, resulting in modest overheads (e.g., training throughput $\sim$1.5x slower than GDN at 8B scale, but still $6\times$ faster than full-attention decoding).

Architectural Extensions and Hybridization

Experiments with hybrid architectures blending Sliding Window Attention (SWA), FullAttn, and SDM layers provide evidence for complementarity between SDM and full attention. At large scale, an interleaved SDM/FullAttn model outperforms pure SDM or FullAttn variants on scaling laws, suggesting the potential for further performance/efficiency tradeoff optimization.

Implications and Future Prospects

SDM demonstrates that the scaling limitations of linear RNNs and linear attention mechanisms are not inherent, but rather architectural. By leveraging sparse, content-addressable memory, it is possible to attain recall and in-context learning capabilities competitive with, or superior to, large transformers while maintaining constant per-token compute.

Theoretical Impact: The results challenge the paradigm that full-softmax attention is necessary for strong long-context modeling, and provide new scaling laws for memory-augmented neural architectures.

Practical Impact: SDM offers a promising route to models capable of long-sequence inference (e.g., million-token contexts) with bounded memory and compute per token, directly addressing the bottlenecks faced by agentic, retrieval-augmented, or online learning systems.

Technical Outlook: Further progress will hinge on engineering memory access kernels to close the bandwidth gap; integrating SDM-like memory into multimodal or sensorimotor models; and exploring explicit memory compression/growth strategies as model and data scale beyond current benchmarks.

Conclusion

Sparse Delta Memory substantively advances the state of the art in sequential modeling by scaling the state of linear RNNs through sparsity, overcoming foundational limitations of prior approaches. Its empirical superiority across a range of benchmarks suggests that sparsely indexed, large-capacity memory is a critical ingredient for efficient, scalable, and effective long-context neural sequence modeling, with broad implications for the future of large language and agent systems (2607.07386).