Pair-Addressed Memory for Long Context Linear Attention
This presentation explores the Kronecker Delta Rule, a novel approach to recurrent linear attention that treats memory as a collection of scalar associations indexed by joint key-value pairs. Unlike traditional methods that revise entire value vectors, this bilinear correction minimizes interference by updating only along the current pair direction. The resulting KronLSTM architecture achieves bounded memory norms, exact least-squares convergence under stochastic conditions, and strong empirical performance on language modeling, long-context retrieval, and real-world recall tasks at scale.Script
Linear attention models store a fixed-size memory matrix, but how do you revise one association without corrupting all the others? The Kronecker Delta Rule solves this by correcting scalar bilinear associations rather than entire vectors.
Traditional delta rules correct the retrieved value vector, causing interference with any association sharing the same key direction. KronLSTM instead corrects the scalar quantity retrieved by the bilinear probe, so interference requires overlap in both the key and the value.
The architecture separates three mechanisms: a diagonal forget gate per key channel, a scalar erase gate controlling how much of the current association to remove, and a learned scalar write gate determining the target strength. Decoupling erase and write turns out to be essential for retrieval performance.
Under independent input pairs, the expected memory state converges geometrically to the least-squares solution with no finite step-size bias. The full stochastic process has a unique stationary distribution whose covariance scales linearly with step size, and the expected memory norm remains bounded independently of sequence length.
On natural-text needle retrieval, KronLSTM's advantage over the DeltaNet baseline grows with context length. At 32,000 tokens after extension training, it leads on every natural-text task, though both models still degrade substantially at that scale. The results suggest reduced interference improves the degradation curve but does not eliminate the fundamental capacity limits of fixed-size recurrent memory.
KronLSTM reframes recurrent memory as a collection of up to d squared scalar associations, each indexed by a key-value pair and retrieved through a bilinear probe. You can explore the full architecture, proofs, and extended context results at EmergentMind.com, where you can also create your own research video presentations.
