KronLSTM: Sparsity and speed in comcise

• The result achieves state-of-the-art performance in linear models on the Pure-linear natural estimator of average of six commonsense tasks with an average of 59.72 and LAMBDA text perplexity average of 11.72.
• The model now uses a Kronecker Delta Rule to correct memory associations selectively, only changing along the key-value direction which saves the redundancy.
• It allows for efficient chunkwise-parallel computation and maintains high throughput on long contexts

KronLSTM is a recurrent linear-attention architecture introduced as an extension of the xLSTM family. Its associative memory is a matrix updated by the Kronecker Delta Rule, which revises a scalar key–value association along a single rank-one direction. The architecture combines this update with key-direction forget gating, scalar erase and write gates, gated readout, normalized query–key–value projections, and chunkwise-parallel computation. Unlike conventional softmax attention, KronLSTM maintains a fixed-size recurrent state whose memory footprint is independent of sequence length. The name should be distinguished from earlier uses of LSTM-structured Gaussian processes and recurrent kernel machines: those works do not introduce the KronLSTM architecture or Kronecker-factorized recurrent updates (Al-Shedivat et al., 2016, Liang et al., 2019, Shirdel et al., 2021). The architecture described by the name KronLSTM is introduced in "Kronecker Delta Rule for Long Context Linear Attention" (Hartl et al., 19 Oct 2026).

1. Conceptual origin and scope

KronLSTM addresses the memory-update problem in recurrent linear attention. In a standard causal softmax Transformer, attention has quadratic sequence-length cost and the key–value cache grows linearly with context. Linear attention replaces this cache with a fixed-size matrix memory $\BS_t\in\mathbb{R}^{d_v\times d_k}$, producing linear computational complexity in sequence length and constant-memory recurrent inference.

The basic Hebbian linear-attention update is

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$

with key $\Bk_t\in\mathbb{R}^{d_k}$, value $\Bv_t\in\mathbb{R}^{d_v}$, query $\Bq_t\in\mathbb{R}^{d_k}$, and readout

$\Bh_t=\BS_t\Bq_t.$

The rank-one outer-product write $\Bv_t\Bk_t^\top$ accumulates associations, but it does not explicitly revise an existing association. Later writes can bury earlier information, and overlapping keys produce crosstalk. Forget mechanisms can control memory growth, but long retention requires forget rates close to one.

The standard Delta Rule improves revision by correcting the vector retrieved at the current key:

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$

This update imposes a vector-valued constraint, $\BS\Bk_t=\Bv_t$, and its correction direction depends on the current memory. Consequently, it can modify multiple value directions whenever their keys overlap with $\Bk_t$.

KronLSTM instead treats the key–value pair as a scalar associative probe. Its memory stores the bilinear quantity

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$0

rather than requiring the entire retrieved vector $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$1 to equal $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$2. This produces a more selective update: interference with an existing association depends on both key overlap and value overlap.

Earlier papers supplied in connection with the name KronLSTM concern different constructions. "Learning Scalable Deep Kernels with Recurrent Structure" places a Gaussian process over deterministic LSTM sequence embeddings and uses Kronecker, Toeplitz, circulant, and structured-kernel interpolation algebra for GP scalability (Al-Shedivat et al., 2016). "Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods" derives RKM-LSTM and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$3-gram RKM-LSTM models from recurrent kernel machines, but contains no Kronecker products or Kronecker-factorized recurrent matrices (Liang et al., 2019). "Deep Learning with Kernel Flow Regularization for Time Series Forecasting" applies a kernel-flow auxiliary loss to ordinary LSTM hidden representations, without modifying recurrent matrices (Shirdel et al., 2021).

2. The Kronecker Delta Rule

Let $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$4 be a scalar target and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$5 an erase strength. The Kronecker Delta Rule is

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$6

with residual

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$7

The update can be separated into erase and write components:

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$8

KronLSTM uses a decoupled parameterization in which $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$9 controls erasure and $\Bk_t\in\mathbb{R}^{d_k}$0 controls writing:

$\Bk_t\in\mathbb{R}^{d_k}$1

The rule is one gradient-descent step on the scalar loss

$\Bk_t\in\mathbb{R}^{d_k}$2

Its matrix gradient is

$\Bk_t\in\mathbb{R}^{d_k}$3

Thus, the update corrects one scalar constraint,

$\Bk_t\in\mathbb{R}^{d_k}$4

rather than the $\Bk_t\in\mathbb{R}^{d_k}$5 scalar constraints imposed by vector retrieval.

The Kronecker interpretation follows by defining

$\Bk_t\in\mathbb{R}^{d_k}$6

Because

$\Bk_t\in\mathbb{R}^{d_k}$7

the vectorized update becomes

$\Bk_t\in\mathbb{R}^{d_k}$8

This is the Widrow–Hoff or LMS delta rule applied to the Kronecker feature $\Bk_t\in\mathbb{R}^{d_k}$9. The homogeneous transition changes the memory only along the single direction $\Bv_t\in\mathbb{R}^{d_v}$0 in the $\Bv_t\in\mathbb{R}^{d_v}$1-dimensional vectorized memory space. Its eigenvalue along $\Bv_t\in\mathbb{R}^{d_v}$2 is

$\Bv_t\in\mathbb{R}^{d_v}$3

while directions orthogonal to $\Bv_t\in\mathbb{R}^{d_v}$4 have eigenvalue $\Bv_t\in\mathbb{R}^{d_v}$5.

The non-expansive range is

$\Bv_t\in\mathbb{R}^{d_v}$6

For

$\Bv_t\in\mathbb{R}^{d_v}$7

the targeted residual contracts. Repeated presentation of the same pair gives

$\Bv_t\in\mathbb{R}^{d_v}$8

The Newton step

$\Bv_t\in\mathbb{R}^{d_v}$9

sets the residual to zero in one update.

3. Selective revision and interference

For a previously stored pair $\Bq_t\in\mathbb{R}^{d_k}$0, the change in its scalar readout under a current update is

$\Bq_t\in\mathbb{R}^{d_k}$1

Interference therefore depends on the product of value overlap and key overlap. If either the keys or the values are orthogonal, the earlier association is unchanged. If the current association is already correct, $\Bq_t\in\mathbb{R}^{d_k}$2, and the update produces no interference.

This differs from the main alternatives as follows:

• Hebbian linear attention: interference is governed primarily by key overlap and the memory is accumulative.
• Standard DeltaNet: vector retrieval is corrected, with interference governed primarily by key overlap.
• KronLSTM: scalar pair associations are corrected, and interference requires both key and value overlap.

The rank-one correction is also minimal in a precise geometric sense. Let $\Bq_t\in\mathbb{R}^{d_k}$3 be any correction satisfying

$\Bq_t\in\mathbb{R}^{d_k}$4

The Newton correction is

$\Bq_t\in\mathbb{R}^{d_k}$5

For every admissible correction,

$\Bq_t\in\mathbb{R}^{d_k}$6

with equality only for $\Bq_t\in\mathbb{R}^{d_k}$7. The same correction is optimal for every unitarily invariant matrix norm, including Schatten-$\Bq_t\in\mathbb{R}^{d_k}$8 norms. It is also the unique correction whose column space lies in $\Bq_t\in\mathbb{R}^{d_k}$9 and whose row space lies in $\Bh_t=\BS_t\Bq_t.$0.

The scalar nature of the memory is an important qualification. KronLSTM has a memory space of dimension $\Bh_t=\BS_t\Bq_t.$1, but this should not be interpreted as $\Bh_t=\BS_t\Bq_t.$2 independent key-to-vector retrievals. Its basic operation stores scalar bilinear associations rather than directly enforcing $\Bh_t=\BS_t\Bq_t.$3.

4. Architecture and recurrent update

KronLSTM embeds the Kronecker Delta Rule in an xLSTM-style sequence mixer. The equations are defined per attention head. Given layer input $\Bh_t=\BS_t\Bq_t.$4, the model constructs keys, queries, and values from separate linear projections. Each projection is followed by a short causal depthwise convolution, a SiLU activation, and $\Bh_t=\BS_t\Bq_t.$5-normalization:

$\Bh_t=\BS_t\Bq_t.$6

The normalization gives approximately

$\Bh_t=\BS_t\Bq_t.$7

The matrix memory is read using

$\Bh_t=\BS_t\Bq_t.$8

Forget gate

The forget gate acts independently on key directions:

$\Bh_t=\BS_t\Bq_t.$9

Its preactivation is

$\Bv_t\Bk_t^\top$0

The gate is

$\Bv_t\Bk_t^\top$1

with the exponent applied componentwise. The scalar $\Bv_t\Bk_t^\top$2 is learned per head and broadcast across key dimensions. Because the gate multiplies the memory from the right, $\Bv_t\Bk_t^\top$3 decays key directions independently.

Erase and write gates

The erase and write gates are scalar per head:

$\Bv_t\Bk_t^\top$4

$\Bv_t\Bk_t^\top$5

The gates are decoupled: $\Bv_t\Bk_t^\top$6 determines how much of the current association is erased, whereas $\Bv_t\Bk_t^\top$7 determines how much new scalar target is written.

Memory recurrence

The KronLSTM recurrence is

$\Bv_t\Bk_t^\top$8

or, equivalently,

$\Bv_t\Bk_t^\top$9

The scalar score being corrected is computed on the decayed memory:

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$0

With normalized keys and values and $\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$1,

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$2

placing the update in the non-expansive regime while the forget gate provides additional contraction.

Output processing

The output gate is value-channel-wise:

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$3

The gated readout is passed through per-head RMSNorm and an output projection:

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$4

Unlike the matrix-LSTM component of xLSTM, KronLSTM carries no normalizer state or stabilizer state. Its recurrent state consists only of matrix memories $\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$5.

5. Stability theory and computational properties

The theoretical analysis considers the ungated Kronecker Delta Rule under i.i.d. triples $\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$6, independent sampling, fixed step size $\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$7, and full-rank feature covariance. Define

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$8

$\BS_{t+1} = \BS_t-\beta_t(\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$9

The vector $\BS\Bk_t=\Bv_t$0 is the least-squares memory, and its matrix reshaping is $\BS\Bk_t=\Bv_t$1.

The expected memory recursion is

$\BS\Bk_t=\Bv_t$2

Its unique fixed point is $\BS\Bk_t=\Bv_t$3. Under the step-size condition

$\BS\Bk_t=\Bv_t$4

the expected state converges geometrically to the least-squares memory. The convergence is exactly unbiased: the expected stationary state equals $\BS\Bk_t=\Bv_t$5 without a step-size-dependent bias.

The state process is a time-homogeneous Markov chain on $\BS\Bk_t=\Bv_t$6. Under the stated assumptions, the iterates have a unique stationary distribution $\BS\Bk_t=\Bv_t$7 with finite second moment, and their distribution converges geometrically to it in Wasserstein-2 distance. The stationary mean is

$\BS\Bk_t=\Bv_t$8

With a constant step size, an inconsistent or overloaded association stream generally produces a stationary distribution rather than convergence to a single matrix. The stationary variance is $\BS\Bk_t=\Bv_t$9: larger $\Bk_t$0 enables faster adaptation but produces greater stationary jitter, while smaller $\Bk_t$1 reduces steady-state error but slows adaptation. Diminishing step sizes or iterate averaging recover exact convergence to $\Bk_t$2.

The expected memory norm is uniformly bounded under the assumptions. In the non-expansive regime, $\Bk_t$3, the stationary bound is

$\Bk_t$4

For a process initialized at $\Bk_t$5, the expected squared norm is bounded independently of sequence length. This is an expectation-level statement rather than a deterministic bound on every realization.

Recurrent inference

Each head stores a $\Bk_t$6 matrix. Memory updates and reads require matrix–vector and rank-one operations with approximate per-token cost

$\Bk_t$7

and recurrent memory

$\Bk_t$8

per head. The memory footprint is independent of sequence length.

Chunkwise-parallel computation

The implementation does not explicitly form the $\Bk_t$9-dimensional Kronecker features. Instead, it uses key Gram matrices, value Gram matrices, their Hadamard product, and a unit lower-triangular solve. For a chunk of length $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$00, reported per-head costs include key Gram computation $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$01, value Gram computation $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$02, triangular solve $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$03, memory products $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$04, and intra-chunk output products $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$05.

Over sequence length $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$06, the stated chunkwise complexity is

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$07

Cumulative forget factors are maintained in log space, decay ratios are computed through exponential differences, triangular solves are carried out in FP32, and other products use tensor-core-compatible mixed precision. The kernels were numerically checked against the token-by-token recurrence, with forward and backward computations matching the reference implementation to approximately $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$08 relative error in double precision for the tested configurations.

6. Empirical evaluation and limitations

The primary experiments used FineWeb-Edu with 100 billion tokens, the Llama 2 tokenizer with vocabulary size $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$09, sequence length $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$10, decoder-only models, bf16 mixed precision with FP32 optimizer state, AdamW, a peak learning rate of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$11, 10% linear warmup, cosine decay to $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$12, weight decay $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$13, global gradient clipping at $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$14, and a global batch of 128 sequences. Training used 16 nodes with four NVIDIA GH200 GPUs each.

The 1.5B configuration used 24 layers, hidden size $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$15, 16 heads, and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$16. The 400M configuration used 24 layers, hidden size $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$17, four heads, and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$18.

Language modeling and retrieval

For 1.5B models trained on 100 billion tokens, KronLSTM achieved the highest reported linear-model average across six commonsense tasks:

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$19

In the hybrid setting with sliding-window attention of size 2048, it achieved

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$20

On the real-world recall suite consisting of SWDE, SQuAD, FDA, TriviaQA, Natural Questions, and DROP, KronLSTM obtained a recall average of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$21 in the pure-linear setting and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$22 in the hybrid setting. It was best on three of six tasks in both settings.

On RULER at 4k, 8k, and 16k contexts, KronLSTM was generally the best or second-best linear model except on S1, an artificial task with frequently repeated noise sentences. At 16k, it was the strongest hybrid model on every RULER task. Its advantage increased with context length on several natural-text retrieval tasks.

For long-context extension, 1.5B KronLSTM and Gated DeltaNet-2 models were continued from 8k to 32k sequence length. At 32k, KronLSTM achieved linear Avg-6 of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$23, linear LAMBADA perplexity of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$24, and linear recall average of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$25. In the hybrid setting, it achieved Avg-6 of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$26, LAMBADA perplexity of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$27, and recall average of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$28.

On state-tracking tasks, KronLSTM showed behavior similar to Gated DeltaNet on parity and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$29 permutation composition, while Mamba-3 was best on modulo arithmetic. State tracking permitted

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$30

which allows negative eigenvalues of the transition factor and supports alternating or counting-like dynamics.

At 400M scale and 40 billion training tokens, KronLSTM obtained Avg-6 of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$31 and LAMBADA perplexity of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$32, compared with $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$33 and $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$34 for mLSTM.

Ablations and throughput

The base 1.5B KronLSTM model had Avg-6 of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$35, LAMBADA perplexity of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$36, and recall average of $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$37. Removing $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$38 normalization on values reduced Avg-6 to $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$39 and increased LAMBADA perplexity to $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$40. Coupling $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$41 to both erase and write slightly improved ordinary language modeling but degraded long-context retrieval. Fixing the target to $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$42 reduced Avg-6 to $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$43, increased LAMBADA perplexity to $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$44, and substantially damaged needle-in-a-haystack performance.

On a single H100-80GB, pure-linear 1.5B models processed 16,384 tokens per step. KronLSTM achieved approximately $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$45k–$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$46k tokens per second, was 11–18% faster than Gated DeltaNet-2, and was within 4–8% of Kimi Delta Attention up to the $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$47 shape. Gated DeltaNet remained fastest because its scalar decay yields cheaper kernels.

Limitations

KronLSTM underperformed on RULER S1, which the paper attributes to residual accumulation for frequently recurring keys. The reduced-interference claim is therefore strongest for distinct or semantically meaningful key–value pairs rather than adversarial repeated-key noise.

The theoretical guarantees depend on stochastic assumptions, including i.i.d. inputs, independent sampling, fixed step size, and full-rank feature covariance. The bounds depend on $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$48; poor coverage of memory space makes this quantity small and weakens the guarantees. For independent keys and values,

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$49

so poor conditioning in either feature family affects convergence and memory retention.

The permissible step size depends on key and value norms:

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top,$50

Large value norms reduce the stable range, motivating value normalization. Constant step size also implies stationary fluctuations rather than pointwise convergence in overloaded or inconsistent settings.

Finally, the principal experiments use 1.5B models trained at 8k context, with later extension experiments at 32k. The reported results do not establish that the advantage persists at frontier-scale parameter counts or at contexts substantially longer than 32k. The main comparisons are against recurrent linear-attention architectures rather than full softmax Transformers.

KronLSTM’s defining contribution is therefore the combination of a scalar, Kronecker-feature delta rule with gated matrix memory. Its distinguishing property is selective revision: the memory changes along the joint key–value direction, protecting an existing association whenever either its key or value is sufficiently dissimilar. This provides a recurrent fixed-state alternative that combines linear-time sequence processing, explicit associative revision, and theoretically analyzable memory dynamics (Hartl et al., 19 Oct 2026).