Papers
Topics
Authors
Recent
Search
2000 character limit reached

Kronecker Delta Rule for Long Context Linear Attention

Published 19 Oct 2026 in cs.LG and cs.AI | (2610.26834v1)

Abstract: We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space. This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling. The resulting memory dynamics can be mathematically described as a Markov chain that converges to a steady-state distribution, while the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence. Hence, KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state. Experiments show that KronLSTM is competitive with, and often outperforms, existing linear attention architectures, including models of the (Gated) DeltaNet family, with particularly strong performance on long context benchmarks. These results establish the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear attention architectures. Our code (https://github.com/NX-AI/KronLSTM) and models (https://huggingface.co/collections/NX-AI/kronlstm) are open-source.

Summary

  • The paper proposes the Kronecker Delta Rule, which treats key-value pairs as joint addresses, reducing memory revise interference and stabilizing long context retrieval.
  • When compared to other models, KronLSTM consistently achieves the highest aggregate score on language tasks, reducing LAMBADA perplexity and improving recall in hybrid settings.
  • The Kronecker Delta Rule enhances memory capacity up to square theta values due to the unique built-in normalization reducing the state update size, thus optimizing throughput.
  • related_questions

Problem setting and central contribution

“Kronecker Delta Rule for Long Context Linear Attention” (2610.26834) addresses a central failure mode of recurrent linear-attention models: a fixed-size associative memory must revise stored information without either accumulating unbounded state norms or perturbing unrelated associations. Standard Hebbian linear attention writes outer products additively, so memory magnitude can grow with sequence length. Delta-rule architectures constrain memory growth through error-correcting updates, but their vector-valued correction can interfere with all associations sharing key directions.

The paper proposes the Kronecker Delta Rule, which treats a key–value pair as a joint address and corrects a scalar bilinear association rather than an entire retrieved value vector. Given memory $\BS \in \mathbb{R}^{d_v \times d_k}$, key $\Bk$, value $\Bv$, and target cc, the update is

$\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$

The corrected quantity is the scalar $\Bv_t^\top\BS_t\Bk_t$, and the update direction is fixed by the current pair $\Bv_t\Bk_t^\top$, rather than by the current retrieval residual $\BS_t\Bk_t-\Bv_t$. The resulting KronLSTM architecture embeds this rule in an xLSTM-style sequence mixer with separate forget, erase, write, and output gates.

The paper’s principal claim is that targeted scalar correction provides lower interference than DeltaNet while retaining the bounded-memory and parallelization properties of delta-rule attention. The claim is both theoretical and empirical: the authors derive stability and convergence results for the stochastic recurrence and evaluate 1.5B-parameter models on language modeling, real-world recall, synthetic retrieval, context extension, state tracking, and throughput.

From Hebbian writing to pair-addressed correction

Linear attention represents the prefix of a sequence through a matrix state. In its simplest form,

$\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top, \qquad \Bh_t=\BS_t\Bq_t.$

This update stores key–value correlations but cannot explicitly revise a previous association. Later writes can bury an earlier value through superposition, yet there is no residual-based mechanism that drives a stored association toward a new target.

DeltaNet introduces an error-correcting update,

$\BS_{t+1} = \BS_t - \beta_t (\BS_t\Bk_t-\Bv_t)\Bk_t^\top.$

This is stochastic gradient descent on the vector loss $\Bk$0. Its correction is appropriate when the memory must retrieve the entire vector $\Bk$1 from $\Bk$2, but the residual $\Bk$3 determines the update direction. Consequently, an update at $\Bk$4 can alter other associations whose keys overlap with $\Bk$5.

The Kronecker Delta Rule instead defines the Kronecker feature

$\Bk$6

and regards the vectorized memory $\Bk$7 as a linear regressor for a scalar target $\Bk$8:

$\Bk$9

Thus, the update is exactly the Widrow–Hoff delta rule applied to the pair feature $\Bv$0. Equivalently, it is gradient descent on

$\Bv$1

or a relaxed Kaczmarz projection onto the hyperplane $\Bv$2.

This reformulation clarifies the paper’s interpretation of the memory. KronLSTM is not primarily a retrieval memory in the DeltaNet sense; it is a pair-addressed scalar memory. It stores a scalar at the joint address $\Bv$3 and reads that scalar through a bilinear probe. Under linearly independent Kronecker features, up to $\Bv$4 scalar constraints can be represented exactly. In the square case $\Bv$5, the nominal capacity is therefore $\Bv$6 scalar associations rather than $\Bv$7 value vectors.

Figure 1

Figure 1: The Kronecker update restricts revision to the joint key–value direction, whereas mLSTM and DeltaNet induce progressively broader interference patterns.

The distinction has a direct interference consequence. For a previously stored pair $\Bv$8, an update at $\Bv$9 changes its scalar readout by

cc0

Interference is therefore proportional to the product of value overlap and key overlap. Orthogonality in either factor protects the earlier association. In contrast, the paper characterizes Hebbian and DeltaNet interference as controlled by key overlap alone. This is the formal basis for the claim that the Kronecker update is “minimally invasive.”

The appendix strengthens this claim. At the Newton step

cc1

the update is the unique correction satisfying the target constraint while minimizing the Frobenius norm. The same rank-one correction simultaneously minimizes every unitarily invariant norm. Under isotropic probe statistics, minimizing correction norm is equivalent to minimizing expected squared interference with unrelated associations. The result is conditional on the task definition: the Kronecker rule minimally modifies the memory for a scalar bilinear constraint, whereas DeltaNet solves the different and stronger requirement of reconstructing an entire value vector.

KronLSTM architecture

KronLSTM extends the mLSTM matrix memory but replaces its Hebbian write with the gated Kronecker Delta Rule. The update separates three mechanisms:

cc2

Here, cc3 is a diagonal key-channel forget gate, cc4 is a scalar erase gate, and cc5 is a scalar write or target gate. The decoupling of cc6 and cc7 is important: cc8 controls how much of the current association is erased, while cc9 controls the amount written. The ablation results later show that replacing the learned target with $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$0 substantially harms retrieval.

The model uses $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$1-normalized keys and values. Since $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$2, this gives

$\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$3

placing the memory update in the non-expansive regime. The architecture therefore does not require the normalizer and stabilizer states used by mLSTM. The memory is read by $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$4, passed through an output gate and RMSNorm, and then projected back into the residual stream.

Figure 2

Figure 2: KronLSTM combines normalized query, key, and value projections with diagonal key-wise forgetting, scalar erase/write gates, bilinear memory revision, and gated readout.

The update also admits chunkwise-parallel training. After absorbing cumulative key-wise decay into asymmetric keys, the tokenwise recurrence reduces to a unit lower-triangular system in the scalar write coefficients. The relevant Gram matrix is the Hadamard product of a decay-weighted key Gram matrix and a value Gram matrix. Importantly, the $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$5-dimensional Kronecker features are never explicitly materialized. This preserves the hardware-oriented structure of DeltaNet-style implementations while changing the internal coupling from vector readouts to scalar bilinear scores.

The throughput measurements support the practicality of this construction. On a single NVIDIA H100, KronLSTM reaches 36.8–44.5K tokens/s, reported as 11–18% faster than GDN-2 across the tested shapes and within 4–8% of KDA up to the $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$6 configuration. GDN remains faster because its scalar decay produces cheaper kernels. At $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$7, all models slow because inter-chunk recurrence parallelizes primarily over sequences and heads.

Stability and stochastic memory dynamics

The theoretical analysis models the input stream as i.i.d. triples $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$8 with fixed step size $\BS_{t+1} = \BS_t - \beta_t \left( \Bv_t^\top \BS_t \Bk_t-c_t \right) \Bv_t\Bk_t^\top .$9. Let

$\Bv_t^\top\BS_t\Bk_t$0

The least-squares memory $\Bv_t^\top\BS_t\Bk_t$1 is the optimum for predicting the scalar target from the Kronecker feature. Under full-rank features and the step-size condition

$\Bv_t^\top\BS_t\Bk_t$2

the expected iterate follows the exact recursion

$\Bv_t^\top\BS_t\Bk_t$3

Its unique fixed point is exactly $\Bv_t^\top\BS_t\Bk_t$4, with no finite-step-size bias. The mean converges geometrically at rate

$\Bv_t^\top\BS_t\Bk_t$5

The paper’s stronger result concerns the full stochastic process. The memory states form a Markov chain with a unique stationary distribution $\Bv_t^\top\BS_t\Bk_t$6, and the distribution converges geometrically in Wasserstein-2 distance. The proof uses synchronous coupling: two chains receive the same random pair, so the additive target-dependent terms cancel in their difference. The remaining homogeneous dynamics contract in expectation because the feature covariance $\Bv_t^\top\BS_t\Bk_t$7 is positive definite.

This distinction is important. Individual transition matrices have eigenvalue $\Bv_t^\top\BS_t\Bk_t$8 on the subspace orthogonal to the current Kronecker feature and therefore need not be strict contractions. Contraction occurs at the level of the random transition kernel under a common-noise coupling, not through a uniform spectral norm bound on every individual update.

The stationary mean is exactly $\Bv_t^\top\BS_t\Bk_t$9, while the stationary covariance is $\Bv_t\Bk_t^\top$0. The steady-state excess loss obeys the LMS-type scaling

$\Bv_t\Bk_t^\top$1

where $\Bv_t\Bk_t^\top$2 is the irreducible least-squares residual. Thus, overload beyond the nominal $\Bv_t\Bk_t^\top$3 capacity does not cause divergence. Instead, the state fluctuates around the least-squares solution, with fluctuation magnitude controlled by the step size and the residual energy.

The norm analysis yields a sequence-length-independent bound in the non-expansive regime. In particular, when $\Bv_t\Bk_t^\top$4,

$\Bv_t\Bk_t^\top$5

For the online iterates initialized at zero, the paper obtains

$\Bv_t\Bk_t^\top$6

The implication is precise: the expected memory norm is bounded independently of sequence length from the first update onward, subject to the i.i.d. input model, feature-rank assumptions, and step-size constraints. The forget gate further contracts the state.

The analysis does not establish boundedness for arbitrary dependent data streams, arbitrary learned time-varying gates, or pathological feature distributions without the required covariance rank. Those conditions matter because the proof relies on independence from the past, positive definiteness of $\Bv_t\Bk_t^\top$7, and a uniform step-size bound.

Language modeling and recall performance

The main experiments train 1.5B-parameter models for 100B tokens on FineWeb-Edu with 8K training contexts. Models are evaluated both as pure linear-attention architectures and in a hybrid configuration containing sliding-window attention with a 2048-token window at a $\Bv_t\Bk_t^\top$8 linear-to-SWA layer ratio.

On six downstream tasks—HellaSwag, PIQA, ARC-easy, ARC-challenge, WinoGrande, and LAMBADA—KronLSTM obtains the strongest aggregate performance among the compared models.

Model Linear Avg-6 Linear LAMBADA PPL Hybrid Avg-6 Hybrid LAMBADA PPL
GDN 57.73 12.58 57.99 12.95
Mamba-3 57.31 12.95 58.54 13.01
KDA 58.12 12.67 57.85 12.60
GDN-2 57.94 12.54 58.79 11.85
mLSTM 57.18 13.87 57.35 14.38
KronLSTM 59.72 11.72 59.56 11.18

As a pure linear model, KronLSTM leads on five of six tasks and improves the average over KDA by 1.6 accuracy points. In the hybrid setting, it leads on four of six tasks and achieves the lowest LAMBADA perplexity. The result is notable because adding SWA changes model averages by approximately one point or less, whereas KronLSTM remains competitive in both regimes rather than depending on local attention.

On the 2K real-world recall suite, KronLSTM reaches 30.31 average accuracy as a linear model, ahead of GDN at 29.81 and GDN-2 at 29.39. In the hybrid setting it reaches 37.40, ahead of Mamba-3 at 36.96 and GDN-2 at 35.85. It is the best method on three of six tasks in each setting.

These results support the paper’s central mechanism-level interpretation: when the benchmark requires preserving specific associations, the pairwise interference profile of the update can matter more than generic language-model perplexity. However, the recall advantage is not universal. On DROP, KronLSTM underperforms several baselines, and the paper does not provide a task-level causal analysis sufficient to attribute all differences to the update rule rather than optimization or representation effects.

Needle retrieval and context extension

The RULER needle-in-a-haystack experiments evaluate 4K, 8K, and 16K contexts after 8K pretraining. KronLSTM is generally the best or second-best pure linear model on natural-text retrieval tasks and leads the hybrid models when averaged across lengths. Its advantage increases on several multi-needle tasks as context length grows.

The paper emphasizes the comparison with GDN. On hybrid MQ retrieval, KronLSTM’s lead increases from 2.4 points at 4K to 4.1 points at 16K. On MV, it is 3.5 points behind GDN at 4K, but leads by 1.1 points at 8K and 3.8 points at 16K. These trends are consistent with accumulated key-only interference in DeltaNet-style updates and doubly gated interference in KronLSTM.

The principal exception is S1, an artificial task containing frequently repeated noise sentences. The authors attribute KronLSTM’s degradation to residual accumulation for recurring keys. This exception is important: the paper’s claim is not that pair-addressed correction dominates for every retrieval distribution. Repeated or highly correlated pair features can still overload the scalar memory.

Figure 3

Figure 3: After 32K context extension, KronLSTM retains a consistent advantage over GDN-2 on natural-text NIAH tasks, although both models degrade substantially at the longest length.

For long-context training, the authors branch 1.5B GDN-2 and KronLSTM models after 75B tokens and continue pretraining for 25B tokens at 32K context. KronLSTM maintains higher average downstream accuracy and lower LAMBADA perplexity:

Setting Model Avg-6 LAMBADA PPL Recall average
Linear GDN-2 58.75 8.36 29.95
Linear KronLSTM 60.19 7.66 31.08
Hybrid GDN-2 59.32 8.27 35.21
Hybrid KronLSTM 60.10 7.64 36.23

On the extended RULER evaluation, KronLSTM exceeds GDN-2 on all natural-text tasks from 16K onward. At 32K, for example, the linear models obtain:

Task GDN-2 KronLSTM
S2 2.6 5.6
S3 0.8 8.6
MK 3.0 5.0
MQ 1.5 3.0
MV 7.2 11.3

Both systems experience a substantial reduction from 16K to 32K, so the result demonstrates a relative advantage rather than robust retrieval at 32K. The hybrid results also remain low at 32K, although KronLSTM is consistently better than GDN-2. The implication is that reduced interference improves degradation curves but does not eliminate the finite representational and optimization limits of fixed-size recurrent memory.

State tracking and smaller-scale validation

KronLSTM can enter the negative-eigenvalue regime by allowing $\Bv_t\Bk_t^\top$9 rather than restricting it to $\BS_t\Bk_t-\Bv_t$0. In the vectorized recurrence, the transition has eigenvalue $\BS_t\Bk_t-\Bv_t$1 along the current Kronecker feature. Values in $\BS_t\Bk_t-\Bv_t$2 produce a negative eigenvalue, a mechanism previously associated with state-tracking capabilities in DeltaNet-like models.

On parity and $\BS_t\Bk_t-\Bv_t$3 permutation tracking, KronLSTM performs similarly to GDN and ahead of Mamba-3. Mamba-3 is strongest on modular arithmetic.

Figure 4

Figure 4: KronLSTM matches GDN on parity and permutation tracking while Mamba-3 has the strongest modular-arithmetic performance.

The state-tracking results establish that the scalar bilinear correction does not remove the dynamical expressivity associated with sign-changing transition modes. They do not, however, show that KronLSTM is uniformly superior on formal-language or algorithmic tasks: its performance is task-dependent, and Mamba-3 retains an advantage on modular arithmetic.

At 400M parameters, the differences narrow, but KronLSTM remains strongest on average among the pure linear models. Relative to mLSTM, it improves Avg-6 by 2.1 points and reduces LAMBADA perplexity from 43.40 to 28.28. Mamba-3 is best on PIQA, KDA on LAMBADA accuracy, and GDN-2 ties KronLSTM on WinoGrande. These results suggest that the update rule remains useful below the 1.5B scale, while also showing that the ranking is not invariant to model size or task.

Limitations and open questions

The theoretical guarantees depend on assumptions that are stronger than the actual language-model setting. The stochastic analysis assumes i.i.d. input triples and independence of the current sample from the memory state. Natural language violates both assumptions through temporal dependence and distribution shift. The exact unbiasedness and stationary covariance results therefore characterize an analytically controlled model of the recurrence, not arbitrary autoregressive streams.

The norm bounds also require feature coverage through positive-definite $\BS_t\Bk_t-\Bv_t$4, or a corresponding restriction to the feature span in the rank-deficient case. Poorly conditioned Kronecker features can make $\BS_t\Bk_t-\Bv_t$5 small and weaken the quantitative bounds. Moreover, the cleanest bound assumes $\BS_t\Bk_t-\Bv_t$6; state tracking deliberately operates beyond this range, although still below the formal stability threshold of $\BS_t\Bk_t-\Bv_t$7.

The empirical comparison is restricted to 1.5B and 400M models, with 8K pretraining followed in a separate experiment by 32K continuation training. The paper does not evaluate full softmax-attention hybrids, larger-scale models, or context lengths beyond 32K. The authors specifically leave open whether the observed advantages persist when KronLSTM is scaled substantially and when it is combined with full attention layers.

The most direct unresolved technical question is how the proposed scalar recognition memory interacts with the retrieval demands of general language modeling. The architecture has nominally $\BS_t\Bk_t-\Bv_t$8 scalar degrees of freedom, but each write requires a jointly informative key–value pair and a bilinear readout. The S1 failure and the relatively low absolute 32K retrieval scores indicate that nominal capacity does not directly translate into reliable arbitrary-location retrieval. A useful next analysis would separate feature geometry, target statistics, gate behavior, and optimization effects in determining when the $\BS_t\Bk_t-\Bv_t$9 capacity is accessible.

Conclusion

The paper introduces a bilinear, pair-addressed alternative to the vector-valued Delta Rule and embeds it in the gated KronLSTM sequence mixer. Its main technical contribution is the identification of an update that corrects one scalar association along one rank-one matrix direction, yielding interference proportional to both key and value overlap. The associated stochastic recurrence has an exact least-squares mean, a unique stationary distribution, bounded expected memory norm, and LMS-like $\BS_{t+1}=\BS_t+\Bv_t\Bk_t^\top, \qquad \Bh_t=\BS_t\Bq_t.$0 steady-state error under explicit assumptions.

Empirically, KronLSTM achieves the strongest aggregate language-modeling results among the evaluated linear-attention baselines, improves real-world recall, and shows increasingly favorable relative performance under 16K–32K context extension. These results support the paper’s narrower claim: pair-addressed scalar correction is an effective and analytically tractable memory mechanism for stable long-context linear attention, although its benefits remain conditioned by feature correlations, task structure, model scale, and the assumptions underlying the stability analysis.

Whiteboard

Explain it Like I'm 14

1. What is this paper about?

This paper introduces a new way for artificial intelligence systems to remember information from long pieces of text.

Most modern LLMs use a method called Transformer attention. It helps the model decide which earlier words are important when understanding a new word. However, this method becomes increasingly expensive when the text gets longer.

The authors present a new system called KronLSTM. It is designed to:

  • remember important information over long texts,
  • use less computer memory and processing power,
  • avoid accidentally damaging memories it already stored, and
  • remain stable while learning.

KronLSTM is based on a new memory-updating method called the Kronecker Delta Rule.

2. What questions are the researchers asking?

The researchers mainly wanted to find out:

  1. Can a LLM remember information from very long texts without becoming too expensive to run?
  2. Can its memory be updated without disturbing other information it has already learned?
  3. Can the model’s memory remain stable instead of growing uncontrollably?
  4. Does the new model perform as well as, or better than, other fast language-model architectures?
  5. Can it retrieve information from texts much longer than the texts it saw during training?

A simple example is a story containing many characters and events. If the model is asked about something that happened near the beginning, it should be able to find the answer even after reading thousands of additional words.

3. How did the researchers approach the problem?

The basic memory problem

KronLSTM stores information in a mathematical object called a memory matrix. You can imagine this matrix as a large filing cabinet:

  • a key is like a label on a folder,
  • a value is the information stored inside,
  • a query is a question asking which folder to open.

When a new word arrives, the model uses these keys and values to update its memory.

Older ways of updating memory

The paper compares several memory-update methods.

Hebbian updates add new information to memory. This is like putting another note into a filing cabinet without removing anything. The problem is that the cabinet can become too full, and newer notes may interfere with older ones.

The Delta Rule tries to correct an old memory. It checks what the model currently remembers and changes it toward the correct answer. This is better than simply adding information, but the correction can unintentionally affect other memories with similar keys.

The new Kronecker Delta Rule

The paper’s new method is more targeted. It changes the strength of a specific key–value pair, rather than changing a whole group of related memories.

An analogy is editing a spreadsheet:

  • a broad update might accidentally change several cells in the same row,
  • the Kronecker Delta Rule tries to change only the particular cell that needs fixing.

The method has three main controls, called gates:

  • Forget gate: decides how much old information should fade.
  • Erase gate: decides how much of a particular memory should be removed.
  • Write gate: decides how much new information should be stored.

These gates allow the model to keep useful information for a long time while still replacing information when necessary.

Mathematical analysis

The researchers also studied the behavior of the memory using mathematics. They showed that:

  • the memory tends toward a stable long-term condition,
  • the average memory size stays bounded,
  • the memory does not grow without limit, and
  • the model can learn without needing extra safety mechanisms used by some other systems.

The paper describes the memory updates as a Markov chain. In everyday language, this means the next memory state depends mainly on the current memory state and the new information arriving now.

Experiments

The researchers trained and tested 1.5-billion-parameter models using large text datasets. They compared KronLSTM with other fast sequence models, including:

  • Gated DeltaNet,
  • Kimi Delta Attention,
  • Mamba-3,
  • Gated DeltaNet-2, and
  • mLSTM.

They tested the models on:

  • general language understanding,
  • commonsense reasoning,
  • recalling information from documents,
  • finding hidden “needle” facts inside long texts,
  • tracking changing states, and
  • training speed.

They tested both purely linear models and hybrid models, which combine linear attention with a smaller amount of regular attention.

4. What did the researchers find?

Strong language understanding

KronLSTM achieved the highest average score on the six main language tasks among the compared linear models.

For example, in the reported results:

  • KronLSTM’s average score for the six tasks was 59.72%,
  • the next strongest linear model, Kimi Delta Attention, scored 58.12%.

KronLSTM also had the lowest LAMBADA perplexity. Perplexity is a measure of how surprised a LLM is by text, so a lower score generally means better prediction.

Better long-text recall

KronLSTM was especially strong at finding information hidden inside long passages.

On a real-world recall test, it achieved the best average score among the purely linear models. It also performed well when the text became longer than the 8,000-token context used during its original training.

At 16,000 tokens, KronLSTM often performed better than competing systems. After additional training on 32,000-token texts, it continued to outperform Gated DeltaNet-2 on the natural-language retrieval tasks tested.

This suggests that its more careful memory updates reduce the damage caused by long sequences.

Less interference between memories

The mathematical analysis predicts that KronLSTM changes other memories less than the ordinary Delta Rule.

This was supported by the experiments. When many pieces of information were stored, KronLSTM was generally better at preserving useful earlier information.

However, it was not best at every task. On one artificial test containing many repeated noise sentences, KronLSTM performed worse. This shows that the method still has weaknesses, especially when the same kinds of keys appear repeatedly in unusual patterns.

Stable memory

The researchers proved that the expected size of the memory remains bounded. In simple terms, the memory should not grow larger and larger until the computer becomes unstable.

This is important because unstable memories can lead to:

  • very large numbers,
  • unreliable training,
  • sudden learning failures, and
  • poor predictions.

Unlike some versions of xLSTM, KronLSTM does not need separate normalizer and stabilizer states to control its memory.

Competitive training speed

KronLSTM was faster to train than Gated DeltaNet-2 in the reported tests. It processed about 11–18% more tokens per second than Gated DeltaNet-2, although Gated DeltaNet was still faster overall in some settings.

5. Why is this research important?

Long-context LLMs could be useful for:

  • reading books and long reports,
  • searching large collections of documents,
  • understanding computer programs,
  • following long conversations,
  • analyzing scientific papers, and
  • remembering information from earlier parts of a story.

The main contribution of this paper is not just a new model, but a new idea about how memory should be changed. The Kronecker Delta Rule tries to make small, precise corrections instead of rewriting large parts of memory.

This could help future LLMs become:

  • cheaper to run on long texts,
  • better at remembering distant information,
  • more stable during training, and
  • less likely to confuse one memory with another.

Still, the results do not prove that KronLSTM will replace Transformers everywhere. The experiments used particular model sizes, datasets, and tasks. The authors also identify future work, including testing larger models and combining KronLSTM with full attention layers.

In short, the paper presents KronLSTM as a faster and more stable way for LLMs to remember information over long contexts, with especially promising results when the model must retrieve important facts from very long texts.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper leaves the following issues unresolved:

  • Mismatch between theoretical assumptions and the trained model: The convergence and bounded-norm results assume i.i.d. key–value–target triples, fixed step size, and a positive-definite feature covariance, whereas KronLSTM uses input-dependent gates, learned projections, causal convolutions, normalization, forget matrices, and highly correlated language sequences.
  • No stability theorem for the complete KronLSTM block: The analysis covers the associative-memory update, but does not establish convergence, bounded activations, or gradient stability for the full deep network with learned gates, RMSNorm, output projections, residual connections, and backpropagation through time.
  • Unclear implications of input-dependent gates: The theoretical guarantees do not directly cover the learned, input-dependent βt\beta_t, ctc_t, and $\BF_t$ used in experiments, particularly when gate distributions vary substantially across layers, heads, token types, or training stages.
  • Insufficient characterization of stationary memory distributions: The paper proves existence and uniqueness of a stationary distribution under its assumptions but does not characterize its variance, tails, mixing time, dependence on β\beta, or behavior under non-i.i.d. and heavy-tailed inputs.
  • No robustness analysis for distribution shift: It remains unknown whether bounded memory and retrieval quality persist when inference inputs differ from the training distribution, contain adversarially chosen key–value patterns, or exhibit unusually long repetitions.
  • Capacity claims are primarily algebraic rather than end-to-end: The stated capacity of up to dkdvd_kd_v scalar associations requires linearly independent Kronecker features, but the paper does not measure the effective capacity of learned representations on realistic language data or quantify how capacity changes with correlated keys and values.
  • The interference advantage is not isolated experimentally: Although the paper attributes improvements to reduced key-and-value-overlap interference, it does not provide controlled experiments that vary key overlap, value overlap, association density, or update order while holding model size and optimization conditions fixed.
  • Residual accumulation for repeated keys remains unresolved: The paper identifies degraded performance on the RULER S1 task but does not develop a mechanism, theoretical bound, or architectural intervention for recurring keys and repeated noisy associations.
  • Long-context evaluation remains limited: The main retrieval experiments reach 32k tokens after context extension, leaving performance at substantially longer contexts—such as 64k, 128k, or 1M tokens—unestablished.
  • Context extension is not compared fairly across more alternatives: The long-context extension experiment compares only KronLSTM with GDN-2, so it does not determine whether the advantage persists against other linear-attention, state-space, recurrent, or hybrid Transformer baselines trained with the same extension procedure.
  • Limited evaluation of full softmax-attention systems: The experiments compare recurrent linear architectures and sliding-window hybrids but do not include equivalently sized full-attention or full-attention-plus-local-attention models, leaving the quality–efficiency trade-off against standard Transformers unclear.
  • No evaluation of hybrid models with global softmax layers: The conclusion identifies hybrids containing full softmax attention as future work, so the paper does not establish whether KronLSTM remains advantageous in the deployment configurations used by several contemporary LLMs.
  • Scale generalization is unverified: Most reported results use 1.5B-parameter models trained on 100B tokens, with only limited 400M results in the appendix; it is unknown whether the gains persist at 7B parameters and beyond or under substantially larger token budgets.
  • Training-data diversity is narrow: Pretraining uses FineWeb-Edu and a specific long-document mixture, so the method’s behavior on multilingual, code, mathematical, conversational, multimodal, or domain-specific corpora is not established.
  • Benchmark coverage does not test many long-context capabilities: Needle retrieval, recall suites, commonsense tasks, and synthetic state tracking do not adequately measure long-document question answering, multi-hop reasoning, temporal reasoning, summarization, document editing, in-context learning, or instruction following.
  • Synthetic state-tracking results are only partially explored: State tracking is evaluated up to length 1024 after training to length 128, but the paper does not investigate more complex state spaces, noisy transitions, multiple simultaneous variables, or state tracking at the long-context lengths emphasized elsewhere.
  • Ablation coverage is insufficient for attributing gains: The relative contributions of the Kronecker update, vector forget gate, erase gate, target gate, normalization choices, short convolutions, head timescales, and output gating are not comprehensively separated.
  • Gate behavior is not empirically analyzed: The paper does not report learned gate distributions, retention timescales, saturation rates, or how gates differ across layers and heads, making it difficult to understand what memory policy KronLSTM learns.
  • The impact of normalized keys and values is unclear: The theoretical capacity and interference properties are presented generally, but the practical consequences of ℓ2\ell_2 normalization—including reduced representational flexibility and the handling of magnitude information—are not investigated.
  • Numerical stability is asserted more strongly than demonstrated: The paper reports a learning-rate stress test but does not provide systematic measurements of overflow, underflow, gradient statistics, loss spikes, or failure rates across precisions such as FP32, BF16, FP16, and quantized inference.
  • Inference efficiency is not fully evaluated: Training throughput is measured on one H100 configuration, but the paper does not report end-to-end decoding latency, prefilling cost, memory bandwidth, energy use, batch-size scaling, or performance on other hardware.
  • The computational cost of the matrix state is not fully contextualized: Because each head stores a dv×dkd_v \times d_k matrix, the paper does not quantify how state size and update cost compare with KV caches, sliding-window attention, and other linear-attention states across realistic model configurations.
  • Chunkwise-parallel training equivalence is underexplored: The paper states that the vectorized rule permits a DeltaNet-style chunkwise formulation, but does not quantify numerical discrepancies, parallel-scan efficiency, or gradient differences between recurrent and chunkwise implementations.
  • No study of very long-document memory degradation over repeated batches: The experiments focus on single-context retrieval, leaving open how memory resets, document boundaries, packing strategies, and recurrent state carryover affect training and inference.
  • The relationship between reduced interference and language-model perplexity is not established causally: Improvements on benchmark scores are reported, but the paper does not show whether they arise from better associative memory, optimization dynamics, architectural parameterization, or differences in effective capacity.
  • Baseline comparability may remain uncertain: The paper does not provide full evidence that all baselines use identical parameter counts, training schedules, initialization, optimizer settings, kernel implementations, and hyperparameter tuning budgets, which limits the strength of architectural comparisons.
  • No analysis of failure modes under adversarial or pathological associations: It remains unknown how KronLSTM behaves when values are nearly collinear, keys and values are deliberately anti-correlated, targets alternate rapidly, or an adversary repeatedly writes and erases the same association.
  • Theoretical results do not address learned representation collapse: The positive-definiteness assumption on E[aa⊤]\mathbb{E}[\mathbf a\mathbf a^\top] may fail or become ill-conditioned in trained networks, but the paper does not analyze near-singular feature covariance, conditioning, or its effect on memory norms and learning speed.
  • No formal comparison of expressive power with DeltaNet and softmax attention: The paper provides capacity and interference arguments but does not characterize which sequence functions KronLSTM can represent that competing architectures cannot, or the representational cost of achieving equivalent accuracy.
  • The long-term behavior of gradients remains incompletely justified: Bounded expected memory norms do not by themselves guarantee non-vanishing or well-conditioned gradients over very long sequences; gradient spectra, Jacobian products, and trainability at million-token horizons remain open questions.
  • Practical deployment constraints are not assessed: The paper does not evaluate quantization, pruning, model compression, fault tolerance, or deployment on edge and memory-constrained devices.
  • Reproducibility of the strongest claims needs broader confirmation: Results are reported for a limited number of model sizes, datasets, hardware settings, and random seeds; multi-seed variance and independent replication are not presented for the main comparisons.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Long-context language-model inference and chat assistants — software, search, and customer support. KronLSTM can be evaluated as a drop-in replacement for Transformer attention or other recurrent linear-attention layers in assistants that process long conversations, technical manuals, support tickets, or code repositories. Its fixed-size recurrent memory removes the context-length-dependent key–value cache and provides linear-time processing, while the Kronecker Delta Rule reduces interference between associations.
    • Potential tools: open-source KronLSTM checkpoints, inference kernels, and hybrid models combining KronLSTM with sliding-window attention.
    • Dependencies: the reported results are primarily for 400M–1.5B-parameter models and specific training data; production deployment requires validation on larger models, domain-specific data, latency, quality, and safety benchmarks. Memory is fixed-size but not equivalent to lossless storage of every token.
  • Long-document retrieval and question answering — legal, enterprise, biomedical, and research workflows. A KronLSTM-based model could process contracts, clinical literature, regulatory filings, incident reports, or scientific papers while retaining relevant facts over long sequences. The paper reports strong performance on real-world recall tasks and needle-in-a-haystack evaluations, especially as context length increases.
    • Potential workflow: stream a document once, maintain the recurrent state, and answer multiple queries without recomputing a full attention cache.
    • Dependencies: retrieval accuracy is uneven across tasks; the artificial repeated-noise S1 benchmark exposes degradation caused by recurring keys. High-stakes applications still require explicit retrieval, citations, human review, and evaluation for hallucinations.
  • Memory-efficient edge and on-device LLMs — mobile, embedded, and private computing. Constant-size recurrent state can reduce inference memory relative to full-context Transformers, enabling long-session summarization, offline assistants, document readers, or coding tools on constrained hardware. The architecture’s bounded memory dynamics may also reduce the need for extra normalization or stabilization states.
    • Potential products: offline note-taking assistants, private voice/text agents, embedded technical-support systems, and long-running device copilots.
    • Dependencies: actual suitability depends on kernel optimization, quantization, model size, recurrent-state storage, and energy measurements. The paper reports throughput on an NVIDIA H100, not on mobile CPUs, NPUs, or microcontrollers.
  • Hybrid long-context model serving — cloud inference and enterprise AI platforms. KronLSTM can replace selected attention layers while retaining sliding-window attention for local token interactions. This offers a practical deployment path: use recurrent linear attention for persistent context and local attention for precise nearby reasoning.
    • Potential workflow: use a 5:1-style ratio of linear-attention to sliding-window layers as an initial engineering configuration, then tune the ratio for latency and accuracy.
    • Dependencies: the reported hybrid configuration uses a 2,048-token window and is not necessarily optimal for every workload. Serving systems must support recurrent state management, batching, sequence resets, and state isolation between users.
  • Efficient continued pretraining and domain adaptation — academia and industry research. Researchers can use the open-source implementation and models to investigate context extension, recurrent memory, and alternatives to quadratic attention. The reported 32k-context continuation training provides a practical starting point for adapting models to long documents.
    • Potential tools: reproducible training pipelines, ablation suites comparing KronLSTM with Gated DeltaNet, Mamba, KDA, and Transformer hybrids, and benchmarks for retrieval interference.
    • Dependencies: context extension requires additional training data and compute; performance may depend strongly on the document mixture, tokenizer, sequence length, and gate configuration.
  • Stable training of recurrent sequence models — machine-learning engineering. The bounded-memory analysis and non-expansive parameterization can guide implementations of recurrent associative memories. In particular, normalized keys and values plus sigmoid erase gates ensure the update remains in the theoretically favorable regime described in the paper.
    • Actionable practice: monitor βt∥kt∥2∥vt∥2\beta_t\lVert \mathbf{k}_t\rVert^2\lVert \mathbf{v}_t\rVert^2, memory norms, gradient norms, and retrieval accuracy during training; use these diagnostics to detect instability or excessive forgetting.
    • Dependencies: the theoretical bounds assume distributional conditions such as positive-definite feature covariance and bounded update magnitudes. Learned gates and real training distributions may violate idealized assumptions.
  • Online associative memory for streaming data — anomaly detection, telemetry, and adaptive systems. The Kronecker Delta Rule supports targeted correction and erasure of associations through its target value: approximately 1 for writing and 0 for deletion. This can be used in streaming models that must update continuously without retraining on the full history.
    • Potential applications: adaptive sensor models, log-stream summarization, changing-user preference models, and online classification features.
    • Dependencies: the rule stores scalar associations in a matrix memory rather than arbitrary records; capacity depends on the independence and dimensionality of key–value feature pairs. Data drift and correlated features may cause interference.
  • Teaching and research in machine learning — academic training. The paper provides a compact example linking associative memory, the Widrow–Hoff Delta Rule, Kronecker products, Markov-chain convergence, and recurrent neural architectures. It can support graduate-level assignments or research prototypes on stable online learning.
    • Potential exercises: implement the naive and chunkwise-parallel updates, verify the contraction condition, measure interference under key/value overlap, and reproduce long-context retrieval experiments.
    • Dependencies: the manuscript contains complex notation and some results require careful implementation and independent verification; reproduction should use the released code and reference kernels.

Long-Term Applications

  • Large-scale long-context foundation models — general-purpose AI platforms. If scaling experiments confirm the reported advantages, KronLSTM could become a component of foundation models that handle books, repositories, multimodal event streams, or extended interactions with lower inference cost than full softmax attention. Its fixed-size state is particularly attractive for long-running sessions and streaming generation.
    • Potential product: a hybrid architecture with KronLSTM for persistent memory, local attention for detailed nearby reasoning, and occasional full attention or retrieval modules for high-precision access.
    • Dependencies: the paper explicitly identifies larger-scale training and hybrids with full softmax attention as future work. Scaling may reveal optimization, expressivity, or quality limitations not visible at 1.5B parameters.
  • Persistent conversational and personal agents — consumer software and workplace automation. Targeted erase/write behavior could support agents that retain useful preferences while revising outdated information with less disruption to unrelated memories. This may enable long-lived personal assistants, project agents, or enterprise copilots.
    • Potential workflow: encode user facts and task state into recurrent memory, use explicit target or gate signals for updates and deletions, and maintain external audit logs for user control.
    • Dependencies: model-internal memory is not automatically interpretable, persistent, or reliably deletable. Privacy, consent, memory inspection, adversarial prompting, and guaranteed erasure require external memory controls and formal auditing.
  • Real-time robotics and autonomous systems — robotics, vehicles, and industrial control. The architecture’s constant-memory recurrence and reported state-tracking ability could support long sensor histories, task state, and event tracking in systems with strict latency or power constraints.
    • Potential applications: robot manipulation histories, navigation state, predictive maintenance, and multi-step control policies.
    • Dependencies: the experiments are language-centric and do not establish control stability, safety, or performance under continuous noisy sensor streams. Deployment requires closed-loop testing, bounded latency guarantees, sensor-fusion studies, and fail-safe mechanisms.
  • Streaming multimodal models — speech, video, IoT, and industrial monitoring. The matrix-memory update is modality-agnostic in principle and could be applied to sequences of audio, visual, or sensor embeddings. Targeted revisions may help preserve distinct events when feature representations have limited overlap.
    • Potential products: always-on meeting transcription, video-event monitoring, wearable assistants, and industrial anomaly monitors.
    • Dependencies: keys and values must be learned so that relevant events are sufficiently separable. Correlated or highly repetitive streams may produce residual accumulation and interference, as suggested by the S1 results.
  • Low-latency financial and operational forecasting — finance and enterprise analytics. A stable streaming associative memory could process transaction, market, supply-chain, or demand sequences without retaining the entire historical window. This may reduce serving cost for high-frequency or resource-constrained forecasting systems.
    • Potential workflow: update the recurrent state as events arrive, query it for predictions, and reset or reinitialize it at regime boundaries.
    • Dependencies: the paper does not evaluate forecasting, financial data, distribution shift, or adversarial manipulation. Strong governance, backtesting, calibration, and explicit safeguards would be required before consequential use.
  • Energy-efficient data-center inference — cloud infrastructure and sustainability. If the theoretical constant-memory advantage translates to large-scale serving, KronLSTM could reduce key–value-cache memory, communication overhead, and possibly energy consumption for long-context workloads.
    • Potential infrastructure: recurrent-state-aware batching, state compression, specialized GPU/TPU kernels, and scheduling systems that reuse states across streaming requests.
    • Dependencies: total cost also includes recurrent computation, model weights, training, kernel efficiency, and batching limitations. The paper reports promising H100 training throughput but does not provide end-to-end serving energy or cost measurements.
  • Adaptive memory modules beyond language modeling — neuroscience-inspired computing and general sequence learning. The Kronecker Delta Rule can serve as a general-purpose learning rule for associative matrix memories, with explicit control over correction, erasure, and write strength. It may be useful in differentiable memory, continual learning, and neural algorithmic reasoning.
    • Potential research directions: memory modules for program execution, symbolic state tracking, continual classification, and differentiable databases.
    • Dependencies: the capacity result assumes linearly independent Kronecker features, and a single key may retrieve the sum of multiple orthogonal values. Scaling capacity and preventing interference in realistic, correlated representations remain open problems.
  • Policy and infrastructure planning for efficient AI. Policymakers and public-sector technology programs could support benchmarks and procurement criteria for long-context efficiency, comparing quadratic-attention and recurrent linear-attention systems on quality, energy, memory, privacy, and reproducibility rather than relying only on parameter count.
    • Potential outputs: standardized long-context retrieval tests, energy-per-token reporting, state-isolation requirements for hosted models, and audit protocols for persistent model memory.
    • Dependencies: policy recommendations should remain technology-neutral until independent studies reproduce the results across model scales, hardware platforms, languages, and safety-critical domains.
View Paper Prompt  View All Prompts 

Glossary

  • Associative memory: A memory system that stores relationships between input keys and corresponding values. “Therefore, $\BS$ is an associative memory”
  • Causal depthwise convolution: A convolution applied independently to each channel while using only current and preceding inputs. “each followed by a short causal depthwise convolution”
  • Chunkwise-parallel training: A method that processes sequential updates in parallel across chunks of a sequence. “which admits chunkwise-parallel training”
  • Contraction: A transformation that reduces distances or errors between successive states. “the kernel contracts on laws with finite second moment”
  • Delta Rule: An error-correcting memory update that adjusts stored information according to the difference between a retrieved value and a target value. “In contrast, the Delta Rule”
  • Eigenvalue: A scalar describing how a linear transformation scales a corresponding eigenvector. “the transition factor has a negative eigenvalue”
  • Exponential input gate: A gate using an exponential function to control how strongly new information is written to memory. “under a sigmoid forget gate and an exponential input gate”
  • Finite second moment: A probabilistic condition in which a random variable has a finite expected squared norm. “contracts on laws with finite second moment”
  • Forget gate: A learned mechanism that controls how much previously stored information is retained. “The forget gate is a diagonal matrix”
  • Frobenius norm: A matrix norm equal to the square root of the sum of the squares of all matrix entries. “the steady state and --- started at $\BS_0 = \BZe$ --- every iterate satisfies”
  • Generalized Householder matrix: A matrix transformation related to a reflection or rank-one update, used here to characterize memory transitions. “the transition $\BI - \beta_t\, \Bk_t \Bk_t^\top$ is a generalized Householder matrix”
  • Gradient flow: The propagation of optimization gradients through a neural network during training. “This targeted memory revision results in a less noisy gradient flow over time.”
  • Hebbian rule: A learning rule that strengthens associations by adding an outer product of an input and its associated value. “with a Hebbian rule, i.e., an outer product update rule.”
  • Hybrid model: A model combining linear attention layers with another attention mechanism, such as sliding-window attention. “Therefore, we evaluate every architecture in two settings: as a pure linear attention model, and as a hybrid with SWA layers.”
  • Irreducible residual: The prediction error that cannot be eliminated by the chosen model or representation. “ρ\rho the irreducible residual”
  • Kronecker product: An operation combining two matrices or vectors into a larger block-structured matrix or vector. “With the Kronecker product $\Ba_t = \Bk_t \otimes \Bv_t$”
  • Least-squares memory: A memory state that minimizes the expected squared association error. “the least-squares memory”
  • Linear attention: An attention mechanism whose computational and memory costs scale linearly with sequence length by maintaining a fixed-size summary. “Linear attention compresses the context into a fixed-size memory matrix”
  • Markov chain: A stochastic process in which the distribution of the next state depends only on the current state. “the iterates form a time-homogeneous Markov chain”
  • Memory interference: Unwanted modification of previously stored information when new information is written. “reducing interference with previously stored associations”
  • Memory norm: A numerical measure of the magnitude of a memory matrix or state. “We will present expected bounds on the memory norm”
  • Non-expansive: A property of a transformation that does not increase the distance or norm of its input. “In the non-expansive regime κ≤1\kappa \le 1”
  • Normalizer state: An auxiliary state used to control the scale of a memory or readout in a recurrent architecture. “there is no normalizer state and no stabilizer state as in the mLSTM.”
  • Numerical instability: Computational behavior in which small errors or updates produce uncontrolled or unreliable numerical values. “can itself lead to numerical difficulties and learning instabilities”
  • Outer product: The matrix formed by multiplying a column vector by the transpose of another vector. “i.e., an outer product update rule.”
  • Perplexity: A measure of how well a LLM predicts a sequence, with lower values indicating better predictions. “and the lowest perplexity on LAMBADA.”
  • Rank-one update: A matrix modification formed from the outer product of two vectors, changing the matrix along a single matrix-space direction. “it modifies the memory only along a single direction in matrix space”
  • Readout: The operation of retrieving an output from a memory state using a query or input vector. “its readout can become arbitrarily large”
  • Recurrent linear attention: Linear attention implemented through sequential state updates rather than explicit full-context attention. “recurrent linear attention architectures”
  • Residual: The difference between a model’s predicted quantity and its target. “a scalar is corrected”
  • RMSNorm: A normalization method that rescales activations according to their root-mean-square magnitude. “a per-head RMSNorm”
  • Scalar association: A single numerical strength representing the relationship between a key and a value. “The Kronecker Delta Rule corrects this scalar towards the target”
  • Sequence-mixer block: A neural-network layer that transforms and combines information across sequence positions. “the Kronecker Delta Rule is embedded in the KronLSTM sequence-mixer block.”
  • Sigmoid gate: A gate whose output lies between zero and one and controls the flow of information. “The mLSTM places the Hebbian write \eqref{eq:linatt} under a sigmoid forget gate”
  • Sliding-window attention: An attention mechanism that restricts each token to attending within a fixed-size local window. “or use sliding-window attention (SWA) instead”
  • Softmax attention: The standard Transformer attention mechanism that uses softmax-normalized query–key similarities. “whose softmax attention mechanism has computational costs that grow quadratically”
  • Stable state-space ordinary differential equation: A differential equation whose dynamics constrain state growth and remain mathematically stable. “through a forget mechanism derived from a stable state-space ordinary differential equation.”
  • Stabilizer state: An auxiliary state used to prevent numerical instability by controlling the scale of recurrent memory computations. “there is no normalizer state and no stabilizer state as in the mLSTM.”
  • Stationary distribution: A probability distribution that remains unchanged under the transition dynamics of a stochastic process. “it has a unique stationary distribution πβ\pi_\beta”
  • Steady state: A long-term state or distribution toward which a dynamical system converges. “whose dynamics converge to a steady-state distribution”
  • State tracking: The ability of a sequence model to maintain and update information about an evolving latent state. “KronLSTM can perform state tracking”
  • Target gate: A learned gate that controls the target strength of an association written to memory. “equipped with a vector forget gate, a β\beta gate, and a target gate.”
  • Time-homogeneous Markov chain: A Markov chain whose transition rule does not change over time. “the iterates form a time-homogeneous Markov chain”
  • Vectorized form: A representation in which a matrix is converted into a single vector, often to simplify linear-algebraic analysis. “Delta Rule in vectorized form”
  • Widrow–Hoff Delta Rule: An incremental least-mean-squares update rule that adjusts parameters toward a target using an input vector and prediction error. “the Widrow--Hoff Delta Rule \citep{Widrow:60} with input $\Ba_t$ and target ctc_t”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 2 tweets with 594 likes about this paper.

Sign Up for Free

HackerNews

  1. Kronecker Delta Rule for Long Context Linear Attention (2 points, 0 comments)