Fast Weight Attention

• Fast Weight Attention is a family of recurrent attention mechanisms that rapidly write and query key–value associations in dynamically updated parameters or states, enabling sequence-length-independent memory.
• Outer-product, delta-rule, decay-based, tensor-product, sparse product-key, and gradient-programmed variants trade off retrieval capacity, interference, forgetting, stability, and computational cost.
• Applications include associative reasoning, language modeling, reinforcement learning, long-context retrieval, and dynamic evaluation, but fixed-state compression and hardware-efficient routing remain key challenges.

Fast Weight Attention is a family of attention and recurrent-memory mechanisms in which a slowly trained controller dynamically writes and reads rapidly changing parameters or associative states. The fast state commonly stores key–value bindings through outer-product updates and retrieves them with a query, producing a recurrent or linearized form of attention with state size independent of sequence length. Depending on the architecture, fast weights may be additive, decayed, selectively overwritten by a delta rule, organized as a tensor-product memory, sparsely indexed in a product-key bank, or interpreted as online gradient updates. The term therefore denotes a design space rather than one unique operator.

1. Conceptual foundations and historical development

Fast Weight Attention separates two temporal scales. Slow weights are ordinary trainable parameters, such as projection matrices, recurrent parameters, embeddings, and biases, optimized across examples by gradient descent. Fast weights are dynamically changing parameters or state variables generated or modified during processing of a particular sequence, document, episode, or context. A slow network acts as a programmer: it produces keys, values, queries, gates, update rates, or candidate fast parameters. A fast network or memory then uses the current state to generate outputs.

The classical fast-weight formulation associates a key $k_t$ with a value $v_t$ by writing an outer product into a matrix:

$W_t=W_{t-1}+v_t k_t^\top.$

A query retrieves the accumulated content through

$y_t=W_tq_t.$

Starting from $W_0=0$ gives

$$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$

The matrix is an associative memory: similarity between a query and a stored key determines the contribution of the corresponding value. The formulation descends from fast-weight proposals by Hinton and Plaut, Schmidhuber’s fast-weight programmers, and the fast-weight mechanism adopted by Ba et al. for attending to the recent past. These approaches treat rapidly changing connections as short-term memory or as a context-dependent program.

The modern connection to attention follows from reassociating matrix multiplication. For causal, unnormalized attention with projected keys, values, and queries,

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$

Thus causal linear attention is an additive fast-weight programmer: the sequence-dependent matrix $\sum_jv_jk_j^\top$ is a fast associative memory, while the projections generating $k_j$, $v_j$, and $v_t$0 are slow parameters. This formal equivalence is developed in "Linear Transformers Are Secretly Fast Weight Programmers" (Schlag et al., 2021) and extended to recurrent fast networks and recurrent programmers in "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers" (Irie et al., 2021).

The architecture should not be conflated with all forms of dynamic parameterization. "Modeling Rapid Contextual Learning in the Visual Cortex with Fast-Weight Deep Autoencoder Networks" (Li et al., 7 Aug 2025) uses LoRA adapters in ViT attention as a fast-weight-like slow/fast architecture, but its adapters are optimized during familiarity training rather than updated online through an explicit associative-memory recurrence. Similarly, vertical reuse of attention weights in the Shared Attention Network is attention-state reuse across Transformer depth, not canonical fast-weight programming (Xiao et al., 2019).

2. Associative-memory operators

Additive outer-product memory

The basic update is

$v_t$1

It stores a superposition of bindings. The readout

$v_t$2

is an unnormalized linear-attention computation. A feature-map version replaces $v_t$3 and $v_t$4 by $v_t$5 and $v_t$6 and may maintain a normalizer:

$v_t$7

$v_t$8

The recurrent state has fixed dimensions with respect to sequence length, giving linear sequence processing and constant recurrent-state size. Its effective capacity is constrained by key or feature dimensionality. Nonorthogonal keys interfere, and additive updates cannot explicitly remove obsolete associations.

"Linear Transformers Are Secretly Fast Weight Programmers" (Schlag et al., 2021) analyzes this limitation through an orthogonality argument. If projected keys must be mutually orthogonal in a feature space of dimension $v_t$9, then at most $W_t=W_{t-1}+v_t k_t^\top.$0 independent associations can be represented without interference under the idealized criterion. Its experiments report degradation near 60 associations for Linear Attention with $W_t=W_{t-1}+v_t k_t^\top.$1, and near nominal capacities of 128, 256, and 384 for DPFP feature maps with corresponding settings.

Auto-associative fast weights

"Fast Weight Long Short-Term Memory" (Keller et al., 2018) integrates additive fast weights into an LSTM. Its fast matrix is

$W_t=W_{t-1}+v_t k_t^\top.$2

where $W_t=W_{t-1}+v_t k_t^\top.$3 is an LSTM candidate activation, $W_t=W_{t-1}+v_t k_t^\top.$4 is the writing rate, and $W_t=W_{t-1}+v_t k_t^\top.$5 is the retention coefficient. With $W_t=W_{t-1}+v_t k_t^\top.$6,

$W_t=W_{t-1}+v_t k_t^\top.$7

Reading with $W_t=W_{t-1}+v_t k_t^\top.$8 gives

$W_t=W_{t-1}+v_t k_t^\top.$9

The same activation functions as both write vector and query make this mechanism auto-associative. The reported experiments used $y_t=W_tq_t.$0 and $y_t=W_tq_t.$1.

The fast-weight LSTM combines the matrix memory with the conventional LSTM cell:

$y_t=W_tq_t.$2

followed by

$y_t=W_tq_t.$3

The LSTM gates regulate candidate writing, conventional-state retention, incorporation of retrieved memory, and output exposure. The resulting model has vector-valued gated memory and matrix-valued associative memory operating on different temporal scales.

On the modified associative retrieval task, FW-LSTM substantially exceeded both layer-normalized LSTM and fast-weight RNN baselines. At hidden size $y_t=W_tq_t.$4 and mART length $y_t=W_tq_t.$5, its reported test accuracy was $y_t=W_tq_t.$6, compared with $y_t=W_tq_t.$7 for LN-LSTM and $y_t=W_tq_t.$8 for FW-RNN. At hidden size $y_t=W_tq_t.$9, FW-LSTM achieved $W_0=0$0, compared with $W_0=0$1 and $W_0=0$2, respectively. These results support architectural synergy, although they do not formally isolate the causal contribution of each gate or normalization choice.

Delta-rule memory

A delta rule first retrieves the memory’s current prediction and then writes only the error:

$W_0=0$3

$W_0=0$4

The update strength is learned, commonly as

$W_0=0$5

This permits selective replacement: $W_0=0$6 strongly corrects the existing mapping, whereas $W_0=0$7 preserves it. Unlike global decay, correcting one key need not uniformly weaken unrelated associations.

For orthonormal keys, if

$W_0=0$8

and a new value $W_0=0$9 arrives for $$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$0, then

$$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$1

preserves the association for $$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$2 while interpolating the association for $$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$3.

The Delta Net and DPFP experiments in (Schlag et al., 2021) show that delta updates outperform purely additive updates when keys recur with different values. In language modeling, the delta rule improved both deterministic Linear Transformer and Performer variants, including in settings that were not over the nominal memory-capacity limit.

3. Recurrent architectures and learning objectives

Fast Weight Attention can be viewed as an online learning algorithm embedded in a sequence model. This perspective is developed explicitly in "Fast Weight Attention for Continual Learning" (Zhang et al., 27 Aug 2026), which treats the recurrent state as a predictor updated by local optimization.

For a memory $$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$4, a feature $$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$5, and target $$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$6, the memory predicts

$$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$7

The regression residual is

$$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$8

A normalized squared-error update is

$$W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t).$$9

with

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$0

This is a normalized least-mean-squares update when $$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$1 and $$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$2. The paper distinguishes scalar plasticity, per-value-channel plasticity, and sliding-window mini-batch updates in the Falcon-1, Falcon-2, and Falcon-3 families. Suffix A denotes the corresponding inner-product objective rather than squared-error regression.

A central issue is temporal alignment. Under read-after-write semantics, the local example for prefix prediction is

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$3

The preceding feature predicts the newly revealed value. The same-step association $$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$4 remains causal but optimizes a different internal objective. This distinction separates temporal alignment from plasticity, forgetting, and bounded rehearsal.

The Falcon updates admit recurrent, masked-parallel, and chunk-parallel forms. Scalar-decay recurrences can be unrolled into decay-weighted causal attention:

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$5

Regression variants require more complex causal triangular solves, whereas additive inner-product variants reduce to masked linear-attention expressions. Chunk-parallel execution propagates only boundary states sequentially while computing intra-chunk interactions in parallel.

Other recurrent fast-weight programmers alter the architecture of the slow or fast network. Delta RNN adds a recurrent fast matrix; Delta LSTM dynamically programs fast gated recurrent matrices; Recurrent Delta Net makes the slow programmer depend on the previous fast output (Irie et al., 2021). The resulting design space independently varies slow-network recurrence, fast-network recurrence, update rule, activation placement, normalization, and number of fast matrices.

In "Learning Associative Inference Using Fast Weight Memory" (Schlag et al., 2020), the memory is a third-order tensor-like map. Two keys form a tensor-product address:

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$6

and the memory stores a value at that address. A delta-style write is

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$7

Multiple recurrent reads permit chained inference. A retrieved value becomes part of the next query, supporting mappings such as

$$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$8

The tensor-product address has $$y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t.$$9 features and the explicit third-order memory has $\sum_jv_jk_j^\top$0 storage and computation under dense implementation. In concatenated-bAbI, FWM achieved $\sum_jv_jk_j^\top$1 QA accuracy, compared with $\sum_jv_jk_j^\top$2 for LSTM and $\sum_jv_jk_j^\top$3 for Transformer-XL. Its benefits were strongest in compositional reasoning and continual streams containing obsolete or context-dependent facts.

4. Forgetting, plasticity, and memory capacity

Fast-weight systems differ substantially in how they control retention.

Fixed decay multiplies the previous memory by a scalar:

$\sum_jv_jk_j^\top$4

It provides geometric recency weighting but applies the same retention coefficient to every memory component.

Scalar gating uses a learned gate, often

$\sum_jv_jk_j^\top$5

to interpolate the old state and a new write. Its uniform action limits fine-grained control over value and feature dimensions.

Element-wise decay gives each memory element a learned coefficient. "Fine-Tuning Pre-trained Transformers into Decaying Fast Weights" (Mao, 2022) uses

$\sum_jv_jk_j^\top$6

where

$\sum_jv_jk_j^\top$7

The resulting $\sum_jv_jk_j^\top$8 decay matrix is low-rank parameterized at each step, while every state entry has its own coefficient in $\sum_jv_jk_j^\top$9. The new write is added directly rather than multiplied by a complementary gate. Sigmoid-bounded decay is necessary for stability; removing the sigmoid caused divergence in the reported experiments.

The final decaying fast-weight model removes the nonlinear feature map and attention normalizer, using a linear feature projection that can be absorbed into the key or query projection. It therefore maintains only $k_j$0 and reads

$k_j$1

For autoregressive generation, its per-token computation and state are $k_j$2, independent of context length $k_j$3. Standard cached self-attention requires $k_j$4 work per token and $k_j$5 cache memory. Training remains more demanding: because the decay recurrence is non-reversible, the reported training memory is $k_j$6 and parallel time is $k_j$7 over a sequence.

Fine-tuning GPT-2 with decaying fast weights reached validation perplexity $k_j$8 at fast-weight dimension $k_j$9, compared with $v_j$0 for the GPT-2 baseline. The method therefore recovered $v_j$1 of the baseline performance according to the paper, while replacing context-length-dependent attention with a fixed-size recurrent state. Its success depends on fine-tuning; conversion without additional training is not established.

Capacity remains a fundamental limitation. Additive matrix memories superpose associations, so similar keys cause crosstalk. Tensor-product memories increase representational capacity but incur cubic cost. Sparse banks increase the number of addressable slots but introduce routing and update complexity. All fixed-state mechanisms compress history and cannot reproduce arbitrary softmax attention exactly.

5. Sparse and gradient-programmed fast memories

Fast-weight programming need not use a dense matrix accumulator.

"Fast-weight Product Key Memory" (Zhao et al., 2 Jan 2026) transforms Product Key Memory into a dynamic episodic memory. A query is split into two subqueries, each addressing a smaller key table. Product-key combinations define $v_j$2 conceptual slots, while retrieval uses top-$v_j$3 selection over each subtable and searches only the resulting $v_j$4 candidate pairs.

The module uses inverse-distance weighting:

$v_j$5

FwPKM updates its key and value parameters through local chunk-level gradient descent. For a target value $v_j$6 and retrieval $v_j$7, it minimizes

$v_j$8

Value rows are updated by local MSE gradients, while key tables receive a marginal-entropy addressing loss to prevent memory collapse. The module combines episodic retrieval with the ordinary slow pathway:

$v_j$9

Its memory is explicitly indexed rather than compressed into a fixed-dimensional superposition. Experiments used $v_t$00 conceptual slots and top-$v_t$01 retrieval for FwPKM. The method improved long-context behavior and remained effective in Needle in a Haystack evaluations at 128K tokens despite training on 4K-token sequences. Exact perplexity and NIAH tables are not supplied in the cited data, and sparse top-$v_t$02 gathering reduced hardware throughput despite favorable arithmetic scaling.

Fast Weight Layers provide another interpretation: fast weights can be generated by gradient descent itself. "Meta-Learning Fast Weight LLMs" (Clark et al., 2022) places a small FWL after the last Transformer attention layer. It computes per-position gradients of the FWL loss and applies them to fast copies of its parameters:

$v_t$03

For a matrix multiplication, each gradient is rank one:

$v_t$04

Consequently, adapted outputs can be written as a causal linear-attention operation whose keys are prior FWL inputs and whose values are output-gradient signals. The retrieved value is therefore an error-driven correction rather than an independently projected content vector.

FWLs approximate dynamic evaluation while updating only a small added component. In the reported Transformer-XL experiment, FWL reduced test perplexity from $v_t$05 to $v_t$06, compared with $v_t$07 for full dynamic evaluation, while achieving 1340 tokens/s versus 510 tokens/s for dynamic evaluation. The paper reports less than 30% additional FLOPs and less than 20% wall-clock overhead in the relevant settings. Training uses second-order gradients, but the FWL remains small and avoids differentiating recurrently through the entire Transformer.

6. Efficient attention, applications, and unresolved issues

Fast Weight Attention offers a general route to reducing attention’s dependence on sequence length. Full softmax attention stores or accesses all previous key–value pairs and permits arbitrary pairwise interactions, but autoregressive generation requires $v_t$08 work per token. Fast-weight and linear-attention methods replace this history with a recurrent sufficient statistic. The resulting state is constant with respect to $v_t$09, although its size may be quadratic or cubic in feature dimensions.

The same perspective supports compression and routing strategies. MiTA interprets full attention as an $v_t$10-width dynamically instantiated two-layer MLP, with one hidden unit per key–value pair. "MiTA Attention: Efficient Fast-Weight Scaling via a Mixture of Top-$v_t$11 Activations" (Wen et al., 1 Feb 2026) compresses the $v_t$12-width memory into $v_t$13 landmark-based global pairs and constructs deformable experts from top-$v_t$14 activated original pairs. Each query receives $v_t$15 attended pairs instead of $v_t$16.

With fixed $v_t$17, $v_t$18, and routing count $v_t$19, the output path costs approximately

$v_t$20

rather than $v_t$21. The method combines global compression with content-dependent sparse retrieval. Its principal systems limitation is irregular top-$v_t$22 gathering and random memory access. Preliminary vision experiments reported MiTA-ViT-T and MiTA-ViT-S ImageNet-1K accuracies of $v_t$23 and $v_t$24, and Long Range Arena average accuracy of $v_t$25, close to standard attention’s $v_t$26. The paper does not establish large-scale language-model pretraining results.

Fast-weight methods have been evaluated across several domains:

• Associative sequence learning: FW-LSTM improves difficult long-distance retrieval, especially on mART (Keller et al., 2018).
• Compositional language reasoning: FWM supports mutable bindings and multi-hop inference on catbAbI (Schlag et al., 2020).
• Language modeling: Delta Networks, decaying fast weights, FWLs, Falcon variants, and recurrent FWPs improve or approach Transformer baselines under different settings (Schlag et al., 2021, Mao, 2022, Clark et al., 2022, Zhang et al., 27 Aug 2026).
• Reinforcement learning: recurrent fast-weight programmers improve performance over LSTM in several Atari games and generalize across randomly generated POMDP graphs (Irie et al., 2021, Schlag et al., 2020).
• Visual familiarity: LoRA-mediated adaptation broadens attention scope, strengthens early-layer global alignment, and improves robustness to image noise, although it is not an explicit online fast-weight recurrence (Li et al., 7 Aug 2025).
• Traffic-matrix forecasting: gated quantum-inspired fast-weight programmers provide compact recurrent temporal models, with G-QKANFWP attaining pooled RMSE $v_t$27 using 8,189 parameters under the reported Abilene protocol (Peng et al., 26 Jun 2026).
• Long-context episodic retrieval: FwPKM dynamically rewrites a product-key memory and extrapolates to 128K-token Needle in a Haystack contexts (Zhao et al., 2 Jan 2026).

Several issues remain unresolved. Dense fast-weight matrices have quadratic state size, tensor-product memories have cubic cost, and sparse memories incur routing and memory-access overhead. Additive updates suffer interference and cannot remove obsolete bindings; delta rules require additional reads and update computations; decay rules trade retention against accumulation and stability. Fixed-state memories also compress context and therefore lack the unbounded addressability of explicit softmax attention.

The distinction between fast-weight attention and related mechanisms is consequently important. Shared attention weights reuse attention computations across Transformer layers but do not create a recurrent associative memory (Xiao et al., 2019). LoRA adapters provide isolated low-rank plasticity but are not necessarily updated online (Li et al., 7 Aug 2025). Product-key fast memories use explicit sparse banks and chunk-level optimization rather than dense linear-attention accumulators (Zhao et al., 2 Jan 2026). FWLs implement gradient descent as linear attention, so their retrieved values are error signals rather than conventional value projections (Clark et al., 2022).

Current research therefore separates four design questions: what information the fast state stores, how temporal alignment defines its local objective, how plasticity and overwriting are controlled, and how forgetting or bounded rehearsal is implemented. Fast Weight Attention is best understood as the resulting framework: an online-programmable, context-dependent memory that connects associative storage, recurrent computation, linear attention, continual learning, dynamic evaluation, selective state-space modeling, and sparse episodic retrieval.