--- title: Fast Weight Attention url: https://www.emergentmind.com/topics/fast-weight-attention type: topic --- # Fast Weight Attention Fast Weight Attention is a family of attention and recurrent-memory mechanisms in which a slowly trained controller dynamically writes and reads rapidly changing parameters or associative states. The fast state commonly stores key–value bindings through outer-product updates and retrieves them with a query, producing a recurrent or linearized form of attention with state size independent of sequence length. Depending on the architecture, fast weights may be additive, decayed, selectively overwritten by a delta rule, organized as a tensor-product memory, sparsely indexed in a product-key bank, or interpreted as online gradient updates. The term therefore denotes a design space rather than one unique operator. ## 1. Conceptual foundations and historical development Fast Weight Attention separates two temporal scales. Slow weights are ordinary trainable parameters, such as projection matrices, recurrent parameters, embeddings, and biases, optimized across examples by gradient descent. Fast weights are dynamically changing parameters or state variables generated or modified during processing of a particular sequence, document, episode, or context. A slow network acts as a programmer: it produces keys, values, queries, gates, update rates, or candidate fast parameters. A fast network or memory then uses the current state to generate outputs. The classical fast-weight formulation associates a key $k_t$ with a value $v_t$ by writing an outer product into a matrix: $$ W_t=W_{t-1}+v_t k_t^\top. $$ A query retrieves the accumulated content through $$ y_t=W_tq_t. $$ Starting from $W_0=0$ gives $$ W_t=\sum_{j=1}^{t}v_jk_j^\top, \qquad y_t=\sum_{j=1}^{t}v_j(k_j^\top q_t). $$ The matrix is an associative memory: similarity between a query and a stored key determines the contribution of the corresponding value. The formulation descends from fast-weight proposals by Hinton and Plaut, Schmidhuber’s fast-weight programmers, and the fast-weight mechanism adopted by Ba et al. for attending to the recent past. These approaches treat rapidly changing connections as short-term memory or as a context-dependent program. The modern connection to attention follows from reassociating matrix multiplication. For causal, unnormalized attention with projected keys, values, and queries, $$ y_t=V_{1:t}K_{1:t}^{\top}q_t = \left(\sum_{j=1}^{t}v_jk_j^\top\right)q_t. $$ Thus causal linear attention is an additive fast-weight programmer: the sequence-dependent matrix $\sum_jv_jk_j^\top$ is a fast associative memory, while the projections generating $k_j$, $v_j$, and $q_t$ are slow parameters. This formal equivalence is developed in "Linear Transformers Are Secretly Fast Weight Programmers" [2102.11174] and extended to recurrent fast networks and recurrent programmers in "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers" [2106.06295]. The architecture should not be conflated with all forms of dynamic parameterization. "Modeling Rapid Contextual Learning in the Visual Cortex with Fast-Weight Deep Autoencoder Networks" [2508.04988] uses LoRA adapters in ViT attention as a fast-weight-like slow/fast architecture, but its adapters are optimized during familiarity training rather than updated online through an explicit associative-memory recurrence. Similarly, vertical reuse of attention weights in the Shared Attention Network is attention-state reuse across Transformer depth, not canonical fast-weight programming [1906.11024]. ## 2. Associative-memory operators ### Additive outer-product memory The basic update is $$ W_t=W_{t-1}+v_tk_t^\top. $$ It stores a superposition of bindings. The readout $$ W_tq_t=\sum_{j=1}^{t}v_j(k_j^\top q_t) $$ is an unnormalized linear-attention computation. A feature-map version replaces $k_t$ and $q_t$ by $\phi(k_t)$ and $\phi(q_t)$ and may maintain a normalizer: $$ W_t=W_{t-1}+v_t\phi(k_t)^\top, \qquad z_t=z_{t-1}+\phi(k_t), $$ $$ y_t= \frac{W_t\phi(q_t)} {z_t^\top\phi(q_t)}. $$ The recurrent state has fixed dimensions with respect to sequence length, giving linear sequence processing and constant recurrent-state size. Its effective capacity is constrained by key or feature dimensionality. Nonorthogonal keys interfere, and additive updates cannot explicitly remove obsolete associations. "Linear Transformers Are Secretly Fast Weight Programmers" [2102.11174] analyzes this limitation through an orthogonality argument. If projected keys must be mutually orthogonal in a feature space of dimension $d_{\text{dot}}$, then at most $d_{\text{dot}}$ independent associations can be represented without interference under the idealized criterion. Its experiments report degradation near 60 associations for Linear Attention with $d_{\text{dot}}=64$, and near nominal capacities of 128, 256, and 384 for DPFP feature maps with corresponding settings. ### Auto-associative fast weights "Fast Weight Long Short-Term Memory" [1804.06511] integrates additive fast weights into an LSTM. Its fast matrix is $$ \mathbf A_t = \lambda\mathbf A_{t-1} + \eta\mathbf g_t\mathbf g_t^\top, $$ where $\mathbf g_t$ is an LSTM candidate activation, $\eta$ is the writing rate, and $\lambda$ is the retention coefficient. With $\mathbf A_0=0$, $$ \mathbf A_t = \eta\sum_{s=1}^{t} \lambda^{t-s}\mathbf g_s\mathbf g_s^\top. $$ Reading with $\mathbf g_t$ gives $$ \mathbf A_t\mathbf g_t = \eta\sum_{s=1}^{t} \lambda^{t-s} \mathbf g_s (\mathbf g_s^\top\mathbf g_t). $$ The same activation functions as both write vector and query make this mechanism auto-associative. The reported experiments used $\eta=1.0$ and $\lambda=0.99$. The fast-weight LSTM combines the matrix memory with the conventional LSTM cell: $$ \mathbf c_t = \mathbf f_t\odot\mathbf c_{t-1} + \mathbf i_t\odot \left( \hat{\mathbf g}_t+\mathbf A_t\mathbf g_t \right), $$ followed by $$ \mathbf h_t=\mathbf o_t\odot\tanh(\mathbf c_t). $$ The LSTM gates regulate candidate writing, conventional-state retention, incorporation of retrieved memory, and output exposure. The resulting model has vector-valued gated memory and matrix-valued associative memory operating on different temporal scales. On the modified associative retrieval task, FW-LSTM substantially exceeded both layer-normalized LSTM and fast-weight RNN baselines. At hidden size $50$ and mART length $K=16$, its reported test accuracy was $93.3\%$, compared with $25.7\%$ for LN-LSTM and $29.0\%$ for FW-RNN. At hidden size $100$, FW-LSTM achieved $92.6\%$, compared with $22.5\%$ and $30.5\%$, respectively. These results support architectural synergy, although they do not formally isolate the causal contribution of each gate or normalization choice. ### Delta-rule memory A delta rule first retrieves the memory’s current prediction and then writes only the error: $$ \bar v_t=W_{t-1}\phi(k_t), $$ $$ W_t = W_{t-1} + \beta_t \left(v_t-\bar v_t\right) \otimes\phi(k_t). $$ The update strength is learned, commonly as $$ \beta_t=\sigma(w_\beta x_t). $$ This permits selective replacement: $\beta_t\approx1$ strongly corrects the existing mapping, whereas $\beta_t\approx0$ preserves it. Unlike global decay, correcting one key need not uniformly weaken unrelated associations. For orthonormal keys, if $$ W=v_1k_1^\top+v_2k_2^\top $$ and a new value $v_3$ arrives for $k_2$, then $$ W'=W+\beta(v_3-v_2)k_2^\top $$ preserves the association for $k_1$ while interpolating the association for $k_2$. The Delta Net and DPFP experiments in [2102.11174] show that delta updates outperform purely additive updates when keys recur with different values. In language modeling, the delta rule improved both deterministic Linear Transformer and Performer variants, including in settings that were not over the nominal memory-capacity limit. ## 3. Recurrent architectures and learning objectives Fast Weight Attention can be viewed as an online learning algorithm embedded in a sequence model. This perspective is developed explicitly in "Fast Weight Attention for Continual Learning" [2608.27763], which treats the recurrent state as a predictor updated by local optimization. For a memory $\mathbf S_t\in\mathbb R^{d_x\times d_v}$, a feature $\mathbf x_t$, and target $\mathbf y_t$, the memory predicts $$ \widehat{\mathbf y}_t = \mathbf S_{t-1}^{\top}\mathbf x_t. $$ The regression residual is $$ \mathbf r_t = \mathbf y_t-\widehat{\mathbf y}_t. $$ A normalized squared-error update is $$ \mathbf S_t = (1-\eta_t\lambda_t)\mathbf S_{t-1} + \eta_t\mathbf x_t\mathbf r_t^\top, $$ with $$ \eta_t = \frac{\beta_t} {\|\mathbf x_t\|_2^2+\lambda_t+\varepsilon}, \qquad \beta_t\in(0,2). $$ This is a normalized least-mean-squares update when $\lambda_t=0$ and $\varepsilon=0$. The paper distinguishes scalar plasticity, per-value-channel plasticity, and sliding-window mini-batch updates in the Falcon-1, Falcon-2, and Falcon-3 families. Suffix A denotes the corresponding inner-product objective rather than squared-error regression. A central issue is temporal alignment. Under read-after-write semantics, the local example for prefix prediction is $$ \mathbf x_t=\phi(\mathbf k_{t-1}), \qquad \mathbf y_t=\mathbf v_t. $$ The preceding feature predicts the newly revealed value. The same-step association $(\phi(\mathbf k_t),\mathbf v_t)$ remains causal but optimizes a different internal objective. This distinction separates temporal alignment from plasticity, forgetting, and bounded rehearsal. The Falcon updates admit recurrent, masked-parallel, and chunk-parallel forms. Scalar-decay recurrences can be unrolled into decay-weighted causal attention: $$ \mathbf S_t = \gamma_t\mathbf S_{t-1} + \eta_t\mathbf x_t\mathbf v_t^\top, \qquad \gamma_t=1-\eta_t\lambda_t. $$ Regression variants require more complex causal triangular solves, whereas additive inner-product variants reduce to masked linear-attention expressions. Chunk-parallel execution propagates only boundary states sequentially while computing intra-chunk interactions in parallel. Other recurrent fast-weight programmers alter the architecture of the slow or fast network. Delta RNN adds a recurrent fast matrix; Delta LSTM dynamically programs fast gated recurrent matrices; Recurrent Delta Net makes the slow programmer depend on the previous fast output [2106.06295]. The resulting design space independently varies slow-network recurrence, fast-network recurrence, update rule, activation placement, normalization, and number of fast matrices. In "Learning Associative Inference Using Fast Weight Memory" [2011.07831], the memory is a third-order tensor-like map. Two keys form a tensor-product address: $$ a_t=\operatorname{vec}(k_{1,t}\otimes k_{2,t}), $$ and the memory stores a value at that address. A delta-style write is $$ W'_t = W_{t-1} + \beta_t a_t (v_t-v_{\mathrm{old},t})^\top. $$ Multiple recurrent reads permit chained inference. A retrieved value becomes part of the next query, supporting mappings such as $$ a\rightarrow b,\qquad b\rightarrow c \quad\Longrightarrow\quad a\rightarrow c. $$ The tensor-product address has $O(d_{\mathrm{FWM}}^2)$ features and the explicit third-order memory has $O(d_{\mathrm{FWM}}^3)$ storage and computation under dense implementation. In concatenated-bAbI, FWM achieved $96.75\%$ QA accuracy, compared with $80.88\%$ for LSTM and $87.66\%$ for Transformer-XL. Its benefits were strongest in compositional reasoning and continual streams containing obsolete or context-dependent facts. ## 4. Forgetting, plasticity, and memory capacity Fast-weight systems differ substantially in how they control retention. **Fixed decay** multiplies the previous memory by a scalar: $$ A_t=\lambda A_{t-1}+\eta v_tk_t^\top. $$ It provides geometric recency weighting but applies the same retention coefficient to every memory component. **Scalar gating** uses a learned gate, often $$ g_t=\sigma(W_gx_t), $$ to interpolate the old state and a new write. Its uniform action limits fine-grained control over value and feature dimensions. **Element-wise decay** gives each memory element a learned coefficient. "Fine-Tuning Pre-trained Transformers into Decaying Fast Weights" [2210.04243] uses $$ S_t=G_t\otimes S_{t-1}+v_t\phi(k_t)^\top, $$ where $$ G_t= \sigma(W_zx_t+b_z) \sigma(W_fx_t+b_f)^\top. $$ The resulting $d\times m$ decay matrix is low-rank parameterized at each step, while every state entry has its own coefficient in $(0,1)$. The new write is added directly rather than multiplied by a complementary gate. Sigmoid-bounded decay is necessary for stability; removing the sigmoid caused divergence in the reported experiments. The final decaying fast-weight model removes the nonlinear feature map and attention normalizer, using a linear feature projection that can be absorbed into the key or query projection. It therefore maintains only $S_t$ and reads $$ y_t=S_t\phi(q_t). $$ For autoregressive generation, its per-token computation and state are $O(dm)$, independent of context length $T$. Standard cached self-attention requires $O(Td)$ work per token and $O(Td)$ cache memory. Training remains more demanding: because the decay recurrence is non-reversible, the reported training memory is $O(Tdm)$ and parallel time is $O(T)$ over a sequence. Fine-tuning GPT-2 with decaying fast weights reached validation perplexity $14.6$ at fast-weight dimension $m=32$, compared with $14.5$ for the GPT-2 baseline. The method therefore recovered $99\%$ of the baseline performance according to the paper, while replacing context-length-dependent attention with a fixed-size recurrent state. Its success depends on fine-tuning; conversion without additional training is not established. Capacity remains a fundamental limitation. Additive matrix memories superpose associations, so similar keys cause crosstalk. Tensor-product memories increase representational capacity but incur cubic cost. Sparse banks increase the number of addressable slots but introduce routing and update complexity. All fixed-state mechanisms compress history and cannot reproduce arbitrary softmax attention exactly. ## 5. Sparse and gradient-programmed fast memories Fast-weight programming need not use a dense matrix accumulator. "Fast-weight Product Key Memory" [2601.00671] transforms Product Key Memory into a dynamic episodic memory. A query is split into two subqueries, each addressing a smaller key table. Product-key combinations define $N=(\sqrt N)^2$ conceptual slots, while retrieval uses top-$k$ selection over each subtable and searches only the resulting $k^2$ candidate pairs. The module uses inverse-distance weighting: $$ s_i^{\mathrm{IDW}} = -\log\left(\varepsilon+\|q-K_i\|_2^2\right), \qquad \varepsilon=10^{-3}. $$ FwPKM updates its key and value parameters through local chunk-level gradient descent. For a target value $v_t$ and retrieval $\widehat v_t$, it minimizes $$ \mathcal L_{\mathrm{MSE}} = \frac12\|v_t-\widehat v_t\|_2^2. $$ Value rows are updated by local MSE gradients, while key tables receive a marginal-entropy addressing loss to prevent memory collapse. The module combines episodic retrieval with the ordinary slow pathway: $$ o_t=g_t\widehat v_{t+1}+(1-g_t)v_t. $$ Its memory is explicitly indexed rather than compressed into a fixed-dimensional superposition. Experiments used $512^2\approx262{,}000$ conceptual slots and top-$8$ retrieval for FwPKM. The method improved long-context behavior and remained effective in Needle in a Haystack evaluations at 128K tokens despite training on 4K-token sequences. Exact perplexity and NIAH tables are not supplied in the cited data, and sparse top-$k$ gathering reduced hardware throughput despite favorable arithmetic scaling. Fast Weight Layers provide another interpretation: fast weights can be generated by gradient descent itself. "Meta-Learning Fast Weight Language Models" [2212.02475] places a small FWL after the last Transformer attention layer. It computes per-position gradients of the FWL loss and applies them to fast copies of its parameters: $$ \theta'_t = \theta-\alpha\circ \sum_{i