Fast Weight Attention for Continual Learning

Abstract: Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step $t$ is the prefix-aligned pair $$(\mathbf{x}<em>t,\mathbf{y}_t)=(φ(\mathbf{k}</em>{t-1}),\mathbf{v}_t)$$. The common same-step association $(φ(\mathbf{k}_t),\mathbf{v}_t)$ remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.

• The paper presents fast recurrent fast-weight sequence models for continual learning and online optimization using state transitions to predict new values based on preceding information.
• falcons are designed as taxonomies via two local objectives (squared-error regression and negative inner product), three update granularities and normalization and decay mechanisms
• Fast convergent deduction improves models predictive success on key-tasks limited by state information complexity

Problem formulation and central claim

The paper frames recurrent fast-weight sequence models as continual learners whose recurrent state is updated by an online optimization rule. Its central claim is that the temporal alignment of the local fast-memory training pair is not merely an indexing convention. Under read-after-write autoregressive semantics, the state available before observing token $t$ should predict the newly revealed value using the prefix feature $\phi(\mathbf{k}_{t-1})$. The corresponding causal pair is therefore

$$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$

rather than the more common same-step association $(\phi(\mathbf{k}_t),\mathbf{v}_t)$. The latter remains causal, but it defines a different internal objective: it associates a token’s value with a feature generated from that same token, whereas the shifted rule trains the state to predict the newly revealed value from information available in the preceding prefix.

This distinction provides the organizing principle for the Falcon family. The fast state $\mathbf{S}_t$ is a fixed-size matrix that maps write features to value predictions and is read after each update through $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$. Consequently, the model does not store an expanding KV cache. Instead, it performs bounded online learning inside the forward pass, with the state transition controlling plasticity, interference, forgetting, and the effective temporal receptive field. The paper’s formulation connects this view to Delta Networks, linear attention, selective SSMs, test-time regression, and internal-memory approaches such as Titans and ATLAS (2608.27763).

The contribution is therefore broader than a single recurrence. It is a taxonomy of state updates obtained by combining two local objectives, three update granularities, and explicit normalization and decay mechanisms:

Objective	Scalar update	Per-channel update	Sliding-window update
Squared-error regression	Falcon-1	Falcon-2	Falcon-3
Negative inner product	Falcon-1A	Falcon-2A	Falcon-3A

The numerical index denotes the update structure: one scalar gain, one gain per value channel, or a finite-history mini-batch. The suffix “A” denotes the inner-product family.

Objective alignment and online regression

The regression family begins with an instantaneous ridge objective,

$$\ell_t(\mathbf{S}) = \frac{1}{2} \left\| \mathbf{S}^{\top}\mathbf{x}_t-\mathbf{y}_t \right\|_2^2 + \frac{\lambda_t}{2}\|\mathbf{S}\|_F^2.$$

Evaluated at the pre-update state, the residual is

$$\mathbf{r}_t = \mathbf{y}_t-\mathbf{S}_{t-1}^{\top}\mathbf{x}_t.$$

A single gradient step gives

$$\mathbf{S}_t = (1-\eta_t\lambda_t)\mathbf{S}_{t-1} + \eta_t\mathbf{x}_t\mathbf{r}_t^\top.$$

With the shifted assignment $\mathbf{x}_t=\phi(\mathbf{k}_{t-1})$ and $\phi(\mathbf{k}_{t-1})$0, this is Falcon-1. Equivalently, the update consists of a rank-one feature-direction edit followed by a target write:

$\phi(\mathbf{k}_{t-1})$1

The term $\phi(\mathbf{k}_{t-1})$2 is the Hessian action of the instantaneous squared-error loss. It selectively modifies the predictor in the current write-feature direction before injecting the new target. This explains the principal distinction from additive linear attention: regression writes are error-driven and can overwrite an existing association, whereas additive inner-product writes accumulate correlations unless explicit decay is applied.

The proposed step size is an NLMS-style normalization,

$\phi(\mathbf{k}_{t-1})$3

The denominator is the local smoothness scale of the ridge objective. For nondegenerate inputs, the objective is $\phi(\mathbf{k}_{t-1})$4-smooth with $\phi(\mathbf{k}_{t-1})$5, so $\phi(\mathbf{k}_{t-1})$6 implies $\phi(\mathbf{k}_{t-1})$7 and yields pointwise descent in the instantaneous objective. The guarantee is explicitly local: it does not imply monotone decrease of the cumulative online loss or of the outer autoregressive likelihood. This qualification is important because the state is continually evaluated on changing examples, and because the slow network parameters are trained through the update.

Falcon-2 extends this construction by assigning a separate gain $\phi(\mathbf{k}_{t-1})$8 to each value channel. The residual and write feature remain shared, but the columns of $\phi(\mathbf{k}_{t-1})$9 have independent plasticity and decay trajectories:

$$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$0

Since the squared-error objective decomposes over value coordinates, the per-step descent argument applies column-wise when each $$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$1. Falcon-2 is thus not a full matrix-valued adaptive optimizer; it is a collection of independent scalar OGD updates sharing the same feature-direction statistics.

Inner-product updates and their interpretation

The second objective is

$$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$2

Its gradient produces the direct-write recurrence

$$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$3

This is Falcon-1A. When $$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$4, it reduces to the standard additive fast-weight or denominator-free linear-attention update, except that the write feature is shifted to $$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$5. The paper emphasizes that this objective is structurally different from regression. With $$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$6, the inner-product objective is linear and unbounded below, so the update is not minimizing a finite optimum. It should instead be interpreted as Hebbian correlation accumulation with magnitude control.

Although the inner-product objective has curvature determined only by $$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$7, Falcon-A uses

$$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$8

to normalize the write magnitude. The feature-energy term is therefore not required for a curvature-based descent guarantee; it is a scale stabilizer that provides a sensible limit as $$(\mathbf{x}_t,\mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}),\mathbf{v}_t),$$9. When $(\phi(\mathbf{k}_t),\mathbf{v}_t)$0, the resulting step remains within the relevant smoothness range for the regularized objective.

Falcon-2A introduces per-channel gains and per-channel decay:

$(\phi(\mathbf{k}_t),\mathbf{v}_t)$1

Unlike Falcon-2, there is no residual subtraction. Each value channel directly accumulates its target under a channel-specific gain. This yields an unrolled representation with a distinct decay-weighted causal mask for every output channel. The result is more expressive than a scalar decay but also introduces additional mask and kernel structure in the parallel formulation.

The use of RMS normalization for query and key projections is another important implementation decision. Fast-weight reads scale with query and state norms, while direct writes scale with feature norms. RMS-normalizing the projections bounds the feature energy and maintains coordinate magnitudes suitable for mixed-precision execution. Values are not normalized by default, leaving value-scale control to the learned projections and update gains.

Sliding-window continual learning

Falcon-3 and Falcon-3A replace the single-example update with a mini-batch step over the most recent $(\phi(\mathbf{k}_t),\mathbf{v}_t)$2 causal pairs. For regression, the windowed objective is the average squared error plus ridge regularization. Its gradient is

$(\phi(\mathbf{k}_t),\mathbf{v}_t)$3

where $(\phi(\mathbf{k}_t),\mathbf{v}_t)$4 is the average feature covariance and $(\phi(\mathbf{k}_t),\mathbf{v}_t)$5 is the average feature-value cross-covariance. The update is

$(\phi(\mathbf{k}_t),\mathbf{v}_t)$6

All residuals are evaluated at the pre-update state. This is a genuine mini-batch gradient step, not a sequence of sequential single-example updates. Its normalization uses the spectral smoothness scale

$(\phi(\mathbf{k}_t),\mathbf{v}_t)$7

The averaging convention is consequential. Neither the cross-covariance magnitude nor the smoothness statistic grows linearly with the nominal window size $(\phi(\mathbf{k}_t),\mathbf{v}_t)$8. With RMS-normalized features, the spectral scale remains controlled by the feature dimension rather than by $(\phi(\mathbf{k}_t),\mathbf{v}_t)$9. Thus increasing the rehearsal window does not systematically amplify either the write injection or the decay fraction. The window acts as bounded rehearsal: it reduces update noise and retains a finite recent history without requiring an unbounded external memory.

Falcon-3A applies the same windowing principle to direct inner-product writes:

$\mathbf{S}_t$0

with

$\mathbf{S}_t$1

where $\mathbf{S}_t$2 is the average feature energy over the active window. Falcon-3A reduces to Falcon-1A when $\mathbf{S}_t$3. Its window operator is local, but after recurrence unrolling, decay can induce a dense lower-triangular influence pattern. This distinction matters for implementation: the explicit window is bounded, while the effective contribution of an earlier write can persist through subsequent decay factors.

Recurrent, parallel, and chunk-parallel equivalence

A major technical component of the paper is the derivation of equivalent execution forms. The recurrent forms provide causal scans with fixed-size state. For scalar direct writes, unrolling gives

$\mathbf{S}_t$4

where $\mathbf{S}_t$5 and $\mathbf{S}_t$6. The readout is therefore equivalent to a causal attention operation with a decay-weighted mask. This establishes an exact algebraic bridge between recurrent fast-weight memory and masked linear attention.

For Falcon-1 and Falcon-2, the residual recurrence is more involved because each update edits the previous state in a feature-dependent direction. The paper uses WY-style factorizations and triangular solves to express the sequence of rank-one edits as a masked parallel computation. In a chunked implementation, intra-chunk interactions are evaluated through Gram matrices and causal triangular systems, while the fixed-size state is propagated between chunks. Falcon-2 shares the feature Gram matrix across value channels but solves channel-specific triangular systems, enabling batched GPU execution.

Falcon-3 is a rank-$\mathbf{S}_t$7 affine recurrence after zero-padding the active windows. Its chunk-parallel implementation uses a three-phase structure:

1. parallel intra-chunk construction of window statistics and local triangular solves;
2. sequential or associative inter-chunk propagation of the fixed-size state;
3. parallel materialization of token outputs.

Falcon-3A admits a simpler masked-attention interpretation because its update is additive. Its mask combines the bounded window operator with cumulative decay. For exact continuation across segments, however, the recurrent matrix state alone is insufficient: the last $\mathbf{S}_t$8 causal pairs must also be retained. This is a concrete memory requirement of the sliding-window design and distinguishes it from strictly state-only recurrences.

The implementation introduces positive-decay renormalization. Since log-space unrolling requires $\mathbf{S}_t$9, the raw decay fraction $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$0 is clamped below one. The paper is explicit that the per-step descent result applies to the unclamped recurrence. When the clamp activates, the implementation is instead interpreted as using an effective ridge coefficient $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$1. This preserves numerical stability but changes the exact optimization objective at those steps.

Empirical evaluation

The experiments evaluate 124M–130M parameter models trained on approximately 49.2B FineWeb-Edu tokens, using sequence length $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$2, global batch size $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$3, and $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$4 optimization steps. The main comparisons include a Transformer, RetNet/LightningAttn, Mamba-2, DeltaNet, Gated DeltaNet, Falcon-1A variants, Falcon-3A.3, and Falcon-1.3. The paper does not benchmark every proposed variant in the main experiments: Falcon-2, Falcon-2A, and Falcon-3 regression are defined technically but are not separately evaluated there.

The strongest language-modeling result is obtained by the regression ablation Falcon-1.3, not by an inner-product variant. On FineWeb-Edu, Falcon-1.3 reaches perplexity 17.10, compared with 17.32 for Gated DeltaNet, 17.38 for the Transformer, and 17.40 for Falcon-1A.3. This is a meaningful result for the paper’s optimization perspective: the error-driven shifted regression update is competitive with established recurrent baselines and slightly outperforms the listed Transformer baseline under the matched small-model setup. However, the result does not establish a uniform advantage for the Falcon family, since the best-performing configuration is a particular regression ablation and the gains are not consistent across all evaluation datasets.

On WikiText, Falcon-1.3 obtains perplexity 33.00, compared with 30.99 for Gated DeltaNet and 33.25 for the Transformer. On LAMBADA, it reaches 48.70, essentially matching Mamba-2 at 48.74 but trailing Gated DeltaNet at 46.70. These results reinforce the paper’s more cautious empirical conclusion: the proposed updates preserve competitive language-model quality, but they do not dominate recurrent baselines across metrics.

The downstream results similarly favor different variants under different evaluation protocols. Falcon-1A.2 achieves the highest listed small-model zero-shot average, 49.30, while Falcon-1.3 achieves the strongest recurrent one-shot average, 49.54. The Transformer obtains 49.67 one-shot average, so Falcon-1.3 does not exceed the Transformer on that aggregate. The results instead indicate that context-conditioned gains, normalization choices, and the distinction between $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$5-conditioning and direct $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$6-conditioning materially affect performance.

The clearest advantage appears in variable-length reversed multi-digit addition. Models are trained on widths from 1 to 32 digits and evaluated on teacher-forced target suffixes averaging widths 33 to 48. Falcon-3A.3 achieves the best mean accuracy, 87.2, followed by Falcon-1A.3 at 85.9. The Transformer reaches 65.8, Mamba-2 75.2, and RetNet/LightningAttn 82.9. At 48 digits, Falcon-1A.3 and Falcon-3A.3 both achieve 69.0%, compared with 49.0% for the Transformer and 51.0% for Mamba-2.

The arithmetic result supports the claim that shifted, normalized writes can improve extrapolation when the task depends on causal storage and carry propagation. It does not, however, isolate the effect of temporal alignment from all other factors: the evaluated Falcon systems also use architectural, normalization, conditioning, and decay choices. The paper appropriately characterizes this experiment as controlled evidence about memory-writing behavior rather than as a general language-modeling result.

Limitations and open questions

The strongest theoretical guarantee is limited to per-step descent in the instantaneous regularized objective. It does not establish stability of the nonstationary online process, convergence of the cumulative loss, preservation of useful memories under continual distribution shift, or improvement of the outer autoregressive objective. The inner-product objective is particularly dependent on interpretation: when $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$7, it is unbounded below, so its normalized update is a magnitude-controlled Hebbian rule rather than optimization toward a finite minimizer.

The positive-decay clamp introduces another explicit qualification. The unclamped recurrence has the stated gradient interpretation, but the clamped implementation uses an effective shrinkage coefficient and therefore no longer exactly follows the original ridge objective. The paper does not quantify how frequently the clamp activates in the reported models or how much it contributes to empirical behavior.

The sliding-window methods require auxiliary state for exact segment continuation. Falcon-3 and Falcon-3A need the matrix state together with a tail of recent causal pairs, and their computational cost scales with the window size. The paper therefore does not establish that bounded rehearsal is preferable to simpler learned decay or to an unwindowed recurrence at matched hardware and memory budgets.

Empirically, the evaluation is concentrated on small models and a single 50B-token training budget. The per-channel Falcon-2 and Falcon-2A variants, despite being a substantial part of the methodological contribution, are not independently benchmarked in the main experiments. Consequently, the claimed benefits of channel-specific plasticity remain primarily theoretical and implementation-oriented. The paper also leaves open whether the shifted alignment continues to help when the model’s slow network can already encode strong same-step associations, and whether the arithmetic gains persist under free-running rather than teacher-forced evaluation.

Conclusion

“Fast Weight Attention for Continual Learning” (2608.27763) presents recurrent fast-weight sequence modeling as online optimization over an explicitly aligned local memory objective. Its main conceptual contribution is the identification of the prefix-aligned pair $\mathbf{S}_t^\top\phi(\mathbf{q}_t)$8 under read-after-write semantics. From this alignment, it derives regression, inner-product, per-channel, and sliding-window updates with normalized gains, explicit forgetting, and exact recurrent-to-parallel transformations.

The empirical evidence is differentiated rather than uniform. Falcon-1.3 attains the best reported FineWeb-Edu perplexity at 17.10, while Falcon-3A.3 reaches 87.2% mean accuracy on variable-length arithmetic extrapolation. These results support the utility of objective-aligned fast-weight updates, particularly for causal storage and extrapolation, while leaving the relative value of per-channel plasticity, windowed rehearsal, and alignment choices open for broader-scale evaluation.