Linear Associative Memory

• Linear associative memory is a content-addressable system that encodes associations via superpositions of outer products and retrieves them using linear scoring.
• It serves as a baseline model for factual recall in linear networks and informs the theoretical foundation of attention mechanisms in Transformer architectures.
• Recent analyses quantify its capacity, retrieval dynamics, and optimal margins through Hebbian rules and extreme-value theory, guiding practical enhancements.

Searching arXiv for papers on linear associative memory and related associative-memory views in modern architectures. Linear associative memory (LAM) is a content-addressable memory in which associations between cues and targets are stored in a linear operator and retrieved by linear scoring or matrix-vector application. In the canonical hetero-associative construction, the memory matrix is a superposition of outer products of values and keys, while auto-association uses the same mechanism with identical cues and targets. Although this is a classical Hebbian model, recent work treats LAM as a precise object of asymptotic analysis, a baseline for factual recall in linear networks, and a unifying abstraction for attention, feed-forward networks, and fast-weight memory in Transformers (Krotov et al., 8 Jul 2025, Giorlandino et al., 11 May 2026, 2505.19488).

1. Core formalism

In its simplest hetero-associative form, LAM stores training pairs $\{(x_i,y_i)\}_{i=1}^p$ with $x_i\in\mathbb R^d$, $y_i\in\mathbb R^m$ in a memory matrix

$M=\sum_{i=1}^p y_i x_i^\top,$

and retrieves by

$\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$

Auto-association is the specialization $M=\sum_{i=1}^p x_i x_i^\top$, with $\hat x = Mx$. When exact recall of the training pairs is feasible, least-squares and ridge-regularized forms appear as

$$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$

where $X=[x_1,\dots,x_p]$ and $Y=[y_1,\dots,y_p]$ (Krotov et al., 8 Jul 2025).

A more recent formulation studies LAM as a score-based retrieval system over embedded inputs and outputs. With embeddings $x_i\in\mathbb R^d$0 and a linear map $x_i\in\mathbb R^d$1, scores are

$x_i\in\mathbb R^d$2

Strict separation requires

$x_i\in\mathbb R^d$3

equivalently

$x_i\in\mathbb R^d$4

Because these constraints are homogeneous in $x_i\in\mathbb R^d$5, the problem is invariant under the scaling $x_i\in\mathbb R^d$6; in analysis and experiments, scale is fixed by normalizing non-target scores to have unit variance (Giorlandino et al., 11 May 2026).

LAM also appears in recurrent form. In one benchmarked realization, the recall field is

$x_i\in\mathbb R^d$7

followed by winner-take-all dynamics,

$x_i\in\mathbb R^d$8

The architecture may be non-modular, with global $x_i\in\mathbb R^d$9-WTA, or modular, with local WTA inside each hypercolumn; in the modular case intra-hypercolumn synapses are removed and one winner is enforced per module (Lansner et al., 1 May 2026).

2. Retrieval criteria and capacity laws

The capacity of a linear associative memory depends strongly on the retrieval criterion. In a kernel view of retrieval with i.i.d. Gaussian keys and values, the inverse signal-to-noise ratio takes the form

$y_i\in\mathbb R^m$0

For the linear kernel $y_i\in\mathbb R^m$1, one obtains $y_i\in\mathbb R^m$2. For the ReLU kernel $y_i\in\mathbb R^m$3, $y_i\in\mathbb R^m$4. For the exponential kernel $y_i\in\mathbb R^m$5 with $y_i\in\mathbb R^m$6, the estimate becomes

$y_i\in\mathbb R^m$7

Holding SNR constant yields $y_i\in\mathbb R^m$8 for linear kernels and $y_i\in\mathbb R^m$9 for the exponential kernel (2505.19488).

For factual recall with strict winner-take-all decoding, the asymptotics are different because each association must beat many distractors. In the statistical-physics analysis of a linear memory that stores $M=\sum_{i=1}^p y_i x_i^\top,$0 Gaussian input-output associations in $M=\sum_{i=1}^p y_i x_i^\top,$1, the relevant load is

$M=\sum_{i=1}^p y_i x_i^\top,$2

and the decoupled model has a sharp threshold

$M=\sum_{i=1}^p y_i x_i^\top,$3

Finite-size corrections are of order $M=\sum_{i=1}^p y_i x_i^\top,$4, and the paper gives numerical and analytical evidence that the decoupled model and the original shared-output problem are equivalent in storage capacity, spectra of learned weights, and storage mechanism (Giorlandino et al., 11 May 2026).

A complementary asymptotic theory emphasizes that top-1 and listwise retrieval have different intrinsic scales. For the correlation matrix memory

$M=\sum_{i=1}^p y_i x_i^\top,$5

top-1 retrieval exhibits a sharp phase transition on the scale $M=\sum_{i=1}^p y_i x_i^\top,$6: if $M=\sum_{i=1}^p y_i x_i^\top,$7, then $M=\sum_{i=1}^p y_i x_i^\top,$8 succeeds for $M=\sum_{i=1}^p y_i x_i^\top,$9 and fails for $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$0. More generally, if

$\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$1

for sufficiently large $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$2, then with high probability no linear memory achieves top-1 retrieval. By contrast, under the Tail-Average Margin (TAM), the logarithmic penalty disappears and capacity follows the quadratic scale $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$3, with ridgeless critical load

$\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$4

This suggests that LAM does not have a single universal “capacity”; the relevant law depends on whether one asks for linear reconstruction, strict top-1 separation, or listwise certification (Barnfield et al., 6 May 2026).

3. Mechanistic picture: extreme values, optimal margins, and spectra

The recent strict-separation theory gives a detailed mechanistic account of how an optimal linear memory differs from naive superposition. The original problem reuses one output bank across all inputs, whereas the decoupled problem assigns each input its own independent set of competing outputs. Empirically, the load-accuracy curves for the original and decoupled problems coincide across $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$5 and rank ratio $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$6, and their learned singular-value distributions and effective-margin histograms also match closely (Giorlandino et al., 11 May 2026).

The central mechanism is not broad mean separation but calibration to an extreme-value threshold. For the naive Hebbian rule

$\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$7

normalized so that non-target scores have unit variance, the diagonal score distribution has mean $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$8 and variance $\hat y = Mx = \sum_{i=1}^p (x_i^\top x)\, y_i.$9, while off-target scores are $M=\sum_{i=1}^p x_i x_i^\top$0. A heuristic analysis gives a critical load $M=\sum_{i=1}^p x_i x_i^\top$1, well below the optimal $M=\sum_{i=1}^p x_i x_i^\top$2. The optimal solution instead leaves non-target scores approximately $M=\sum_{i=1}^p x_i x_i^\top$3 and raises the correct scores only to the extreme-value threshold generated by the competitor maxima, about $M=\sum_{i=1}^p x_i x_i^\top$4. In the replica-symmetric description, the energetic term satisfies

$M=\sum_{i=1}^p x_i x_i^\top$5

which yields the sharp balance at $M=\sum_{i=1}^p x_i x_i^\top$6 (Giorlandino et al., 11 May 2026).

The same analysis extends to rank-constrained two-layer linear architectures $M=\sum_{i=1}^p x_i x_i^\top$7 with rank $M=\sum_{i=1}^p x_i x_i^\top$8. In this setting,

$M=\sum_{i=1}^p x_i x_i^\top$9

with reported values $\hat x = Mx$0 for $\hat x = Mx$1. Near capacity, the singular-value density normalized to top singular value $\hat x = Mx$2 converges to

$\hat x = Mx$3

so the nonzero spectrum occupies only the upper tail of the quarter-circle law. The interpretation offered in the paper is that the memory concentrates its usable degrees of freedom into the top singular modes that are needed to satisfy extreme-value constraints with minimal excess margin (Giorlandino et al., 11 May 2026).

4. Attention, FFNs, and the kernel view of LAM

A major contemporary role of LAM is as a mathematical normal form for Transformer components. In the kernel formulation of attention, one writes

$\hat x = Mx$4

and then introduces a feature map $\hat x = Mx$5 such that $\hat x = Mx$6. This gives the associative-memory form

$\hat x = Mx$7

Softmax attention corresponds to the exponential kernel, with an infinite-dimensional feature map obtained from the power-series expansion, while linear attention uses kernels with finite-dimensional $\hat x = Mx$8. Feed-forward networks also fit this template: a two-layer FFN without bias,

$\hat x = Mx$9

can be written as

$$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$0

so FFNs function as static kernel associative memories, whereas attention stores dynamic associations in the KV cache (2505.19488).

This viewpoint becomes more concrete in Transformer parameterization. In the attention block,

$$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$1

the $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$2 pathway determines addressing, while $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$3 and $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$4 are treated as associative memory parameters. The same paper treats FFNs as associative memories through

$$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$5

and its gated variant. Prior work cited there shows that when $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$6 is fixed, $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$7 acts as a linear associative memory; because $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$8 and $$M^*=YX^\dagger,\qquad M_\lambda = YX^\top (XX^\top+\lambda I_d)^{-1},$$9 play symmetric roles, both are included in the associative-memory subsystem, and FFN weights are likewise regarded as key-value stores well approximated by linear associative memories (Wang et al., 30 Sep 2025).

5. Memory update dynamics and optimization

The associative-memory lens is equally useful for update dynamics. Under $X=[x_1,\dots,x_p]$0-linearization, normalized Softmax attention induces the recurrence

$X=[x_1,\dots,x_p]$1

whereas sliding-window attention with window $X=[x_1,\dots,x_p]$2 gives

$X=[x_1,\dots,x_p]$3

The first recurrence corresponds to a training objective with Frobenius regularization but yields normalization-induced memory shrinkage and gradient vanishing, since

$X=[x_1,\dots,x_p]$4

The second corresponds to a linear objective that is effectively unbounded below in $X=[x_1,\dots,x_p]$5, so the “enter minus leave” update can amplify memory and destabilize gradients (Liu et al., 8 Dec 2025).

GatedFWA modifies the sliding-window update by introducing a learnable, non-negative gate $X=[x_1,\dots,x_p]$6, accumulated as a decay bias in the logits. The resulting recurrence is

$X=[x_1,\dots,x_p]$7

This adds a learnable contraction on the carried state and a softened erasure of the leaving token. The Jacobian becomes

$X=[x_1,\dots,x_p]$8

so gradient flow is neither forced to vanish as in normalized Softmax nor left unbounded as in plain SWA. The implementation preserves linear-time behavior, with training arithmetic $X=[x_1,\dots,x_p]$9 and decode complexity $Y=[y_1,\dots,y_p]$0 (Liu et al., 8 Dec 2025).

Optimization of learned associative memories shows a related matrix-structural effect. In a one-layer LAM under class-imbalanced data, Muon updates a gradient $Y=[y_1,\dots,y_p]$1 by its orthogonal factor $Y=[y_1,\dots,y_p]$2, thereby equalizing magnitudes across singular directions. The paper proves that under orthonormal embeddings and two-class imbalance, Muon achieves balanced learning across classes regardless of feature embeddings, whereas Adam simplified to SignGD can create large disparities depending on embedding geometry. Empirically, Muon’s gains are concentrated in the associative-memory-heavy components of Transformers—VO attention weights and FFNs—and are accompanied by a more isotropic singular spectrum and substantially better optimization of tail classes on heavy-tailed corpora (Wang et al., 30 Sep 2025).

6. Benchmarks, limitations, and adjacent generalizations

Classical recurrent LAMs with local Hebbian plasticity remain an active benchmark domain. Studies comparing WILL, HEBB, HOPF, COV, PRCOV, and Bayesian rules such as BCPNN, BCP, and BOMs on sparse binary patterns reach a consistent ranking: original additive Hebb has the worst capacity, covariance learning is robust but moderate, Willshaw performs well for direct storage yet fails in prototype extraction, and Bayesian-Hebbian rules attain the highest capacity in almost all tested conditions (Lansner et al., 2023, Lansner et al., 1 May 2026). Quantitatively, one benchmark reports BCPNN at $Y=[y_1,\dots,y_p]$3 and $Y=[y_1,\dots,y_p]$4 bits per free weight on standard random patterns in non-modular and modular architectures, respectively, while another reports BCP and BOMs at $Y=[y_1,\dots,y_p]$5–$Y=[y_1,\dots,y_p]$6 bits per weight under $Y=[y_1,\dots,y_p]$7 input noise; both studies also find that prototype extraction is substantially harder than direct storage, and that PRCOV is especially robust under correlated inputs (Lansner et al., 2023, Lansner et al., 1 May 2026).

Several limits are now well delineated. Sharp asymptotic capacity results typically assume i.i.d. Gaussian inputs and outputs; universality is suggested but proofs remain open. In the strict-separation theory, equivalence between the original shared-output problem and the decoupled problem is a conjecture supported by evidence rather than a theorem, and the replica-symmetric analysis for rank-constrained models remains non-rigorous. Capacity is also sensitive to robustness requirements: because optimal solutions place target scores just above the extreme-value threshold, any added safety margin reduces the number of storable associations (Giorlandino et al., 11 May 2026).

A distinct response to the lack of dynamical basins in strict LAM is to retain linear encoder/decoder interfaces while replacing direct recall by threshold-linear latent dynamics. In such systems, patterns are associated with attractors of a threshold-linear network, corrupted inputs are driven into regions of attraction, and retrieval receives formal robustness guarantees through SDP- or LP-based certification of those regions. This line of work does not remain strictly linear at the latent level, but it uses LAM as the reference architecture from which attractor-based generalizations are constructed (Qin et al., 30 Mar 2026).

In that sense, LAM functions both as a concrete memory mechanism and as a reference theory. It provides explicit storage rules, exact retrieval equations, tractable asymptotics, and interpretable spectra; at the same time, its limitations under interference, extreme-value competition, and lack of autonomous correction clarify why modern architectures add kernelization, gating, rank structure, or attractor dynamics rather than abandoning the linear memory perspective altogether.