Dense Associative Memory with biased patterns: a Replica Symmetric analysis

Abstract: We investigate dense higher-order associative memories in the high storage regime when the stored patterns are biased, namely when the entries of the patterns are not symmetrically distributed around zero. In this setting, the standard Hebbian prescription must be modified by recentering and rescaling the pattern entries, and an additional term must be introduced in the Hamiltonian to enforce consistency between the average activity of the network and that of the stored patterns. As a first step, we perform a signal-to-noise analysis in the zero-temperature limit and show that the bias reduces the effective storage capacity through a multiplicative correction factor (1-b^2)^P, while preserving the superlinear scaling with the system size. We then derive the quenched statistical pressure within the Replica Symmetric framework by means of Guerra's interpolation method and obtain the corresponding self consistency equations for the relevant order parameters. The analytical treatment confirms the heuristic prediction of the signal-to-noise argument, showing that the same bias dependent renormalization naturally emerges in the variance of the cross-talk noise. Finally, we discuss the resulting phase behavior of the model and its implications for retrieval performance in the model.

• The paper demonstrates that nonzero pattern bias reduces storage capacity by a factor of (1-b²)^P while preserving the superlinear scaling of O(N^(P-1)).
• It employs zero-temperature signal-to-noise analysis alongside Replica Symmetry techniques using Guerra's interpolation to derive precise self-consistency equations.
• Numerical simulations confirm that dense higher-order networks effectively retrieve biased patterns, ensuring robust attractor dynamics even near saturation.

Dense Associative Memory with Biased Patterns: Replica Symmetry, Storage Capacity, and Retrieval

Introduction and Theoretical Framework

This work provides an in-depth analysis of dense higher-order associative memory neural networks in the high-storage regime when the entries of stored patterns are subject to nonzero bias. The study addresses an essential gap in associative memory theory, extending beyond the conventional symmetric (unbiased) pattern assumption by introducing a formalism for biased memories. The model generalizes the $P$-spin Hopfield network, using recentered and rescaled Hebbian interaction tensors and imposing an additional quadratic penalty to constrain the average network activity to match that of the patterns.

The authors employ a signal-to-noise analysis in the zero-temperature limit and a full Replica Symmetric (RS) statistical mechanics approach, leveraging Guerra's interpolation technique to derive explicit self-consistency equations. The central question is quantifying the impact of pattern bias on retrieval threshold, network capacity, and phase diagram structure.

Model Specification and Modifications for Pattern Bias

The model is comprised of $N$ binary neurons storing $K$ patterns $\{\boldsymbol{\xi}^\mu\}$, each sampled with a tunable mean $b$, such that $\mathbb{P}(\xi=+1) = (1+b)/2$ and $\mathbb{P}(\xi=-1) = (1-b)/2$. The standard Hebbian learning rule is recentered and rescaled via the transformation $\eta_i^\mu = (\xi_i^\mu - b)/\sqrt{1-b^2}$ ensuring zero mean and unit variance, as required for unbiased fluctuation analysis.

However, this is insufficient to guarantee retrieval of patterns with nonzero mean activity. Thus, the Hamiltonian is augmented by a quadratic penalty term proportional to $(M(\boldsymbol{\sigma})-b)^2$, where $M(\boldsymbol{\sigma})$ denotes the average spin activity. The constraint strength $N$0 is tuned explicitly, controlling network compliance with the imposed activity.

Zero-Temperature Signal-to-Noise Analysis

At $N$1, initializing the neuron state to a stored pattern, the local field is decomposed into a deterministic signal from the retrieved pattern and a crosstalk noise from the remaining $N$2 patterns. Estimating the SNR shows pattern bias reduces network capacity by a multiplicative correction $N$3—yet does not alter the underlying $N$4 superlinear scaling:

$N$5

This deduction is robust, grounded in central limit arguments and large deviation estimates, indicating that superlinear scaling persists but is suppressed for increasing bias.

Replica Symmetric Statistical Mechanics and Guerra Interpolation

The analytical backbone uses a Replica Symmetric computation of the quenched free energy (statistical pressure) via Guerra's interpolation. The partition function is recast, segregating the retrieval signal and representing the crosstalk from nonretrieved patterns by a Gaussian noise whose variance encodes the core combinatorial effects of bias and network order. The full RS free energy is derived as a function of the order parameters: Mattis magnetization, mean activity, and two-replica overlap $N$6.

Self-consistency equations, central to RS analysis, are computed. They explicitly exhibit the bias-induced renormalization of the variance in the crosstalk noise, precisely as predicted by the signal-to-noise analysis:

$N$7

This formal correspondence across two methodological lines demonstrates the universality of the effect of bias on the phase diagram structure and retrieval load.

Figure 1: Solution surfaces of the self-consistency equations at zero temperature for varying $N$8 and $N$9 (upper row: $K$0; lower row: $K$1); substantial shifts in retrieval regions and transitions are evident under pattern bias.

Phase Diagram and Storage Capacity

Numerical resolution of the zero-temperature self-consistency equations reveals precise modulation of retrieval capacity by bias and activity constraint $K$2. Critical load curves in the $K$3 plane showcase:

• As $K$4 increases, retrieval capacity is moderately enhanced ("booster effect"), but the effect diminishes for higher $K$5.
• Increasing the pattern bias $K$6 leads to systematically reduced critical load and, consequently, storage capacity.

Figure 2: Critical load $K$7 as a function of $K$8 and $K$9 for different $\{\boldsymbol{\xi}^\mu\}$0 values; the capacity loss with increasing $\{\boldsymbol{\xi}^\mu\}$1 is more pronounced at higher interaction order.

Numerical Simulations: Retrieval and Stability

Extensive MCMC simulations were conducted to validate theoretical predictions in both the dense ($\{\boldsymbol{\xi}^\mu\}$2) and pairwise ($\{\boldsymbol{\xi}^\mu\}$3) regimes. In the dense regime ($\{\boldsymbol{\xi}^\mu\}$4, $\{\boldsymbol{\xi}^\mu\}$5, $\{\boldsymbol{\xi}^\mu\}$6, $\{\boldsymbol{\xi}^\mu\}$7), the network consistently retrieves the pattern and stabilizes mean activity at the target bias, illustrating the efficiency of dense associative memories in overcoming the limitations of standard Hopfield nets under bias conditions.

Figure 3: Time evolution of magnetization and mean activity for different $\{\boldsymbol{\xi}^\mu\}$8; the dense network ($\{\boldsymbol{\xi}^\mu\}$9) successfully retrieves and maintains the biased pattern, unlike the pairwise network.

When approaching the saturation limit ($b$0), dense networks with $b$1 continue to demonstrate robust attractor behavior, while pairwise Hopfield networks rapidly lose both the overlap with the stored pattern and appropriate mean activity.

Figure 4: Retrieval stability near saturation load; dense ($b$2) architecture maintains pattern recovery and correct mean activity, pairwise network fails.

Theoretical and Practical Implications

The findings advance the understanding of associative memory storage in non-idealized scenarios, where stored patterns may be statistically imbalanced—a feature commonly observed in biological and engineered systems. The persistence of superlinear storage scaling, given proper Hebbian recentering/scaling and activity constraint, substantially broadens the range of applicability of higher-order associative memories.

On the theoretical front, the confirmation of bias-induced variance renormalization via both heuristic (signal-to-noise) and rigorous (replica) analyses provides a precise mapping between bias and critical network properties. This lays groundwork for future extensions into Replica Symmetry Breaking (RSB), dynamical basin size characterization, and the behavior of modern exponential or diluted Hopfield networks in machine learning environments.

Speculation on Future Directions

Potential extensions include:

• Analytical and numerical exploration of RSB regimes, necessary for describing glassy or nonretrieval phases.
• Investigation of dynamic properties such as basin size and convergence rates, especially relevant for applications in rapid inference.
• Inclusion of hetero-associative (modular) and diluted architectures, allowing for richer storage and interaction structures.
• Adaptation to modern Hopfield variants and their integration as modules within deep or recurrent learning systems, where pattern bias and structured activity constraints are prevalent.

Conclusion

This study provides a thorough Replica Symmetric analysis of dense associative memory in the presence of biased patterns. The core result—capacity reduction governed by $b$3 without altering superlinear scaling—highlights a universal effect of bias that is both analytically tractable and numerically robust. Dense high-order associative memories, with proper activity constraints and rule modification, support efficient retrieval even under strong pattern bias; these findings are directly relevant for both theoretical neuroscience and modern AI architectures.

(2604.02789)