Hopfield–Fenchel–Young Networks

• Hopfield–Fenchel–Young Networks are a unified framework that generalizes classical Hopfield networks and modern attention mechanisms.
• They use energy-based models with Fenchel–Young losses to enable differentiable, margin-controlled inference and structured memory retrieval.
• Practical applications include enhanced recall accuracy in image recognition, text rationalization, and multiple instance learning tasks.

Hopfield–Fenchel–Young (HFY) networks are a unified theoretical and algorithmic framework for associative memory retrieval and structured attention, encompassing classical Hopfield networks, their modern continuous and sparse variants, and generalizations to structured memory operations. The core of the HFY formulation is an energy-based model in which inference and learning are governed by Fenchel–Young losses and their associated convex-analytic properties. This construction enables differentiable, margin-controlled, and structured inference and learning in memory-augmented neural and statistical models (Blondel et al., 2022, Santos et al., 2024, Santos et al., 2024).

1. Formal Energy-Based Construction

An HFY network operates over a memory matrix $X \in \mathbb{R}^{N \times D}$ containing $N$ patterns (rows $x_1,\ldots,x_N$) and a query (state) $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$. Two closed convex functions, $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$ and $\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$, along with their Fenchel conjugates $\Omega^*$ and $\Psi^*$, define the HFY energy: $E(q) = -\Omega^*(Xq) + \Psi(q)$ Alternatively, the energy can be written as a difference of two Fenchel–Young losses,

$E(q) = -L_\Omega(Xq; u) + L_\Psi(X^Tu; q)$

where $N$0 is a reference (e.g., $N$1 for probability-simplex domains), and

$N$2

is the Fenchel–Young loss induced by a convex function $N$3. The first term, $N$4, provides a concave Hopfield scoring mechanism; the second, $N$5, imposes a convex-analytic post-transformation, such as $N$6-normalization or more general constraints (Santos et al., 2024).

2. Fenchel–Young Losses, Generalized Entropies, and Sparsity

The Fenchel–Young loss framework admits rich choices for $N$7 and $N$8, controlling the network's dynamics and sparsity. Popular selections include:

• Tsallis $N$9-negentropy ($x_1,\ldots,x_N$0): for $x_1,\ldots,x_N$1,

$x_1,\ldots,x_N$2

with the dual inducing $x_1,\ldots,x_N$3. For $x_1,\ldots,x_N$4, this recovers softmax; for $x_1,\ldots,x_N$5, sparsemax is obtained.

• Norm negentropy ($x_1,\ldots,x_N$6):

$x_1,\ldots,x_N$7

inducing the $x_1,\ldots,x_N$8-normmax transformation.

For continuous-valued or constrained $x_1,\ldots,x_N$9, $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$0 can be, for example, quadratic ($q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$1) or an indicator over a norm-ball, leading to explicit post-transformations like $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$2- or layer-normalization.

The regularizer $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$3 determines both the sparsity and the existence of a loss margin (i.e., finite interval of exact retrieval), with larger $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$4 or $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$5 yielding sparser updates and explicit margin control (Blondel et al., 2022, Santos et al., 2024, Santos et al., 2024).

3. Inference, Update Rules, and Differentiability

HFY networks implement memory retrieval by minimizing the energy $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$6 over $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$7, which decomposes into concave and convex components. In practice, the concave–convex procedure (CCCP) yields the canonical update: $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$8 where $q \in \text{dom}\,\Psi \subseteq \mathbb{R}^D$9 is the transformation induced by the conjugate of $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$0 (e.g., softmax, entmax, normmax, SparseMAP). For classical softmax-based Hopfield/attention (with entropy regularizer), this reduces to standard attention pooling. For Tsallis or norm-based entropies, updates are end-to-end differentiable and sparse.

No unrolled backpropagation through the inner optimization loop is necessary. Gradients follow by invoking the envelope theorem: the chain rule composes $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$1 directly with differentiable argmaxes or proximal operators, enabling scalable and efficient learning (Blondel et al., 2022, Santos et al., 2024).

4. Exact Retrieval, Margins, and Storage Capacity

A central property of HFY networks is the precise relationship between the choice of regularizer, margin, and memory capacity. For Fenchel–Young losses induced by Tsallis or norm entropies with $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$2 or $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$3, there exists a finite margin $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$4 such that exact retrieval of a stored pattern $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$5 from an initial query $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$6 is guaranteed if the coherence gap

$$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$7

and $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$8 is sufficiently close. For the structured case (e.g., k-subset or sequence retrievals), analogous structured margins and retrieval results hold using combinatorial polytopes (Santos et al., 2024, Santos et al., 2024).

Under these conditions, exponential storage capacity is retained: for example, $$\Omega : \mathbb{R}^N \to \mathbb{R}\cup\{+\infty\}$$9 patterns can be stored with provable one-step exact retrieval (worst-case packing); for random patterns, exponential scaling in $\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$0 is observed with high probability.

5. Extensions: Structured Memory and Post-Transformations

HFY networks generalize beyond single-pattern attractors to structured associations via the SparseMAP transformation, applicable when the memory domain is the convex hull of combinatorial structures $\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$1 (e.g., k-subsets, sequences, matchings). The SparseMAP update

$\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$2

produces a sparse convex combination of structures, enabling retrieval of pattern groups with structured margins and empirical performance gains on tasks requiring set or sequential pattern recall (Santos et al., 2024, Santos et al., 2024).

Post-transformations such as $\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$3-normalization and generalized layer normalization are implemented by appropriate choices of $\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$4, e.g., as an indicator function over norm-balls or subspaces, yielding differentiable and explicit updates in network state space (Santos et al., 2024).

6. Special Cases and Unification

The HFY formalism subsumes:

• Classical Hopfield networks: quadratic energy with quadratic regularizer; binary or continuous attractors.
• Modern Hopfield/attention mechanisms: softmax-induced energy with quadratic proximity—recovering transformer-style attention via energy minimization.
• Sparse Hopfield and attention: entmax or normmax in place of softmax, resulting in sparse state updates and improved margin properties.
• Structured Hopfield networks: via convex-hull regularizers (SparseMAP), enabling the retrieval of structured pattern sets, sequences, or associations.

The following table summarizes special cases:

Scenario	$\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$5	$\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$6	$\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$7	$\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$8
Binary Hopfield	$\Psi : \mathbb{R}^D \to \mathbb{R}\cup\{+\infty\}$9	Identity	$\Omega^*$0 indicator	$\Omega^*$1
Softmax attention	entropy	softmax	$\Omega^*$2	Identity
Sparse attention	Tsallis/entmax	entmax$\Omega^*$3	$\Omega^*$4	Identity
$\Omega^*$5-normalized state	Entropy, quadratic, etc.	softmax/entmax/sparse	$\Omega^*$6	$\Omega^*$7
Structured (SparseMAP)	SparseMAP regularizer	SparseMAP	Quadratic/other	Identity or normalized

7. Empirical Results and Applications

HFY networks have demonstrated robust empirical performance across various tasks, including synthetic and real data:

• Recall and capacity experiments: On MNIST, CIFAR-10, and Tiny ImageNet, networks using sparse $\Omega^*$8-entmax and normmax outperform softmax and classical Hopfield on recall accuracy under noise or partial masking. Layer- and $\Omega^*$9-norm post-transformations further improve robustness (Santos et al., 2024).
• Multiple Instance Learning (MIL): Replacing attention pooling with HFY pooling (entmax, normmax, SparseMAP) yields state-of-the-art performance on MNIST and image-bag MIL benchmarks, especially when the structure of pooling matches the MIL threshold (Santos et al., 2024).
• Text rationalization: Using sequential SparseMAP for rationale selection in text classification (SST, IMDB, AgNews, BeerAdvocate) matches or exceeds baselines, producing rationales with higher F1 overlap with human annotations and better contiguity (Santos et al., 2024).
• Free and sequential recall: With suitable transformations (constrained sparsemax, penalized entmax), HFY networks achieve near-perfect unique-recall and model replay-like dynamics (Santos et al., 2024).

These findings indicate that the HFY framework not only unifies and generalizes associative memory and attention models but also provides practical advantages—differentiable, sparse, and structured updates; explicit memory margins; and broad applicability. The paradigm enables both theoretical analysis and scalable deployment in learning systems (Blondel et al., 2022, Santos et al., 2024, Santos et al., 2024).