---
title: Self-Attention Fermi Neural Network
url: https://www.emergentmind.com/topics/self-attention-fermi-neural-network
type: topic
---

# Self-Attention Fermi Neural Network

“Self-Attention Fermi Neural Network” is a broad category label for determinant-based fermionic neural-network wavefunctions in which electron or particle correlations are generated primarily by self-attention over particle embeddings, while fermionic antisymmetry is enforced by Slater determinants rather than by making the entire network antisymmetric directly. In the cited literature, the label is informal rather than universal: the molecular architecture is named **Psiformer** [2211.13672], the periodic-solid adaptation is **Psi-Solid** [2502.05383], one study explicitly uses the phrase **self-attention Fermi neural network** for chiral superconductivity [2509.03683], and a later large-scale 2D electron-gas study presents an attention-based determinant ansatz of the same general type [2607.08616]. Across these works, the recurring theme is that self-attention serves as the correlation engine inside a variational Monte Carlo wavefunction for continuum fermions.

## 1. Definition and formal structure

In this research line, the many-body wavefunction is written as a sum of determinants of configuration-dependent orbitals. The common structural idea is that the orbital assigned to particle \(j\) is not a fixed one-body orbital \(\phi_\mu(\mathbf x_j)\), but a generalized orbital depending on the full configuration, such as \(\Phi^k_\mu(\mathbf x_j;\{\mathbf x_{/j}\})\) or \(\phi_i^{(k)}(\mathbf r_j,\{\mathbf r_{\neq j}\})\) [2509.03683, 2607.08616]. This places the ansatz in the same broad determinant-based family as FermiNet and related neural quantum states, but with self-attention replacing the more hand-structured interaction backbone [2211.13672, 2502.05383].

A representative form appears in several papers:
\[
\Psi(\mathbf X)=\sum_{k=1}^{N_{\rm det}} \det\!\bigl[\Phi^k_\mu(\mathbf x_j;\{\mathbf x_{/j}\})\bigr],
\]
with variants that optionally include a Jastrow factor,
\[
\Psi_\theta(x)=\exp\!\big(\mathcal J_\theta(x)\big)\sum_{k=1}^{N_{\mathrm{det}}}\det\!\big[\mathbf \Phi_\theta^k(x)\big]
\]
[2211.13672, 2509.03683, 2412.00618]. Psi-Solid uses a determinant sum without a Jastrow factor in its final reported architecture, after the authors found little gain for their moiré systems [2502.05383]. The large-scale 2D homogeneous electron gas study likewise writes the wavefunction directly as a determinant sum and explicitly notes that it is not built from a prescribed orbital basis [2607.08616].

The symmetry logic is consistent across the fermionic papers. The neural network mapping from the set of particle coordinates to per-particle latent states is permutation equivariant, because identical transformations are applied to every particle and attention acts over the set of particles [2502.05383, 2509.03683]. Antisymmetry is then imposed by the determinant. This is the same conceptual strategy emphasized in comparisons with FermiNet and PsiFormer: equivariant orbitals first, determinant antisymmetrization second [2502.05383].

The phrase itself should be used carefully. Psi-Solid is described as “very close in spirit to what one might informally call a ‘self-attention Fermi neural network,’” but that paper does not introduce the phrase as the formal model name [2502.05383]. By contrast, the chiral-superconductivity paper explicitly frames its architecture as “a general-purpose self-attention Fermi neural network” [2509.03683]. A plausible editorial interpretation is that the term denotes a family of transformer-like fermionic neural wavefunctions rather than a single canonical architecture.

## 2. Architecture and self-attention as a correlation mechanism

The architectural core is multi-head self-attention over particle tokens. In Psiformer, each electron is represented by a hidden state \(\mathbf h_i^\ell\), and each layer applies multi-head self-attention followed by a residual \(\tanh\)-MLP update:
\[
\mathbf f_i^{\ell+1}=\mathbf h_i^\ell+\mathbf W_o^\ell \,\mathrm{concat}_h\!\Big[\mathrm{SelfAttn}_i(\mathbf h_1^\ell,\dots,\mathbf h_N^\ell)\Big],
\]
\[
\mathbf h_i^{\ell+1}=\mathbf f_i^{\ell+1}+\tanh\!\left(\mathbf W^{\ell+1}\mathbf f_i^{\ell+1}+\mathbf b^{\ell+1}\right)
\]
[2211.13672]. The attention is standard scaled dot-product attention over electrons, with learned query, key, and value projections [2211.13672]. The chiral-superconductivity architecture uses the same pattern on periodic coordinate embeddings of particles in continuous space [2509.03683].

Psi-Solid adapts this design to periodic solids. Raw coordinates are first mapped to periodic features,
\[
{\rm feature}(\mathbf r)=
\begin{pmatrix}
\sin(\mathbf G_1^T\mathbf r)\\
\sin(\mathbf G_2^T\mathbf r)\\
\cos(\mathbf G_1^T\mathbf r)\\
\cos(\mathbf G_2^T\mathbf r)
\end{pmatrix},
\]
with \(\mathbf G_i^T\mathbf L_j=2\pi\delta_{ij}\), and self-attention is then inserted between residual per-electron blocks [2502.05383]. Its particlewise attention rule is written directly as
\[
{\rm SelfAttn}_i=
\frac{1}{\mathcal N}\sum_{j=1}^N \exp\!\left(\mathbf q_j^{lh}\!\cdot\! \mathbf k_i^{lh}\right)\mathbf v_j^{lh},
\]
with a nonstandard normalization \(\mathcal N\) [2502.05383]. The authors explicitly interpret the resulting correlated orbitals \(\phi_j(\mathbf r_i;\{\mathbf r_{/i}\})\) as backflow-like, because the orbital assigned to one electron depends on the coordinates of all the others [2502.05383].

The FQH study also adapts Psiformer into a complex-valued fermionic ansatz. Each spin-polarized electron is treated as a token with coordinate-derived features, attention generates many-body orbitals, and real and imaginary parts are created in separate streams before determinant evaluation [2412.00618]. The chiral-superconductivity paper follows the same basic template: periodic coordinate embedding, \(L\) layers of multi-head self-attention and MLP, projection to complex \(N\times N\) orbital matrices, and determinant summation [2509.03683]. In that work, the hyperparameter table lists \(6\) network layers, \(3\) attention heads per layer, attention dimensions \(16\) for queries/keys and values, perceptron dimension \(128\), one perceptron per layer, and \(4\) determinants [2509.03683].

The 2D homogeneous electron-gas paper introduces an additional physically motivated modification: **spatially weighted particle attention**,
\[
\alpha_{ij}=(K_i^TQ_j)e^{-r_{ij}/\lambda},
\]
where \(r_{ij}\) is the interparticle distance and \(\lambda\) is a trainable length scale initialized from the average interparticle distance [2607.08616]. This preserves dense all-to-all attention while biasing attention toward locality. The paper presents the model as a stack of self-attention and perceptron layers, repeated \(L\) times, followed by projection into generalized orbitals and determinant evaluation [2607.08616].

A recurrent architectural consequence is that self-attention functions as a learned many-body backflow or interaction-gating mechanism. Psiformer motivates attention by arguing that prior FermiNet- and PauliNet-style models “did not include an attention-like mechanism for gating interactions between electrons” [2211.13672]. Psi-Solid makes the stronger claim that electron correlation is produced entirely by self-attention in the core model [2502.05383]. This suggests that, within this family, self-attention is treated not as an auxiliary feature mixer but as the main representation of fermionic correlation.

## 3. Variational Monte Carlo formulation and optimization

All of the fermionic self-attention wavefunctions in the cited literature are trained within variational Monte Carlo. The objective is the variational energy
\[
E_\theta=\frac{\langle \Psi_\theta|\hat H|\Psi_\theta\rangle}{\langle \Psi_\theta|\Psi_\theta\rangle},
\]
estimated by Monte Carlo sampling from \(|\Psi_\theta|^2\) using the local energy
\[
E_{\rm loc}(\mathbf R)=\Psi_\theta^{-1}(\mathbf R)\hat H\Psi_\theta(\mathbf R)
\]
[2502.05383, 2211.13672, 2509.03683, 2412.00618]. Sampling is performed with Metropolis–Hastings in the cited studies [2502.05383, 2211.13672, 2509.03683].

A common optimization strategy is natural-gradient descent or stochastic reconfiguration approximated by KFAC. Psi-Solid writes the wavefunction-space metric as the quantum geometric tensor
\[
g_{nm}(\boldsymbol\theta)=
\langle \partial_{\theta_n}\Psi_{\boldsymbol\theta}|
(1-|\Psi_{\boldsymbol\theta}\rangle\langle\Psi_{\boldsymbol\theta}|)
|\partial_{\theta_m}\Psi_{\boldsymbol\theta}\rangle,
\]
and uses KFAC because exact inversion is too expensive [2502.05383]. It notes an important caveat: the current KFAC implementation acts only on the wavefunction magnitude and neglects the phase, effectively reducing the metric to the Fisher information matrix [2502.05383]. The chiral-superconductivity paper gives the analogous stochastic-reconfiguration matrix
\[
S_{ab}=
\mathbb E[O_a^*O_b]-\mathbb E[O_a^*]\mathbb E[O_b],
\qquad
\Delta\boldsymbol\theta=-\alpha S^{-1}\mathbf g,
\]
and then uses KFAC as an efficient approximation [2509.03683].

The training setups are concrete and differ by application. Psiformer uses KFAC for main training, LAMB for pretraining to Hartree–Fock orbitals computed by PySCF, \(2\times 10^5\) optimization iterations, batch size \(4096\), \(30\) MCMC decorrelation steps, KFAC norm constraint \(10^{-3}\), damping \(10^{-3}\), local-energy clipping parameter \(\rho=5.0\), and learning rate
\[
lr(t)=lr_0\left(1+\frac{t}{t_0}\right)^{-1},
\quad
lr_0=0.05,\ t_0=10^5
\]
[2211.13672]. Psi-Solid uses \(1.5\times 10^5\) training iterations, \(3\) network layers, \(6\) attention heads per layer, attention dimension \(16\), perceptron dimension \(64\), \(N_{\rm det}=4\), batch size \(4096\), KFAC norm constraint \(10^{-3}\), damping \(10^{-3}\), local-energy clipping \(\rho=5.0\), and
\[
\eta(t)=\eta_0\left(1+\frac{t}{t_0}\right)^{-1},
\quad
\eta_0=10,\ t_0=10^5
\]
[2502.05383]. The chiral-superconductivity study uses random initialization with no pretraining, batch size \(1024\), burn-in steps \(200\), sample move width \(0.2\), \(40000\) training iterations, initial learning rate \(\eta_0=1\), delay \(t_0=10^5\), local-energy clipping \(\rho=5.0\), KFAC norm constraint \(10^{-3}\), and KFAC damping \(10^{-3}\) [2509.03683].

The role of auxiliary physical factors differs across works. Psiformer includes an explicit electron-electron Jastrow because its attention network does not receive electron-electron distances directly, and ablations show that removing the Jastrow makes energies very noisy [2211.13672]. The FQH paper also includes a Jastrow factor to reproduce the Coulomb cusp and short-distance nonanalyticity beyond lowest-Landau-level holomorphic forms [2412.00618]. By contrast, Psi-Solid reports that a simple cusp-enforcing Jastrow gave no significant improvement in the spin-polarized moiré systems while increasing GPU time per step by about \(10\)–\(20\%\), and therefore omits it in the final architecture [2502.05383].

## 4. Major realizations and physical applications

The earliest direct realization in the supplied corpus is Psiformer, introduced for ab initio quantum chemistry as a self-attention-based drop-in replacement for the neural-network component of FermiNet- and PauliNet-style wavefunctions [2211.13672]. It retains determinant antisymmetrization, envelope functions, and a Jastrow factor, but replaces the FermiNet interaction backbone with a self-attention network over electrons [2211.13672]. The paper emphasizes that the gains become larger on larger molecules, with improvements over FermiNet by dozens of kcal/mol and up to about \(120\) mHa \(\approx 75\) kcal/mol on CCl\(_4\) [2211.13672].

A second realization appears in fractional quantum Hall physics, where an attention-based fermionic neural network adapts Psiformer to \(N\) spin-polarized electrons in 2D under magnetic field, working directly with the full Hilbert space in disk geometry [2412.00618]. There the architecture is a complex, determinant-based real-space ansatz with self-attention-generated many-body orbitals and a Jastrow factor for cusp physics [2412.00618]. The paper reports energies consistently lower than lowest-Landau-level projected exact diagonalization and uses the learned wavefunction to visualize microscopic zero structures, Landau-level mixing effects, and a transition from FQH liquid to crystal state at strong LL mixing [2412.00618].

Psi-Solid extends the approach to periodic continuum solids, specifically a continuum model of electrons in the WSe\(_2\)/WS\(_2\) moiré semiconductor heterobilayer [2502.05383]. The Hamiltonian is
\[
H=\sum_i\left(-\frac12\nabla_i^2+V(\mathbf r_i)\right)
+\frac12\sum_i\sum_{i\neq j}\frac{1}{|\mathbf r_i-\mathbf r_j|},
\]
with periodic boundary conditions and Ewald-summed Coulomb interactions [2502.05383]. The benchmark systems are \(9\)-, \(12\)-, and \(27\)-site supercells at filling \(\nu=2/3\), corresponding to \(6\), \(8\), and \(18\) spin-polarized electrons [2502.05383]. The work frames Psi-Solid as a self-attention NN wavefunction inspired by PsiFormer rather than as a formal FermiNet variant [2502.05383].

The chiral-superconductivity study applies a general-purpose self-attention Fermi neural network to a spin-polarized two-dimensional attractive Fermi gas with periodic boundary conditions and Gaussian attraction,
\[
H=-\sum_{i=1}^N\frac{\nabla_i^2}{2m_e}+\sum_{i>j}V(\mathbf r_i-\mathbf r_j),
\qquad
V(\mathbf r)=\frac{U}{2\pi\sigma_U^2}\exp\!\left(-\frac{|\mathbf r|^2}{2\sigma_U^2}\right),
\quad U<0
\]
[2509.03683]. The paper’s central methodological point is that the architecture has no explicit pairing structure, unlike Pfaffian, geminal, or BCS-like forms, and therefore asks whether a general-purpose antisymmetric network can discover chiral superconductivity by energy minimization alone [2509.03683].

The most recent large-scale realization in the supplied material studies the fully spin-polarized 2D homogeneous electron gas up to \(N=169\) particles in a periodic triangular cell with Ewald interaction [2607.08616]. The paper presents the model as an alternative to canonical FermiNet-style architectures, still determinant-based but driven by self-attention and not by a prescribed orbital basis or backflow on fixed plane-wave orbitals [2607.08616].

## 5. Empirical behavior, observables, and scaling claims

The empirical record across these papers is centered on lower variational energies and on direct access to wavefunction-dependent observables.

In quantum chemistry, Psiformer consistently outperforms FermiNet on the standard small-molecule benchmarks from the original FermiNet paper [2211.13672]. Representative energies from the appendix include ozone,
\[
\text{FermiNet large}: -225.4268(1)\ {\rm Ha},\qquad
\text{Psiformer large}: -225.43231(8)\ {\rm Ha},
\]
and bicyclobutane,
\[
\text{FermiNet large}: -155.9432(1)\ {\rm Ha},\qquad
\text{Psiformer large}: -155.94836(7)\ {\rm Ha}
\]
[2211.13672]. On larger molecules the reported differences are larger: for benzene, Psiformer gives \(-232.2400(1)\) Ha versus \(-232.2205(2)\) Ha for FermiNet; for naphthalene, \(-385.8685(2)\) Ha versus \(-385.8147(4)\) Ha; and for CCl\(_4\), \(-1878.804(1)\) Ha versus \(-1878.684(1)\) Ha, a gain of about \(120\) mHa [2211.13672]. The same paper notes that absolute-energy improvements do not automatically imply uniformly improved relative quantities such as binding energies or ionization potentials [2211.13672].

In periodic moiré solids, Psi-Solid outperforms both self-consistent Hartree–Fock and band-projected exact diagonalization in the tested systems [2502.05383]. For the \(27\)-site system at \(\nu=2/3\) and \(\epsilon=10\), the reported total ground-state energies are
\[
E_{\rm Psi\text{-}Solid}=-32.070(7),\quad
E_{\rm SlaterNet}=-31.35(2),\quad
E_{\rm BP\text{-}ED}=-31.32443.
\]
For \(\epsilon=5\),
\[
E_{\rm Psi\text{-}Solid}=-59.127(9),\quad
E_{\rm SlaterNet}=-58.01(3),\quad
E_{\rm BP\text{-}ED}=-57.80848.
\]
For this larger system, BP-ED is restricted to a single band, and the neural ansatz is about \(2.5\%\) lower in energy than BP-ED [2502.05383]. The same optimized wavefunctions yield charge density and pair correlation functions showing a Fermi-liquid-like state at weaker interactions and a generalized Wigner crystal at stronger interactions [2502.05383].

In chiral superconductivity, the evidence is not only variational. The paper defines the pair-binding energy
\[
E_B(N)=E(N)+E(N+2)-2E(N+1)
\]
and reports the reversed odd-even pattern expected for a topological spinless \(p_x\pm i p_y\) superconductor for \(N=29,\dots,37\): \(E_B<0\) for odd \(N\) and \(E_B>0\) for even \(N\) [2509.03683]. It further performs \(C_4\) projection,
\[
\tilde\Psi_m(\mathbf X)=\frac14\sum_{k=0}^3e^{-ikm\pi/2}\Psi(\mathbf R_{\pi/2}^k\mathbf X),
\]
finding the lowest-energy sector consistent with
\[
m\equiv \pm (N-1)/2\pmod 4
\]
[2509.03683]. The strongest diagnostic is the two-body reduced density matrix, whose leading detached eigenvalue indicates ODLRO and whose leading eigenvector has a \(2\pi\) phase winding around the origin, directly identifying chiral \(p_x+i p_y\) symmetry [2509.03683].

In the 2D homogeneous electron gas, the large-scale self-attention wavefunction reaches energies below state-of-the-art fixed-node DMC based on optimized Slater–Jastrow–backflow trial states [2607.08616]. For \(N_e=91,\ r_s=30\), the reported energies are
\[
E_{\rm SJB\text{-}VMC}=-0.0318792(2)\ {\rm Ha/el},\quad
E_{\rm SJB\text{-}DMC}=-0.0319370(1)\ {\rm Ha/el},\quad
E_{\rm NN\text{-}VMC}=-0.03193763(3)\ {\rm Ha/el}.
\]
For \(N_e=91,\ r_s=40\),
\[
E_{\rm NN\text{-}VMC}=-0.02443814(2)\ {\rm Ha/el},
\]
and for \(N_e=169,\ r_s=30\),
\[
E_{\rm NN\text{-}VMC}=-0.03192077(3)\ {\rm Ha/el}
\]
[2607.08616]. The paper quantifies the gain over SJB-VMC in recovered correlation energy as \(1.263\%\) for \(N_e=91,\ r_s=30\), \(1.466\%\) for \(N_e=91,\ r_s=40\), and \(1.401\%\) for \(N_e=169,\ r_s=30\), emphasizing that the gain does not deteriorate with increasing system size [2607.08616]. It also computes
\[
S(\mathbf q)=\frac{\langle \rho_{\mathbf q}\rho_{-\mathbf q}\rangle}{N_e}
\]
and reconstructs a Feynman upper bound
\[
\varepsilon(\mathbf q)\lesssim \frac{\hbar^2 q^2}{2m\,S(\mathbf q)},
\]
recovering the small-\(q\) plasmon branch and a roton-like minimum near \(q\approx 2k_F\) [2607.08616].

The most explicit scaling claim in this corpus comes from Psi-Solid. It does **not** claim that training cost scales as \(N^2\); rather, it defines a saturation threshold \(N_{\rm par}^*\) for the smallest parameter count beyond which converged energies remain within one standard deviation of the best energy found, and fits
\[
N_{\rm par}^*\approx 320\times N^{2.1}
\]
across \(9\)-, \(12\)-, and \(27\)-site systems at \(\nu=2/3\) [2502.05383]. The same study cautions that parameter count alone is not enough, since roughly \(2\)–\(3\) attention layers and a sufficient number of heads were necessary to reach the best energies [2502.05383]. A plausible implication is that, within the reported architecture family, depth and head count are part of the effective scaling law, not merely width.

## 6. Relation to FermiNet, terminology, and limitations

Self-attention Fermi neural networks are best understood as transformer-like alternatives to FermiNet-family architectures rather than as a complete break with determinant-based neural quantum states. The commonality with FermiNet is determinant antisymmetrization, many-body-dependent orbitals, and VMC optimization [2211.13672, 2502.05383, 2607.08616]. The architectural difference is that FermiNet traditionally uses specially designed one-electron and two-electron streams with permutation-equivariant mixing, whereas Psiformer and Psi-Solid use self-attention as the central correlation mechanism [2211.13672, 2502.05383]. The 2020 FermiNet engineering paper is therefore an important baseline for accuracy, cost, determinant structure, and envelope design, but it does not introduce self-attention [2011.07125].

The limitations are also consistent across the literature. Attention is dense and remains \(O(N^2)\) in particle number at the attention-map level in naive implementations, while determinant evaluation remains a major cost [2502.05383, 2607.08616]. The large-scale HEG paper explicitly frames scalability as a central question because self-attention wavefunctions have increased computational complexity relative to traditional variational ansätze [2607.08616]. Psi-Solid does not present a formal operation-count analysis and emphasizes that its favorable \(N^{2.1}\) law concerns required parameter count for energy saturation, not runtime or memory [2502.05383]. Psiformer also notes that attention has the usual quadratic dependence on number of tokens within a layer, though overall VMC cost involves more than just attention [2211.13672].

The physical scope of published demonstrations remains narrower than the phrase might suggest. Psi-Solid treats a specific continuum moiré model with spin-polarized electrons and relatively simple band structure rather than generic ab initio solids [2502.05383]. The chiral-superconductivity work studies a spinless attractive Fermi gas with periodic boundary conditions [2509.03683]. The large-scale HEG study is fully spin-polarized [2607.08616]. Psiformer is molecular and not periodic [2211.13672]. This suggests that the phrase denotes a methodological class with successful demonstrations across multiple fermionic domains, but not yet a single universal architecture validated on all realistic multiorbital, spinful, material-specific Hamiltonians.

A related terminological confusion arises with non-fermionic attention models inspired by physics. “Fractional neural attention” replaces dot-product self-attention by a fractional-diffusion kernel and is explicitly not a fermionic or Fermi–Dirac attention model [2511.10208]. Its relevance to self-attention Fermi neural networks is only conceptual, in the sense that both are physics-inspired modifications of attention [2511.10208].

Taken together, the literature supports a precise characterization. A self-attention Fermi neural network is a determinant-based neural quantum state for fermions in which particles are treated as tokens, correlations are learned through self-attention across particles, permutation equivariance is preserved through the internal network, and antisymmetry is enforced by determinant structure. In published realizations, this design has been used to lower molecular VMC energies below FermiNet on many benchmarks [2211.13672], to outperform Hartree–Fock and band-projected exact diagonalization in a moiré solid [2502.05383], to identify chiral \(p_x\pm i p_y\) superconductivity without explicit pairing bias [2509.03683], to solve fractional quantum Hall states in real space [2412.00618], and to reach large-scale 2D electron-gas calculations with energies below fixed-node DMC [2607.08616].

Source: https://www.emergentmind.com/topics/self-attention-fermi-neural-network