Attention-Based DNN for Quantum Simulations

• Attention-based deep neural networks are models that incorporate mechanisms to selectively emphasize key input features and effectively represent highly entangled quantum states.
• They employ architectures such as RBM and CNN, using variational Monte Carlo to optimize complex wavefunctions with polynomial computational scaling over traditional methods.
• These methods enable accurate quantum-state tomography and extend to quantum chemistry, offering breakthroughs in simulating strongly correlated and nonlocal interactions.

An attention-based deep neural network in the context of quantum many-body simulation refers to a neural-network variational quantum state (NQS) ansatz wherein the network architecture incorporates mechanisms that prioritize or weight specific components, features, or correlations within input configurations—typically via non-linear activation functions or convolutional layers with built-in symmetry constraints. These attention-like mechanisms enable the representation and efficient optimization of complex, highly entangled wavefunctions in variational Monte Carlo (VMC), especially for spin and fermion systems. Such networks have demonstrated substantial advantages in capturing nonlocal interactions and flattened computational scaling in contrast to classical tensor-network or quantum Monte Carlo methods.

1. Neural Quantum State Ansatz and Architecture

In neural-network-based VMC, the wavefunction $\psi_\theta(S)$ of a many-body spin or fermion system is parameterized by network parameters $\theta$ and input configuration $S=(S_1,\dotsc,S_N)$ (Song, 3 Jun 2024). Two principal classes of attention-enabled architectures are commonly deployed:

• Restricted Boltzmann Machine (RBM):

$$\Psi_{\rm RBM}(S;\theta) = e^{\sum_j a_j S_j} \prod_{i=1}^M 2\cosh\!\left(b_i + \sum_j W_{ij} S_j\right)$$

Here, the hidden-unit density $\alpha=M/N$ plays a role analogous to the bond dimension in tensor networks.

• Deep Feedforward / Convolutional Neural Networks:

Layers are constructed as

$$z_j^{(n+1)} = \sum_k w_{jk}^{(n+1)}\,y_k^{(n)} + b_j^{(n+1)}, \quad y_j^{(n+1)} = f(z_j^{(n+1)})$$

where $f$ is a non-linear activation such as sigmoid, ReLU, or SELU. The network mapping $F_\theta(S)$ defines real/imaginary parts of the log-amplitude, and the output wavefunction is

$$\psi_\theta(S) = \exp\left[\mathrm{Re}\,F_\theta(S)\right] \exp\left[i\,\mathrm{Im}\,F_\theta(S)\right]$$

Complex-valued non-linearities such as SELU are used for expressivity in representing phases.

Feature prioritization or "attention" emerges in the convolutional layers, symmetry-aware pooling, and activation patterns, enabling the network to selectively focus on relevant spatial/spin configurations and correlations.

2. Variational Energy and Local Energy Formulation

Identification of the ground state or low-lying eigenstates is formulated via minimization of the energy Rayleigh quotient,

$$E[\theta] = \frac{\langle\Psi_\theta|H|\Psi_\theta\rangle}{\langle\Psi_\theta|\Psi_\theta\rangle} = \sum_S \left(\sum_{S'} H_{S,S'} \frac{\psi_\theta(S')}{\psi_\theta(S)} \right) \frac{|\psi_\theta(S)|^2}{\sum_{S''} |\psi_\theta(S'')|^2} \approx \left\langle E_{\rm loc}(S) \right\rangle_{S\sim|\psi_\theta|^2}$$

where the local energy is

$$E_{\rm loc}(S) = \sum_{S'} H_{S,S'} \frac{\psi_\theta(S')}{\psi_\theta(S)}$$

The expectation value is estimated via Markov-chain Monte Carlo (MCMC) sampling of configurations distributed as $|\psi_\theta(S)|^2$.

3. Gradient Estimation and Optimization Strategies

Stochastic optimization proceeds by estimating the gradient of $E[\theta]$ with respect to $\theta$ using the log-derivative trick,

$$O_k(S) = \frac{\partial}{\partial\theta_k}\ln\psi_\theta(S)$$

yielding

$$\frac{\partial E}{\partial\theta_k} = 2 \left( \langle E_{\rm loc}(S)\,O_k(S)\rangle - \langle E_{\rm loc}(S)\rangle\,\langle O_k(S)\rangle \right)$$

This covariance structure is particularly suited to first-order optimizers (SGD, Adam) and second-order methods such as stochastic reconfiguration (SR),

$S_{kk'}\,\delta\theta_{k'} = \tau\,F_k,$

with the quantum-geometric tensor $$S_{kk'} = \langle O_k^* O_{k'} \rangle - \langle O_k^* \rangle \langle O_{k'} \rangle$$ and force vector $$F_k = \langle O_k^* H \rangle - \langle O_k^* \rangle \langle H \rangle$$. Regularization and learning-rate annealing are used for stability and convergence.

4. Monte Carlo Sampling and Symmetry Constraints

Sampling is performed by constructing an MCMC chain with detailed balance. For a symmetric proposal $S(S\to S') = S(S'\to S)$, the acceptance probability is

$$A(S\to S') = \min\left(1, \frac{|\psi_\theta(S')|^2}{|\psi_\theta(S)|^2}\right)$$

Efficient sampling, chain thinning, and block-averaging are used to reduce estimator variance and autocorrelation. Imposing symmetry constraints (translational, point-group) and using group-equivariant convolutions reduces the number of network parameters and ensures physical invariances.

5. Benchmarks and Computational Scaling ("Flattening" Advantage)

Attention-based deep NQS architectures deliver significant advantages over tensor-network or QMC methods in representing highly nonlocal, entangled states, especially in high dimensions:

• RBM ansatz with hidden-unit density $\alpha = 1\!-\!8$ achieves relative energy error $\varepsilon_{\rm err} < 10^{-3}$ for $5\times5$ Ising model (Song, 3 Jun 2024).
• Accurate location of critical points and correlation functions within a few percent for 1D chains up to $L=60$.
• For the frustrated $J_1$-$J_2$ Heisenberg model on $4\times4$ and $6\times6$ lattices, CNNs distinguish Néel/plaquette/columnar-VBS by structure factors.
• Sampling and optimization costs grow only polynomially with system size: $\mathcal O(10^5)$ samples, $10^2$-$10^3$ gradient steps per run, total wall time of hours on a single GPU.

Deep/wide NQS with $\mathcal O(10^4$–$10^5)$ parameters efficiently flatten the scaling barrier that plagues tensor networks (exponential bond-dimension growth) and QMC (sign problems).

6. Quantum-State Tomography and Extension to Quantum Chemistry

Neural quantum states as attention-based deep neural networks excel in quantum-state tomography and ab initio quantum chemistry:

• Neural networks generalize the representation of interactions, including nonlocal correlations inherent in many-body systems.
• In quantum-state tomography, NQS representation has achieved significant results, enabling physical characterization of larger systems.
• These methods extend to fermionic systems and positronic chemistry, with architectures such as FermiNet demonstrating accurate ground-state and annihilation observables across varied molecules (Cassella et al., 2023).
• The ability to encode cusp conditions, sign structures, and many-body correlations is critical in quantum chemistry applications.

7. Implications and Frontiers

The success of attention-based deep neural network NQS in VMC suggests several directions:

• Fully exploiting network non-linearity and expressive capacity is vital, especially when network width/depth can be increased to match physical complexity.
• Incorporation of symmetry via convolutional and equivariant layers further optimizes both expressiveness and computational efficiency.
• These architectures are expected to scale to much larger system sizes, enable tomography of complex quantum states, and deliver new insights into strongly correlated materials.
• The precise flattening advantage and ability to represent highly entangled, nonlocal states are key breakthroughs in the study of quantum many-body systems.

In summary, attention-based deep neural networks in VMC—such as RBM, feedforward, and convolutional NQS—constitute a powerful, scalable, and expressive framework for quantum many-body simulation. Their methodological innovations in network construction, sampling, and optimization have set new benchmarks in accuracy and efficiency for both model and ab initio systems (Song, 3 Jun 2024).