Σ-Attention in Quantum Operator Learning

• Σ-Attention is a unified framework that uses Transformer-based operator learning to approximate self-energy operators in quantum many-body systems.
• It integrates data from perturbation theory, strong-coupling expansion, and exact diagonalization to accurately capture interaction effects.
• The approach employs spectral covariance analysis to optimize pooling in sequence models, enabling efficient scaling to larger system sizes.

Σ-Attention denotes both a class of Transformer-based operator-learning frameworks for approximating self-energy operators in strongly correlated quantum systems and, more generally, a theoretical approach to attention mechanisms via spectral and covariance analysis in the high-dimensional regime. In these contexts, Σ-Attention captures the interface between signal-processing optimality, free probability, and scalable learning architectures, unifying attention mechanisms with operator discovery from many-body data. The methodology is employed for universal self-energy approximation, efficient scaling for large systems, and theoretical justification for optimal pooling in sequence models (Zhu et al., 20 Apr 2025, Seddik, 7 May 2026).

1. Operator Learning for Self-Energy in Strongly Correlated Systems

At its core, Σ-Attention in quantum many-body physics infers the self-energy operator $\Sigma(i\omega_n)$—a central object in the Matsubara Green’s function formalism, where interaction effects are encoded in the relation

$$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$

with $G_0$ the non-interacting propagator and $G$ the fully interacting Green’s function [(Zhu et al., 20 Apr 2025), Eq. 3–4]. The challenge in correlated systems is finding $\Sigma$ efficiently and accurately across distinct parameter regimes where traditional analytic or numerical approaches are limited.

Σ-Attention addresses this challenge by deploying an encoder-only Transformer as a universal ansatz for $\Sigma$, ingesting a hybrid dataset that combines results from:

• Many-Body Perturbation Theory (MBPT) (e.g., 2nd-Born, GW) for weak coupling and large sizes,
• Strong-Coupling Expansion (SCE) for strong interaction regimes,
• Exact Diagonalization (ED) for small systems across couplings.

The model synthesizes these disparate data regimes into a single learnable map, using attention mechanisms to generalize self-energy functionals across the entire phase space, as shown in the unified density-of-states and Mott transition predictions for the 1D Hubbard model [(Zhu et al., 20 Apr 2025), Table I; Figs. 2–3].

2. Transformer Architecture and Mathematical Ansatz

Σ-Attention employs a momentum- or site-indexed encoder-only Transformer. Tokens correspond to discrete momenta $k$ (or sites $i$), with each feature vector composed of:

• The bare Green’s function $G_0(k, \tau_j)$ sampled on a grid,
• The (broadcasted) interaction strength $U$,
• A momentum-dependent positional encoding $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$0.

The full input matrix is $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$1, where $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$2 is the number of momentum points and $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$3 the number of imaginary-time grid points [(Zhu et al., 20 Apr 2025), Fig. 1].

The attention modules are standard multi-head architectures, using $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$4 heads ($$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$5), residual and LayerNorm blocks, and a shallow ($$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$6 layers) configuration. After multi-head self-attention and feed-forward mapping, the network outputs the scaled self-energy $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$7 across the $$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$8 grid.

Mathematically, the network realizes a map

$$G^{-1}(i\omega_n) = G_0^{-1}(i\omega_n) - \Sigma(i\omega_n)$$9

where $G_0$0 denotes the representation after $G_0$1 Transformer layers and $G_0$2 is a learned projection (Zhu et al., 20 Apr 2025).

3. Training, Dataset Construction, and Optimization

Σ-Attention’s training set consists of tens of thousands of (input, target) pairs sampled over the parameter grid:

• MBPT covers $G_0$3 for $G_0$4 to $G_0$5,
• SCE populates $G_0$6 and $G_0$7 up to $G_0$8,
• ED supplies data for $G_0$9 to $G$0, $G$1 to $G$2.

Overlap domains allow cross-validation, ensuring no regime exceeds $G$3 errors $G$4 relative to ED. During training, Σ is scaled as

$G$5

to ensure magnitude compatibility with $G$6. The loss combines $G$7 and $G$8 terms

$G$9

With Adam optimizer and CosineAnnealingLR learning rate scheduling, no explicit regularization (e.g., dropout) is used beyond stochastic batching (Zhu et al., 20 Apr 2025).

4. Scaling, Generalization, and Theoretical Underpinnings

The model’s architecture admits scaling to larger system sizes owing to the attention block’s invariance to sequence length $\Sigma$0—projection weights remain constant irrespective of input width. This confers an advantage relative to auxiliary-field QMC, whose $\Sigma$1 cost per iteration contrasts with the Transformer’s $\Sigma$2 per layer, enabling practical simulation of $\Sigma$3 sites with negligible cost (Zhu et al., 20 Apr 2025).

Beyond application, Σ-Attention also provides a principled spectral-theoretic view of attention in the high-dimensional limit (Seddik, 7 May 2026). Consider a pooled representation $\Sigma$4, with tokens from a Gaussian mixture embedding table and attention weights $\Sigma$5. The sample covariance $\Sigma$6 has population-level structure

$\Sigma$7

where $\Sigma$8, $\Sigma$9, $\Sigma$0 is the positional correlation, and $\Sigma$1 is the effective spike-to-noise ratio.

The spectrum of $\Sigma$2 converges to $\Sigma$3 (a free multiplicative convolution of Marchenko–Pastur laws), and signal recovery is characterized by BBP-type phase transitions:

• Spike outlier appears when $\Sigma$4,
• Optimal recovery is achieved by maximizing $\Sigma$5.

The optimal weights correspond to the normalized top eigenvector $\Sigma$6 of $\Sigma$7: $\Sigma$8 This formalizes the attention mechanism as a covariant pooling that maximizes signal recovery from sequences [(Seddik, 7 May 2026), Theorems 2, 5].

5. Empirical Performance and Physical Applications

On the 1D half-filled Hubbard model ($\Sigma$9), Σ-Attention accurately predicts the Matsubara Green’s function $k$0 across interaction strengths $k$1 to $k$2, retaining $k$3-norm errors $k$4–$k$5 relative to AFQMC (see log-scale error plots). Crucially, analytic continuation of the output recovers the Mott gap opening at $k$6, matching QMC benchmarks, while conventional 2nd-Born theory fails to capture this phase transition [(Zhu et al., 20 Apr 2025), Figs. 2–3]. Thus, Σ-Attention demonstrates high-fidelity interpolation between weak- and strong-coupling regimes, with charge-gap prediction errors $k$7.

Standard causal self-attention, particularly in the high-dimensional limit with dot-product score scaling ($k$8), yields deterministic harmonic attention weights that allocate higher mass to early positions. For signals concentrated at the start of the sequence, causal Σ-Attention strictly outperforms mean pooling, as formalized by the ratio $k$9 versus $i$0 for $i$1 [(Seddik, 7 May 2026), Section 5].

6. Limitations and Prospects

While Σ-Attention architecture efficiently approximates the self-energy operator and generalizes to larger system sizes, it does not strictly enforce the correct analytic (causal) structure of $i$2 and $i$3, resulting in small causality violations for $i$4. Possible remediation involves embedding spectral sum rules or Nevanlinna constraints into the architecture. Current models implement a "bare" ansatz $i$5; incorporating a "renormalized" ansatz $i$6 within a fixed-point iteration is a prospective extension.

Application to other quantum lattice models, multiple orbitals, finite doping, and temperature tuning is possible by expanding the input matrix with additional Hamiltonian parameters, with training data generated from cluster DMFT, diagrammatic QMC, or ED. Further generalization includes learning two-particle vertices or dynamical susceptibilities, potentially enabling operator learning for observables beyond the single-particle regime (Zhu et al., 20 Apr 2025).

7. Unified View: Covariance Perspective on Attention

Σ-Attention unifies practical operator learning and theoretical principles underlying attention. By interpreting pooling via attention as a spectral operation on covariance matrices, the framework connects free probability, random matrix theory, and BBP phase transitions with architectural and optimization choices in neural sequence models. Maximizing the Rayleigh quotient $i$7 defines the optimal pooling strategy, recoverable as the leading eigenvector of the positional correlation $i$8. This provides both (i) a blueprint for constructing data-efficient operator learners in physics and (ii) a rigorous statistical rationale for observed empirical advantages of attention mechanisms over uniform pooling in high-dimensional settings (Zhu et al., 20 Apr 2025, Seddik, 7 May 2026).