PeSANet: Physics-Encoded Spectral Attention Network

• PeSANet is a neural network that fuses physical constraints with spectral attention to accurately model and forecast complex PDE-governed systems.
• It alternates between local physics encoding and global spectral transformations, enabling effective learning on both structured and unstructured mesh data.
• Empirical results show that PeSANet outperforms traditional spectral and attention-based models, particularly in data-scarce and zero-shot scenarios.

The Physics-Encoded Spectral Attention Network (PeSANet) is a neural framework that incorporates both physics-based constraints and spectral-domain attention to model and forecast complex systems governed by partial differential equations (PDEs). It provides a unified strategy for integrating local physical dynamics and global frequency-domain interactions, allowing for robust generalization and adaptability, especially in data-scarce or partially known physical regimes. PeSANet is distinguished from other neural solvers by combining operator learning with spectral attention mechanisms, leveraging physically meaningful bases, and calibrating their local activation pointwise. This approach has demonstrated superior performance on a diverse range of scientific PDE tasks, outperforming both spectral-only and attention-only models in multiple regimes (Wan et al., 3 May 2025, Yue et al., 2024).

1. Theoretical Foundation and Motivation

Operator learning for PDEs has historically bifurcated into two families. Attention-based models (e.g., Transformers) excel in point-wise adaptation but lack global spectral structure, making them vulnerable to poor generalization under distribution or resolution shift. Conversely, spectral neural operators (e.g., FNOs) encode global continuity and enable zero-shot generalization to new meshes but show limited flexibility within complex domains and struggle with sharp local features.

PeSANet seeks to bridge this dichotomy by explicitly encoding both the physical structure (via domain-specific spectral bases) and adaptive attention across frequencies or spatial locations. It is motivated by the limitations of traditional numerical methods in the face of missing or incomplete laws and the challenge neural models face when data is limited and global structure is crucial (Wan et al., 3 May 2025, Yue et al., 2024).

2. Architectural Principles

PeSANet alternates between physics-encoded blocks and spectral attention mechanisms to fuse local and global modeling capabilities. The physics-encoded component imposes hard physical constraints, approximating local differential operators from data even when those operators or boundary conditions are only partially observed. The spectral component projects latent representations into a physically meaningful frequency space, typically the eigenfunctions of the Laplace–Beltrami operator for general domains or standard Fourier bases for regular grids (Wan et al., 3 May 2025, Yue et al., 2024).

A prototypical PeSANet block, as implemented in the Holistic Physics Mixer (HPM) variant, involves:

• Local embedding of input fields into a latent space.
• Point-Calibrated Spectral Transform: Multi-head mixing where at each point, a learned “gate” modulates the contribution of each spectral basis function.
• Spectral-domain linear transformations (shared or per-head).
• Inverse projection back to physical space using the same gates.
• Channel-wise feedforward mixing.
• Residual connections and normalization layers for stability and expressivity.

This cycle can be repeated several times (L blocks), yielding multi-scale, globally aware, yet locally adaptive representations (Yue et al., 2024).

3. Mathematical Formalism

Let $X \in \mathbb{R}^{N \times d_v}$ be the latent feature matrix for $N$ mesh points. The PeSANet core spectral attention module is defined as follows:

• Compute the first $N^k$ Laplace–Beltrami eigenfunctions $\{\phi_i\}_{i=1}^{N^k}$ for the domain.
• Predict a point-wise gate matrix $G \in [0,1]^{N \times N^k}$ using

$$G = \text{Softmax}_k(\mathrm{MLP}_{\text{gate}}(X))$$

• Perform the forward transform:

$$\hat{X}_i = \phi_i^\top (G_{:,i} \odot X), \quad i=1\,\ldots, N^k$$

• Inverse transform:

$$X' = \sum_{i=1}^{N^k} (G_{:,i} \odot \phi_i)\, \hat{X}_i$$

Each head in a multi-head architecture is processed independently, concatenated, and updated through further residual and feed-forward layers. Optional standard attention can be introduced in parallel to the spectral mixing step (Yue et al., 2024).

4. Physics Encoding and Spectral Adaptivity

The use of Laplace–Beltrami eigenfunctions as basis functions encodes the domain’s geometry and boundary conditions directly into the latent space. Low-frequency (low-eigenvalue) modes capture smooth, globally coherent features, while high-frequency modes enable modeling of sharp gradients, boundary layers, or localized phenomena.

Pointwise gates $G$ allow each mesh node to recruit the optimal mixture of spectral components, controlling the degree of local adaptivity. This mechanism ensures that the network can flexibly prioritize long-range regularities or fine-scale details as dictated by the local PDE solution manifold (Yue et al., 2024).

Furthermore, the spectral attention mechanism supports strong zero-shot generalization to unseen resolutions, as the basis functions and mixing process remain meaningful without retraining, a property difficult to achieve with purely attention-based or convolutional networks.

5. Training Methodology and Loss Functions

PeSANet is amenable to standard function-space supervision using relative L2 loss:

$$\mathcal{L}_{\text{rec}} = \frac{1}{N_\mathrm{test}} \sum_{i=1}^{N_\mathrm{test}} \frac{\| u_\text{pred}^{(i)} - u_\text{true}^{(i)} \|_2}{\| u_\text{true}^{(i)} \|_2}$$

No explicit PDE residual or boundary penalty is required, as the spectral basis encodes appropriate physical structure (Yue et al., 2024). Training is typically performed with AdamW and OneCycleLR or StepLR schedules; hyperparameter selection depends on the specific PDE and data regime (Wan et al., 3 May 2025, Yue et al., 2024).

Hardware and optimization protocols are as follows (Wan et al., 3 May 2025):

• Single NVIDIA A100 GPU (80GB), Intel Xeon Platinum 8380 (64 cores).
• Problem-specific batch sizes, epochs, and initial learning rates, e.g.:

Case	Batch Size	Epochs	Initial LR
Burgers	8	5,000	$1\times 10^{-4}$
Gray–Scott	8	5,000	$5\times 10^{-4}$
FitzHugh–Nagumo	32	8,000	$5\times 10^{-4}$
Navier–Stokes	32	8,000	$1\times 10^{-4}$

Learning rates are decayed via StepLR at problem-specific rates.

6. Empirical Capabilities and Benchmark Performance

Extensive studies across both structured (e.g., Darcy flow, Navier–Stokes) and unstructured (e.g., irregular Darcy, pipe turbulence) meshes demonstrate that PeSANet achieves superior relative L2 error compared to state-of-the-art baselines, including:

• Spectral-only (FNO, WNO, LSM, NORM)
• Attention-only (Transolver, GNOT)

Key empirical findings include:

• When training data is limited (200–400 samples), PeSANet matches or exceeds the performance of fixed-basis spectral approaches while strongly outperforming attention models.
• With abundant data, PeSANet continues to improve similarly to attention models, whereas spectral-only methods saturate.
• In zero-shot generalization (testing on new grid resolutions), PeSANet maintains low error (∼$10^{-2}$), whereas attention-only baselines show error increases by an order of magnitude.
• Ablations show that pointwise calibration via softmax gates is critical, particularly when only a small number of frequencies are retained (Yue et al., 2024).

As a case study in the original work, PeSANet outperformed FNO, PeRCNN, Factorized FNO, and FactFormer on Burgers’, Gray–Scott, FitzHugh–Nagumo, and incompressible Navier–Stokes systems with periodic boundaries across all metrics, especially in long-term forecasting—though detailed quantitative tables are not available in extracted appendix content (Wan et al., 3 May 2025).

7. Representative Applications and Broader Impact

The architecture has been applied successfully to a range of canonical PDEs in two spatial dimensions, including:

• 2D Burgers’ Equation
• 2D Gray–Scott Reaction–Diffusion
• 2D FitzHugh–Nagumo
• 2D incompressible Navier–Stokes (high Reynolds number, periodic BC)

A plausible implication is that PeSANet can be adapted to both structured and unstructured meshes, enabling applicability to real-world physical systems with complex geometries. Its principled balance of spectral and spatial modeling provides a pathway toward neural solvers that are both robust to data scarcity and capable of strong generalization across domain geometries and discretizations (Wan et al., 3 May 2025, Yue et al., 2024).

• PeSANet: Physics-encoded Spectral Attention Network for Simulating PDE-Governed Complex Systems (Wan et al., 3 May 2025)
• Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space (Yue et al., 2024)
• Physics-Informed Neural Networks with Fourier Features and Attention-Driven Decoding (Arni et al., 6 Oct 2025)