Spectrum-Weighted RFF-MSPINNs for PDEs

• The paper introduces a novel multistage physics-informed neural network that integrates spectrum-weighted random Fourier features to overcome spectral bias in PDE solvers.
• It dynamically samples Fourier frequencies based on the residual's power spectrum, directing network capacity toward high-frequency and multiscale components.
• Empirical results show one to two orders of magnitude error reduction over standard PINNs, enhancing both accuracy and training efficiency in complex PDE applications.

Spectrum-Weighted Random Fourier Feature Multistage Physics-Informed Neural Networks (RFF-MSPINNs) are a class of neural PDE solvers designed to overcome spectral bias in standard Physics-Informed Neural Networks (PINNs) by incorporating spectrum-informed random Fourier features and a multistage residual learning strategy. These approaches systematically adapt the representation capacity and optimization trajectory of PINNs to capture high-frequency and multiscale solution components with significantly improved accuracy and efficiency, supported by formalism and experiments across multiple recent works (Li et al., 25 Aug 2025, Feng et al., 21 Dec 2025, Wu et al., 22 Oct 2025).

1. Motivation: Spectral Bias and PINNs

PINNs approximate solutions to partial differential equations (PDEs) by optimizing a composite loss measuring how well the candidate solution satisfies the PDE operator, boundary, and (if applicable) initial conditions. A key limitation of standard PINNs is spectral bias: they fit low-frequency components of the solution rapidly, while high-frequency components converge slowly or are missed entirely. This hinders performance on wave propagation, turbulence, and scattering problems containing rich high-frequency content (Li et al., 25 Aug 2025).

Random Fourier features (RFFs) partially address spectral bias by embedding high-frequency components at the input stage. However, conventional RFFs sample frequencies $\omega$ from a static Gaussian distribution, which is agnostic to the multiscale spectral properties of the solution or residuals. RFF-MSPINNs build upon this by dynamically sampling frequencies and phases based on the power spectral density (PSD) of the current PDE residual, thereby directing model capacity toward dominant, energetically relevant frequency bands.

2. Architecture: Spectrum-Weighted RFF Input Embedding

At each stage $i$ in the multistage scheme, an RFF layer precedes the main neural network:

$$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$

• Without spectrum weighting, $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$, $b_j \sim \text{Unif}[0, 2\pi]$.
• In RFF-MSPINNs, frequencies $\omega_j$ are selected with probability proportional to the power in the corresponding spectral band of the residual:

1. Compute DFT of the stage residual $r_{i-1}$:

   $$\hat{r}_{i-1}(k) = \sum_{n=1}^{N_d} r_{i-1}(x^{(n)}) e^{-2\pi i\,k \cdot x^{(n)}}$$

2. Form the unnormalized power spectrum $S_{i-1}(k) = |\hat{r}_{i-1}(k)|^2$.
3. Normalize to obtain a frequency PMF: $p_{i-1}(k) = S_{i-1}(k)/\sum_{k'} S_{i-1}(k')$.
4. Draw $i$0 frequencies $i$1, and $i$2 uniformly.

This sampling biases the input representation toward residual-dominant modes, enabling rapid targeted reduction in error at challenging frequencies (Li et al., 25 Aug 2025).

Extensions further enhance input embeddings with cross-attention, multi-scale banks with learnable amplitudes, and incremental enrichment mechanisms, as detailed in (Feng et al., 21 Dec 2025).

3. Multistage Residual Learning Framework

The multistage paradigm decomposes the solution $i$3 as a sum of stage-wise corrections. At stage $i$4:

• The model $i$5, where $i$6 is the RMS of the previous residual.
• $i$7 is a fully-connected network applied to the spectrum-informed RFF embedding $i$8.
• The loss per stage is:

$i$9

The process continues until the RMS of the stage residual decreases below a prescribed threshold relative to the previous stage. This approach systematically improves the approximation of both low- and high-frequency components (Li et al., 25 Aug 2025).

4. Spectrum-Weighted RFF Variants and Further Enhancements

Several architectural variants extend the RFF-MSPINN principle:

• Learnable amplitude scaling: The input RFF bank is augmented by amplitude factors $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$0, with $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$1 trainable to focus the representational power on relevant frequencies (Feng et al., 21 Dec 2025).
• Cross-attention reweighting: RFF outputs are partitioned into tokens, which interact with a query through cross-attention, adaptively routing capacity toward the most informative scales for each input.
• Incremental spectral enrichment: Dominant modes identified from intermediate DFT analysis of the current approximation are appended to the RFF bank and smoothly introduced via soft-masking into the attention mechanism.
• Two-network PDE mixing: Separates the modeling of high-frequency (via RFF-cross-attention) and low-frequency (via MLP) solution components, with the final output $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$2. The mixing factor $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$3 is either learned or set by a physics-motivated closed-form formula at each training epoch (Feng et al., 21 Dec 2025).

These enhancements further accelerate convergence on oscillatory and discontinuous solutions and allow the architecture to adapt to evolving frequency-domain demands during training.

5. Algorithmic Description

A high-level pseudocode for standard RFF-MSPINN training (Li et al., 25 Aug 2025):

$r_{i-1}$5

Typical hyperparameters include $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$4–$$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$5 features, $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$6–$$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$7 hidden layers of $$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$8–$$\phi_i(x; \{\omega_j, b_j\}_{j=1}^M) = \sqrt{\frac{2}{M}} \left[ \cos(\omega_1^T x + b_1), \ldots, \cos(\omega_M^T x + b_M) \right]^T$$9 units, $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$0 or GELU activations, and a stage count $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$1–$\omega_j \sim \mathcal{N}(0, \sigma^2 I)$2 (Li et al., 25 Aug 2025).

For variants with cross-attention and amplitude scaling, the procedure includes construction of multiscale RFF banks, grouping into frequency tokens, and staged introduction of posterior DFT frequencies (Feng et al., 21 Dec 2025). Iterative “θ‐solve + ω‐gradient” upper-lower loop training is also used, especially when optimizing frequency bases (Wu et al., 22 Oct 2025).

6. Empirical Performance and Benchmarks

RFF-MSPINNs have demonstrated one to two orders of magnitude lower error than conventional PINNs and standard Multistage Neural Networks (MSNNs) across several canonical PDEs:

• Burgers Equation (shock regime):
  • PINN: $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$3
  • MSNN: $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$4
  • RFF-MSPINNs (3 stages): $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$5
• 2D Helmholtz Scattering: For $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$6, the $\omega_j \sim \mathcal{N}(0, \sigma^2 I)$7 relative errors for real ($\omega_j \sim \mathcal{N}(0, \sigma^2 I)$8) and imaginary ($\omega_j \sim \mathcal{N}(0, \sigma^2 I)$9) fields are

$b_j \sim \text{Unif}[0, 2\pi]$0	Method	$b_j \sim \text{Unif}[0, 2\pi]$1	$b_j \sim \text{Unif}[0, 2\pi]$2
1	PINN	$b_j \sim \text{Unif}[0, 2\pi]$3	$b_j \sim \text{Unif}[0, 2\pi]$4
	MSNN	$b_j \sim \text{Unif}[0, 2\pi]$5	$b_j \sim \text{Unif}[0, 2\pi]$6
	RFF-MSPINNs	$b_j \sim \text{Unif}[0, 2\pi]$7	$b_j \sim \text{Unif}[0, 2\pi]$8
1.5	PINN	$b_j \sim \text{Unif}[0, 2\pi]$9	$\omega_j$0
	MSNN	$\omega_j$1	$\omega_j$2
	RFF-MSPINNs	$\omega_j$3	$\omega_j$4
2	PINN	$\omega_j$5	$\omega_j$6
	MSNN	$\omega_j$7	$\omega_j$8
	RFF-MSPINNs	$\omega_j$9	$r_{i-1}$0

• High-frequency and multi-scale PDEs: Spectrum-guided RFF enrichment enables RFF-MSPINN and its iterative Fourier-augmented relatives to maintain accuracy in cases where vanilla PINNs diverge or entirely lose high-frequency content (Li et al., 25 Aug 2025, Wu et al., 22 Oct 2025).

Training time is also reduced relative to Spectrum-Informed Multistage PINNs (SI-MSPINNs); e.g., 700 s (RFF-MSPINNs) vs. 1300 s (SI-MSPINNs) for three-stage training on Burgers equation (Li et al., 25 Aug 2025).

7. Theoretical Guarantees and Expressivity

Random Fourier Feature embedding strictly enlarges the hypothesis class of a PINN. As the feature count $r_{i-1}$1 grows and with sufficient training, RFF-MSPINNs achieve universal $r_{i-1}$2 approximation on PDE solution spaces—assuming sufficient depth and expressivity in the base network (Wu et al., 22 Oct 2025).

Moreover, for linear operators $r_{i-1}$3, the two-stage iterative (θ-solve, ω-update) scheme converges provably to a stationary point of the loss, and the linear regression problem admits a closed-form solution for the readout weights, further accelerating convergence (Wu et al., 22 Oct 2025).

A spectrum analysis ablation confirms that, as the number of Fourier features increases, the normalized DFT magnitude of the network output at high frequencies approaches $r_{i-1}$4, whereas standard PINNs remain near zero—direct evidence of effective mitigation of spectral bias (Wu et al., 22 Oct 2025).

---

References:

• (Li et al., 25 Aug 2025)
• (Feng et al., 21 Dec 2025)
• (Wu et al., 22 Oct 2025)