Spectral Bias Mitigation via xLSTM-PINN: Memory-Gated Representation Refinement for Physics-Informed Learning (2511.12512v1)

Abstract: Physics-informed learning for PDEs is surging across scientific computing and industrial simulation, yet prevailing methods face spectral bias, residual-data imbalance, and weak extrapolation. We introduce a representation-level spectral remodeling xLSTM-PINN that combines gated-memory multiscale feature extraction with adaptive residual-data weighting to curb spectral bias and strengthen extrapolation. Across four benchmarks, we integrate gated cross-scale memory, a staged frequency curriculum, and adaptive residual reweighting, and verify with analytic references and extrapolation tests, achieving markedly lower spectral error and RMSE and a broader stable learning-rate window. Frequency-domain benchmarks show raised high-frequency kernel weights and a right-shifted resolvable bandwidth, shorter high-k error decay and time-to-threshold, and narrower error bands with lower MSE, RMSE, MAE, and MaxAE. Compared with the baseline PINN, we reduce MSE, RMSE, MAE, and MaxAE across all four benchmarks and deliver cleaner boundary transitions with attenuated high-frequency ripples in both frequency and field maps. This work suppresses spectral bias, widens the resolvable band and shortens the high-k time-to-threshold under the same budget, and without altering AD or physics losses improves accuracy, reproducibility, and transferability.

• The paper introduces xLSTM-PINN, which integrates memory gating and intra-layer recursion to mitigate spectral bias in PINNs.
• It reshapes the neural tangent kernel eigen-spectrum, improving high-frequency convergence and reducing error metrics by up to three orders of magnitude.
• Empirical evaluations on various PDEs, including 1D advection-reaction and 2D Laplace, demonstrate more rapid, stable convergence compared to traditional PINNs.

Spectral Bias Mitigation in Physics-Informed Neural Networks with xLSTM-PINN

Introduction

Physics-Informed Neural Networks (PINNs) have established themselves as a viable mesh-free paradigm for solving forward and inverse problems involving partial differential equations (PDEs), embedding physical constraints directly into the learning objective. A well-documented fundamental limitation of conventional PINNs is pronounced spectral bias: a tendency to learn low-frequency solution components faster than high-frequency content. This limitation is particularly deleterious for problems involving sharp gradients, oscillatory regimes, or multi-scale physics, as the network struggles to capture and resolve high-wavenumber features, regardless of its overall parameter count or training budget.

The paper "Spectral Bias Mitigation via xLSTM-PINN: Memory-Gated Representation Refinement for Physics-Informed Learning" (2511.12512) addresses these challenges by proposing xLSTM-PINN, an architectural refinement that introduces memory gating and intra-layer recursion (residual micro-steps) via xLSTM blocks. Crucially, this approach modifies only the representation subnetwork within PINNs, leaving physics losses and automatic differentiation (AD) pathways unchanged, thereby providing a clean decoupling of architectural spectral engineering and domain constraints.

xLSTM-PINN Architecture: Memory Gating and Residual Micro-Steps

The core innovation resides at the representation level, where each layer is replaced by an xLSTM block. Rather than a conventional feedforward stack, xLSTM blocks implement intra-layer recursions over a "micro-time" axis, with gating mechanisms analogous to LSTM but adapted for spatially extended representations of physical coordinates.

Figure 1: xLSTM-PINN architecture overview, illustrating the injection of memory gating and intra-block micro-steps into the representation pipeline while retaining standard PINN physics loss and AD routes.

Within each block, a set number $S$ of micro-steps recursively refine intermediate feature states through memory update equations, residual merges, and channel-wise feedforward gating. This effectively increases the functional depth of the representation without introducing additional trainable parameters. The design maintains $\mathcal{O}(L W^2)$ parameter scaling (with $L$ layers of width $W$) while compute cost scales as $\mathcal{O}(L S W^2)$, with $S$ controlling the intra-layer recursion depth.

Importantly, the architecture supports direct compatibility with the standard PINN training loop: input coordinates $\mathbf{x}$, physics-based loss construction via AD, and multi-term empirical risk over PDE residuals and boundary conditions.

Theoretical Analysis and Spectral Bias Mitigation

The spectral bias in neural PDE solvers can be understood formally via the eigen-spectrum of the neural tangent kernel (NTK) induced by the representation. xLSTM-PINN is shown to reshape this spectrum by lifting high-frequency eigenvalues through a residual linearization and memory gating mechanism. Explicit bounds are derived for the ratio of kernel eigenvalues between xLSTM-PINN and baseline PINNs, showing that if the frequency gain term $\alpha(\mathbf{k})$ grows with wavenumber magnitude $|\mathbf{k}|$, operating $S$ micro-steps per block monotonically increases high-frequency convergence.

The linearized training dynamics for frequency error components $e_{\mathbf{k}}$ show that xLSTM-PINN achieves:

• Exponentially faster decay of high-frequency errors,
• Lower endpoint frequency error across a larger spectrum,
• A right-shifted "resolvable bandwidth" $k^*(\varepsilon)$ for any fixed error threshold.

This is validated by a benchmarking suite probing the spectral response under fixed training budget and model capacity.

Figure 2: Spectral benchmarking of xLSTM-PINN reveals suppression of spectral bias, with reduced endpoint errors, improved high-frequency gain, and more rapid time-to-threshold across wavenumbers.

Empirical Evaluation: Benchmark PDEs

1D Advection–Reaction (Drift–Decay) Equation

A constant-coefficient first-order PDE serves as an initial testbed. xLSTM-PINN displays a narrow, characteristic-aligned error band and low background error, while the baseline PINN shows thickened error stripes and hotspot localization. Quantitatively, xLSTM-PINN achieves order-of-magnitude lower MSE, RMSE, and MAE versus the baseline under identical sampling and optimization.

Figure 3: Absolute error maps for the 1D advection-reaction equation demonstrate reduced high-wavenumber error along characteristic directions when using xLSTM-PINN.

Figure 4: Training loss trajectories. xLSTM-PINN reaches and sustains a lower loss floor more rapidly than baseline PINN, with faster late-phase convergence.

2D Laplace Equation with Mixed Boundary Conditions

Under mixed Dirichlet–Neumann boundary forcing, xLSTM-PINN produces accurate isocontours and error maps at the level of numerical noise ($< 4\times 10^{-4}$), while the baseline exhibits structure-bias and order-of-magnitude larger errors. All primary metrics (MSE, RMSE, MAE, MaxAE) improve by up to three orders of magnitude.

Figure 5: Solution, prediction, and absolute error for the 2D Laplace problem, showing the sharp reduction of field errors by xLSTM-PINN.

Figure 6: xLSTM-PINN demonstrates earlier and cleaner convergence on the 2D Laplace benchmark, sustaining a lower error basin throughout training.

Steady-State Heat Conduction in a Disk

On a geometry with Robin-type convective boundary conditions, xLSTM-PINN again improves contour alignment and shrinks boundary error rings to the $10^{-4}$ level, while the PINN’s spatial error remains an order of magnitude higher.

Figure 7: Steady-state heat conduction in a disk, with xLSTM-PINN eliminating large near-boundary error bands present in the PINN solution.

Figure 8: Loss curves confirm more efficient and stable convergence for xLSTM-PINN in the disk heat conduction case.

Anisotropic Poisson–Beam Equation (Fourth-Order PDE)

For a fourth-order, spectrally stiff PDE with both Dirichlet and higher-derivative conditions, xLSTM-PINN reduces global and maximum error by up to an order of magnitude and sharply attenuates high-frequency error ripples.

Figure 9: xLSTM-PINN suppresses high-frequency oscillatory error components in the anisotropic Poisson-beam case beyond the PINN baseline.

Figure 10: Loss history comparison for the high-order PDE further highlights the improved late-phase convergence and suppression of oscillations in xLSTM-PINN.

Implications and Future Directions

xLSTM-PINN demonstrates that spectral bias in physics-informed neural representation can be mitigated at the architectural level through memory gating and intra-layer recursion. This approach is orthogonal to changes in physics loss construction or AD implementation, thereby offering broad compatibility with advancements in PINN strategies, domain decomposition, and multitask physics learning. Strong numerical evidence across four representative PDE types, including challenging high-order and nontrivial boundary configurations, establishes the generality and robustness of this spectral engineering paradigm.

The methodology suggests a pathway for hybrid architectures incorporating frequency curriculum, multi-scale feature routing, or operator-based priors, especially in regimes previously limited by spectral inefficiency. The decoupling of representation dynamics from domain constraints further supports modular extension to operator learning, parametric PDE families, and extrapolative generalization.

Conclusion

By embedding xLSTM blocks with gated memory and residual micro-steps into the representation stack of standard PINNs, xLSTM-PINN achieves systematic and reproducible suppression of spectral bias, lifting high-frequency kernel weights, expanding the resolvable bandwidth, and accelerating high-frequency error convergence for PDE-constrained learning. These architectural enhancements yield consistent improvements in all major accuracy metrics across diverse PDEs, supporting both theoretical analysis and empirical benchmarks. The work substantiates memory-gated representation refinement as a viable route toward improving both the fidelity and computational efficiency of data-driven physics solvers.