Curvature-aware dynamic precision approach for physics-informed neural networks

Abstract: Physics-informed neural networks (PINNs) have become a promising framework for simulating partial differential equations (PDEs) by embedding physical laws directly into neural network training. However, recent studies show that PINN optimisation is sensitive to numerical precision. Existing implementations commonly use either single precision (FP32), which is computationally efficient but prone to failure modes, or double precision (FP64), which is robust but substantially expensive. This creates a trade-off between computational efficiency and numerical accuracy. To reduce the computational cost of double-precision training while retaining prediction accuracy, we propose a curvature-aware precision controller that adapts numerical precision during training rather than treating it as a fixed implementation choice. The proposed method reuses curvature information derived from the limited-memory BFGS (L-BFGS) optimiser to construct a precision controller, retaining FP32 when lower precision is sufficient and promoting computation to FP64 when the training dynamics indicate numerical sensitivity or precision-limited stagnation. We evaluate the proposed approach on four canonical PINN failure-mode benchmarks and an irradiance-driven ordinary differential equation example. We further test the proposed approach across different neural network architectures. The method consistently matches or even slightly exceeds full FP64 solution accuracy while reducing training time relative to full double-precision training on all benchmark equations. The obtained results indicate that precision sensitivity in PINN optimisation is phase-dependent, and that selectively applying higher precision only during numerically critical stages can lower computational cost without sacrificing predictive accuracy.

• The paper introduces a curvature-aware controller that reuses L-BFGS secant history to switch between FP32 and FP64 as optimization conditioning changes.
• Across Convection, Reaction, Wave, and Allen–Cahn benchmarks, dynamic precision matches or exceeds FP64 accuracy while reducing wall-clock time by roughly 2–10%; fixed FP32 often fails to converge.
• The approach generalizes across MLP, PINNsFormer, PINNMamba, and KAN architectures, but still depends on problem-specific curvature thresholds and limited-memory Hessian estimates.

Physics-informed neural networks (PINNs) embed PDE residuals, boundary conditions, and initial conditions into a single training objective, but their optimisation is known to be sensitive to numerical precision. Fixed FP32 training is fast yet prone to failure modes in which the loss appears to converge while the predicted solution remains inaccurate; fixed FP64 is robust but computationally expensive. This paper proposes a curvature-aware dynamic precision controller that reuses curvature information already maintained by the L-BFGS optimiser to switch between FP32 and FP64 during training, retaining double-precision accuracy at reduced wall-clock cost (2606.04736).

Motivation and positioning

The starting point is the observation that some PINN failures are attributable to premature termination of L-BFGS under insufficient precision: when function or gradient evaluations are noisy, quasi-Newton updates can be corrupted and line searches become unreliable. Prior remedies—mixed-precision training (AMP), adaptive collocation sampling, loss reweighting, or specialised architectures—either do not adapt precision to training phases or target spatial heterogeneity rather than temporal variation in optimisation difficulty. The authors' key premise is that numerical sensitivity in PINN training is phase-dependent: a model may escape a stagnation simply by promoting to higher precision at the right moment. Unlike layer-wise or static precision allocation schemes (e.g., Hessian-aware quantisation such as HAWQ), the proposed method schedules precision online based on optimisation geometry.

Curvature signal from L-BFGS history

The controller exploits quantities that L-BFGS already stores. For each iteration $k$, the secant pairs $s_k = \theta_{k+1} - \theta_k$ and $$y_k = \nabla_\theta \mathcal{L}_{k+1} - \nabla_\theta \mathcal{L}_k$$ yield a directional curvature estimate

$\kappa_k = \frac{s_k^\top y_k}{s_k^\top s_k},$

interpretable as a Rayleigh quotient of the local Hessian approximation along $s_k$. A conditioning proxy is formed as the ratio of maximum to minimum $\kappa_k$ over valid stored curvature pairs (those satisfying $s_k^\top y_k > 0$, nonzero denominator, finite value). The proxy is log-transformed and smoothed by an exponential moving average ($\alpha = 0.9$), giving $\tilde{z}_j$, whose slope $\Delta\tilde{z}_j$ captures rapid changes in conditioning.

The switching logic is deliberately simple: FP32 is used only when $s_k = \theta_{k+1} - \theta_k$0 is below a threshold $s_k = \theta_{k+1} - \theta_k$1 and $s_k = \theta_{k+1} - \theta_k$2 is small; the controller returns to FP64 when $s_k = \theta_{k+1} - \theta_k$3 grows large, rises quickly, or FP32 parameter updates become vanishingly small (maximum relative update below $s_k = \theta_{k+1} - \theta_k$4). Decisions are evaluated every 10 L-BFGS steps to prevent oscillation, and training terminates via early stopping when the total loss fails to improve by more than $s_k = \theta_{k+1} - \theta_k$5 for 50 consecutive steps after 100 warm-up steps. The overhead of the controller is negligible since it reuses existing optimiser state.

Benchmark results

Evaluation covers four canonical PINN failure-mode benchmarks—the Convection equation (speed 50), Reaction equation, Wave equation, and Allen–Cahn equation—on a $s_k = \theta_{k+1} - \theta_k$6 collocation grid, plus an irradiance-driven logistic ODE motivated by plant-growth modelling. All experiments use a single NVIDIA A100 GPU, five random seeds, and identical stopping criteria across precision strategies.

Averaged over seeds, the dynamic approach matches or slightly exceeds full FP64 accuracy on all four PDEs while reducing training time:

Equation	Dynamic rRMSE	FP64 rRMSE	FP32 rRMSE	Speedup vs FP64
Convection	0.0091	0.0108	0.8776	1.02×
Reaction	0.0531	0.0578	0.9779	1.07×
Wave	0.0094	0.0147	0.0407	1.04×
Allen–Cahn	0.0495	0.0503	0.9239	1.10×

Pure FP32 is substantially faster (up to ~70× on Allen–Cahn) but frequently fails to converge to an acceptable solution, producing errors one to two orders of magnitude larger. The speedups from dynamic precision are modest—roughly 2–10% wall-clock reduction—which is an honest limitation of the headline claim: the method preserves FP64-level robustness at somewhat lower cost, rather than approaching FP32-class speed. Notably, the dynamic runs sometimes achieve lower error than pure FP64 despite using the same model class and loss. The authors attribute this to precision switching acting as a numerical perturbation that alters the L-BFGS trajectory near convergence, allowing termination at a different point.

Architecture generality

The controller is architecture-agnostic and was tested on vanilla MLP PINNs, PINNsFormer, PINNMamba, and KAN. A striking finding is that the advanced architectures, trained purely in FP32, exhibit inconsistent convergence across seeds—for example, PINNMamba achieves 0% convergence rate (rRMSE < 0.3) on both Convection and Allen–Cahn, and KAN fails entirely on Convection—whereas the dynamic-precision MLP converges reliably (100%) on all four equations. This suggests that architectural improvements alone do not eliminate precision-related instability and can add substantial computational cost (e.g., PINNsFormer requires ~3,360 s versus ~1,283 s for the dynamic MLP on the Wave equation).

Applying the controller to PINNsFormer and PINNMamba yields consistent speedups (1.03×–1.33×) with comparable accuracy to their FP64 counterparts. Interestingly, for these architectures the gains appear partly in step efficiency, not only per-iteration cost; the authors hypothesise that better-conditioned architectures produce smoother curvature trajectories, which the controller exploits more effectively. A wider, shallower three-layer MLP also shows larger improvements (up to 1.28× on Wave), indicating that network shape influences the precision sensitivity of training through its effect on loss-landscape geometry.

Phase-dependent precision sensitivity

Trajectory analysis supports the central thesis. The smoothed curvature proxy follows non-monotonic paths across all benchmarks, and visualised training segments show that FP32 can be faster than FP64 in one phase (low, smooth curvature) yet slower in another (elevated or fluctuating curvature), even within a single run. The controller's behaviour aligns with this: Convection and Allen–Cahn, which exhibit higher and more fluctuating curvature signals, trigger more frequent switching and longer FP64 residence, whereas Reaction and Wave remain in FP32 longer and show larger speed–accuracy improvements. The irradiance ODE, with its smoother low-dimensional dynamics, shows the same qualitative pattern with smaller error variance under dynamic precision than fixed FP32, which occasionally stalls near convergence even in this benign setting.

Limitations and open questions

The paper is candid about several constraints. The curvature signal derives from a limited-memory secant approximation along recent update directions, which may not capture the full local optimisation geometry in highly nonconvex regimes. The controller uses only curvature information, ignoring other determinants of reduced-precision stability such as gradient magnitudes, tensor dynamic range, and underflow/overflow effects. Most practically, the switching threshold $s_k = \theta_{k+1} - \theta_k$7 is treated as a problem-dependent hyperparameter tuned per equation—an ablation across thresholds (2.0, 2.5, 3.0) shows results are fairly stable but not threshold-free—and the condition number itself varies across PDEs and architectures, so no universal signal is currently offered. Open questions include extending the approach to other second-order optimisers used for PINNs (self-scaled BFGS, Broyden-type methods), incorporating richer numerical signals into the controller, moving beyond binary FP32/FP64 switching to multi-level precision, and testing whether similar mechanisms transfer to other numerically sensitive scientific machine learning settings.

Conclusion

This work reframes numerical precision in PINN training as a controllable, phase-dependent quantity rather than a fixed implementation choice. By reusing L-BFGS curvature history as a lightweight conditioning proxy, the proposed controller recovers FP64-level accuracy across four standard failure-mode benchmarks and an applied ODE problem at modestly reduced cost, avoids the convergence failures of fixed FP32, and generalises across architectures without modification. The empirical evidence that precision sensitivity tracks curvature regimes provides a concrete link between loss-landscape conditioning and arithmetic requirements in PINN optimisation, though the dependence on a per-problem threshold and the limited-memory nature of the curvature signal leave room for more principled, universally applicable controllers.