Value of physical inductive biases at scale

Determine whether physical inductive biases for neural surrogates remain valuable at sufficient scale or are subsumed by model capacity and data diversity.

Background

The thesis compares neural architectures that encode different physical priors, including locality in convolutional networks, periodicity in Fourier Neural Operators, and resolution hierarchies in UNets. Vanilla transformers introduce substantially less physical structure and instead rely primarily on model capacity and data.

The unresolved issue is whether explicitly encoding physical inductive biases continues to improve neural PDE surrogates as models become larger and are trained on more diverse data. This question is relevant to the design of scientific foundation models.

References

To date, it remains open to what extend the new probabilistic models reproduce the cross-scale energy transfer and the butterfly effect.

— Butterfly Effect and the Kinetic Energy Cascade in Probabilistic Machine Learning Weather Prediction Models  (2609.18489 - Chen et al., 16 Sep 2026) in Section 2.2, subsection “Upscale energy cascade and kinetic energy spectrum”

Whether physical inductive biases for neural surrogates remain valuable at sufficient scale, or whether they are subsumed by sheer model capacity and data diversity, is an open question.

— From Numerical Simulators of PDEs to Neural Emulators and Back  (2608.24547 - Koehler, 25 Aug 2026) in Section 3, subsection “Transformer-based Architecture”

Understanding what types of structures aid OOD generalization and, critically, how to enforce such structure in SciML models is a significant open problem.

— Into the danger zone: stable extrapolation in high-dimensional function and operator learning  (2609.36709 - Adcock et al., 29 Sep 2026) in Section 7, subsection “Structure beyond regularity”

The trade-off between bias and capacity is one of the more interesting open questions of current model design, and it intersects directly with the discussion of how to test models (Sec.~\ref{sec:assess}): emergent physical constraints, when they appear, may behave differently from hard-coded ones in out-of-distribution regimes, and require their own validation.

— A strategic roadmap for an atomistic machine-learning ecosystem  (2609.39090 - Behler et al., 30 Sep 2026) in Section 3.2, “Model architectures”

A natural next step is to test whether these patterns persist in spatially extended systems and for state-of-the-art architectures of current emulators.

— A Response Theory Probe for Learned Stochastic AI Simulators, Tested on Lorenz-63  (2610.06798 - Böger et al., 5 Oct 2026) in Discussion