- The paper introduces a bias-aware benchmark that evaluates physics foundation models across eight PDE regimes, revealing significant performance variances.
- The study finds that pretraining and model scaling do not guarantee uniform improvements, with some architectures experiencing negative transfer.
- The results emphasize the need for architecture-specific design and further research into transferring explicit physical knowledge for robust generalization.
Conditional Generalization of Physics Foundation Models: A Bias-Aware Analysis
Problem Context and Motivation
Physics foundation models, such as operator transformers and neural PDE solvers, are increasingly proposed as general solutions to spatiotemporal forecasting and scientific simulation, extending the foundational paradigm from language/vision to the physical sciences. The motivating ambition is a reusable model with generalizable physical knowledge, capable of high accuracy across diverse dynamics, scales, and regimes. However, standard evaluation protocols often collapse model performance into a single average score under a fixed training distribution, which obscures conditional capabilities and potential biases.
This paper introduces a comprehensive, bias-aware benchmark for quantifying the conditional generalization of state-of-the-art physics foundation models. The benchmark spans eight PDE regimes, three training distributions, and 25 test regimes arranged over dynamic-scale and initial-condition complexity. It enables factorized diagnosis of both in-distribution and out-of-distribution (OOD) generalization, highlighting how model architecture, model variant (from-scratch vs. various pretrained sizes), and common improvement strategies impact performance across the full test grid.
Benchmark and Evaluation Protocol
The evaluation protocol is designed to explicitly uncover systematic biases and generalization boundaries. Each model is assessed along six major axes:
- Physical dynamics (8 PDE classes): including reaction-diffusion, fluid-like, chaotic, and wave-like systems.
- Training distribution: three mixtures, ranging from simple, balanced, to complex regime emphasis.
- Distribution shifts: a 5×5 grid over dynamic-scale and initial-condition complexity, allowing decomposition into compositional, dynamic-OOD, IC-OOD, and joint-OOD shifts.
- Prediction horizon: both short-term (in-horizon) and longer-term (OOD rollout) forecasting.
- Model variant: scratch and pretrained (small, medium, large), for all five architectures (DPOT, GPhyT, MORPH, MPP, Poseidon).
- Model size: three finetuned sizes per architecture.
Overall, the study aggregates 60,000 quantitative assessments, using relative L2 error and derived robustness/diagnostic metrics. Notably, the design ensures that every model is evaluated over the same full span of regimes and shifts, permitting direct comparison of both overall accuracy and fine-grained robustness.
Key Findings
Systematic Physical-Regime and Temporal Bias
Across all models and variants, the results demonstrate that physics foundation models are conditional generalists. Their capabilities strongly depend on the particular PDE regime, temporal scale, initial-condition complexity, and architecture. Even in train-seen conditions with balanced data, error can vary by up to two orders of magnitude between PDEs; e.g., Fisher-KPP and Gray-Scott are consistently easiest, while Kolmogorov, Kuramoto-Sivashinsky, Decay, and Wave regimes drive the largest errors. This pattern is not eliminated by pretraining or scaling—architecture-dependent inductive biases persist, controlling relative strengths and weaknesses.
Temporal stability is also unequally distributed. Early-frame accuracy does not imply rollout stability; some models rapidly amplify initial errors, and temporal robustness can be decoupled from short-term accuracy. The structure of these amplification patterns is both PDE-dependent and architecture-dependent.
Conditional Robustness to Distribution Shift
Out-of-distribution robustness is governed by the direction and magnitude of the physical shift. Dynamic-scale and joint dynamic/initial-condition shifts incur the largest relative error amplifications (mean ShiftDamage up to 6–8× the train-seen reference, in some cases). Conversely, many IC-only shifts have a lesser or even negative robustness gap. Importantly, moving to a more complex or balanced training distribution increases raw accuracy in select regions, particularly large-dynamic regimes, but does not close the normalized robustness gap. Gains on train-seen regimes do not translate equivalently to OOD cells, and in some cases, more complex training increases OOD ShiftDamage.
Limitations of Pretraining and Scaling
Neither pretraining nor parameter count guarantees uniform improvement. For matched-size comparisons, 37.5% of architecture–PDE pairs show negative transfer from pretraining, and inverse scaling is frequent (Larger models perform worse than S-baseline in 25% of cases). Scaling and pretraining tend to reinforce architecture-specific regime preferences rather than yielding universally transferable gains.
Architecture-Specific Failure Modes
Each model architecture exposes unique combinations of weaknesses: DPOT is most sensitive to distribution shift; MORPH is dominated by rollout and scaling instability; MPP suffers from high rates of negative pretraining transfer. Poseidon and GPhyT display more balanced, but still uneven, failure profiles. These qualitative differences in failure mechanisms underscore that model choice should be made with respect to the target regime, rather than solely on average benchmarking scores.
Implications and Outlook
Practical Usage
The conditional performance landscape has direct practical implications. Model selection for physical forecasting and surrogate modeling must consider the target PDE family, dynamic/IC regime, and relevant robustness requirements—one-size-fits-all is empirically unjustified. A sophisticated deployment should utilize a multiaxial capability profile rather than a single error metric.
Theoretical Progression
The results challenge the sufficiency of larger data, more pretraining, or model scaling as levers for robustly generalizable physics. The persistence of organized robustness gaps, even with complex training and high parameter counts, suggests that core limitations reside in the models' ability to represent and transfer physical structure across regimes, rather than mere capacity or coverage. Further theoretical understanding is required to design mechanisms for explicit physical knowledge transfer and context recognition.
Future Directions
To realize the foundational promise in scientific machine learning, research must move beyond brute-force scaling. Promising directions include:
- Inductive architectural innovations, potentially tailored to encode transferable physical symmetries and invariants across regimes.
- Systematic analysis of internal model representations, to elucidate what physical features are captured, missed, or overly biased by training biases.
- Meta-learning approaches that enable regime inference and adaptive representation selection from limited context.
- Better benchmarks and diagnostic metrics, aligned with the compositional and extrapolative structure of physical generalization.
Conclusion
This benchmark-driven study shows that the current generation of physics foundation models is not universally generalizable across physical regimes, scales, and distribution shifts. Conditional generalization and architecture-specific failure fingerprints dominate practical capability. Research progress will require advances in the representation and transfer of physical knowledge, not just expansion of data or model size.
Reference: "Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts" (2605.29283)