Towards a Physics Foundation Model (2509.13805v1)

Abstract: Foundation models have revolutionized natural language processing through a ``train once, deploy anywhere'' paradigm, where a single pre-trained model adapts to countless downstream tasks without retraining. Access to a Physics Foundation Model (PFM) would be transformative -- democratizing access to high-fidelity simulations, accelerating scientific discovery, and eliminating the need for specialized solver development. Yet current physics-aware machine learning approaches remain fundamentally limited to single, narrow domains and require retraining for each new system. We present the General Physics Transformer (GPhyT), trained on 1.8 TB of diverse simulation data, that demonstrates foundation model capabilities are achievable for physics. Our key insight is that transformers can learn to infer governing dynamics from context, enabling a single model to simulate fluid-solid interactions, shock waves, thermal convection, and multi-phase dynamics without being told the underlying equations. GPhyT achieves three critical breakthroughs: (1) superior performance across multiple physics domains, outperforming specialized architectures by up to 29x, (2) zero-shot generalization to entirely unseen physical systems through in-context learning, and (3) stable long-term predictions through 50-timestep rollouts. By establishing that a single model can learn generalizable physical principles from data alone, this work opens the path toward a universal PFM that could transform computational science and engineering.

• The paper introduces GP_hyT, a hybrid transformer-numerical integrator that learns generalizable physics principles from extensive simulation data.
• The model achieves superior parameter efficiency and accuracy with up to 29× MSE reduction compared to FNO and UNet on diverse physical systems.
• The work demonstrates zero-shot in-context learning that enables accurate predictions on novel boundary conditions and unseen physical phenomena.

Towards a Physics Foundation Model: The General Physics Transformer

Introduction

The paper "Towards a Physics Foundation Model" (2509.13805) introduces the General Physics Transformer (GP\textsubscript{hy}T), a transformer-based architecture trained on 1.8 TB of diverse simulation data spanning fluid dynamics, heat transfer, and multi-phase systems. The central thesis is that a single, large-scale transformer can learn generalizable physical principles from data alone, enabling zero-shot generalization to unseen physical systems and boundary conditions via in-context learning. This work addresses the limitations of existing physics-aware machine learning models, which are typically restricted to narrow domains and require retraining for each new system, and demonstrates that foundation model capabilities are achievable for physics.

Model Architecture

GP\textsubscript{hy}T is a hybrid architecture integrating a transformer-based neural differentiator with a classical numerical integrator. The neural differentiator predicts the time derivative of the physical state, $\frac{\partial X}{\partial t}$, where $X$ comprises multiple physical fields (e.g., pressure, temperature, velocity). The input consists of a stack of prior time snapshots, tokenized into non-overlapping spatiotemporal patches (tubelets), with absolute positional encodings. The transformer applies unified spatiotemporal attention, eschewing factorized approaches to maximize expressivity and capture non-separable phenomena such as turbulence and shockwave interactions.

Figure 1: General architecture of GP\textsubscript{hy}T, showing the neural differentiator and numerical integrator pipeline.

To enhance local information, first-order spatial and temporal derivatives are computed via central differences and concatenated with the input fields. The output of the neural differentiator is advanced in time using a numerical integrator, with the Forward Euler method selected for computational efficiency. Ablation studies confirm that higher-order integrators (e.g., RK4) do not yield significant accuracy improvements in this framework.

Training Data and Augmentation

The training corpus comprises over 2.4 million simulation snapshots from eight datasets, including incompressible and compressible flows, thermal convection, flows around obstacles, heated flows, and multi-phase dynamics. Data augmentation strategies include variable time increments (forcing the model to infer temporal scale from context) and per-dataset normalization (preserving relative physical quantities while requiring inference of absolute magnitudes and spatial scales). This design explicitly encourages in-context learning rather than memorization of dataset-specific patterns.

Figure 2: Physical domains and boundary conditions for the-well datasets, illustrating the diversity of simulated systems.

Multi-Physics Learning Performance

GP\textsubscript{hy}T is benchmarked against Fourier Neural Operator (FNO) and UNet architectures, trained identically for single-step prediction across all test sets. GP\textsubscript{hy}T achieves a 5× reduction in median MSE compared to UNet and a 29× reduction compared to FNO at equivalent model sizes. Even the smallest GP\textsubscript{hy}T variant (10% of the parameters) outperforms full-sized FNO and UNet in median MSE, indicating superior parameter efficiency and cross-scale interaction modeling.

Figure 3: Average and median MSE across all test sets for each model; GP\textsubscript{hy}T demonstrates substantial performance gains.

Qualitative results show that GP\textsubscript{hy}T maintains sharp shock fronts and accurately tracks convective plumes in complex systems, outperforming FNO (which produces overly smoothed predictions) and UNet (which diffuses sharp features).

Figure 4: Next timestep prediction (velocity magnitude) of GP\textsubscript{hy}T, UNet, and FNO against ground truth for representative systems.

Zero-Shot In-Context Generalization

A defining feature of foundation models is zero-shot generalization via in-context learning. GP\textsubscript{hy}T is evaluated on systems with boundary conditions and physical phenomena absent from the training data, including open boundaries, supersonic flows, and turbulent radiative layers. The model infers new boundary conditions from the prompt, achieving MSE nearly identical to known systems. For entirely novel physics, while MSE increases, GP\textsubscript{hy}T still produces physically plausible predictions, capturing essential dynamics such as bow shock formation and turbulent mixing.

Figure 5: Qualitative results for in-context learning; GP\textsubscript{hy}T adapts to unseen boundary conditions and novel physics.

Long-Range Rollout Stability

Autoregressive rollouts over 50 timesteps test the model's ability to maintain physical consistency and stability. GP\textsubscript{hy}T-XL preserves global dynamics and boundary interactions, with error accumulation manifesting primarily as loss of high-frequency detail. Error growth is near-linear for most systems, with modified boundary conditions tracking their training counterparts closely. For novel physics, error growth rates are higher, underscoring the challenge of long-term stability in foundation models.

Figure 6: Autoregressive rollout prediction for velocity magnitude of GP\textsubscript{hy}T-XL on Euler shockwaves; error accumulation and stability are visualized.

Ablation Studies and Hyperparameter Analysis

Prompt length and temporal patch size are critical for in-context learning. Increasing the number of input timesteps yields log-linear performance improvements, with diminishing returns beyond two timesteps. Larger temporal patch sizes allow compression of temporal information, reducing computational cost with minimal accuracy loss.

Figure 7: Average MSE over the test set for varying input timesteps and temporal patch sizes, illustrating the trade-off between context and efficiency.

Integrator ablation confirms that the differentiator-integrator framework is superior to direct next-step prediction, with minimal difference between explicit integrator schemes.

Figure 8: Median MSE for different integration strategies; the differentiator-integrator approach is clearly superior.

Detailed Predictions Across Systems

GP\textsubscript{hy}T produces high-fidelity next-step predictions for a wide range of physical systems, including Euler shocks, incompressible flows, Rayleigh–Bénard convection, two-phase flows, heated flows, and shear flows. For novel systems, the model extrapolates to physically plausible states, even in the absence of training data for those regimes.

Figure 9: Next step predictions and ground truth for the Euler dataset; GP\textsubscript{hy}T maintains sharp discontinuities and accurate field evolution.

Figure 10: Next step predictions for incompressible flow with obstacles; pressure and velocity fields are accurately reproduced.

Figure 11: Next step predictions for Rayleigh–Bénard convection; density, pressure, and velocity fields are well captured.

Figure 12: Next step predictions for two-phase flow; density and pressure fields are consistent with ground truth.

Figure 13: Next step predictions for heated flow; temperature and velocity fields are accurately modeled.

Figure 14: Next step predictions for shear flow; pressure and velocity magnitude are preserved.

Limitations and Future Directions

GP\textsubscript{hy}T is currently restricted to 2D systems and fluid/thermal domains. Extension to 3D, incorporation of additional physical domains (mechanics, chemistry, molecular dynamics, optics), and variable-resolution capabilities are necessary for a comprehensive Physics Foundation Model. Long-term stability remains a challenge, with error accumulation limiting practical engineering applications. Scaling model capacity and training data diversity is expected to yield further emergent capabilities.

Conclusion

The General Physics Transformer establishes that transformer-based architectures can serve as foundation models for physics, learning generalizable principles from diverse simulation data and adapting to novel tasks via in-context learning. GP\textsubscript{hy}T outperforms specialized models in both accuracy and generalization, demonstrating the feasibility of a "train once, deploy anywhere" paradigm for physics simulation. The implications are substantial: democratized access to high-fidelity simulations, accelerated scientific discovery, and reduced reliance on specialized solver development. Future work should address extension to broader physical domains, improved long-term stability, and scalable deployment strategies, paving the way for universal physics engines in computational science and engineering.