Physics-Informed Multimodal Foundation Model

• PI-MFM is a framework that integrates multimodal data encoding with physics-aware objectives to create universal surrogates for complex physical systems.
• The architecture leverages dual spatial-spectral tokenization, cross-modal fusion with FiLM conditioning, and state-space backbones for efficient PDE operator learning.
• Combined training objectives—including PDE residuals, boundary conditions, and data-fit losses—yield robust zero-shot transfer and heightened accuracy in sparse or noisy data regimes.

A Physics-Informed Multimodal Foundation Model (PI-MFM) is an architectural and training paradigm for universal surrogates of physical systems, emphasizing both multimodal data integration and explicit enforcement of governing physical laws during pretraining and adaptation. PI-MFM frameworks generalize classical data-driven operator-learning by incorporating physics-aware objectives and specialized token fusion mechanisms. Recent instantiations, particularly PDE-FM and related models, target scalable and data-efficient modeling of heterogeneous partial differential equation (PDE) domains, enabling robust transfer and zero-shot generalization, especially in regimes with sparse or noisy supervision (Zhu et al., 28 Dec 2025, Soares et al., 26 Nov 2025).

1. Architectural Components and Input Encoding

PI-MFM architectures combine modular multimodal encoding with symbolic physics integration. The backbone typically consists of:

• Dual Spatial–Spectral Tokenization: Input fields $x_i \in \mathbb{R}^{C_i \times H \times W}$ are normalized via $1 \times 1$ convolution ($A_i^{\mathrm{in}}$) to a shared latent space. Spatial patches ($p_s \times p_s$) are encoded by shallow ConvNets and projected to tokens $T_{\mathrm{spatial}}$. Spectral content is captured by truncated FFT per channel ($T_{\mathrm{spectral}}$), stacking global low-frequency information; mathematically, FFT truncation is $$U(k) = \int_{\Omega} u(x)e^{-2\pi i k \cdot x}\;dx, \;\; |k_x| \leq m_x$$.
• Physics-Aware Conditioning: Boundary condition and physical metadata $c \in \mathbb{R}^p$ are injected via FiLM—scalars output by small MLPs $\gamma(c), \beta(c)$ modulate each patch token: $$\tilde{T}_{\mathrm{spatial}} = T_{\mathrm{spatial}} \odot (1 + \gamma(c)) + \beta(c)$$.
• Cross-Modal Fusion and Dual Encoder: Spatial and spectral streams are encoded separately (ConvNeXt blocks, MLPs) and fused through bidirectional cross-attention.
• State-Space Backbone (Mamba): Tokens traverse $L$ layers of a linear-time state-space model (MambaLayer), expressed as discretized linear ODEs ($s' = As + Bx; y = Cs + Dx$), achieving $\mathcal{O}(N_p d)$ compute per layer (Soares et al., 26 Nov 2025).
• Operator-Theoretic Decoder (FNO Head): Final tokens are reshaped and upsampled onto the original grid, with a Fourier Neural Operator layer projecting to the target physical field $\hat{u}(x)$, ensuring smoothness and global coherence.
• Symbolic PDE Encoding (for PINN-Style PI-MFM): PDEs are encoded as prefix-notation token sequences, e.g., add u_t mul q u_x, parsed into trees for loss assembly. Symbol and data streams are cross-attended before decoding (Zhu et al., 28 Dec 2025).

2. Physics-Informed Training Objectives and Loss Construction

PI-MFM leverages multiline objective functions combining both data and physics prior terms:

• PDE Residual Loss: At collocation points $(t_i,x_i)$, residuals are computed by applying the parsed PDE tree to model predictions $\hat{u}(t,x)$: $$L_{\mathrm{phys}} = \frac{1}{N_c} \sum_{i=1}^{N_c} |\mathcal{R}[\hat{u}](t_i, x_i)|^2$$.
• Initial/Boundary Condition Losses: Enforce correct field values and derivatives at initial/boundary points, e.g., $$L_{\mathrm{IC}} = \frac{1}{N_{\mathrm{ic}}}\sum_j |\hat{u}(0, x_j) - u_0(x_j)|^2$$.
• Data-Fit Loss: Where solution samples are available, standard $L^2$ loss: $$L_{\mathrm{data}} = \frac{1}{N_d}\sum_{k=1}^{N_d} |\hat{u}(t_k, x_k) - u_{\mathrm{true}}(t_k, x_k)|^2$$.
• Total Training Objective: Linear combination with weights $$\omega_{\mathrm{phys}}, \omega_{\mathrm{IC}}, \omega_{\mathrm{data}}, \omega_{\mathrm{IC'}}$$, potentially cosine-annealed; in practice, $\omega_{\mathrm{data}}$ is set to 1, others typically in the range 1–10.

Multiple recent works employ hybrid spatial–spectral loss, adding frequency-weighted penalties to enhance high-frequency fidelity (Soares et al., 26 Nov 2025). Physics invariants (conservation penalties), e.g., mass or energy, are optionally included.

3. Derivative Computation and Physics Loss Assembly

Automatic differentiation (AD) and finite-difference (FDM) schemes underpin loss construction:

• AD Strategy: Vectorized JVPs produce all time/space derivatives required by symbolic loss trees. Forward-mode AD scales with coordinate/derivative orders ($$\mathcal{O}((d_{\mathrm{dir}})(d_{\mathrm{order}}))$$); reverse-mode AD is prohibitive for batchwise collocation evaluation.
• FDM Strategy: Forward passes at shifted collocation points yield staggered grids; central difference stencils reconstruct required $\partial u/\partial t$ and higher derivatives.
• Tradeoffs: AD is hyperparameter-free, but memory-intensive; FDM demands explicit step-size $\Delta$ tuning, with truncation ($E_{\mathrm{trunc}}\sim\mathcal{O}(\Delta^2)$), and round-off ($$E_{\mathrm{round}}\sim\mathcal{O}(\epsilon/\Delta^2)$$) errors. Empirically, FDM (float32) and AD (float16) achieve comparable error rates ($\sim1.04\%$ vs. $1.02\%$ $L^2$ relative error) but differ by 2× in runtime (Zhu et al., 28 Dec 2025).
• Loss Assembly Pseudocode:

Input: batch {(u0_b, s_b)}_{b=1}^B, collocation points TX = {(t_m,x_m)}_{m=1}^M 1. U = G_θ(u0_b, s_b)(t_m, x_m) ∈ ℝ^{B×M} 2. For derivative types d ∈ D: Compute U_d (AD or FDM) 3. Parse s_b → expression tree T_b 4. Evaluate T_b on computed [U, U_{u_t}, U_{u_x}, …] → residuals 5. Loss = mean_b,m (residual[m])^2

(Zhu et al., 28 Dec 2025)

4. Pretraining, Transfer, and Adaptation

PI-MFM pretraining employs diverse, high-resolution physics datasets, typically spanning domains such as hydrodynamics (Navier–Stokes), reaction–diffusion, radiative turbulence, viscoelasticity, linear acoustics, and astrophysics (MHD). Datasets such as The Well include both 2D and 3D PDE regimes (Soares et al., 26 Nov 2025):

• Sampling Strategies: Batch sampling is proportional to $( |\mathcal{D}_i| )^{0.5}$ to balance dataset sizes. Training utilizes small and large batch/epoch configurations for ablation and full-scale comparison.
• Domain Adaptation: Transfer to new physics domains involves only learning new input/output adapters ($1 \times 1$ convolutions) and adjusting FiLM metadata injections. The backbone and tokenizer remain frozen, supporting rapid and architectural-invariant adaptation (Soares et al., 26 Nov 2025).
• Zero-Shot Physics-Informed Fine-Tuning: Pretrained models can be adapted to entirely new PDE families or regimes using only physics residual and initial/boundary loss terms ($$L_{\mathrm{phys}} + L_{\mathrm{IC}} + L_{\mathrm{IC'}}$$). Empirically, zero-shot adaptation achieves $\sim1\%$ $L^2$ error in 3k gradient steps, outperforming physics-only training from scratch (Zhu et al., 28 Dec 2025).

5. Experimental Benchmarks and Quantitative Evaluation

Empirical evaluation encompasses sparse, partial, and noisy-label regimes, as well as cross-domain robustness:

• Sparse-Label Supervision: PI-MFM reduces test errors from $\sim20\%$ ($L^2$, data-only at $8 \times 32$ grid) to $<5\%$, with largest gains at lowest resolution (Zhu et al., 28 Dec 2025). Data-efficient function pair learning demonstrates <1% error with $\sim2000$ labeled samples vs. >20,000 for data-only.
• Physical Consistency: Weighted spectral losses enhance high-frequency accuracy. Conservation constraint terms (mass, energy) further regularize solutions.
• State-of-the-Art Comparison Table (Soares et al., 26 Nov 2025):

| Dataset | FNO | TFNO | CNextU-net | PhysiX | PI-MFM | |-------------------------------|---------|---------|------------|--------------|----------| | Shear Flow | 1.1890 | 1.4720 | 0.8080 | 0.0700 | 0.0345 | | Rayleigh–Bénard | 0.8395 | 0.6566 | 0.6699 | 0.1470 | 0.0415 | | Turbulence Gravity Cooling | 0.2429 | 0.2673 | 0.2096 | — | 0.0796 | | Acoustic Scattering | 0.5062 | 0.5057 | 0.0153 | 0.0960 | 0.0487 | | Viscoelastic Instability | 0.7212 | 0.7102 | 0.2499 | 0.2370 | 0.5204 |

Ablation studies confirm performance gains from Mamba backbone, spectral tokenization, cross-attention, and FiLM conditioning (Soares et al., 26 Nov 2025). Primary failure mode remains in viscoelastic turbulence, suggesting a lack of inductive bias or explicit memory mechanisms.

6. Limitations, Trade-Offs, and Future Directions

Key limitations of PI-MFM include:

• Computational Cost: AD yields high memory/runtime demand; FDM requires step-size tuning. For high-dimensional or large-domain problems, physics residual evaluation overhead remains significant (Zhu et al., 28 Dec 2025, Soares et al., 26 Nov 2025).
• Domain Coverage: Most experiments focus on 1D time-dependent or periodic 2D/3D PDEs; extension to nonperiodic, complex boundary geometries remains an open challenge.
• Physics Representation: Integration of nonlocal, delayed, or elastic behaviors (e.g., viscoelastic turbulence) is suboptimal without specialized architectural components. Purely convolutional surrogates sometimes outperform in stationary or highly stiff regimes.

Future research directions include adaptive collocation (dynamic point resampling), higher-order gradient regularization, curriculum-based multi-physics pretraining, meta-learning of physics weights, and hybridization with traditional solvers (finite-element or mesh-based integration). Uncertainty quantification, federated/distributed training, and retrieval-augmented physics modules also present viable paths forward (Zhu et al., 28 Dec 2025, Soares et al., 26 Nov 2025, Farhadloo et al., 20 Feb 2025).

7. Theoretical Context and Extensions

PI-MFM generalizes the concept of Physics-Guided Foundation Models (PGFM) by fusing large-scale multimodal pretraining, physics-constrained loss regularization, and physics-aware architectural biases (Farhadloo et al., 20 Feb 2025). The use of symbolic PDE encoding and automatic physics-loss assembly builds on techniques from physics-informed neural networks (PINNs) and DeepONets but extends them to the multimodal, foundation-model scale. Product-of-Experts fusion for unsupervised disentanglement, as utilized in PIMA (Trask et al., 2022), can serve as a template for scalable embedding of scientific fingerprints.

A plausible implication is that PI-MFM models, by leveraging physics loss as regularizer and transfer enabler, mark a significant step toward universal, robust, and data-efficient multi-operator solvers for scientific discovery and simulation. However, practical deployment at full scale will require new advances in physics-token integration, operator decoding, and adaptive training across multi-domain, multi-modal regimes.