Physics-Informed Transformer Models

• Physics-Informed Transformer is a family of models that incorporate physical laws and constraints into transformer architectures for enhanced representation learning.
• These models embed physics through methods like symbolic equation conditioning, physics-residual optimization, differentiable decoders, and attention bias using domain priors.
• They demonstrate practical success in fields from PDE solving and inverse design to climate forecasting and robotics, while addressing computational complexities.

Physics-Informed Transformer denotes a family of transformer-based models in which physical laws, mechanistic simulators, or domain priors are built into representation learning, attention, decoding, or optimization. In the recent literature, this label covers markedly different constructions: transformers that ingest tokenized governing equations, transformer neural operators trained only with PDE residuals, encoder–decoder architectures regularized by rigid-body or contact dynamics, variational models whose decoder is a differentiable radiative-transfer model, and attention mechanisms biased by physical priors such as teleconnections or phase synchrony. The unifying feature is not a single architecture, but the deliberate coupling of transformer sequence or token processing with physically meaningful constraints or variables (Lorsung et al., 2023, Mensah et al., 13 Nov 2025, Boya et al., 2024).

1. Scope and definitions

The term is used broadly rather than uniformly. In some formulations, physics appears as an explicit input representation. PITT, for example, tokenizes the governing PDE, its parameters, boundary conditions, and target time, and conditions a neural-operator update on that symbolic representation; the paper is explicit that this is a physics-aware architecture and input design rather than a model with an explicit PDE residual loss (Lorsung et al., 2023). In other formulations, physics appears as the optimization criterion itself. PINNsFormer, PhysicsFormer, PINTO, PITO, and PIITO are all trained by losses built from PDE residuals together with initial or boundary constraints, although their architectural choices differ substantially (Zhao et al., 2023, Barman et al., 7 Jan 2026, Boya et al., 2024, Guo, 27 Jan 2026).

A second major usage of the term refers to physically structured latent variables or decoders. In the Sentinel-2 inversion model, the Transformer-VAE encoder predicts distributions over PROSAIL parameters, and PROSAIL itself is used as the fixed, differentiable decoder, so the latent variables are actual canopy and leaf parameters rather than arbitrary codes (Mensah et al., 13 Nov 2025). A related but distinct usage appears in neural quantum states, where a Transformer models corrections to a physically motivated reference state in a physics-informed basis, improving interpretability and sampling efficiency without using a PDE-style residual (Sobral et al., 2024).

A third usage concerns attention priors. Pi-Transformer introduces a data-driven series attention together with a smoothly evolving prior attention encoding scale-related self-similarity and phase synchrony, while TelePiT modifies attention logits with a teleconnection bias derived from learnable global climate patterns (Maleki et al., 24 Sep 2025, Lyu et al., 8 Jun 2025). In such cases, the model is “physics-informed” because the attention mechanism itself is constrained or biased by physically motivated structure.

Formulation	Representative paper	Physics enters through
Symbolic equation conditioning	PITT (Lorsung et al., 2023)	PDE tokens, parameters, BCs, target time
Differentiable physical decoder	PROSAIL Transformer-VAE (Mensah et al., 13 Nov 2025)	Fixed PROSAIL decoder, bounded latent parameters
Physics-only operator learning	PINTO (Boya et al., 2024)	PDE residual and initial/boundary losses
Residual-trained turbulence operator	PITO/PIITO (Guo, 27 Jan 2026)	LES equations in the loss
Prior- or bias-modulated attention	Pi-Transformer (Maleki et al., 24 Sep 2025)	Smooth prior attention and mismatch scoring

A common misconception is that all physics-informed transformers are simply PINNs with attention. The literature contradicts this. Some models are classical residual-based PINN variants, some are simulation-only inverse models with differentiable forward physics, some are operator learners trained without simulation labels, and some are attention models whose “physics” is encoded as priors rather than as residual minimization.

2. Recurrent architectural patterns

Several architectural motifs recur across the literature. One is the conversion of pointwise spatiotemporal inputs into short pseudo-sequences so that self-attention can model temporal dependence directly. PINNsFormer introduces a Pseudo Sequence Generator that maps a single input $[\mathbf{x}, t]$ to $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$, and then uses a Transformer encoder–decoder in which decoder self-attention is omitted and encoder–decoder attention carries the temporal context (Zhao et al., 2023). PhysicsFormer adopts the same general pseudo-sequential idea but emphasizes a lighter encoder–decoder with multi-head cross-attention, a high-dimensional projection layer, and a weighted-sine activation (Barman et al., 7 Jan 2026).

A second motif is query-conditioned cross-attention as a neural-operator kernel. PINTO encodes a domain query point and repeatedly updates it through cross-attention against encoded initial or boundary points and values. Its cross-attention units are presented as iterative kernel integral operators that transform query points into boundary-aware representations, allowing one trained model to generalize to unseen initial and boundary conditions (Boya et al., 2024). This operator viewpoint is echoed in PITO, where self-attention over 3D volumetric patches is interpreted as a discretized integral operator for the evolution of filtered turbulence fields (Guo, 27 Jan 2026).

A third motif is patch or token design matched to physical geometry. PITO partitions a $3\mathrm{D}$ flow domain into non-overlapping cubic patches of size $P \times P \times P$, while the topology-optimization transformer patchifies a $64\times 64$ two-channel image of strain energy density and von Mises stress and appends a conditioning token containing volume fraction, load coordinates, load vector components, and boundary condition specification (Guo, 27 Jan 2026, Lutheran et al., 4 Apr 2026). In climate forecasting, TelePiT uses spherical harmonic embedding and zonal tokenization so that the transformer operates on a geometry-aware representation of global atmospheric variables (Lyu et al., 8 Jun 2025).

Hybridization is also common. HMT-PF is explicitly not a pure transformer: it uses a Transformer-like spatial encoder with Galerkin self-attention and a Galerkin cross-attention decoder, but replaces a standard temporal transformer with a Mamba latent propagation module (Du et al., 16 May 2025). TE-PINN uses a transformer encoder over IMU sequences, but couples it with quaternion normalization, RK4 propagation, and rigid-body rotational dynamics (Golroudbari, 2024). The family is therefore architecturally heterogeneous even when the same label is used.

3. Modes of physical integration

The most direct mode of integration is the use of a physical forward model as decoder. In the Transformer-VAE for Sentinel-2 inversion, the encoder predicts a truncated-normal posterior $q_\phi(z|x)$ over PROSAIL parameters, and the sampled latent vector is passed to PROSAIL as a deterministic and differentiable decoder. Training minimizes the standard VAE objective,

$$\mathcal{L} = \mathcal{L}_{rec} + \beta_{KL}\,\mathcal{L}_{KL},$$

with the prior defined as a product of independent uniform distributions over valid physiological ranges, so both the latent support and the decoder are physically constrained (Mensah et al., 13 Nov 2025).

The dominant mode in PDE-oriented work is physics-residual training. PINNsFormer replaces the pointwise PINN loss with a sequential loss,

$$\mathcal{L}_{PINNsFormer} = \lambda_{res}\mathcal{L}_{res} + \lambda_{ic}\mathcal{L}_{ic} + \lambda_{bc}\mathcal{L}_{bc},$$

where residual and boundary terms are enforced across all pseudo-time copies, while the initial-condition term is enforced only at $t=0$ (Zhao et al., 2023). PhysicsFormer retains the same residual, boundary, initial, and data components but evaluates them over pseudo-sequences generated by its data embedder (Barman et al., 7 Jan 2026). PINTO learns a boundary-to-solution operator in a simulation-free setting by minimizing only PDE-residual and initial or boundary losses across families of conditions, rather than a supervised solution mismatch (Boya et al., 2024). PITO and PIITO embed the LES equations directly in the loss and remove supervised target-field loss entirely, learning a flow-evolution operator from the residuals of filtered incompressible Navier–Stokes dynamics (Guo, 27 Jan 2026).

Another mode is attention bias from physical diagnostics or priors. PIBERT modifies self-attention logits by subtracting a nonnegative PDE residual proxy, so higher-residual tokens are monotonically down-weighted in attention (Chakraborty et al., 23 Jun 2026). Pi-Transformer defines a symmetric KL divergence between a learned series attention and a physics-informed prior attention, regularizes the prior to evolve smoothly, and uses both the weighted reconstruction signal and the series–prior mismatch as anomaly indicators (Maleki et al., 24 Sep 2025). TelePiT constructs a global teleconnection vector and adds a teleconnection-derived bias term to standard attention logits, making the attention climate-mode-aware rather than purely content-based (Lyu et al., 8 Jun 2025).

A fourth mode is physics-informed state representation. PhysPT builds a differentiable physics-based body representation, Phys-SMPL, with body-part masses and inertia tensors derived from SMPL geometry, and combines it with a continuous contact-force model and an Euler–Lagrange loss (Zhang et al., 2024). TE-PINN embeds quaternion kinematics, Euler’s rotational equation, sensor correction models, RK4 propagation, and unit-quaternion normalization in its loss and state update (Golroudbari, 2024). In these models, transformer attention is only one component of a larger hybrid system whose physical meaning is anchored by state equations.

4. Supervision regimes and optimization strategies

A striking feature of the area is the diversity of supervision regimes. Several models are trained with no labeled solutions. PINTO is explicitly simulation-free and uses only collocation points and initial or boundary samples; ground-truth numerical or analytical solutions are reserved for validation and testing (Boya et al., 2024). PITO and PIITO likewise learn next-step turbulence operators without labeled future flow fields, relying solely on LES residuals (Guo, 27 Jan 2026). This operator-learning interpretation departs from the usual supervised transformer paradigm.

Other models are simulation-only rather than simulation-free. The PROSAIL Transformer-VAE is trained exclusively on simulated PROSAIL data: 500,000 simulated samples for training and 20,000 validation simulations, with no real Sentinel-2 images, no real-image self-supervision, no in-situ labels, and no calibration on satellite imagery (Mensah et al., 13 Nov 2025). The ring-resonator inverse-design transformer is trained on more than 30,000 rows of computationally generated wavelength/$n_{\mathrm{eff}}$/$$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$0 data, with a physics-informed custom loss that penalizes both design-parameter mismatch and mismatch in resonant wavelengths computed from the resonance equation (Lim et al., 23 Apr 2025).

Self-supervised fine-tuning also appears. HMT-PF freezes its backbone and trains only a residual encoder and a fine-tuning feedforward network using a self-supervision term on randomly masked points plus a physics conservation term based on PDE residuals at query points (Du et al., 16 May 2025). PIBERT pretrains with Masked Physics Prediction and Equation Consistency Prediction before supervised reconstruction with divergence and Laplacian regularizers (Chakraborty et al., 23 Jun 2026). PhysPT is trained self-supervised on motion-capture data only and derives pseudo force labels by solving an optimization problem based on the Euler–Lagrange equation, without paired video-force labels or 3D-annotated videos (Zhang et al., 2024).

Optimization strategies are equally varied. PINNsFormer reports L-BFGS with Strong Wolfe line search for 1000 iterations and loss weights set to 1 (Zhao et al., 2023). PhysicsFormer uses Adam followed by L-BFGS fine-tuning (Barman et al., 7 Jan 2026). PhyTF-GAN alternates between a decoder-only physics-informed transformer and a GAN-based residual-guided sampler, adds a causal penalty

$$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$1

and reports that the full framework addresses both residual oversight and temporal-causality violations in time-dependent PDEs (Zhang et al., 15 Jul 2025). PIDT, despite the name, is not trained by a physics residual at all; it is an offline GPT-style decision transformer trained by cross-entropy on logged restoration trajectories, conditioned on a physics-informed return-to-go tied to restored power (Zhao et al., 2024).

5. Domains of application and reported results

The application range is unusually broad. In remote sensing, the PROSAIL Transformer-VAE performs zero-shot Sentinel-2 inversion on FRM4Veg and BelSAR and reports overall LAI RMSE $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$2 with $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$3, outperforming the SNAP biophysical processor and a PROSAIL-VAE baseline for LAI, while remaining worse than PROSAIL-VAE for CCC retrieval (Mensah et al., 13 Nov 2025). In robotics, TE-PINN reports up to 15% reduction in mean quaternion error, 36.84% reduction in mean Euler angle error, 35.04% lower error during rapid rotations and accelerations, 20% improvement in gyroscope bias estimation, and 73.44% better uncertainty calibration than baseline methods (Golroudbari, 2024). In photonic inverse design, the ring-resonator transformer reports less than 6 nm errors for predicted designs corresponding to target resonant wavelengths (Lim et al., 23 Apr 2025).

In climate and anomaly detection, the transformer modifications are tailored to long-range structure rather than pointwise physics. TelePiT reports a 57.7% reduction in RMSE for 2-meter temperature compared to previous best models and outperforms operational numerical weather prediction systems across almost all shown pressure levels and variables (Lyu et al., 8 Jun 2025). Pi-Transformer reports F1 scores of 91.23 on SMD, 95.96 on MSL, 97.02 on SMAP, 96.82 on SWaT, and 98.08 on PSM, with particular strength on timing and phase-breaking anomalies (Maleki et al., 24 Sep 2025).

PDE and CFD applications remain central. PINNsFormer reports superior generalization and accuracy across convection, reaction, wave, and 2D Navier–Stokes settings, with approximately $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$4 compute overhead and $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$5 memory overhead at $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$6 (Zhao et al., 2023). PhysicsFormer reports mean squared errors on the order of $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$7 on Burgers’ equation and Navier–Stokes flow reconstruction, and in the inverse Navier–Stokes problem reports zero identification error for both $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$8 and $$\{[\mathbf{x},t], [\mathbf{x},t+\Delta t], \ldots, [\mathbf{x},t+(k-1)\Delta t]\}$$9 for clean data and errors of 0.07% for $3\mathrm{D}$0 and 0% for $3\mathrm{D}$1 under 1% Gaussian noise (Barman et al., 7 Jan 2026). PITO and PIITO report stable long-term extrapolation exceeding 25 times the training horizon in decaying homogeneous isotropic turbulence, with GPU-memory reductions of 79.5% and 91.3% and parameter requirements of only 31.5% and 3.1% of PIFNO, respectively (Guo, 27 Jan 2026). PIBERT reports all-channel NMSE $3\mathrm{D}$2 and LPCC $3\mathrm{D}$3 on cylinder-real, and all-channel NMSE $3\mathrm{D}$4 on FSI-real, with stronger localized wake recovery than its baselines (Chakraborty et al., 23 Jun 2026).

Design and inverse problems form another cluster. The topology-optimization transformer treats topology optimization as a single-pass operator-learning problem and reports a compliance error of 1.86%, median compliance error of 0.32%, and floating material of 6.60% for ViT-Small-$3\mathrm{D}$5; post-processing with floating-material-loss gradients reduces floating material from 6.60% to 0.8% (Lutheran et al., 4 Apr 2026). PhysPT, applied as a plug-in refinement module on top of kinematics-based monocular video estimates, reports large reductions in acceleration error, foot skating, and ground penetration on Human3.6M and 3DOH, and improves downstream action recognition to 98.0% when joints and inferred forces are combined (Zhang et al., 2024).

6. Limitations, distinctions, and open issues

The primary limitation is conceptual heterogeneity. “Physics-informed” does not denote a single level of physical fidelity. PITT explicitly notes that its physics comes from symbolic equation tokens and a numerical-method-inspired update, not from an explicit PDE residual loss (Lorsung et al., 2023). TelePiT states that its latent ODE is inspired by atmospheric transport rather than being a full governing-PDE solver (Lyu et al., 8 Jun 2025). HMT-PF is described by its authors as a hybrid Mamba-Transformer rather than a pure transformer (Du et al., 16 May 2025). Comparisons across papers therefore require care.

A second limitation is that physics-informed design does not guarantee uniformly better results on every target or regime. In the PROSAIL inversion study, the Transformer-VAE is better than SNAP for CCC but worse than PROSAIL-VAE for CCC, even while achieving strong zero-shot LAI performance (Mensah et al., 13 Nov 2025). In topology optimization, dynamic loading remains more difficult: the best dynamic model reports average compliance error 4.81% but floating material around 48% (Lutheran et al., 4 Apr 2026). In volatility-surface forecasting, PI-ConvTF achieves the best overall MAPE in the main setting, but transformer-based models degrade sharply when evaluated on volatile regimes not represented in training, such as the Subprime Mortgage Crisis or the initial COVID-19 period (Kim et al., 2022).

A third limitation is computational or optimization complexity. PINNsFormer improves failure modes but incurs extra sequence-modeling cost (Zhao et al., 2023). PhyTF-GAN addresses hard residual regions and time causality, yet its authors note higher computational cost, GAN training instability, and pretraining requirements (Zhang et al., 15 Jul 2025). PhysicsFormer reduces cost relative to PINNsFormer, but its performance depends on the sensitivity of $3\mathrm{D}$6 and $3\mathrm{D}$7 (Barman et al., 7 Jan 2026). Some models remain fully simulation-based, and others still await real-world validation beyond synthetic or benchmark settings; CFRTransformer, for example, is evaluated in simulation and explicitly notes future extension to measured 5G NR and 802.11 field-trial datasets (Zubow et al., 2 Apr 2026).

A plausible implication is that the field is less a single methodology than a design space. Recent work shows at least five non-equivalent strategies: symbolic conditioning, residual-constrained operator learning, differentiable physical decoding, prior- or diagnostic-biased attention, and physics-grounded latent-state modeling. Future progress will likely depend less on the generic presence of a transformer than on how tightly the inductive bias matches the governing structure of the target domain.