Multimodal Physics-Informed Neural Network

• Multimodal PINN is a neural framework that integrates heterogeneous inputs, including meteorological, environmental, and image data, under embedded physical laws.
• It employs specialized preprocessing pipelines and dual pathway fusion to combine numeric metadata with fisheye image features for accurate thermal predictions.
• The model balances data loss with physics-based residuals, yielding improved interpretability and performance in urban thermal comfort assessments.

A multimodal Physics-Informed Neural Network (PINN) is a neural architecture in which heterogeneous data channels are fused within a training objective constrained by governing physical relations. In the most explicit formulation represented in recent arXiv work, multimodality refers to the joint use of meteorological variables, built-environment descriptors, and image-derived shading information to predict Mean Radiant Temperature, while a non-trainable computation layer and a physics residual enforce radiative-balance consistency (Shaeri et al., 11 Mar 2025). More broadly, the term also appears in adjacent PINN literature for settings that combine heterogeneous sensor modalities, multiple physical constraints, auxiliary point clouds, or multiple PDE families within one trainable framework.

1. Definition and terminological scope

In the outdoor thermal-comfort setting, the multimodal PINN is defined by three input modalities: meteorological and solar variables, built-environment descriptors, and fisheye image-derived shading metrics. The objective is to predict $T_{mrt}$ while embedding a six-direction radiative model directly into the network and loss (Shaeri et al., 11 Mar 2025).

Related PINN literature uses closely adjacent meanings. In physics-informed representations for PDE control, multimodality refers to sparse observations with three possible modalities, $(u^x,u^y,p)$, under missing or uncertain sensor readings, with masks used to represent absent channels (Feng et al., 2024). In multi-objective collocation PINNs, each enforced requirement—PDE residual, boundary condition mismatch, auxiliary point-cloud fit, thermodynamic inequality, symmetry, or invariant property—constitutes one component of a vector-valued optimization problem (Bahmani et al., 2021). In multi-PDE learning, a single backbone is shared across the Simple Harmonic Oscillator, the 1D Heat Equation, the 1D Wave Equation, and the 2D Laplace Equation, with the loss selecting the appropriate operator per task (Park, 11 Feb 2025).

This suggests that “multimodal PINN” is not yet a fully standardized term. In current usage, it can denote multimodal sensing, multimodal supervision, or unified learning across distinct physical regimes. The common element is the simultaneous treatment of multiple heterogeneous information sources under explicit physical constraints.

2. Physical formulation and loss construction

The mean-radiant-temperature formulation in the urban-comfort model is based on six directional radiative components: up, down, north, east, south, and west. Shortwave flux $S$ and longwave flux $L$ from these directions are aggregated into a total radiative flux, and $T_{mrt}$ is then obtained through a Stefan–Boltzmann balance. The human-body shortwave and longwave absorption coefficients are denoted by $a_k$ and $a_l$, directional weighting factors by $W$, and the computation layer is fixed rather than trainable (Shaeri et al., 11 Mar 2025).

The loss has the canonical PINN decomposition

$$\mathcal{L} =\alpha\,\mathcal{L}_{data} +\beta\,\mathcal{L}_{phys},$$

where $\mathcal{L}_{data}$ is the mean squared error between predicted and measured $(u^x,u^y,p)$0, and $(u^x,u^y,p)$1 penalizes violations of the radiative transfer equations through a residual formed from predicted total flux and predicted $(u^x,u^y,p)$2. In the reported experiments, $(u^x,u^y,p)$3 and $(u^x,u^y,p)$4 were found effective through random hyperparameter search (Shaeri et al., 11 Mar 2025).

This structure aligns with broader PINN practice. In multimodal policies with physics-informed representations, the PDE-informed component is written as

$(u^x,u^y,p)$5

with separate terms for initial conditions, boundary conditions, and interior residuals, while sparse-data loss is backpropagated through both encoder and decoder (Feng et al., 2024). In mixed-formulation PINNs for thermo-mechanically coupled systems, the total coupled loss combines strong-form residuals, energy-form residuals, boundary-condition terms, and connection terms linking auxiliary stresses or fluxes to the physical fields (Harandi et al., 2023).

A recurrent design principle is therefore that multimodal PINNs do not simply concatenate inputs. They organize heterogeneous evidence around a physics-compatible latent structure, usually by combining data-fit and residual-based terms in a single objective.

3. Modalities, encoders, and fusion mechanisms

The outdoor $(u^x,u^y,p)$6 model uses three modalities with distinct preprocessing pipelines. The meteorological and solar variables include air temperature, relative humidity, wind speed, timestamp, latitude, longitude, altitude, solar altitude, azimuth, and minutes from sunrise; preprocessing uses temporal nearest-neighbor interpolation for missing data, outlier removal at $(u^x,u^y,p)$7, and min–max normalization. Built-environment descriptors include Sky View Factor, percent tree canopy, percent building cover, and impervious versus pervious surfaces, normalized alongside meteorological inputs. Fisheye-image-derived shading metrics are produced by reprojecting six-directional raw cube maps into a single hemispherical fisheye view via equiangular warping, applying SegFormer to generate a sky-mask and sun-exposed versus shaded segmentation, and using a fine-tuned ResNet-50 backbone with the last 30 layers unfrozen to extract a 2,048-dimensional feature vector (Shaeri et al., 11 Mar 2025).

The network architecture separates shortwave and longwave pathways. The image backbone is a ResNet-50 pretrained on ImageNet and fine-tuned on fisheye inputs. The shortwave MLP has three fully connected layers with 128, 256, and 128 units and ReLU activations, taking as input the concatenation of the 2,048-dimensional image vector and numeric metadata. The longwave MLP has the same layer widths and activations but consumes only numeric metadata. A fusion layer concatenates the last hidden representations of the two MLPs and applies one linear layer to output six directional flux predictions for both shortwave and longwave radiation. A fixed $(u^x,u^y,p)$8 computation layer then aggregates the predicted fluxes through the embedded physics equation (Shaeri et al., 11 Mar 2025).

A distinct multimodal fusion strategy appears in PDE-control settings. There, each sensor contributes a vector containing spatial coordinates, time, observed values or zeros, and binary masks for modality presence. A sensor MLP maps each sensor vector to a latent feature, mean or sum pooling aggregates across sensors, and a second MLP produces a time-indexed representation $(u^x,u^y,p)$9. The decoder takes $S$0 as input and outputs $S$1, while missing modalities are handled by zero-filled channels plus masks rather than by architectural branching (Feng et al., 2024).

These two cases illustrate two canonical multimodal PINN patterns. One pattern fuses image and tabular metadata to infer physically structured latent quantities such as directional radiative fluxes. The other maps sparse, masked sensor sets into a latent state that must be sufficient for PDE-consistent field reconstruction.

4. Optimization, sampling, and training strategies

For $S$2 modeling, the reported dataset contains 1,130 observations with matching fisheye images from MaRTy in Tempe, Arizona, summer 2018. Training uses an 80% train and 20% validation split, with additional 3-fold cross-validation for robustness. Optimization uses Adam with initial learning rate $S$3 decayed on plateau, batch size 32, weight decay $S$4, early stopping on validation loss, and dropout with $S$5 in the MLPs. Hyperparameters are selected by random search over $S$6 and MLP widths. No augmentation is used beyond standard image normalization because fisheye warping yields consistent geometry (Shaeri et al., 11 Mar 2025).

Other multimodal PINN training regimes emphasize the optimization difficulty created by multiple competing objectives. In multi-objective collocation PINNs, training is explicitly cast as vector-valued optimization, with one objective per PDE residual, boundary condition, compatibility constraint, symmetry constraint, thermodynamic constraint, or auxiliary point-cloud fit. The paper introduces three mechanisms: pre-training on auxiliary data stored in point clouds, Net2Net teacher-to-student initialization, and gradient surgery that projects away pairwise gradient conflicts when $S$7 (Bahmani et al., 2021).

Sample selection itself can also become part of the multimodal PINN pipeline. In multi-PDE learning, Monte Carlo Dropout is used for uncertainty estimation within an active-learning loop that selects the highest-variance points from a candidate pool, retrains or fine-tunes the shared PINN, and repeats until the desired budget is reached. The shared backbone is a three-layer tanh MLP of width 20 with dropout rate $S$8 at each hidden layer during both training and uncertainty estimation (Park, 11 Feb 2025).

For coupled multiphysics, mixed-formulation PINNs introduce yet another training axis: fully coupled optimization of all fields together versus sequential or staggered optimization in which thermal and mechanical subnetworks are alternately minimized while the other is frozen. Hard boundary constraints combined with L–BFGS are also reported, reducing training time from hours under Adam to approximately 40 minutes in the cited thermo-mechanical study (Harandi et al., 2023).

5. Empirical performance and interpretability

For urban thermal-comfort prediction, the principal validation results are summarized below.

Model	RMSE	$S$9
Baseline multimodal NN (no physics loss)	3.65 °C	0.879
PINN (numeric + metadata with $L$0)	3.63 °C	0.881
Multimodal PINN (fused with fisheye shading features)	3.50 °C	0.889
XGBoost on WebMRT feature set	$L$1 °C	$L$2

The same experiments report MAPE and MBE. The baseline multimodal NN yields MAPE $L$3 and MBE $L$4 °C; the PINN using numeric inputs and metadata with physics loss yields MAPE $L$5 and MBE $L$6 °C; the multimodal PINN yields MAPE $L$7 and MBE $L$8 °C. The authors conclude that the multimodal PINN matches state-of-the-art accuracy while enforcing physical consistency (Shaeri et al., 11 Mar 2025).

Interpretability is treated as an architectural and loss-level property rather than a post hoc attribution exercise. In the $L$9 system, interpretability is enhanced by $T_{mrt}$0 because predicted fluxes must honor radiative-balance laws, and the separate shortwave and longwave MLPs permit inspection of which inputs drive each component. This is a narrower claim than full causal identifiability, but it does distinguish the model from purely black-box multimodal regressors (Shaeri et al., 11 Mar 2025).

A related but distinct empirical picture appears in multimodal policies with physics-informed representations. There, representation consistency measured against an autoencoder baseline is reported as 0.7625 for AE under full modality and 0.8091 under missing modality, compared with 0.4024 and 0.3659 for PIR. The same work reports that SAC combined with PIR converges approximately $T_{mrt}$1 faster, improves vortex-street navigation success rate by approximately 10–15%, and decreases average time-to-target by approximately 20% (Feng et al., 2024). These results concern control rather than direct PDE solution, but they reinforce the broader point that multimodal physics-informed representations can remain useful under modality dropout.

6. Applications, limitations, and research directions

The most developed application in the supplied literature is urban thermal-comfort assessment in hot desert climates. The multimodal PINN for $T_{mrt}$2 is presented as a surrogate that bridges field measurement, radiation modeling, and deployment-scale inference. Its scalability claim is specific: fisheye-based shading extraction replaces expensive radiometric sensors or detailed GIS metadata, permitting deployment on mobile devices or drones with only a camera and standard weather data. At the same time, the physics residual increases training time by approximately 15%, and real-time inference is described as feasible on modern GPUs but potentially challenging on low-power edge hardware (Shaeri et al., 11 Mar 2025).

The current limitations are likewise domain specific. The cited $T_{mrt}$3 model is trained on hot desert climates, and generalization to temperate or humid zones is said to require retraining on new data. This is consistent with the broader PINN observation that physics constraints reduce, but do not eliminate, sensitivity to domain shift in forcing distributions, material properties, or boundary regimes (Shaeri et al., 11 Mar 2025).

Across neighboring lines of work, multimodal PINNs are already extending into several directions. One direction is task unification: a single shared network can solve the SHO, Heat1D, Wave1D, and Laplace2D equations without redesigning the backbone, which suggests the feasibility of a more generalizable PINN-based foundation model (Park, 11 Feb 2025). Another is coupled multiphysics: mixed-formulation PINNs for stationary thermo-mechanically coupled systems use auxiliary stresses and heat fluxes, strong-form plus energy-form losses, and parametric learning to handle heterogeneous discontinuities and unseen parameter ratios (Harandi et al., 2023). A third is downstream decision-making: physics-informed multimodal representations can serve directly as latent states for reinforcement learning in PDE-governed environments (Feng et al., 2024).

Future extensions reported for the $T_{mrt}$4 system include incorporating transformer-based vision encoders for richer spatial context, expanding the dataset across seasons and climates, integrating six-directional imaging directly rather than a single fisheye, coupling the PINN surrogate with meso-scale urban climate simulators for accelerated scenario studies, and adapting the framework to building-energy modeling and façade radiation analysis (Shaeri et al., 11 Mar 2025). A plausible implication is that the next generation of multimodal PINNs will be defined less by a single architectural recipe than by a family of design choices for fusing heterogeneous observations, selecting informative samples, and enforcing the appropriate physics at the correct level of abstraction.