Multi-Objective PINNs

• Multi-objective PINNs are a deep learning framework that embeds physical laws into neural networks by treating PDE, IC, BC, and data-fit terms as distinct optimization objectives.
• The approach leverages methods like NSGA-PINN, combining evolutionary Pareto-front search with gradient refinement to achieve lower residuals and enhanced constraint satisfaction.
• Adaptive weighting and constrained optimization strategies efficiently balance trade-offs among conflicting loss scales, proving effective even in noisy and multi-fidelity applications.

Physics-Informed Neural Networks (PINNs) provide a variational deep learning approach to solving differential equations by embedding the governing physical laws, auxiliary conditions, and data constraints directly into the neural network loss. In practice, the PINN framework naturally leads to a multi-objective optimization problem, as each governing equation, initial or boundary constraint, measurement data, or auxiliary criterion introduces a distinct loss term. The multi-objective nature of PINN training presents optimization and modeling challenges beyond traditional single-objective machine learning, including balancing objectives of vastly differing scale, efficiently exploring trade-offs, and ensuring precise satisfaction of hard physical constraints. Recent methods address these issues using scalarization, adaptive and dynamic weighting, evolutionary search for Pareto fronts, augmented Lagrangian techniques, constrained optimization, and gradient-conflict resolution, among others. This article surveys the mathematical formulation, optimization algorithms, empirical findings, and practical guidelines for multi-objective PINN training, with emphasis on developments as exemplified by NSGA-PINN (Lu et al., 2023), as well as related advances in adaptive balancing and constrained formulations.

1. Mathematical Formulation: Multi-Objective PINN Losses

A generic PINN for solving a PDE problem is parameterized by network weights $\theta$, designed so that its output $u_\theta(x,t)$ approximates the desired solution $u(x,t)$. Training loss terms typically encode:

• Physics (PDE) residual: $$\mathrm{MSE}_f = \frac{1}{N_f} \sum_{i=1}^{N_f} \| \mathcal{N}_\theta(x^i_f, t^i_f) \|^2$$;
• Initial condition (IC) loss: $$\mathrm{MSE}_{\mathrm{ic}} = \frac{1}{N_{\mathrm{ic}}}\sum_{i=1}^{N_{\mathrm{ic}}} \|u_\theta(x^i_{\mathrm{ic}},0) - u_0(x^i_{\mathrm{ic}}) \|^2$$;
• Boundary condition (BC) loss: $$\mathrm{MSE}_{\mathrm{bc}} = \frac{1}{N_{\mathrm{bc}}}\sum_{i=1}^{N_{\mathrm{bc}}} \|u_\theta(x^i_{\mathrm{bc}}, t^i_{\mathrm{bc}}) - g(x^i_{\mathrm{bc}}, t^i_{\mathrm{bc}}) \|^2$$;
• Data-mismatch or auxiliary losses: e.g., $$\mathrm{MSE}_u = \frac{1}{N_u} \sum_{i=1}^{N_u} \|u_\theta(x^i_u, t^i_u) - \widehat u^i\|^2$$ for fitting observed data.

Each term corresponds to a distinct objective. In vectorized form, define $$F(\theta) = [f_1(\theta), f_2(\theta), f_3(\theta), f_4(\theta)]^\top$$, where $f_1$–$f_4$ are the losses defined above (Lu et al., 2023). The classical approach aggregates these by a weighted sum $L(\theta) = \sum_{j} w_j f_j(\theta)$, but multi-objective PINNs seek to optimize all objectives simultaneously, exploring the Pareto front—the set of parameter vectors that cannot be improved in one objective without worsening at least one other.

2. Multi-Objective Optimization Algorithms

2.1. Population-Based Pareto Front Discovery: NSGA-PINN

NSGA-PINN (Lu et al., 2023) solves for Pareto-optimal PINN parameters via the Non-dominated Sorting Genetic Algorithm II (NSGA-II), hybridized with local stochastic gradient refinements:

• Maintain a population $P_t$ of candidate parameter vectors.
• Evaluate each on $F(\theta)$, sort by non-domination level and crowding distance.
• Apply simulated binary crossover and polynomial mutation (with standard settings: $N_p=50$, $G=100$, $p_c=0.9$, $p_m=1/d$, tournament size 2).
• Locally refine each offspring via $K$ steps of ADAM or other gradient optimizer to exploit gradient information.
• Merge, re-sort, and truncate by Pareto rank/crowding distance to form $P_{t+1}$.
• Output the final Pareto set, among which models with precisely minimized IC/BC losses can be selected directly, reliably enforcing physical constraints.

2.2. Scalarization and Adaptive Weighting

Weighted-sum scalarizations, as in standard PINN training, select weights statically ($w_j$) or with online adaptation. The effectiveness of the solution strongly depends on balancing the residual and constraint losses, which can differ in scale by orders of magnitude. Adaptive weighting algorithms address this by tuning $w_j$ based on instantaneous or historical loss or gradient statistics (Farea et al., 17 Sep 2025). Representative strategies include Residual-Based Attention (RBA), Self-Adaptive (SA), and Learning-Rate Annealing (LRA)—each leveraging the relative magnitude of losses and their gradients to promote balanced optimization progress for each term (Farea et al., 17 Sep 2025).

2.3. Constrained Optimization and Hybrid Methods

Constraint-based methods penalize physics violations softly but enforce boundary/initial/data constraints via equality constraints with Lagrange multipliers, as in the Augmented Lagrangian Method (ALM) (Basir et al., 2021). For forward and inverse problems with multi-fidelity data, PINNs may be trained as: $$\min_\theta \mathcal{J}_\mathcal{F}(\theta) + \mathcal{J}_\mathcal{M}(\theta), \quad \text{s.t. } \phi(B_j(\cdot)) = 0, \; \phi(I_k(\cdot)) = 0$$ with ALM updates for multipliers and penalties driving exact satisfaction of hard constraints (e.g., boundary values), while allowing soft fitting to noisy or low-fidelity data.

3. Empirical Performance and Guidelines

Extensive experiments confirm that multi-objective PINN methods improve both the enforcement of physical constraints and prediction accuracy compared to scalarized or single-objective baselines:

• On prototypical ODE/PDE benchmarks (Burgers', Poisson, inverse parameter recovery with noise), NSGA-PINN finds solutions with one order of magnitude lower physics residual and substantially reduced $L^2$ errors, and achieves $<2\%$ parameter recovery error, compared to $>10\%$ for standard PINN (Lu et al., 2023).
• With increasing noise (1–5%), NSGA-PINN achieves lower estimation bias and variance due to its ability to directly optimize data-fit and physics consistency as separate objectives.
• In multi-fidelity and multiphysics problems, constraint-based or Pareto-front-driven approaches robustly enforce IC/BC/data constraints without suffering from delicate loss-balancing hyperparameter tuning, a notorious challenge in standard PINN training (Basir et al., 2021).

Practical recommendations synthesized from empirical studies include:

• Treat IC and BC losses as separate objectives whenever high-precision constraint enforcement is required.
• Begin with moderate population sizes and generation counts (e.g., $N_p = 50$, $G = 50$ in NSGA-PINN), increasing only if Pareto convergence is inadequate.
• Use a small number (5–20) of local gradient steps per evolutionary generation to leverage both global exploration and local refinement.
• Select final models from the Pareto set by the targeted trade-off between residual losses and constraint satisfaction.
• When possible, include data-mismatch terms explicitly as objectives in inverse or data-driven applications (Lu et al., 2023).

4. Trade-Offs, Constraint Enforcement, and Pareto Selection

In the multi-objective setting, explicit trade-offs among objectives can be visualized as the Pareto front in the loss space. For example, reducing data loss may increase PDE-residual loss and vice versa; the choice of final model thus becomes a principled trade-off rather than a side effect of arbitrary weight selection. This is particularly effective for constraint satisfaction: splitting IC and BC losses as standalone objectives ensures that some solutions in the Pareto set achieve arbitrarily low boundary errors, making this approach robust compared to tuning scalar weights in a composite loss (Lu et al., 2023). In practice, plotting the Pareto front and selecting the desired model enables systematic handling of the data–physics compromise.

5. Connections to Broader Multi-Objective and Adaptive Strategies

The multi-objective PINN paradigm aligns naturally with broader machine learning multi-task and multi-objective optimization trends, including the use of evolutionary methods (e.g., NSGA-II as in NSGA-PINN) for Pareto-front discovery, adaptive and dynamic weighting strategies, constraint-based augmented Lagrangian methods, and variational approaches for regularization and enforcing hard constraints (Basir et al., 2021, Farea et al., 17 Sep 2025). Studies highlight the criticality of handling gradient and loss scale imbalances, and the computational challenges in high-dimensional or stiff systems. NSGA-PINN demonstrates a hybrid of evolutionary exploration and gradient-based exploitation that is representative of state-of-the-art methodological trends in PINN optimization.

6. Applications, Limitations, and Practical Outlook

Multi-objective PINNs are applicable to both forward (solution approximation) and inverse (parameter identification) problems, especially under regimes of noise, multi-fidelity data, or highly imbalanced loss scales. These approaches are robust to local minima, effective for high-precision constraint enforcement, and yield natural mechanisms for handling noisy/uncertain observations. The main computational overhead arises from maintaining and evolving populations for Pareto estimation, as well as managing post-processing and model selection from the Pareto set. For large-scale or real-time applications, computational costs may be mitigated by reduced population sizes or by introducing more efficient surrogate or adaptive methods.

Key implementation steps include identifying the relevant loss terms to be separated as objectives, maintaining and updating the population with both evolutionary and local refinement steps, periodically logging and analyzing the Pareto front, and selecting models based on the operational criteria for the problem at hand (e.g., constraint satisfaction vs. global accuracy).

7. Summary Table: NSGA-PINN Training Components

Component	Description	Typical Hyperparameters / Notes
Objectives	$f_1$=PDE, $f_2$=IC, $f_3$=BC, $f_4$=data-mismatch	Each as MSE over selected points
Loss vector $F(\theta)$	$[f_1, f_2, f_3, f_4]^\top$	Each optimized directly, not summed
Evolutionary step	SBX crossover, polynomial mutation (NSGA-II)	$N_p=50$, $p_c=0.9$, $p_m=1/d$
Local refinement	ADAM updates for each child in each generation	$K=5-20$ steps
Selection	Non-dominated sorting, crowding distance	Tournament size=2
Output	Pareto-optimal set for trade-off selection	Post-processed for application

This organization of the training process minimizes manual hyperparameter tuning, enhances robustness to non-convexity and ill-scaled optimization, and provides a transparent, physically interpretable trade-off among simultaneous objectives (Lu et al., 2023, Farea et al., 17 Sep 2025, Basir et al., 2021).