Physics-Informed Initialization Strategy

• Physics-informed initialization is a family of techniques that leverages governing equations, boundary and initial conditions, and source tasks to effectively shape the starting state of neural models.
• It encompasses methods such as warm-start transfers, spectral architectural modifications, and meta-learning approaches to mitigate optimization issues inherent in standard PINNs.
• Empirical results show significant gains including faster convergence speeds, lower mean squared errors, and improved robustness across various PDEs and operator learning frameworks.

Searching arXiv for recent and foundational papers on physics-informed initialization strategies in PINNs and related operator-learning models. Physics-informed initialization strategy denotes a class of methods that chooses initial parameters, features, or latent representations for physics-informed models using governing equations, boundary or initial conditions, source-task solutions, operator families, or physics-consistent auxiliary data, rather than relying only on generic random schemes such as Xavier or Glorot initialization. In the recent literature, the term covers warm-start transfer in PINNs, short-time-slab pretraining for evolution equations, sinusoidal and Fourier-feature initialization designed to raise input-gradient variability, meta-learned initializations across PDE families, distributed operator pretraining followed by zero-shot physics-informed fine-tuning, diffusion-generated parameter priors, orthogonality-regularized feature pretraining, and basis-aware variance-preserving schemes for KANs (Wong et al., 2021, Guo et al., 2022, Wong et al., 2021, Zhang et al., 2024, Cheng et al., 31 May 2025, Jia et al., 3 Feb 2026, Rigas et al., 27 Oct 2025).

1. Conceptual basis

The central motivation is that standard PINN training is often dominated by optimization pathologies that appear already at initialization. Several works attribute poor performance to low-quality starting states: standard tanh PINNs can induce an initial bias around flat output functions, creating a local minimum that minimizes PDE residuals while remaining far from the true solution satisfying initial and boundary conditions (Wong et al., 2021); deep MLP PINNs can suffer from poor derivative trainability because derivative networks are near-constant, ill-scaled, or unstable under depth (Wang et al., 2024); and training curves can show a staircase phenomenon in which meaningful loss drops coincide with increases in $\varepsilon$-rank, implying that early feature diversity is itself an initialization bottleneck (Tang et al., 16 Jul 2025).

From this perspective, initialization is not merely a numerical convenience. It determines the initial hypothesis space, the initial derivative scales, and often the basin of attraction of the nonconvex residual loss. This suggests that “physics-informed initialization strategy” is best understood as a family of mechanisms for aligning the starting model with operator structure, admissible solution manifolds, or transferable physics, rather than as a single algorithm.

A useful organizing distinction is between four major families. One family transfers parameters from previously solved source tasks to new targets. A second family modifies the architecture or first-layer features so that the initial network already has suitable derivative or spectral properties. A third family learns reusable feature spaces, operator bases, or basis coefficients before the final solve. A fourth family builds explicit weight-space priors by meta-learning or generative modeling, then uses physics-informed fine-tuning on the new task. Each family appears repeatedly across PINNs, DeepONets, ELMs, and KANs.

2. Warm-start transfer within PINNs

The most direct strategy is to pretrain on one or more source scenarios and then warm-start target optimization. In the lid-driven cavity study “Improved Surrogate Modeling of Fluid Dynamics with Physics-Informed Neural Networks” (Wong et al., 2021), the proposed transfer optimization scheme trains a source PINN jointly on data and physics for $\mathrm{Re}=\{50,100\}$, copies all weights and biases to the target network, and fine-tunes on a new Reynolds number using physics-only loss,

$$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$

No layers are reinitialized or frozen, and end-to-end fine-tuning is used. The paper reports roughly $3\times$ faster convergence and an order-of-magnitude lower test MSE than physics-only training from Xavier initialization, while direct extrapolation without target-side optimization performs poorly. The same study also shows that PINN regularization improves robustness to sparse noisy data and remains beneficial under partial physics, especially when continuity is available but full momentum is not.

A related temporal curriculum appears in PT-PINN for evolution equations (Guo et al., 2022). There, a difficult full-horizon problem is replaced by a sequence of shorter time slabs. A PINN is first trained on $[0,T_p]$, its weights are transferred forward, and pseudo-labels produced on earlier slabs are optionally added through

$$\mathcal{L}_{\mathrm{PT}}(\theta)=\lambda_r\,\mathrm{MSE}_r+\lambda_{ic}\,\mathrm{MSE}_{ic}+\lambda_{bc}\,\mathrm{MSE}_{bc}+\lambda_{\mathrm{sup}}\,\mathcal{L}_{\mathrm{sup}}.$$

This strategy is explicitly intended for strongly nonlinear or high-frequency evolution PDEs. The reported gains include successful solution of Fisher reaction, Allen–Cahn, strongly nonlinear heat, convection with large $\beta$, and oscillatory 2D heat problems where standard PINNs fail or collapse.

IDPINN extends warm-starting to domain decomposition (Si et al., 2024). A single global PINN is first trained on a small dataset to obtain $\theta^{(0)}$, and that same parameter set initializes each subdomain network. The main stage then augments the usual subdomain PDE, boundary, and initial losses with interface continuity and smoothness penalties, including gradient matching of both the solution and the PDE residual across interfaces. On Helmholtz, Poisson, Heat, and Burgers benchmarks, this initialization-enhanced decomposition reduces relative $L^2$ error relative to both vanilla PINN and XPINN baselines.

A different transfer mechanism appears in the student/teacher framework for multi-objective collocation PINNs (Bahmani et al., 2021). A smaller teacher network is first trained with auxiliary point-cloud data and physics objectives, then widened by a Net-to-Net mapping that preserves the represented function exactly. The student resumes physics-informed optimization from this transferred state, together with conflict-aware gradient surgery across PDE, boundary, compatibility, and data objectives. This formulation treats initialization, transfer, and multi-objective optimization as tightly coupled.

3. Architectural and spectral initialization schemes

Another major line of work changes the initial feature map or residual structure so that the network begins with more appropriate gradients, frequencies, or nonlinear interactions. In sf-PINN, inputs are passed through a sinusoidal mapping

$\phi(x)=\sin(2\pi(W_1x+b_1)),$

with $\mathrm{Re}=\{50,100\}$0 and trainable $\mathrm{Re}=\{50,100\}$1 (Wong et al., 2021). The paper’s analysis shows that standard tanh PINNs initialized with Xavier produce vanishing input-gradient variance and therefore fall into a deceptive PDE-residual minimum, while the sinusoidal input mapping raises and spreads gradient variance across the domain. Empirically, the reported gains are large: 3–4 orders of magnitude MSE improvement across multiple forward and inverse problems, with examples including wave, Helmholtz, Navier–Stokes, and convection–diffusion equations.

PirateNets pursue the same goal through adaptive residual connections (Wang et al., 2024). Each deep block is initialized with residual coefficient $\mathrm{Re}=\{50,100\}$2, so the entire network starts as a shallow linear combination of Fourier features rather than as a genuinely deep nonlinear map. The final layer is then initialized by least squares to fit an initial condition or a linearized PDE solution. During training, the $\mathrm{Re}=\{50,100\}$3 coefficients are learned, so the model progressively deepens only as required by the residual landscape. This initialization is explicitly physics-informed because it controls derivative scales at $\mathrm{Re}=\{50,100\}$4 and can encode inductive bias via the embedding and least-squares target.

Deeper-PINN, also described as EM-PINN, uses element-wise multiplication to avoid the deep-linear regime that standard MLP PINNs enter near initialization (Jiang et al., 2024). Under the paper’s linear-regime analysis with zero biases, the derivative of a standard MLP collapses to a product of weight matrices, whereas the multiplicative architecture retains nonlinear dependence on the input. The reported benefit is not a standalone warm start but an architectural initialization remedy: Xavier-normal weights together with multiplicative blocks and shortcut connections preserve nonlinear expressivity and mitigate gradient vanishing in deeper PINNs.

SFLI moves the focus to first-layer feature diversity (Tang et al., 16 Jul 2025). It constructs $\mathrm{Re}=\{50,100\}$5-linearly independent neurons in the input layer by choosing structured orientations, shifts, centers, and a scale parameter

$\mathrm{Re}=\{50,100\}$6

thereby increasing the initial $\mathrm{Re}=\{50,100\}$7-rank. The method is activation-agnostic and is inserted as a “single-line plug-in” for conventional models. The paper ties faster convergence to higher initial $\mathrm{Re}=\{50,100\}$8-rank and reports improvements on function approximation, PirateNet-based Allen–Cahn and cavity flow, Navier–Stokes on a torus, and high-dimensional parabolic PDEs.

A more explicitly spectral variant appears in multistage PINNs with spectral priors (Li et al., 25 Aug 2025). SI-MSPINNs extract the dominant spectral pattern of the current residual by discrete Fourier transform and use those frequencies and phases to initialize a front embedding at each stage, whereas RFF-MSPINNs sample random Fourier features from a residual power spectral density. In both cases the initialization is updated stage by stage as residual energy shifts toward finer scales.

4. Meta-learning and generative parameter priors

A more global interpretation of initialization is to learn a reusable prior over parameters from a distribution of PDE tasks. Meta-PINN does this for seismic wavefield modeling in the frequency domain (Cheng et al., 2024). Tasks are defined by velocity models, each with support and query collocation sets. Inner-loop updates optimize a combined physics and regularization loss on the support set, and outer-loop meta-updates optimize query performance after adaptation. The output is a common initialization $\mathrm{Re}=\{50,100\}$9 for new velocity models. Reported results include reaching the accuracy level of a vanilla PINN trained for about 10,000 epochs in about 2,000 epochs on a layered model, and reaching a loss threshold in about 8,000 epochs versus about 50,000 epochs on the Overthrust model.

NRPINN adapts Reptile to physics-informed settings (Liu et al., 2021). Tasks are sampled from parameterized PDE families and may be supervised, unsupervised, or semi-supervised. Inner loops minimize PDE, boundary, and optional data losses, and the outer Reptile update moves the shared initialization toward task-adapted parameters. The paper reports that NRPINN is faster and more accurate than Xavier, uniform, normal, and random baselines across Poisson, Burgers, Schrödinger, and inverse Burgers tasks, with unsupervised NRPINN often performing best.

DiffPINN models the distribution of converged PINN weights directly (Cheng et al., 31 May 2025). Multiple PINNs are first trained for frequency-domain scattered wavefields, each parameter vector is flattened, a 1D convolutional autoencoder learns a latent code, and a conditional latent diffusion model is trained over those codes using the velocity model and source configuration as conditions. At inference, DDIM sampling produces a latent vector for a new velocity model, the decoder reconstructs full PINN parameters, and a physics-guided latent correction step backpropagates the physics loss through the decoder before final fine-tuning. The paper reports lower physics loss than Meta-PINN and significantly better performance than vanilla PINN on both in-distribution and out-of-distribution velocity models.

These meta-learned and generative methods share one feature with simpler transfer schemes: the target problem is not solved by initialization alone. In Meta-PINN, NRPINN, and DiffPINN, initialization is followed by ordinary physics-informed optimization on the new task. This suggests that learned priors mainly shorten the optimization path and bias training toward better basins, rather than replacing target-side residual minimization.

5. Feature-space, operator, and basis-aware variants

Some methods shift initialization from parameter space to feature space. In distributed DeepONet pretraining (Zhang et al., 2024), multiple operators are learned in a distributed regime such as D2NO or MODNO, local parameters are merged, and the resulting initialization is then adapted to a new operator by zero-shot physics-informed fine-tuning. The fine-tuning loss contains PDE residual, boundary, and initial-condition terms but no labeled downstream solutions. LoRA is also used as a parameter-efficient fine-tuning option. This extends the notion of physics-informed initialization from PINN solution networks to neural operator representations.

PD-OFM goes further by explicitly pretraining a feature basis with a PINN-style residual plus orthogonality regularization,

$$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$0

then freezing the learned features and solving only the final coefficients by least squares (Jia et al., 3 Feb 2026). The paper argues that this produces nearly orthogonal, operator-aware features, improves conditioning, and enhances transfer across source terms and geometries.

GFF-PIELM offers an ELM analogue (Ren et al., 14 Oct 2025). Instead of generic random Fourier features, each hidden neuron receives its own frequency coefficient $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$1 in

$$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$2

Initialization proceeds by starting from a wide interval $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$3, solving for output weights $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$4, and monitoring which frequencies correspond to vanishing or excessively large $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$5. The upper frequency bound is then shrunk to the useful band, often in one or two iterations.

Deep physics-informed KANs adopt a basis-aware Glorot-like rule (Rigas et al., 27 Oct 2025). For basis functions $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$6, forward and backward variance preservation lead to

$$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$7

with $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$8 and $$\mathcal{L}_{\text{target}}(\theta)=\lambda\,\mathcal{L}_{\text{PDE}}^{(\mathrm{Re}=\mathrm{target})}.$$9. RGA KANs then add residual-gated adaptive blocks inspired by PirateNets and initialize the final output layer by physics-informed least squares against the initial condition. Here initialization is simultaneously basis-aware, variance-preserving, and PDE-aware.

Taken together, these approaches indicate that physics-informed initialization can mean initializing coefficients, reusable features, operator bases, or even output-layer least-squares projections. The common principle is that the starting representation should already reflect the operator, the domain, or the relevant spectral content.

6. Reported gains, limitations, and neighboring concepts

The empirical literature reports consistent improvements, but the gains depend strongly on the PDE family, regime shift, architecture, and the form of pretraining.

Method	Setting	Reported effect
Transfer optimization (Wong et al., 2021)	Lid-driven cavity PINN, source $3\times$0 to targets $3\times$1	about $3\times$2 faster convergence and order-of-magnitude lower test MSE than Xavier initialization
sf-PINN (Wong et al., 2021)	Multiple forward and inverse PDEs	3–4 orders of magnitude MSE improvement across problems
PT-PINN (Guo et al., 2022)	Allen–Cahn, high-frequency heat, convection, nonlinear heat	Allen–Cahn $3\times$3 with two slabs versus $3\times$4 for standard PINN
DeepONet distributed pretraining (Zhang et al., 2024)	Burgers-type operator learning	PI-Full random init 21.14% versus PI-Full D2NO 4.99%
Meta-PINN (Cheng et al., 2024)	Seismic wavefield PINN	about $3\times$5 fewer epochs on layered model; about $3\times$6 speedup on Overthrust
SFLI (Tang et al., 16 Jul 2025)	High-dimensional parabolic PDE, $3\times$7	baseline relative error 1.11 versus SFLI $3\times$8
RGA KAN (Rigas et al., 27 Oct 2025)	Allen–Cahn and advection	Allen–Cahn $3\times$9 versus cPIKAN $[0,T_p]$0; advection $[0,T_p]$1 versus cPIKAN $[0,T_p]$2

Several limitations recur. Extrapolation remains difficult under large domain shifts: the cavity-flow transfer study shows that even physics-informed warm starts still struggle at $[0,T_p]$3 (Wong et al., 2021). PT-PINN notes that very short slabs may underfit while longer slabs recover the original difficulty (Guo et al., 2022). DiffPINN depends on a fixed architecture and on a sufficiently broad training distribution over velocity models (Cheng et al., 31 May 2025). DeepONet distributed pretraining notes that standard branch/trunk parameterizations and uniform collocation may not scale well to very high-dimensional PDEs or complex geometries (Zhang et al., 2024). PD-OFM improves conditioning but does not eliminate ill-conditioning entirely (Jia et al., 3 Feb 2026). RGA KANs show that even variance-preserving initialization may be insufficient for deep cPIKANs without residual gating on some PDEs (Rigas et al., 27 Oct 2025).

A common misconception is that physics-informed initialization is synonymous with simple parameter reuse. The literature is broader: it includes frequency-scale selection by monitoring analytically solved output weights in GFF-PIELM (Ren et al., 14 Oct 2025), first-layer $[0,T_p]$4-rank engineering in SFLI (Tang et al., 16 Jul 2025), orthogonality-regularized feature pretraining in PD-OFM (Jia et al., 3 Feb 2026), and least-squares output-layer initialization in PirateNets and RGA KANs (Wang et al., 2024, Rigas et al., 27 Oct 2025). Another misconception is that a good initialization removes the need for target optimization. In the cavity-transfer study, direct extrapolation without fine-tuning performs poorly, and the best results require target-side physics-only training (Wong et al., 2021).

An adjacent but distinct line of work emphasizes that initialization also determines which solution branch a PINN may discover. Deep ensembles with different random initializations can recover multiple stable and unstable solutions of nonlinear equations such as Allen–Cahn and cavity flow, and the resulting PINN fields can then initialize conventional solvers (Zou et al., 8 Mar 2025). This is not physics-informed initialization in the transfer or spectral sense, but it reinforces the broader point that initialization shapes the basin structure of scientific machine learning.

Overall, the available evidence suggests that physics-informed initialization strategies are most effective when they align initial parameters or features with conserved structure, dominant spectra, admissible boundary behavior, or nearby task families, while still leaving room for subsequent physics-based optimization.