Surrogate Modeling with Neural Networks

• Surrogate modeling with neural networks is a data-driven technique that approximates expensive simulation-based models, particularly those governed by PDEs or complex nonlinear processes.
• Different architectures like DeepONet, FNO, and PCA-Net provide tailored trade-offs between accuracy and computational cost for various applications.
• Training methods incorporate physics-informed losses, multi-fidelity strategies, and uncertainty quantification to enhance model robustness and interpretability.

Surrogate modeling with neural networks is a central methodology in computational science and engineering, providing data-driven, computationally efficient approximations to expensive simulation-based models, particularly those governed by partial differential equations (PDEs) or highly nonlinear black-box processes. Surrogate models replace the original simulator $\mathcal{O}$ with a neural network-based map $\mathcal{S}$ such that for an input $u$, $\mathcal{O}(u) \approx \mathcal{S}(u)$. This enables many-query tasks such as uncertainty quantification, real-time control, Bayesian inference, and optimization by dramatically reducing the computational cost of repeated model evaluation (Hoop et al., 2022).

1. Fundamental Concepts and Definitions

Surrogate modeling is the task of constructing a computationally cheap approximation $\mathcal{S}$ to an expensive operator $\mathcal{O}$, with $\mathcal{O}(u)$ solved via PDEs or other high-fidelity numerical models. Operator learning, a core trend in this domain, aims to approximate mappings $\Psi^\dagger: U \to V$ between infinite-dimensional function spaces. Typically, $U$ and $V$ are discretized, and neural networks are trained to minimize the empirical risk over datasets of the form $\mathcal{S}$0 sampled from a measure $\mathcal{S}$1 on $\mathcal{S}$2 (Hoop et al., 2022).

Surrogate neural networks have been successfully applied to black-box optimization (Ruff et al., 2023), closed-loop controller design (Hirt et al., 12 Dec 2025), sensitivity analysis (Sun et al., 2023), flow field prediction (Trávníková et al., 15 Jul 2025), and high-dimensional Bayesian inverse problems (Yan et al., 2019).

2. Neural Network Architectures for Surrogate Modeling

Multiple neural network architectures and operator-learning frameworks are utilized, each with differing representational capacity, cost, and accuracy trade-offs (Hoop et al., 2022, Sun et al., 2023, Propp et al., 2024):

Architecture	Input Representation	Output Strategy	Strengths
PCA-Net	Project inputs to PCA basis	Outputs in PCA basis	Efficient for smooth outputs, low cost for low output dimension
DeepONet	Branch + trunk nets for basis and location	Output as $\mathcal{S}$3	Universal approximation, handles function-to-function mapping
PARA-Net	Inputs as (basis coeffs, spatial coord)	Fully connected, evaluated at all pts	Flexible pointwise queries but high cost for full fields
Fourier Neural Operator (FNO)	Lifting+Fourier+projection	Spectral convolution layers	Spectral bias, excels for smooth PDEs, efficient with FFTs
Dense Encoder-Decoder CNNs	Field inputs (images, meshes)	Dense+skip connections	Effective on high-dimensional grids (e.g., 2D/3D fields)
RBF/Kernel NNs	Parameter vectors	Kernel-weighted outputs	Fast for parametric surrogates, interpretable active sampling

Architectural and design choices (e.g., number of layers/channels, basis size, spectral vs. spatial representation, activation functions) directly affect accuracy, computational efficiency, and suitability for smooth vs. non-smooth outputs (Hoop et al., 2022, Sun et al., 2023, Sunil et al., 2024).

3. Training, Losses, and Physics Incorporation

Standard surrogate modeling uses supervised learning with data generated from high-fidelity simulations:

• Empirical loss minimization of the form $\mathcal{S}$4 (Hoop et al., 2022).
• Mean absolute error (MAE) is often employed for non-Gaussian or highly nonlinear black-box settings (Ruff et al., 2023).
• Physics-informed losses augment data-driven objectives with PDE residual terms or weak-form constraints, as in PINNs and FE-PINNs (Wong et al., 2021, Sunil et al., 2024, Trávníková et al., 15 Jul 2025). General form:

$\mathcal{S}$5

where $\mathcal{S}$6 penalizes PDE residuals, and $\mathcal{S}$7 balances data fidelity and physics prior.

• Multi-fidelity strategies combine cheap, abundant low-fidelity data with scarce, expensive high-fidelity samples using nested subnetworks or residual neural processes. Examples include MF-DNN (Li et al., 2023) and Multi-Fidelity Residual Neural Processes (MFRNP) (Niu et al., 2024).
• Active learning and adaptive refinement selectively enrich the surrogate's training data in high-error regions by error indicators or acquisition functions (Kapadia et al., 2023, Yan et al., 2019).

Physical boundary conditions and properties can be enforced strictly (via exact map rewriting) or weakly (via loss penalties) depending on the architecture and application (Sunil et al., 2024, Wong et al., 2021).

4. Cost–Accuracy Tradeoffs and Benchmarking

The effectiveness of a surrogate model is determined by both its predictive accuracy and its cost (FLOPs or wall time per evaluation). Key benchmarks and results are as follows (Hoop et al., 2022):

• Cost scaling: FNO attains lowest error for given computational budget in problems with smooth output fields, with cost scaling as $\mathcal{S}$8 per evaluation. PCA-Net/DeepONet achieve similar accuracy for lower-dimensional outputs at lower evaluation cost. PARA-Net is inefficient for full-field outputs due to per-grid-point evaluations.
• Empirical convergence: For smooth 2D PDEs, surrogate error can decay as $\mathcal{S}$9 (number of training data), saturating when network expressivity is exhausted.
• Spectral bias: FNO and PCA-Net are effective when PDE solutions are smooth or bandlimited; DeepONet is more robust to non-smooth outputs due to learned bases.

Practical guidelines include restricting network size to avoid overfitting (small data), utilizing FFT-friendly architectures (FNO) when applicable, and leveraging physics-informed training to greatly reduce required labeled samples (PINN: ~O(10^2) labels yield O(1%) accuracy for flow fields) (Trávníková et al., 15 Jul 2025, Wong et al., 2021, Hoop et al., 2022).

5. Multi-Fidelity and Transfer Learning Approaches

Multi-fidelity methods exploit low-fidelity simulations or data to improve efficiency and scalability:

• Composite surrogates: Two-stage models where a low-fidelity DNN informs a correction network for mapping to high-fidelity outputs; the correction can be additive, multiplicative, or general nonlinear (Li et al., 2023, Yan et al., 2019).
• Residual neural processes: MFRNP explicitly models the residual between aggregated low-fidelity neural process predictions and the high-fidelity solution, achieving up to 90% lower error than classic multi-fidelity Gaussian processes on PDE and climate benchmarks (Niu et al., 2024).
• Transfer learning on dimensionally-reduced problems: Dense encoder-decoder CNNs pre-trained on lower-dimensional approximations (e.g., 1D slices) enable substantial data-generation cost savings and lower error than single-fidelity models under fixed computational budgets (Propp et al., 2024).

In high-dimensional settings, Bayesian neural network surrogates and their infinite-width counterparts (neural tangent kernel GPs) have demonstrated scalability and effective uncertainty quantification where traditional Matérn-kernel GPs fail (Hirt et al., 12 Dec 2025).

6. Uncertainty Quantification and Interpretability

Many surrogate modeling frameworks provide built-in or additional mechanisms for quantifying predictive uncertainty:

• Probabilistic neural networks (PNNs): Output both predictive mean and variance; negative log-likelihood training yields interpretable confidence intervals. Uncertainty grows in regions of data paucity or noise, supporting sensor placement and active learning decisions (Maulik et al., 2020).
• Bayesian neural networks (BNNs): Ensembling and anchored schemes (with functional priors) permit rigorous epistemic UQ, crucial for out-of-distribution extrapolation and physical systems with rare events (Ghorbanian et al., 2024).
• NTK-based surrogates: Empirical and trace neural tangent kernels enable faithful, transparent surrogate emulators for explanation and attribution tasks, correlating closely with true model behavior (Engel et al., 2023).

Interpretability is reinforced by low-rank decompositions (functional surrogates with interpretable spatial bases (Jeon et al., 26 Mar 2025)) and explicit basis network separation.

7. Limitations and Open Challenges

Despite recent successes, open challenges remain:

• Lack of general error theory: Rigorous bounds on surrogate test error versus network size, data volume, and discretization granularity exist mainly for linear operators; nonlinear PDEs and operator networks (DeepONet, FNO) lack comprehensive theoretical guarantees (Hoop et al., 2022).
• Hyperparameter and architecture selection: No widely-accepted techniques for optimal branch/trunk sizing (DeepONet), spectral/truncation parameters (FNO), or balancing subnetwork contributions (multi-fidelity surrogates).
• High-dimensional, unstructured domains: Scaling architectures (e.g., FNO) and accurately transferring between mesh resolutions or geometric parametrizations remain areas of active research (Sunil et al., 2024).
• Robust UQ and out-of-distribution reliability: While BNNs and PNNs advance this front, routine, well-calibrated uncertainty estimation in physically-constrained regimes is still a developing area (Ghorbanian et al., 2024, Maulik et al., 2020).

Continued progress requires advances in architecture theory, adaptive training strategies, uncertainty estimation, and domain-informed (physics or prior) inductive bias integration across large-scale, multi-physics, and multi-fidelity simulation regimes.

---

References

• (Hoop et al., 2022) "The Cost-Accuracy Trade-Off In Operator Learning With Neural Networks"
• (Ruff et al., 2023) "Surrogate Neural Networks for Efficient Simulation-based Trajectory Planning Optimization"
• (Wong et al., 2021) "Improved Surrogate Modeling of Fluid Dynamics with Physics-Informed Neural Networks"
• (Sun et al., 2023) "Surrogate Neural Networks to Estimate Parametric Sensitivity of Ocean Models"
• (Sunil et al., 2024) "FE-PINNs: finite-element-based physics-informed neural networks for surrogate modeling"
• (Niu et al., 2024) "Multi-Fidelity Residual Neural Processes for Scalable Surrogate Modeling"
• (Trávníková et al., 15 Jul 2025) "Quantifying data needs in surrogate modeling for flow fields in 2D stirred tanks with physics-informed neural networks (PINNs)"
• (Li et al., 2023) "Surrogate modelling and uncertainty quantification based on multi-fidelity deep neural network"
• (Yan et al., 2019) "An adaptive surrogate modeling based on deep neural networks for large-scale Bayesian inverse problems"
• (Hirt et al., 12 Dec 2025) "High-Dimensional Surrogate Modeling for Closed-Loop Learning of Neural-Network-Parameterized Model Predictive Control"
• (Ghorbanian et al., 2024) "Empowering Bayesian Neural Networks with Functional Priors through Anchored Ensembling for Mechanics Surrogate Modeling Applications"
• (Maulik et al., 2020) "Probabilistic neural networks for fluid flow surrogate modeling and data recovery"
• (Propp et al., 2024) "Transfer Learning on Multi-Dimensional Data: A Novel Approach to Neural Network-Based Surrogate Modeling"
• (Engel et al., 2023) "Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models"
• (Jeon et al., 26 Mar 2025) "Interpretable Deep Neural Network for Modeling Functional Surrogates"
• (Ogren et al., 2020) "Surrogate Modeling of the CLIC Final-Focus System using Artificial Neural Networks"
• (Kapadia et al., 2023) "Active-Learning-Driven Surrogate Modeling for Efficient Simulation of Parametric Nonlinear Systems"
• (Rosso et al., 29 Jul 2025) "Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling"
• (Li et al., 2022) "Gaussian Process Surrogate Models for Neural Networks"