Natural Language Prompting in AI

• Natural Language Prompting is a technique that uses plain text cues to instruct advanced AI models across diverse applications.
• Recent methodologies focus on prompt optimization, few-shot learning, and contextual calibration to boost model performance.
• Key applications include dialogue systems, content generation, and decision support, underscoring its growing significance in modern AI research.

A deep learning surrogate model is a parameterized neural mapping designed to emulate high-fidelity scientific simulation codes, complex stochastic simulators, or large-scale regression workflows with orders-of-magnitude greater computational efficiency. By training on a set of input–output pairs generated by the (often costly) reference model, deep surrogates can perform field or functional predictions, quantify uncertainties, and enable optimization, uncertainty quantification (UQ), and real-time inversion applications, often exceeding the limitations of traditional response surfaces or polynomial chaos approaches.

1. Mathematical Formulation and Model Architectures

A deep surrogate model typically learns a mapping

$f_{\theta}\!: \mathcal{X} \to \mathcal{Y}$

where $\mathcal{X} \subseteq \mathbb{R}^d$ are input vectors (physical or geometric parameters, fields, control signals) and $\mathcal{Y} \subseteq \mathbb{R}^m$ are outputs (QoIs, solution fields, functional statistics). Architectures are determined by input/output structure:

• Feed-forward MLPs for vector-valued maps: 4–10 hidden layers, hundreds to thousands of hidden units, ReLU or Swish activations, batch normalization, dropout, and sometimes ensemble averaging for robust prediction (Vardhan et al., 2022).
• Convolutional Networks (CNNs, U-Nets) for maps from fields to fields or images: architectures may utilize 2D/3D convolutions, encoder–decoder pathways, skip-connections, and, for time-dependent problems, recurrence modules such as ConvLSTM (Davis et al., 2023, Han et al., 2024).
• Normalizing flows for fully invertible, generative mapping between parameter and data space, with conditional transformations and autoencoder dimensionality reduction for high-dimensional outputs (Shen et al., 2024).
• Latent variable models (e.g., VAEs, conditional GANs): stochastic surrogates express $y = f_{\theta}(x, z)$ with $z\sim\mathcal{N}(0,I)$ as the source of aleatoric uncertainty (Yang et al., 2019, Thakur et al., 2021).
• Graph neural networks or message-passing decoders operate over unstructured spatial discretizations (finite elements, point clouds) where spatial invariance or mesh adaptivity is paramount (Meyer et al., 2023).

Selected surrogates are customized for physically structured outputs (e.g., field-wise conservation, geometry-aware representation) and for application contexts ranging from computational fluids to solid mechanics and operational electricity markets.

2. Training Procedures and Optimization

Surrogate modeling requires a representative dataset

$\mathcal{D} = \{ (x_i, y^*_i) \}_{i=1}^N$

where $y^*_i = G(x_i)$ is computed by the full-order or stochastic simulator $G$.

• Offline batch training is the traditional approach, with data precomputed and stored on disk, followed by mini-batch SGD or Adam optimization on a standard loss (MSE, MAE, negative log-likelihood, or adversarial objectives) (Vardhan et al., 2022, Davis et al., 2023, Dong et al., 2021).
• Online or streaming learning mitigates I/O bottlenecks by coupling a parallel simulation fleet directly to the training process, enabling exposure to $10^4$–$10^5$ unique samples and substantially boosting model generalization (Meyer et al., 2023).
• Active learning employs acquisition functions (e.g., ensemble predictive variance) to select informative next simulations, sharply reducing the number of expensive reference model evaluations required for target accuracy (Pestourie et al., 2020).
• Multi-fidelity and adaptive correction approaches use abundant low-cost or low-fidelity data with a small set of high-fidelity samples, either by hierarchical input fusion (Yang et al., 2019), local shallow correction networks (Yan et al., 2019), or composite network architectures (Jeon et al., 26 Mar 2025).
• Loss selection: For deterministic surrogates, mean squared or absolute error is standard; for stochastic surrogates, loss functions target full conditional distributions using, e.g., maximum mean discrepancy (CMMD) (Thakur et al., 2021), adversarial density matching (Yang et al., 2019), or evidence lower bound maximization with variational approximations (Islam et al., 22 Jan 2025). For functional surrogates, expansions in basis and coefficient networks are employed (Jeon et al., 26 Mar 2025).

Regularization (L2 or weight decay), dropout, early stopping, and ensemble averaging stabilize optimization and improve generalization, especially in small-data regimes.

3. Uncertainty Quantification and Stochastic Surrogates

Rigorous UQ is central to surrogate modeling in engineering and scientific settings. Deep learning surrogates provide several mechanisms for quantifying epistemic and aleatoric uncertainty:

• Latent-variable models with conditional sampling: Multiple forward passes with different latent $z$ yield predictive distributions and non-Gaussian/multimodal statistics (Yang et al., 2019, Thakur et al., 2021, Islam et al., 22 Jan 2025).
• Variational inference and Bayesian networks: Surrogates place priors over weights and approximate the posterior via mean-field or more structured variational approaches, allowing epistemic uncertainty to be propagated through predictions (Islam et al., 22 Jan 2025).
• Monte Carlo dropout: Dropout at training and inference time approximates Bayesian model averaging, providing predictive intervals and credible sets (Jeon et al., 26 Mar 2025).
• Normalizing flows: Fully invertible surrogates enable exact sampling from the learned output density, providing calibrated voxel- or pixel-wise uncertainty and supporting reverse parameter inference (Shen et al., 2024).

Uncertainty metrics include ensemble variance, predictive intervals, coverage probability, Hellinger distance between predicted and true distributions, and calibration curve analysis.

4. Specialized Surrogate Methods and Physical Constraints

Deep surrogates in scientific domains must often encode domain knowledge and physical structure:

• PDE-constrained surrogates: Surrogates for parametric PDEs learn field-wise outputs and sometimes incorporate physics-informed losses (e.g., residual penalties, local conservation) (Song et al., 2021, Davis et al., 2023).
• Geometric surrogates: Inputs such as patient-specific 3D geometries for CFD are parameterized using statistical shape models (PCA/LDDMM) or autoencoders, with DNNs mapping shape latents to field-latent outputs (Du et al., 2022).
• Multi-fidelity fusion: By fusing low- and high-fidelity data (e.g., flow-only plus flow–geomechanics runs), surrogates can minimize the number of expensive simulations required for each solution component (Han et al., 2024).
• Constraint-handling in optimization: Analytical surrogates support embedding in optimization pipelines by providing analytic Jacobians and Hessians for surrogate constraints, enabling interior-point or gradient-based solvers for operational planning (Qiu et al., 2020).
• Surrogates for functional outputs: In problems with outputs that are functions over space, surrogates model the output as a sum $\sum_k B_k(z) \cdot \eta_k(s)$ using DNNs for both $B$ (basis functions of global input) and $\eta$ (coefficient functions over space), preserving interpretability (Jeon et al., 26 Mar 2025).

Surrogates can be applied to support design optimization, inverse modeling, and data assimilation frameworks (e.g., Bayesian MCMC and EnKF), enabling tractable solution of problems previously limited by reference model cost (Millevoi et al., 2024, Han et al., 2024, Yan et al., 2019).

5. Performance, Accuracy, and Computational Impact

Surrogate models achieve high speedups and, with enough data and appropriate architecture/hyperparameters, high predictive accuracy:

• Typical speedup: Deep surrogates are routinely $10^3$–$10^5\times$ faster per inference than finite element, finite volume, or coupled multi-physics simulations (Vardhan et al., 2022, Han et al., 2024, Song et al., 2021).
• Accuracy: Relative test-set errors of $<5\%$ are reported for scalar regression targets (pressure vessel stress), sub-degree RMSE for spatial temperature fields (Davis et al., 2023), and $<4\%$ median error for 3D flow/geomechanical response surrogates (Han et al., 2024). Surrogates also demonstrate robust recovery of full output distributions and quantiles in stochastic settings (Thakur et al., 2021, Islam et al., 22 Jan 2025).
• Sample efficiency: Active-learning approaches can reduce high-fidelity query counts by up to $12\times$ for fixed error (Pestourie et al., 2020); multi-fidelity and adaptive DNNs reduce the number of required expensive evaluations for accurate posterior estimation by two orders of magnitude (Yan et al., 2019).
• Limitations: Surrogates may underperform outside trained domains, underrepresent rare events, or have diminished accuracy in extreme or extrapolative scenarios. Complex physical structures (e.g., time-dependent faults, nontrivial b.c.) require specialized or retrained surrogates (Millevoi et al., 2024).
• Uncertainty-aware surrogates ensure better error calibration and credible interval validity, essential for UQ and risk-critical uses (Islam et al., 22 Jan 2025, Jeon et al., 26 Mar 2025, Shen et al., 2024).

6. Practical Applications and Research Directions

Deep learning surrogate models permeate a broad range of computational science and engineering:

• Optimization/design: Surrogates accelerate PDE-constrained optimization in metasurface photonics (Pestourie et al., 2020), pressure vessel design (Vardhan et al., 2022), and real-time reservoir control (Chen et al., 2024).
• Uncertainty quantification: Surrogates enable tractable forward UQ and global sensitivity, e.g., via Monte Carlo or Sobol indices in ceramic aerogel design (Islam et al., 22 Jan 2025).
• Inverse modeling/data assimilation: Surrogates facilitate rapid history matching and Bayesian inversion for CO₂ storage, seismic fault activation, and hydromechanical systems, integrating into MCMC and EnKF workflows (Han et al., 2024, Millevoi et al., 2024, Yan et al., 2019).
• Scientific discovery: Surrogates allow large-scale parametric studies and functional relationship discovery (e.g., drag scaling laws in shallow water hydraulics (Song et al., 2021)) and support interpretability analysis via learned basis or coefficient functions (Jeon et al., 26 Mar 2025).
• Stochastic simulation: Deep generative surrogates (GANs, flows) capture the distributional variability in stochastic simulators and real-world processes (Thakur et al., 2021, Shen et al., 2024).

Current research seeks to expand surrogate generality (physics-informed networks, equivariant architectures), widen uncertainty quantification rigor, integrate richer physical constraints, and further enhance data efficiency through active, adaptive, and multi-fidelity learning strategies.