Transformer Embedding Surrogates

• Transformer embedding surrogates are data-driven models that convert discrete symbolic closure expressions into continuous embeddings to mimic CFD error landscapes.
• They employ Gaussian process regression and acquisition functions, significantly reducing costly CFD evaluations by screening candidate models.
• This approach enables multi-objective, Pareto-optimal optimization in turbulent flow simulations while preserving interpretability and uncertainty quantification.

A transformer embedding surrogate in modern scientific computation denotes an approach that utilizes learned function approximators—known as surrogates—to mimic the input–output map or error response of full high-fidelity CFD or physical solvers within symbolic or data-driven closure model discovery workflows. Rather than employing expensive direct evaluations for each candidate closure or model correction, the surrogate provides fast, differentiable, and uncertainty-quantified estimates of the governing system’s response, significantly accelerating symbolic regression-based model search and enabling practical multi-objective and multi-model optimization in the CFD-driven modeling paradigm.

1. Conceptual Basis of Surrogate-Augmented Symbolic Model Discovery

Transformer embedding surrogates are situated within the broader context of data-driven closure modeling for turbulent and multiphysics flows, where the generation of candidate algebraic models (often via symbolic regression or gene expression programming) is embedded in a physical simulation loop. In such settings, each candidate closure’s “fitness” or error is typically determined by running a truncated CFD or ROM solve to convergence. However, the prohibitive computational cost (hundreds to thousands of CFD solves per generation) motivates the replacement of direct simulation evaluation with an alternative: a surrogate model trained on the error or fitness landscapes already sampled during earlier CFD runs.

Surrogates of this class differ from standard response surfaces by operating on continuous embeddings of discrete symbolic expressions. Each symbolic closure model is mapped via a featurization procedure—such as averaging the expression’s output across a prescribed set of baseline states—into a vector representation amenable to classical regression or probabilistic machine learning models. The surrogate, typically a Gaussian process (GP) with a suitable kernel, is incrementally trained to predict the error and uncertainty for each new candidate. This architecture enables rapid screening of model populations and directs costly CFD evaluation only toward the most promising or uncertain candidates (Fang et al., 22 Dec 2025).

2. Workflow and Mathematical Formulation

The surrogate-augmented training workflow is characterized by cyclic interaction between symbolic model generation, surrogate evaluation, strategic CFD selection, and incremental updating:

1. Generation: Candidate symbolic closure expressions are created via gene expression programming (GEP) or other symbolic regression grammars.
2. Embedding: Each discrete symbolic expression is evaluated on a fixed batch of baseline CFD states; the mean outputs across these inputs are concatenated to form a continuous feature vector $x_k$.
3. Surrogate Prediction: The current surrogate (typically a GP using a rational quadratic kernel) predicts the mean error $\mu(x_k)$ and standard deviation $\sigma(x_k)$ for each candidate.
4. Selection: Candidates are prioritized for full CFD evaluation based on acquisition functions such as Lower Confidence Bound or Expected Improvement, possibly weighted for multi-objective balancing.
5. CFD Evaluation: Only a subset of candidates (those with lowest predicted error or highest uncertainty) are explicitly evaluated via full CFD, updating the training set.
6. Surrogate Update: The GP surrogate is retrained incorporating new CFD-evaluated data, refining the landscape for the next generation.
7. Optimization: Model fitness, as determined by a combination of surrogate and real CFD errors, guides evolutionary progress.

The multi-output extension replaces the scalar-target kernel with a matrix-valued kernel to enable simultaneous optimization of multiple training objectives, providing per-objective mean and variance estimates and supporting Pareto-front selection (Fang et al., 22 Dec 2025).

3. Embedding and Surrogate Architecture

A defining technical element is the transformation from discrete symbolic models to continuous embeddings. The process involves:

• For a candidate symbolic function $f_k$, evaluation at $m$ baseline inputs $\{(I_i, J_i)\}_{i=1}^m$ yielding pointwise outputs $y_i = f_k(I_i, J_i)$.
• The embedding for GP regression is $x_k = h(f_k) = \frac1m\sum_{i=1}^m y_i$, or an analogous vectorized summary.
• The surrogate model $\mathcal{S}: x \mapsto (\mu(x), \sigma(x))$ is a GP with training on previously full-simulated models.

For multi-objective cases, the kernel function becomes matrix-valued, producing a covariance block for each pair of objectives and enabling a vector-valued error and uncertainty prediction framework.

4. Selection Strategies and Multi-Objective Extension

Selection is guided by acquisition functions directly operating on the surrogate predictions:

• Lower Confidence Bound: $m_{\mathrm{LCB}}(x) = -\mu(x) + \beta \sigma(x)$ for $\beta\ge0$.
• Expected Improvement: $m_{\mathrm{EI}}(x)$ quantifies the expected error reduction over the current best, leveraging the predicted mean and variance.
• Convergence Weighting and Pareto Selection: These mechanisms exclude non-convergent or redundant candidates and ensure diverse coverage in the presence of multiple objectives (e.g., matching both velocity and temperature profiles in coupled flows).

Candidates above a threshold (fixed number, relative value, or Pareto-optimal with respect to multiple fitness objectives) are passed to full CFD; the remainder are evaluated solely via the surrogate (Fang et al., 22 Dec 2025).

5. Quantitative Impact and Performance Metrics

Extensive evaluations on canonical turbulence and coupling test cases illustrate the gains from surrogate augmentation:

• In vertical natural convection, the surrogate-augmented training reduced the required number of CFD solves from $\sim$2000 to $\sim$880 (56% reduction), while preserving the final minimal Nusselt-number error (from 19.4% baseline to 9.5% trained value).
• In horizontal mixed convection, training calls decreased from $\sim$1900 to $\sim$355 (81.3% reduction), with Nusselt-number errors improving from $\sim$70% to $<5\%$ across varying Richardson number cases.
• In a three-dimensional annulus, CFD solves dropped from $\sim$2350 to $\sim$1257 (46.5% reduction), without loss of prediction quality in velocity, temperature, or heat-flux metrics.
• Across all studied cases, the surrogate-based framework delivered 45–80% reduction in direct CFD cost while maintaining Pareto-front hypervolume coverage and the predictive accuracy of the purely CFD-trained models (Fang et al., 22 Dec 2025).

6. Interpretability, Generalization, and Limitations

Surrogate embedding preserves the core interpretability of symbolic closures, as the final models remain explicit algebraic expressions calibrated to high-fidelity objectives. Multi-objective extensions allow simultaneous closure of coupled fields (e.g., turbulent stress and heat flux), with solution diversity captured in the Pareto front.

Generalization is fostered through:

• Embedding snapshot and multi-parameter training strategies, in which Reynolds number or geometric parameters are included as terminal features in the symbolic grammar, and
• Continuous updating of the surrogate as the closure search traverses new regions of model space.

Notable limitations include the dependence on the quality of the baseline input state selection and the possibility that surrogate accuracy may degrade outside regions thoroughly sampled by true CFD runs. The embedding process must be engineered to avoid discarding essential nonlinear or regime-dependent closure behavior.

7. Future Prospects and Research Directions

The introduction of transformer embedding surrogates into CFD-driven symbolic closure discovery has enabled rapid, tractable exploration of complex multi-objective error landscapes in fluid modeling and related PDE systems. Further developments are anticipated in:

• Surrogate architectures for higher-dimensional closure spaces (e.g., tensorial SGS models for 3D flows),
• Adaptive strategies for embedding function selection, emphasizing regime coverage and low-dimensional summary informativeness,
• Integration with parameter-aware symbolic regression frameworks (such as Parameter-Aware Ensemble SINDy) to ensure both robustness and simultaneous parametric generality (Kang et al., 13 Aug 2025),
• Expansion of multi-objective frameworks for real-world, multi-physics and multi-fidelity closure optimization.

Empirical evidence suggests that such surrogates, when carefully embedded and updated, remove a fundamental computational bottleneck from data-driven closure discovery, accelerating the deployment of interpretable, high-fidelity models across turbulent, multiphysics, and plasma regimes (Fang et al., 22 Dec 2025).