Multi-fidelity Surrogates

• Multi-fidelity surrogates are predictive models that fuse low-cost, low-accuracy data with expensive high-fidelity data to enable efficient design exploration.
• The LS-MFS framework applies linear regression by incorporating a scaling factor and discrepancy function, yielding robust parameter estimates and rapid evaluations.
• These models are widely used for design optimization, uncertainty quantification, and sensitivity analysis, offering a computationally lightweight alternative to more complex Bayesian approaches.

Multi-fidelity surrogates are predictive models that synthesize data from computational or experimental sources with varying accuracy and cost. Their primary purpose is to enable efficient and accurate emulation of expensive high-fidelity models by leveraging abundant low-cost, lower-fidelity data. These frameworks have found wide adoption in design optimization, uncertainty quantification, sensitivity analysis, and reliability assessment in computational science and engineering. The following provides a rigorous, detailed exposition of the least-squares multi-fidelity surrogate (LS-MFS) framework, as introduced and analyzed by Zhang et al. (Zhang et al., 2017).

1. Linear Regression Formulation

The central structure of LS-MFS is the approximation of a high-fidelity model $f_H(x)$ using both a scaled low-fidelity model $f_L(x)$ and a discrepancy function expressed in a user-specified basis. The surrogate takes the form: $$f_H(x) \approx \rho\, f_L(x) + \delta(x) \qquad (1)$$ where $\rho$ is a scale factor bridging the trend between fidelities, and $\delta(x)$ is the modeling discrepancy expanded as

$\delta(x) = \sum_{i=1}^p b_i\,X_i(x)$

with $X_1(x), \ldots, X_p(x)$ denoting known basis functions (e.g., polynomials), and $b_i$ their coefficients. Thus, the total surrogate becomes

$$f_H(x) \approx \rho\, f_L(x) + \sum_{i=1}^p b_i\,X_i(x) \qquad (2)$$

This proposal regards $f_L(x)$ as a basis element in the space spanned by the regression model, with $\rho$ as its regression coefficient.

2. Design Matrix and Parameter Estimation

Given $n$ high-fidelity observations $\{(x^{(j)}, y_H^{(j)})\}_{j=1}^n$, one constructs the response vector $y_H = (y_H^{(1)}, ..., y_H^{(n)})^T$. The design matrix $D \in \mathbb{R}^{n \times (p+1)}$ is built as

$$D = \begin{bmatrix} f_L(x^{(1)}) & X_1(x^{(1)}) & \cdots & X_p(x^{(1)}) \ \vdots & \vdots & & \vdots \ f_L(x^{(n)}) & X_1(x^{(n)}) & \cdots & X_p(x^{(n)}) \end{bmatrix}$$

The stacked parameter vector is $\beta = [\rho, b_1, \ldots, b_p]^T$. The regression equation reads

$D\,\beta = y_H + \epsilon$

where $\epsilon$ represents residuals.

The parameter solution is given by the normal equations,

$\hat{\beta} = (D^T D)^{-1} D^T y_H$

provided $D$ has full column rank ($p+1$). This guarantees uniqueness and stability in the parameter estimates. This approach is extendable: for multi-level fidelity schemes, one adds more columns (e.g., $f_{L2}(x)$, $f_{L3}(x)$) with associated scale factors.

3. Extensions and Algorithmic Variants

Several modifications are supported:

• The basis $\{X_i(x)\}$ can be polynomials, radial basis functions, splines, wavelets, or arbitrary feature transformations.
• Regularization (ridge, lasso) can be introduced for noisy or collinear settings.
• For non-polynomial or heavy-tailed errors, robust regression (e.g., Huber loss) or Bayesian linear models are appropriate. Ordinary least squares provides maximum-likelihood estimation under Gaussian noise.
• The core methodology generalizes to nonlinear regression models, such as kernel machines or neural networks, by including $f_L(x)$ as an input feature.
• The method is readily extensible to more than two fidelities.

4. Comparison with Gaussian-Process-Based Approaches

Bayesian multi-fidelity surrogates (e.g., Kennedy–O’Hagan models with Gaussian-process priors) provide:

• Predictive means and covariances, enabling uncertainty quantification.
• Flexible handling of spatial correlations and model discrepancies.

However, Gaussian-process methods require nonlinear hyperparameter optimization (typically by maximum likelihood), which is computationally intensive and susceptible to local minima—especially under noisy data or small sample sizes. In contrast, LS-MFS offers:

• A single closed-form estimation step.
• Robustness under small $n$ or high-data-noise conditions.
• Low computational overhead and direct extensibility to arbitrary regression models.

The primary limitations of LS-MFS are absence of built-in spatial correlation modeling and the need to reconstruct variances/intervals from classical regression theory rather than inheriting them from the surrogate.

5. Application Areas

LS-MFS is applicable to a range of tasks:

• Prediction variance is computed by regression theory, e.g.,

$$\operatorname{Var}(\hat{y}) = \sigma^2 (D^T D)^{-1} D^T D (D^T D)^{-1}$$

where $\sigma^2$ is the sample variance of residuals.

• Design of experiments: D-optimal design can be pursued by maximizing $\det(D^T D)$.
• Uncertainty propagation: once trained, the surrogate enables rapid sampling for downstream uncertainty quantification.
• Design optimization: the surrogate’s algebraic form is differentiable and can be efficiently embedded in gradient-based optimization loops.

6. Numerical Example and Performance

Zhang et al. present a worked example involving the function

$$f_H(x_1, x_2) = 100\frac{1 - \exp(-2x_2)}{300x_3 + 1900x_2 + 2092x_1 + 60} + 100\frac{1 - \exp(-2x_2)}{100x_3 + 500x_2 + 4x_1 + 20}$$

with additive Gaussian noise. The low-fidelity model is a perturbation of $f_H$. Using minimal (2×2, 3×3) factorial designs and polynomial bases for the discrepancy, the LS-MFS dramatically outperforms a pure high-fidelity polynomial response surface: at $n = 4$ LS-MFS achieves relative RMSE $\sim 9\%$ compared to $44\%$ for the high-fidelity surface; at $n = 9$ the advantage grows to $1.5\%$ vs. $26\%$.

7. Significance, Limitations, and Outlook

LS-MFS provides a mathematically transparent, computationally lightweight fusion of low- and high-fidelity models. Its conceptual simplicity—treating $f_L(x)$ as a regression basis and fitting all coefficients in a single linear system—results in unique, interpretable models under standard rank assumptions and facilitates rapid experimentation and design studies.

Key drawbacks are its lack of automatic spatial correlation modeling and the absence of direct Bayesian intervals, limiting its performance for non-polynomially behaving functions or in applications demanding rigorous probabilistic uncertainty quantification. In such cases, advanced Bayesian frameworks may be preferred, despite their greater computational complexity.

A plausible implication is that for rapid design-space exploration or in settings with extremely limited high-fidelity data, LS-MFS may serve as a robust, first-line surrogate, delegating to more sophisticated models only when strict spatial variance modeling or full Bayesian inference is necessary (Zhang et al., 2017).