Surrogate Model Construction

• Surrogate Model Construction is a method to create computationally efficient models that approximate expensive simulators using regression or machine learning techniques.
• Techniques such as ridge regression, random forests, and CNNs offer diverse trade-offs in speed, scalability, and accuracy.
• Hybrid and ensemble approaches, including time-step expert selection and weighted aggregation, significantly reduce RMSE while offering substantial speed gains.

A surrogate model is a computational or statistical construct designed to approximate the response of a more complex and expensive-to-evaluate simulator with respect to a set of controllable inputs. Surrogate modeling is critical in domains where iterative simulation, optimization, or uncertainty quantification over high-dimensional parameter spaces is computationally prohibitive. Construction of a surrogate model involves selecting a functional representation (e.g., regression, machine learning, ensemble), defining loss functions, optimizing hyperparameters, and validating the surrogate by comparison to simulation output. Recent advances emphasize hybridization, active learning, error-aware design, and adaptive refinement to achieve high accuracy at minimal computational cost (Carlier et al., 2022).

1. Formal Problem Statement and Loss Functions

Let $X = \{x_1,\ldots,x_K\}$, $x_k \in \mathbb{R}^M$ denote a training set of input configurations for an expensive deterministic simulator $S^*$, and $Y = \{y_1, \ldots, y_K\}$, $y_k = (y^{(1)}_k, \ldots, y^{(N)}_k)$ the corresponding vector outputs, where each $y^{(n)}_k \in \mathbb{R}^{T_n}$ is a multivariate time series of length $T_n$. The surrogate approximates $S^*$ via a mapping $\hat S: x \mapsto \hat y \approx S^*(x)$. All methods minimize an empirical risk based on squared error or RMSE: $$L(\hat Y, Y) = \frac{1}{K} \sum_{k=1}^K \|\hat Y_k - Y_k\|^2,\qquad \text{RMSE}(u, v) = \sqrt{\frac{1}{n} \sum_{i=1}^n (u_i-v_i)^2},$$ typically aggregated over all scenarios and time steps (Carlier et al., 2022).

2. Classical Surrogate Modeling Techniques

Surrogate construction employs several canonical statistical learning models, each targeting the regression task $x \mapsto y$:

• Ridge Regression (Kernel Ridge Variant): Fits a weight matrix $\Beta\in\mathbb{R}^{M\times T_{\text{total}}}$ by minimizing

$$J(\Beta) = \|Y - X\Beta\|^2_F + \lambda \|\Beta\|^2_F,$$

with $\lambda$ tuned by validation (Carlier et al., 2022). Used for high-dimensional, linear-response surrogates.

• Random Forests (Ensemble of Decision Trees): Construct $B$ regression trees, with predictions averaged:

$\hat y_{RF}(x) = \frac{1}{B} \sum_{b=1}^B T_b(x).$

Hyperparameters include tree count $B$, maximum depth, and feature-split size $m_{\text{try}}$; either as a single multi-output forest or a forest per output series (Carlier et al., 2022).

• Convolutional Neural Networks (CNNs): Input $x\in\mathbb{R}^M$ is transformed through an embedding to a 1D pseudo-time series, convolved by three layers (filter sizes 32–128, kernel size 3, stride 1, ReLU activation), then mapped by fully connected layers to the output vector. Optimized with Adam, $L_2$ loss, early stopping on validation RMSE (Carlier et al., 2022).

Each approach yields different trade-offs in speed, scalability, and accuracy, motivating hybridization.

3. Hybrid and Ensemble Surrogate Construction

No single surrogate dominates all outputs and scenarios; hybridization leverages diversity by combining predictions:

• Weighted Ensemble (Stacking/Blending): A convex combination

$$\hat y_{hybrid}(x) = \sum_{i} w_i \hat y_i(x), \quad \sum_i w_i = 1, \; w_i \geq 0,$$

with $w_i$ optimized to minimize validation loss.

• Time-Stepwise Expert Selection: For each output time step $t$, select the single best expert (model) from a candidate pool, based on validation error—yielding an adaptive, hard-selection hybrid.
• Exponential Weighted Aggregation (EWA): Sequentially updates expert weights at each time step according to loss incurred, with a learning rate $\eta$:

$$w_i \leftarrow \frac{\exp(-\eta \text{cumulative}\,\ell_{i})}{\sum_{j} \exp(-\eta \text{cumulative}\,\ell_{j})}.$$

Combines predictions with dynamic soft weights (Carlier et al., 2022).

Performance is assessed on unseen test data, with hybrids (especially hard selection) generally yielding lowest RMSE.

4. Training Protocols, Validation, and Performance Metrics

Training populates the surrogate with a large, diverse sample ($\sim 10^4$ runs), holds out validation sets for hyperparameter tuning and hybrid fitting, and tests final generalization on unseen data. Empirical protocols report:

• Training times: ridge/krr (0.22s), 4-rf (53s), CNN (59min for 100 time series)
• Prediction times: ridge (0.02s), 4-rf (0.15s), CNN (1.39s)
• RMSE on test data: CNN (1.77e-2), 4-rf (3.42e-2), Hybrid 2 (1.25e-2) (Carlier et al., 2022)

Hybrid 2 (time-stepwise expert selection among CNN, 4-rf, pca-rf) achieves a 29% reduction in mean RMSE compared to the best single model (CNN), and delivers $\sim$120$\times$ speedup over the original simulator at large scale.

5. Best Practices and Recommendations

Key empirical findings and guidelines (Carlier et al., 2022):

• Model diversity: Benchmark multiple base learners (linear, tree, neural, PCA-enhanced) using the same RMSE criterion.
• Validation separation: Always reserve an independent validation set for tuning ensemble weights and selection logic.
• Per-time-step adaption: Use time-step expert selection to adapt to phase-dependent surrogate difficulty.
• Speed/accuracy tradeoff: For iterative contexts (ABC, large-batch runs), select moderate-accuracy, high-speed surrogates.
• Generalization checks: Mandatory independent test set evaluation to guard against overfitting to validation data.

6. Significance in Large-Scale Simulation Workflows

The hybrid surrogate modeling approach outlined delivers two orders of magnitude speedup while achieving low RMSE across high-dimensional inputs and multivariate time series outputs (Carlier et al., 2022). The methodology accommodates heterogeneous data, variable signal phases, and rapidly changing simulation conditions, matching industry requirements for advanced driver-assistance systems and other engineering domains that demand scalable, reliable surrogates for computational simulators.

7. Critical Evaluation and Limitations

The results demonstrate that no universal method suffices across all tasks; hybridization is not only beneficial but necessary. Hard-selection hybrids outperform even sophisticated soft-weighted aggregations on unseen data, indicating sensitivity to overfitting in the latter. Tradeoffs between computational cost and predictive accuracy must be explicitly navigated, as higher accuracy models demand more substantial offline training time (Carlier et al., 2022). Expert selection per output phase can be essential in complex, non-stationary surrogate targets.

---

References:

Construction of a Surrogate Model: Multivariate Time Series Prediction with a Hybrid Model (Carlier et al., 2022)