Kernel Surrogate Models

• Kernel Surrogate Models are data-driven function approximators built on RKHS principles that offer mesh-free interpolation with theoretical error bounds.
• They unify classical interpolation techniques with methods like support vector and Gaussian process regression for robust simulation and optimization.
• Greedy algorithms and sparsity techniques enhance efficiency and scalability, making these models competitive with neural surrogates in high-dimensional settings.

Kernel surrogate models are data-driven function approximators constructed within the framework of reproducing kernel Hilbert spaces (RKHS), leveraging positive-definite kernels to interpolate or regress expensive-to-evaluate black-box functions or simulators. They provide mesh-free, highly flexible, and theoretically grounded surrogates suited to deterministic, stochastic, and high-dimensional problems across optimization, scientific computing, and machine learning. Kernel surrogate modeling unifies classical interpolation, regularized approximation, support vector regression, and Gaussian-process regression, and is competitive with, or provides efficient alternatives to, contemporary neural surrogates in applications where sample efficiency, interpretability, or theoretical error guarantees are critical.

1. Mathematical Foundations and Core Framework

Kernel surrogate models are fundamentally built on RKHS theory. For a domain $\Omega\subset\mathbb{R}^d$ and a positive-definite kernel $k:\Omega\times\Omega\to\mathbb{R}$, the kernel surrogate $s(x)$ for a function $f:\Omega\to\mathbb{R}$ given $m$ samples $\{(x_i,f(x_i))\}_{i=1}^m$ is

$s(x) = \sum_{i=1}^m \alpha_i\,k(x,x_i)$

where the coefficients $\alpha$ are determined by solving

$K \alpha = f(X)$

with Gram matrix $K_{ij}=k(x_i,x_j)$. This linear system arises either from strict interpolation or regularized regression (ridge/Tikhonov). Extensions to vector-valued outputs $f:\Omega\to\mathbb{R}^q$ involve replacing scalars by vectors in $\alpha_i$.

Core properties:

• Mesh-free (scattered data): Do not require tensor or simplicial grids.
• RKHS error bounds: For $f\in\mathcal{H}_k$, the pointwise error satisfies $|f(x)-s(x)|\le\|f\|_{\mathcal{H}_k}P_X(x)$, where the power function $P_X(x)$ quantifies the interpolation error and vanishes as the fill distance decreases.
• Spectral characterization: By Mercer's theorem, smooth kernels permit eigenfunction expansions, linking surrogate expressivity directly to kernel smoothness and design.

Sparse and scalable variants are constructed via greedy basis selection and regularization (VKOGA, f/P/P-greedy, KEA) (Santin et al., 2019, Haasdonk et al., 2020, Wenzel et al., 2024).

2. Kernel Surrogate Models in Optimization

In global and local Bayesian optimization, as well as expensive black-box function minimization, kernel surrogates approximate the unknown objective and its derivatives, guiding the search with theoretically quantifiable error bounds. The Hermite kernel surrogate interpolant, in particular, incorporates both function and gradient information, yielding a model

$$s(x) = \sum_{i=1}^m \alpha_i k(x,x_i) + \sum_{i=1}^m \beta_i^T \nabla_y k(x,y)|_{y=x_i}$$

where the interpolation system simultaneously enforces $s(x_i) = f(x_i)$, $\nabla s(x_i) = \nabla f(x_i)$ (Ullmann et al., 2 Jul 2025). The resulting linear system uses a block Gram matrix. This formulation provides high-fidelity local models for trust-region methods.

Adaptive trust-region methods based on Hermite kernel surrogates exploit explicit RKHS-based error bounds (power-function) to define error-aware regions, guaranteeing that model minimization aligns with true objective decrease. Under standard convexity and smoothness assumptions, the iterates converge to stationary points. Empirical studies confirm that Hermite-kernel trust-region methods can achieve 20–40% reductions in high-fidelity evaluation counts compared to L-BFGS-B and generic trust-region SQP solvers in medium- to high-dimensional PDE-constrained optimization problems (Ullmann et al., 2 Jul 2025).

3. Variable Selection and High-Dimensional Structure Discovery

High-dimensional kernel surrogate modeling is challenged by the curse of dimensionality and overparameterization. Optimal kernel learning approaches construct surrogates by convex combinations of kernels over low-dimensional input subsets: $$k_{opt}(x,x') = \sum_{j=1}^m w_j k^{(S_j)}(x_{S_j},x'_{S_j}), \quad w_j\ge0, \quad \sum_j w_j=1$$ where each $k^{(S_j)}$ acts only on a coordinate subset. The weights $w_j$ are found by penalized marginal likelihood optimized with an $\ell_1$ penalty to enforce sparsity and heredity: $$\min_{w} -\log\det(K_w+\sigma^2I) + y^T(K_w+\sigma^2I)^{-1}y + \lambda\|w\|_1$$ The Fedorov–Wynn algorithm selects kernels (additive and interaction terms) in a stagewise fashion, with strong or weak heredity to control high-order interactions and restrict model complexity (Kang et al., 23 Feb 2025).

This framework enables automated variable selection, interpretable ANOVA-like decomposition, and significant improvements in predictive accuracy, especially when only a low-dimensional subset of variables is active. Empirical benchmarks (Michalewicz, Borehole, satellite-drag) show recovered active sets and improved RMSE compared to MLE-fitted GPs and alternative surrogates.

4. Sparsity, Greedy Algorithms, and Model Finetuning

Sparse kernel surrogate models are achieved by basis selection via greedy algorithms:

• f-greedy: Selects the point with the largest residual error.
• P-greedy: Maximizes the power function.
• f/P-greedy: Balances accuracy and stability (Santin et al., 2019, Haasdonk et al., 2020).
• Kernel Exchange Algorithms (KEA): Interleaves removal of weak centers and insertion of strong ones, finetuning a fixed-size subset, and delivering up to 86% reduction in the maximum test error (mean improvement of 17%) (Wenzel et al., 2024).

The stabilized (γ-restricted) VKOGA additionally restricts the candidate set for new centers based on the current power function, yielding more uniform, stable, and accurate models, as verified on high-dimensional biomechanics datasets (Haasdonk et al., 2020).

5. Specialized and Hybrid Kernel Surrogates

Kernel surrogates provide a unifying principle for several specialized and hybrid approaches:

• Koopman Operator Surrogates: Kernel-based extended DMD (kEDMD) surrogates in the RKHS yield rigorous state- and input-dependent pointwise error bounds for bilinear control representations, enabling robust stability-certified control for unknown nonlinear systems (Strässer et al., 17 Mar 2025).
• Kernel-based Neural Surrogates: Tensor-decomposed low-rank kernel surrogates (as in the KHRONOS architecture) provide highly parameter-efficient surrogate models for multi-fidelity tasks, such as aerodynamic field prediction, by constructing low-rank kernel expansions with B-spline basis—orders of magnitude more efficient than dense neural or graph architectures (Sarker et al., 11 Dec 2025).
• Deep Kernel and Structured Models: Deep kernel networks, such as two-layer kernels with learned linear mappings or structured layerwise compositions, boost the adaptivity and efficiency of kernel surrogates within the certified reduced basis modeling workflow, automatically targeting anisotropic parameter directions and yielding high query efficiency with small training sets (Wenzel et al., 2023).
• Kernel Surrogates for Neural Network Analysis: Empirical and approximate neural tangent kernel surrogates serve as faithful analytic replacements for DNNs in regression, classification, and attribution; trace-NTK and its random projection variants produce scalable surrogates closely matching DNN behavior in test accuracy and prediction ranking (Qadeer et al., 2023, Engel et al., 2023).
• Surrogates for Generative Models and ICA: Structured kernel regression (SKR) is a computationally efficient surrogate for GP priors in VAEs, retaining ICA and disentanglement properties while reducing training cost from $O(L^3)$ to $O(L^2)$, yielding nearly identical max-correlation accuracy in synthetic ICA tasks as full GP-VAEs (Wei et al., 13 Aug 2025).

6. Domain-Specific and Advanced Kernels

Advanced surrogate construction involves the design and composition of kernels to capture domain-specific structure:

• Frequency-aware kernels: Construction of composite kernels for time–frequency structured data (e.g., exponential squared-sine for periodicity, rational quadratic for heavy-tailed spectra), as implemented in the SMT 2.0 framework. Product/sum compositions enable flexible modeling of trend, seasonality, and irregularities (e.g., $k_{SE} * k_{Per} + k_{RQ}$) and analytic gradients/Hessians for optimization (Gonel et al., 13 Jul 2025, Saves et al., 2023).
• Mixed variable and hierarchical GP surrogates: Factorized or algebraic kernel structures for mixed continuous-discrete-categorical input spaces, supporting variable activation/deactivation and partial distance calculations, with applications in auto-tuning and hierarchical search (Saves et al., 2023).
• Task Attribution: In multi-task learning, kernel surrogate models over binary task-selection spaces ($\mathbf{s}\in\{0,1\}^K$) accurately capture nonlinear, synergistic, or antagonistic interactions between training tasks. An RBF kernel in Hamming space allows surrogate regression for task exclusion/inclusion, sharply improving task attribution over linear models and influence-function baselines (Zhang et al., 3 Feb 2026).

7. Practical Considerations, Limitations, and Extensions

Kernel surrogate models are theoretically well founded and broadly applicable, but share several computational and methodological challenges:

• They require solving dense interpolation or regression systems, with training complexity typically $O(n^3)$ (full), $O(nN^2)$ (greedy for $N\ll n$), or $O(N^3)$ (sparse, $N$ centers).
• Hyperparameters (kernel bandwidths, regularization) are critical; optimal or adaptive strategies, such as cross-validation, Fedorov–Wynn selection, and automated sharpness control, are active research areas.
• Highly data-driven models may fail to exploit known physics or structure unless hybridized with physics–informed or reduced-basis components.
• Approximate and hybrid surrogates (e.g., SKR, tensorized SNA, randomized projections for NTK surrogates) mitigate cost while maintaining practical fidelity (Wei et al., 13 Aug 2025, Sarker et al., 11 Dec 2025, Engel et al., 2023).
• Error control is available through power-function and RKHS analysis, enabling integration into optimization and robust control workflows.

Kernel surrogate models continue to evolve with advances in scalable algorithms, composite and deep kernel architectures, domain-specific kernel design, and integration into multi-fidelity and adaptive modeling pipelines.

---

Reference Table: Key Kernel Surrogate Model Variants

Approach/Algorithm	Domain/Application	Notable Features
Hermite Kernel Surrogates	Optimization, trust-region	Gradient-enriched, convergence guarantees
Optimal Kernel Learning	High-dim GP surrogates	Variable selection, functional ANOVA structure
VKOGA/KEA/Greedy/γ-VKOGA	Sparse surrogate modeling	Stagewise or exchange selection, stability
kEDMD Koopman Surrogates	Data-driven nonlinear control	Pointwise error bounds, robust control
Quantum Kernel Surrogate Models	Quantum VQE optimization	Finite Fourier basis features, RKHS alignment
KHRONOS Kernel Neural Surrogates	Multi-fidelity field prediction	Low-rank tensor decomposition, efficiency
Deep Kernel Networks	Certified RB-ML-ROM, PDEs	Hierarchical, adaptivity/predictivity
Task Attribution Kernel Surrogates	Multi-task ML attribution	Encodes synergy/antagonism among tasks
SKR for GP-VAEs	Generative models, ICA	Quadratic cost, ICA-level disentanglement
Frequency-Aware Composite Kernels	Forecasting, time series	Periodic, rational quadratic, product kernels

---

Principal References:

• (Ullmann et al., 2 Jul 2025): Hermite kernel trust-region methods for optimization.
• (Kang et al., 23 Feb 2025): Optimal kernel combination for high-dimensional GPs.
• (Santin et al., 2019): RKHS foundations and sparse kernel approximation algorithms.
• (Wenzel et al., 2024): Kernel Exchange Algorithm for finetuning sparse surrogates.
• (Sarker et al., 11 Dec 2025): KHRONOS: kernel-tensor surrogates for multi-fidelity prediction.
• (Zhang et al., 3 Feb 2026): Kernel surrogate models for task attribution in multi-task learning.
• (Haasdonk et al., 2020): γ-restricted VKOGA for stabilized greedy approximation.
• (Wenzel et al., 2023): Deep kernel models for adaptive reduced basis surrogates.
• (Wei et al., 13 Aug 2025): Structured kernel regression as a surrogate for GP-VAEs.
• (Gonel et al., 13 Jul 2025, Saves et al., 2023): Frequency-aware kernels, SMT Toolbox, and mixed variable surrogates.
• (Strässer et al., 17 Mar 2025): kEDMD kernel surrogates for bilinear Koopman models.
• (Smith et al., 2022): Quantum kernel-based surrogates.
• (Qadeer et al., 2023, Engel et al., 2023): NTK/approximate NTK surrogates for neural network regression and attribution.