---
title: Non-Linear Surrogate Model Extension
url: https://www.emergentmind.com/topics/non-linear-surrogate-model-extension-nl-sme
type: topic
---

# Non-Linear Surrogate Model Extension

Non-Linear Surrogate Model Extension (NL-SME) denotes a class of surrogate-modeling strategies in which a baseline surrogate representation is extended so that nonlinear trajectories, nonlinear dynamics, local regimes, or nonlinear operator structure are represented more faithfully than with a single global linear or weakly nonlinear approximation. In the supplied literature, the term is explicit in the federated-learning attack model of [2509.22082], where a linear surrogate trajectory is replaced by a learnable quadratic Bézier curve. A broader reading of related work suggests the same extension principle in surrogate dynamics warm-starting for nonlinear model predictive control [2604.05566], quadratic-bilinear identification from input-output data [2112.01258], on-the-fly reconstruction of nonlinear dynamics from local linearizations [2504.00276], local regime-aware aerodynamic surrogates [1912.01917], recursive spectral embeddings [2004.04480], state-space Kriging for stochastic dynamics [2409.02462], and conditional Gaussian surrogate or preconditioner constructions for complex nonlinear systems [2112.05226].

## 1. Defining pattern and scope

The common structural feature is that the surrogate is not merely fitted as a static approximation of outputs; instead, it is augmented so that nonlinear structure is introduced where a simpler surrogate is known to fail. In [2509.22082], this means replacing straight-line interpolation in parameter space with a quadratic Bézier path. In [2604.05566], it means learning surrogate dynamics only for an auxiliary optimization layer, while preserving the original full-scale NMPC solve. In [2112.01258], it means extending linear Loewner interpolation by inferring quadratic and bilinear operators after lifting. In [2504.00276], it means reconstructing a nonlinear field from interpolated Jacobians rather than regressing the field directly. In [1912.01917] and [2004.04480], it means replacing a single global surrogate by local or recursive pieces adapted to regime changes, discontinuities, or localized complexity. In [2409.02462] and [2112.05226], it means moving from static emulation to dynamical or conditional-statistics formulations.

This suggests that NL-SME is best understood as an extension principle rather than a single algorithm. The extension may occur in trajectory space, operator space, state space, regime space, or conditional-statistics space. Across these settings, the extension is introduced because a baseline surrogate is too restrictive: the linear trajectory assumption in multi-step gradient inversion, global POD/GPR under transonic regime changes, global spectral expansions under discontinuities, or direct brute-force nonlinear simulation under partial observations and stochastic forcing.

## 2. Main formalizations of nonlinear extension

The supplied works instantiate the extension principle through several mathematically distinct constructions.

| Formulation | Extension mechanism | Reported role |
|---|---|---|
| [2509.22082] | Quadratic Bézier surrogate trajectory with control point \(P_1\) | Multi-step gradient inversion in Federated Learning |
| [2604.05566] | Learned surrogate dynamics \(m_\theta\) inside an auxiliary warm-start NLP | Acceleration of full-scale NMPC |
| [2112.01258] | Loewner fit of linear part plus least-squares recovery of \(H\) and \(N\) | Quadratic-bilinear surrogate identification |
| [2504.00276] | RBF interpolation of local linearizations plus line integral reconstruction | On-the-fly surrogation of nonlinear dynamics |
| [1912.01917] | Regime classifier plus local POD/GPR surrogates | Multi-regime aerodynamic prediction |
| [2004.04480] | Recursive partitioning with local spectral residual expansions | Piecewise continuous surrogate for localized or non-stationary responses |
| [2409.02462] | Componentwise Kriging of \(\dot X=f(X,U)\) in state space | Emulation of stochastic nonlinear dynamics |
| [2112.05226] | Conditional Gaussian split \(u=(X,Y)\) with closed conditional statistics | Cheap surrogate model and fast preconditioner |

At the level of equations, the nonlinear extension takes markedly different forms. The explicit NL-SME attack paper uses
\[
\hat{w}(t,P_1)=(1-t)^2w_0+2(1-t)tP_1+t^2w_T,\qquad t\in[0,1],
\]
which strictly contains linear interpolation as the special case \(P_1=(w_0+w_T)/2\) [2509.22082]. The quadratic-bilinear identification framework uses
\[
\dot{x}(t)=Ax(t)+H\bigl(x(t)\otimes x(t)\bigr)+Nx(t)u(t)+Bu(t),\qquad y(t)=Cx(t),
\]
after lifting analytic nonlinearities into QB form [2112.01258]. The on-the-fly surrogate of [2504.00276] reconstructs nonlinear dynamics by
\[
\hat f(\hat x,u,\eta)=\hat F(\hat x,u,\eta)\begin{bmatrix}\hat x\\u\end{bmatrix},\qquad
\hat F(\hat x,u,\eta)=\int_0^1 \mathcal I^D(\lambda \hat x,\lambda u,\eta)\,d\lambda,
\]
so the surrogate is computed only at evaluation time. The S2K emulator writes the nonlinear stochastic system as \(\dot X(t)=f(X(t),U(t))\) and learns each component of \(Y(t)=\dot X(t)\) by Kriging in state space [2409.02462]. The CGNS construction splits the state as \(u=(X,Y)\) and imposes that, conditional on the history of \(X\), the hidden sector \(Y\) is linear-Gaussian, which yields closed filter and smoother equations [2112.05226].

A plausible implication is that NL-SME is not tied to one surrogate family. Neural surrogates, RBF interpolants, Kriging, Loewner realizations, local POD/GPR blocks, spectral expansions, and conditional Gaussian closures all appear in the supplied literature.

## 3. Explicit NL-SME in multi-step gradient inversion

The most direct use of the term appears in [2509.22082]. The paper addresses multi-step Gradient Inversion Attacks in FedAVG, where the attacker observes an initial model \(w_0\), a final model \(w_T\), and the local sample size \(N\), and seeks to reconstruct private client data from the aggregated update. The baseline SME models a surrogate parameter point by linear interpolation,
\[
\hat{w}(\alpha)=(1-\alpha)w_0+\alpha w_T,\qquad \alpha\in[0,1],
\]
and matches the gradient direction to \(w_0-w_T\). The paper argues that this is fundamentally limited because SGD trajectories on non-convex objectives are generally curved rather than linear, and it states a gradient-direction bias lower bound under Lipschitz assumptions.

NL-SME replaces the line segment by the quadratic Bézier path above and optimizes dummy data \(\tilde D\), the curve parameter \(t\), the control point \(P_1\), and a coordinate-wise scaling vector \(d\). Its objective is
\[
\mathcal{L}_{NL\text{-}SME}(\tilde D,t,P_1)=\mathcal{L}_{cos}(\tilde D,t,P_1)+\lambda_{TV}\mathcal{L}_{TV}(\tilde D)+\lambda_P\mathcal{L}_P(P_1)+\lambda_d\mathcal{L}_d(d),
\]
with total variation regularization, midpoint regularization
\[
\mathcal{L}_P(P_1)=\left\|P_1-\frac{w_0+w_T}{2}\right\|^2,
\]
and dvec scaling
\[
\tilde{\nabla}_{\hat w}\ell(\hat w(t,P_1),\tilde D)=\{d_i\cdot \nabla_{w_i}\ell(\hat w(t,P_1),\tilde D)\}_{i=1}^{|w|},\qquad d_i\in[0.1,10.0].
\]
The initialization is conservative:
\[
P_1^{(0)}=\frac{w_0+w_T}{2},\qquad t^{(0)}=0.5,\qquad d^{(0)}=\mathbf 1,
\]
and the paper specifies hierarchical learning rates satisfying \(\eta>\eta_t>\eta_{P_1}\).

On FEMNIST with \(E=20, N=50, B=10\), the reported similarity loss is \(0.0009\) for NL-SME versus \(0.0380\) for SME, \(0.1791\) for IG, and \(0.2414\) for FEDLEAK, with PSNR/SSIM of \(22.61/0.788\). On CIFAR-100 with the same setting, NL-SME reaches \(\mathcal L_{sim}=0.0029\), PSNR \(26.42\), and SSIM \(0.878\), compared with SME’s \(0.0544\), \(21.29\), and \(0.783\). Runtime and memory remain close to SME: for CIFAR-100 at \(E=20, N=50\), NL-SME uses \(0.223\) GB and \(0.77\) min, while SME uses \(0.131\) GB and \(0.70\) min. The paper therefore treats nonlinear trajectory modeling as a privacy-relevant strengthening of surrogate attacks rather than as a mere architectural variation [2509.22082].

## 4. Surrogate dynamics extension in nonlinear control

In [2604.05566], the nonlinear extension is formulated as Surrogate Dynamics Optimization for full-scale NMPC. The stated motivation is that the practical bottleneck is often repeated evaluation of the embedded process model \(\Phi\), especially for high-dimensional systems, multi-time-scale or stiff dynamics, DAE-based models, long-horizon control problems, and industrial settings where in-house solvers are used and advanced automatic differentiation or sparsity tools may not be available. Standard warm starts such as shift initialization are reported to be insufficient when the exogenous input trajectory \(w\), the cost function \(\ell\) or \(J\), or the constraints \(h\) change abruptly.

The workflow has three stages. First, a surrogate dynamics model \(m_\theta\) is trained offline on trajectories generated by the full-scale DAE solver \(\Phi\), using recursive multi-step loss over a horizon \(L\):
\[
L(m_\theta)=\frac{1}{n_x}\sum_{i=1}^{n_x}L_i(m_\theta)+\lambda R(m_\theta).
\]
Second, at inference time the surrogate replaces \(\Phi\) in a lightweight single-shooting NLP, solved with a first-order method such as projected gradient descent. The surrogate stage is deliberately limited to a small part of the total computational budget, typically \(\le 5\%\), and the paper also reports significant acceleration even with \(1\%\) budget. Third, the surrogate solution \(\hat u\) is used to warm-start the original full NMPC problem
\[
\min_u J(u\mid \Phi),
\]
so the final solution still comes from the trusted full-scale formulation.

The case study is 24-hour optimal load-following control of a pressurized water reactor. The paper evaluates MLP, RNN, and Transformer-based encoder-decoder surrogates and reports that the Transformer performs best because it better captures long-horizon temporal dependencies. The training setup uses \(10{,}000\) transients, each of 24 hours, with an \(80\%\) training and \(20\%\) validation split, hyperparameter optimization with Optuna over 100 trials, a maximum model size of \(30{,}000\) parameters, and training in PyTorch on an NVIDIA Tesla V100-SXM2-32GB. Relative to behavior cloning, surrogate training data generation took less than 1 hour, whereas generating BC training data required solving 300 full-scale OCPs, producing about 1 million trajectories and about 100 hours of cumulative compute. Under a main comparison budget of 20 full-scale iterations, about 20 minutes, SDO consistently outperforms shift initialization and BC in convergence speed and objective value after a fixed number of gradient steps, while preserving safety because the full solver remains the final decision stage [2604.05566].

## 5. Operator lifting and on-the-fly nonlinear reconstruction

The framework of [2112.01258] constructs quadratic-bilinear surrogates non-intrusively from input-output measurements. Its central claim is that many nonlinear systems with analytic nonlinearities can be rewritten exactly as QB systems by lifting, rather than approximately by Taylor truncation or Carleman linearization. The linear part is first identified from first-kernel data using the classical Loewner framework; then the supplementing quadratic and bilinear operators are recovered from second-kernel measurements by least squares, with
\[
\theta=\begin{bmatrix}\mathrm{vec}(H) & \tfrac12\,\mathrm{vec}(N)\end{bmatrix},
\qquad \mathcal L\theta=h,
\]
and the requirement \(K\ge r^3+r^2\). Because the system may be ill-conditioned or rank-deficient, the paper recommends truncated SVD or a Moore–Penrose pseudoinverse in regularized form. In the nonlinear RC ladder benchmark, the reduced linear order is chosen as \(r=10\), the second-kernel approximation error is reported as around \(10^{-8}\) in the infinity norm, and the time-domain response error is about \(10^{-6}\).

The on-the-fly surrogation method of [2504.00276] starts from local linearization snapshots rather than trajectories or output kernels. For the nonlinear system \(\xi x(t)=f(x(t),u(t),\eta)\), it stores distinct operating points \(z^{(i)}\) and local Jacobian-based linearizations \(M_i=\mathcal L_f(x^{(i)},u^{(i)},\eta^{(i)})\). An RBF interpolant \(\mathcal I^D\) is built so that \(\mathcal I^D(z^{(i)})=M_i\), optionally with a polynomial tail, and the nonlinear dynamics are reconstructed by integrating the interpolated Jacobian along the line segment from the origin to the query point. The model is “on-the-fly” because the global surrogate is never written out analytically; it is evaluated numerically only when needed. In the controlled Van der Pol example, exact reconstruction from perfect local linearizations reproduces the true nonlinear trajectories, and a surrogate built from only \(N=9\) snapshots still closely matches them. In the parameter-varying test with \(\eta\in[0.3,0.6]\), \(N=68\) observations suffice for interpolation over parameter space. In the 10-state mass-spring-damper example, the method uses \(N=100\) random linearizations in an 11-dimensional space and reports mean simulation time about 4 seconds, an average of 390 online integral evaluations per simulation, and RMSE mostly below \(10^{-3}\), with some outliers up to \(6\times 10^{-3}\) [2504.00276].

These two works illustrate two distinct NL-SME mechanisms: exact or lifted operator enrichment, and nonlinear field reconstruction from local differential information.

## 6. Local, hierarchical, and stochastic extensions

Several supplied papers extend surrogate modeling by introducing locality, recursive structure, or conditional-statistics structure. In aerodynamics, the Local Decomposition Method of [1912.01917] addresses discontinuous signals, moving shocks, bifurcations, and regime changes by combining a physical shock sensor, Gaussian Mixture Model clustering, Gaussian Process Classification of input space, entropy-based active resampling, and local POD/GPR surrogates. For the RAE2822 transonic airfoil, the reported global results improve from \(\langle Q_2\rangle_\Gamma=0.897\) and \(\langle NRMSE\rangle_\Gamma=1.93\times 10^{-2}\) to \(0.989\) and \(7.48\times 10^{-3}\) for \(C_p\), and from \(0.837\) and \(1.15\times 10^{-2}\) to \(0.908\) and \(9.53\times 10^{-3}\) for \(C_f\). In the transonic regime, \(Q_2\) for \(C_p\) improves from \(0.0974\) to \(0.901\).

The Stochastic Spectral Embedding method of [2004.04480] targets responses that are too nonlinear, localized, or non-stationary for a single global spectral surrogate. It recursively partitions the domain into subdomains \(\mathcal X^{\ell,p}\), fits local spectral residual expansions \(\widehat R_S^{\ell,p}\), and assembles a piecewise continuous approximation
\[
\widehat Y_{\mathrm{SSE}}(X)=\sum_{\ell=0}^{L}\sum_{p=1}^{P_\ell}\mathbf 1_{\mathcal X^{\ell,p}}(X)\,\widehat R_S^{\ell,p}(X).
\]
The paper reports that, at \(N_{ED}=200\), SSE achieves at least an order-of-magnitude smaller error than PCE on a 1D localized nonlinear function, and that it significantly outperforms PCE on a 100D analytical function and on a 3D snap-through truss with discontinuity.

The state space Kriging model of [2409.02462] learns the instantaneous derivative map \(Y(t)=f(X(t),U(t))\) rather than a long lagged input-output mapping. It uses componentwise Kriging with a Matérn-5/2 kernel, sparse sample selection by the acquisition criterion
\[
L_i(t)=(m_i(t)-y_i(t))^2+s_i^2(t),
\]
and tailored training trajectories generated by magnifying excitation variability. Reported errors include \(4.5\times 10^{-4}\) to \(6.1\times 10^{-4}\) for a nonlinear quarter-car benchmark, \(9.68\times 10^{-6}\) and \(9.09\times 10^{-6}\) for displacement and velocity in a Duffing oscillator, and \(1.44\times 10^{-4}\), \(1.65\times 10^{-5}\), and \(2.47\times 10^{-5}\) for displacement, velocity, and hysteretic variable in a Bouc–Wen oscillator.

The Conditional Gaussian Nonlinear System of [2112.05226] provides a different nonlinear extension: it preserves nonlinear self-interactions among resolved variables \(X\), keeps \(X\leftrightarrow Y\) cross-couplings, and approximates only the self-interactions among unresolved variables so that, conditional on the history of \(X\), the law of \(Y\) is Gaussian with explicit filter and smoother formulas. The paper presents CGNS simultaneously as a cheap surrogate model and a fast preconditioner for data assimilation, parameter estimation with uncertainty quantification, PDF reconstruction, trajectory sampling, and linear response prediction.

## 7. Assumptions, limitations, and recurring misconceptions

Across the supplied literature, NL-SME does not imply unconditional replacement of the original model. In [2604.05566], the surrogate is explicitly a warm-start layer and “not a replacement for the true solve”; the final solution remains the original NMPC solution. In [2112.05226], the surrogate is valuable precisely because it preserves much of the nonlinear physics while making part of the system analytically tractable. In [2504.00276], the method is not a reduced-order model and does not simply stitch local linears; it reconstructs a nonlinear field from interpolated Jacobians. In [2509.22082], NL-SME is not a controller or predictor but an attack model for reconstructing private data from multi-step updates.

Several limitations recur. Surrogate quality depends on data coverage and representational adequacy. If surrogate accuracy is too low, SDO may fail to accelerate or may degrade the warm start [2604.05566]. The least-squares system for QB identification may be ill-conditioned, rank-deficient, and sensitive to noise [2112.01258]. The on-the-fly linearization method requires online numerical integration, tuning of the width parameter \(c\), and does not yet provide error bounds [2504.00276]. S2K still faces scalability issues in high-dimensional state spaces and ignores cross-output correlations by modeling components independently [2409.02462]. CGNS depends on the quality of the chosen closure; if the closure is poor, parameter estimates or response predictions can be biased [2112.05226]. In the federated-learning setting, NL-SME remains vulnerable to defenses that alter update directions and magnitudes; the paper explicitly notes that adaptive optimizers such as AdamW can disrupt gradient-direction-based attacks [2509.22082].

A recurring misconception is that “nonlinear extension” necessarily means a monolithic black-box regressor. The supplied literature shows otherwise. The nonlinearity may arise from a Bézier trajectory, lifted quadratic-bilinear operators, a line integral of interpolated Jacobians, hard regime partitioning with local reduced models, recursive local spectral expansions, state-space GP closure, or conditional Gaussian decomposition. Another misconception is that greater expressiveness automatically yields formal guarantees. Several papers emphasize the opposite: strong convexity assumptions in the SDO bound are unlikely to hold in practice, noise sensitivity remains open for LoewnerQB, and the on-the-fly method offers convergence intuition rather than formal error estimates [2604.05566], [2112.01258], [2504.00276].

Taken together, these works show that NL-SME is most coherently understood as a family of structurally nonlinear surrogate extensions introduced when global linearization, global interpolation, or direct policy imitation is too weak for the target dynamics, optimization path, or regime structure.

Source: https://www.emergentmind.com/topics/non-linear-surrogate-model-extension-nl-sme