Regularized Surrogate Cost Function

• Regularized surrogate cost functions are objective functions that approximate complex cost landscapes and enforce structure via penalties like smoothness and sparsity.
• They enable efficient optimization in expensive, nonconvex, or non-differentiable settings by managing trade-offs between empirical data and prior knowledge.
• Applications span active learning, stochastic programming, and compressive sensing, boosting generalization and reducing computational costs.

A regularized surrogate cost function is an objective function used in optimization and statistical learning to approximate, control, or accelerate the search for optimal solutions—especially in settings where the true cost is expensive, nonconvex, or non-differentiable. By regularizing a surrogate (that is, supplementing it with terms to encode preferences such as smoothness, sparsity, or computational efficiency), the approach can enforce structure, facilitate generalization, and manage trade-offs between empiricism, prior knowledge, and computational tractability. This concept encompasses a broad array of techniques across simulation-based science, stochastic programming, combinatorial and structured optimization, and compressive sensing.

1. Mathematical Forms and Instantiations

Regularized surrogate cost functions appear in diverse mathematical forms, with the specific expression determined by the problem class and the desired regularization:

• Cost-regularized acquisition (active learning):

$A_{\mathrm{cost-reg}}(x) = \frac{V(x)}{C(x)}$

where $V(x)$ is a Gaussian Process (GP) posterior variance quantifying uncertainty, and $C(x)$ is an estimated sampling cost (Daningburg et al., 2022).

• Regularized sample average approximation (SAA):

$$\hat F^{\mathrm{reg}}_n(M) = \frac{1}{n}\sum_{i=1}^{n} C(M,\xi_i) + \lambda\,\mathcal{R}(M)$$

where $C(M, \xi)$ is a stochastic cost, $M$ is a matrix decision variable, and $\mathcal{R}(M)$ imposes low-rank or other structural penalties (Liu et al., 2019).

• Linear surrogate cost for combinatorial optimization:

$$L(\{w_i\},\theta; \lambda) = \sum_{i=1}^N f(x^*(w_i); y_i) + \lambda \|w_i - \hat{c}(y_i;\theta)\|_2$$

where $f$ is the original nonlinear cost, $w_i$ are problem-specific surrogates, and $\hat{c}(y_i;\theta)$ is a parametric prior (Ferber et al., 2022).

• Tikhonov-regularized least squares in surrogate regression:

$$\min_{\hat{\mathbf{c}}} \frac{1}{nN}\|\mathbf{A} \hat{\mathbf{c}} - \mathbf{u}\|_2^2 + \lambda^2 \|\mathbf{L} \hat{\mathbf{c}}\|_2^2$$

$\mathbf{L}$ is a roughening matrix built from gradient information to promote smoothness (Validi, 2013).

• Nonconvex sparsity surrogates in compressed sensing:

$$G_{\sigma,\gamma,\rho}(x) = L_{\sigma,\gamma}(Ax) + \lambda\,\rho\,\varphi_\rho(x)$$

where $\varphi_\rho(x)$ covers SCAD or MCP penalties as smooth, exact surrogates for $\ell_0$ (Chen et al., 2023).

• Cost-regularized optimal transport (OT):

$$\min_{M,\pi} -\sum_{i,j} \langle Mx_i, y_j \rangle \pi_{ij} + R(M) + \varepsilon\,\mathrm{KL}(\pi\,\|\,a\otimes b)$$

with $R(M)$ a convex structure-inducing penalty (e.g., sparsity, nuclear norm) (Sebbouh et al., 2023).

• Surrogate sharpness regularization:

$$L_{\mathrm{reg}}(\theta) = L_s(\theta) + \lambda \|\nabla_\theta \left[(1/n)\sum_i g(x_i; \theta)\right]\|_2$$

$L_s$ is the standard empirical loss, and the regularizer penalizes model "sharpness" (Dao et al., 6 Mar 2025).

2. Motivations and Theoretical Justifications

Regularized surrogates are motivated by the need to efficiently explore high-dimensional or costly domains, enforce statistical or physical priors, avoid overfitting, and enable scalable optimization or inference:

• Cost-aware sampling: By maximizing information per unit cost, as in the cost-regularized acquisition function, one exploits heterogeneity in simulation efforts, targeting regimes where sampling is most cost-effective (Daningburg et al., 2022).
• Statistical generalization: Regularization such as nuclear norm or MCP in high-dimensional SAA directly reduces sample complexity from $O(p^2)$ to $O(p)$ up to log factors by exploiting low-rankness (Liu et al., 2019).
• Sample efficiency in combinatorial optimization: Enforcing closeness to a prior (via $\lambda\|w_i-\hat{c}(y_i;\theta)\|_2$) ensures better generalization and smoothness of the cost map, making the surrogate robust across problem instances (Ferber et al., 2022).
• Robustness and model smoothness: Penalizing the gradient norm (sharpness) of a surrogate provably bound its out-of-distribution volatility and enhances optimization robustness (Dao et al., 6 Mar 2025).
• Structural regularization: Group, $\ell_1$, and nuclear penalties in OT yield interpretable structure and improved matching accuracy by inducing sparsity or low-rank mapping in learned transport (Sebbouh et al., 2023).
• Efficient surrogate inference: Tikhonov or roughening-operator regularization in stochastic regression promotes smooth surrogates, combating ill-conditioning and preventing unphysical oscillations (Validi, 2013).
• Convexification for tractability: Surrogates, especially in structured prediction, upper-bound the true loss and can be chosen to be convex or quasi-concave, enabling global optimization algorithms (Choi, 2018).

3. Algorithmic Approaches and Optimization

The solution to regularized surrogate cost functions involves distinctive algorithmic pipelines:

• Alternating maximization or proximal steps: Cost-regularized OT is addressed by alternating between (i) solving for the transport plan $\pi$ via Sinkhorn's algorithm and (ii) updating the cost parameter $M$ by proximal minimization for the chosen regularizer (Sebbouh et al., 2023).
• Block-coordinate or alternating least squares (ALS): Separated surrogate regression alternates among optimizing over each factor block with the associated regularization, together with model complexity selection via error indicators (Validi, 2013).
• Second-order or first-order methods for nonconvex regularizers: S³ONC points are obtained for MCP- or SCAD-regularized SAA via cubic regularization or trust-region Newton methods (Liu et al., 2019); proximal gradient with extrapolation is used for nonconvex surrogates in compressive sensing (Chen et al., 2023).
• Joint optimization of surrogates and priors: SurCo variants are trained by differentiating through the argmin of a linear surrogate solver, optimizing both instance-level surrogates and a shared parametric prior, with the regularization strength controlling the tradeoff (Ferber et al., 2022).
• Dual-augmented Lagrangian for sharpness: IGNITE penalizes surrogate sharpness via a dual ascent scheme on the constraint $$\rho \|\nabla_\theta h_\mathcal{D}(\theta)\|_2 \leq \epsilon$$, employing efficient Hessian-vector products (Dao et al., 6 Mar 2025).
• Efficient oracle-based inference: Bi-criteria structured surrogates in structured prediction are optimized via convex-hull, angular or $\lambda$-search oracles, leveraging decomposability where possible (Choi, 2018).

4. Empirical Findings and Trade-offs

Empirical studies across domains document the gains and trade-offs from regularized surrogates:

• Dramatic cost reduction: In gravitational wave surrogate modeling, cost-regularization yielded a %%%%21$\ell_0$22%%%% reduction in simulation budget for a target accuracy, at the expense of less coverage in the highest-cost corners (Daningburg et al., 2022).
• Sample complexity improvement: RSAA in high-dimensional matrix estimation showed nearly linear dependence on dimension, in contrast to quadratic scaling for unregularized approaches (Liu et al., 2019).
• Speed vs. accuracy in combinatorial surrogates: SurCo plans exhibit a spectrum: hard prior regularization accelerates test-time but may miss instance-specific details; per-instance surrogates provide best accuracy at higher computation; hybrid fine-tuning achieves fastest, best solutions (Ferber et al., 2022).
• Robustness and smoothness: SCAD-type surrogates in one-bit compressed sensing offered superior performance under high-noise or flip-ratio regimes, maintaining stability of mean-squared error over $\lambda$ variations; MCP/SCAD were especially robust compared to $\ell_p$ (Chen et al., 2023).
• Interpretability and scalability: Structured penalties in cost-regularized OT enhanced both convergence and label transfer accuracy in multi-omics alignment, notably under high dimensionality and limited sample size (Sebbouh et al., 2023).
• Sharpness control: Surrogate sharpness regularization reduced out-of-distribution gradient norm by up to $50\%$, resulting in consistently improved offline optimization performance (peak 9.6% improvement across benchmarks) (Dao et al., 6 Mar 2025).

Domain	Regularizer Type	Key Empirical Benefit
Gravitational Wave	Cost penalty	$10\times$ cost savings
High-Dim Stochastic	Low-rank (MCP/NN)	$O(p)$ sample complexity
Combinatorial Opt	$\ell_2$ prior	Best-of-both speed/accuracy hybrid
Compressed Sensing	SCAD/MCP	Robust MSE in high noise/flips
OT/Matching	$\ell_1$, nuclear	Faster, more interpretable solutions
Offline Surrogates	Sharpness (grad)	Systematic performance boost

5. Theoretical Guarantees and Conditions

Generalization and convergence results depend critically on surrogate structure and the employed regularization:

• Surrogate upper bounds: Many surrogates, especially structured ones, are constructed to upper-bound the original loss, guaranteeing that minimizing the surrogate controls true risk (Choi, 2018).
• Sample complexity reduction without RSC: Low-rank regularization via MCP yields near-optimal dimension dependence even in absence of restricted strong convexity, enabling application to nonlinear and non-GLM matrix models (Liu et al., 2019).
• PAC-Bayes and sharpness bounds: Surrogate gradient-norm regularization allows bounding worst-case out-of-distribution sharpness in terms of the empirical gradient norm on training data—a previously unattainable guarantee for offline learning settings (Dao et al., 6 Mar 2025).
• Global and linear convergence: For penalty surrogates with KL property (piecewise polynomial, semi-algebraic structure), proximal algorithms enjoy global convergence, and local linear convergence under mild conditions (e.g., SCAD surrogate with stabilized active set) (Chen et al., 2023).
• Regularization-specific complexity terms: For slack- and margin-rescaling surrogates, generalization bounds directly reveal how the surrogate's structure controls the empirical and complexity contributions (Choi, 2018).

6. Practical Implementation and Guidelines

Successful deployment of regularized surrogate cost functions requires:

• Domain-aware cost modeling: The cost function $C(x)$ in acquisition or OT must be precomputed and smooth; sharp discontinuities or errors in cost estimation can severely bias sampling or mapping (Daningburg et al., 2022).
• Regularization selection: The choice and tuning of the regularization strength (e.g., $\lambda$, prior variance, group strength) critically influences the trade-off between bias, variance, and computational demands, as documented in SurCo and RSAA schemes (Ferber et al., 2022, Liu et al., 2019).
• Complexity selection protocols: Instruments such as perturbation-based error indicators (PEI) or cross-validation of sharpness thresholds (as in IGNITE) are essential for preventing over- or under-regularization (Validi, 2013, Dao et al., 6 Mar 2025).
• Robustness in ill-conditioned settings: Tikhonov or roughening penalties, as well as nonconvex sparsity-inducing surrogates, can stabilize regression or recovery where unregularized approaches would fail to yield meaningful or interpretable solutions (Validi, 2013, Chen et al., 2023).
• Monitoring coverage and bias: Aggressive regularization (e.g., extensive cost avoidance) can undersample critical domains; introducing soft bounds or hybrid strategies avoids pathological coverage gaps (Daningburg et al., 2022, Ferber et al., 2022).

7. Connections and Generalizations

Regularized surrogate cost functions form an organizing principle across domains:

• Active learning and Bayesian optimization: Surrogates with cost/uncertainty trade-offs generalize to experiment design and sequential decision-making (Daningburg et al., 2022).
• Statistical learning and high-dimensional inference: Regularization enables tractable estimation even when the ambient model class is large, superseding older reliance on convexity or strong identifiability (Liu et al., 2019).
• Optimal transport and matching: Imposing structure on cost parameters (sparsity, group, rank) parallels the developments in interpretable and scalable OT; surrogates can enforce alignment with desired tasks or biological priors (Sebbouh et al., 2023).
• Structured prediction and compressive sensing: Surrogates enable convex, efficient optimization or recovery even in nonconvex, discrete, or signal-structured settings; design of the surrogate is tailored to the structure and restrictions of the original loss (Choi, 2018, Chen et al., 2023).
• Offline optimization and reliability: Modern practice leverages sharpness or gradient control for out-of-distribution reliability of surrogate models, emphasizing the need for provable regularization in data-driven acquisition and design (Dao et al., 6 Mar 2025).

Regularized surrogate cost functions are thus fundamental to navigating the competing demands of computational feasibility, statistical efficiency, and practical relevance in modern data-driven science and engineering.