Analytical Evaluation of DCA Convergence Properties for Minimizing Prediction Functions of Gaussian RBF Support Vector Regression

Abstract: For nonconvex optimization problems whose objective is the prediction function of a trained Support Vector Regression (SVR) model with the Gaussian radial basis function (RBF) kernel (RBF-SVR), we present a framework that applies the difference of convex functions (DC) algorithm (DCA) by exploiting the analytical structure of the RBF kernel to construct an explicit DC decomposition. Specifically, we derive in closed form both the lower bound $μ$ of the strong convexity parameter of the DC components and the upper bound $L$ of the gradient Lipschitz constant of the subproblem. Both $μ$ and $L$ are determined solely by the post-training dual-coefficient sum $C_α$ and the RBF kernel parameter $γ$, together with the DC decomposition parameter $ρ$, and they share a common leading term $C_αρ$. Through numerical experiments on six benchmark functions, we show that $C_αρ$ is the primary single quantity characterizing both the convergence properties and the initial-point dependence of DCA, and further demonstrate that it decomposes into two independent pathways, $C \to C_α$ and $γ\to ρ$, with its primary variation governed by the SVR hyperparameters $(C, γ)$. Together, these results allow the convergence properties of DCA on RBF-SVR to be assessed in advance through the single scalar quantity $C_αρ$: approximately from $(C, γ)$ before training, and exactly in closed form after training.

• The paper presents a closed-form analytical framework that rigorously characterizes DCA convergence in Gaussian RBF-SVR models by establishing explicit bounds on strong convexity and smoothness.
• The methodology decomposes the nonconvex SVR prediction function into convex components using the RBF kernel structure, ensuring diagnostic and tuning capabilities via analytical bounds.
• Empirical results demonstrate that the composite quantity Cαρ, governed by SVR hyperparameters, predicts convergence speed and iteration counts across various benchmark functions.

Analytical Convergence Characterization of DCA for RBF-SVR Surrogate Optimization

Introduction

The paper "Analytical Evaluation of DCA Convergence Properties for Minimizing Prediction Functions of Gaussian RBF Support Vector Regression" (2606.03559) rigorously develops a closed-form analytical framework for characterizing the convergence properties of the Difference of Convex (DC) algorithm (DCA) when the target is the prediction function of a trained Support Vector Regression (SVR) model with a Gaussian Radial Basis Function (RBF) kernel. The authors exploit the analytical structure of the RBF kernel to establish explicit bounds on the strong convexity and gradient Lipschitz constants of the convex components in the DC decomposition of the SVR objective, both before and after training. Through extensive numerical experimentation, they demonstrate that a single composite quantity, $C_{\alpha}\rho$, fully characterizes the convergence behavior of DCA, where $C_{\alpha}$ is the sum of dual coefficients after SVR training, $\rho$ is the DC decomposition parameter (explicitly bounded via kernel structure), and both are tightly governed by SVR hyperparameters.

DC Decomposition of RBF-SVR Prediction Functions

The SVR prediction function with a Gaussian RBF kernel is given by: $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$ where $k(\bm{x}_i, \bm{x})$ is the RBF kernel and $(\alpha_i, \alpha_i^*)$ are dual coefficients from SVR training. The function is nonconvex and conventionally optimized via metaheuristics, which ignore algebraic structure and lack convergence guarantees.

Applying DC programming, the prediction function is decomposed as $\hat{f} = G - H$, where both $G$ and $H$ are convex functions constructed analytically by leveraging the RBF kernel's Hessian properties. The decomposition relies on a parameter $\rho$, linked to the kernel width $C_{\alpha}$0, and is structured so that convexity of the subproblems at every iteration is guaranteed when $C_{\alpha}$1 exceeds explicit lower bounds derived from operator theory.

Analytical Bounds: Strong Convexity and Smoothness

The key technical results are the closed-form expressions:

• Strong convexity lower bound: $C_{\alpha}$2,
• Lipschitz upper bound: $C_{\alpha}$3.

Both constants depend on only the sum of SVR duals $C_{\alpha}$4, the kernel width $C_{\alpha}$5, and the DC decomposition parameter $C_{\alpha}$6, and share leading order dependence on the product $C_{\alpha}$7. Thus, the convergence guarantees for DCA (descent at each step, criticality of accumulation points) can be evaluated a priori using these quantities, enabling both pre-training estimation (from hyperparameters) and post-training exact calculation.

Subproblem Structure and Convergence Quantification

The DCA subproblem each iteration is a convex optimization comprising a quadratic proximal term and a kernel-weighted sum, whose geometry is precisely controlled via $C_{\alpha}$8. The explicit completion-of-squares allows full characterization of search dynamics. The evolution of iterates exhibits strong monotone descent, with step sizes and convergence speed quantitatively dictated by $C_{\alpha}$9.

Numerical Verification and Empirical Convergence

Across six benchmark functions with diverse multimodal landscapes:

Figure 1: Heatmap visualizations of Branin, Himmelblau, Rastrigin, Ackley, Levy, and Six-hump Camel functions, illustrating varying degrees of nonconvexity and ridge structure.

Statistical analysis over thousands of SVR models trained with systematically varied $\rho$0 demonstrates:

• Median and variance of DCA iteration counts are monotonic in $\rho$1.
• The dependence of DCA convergence on the initial point is amplified at high $\rho$2.
• The quantity $\rho$3 decomposes into two independent pathways: $\rho$4 controls $\rho$5 (strong rank correlation $\rho$6); $\rho$7 directly influences $\rho$8, confirming the analytic separation.
• Empirical convergence curves exhibit linear (exponential) decay in normalized residuals regardless of underlying objective multimodality.

Practical Implications

The results offer both theoretical and practical implications:

• Theoretical rigor: Closed-form bounds ground DC optimization analysis for nonconvex surrogate functions, bridging function-structure-specific and generic DC theory.
• Optimization protocol design: Convergence properties can be diagnosed, and smoothness parameters selected, at both the model-design (hyperparameter selection) and post-training stages.
• Surrogate modeling: Hyperparameter choices for SVR can be tuned not only for prediction quality but also for optimization tractability, avoiding pathological regimes where DCA converges slowly or is highly sensitive to initial conditions.
• Algorithmic extension: The explicit Lipschitz bound $\rho$9 provides an analytically justified smoothness constant for Frank-Wolfe-based DC optimization variants (DC-FW, Blended Pairwise Conditional Gradients), facilitating scalable and constrained nonconvex optimization (Maskan et al., 11 Mar 2025, Pokutta, 23 Jul 2025).

Figures and Empirical Structure

Figure 1 provides the landscape context; additional empirical plots (scatter of $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$0 versus $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$1, heatmaps over $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$2, normalized convergence trajectories) support the claims quantitatively.

Figure 2: Median of $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$3 in the $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$4 space illustrating how excess in both hyperparameters leads to challenging optimization.

Figure 3: Evolution of the normalized residual $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$5, showing linear convergence behavior for DCA across SVR models grouped by $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$6.

Conclusion

This work formally characterizes and empirically validates the convergence properties of DCA applied to RBF-SVR surrogate minimization via a unified closed-form index, $$\hat{f}(\bm{x}) = \sum_{i=1}^{n} (\alpha_i - \alpha_i^*) k(\bm{x}_i, \bm{x}),$$7. The analytic structure provides direct diagnostic and tuning capabilities for surrogate-based optimization protocols using SVR, as well as principled selection of smoothness constants for modern scalable DC optimization algorithms. The methodology and results are extensible to other kernel-based surrogates and offer a blueprint for bridging machine learning-based surrogate modeling and mathematical programming in engineering design and simulation-driven optimization.

Future Directions

Further empirical assessment of advanced DC-FW variants using these analytical constants, exploration of higher-dimensional settings, and extensions to other smooth kernel architectures remain open avenues. The integration of this analysis with uncertainty-quantified surrogate optimization (e.g., Bayesian acquisition functions) is also a promising direction.