Unified SVM Algorithm Based on LS-DC Loss

Abstract: Over the past two decades, support vector machine (SVM) has become a popular supervised machine learning model, and plenty of distinct algorithms are designed separately based on different KKT conditions of the SVM model for classification/regression with different losses, including the convex loss or nonconvex loss. In this paper, we propose an algorithm that can train different SVM models in a \emph{unified} scheme. First, we introduce a definition of the \emph{LS-DC} (\textbf{l}east \textbf{s}quares type of \textbf{d}ifference of \textbf{c}onvex) loss and show that the most commonly used losses in the SVM community are LS-DC loss or can be approximated by LS-DC loss. Based on DCA (difference of convex algorithm), we then propose a unified algorithm, called \emph{UniSVM}, which can solve the SVM model with any convex or nonconvex LS-DC loss, in which only a vector is computed, especially by the specifically chosen loss. Particularly, for training robust SVM models with nonconvex losses, UniSVM has a dominant advantage over all existing algorithms because it has a closed-form solution per iteration, while the existing algorithms always need to solve an L1SVM/L2SVM per iteration. Furthermore, by the low-rank approximation of the kernel matrix, UniSVM can solve the large-scale nonlinear problems with efficiency. To verify the efficacy and feasibility of the proposed algorithm, we perform many experiments on some small artificial problems and some large benchmark tasks with/without outliers for classification and regression for comparison with state-of-the-art algorithms. The experimental results demonstrate that UniSVM can achieve comparable performance in less training time. The foremost advantage of UniSVM is that its core code in Matlab is less than 10 lines; hence, it can be easily grasped by users or researchers.

• The paper introduces the LS-DC loss to unify multiple SVM models for classification and regression tasks.
• It employs DC decomposition to transform nonconvex optimization into solvable convex subproblems.
• Experimental results show that UniSVM delivers robust performance, especially on large-scale, noise-infused datasets.

Unified SVM Algorithm Based on LS-DC Loss

The paper under discussion introduces a unified algorithm for training various Support Vector Machine (SVM) models using a Least Squares type of Difference of Convex (LS-DC) loss functions framework. This approach leverages the Difference of Convex (DC) programming to handle different types of losses for both classification and regression tasks, presenting a potentially efficient solution for dealing with nonconvex optimization problems prevalent in SVM contexts.

LS-DC Loss and Unified SVM Model

The core innovation of this research is the introduction of the LS-DC loss, which encompasses both convex and nonconvex loss functions used in SVMs. By employing the DC decomposition, the LS-DC loss allows the restructuring of various SVM models into a uniform structure that can be efficiently optimized. The algorithm, termed UniSVM, is capable of solving SVM models using this unified loss framework, significantly enhancing its robustness and computational efficiency.

Figure 1: The plots of some LS-DC losses for classification and their DC decompositions: ``\rm{red Red~curve.

The DC decomposition approach enables the transformation of the complex optimization landscape into a series of convex optimization problems. This transformation is facilitated by the difference of convex algorithm (DCA), which iteratively refines the solution by solving simpler convex problems until convergence.

Implementation of the UniSVM Algorithm

UniSVM's practical implementation can be achieved with remarkable simplicity, as the core computational routine involves solving a system of linear equations iteratively to approximate the SVM solution. The primary computational effort lies in matrix operations, specifically the inversion and multiplication of matrices. For large-scale problems where the kernel matrix is extensive, the algorithm benefits from techniques like the Nyström method or pivoted Cholesky factorization to approximate the matrix efficiently.

UniSVM is implemented as follows in the simplified context where the full kernel matrix is available:

function [alpha] = UniSVM_small(K, y, lambda, A, dloss, eps0) m = length(y); v_old = zeros(m,1); Q = inv(K + lambda * m / A * eye(m)); alpha = Q*y; while 1 Ka = K * alpha; v = - y .* dloss(1-y .* Ka); if norm(v_old - v) < eps0, break; end alpha = Q * (Ka - v *(0.5/A)); v_old = v; end end

The above implementation calculates the parameters using a closed-form iterative scheme. The dloss function is used to calculate the subgradient of the loss function, essential for the iterative update step.

Experimental Results and Discussion

The experiments corroborate that UniSVM achieves competitive results compared to existing SVM frameworks, particularly in handling large-scale problems with noise-infused datasets. It is observed that models utilizing nonconvex losses exhibit robustness against outliers, outperforming standard convex loss SVMs.

Figure 2: Comparison of the related algorithms of SVM with convex losses.

Concluding Remarks

The UniSVM algorithm introduces a versatile and robust framework for SVM training, capable of handling both convex and nonconvex optimization problems effectively. Its simplicity in core implementation, combined with advanced methods for large data handling, positions it as a valuable tool for machine learning practitioners. Future work may explore extending the LS-DC framework to other machine learning models or further enhancing the algorithm's scalability and robustness.