Radial basis function kernel optimization for Support Vector Machine classifiers

Abstract: Support Vector Machines (SVMs) are still one of the most popular and precise classifiers. The Radial Basis Function (RBF) kernel has been used in SVMs to separate among classes with considerable success. However, there is an intrinsic dependence on the initial value of the kernel hyperparameter. In this work, we propose OKSVM, an algorithm that automatically learns the RBF kernel hyperparameter and adjusts the SVM weights simultaneously. The proposed optimization technique is based on a gradient descent method. We analyze the performance of our approach with respect to the classical SVM for classification on synthetic and real data. Experimental results show that OKSVM performs better irrespective of the initial values of the RBF hyperparameter.

• The paper presents OKSVM, which optimizes the RBF kernel hyperparameter γ alongside SVM weights to overcome limitations of traditional grid search.
• It employs an iterative dual optimization using gradient descent with dynamic learning rates and controlled step lengths to ensure robust convergence.
• Experiments on synthetic and real datasets demonstrate significant improvements in F1 scores, recall, precision, and AUC, underscoring its practical effectiveness.

Radial Basis Function Kernel Optimization for Support Vector Machine Classifiers

Support Vector Machines (SVMs) are widely regarded as powerful tools for classification tasks due to their capability to handle complex problems through kernel methods. The Radial Basis Function (RBF) kernel, a commonly used kernel in SVMs, requires fine-tuning of its hyperparameters to achieve optimal performance. The paper presents a novel approach termed OKSVM, which automatically optimizes the RBF kernel hyperparameter $\gamma$ alongside the SVM weights, potentially overcoming the limitations of traditional grid search methods.

Methodology

Support Vector Machines

SVMs operate by transforming input data into a high-dimensional space and finding the optimal hyperplane that maximizes the margin between classes. This margin-based optimization problem can be expressed in both primal and dual forms, with the latter being preferred for datasets with a large feature space due to fewer optimization variables. The dual form includes Lagrange multipliers associated with training samples, providing an efficient way to address soft-margin constraints that allow for misclassification to handle noisy data.

RBF Kernel Optimization

The paper introduces an innovative dual optimization strategy for the RBF kernel hyperparameter $\gamma$. This optimization leverages gradient descent to minimize the upper bound on misclassification probability. The approach alternates between optimizing the Lagrange multipliers $\boldsymbol{\alpha}$ and refining $\gamma$, ensuring stability with controlled gradient descent step lengths. This dual optimization is executed iteratively, adjusting the learning rate dynamically using the bold driver adaptation method.

Computational Implementation

During implementation, computational challenges such as potential horizontal asymptotes in the optimization landscape are managed by limiting $\gamma$ values to prevent numerical errors. Parameters such as $\gamma_{MAX}$ and stagnation checks ensure robust convergence. Moreover, initialization strategies and repeated validation on synthetic datasets with varied dimensions and separability reinforce the methodology's effectiveness.

Experimental Results

Synthetic Datasets

Experiments on synthetic datasets highlight OKSVM's resilience to initial hyperparameter values, consistently yielding better performance than classical SVM across different configurations of data dimensions and class separability. The robustness of OKSVM is further supported by the adaptive nature of the proposed algorithm when hyperparameters are pre-tuned via grid search.

Real Datasets

Evaluations on real-world datasets reveal that OKSVM outperforms traditional SVM implementations in most cases, demonstrating improved F1 scores, recall, precision, and AUC measures. This superiority is particularly evident in datasets with intricate distribution patterns, validating OKSVM’s practical applicability in diverse real-world scenarios.

Conclusion

The proposed OKSVM methodology provides a reliable mechanism for automatic and efficient optimization of SVM classifiers, especially when employed with RBF kernels. By eliminating the dependence on initial hyperparameter settings, OKSVM advances the practice of tuning SVMs, paving the way for its deployment in various complex classification tasks without compromising stability or performance. Future work could explore extending this approach to other kernel types and broader hyperparameter spaces.