Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data structure > labels? Unsupervised heuristics for SVM hyperparameter estimation (2111.02164v2)

Published 3 Nov 2021 in cs.LG

Abstract: Classification is one of the main areas of pattern recognition research, and within it, Support Vector Machine (SVM) is one of the most popular methods outside of field of deep learning -- and a de-facto reference for many Machine Learning approaches. Its performance is determined by parameter selection, which is usually achieved by a time-consuming grid search cross-validation procedure (GSCV). That method, however relies on the availability and quality of labelled examples and thus, when those are limited can be hindered. To address that problem, there exist several unsupervised heuristics that take advantage of the characteristics of the dataset for selecting parameters instead of using class label information. While an order of magnitude faster, they are scarcely used under the assumption that their results are significantly worse than those of grid search. To challenge that assumption, we have proposed improved heuristics for SVM parameter selection and tested it against GSCV and state of the art heuristics on over 30 standard classification datasets. The results show not only its advantage over state-of-art heuristics but also that it is statistically no worse than GSCV.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. Journal of Multiple-Valued Logic & Soft Computing 17.
  2. A systematic review on supervised and unsupervised machine learning algorithms for data science. Supervised and unsupervised learning for data science , 3–21.
  3. Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression. Pattern Recognition 40, 2154–2162. doi:https://doi.org/10.1016/j.patcog.2006.12.015.
  4. Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis. The Journal of Machine Learning Research 18, 2653–2688.
  5. Cross-validation.
  6. The balanced accuracy and its posterior distribution, in: 2010 20th international conference on pattern recognition, IEEE. pp. 3121–3124.
  7. Svm-rbf parameters testing optimization using cross validation and grid search to improve multiclass classification. Scientific Visualization 11, 80–90.
  8. Appearance-based object recognition using SVMs: which kernel should i use?, in: Proc of NIPS workshop on Statistical methods for computational experiments in visual processing and computer vision, Whistler.
  9. Sigest: Hyperparameter estimation for the gaussian radial basis kernel. https://www.rdocumentation.org/packages/kernlab/versions/0.9-29/topics/sigest. RDocumentation, Accessed: 10-06-2021.
  10. A comprehensive survey on support vector machine classification: Applications, challenges and trends. Neurocomputing 408, 189–215.
  11. Model selection for support vector machines, in: Advances in neural information processing systems, pp. 230–236.
  12. Semi-supervised classification by low density separation., in: AISTATS, Citeseer. pp. 57–64.
  13. Application of SVM and ANN for intrusion detection. Computers & Operations Research 32, 2617–2634. doi:https://doi.org/10.1016/j.cor.2004.03.019.
  14. Remote sensing image scene classification meets deep learning: Challenges, methods, benchmarks, and opportunities. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13, 3735–3756.
  15. A spatial-spectral disagreement-based sample selection with an application to hyperspectral data classification. IEEE Geoscience and Remote Sensing Letters 16, 467–471. doi:https://doi.org/10.1109/LGRS.2018.2868862.
  16. Support-vector networks. Machine learning 20, 273–297. doi:https://doi.org/10.1007/BF00994018.
  17. A realistic seizure prediction study based on multiclass SVM. International journal of neural systems 27, 1750006. doi:https://doi.org/10.1142/S012906571750006X.
  18. Support vector regression machines. Advances in neural information processing systems 9, 155–161.
  19. UCI machine learning repository. URL: http://archive.ics.uci.edu/ml.
  20. Make it cheap: learning with o(nd) complexity, in: The 2012 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–4. doi:https://doi.org/10.1109/IJCNN.2012.6252380.
  21. Vision-based pest detection based on SVM classification method. Computers and Electronics in Agriculture 137, 52–58. URL: https://www.sciencedirect.com/science/article/pii/S016816991631136X, doi:https://doi.org/10.1016/j.compag.2017.03.016.
  22. Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research 15, 3133–3181.
  23. OpenML-Python: an extensible Python API for openML. Journal of Machine Learning Research 22, 1–5.
  24. Gamma=‘scale‘ in SVC. https://github.com/scikit-learn/scikit-learn/issues/12741. Accessed: 10-06-2021.
  25. Application of hyperspectral imaging and machine learning methods for the detection of gunshot residue patterns. Forensic Science International 290, 227–237. doi:https://doi.org/10.1016/j.forsciint.2018.06.040.
  26. Artificial intelligence empowered emails classifier for internet of things based systems in industry 4.0. Wireless Networks 28, 493–503. URL: https://doi.org/10.1007/s11276-021-02619-w, doi:10.1007/s11276-021-02619-w.
  27. A comparison of methods for multiclass support vector machines. IEEE transactions on Neural Networks 13, 415–425. doi:https://doi.org/10.1109/72.991427.
  28. Provably efficient machine learning for quantum many-body problems. Science 377, eabk3333.
  29. A discriminative framework for detecting remote protein homologies. Journal of computational biology 7, 95–114. doi:https://doi.org/10.1089/10665270050081405.
  30. Using the fisher kernel method to detect remote protein homologies., in: ISMB, pp. 149–158.
  31. Constructing support vector machine ensemble. Pattern recognition 36, 2757–2767. doi:https://doi.org/10.1016/S0031-3203(03)00175-4.
  32. Multiclass classification of dry beans using computer vision and machine learning techniques. Computers and Electronics in Agriculture 174, 105507.
  33. Svm parameter tuning with grid search and its impact on reduction of model over-fitting, in: Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing: 15th International Conference, RSFDGrC 2015, Tianjin, China, November 20-23, 2015, Proceedings, Springer. pp. 464–474.
  34. Abnormal events detection using unsupervised one-class svm-application to audio surveillance and evaluation, in: 2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), IEEE. pp. 124–129. doi:https://doi.org/10.1109/AVSS.2011.6027306.
  35. Plant disease detection and classification by deep learning—a review. IEEE Access 9, 56683–56698.
  36. A HOG feature and SVM based method for forward vehicle detection with single camera, in: 2013 5th International Conference on Intelligent Human-Machine Systems and Cybernetics, IEEE. pp. 263–266. doi:https://doi.org/10.1109/IHMSC.2013.69.
  37. Effects of SVM parameter optimization on discrimination and calibration for post-procedural pci mortality. Journal of Biomedical Informatics 40, 688–697.
  38. Application of machine learning-based pattern recognition in iot devices: Review, in: Kumar, S., Purohit, S.D., Hiranwal, S., Prasad, M. (Eds.), Proceedings of International Conference on Communication and Computational Technologies, Springer Singapore, Singapore. pp. 669–689.
  39. Study on the impact of partition-induced dataset shift on k𝑘kitalic_k-fold cross-validation. IEEE Transactions on Neural Networks and Learning Systems 23, 1304–1312. doi:https://doi.org/10.1109/TNNLS.2012.2199516.
  40. Supervised learning for insider threat detection using stream mining, in: 2011 IEEE 23rd international conference on tools with artificial intelligence, IEEE. pp. 1032–1039. doi:https://doi.org/10.1109/ICTAI.2011.176.
  41. Limitations of the use of mobile devices and smart environments for the monitoring of ageing people, in: Proceedings of the 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health - HSP, INSTICC. SciTePress. pp. 269–275. doi:10.5220/0006817802690275.
  42. On performing classification using svm with radial basis and polynomial kernel functions, in: 2010 3rd International Conference on Emerging Trends in Engineering and Technology, IEEE. pp. 512–515.
  43. Semi-supervised hyperspectral classification from a small number of training samples using a co-training approach. ISPRS Journal of Photogrammetry and Remote Sensing 121, 60–76. doi:https://doi.org/10.1016/j.isprsjprs.2016.08.011.
  44. From regularization operators to support vector kernels. Adv. Neural Inf. Process. Syst 10, 343–349.
  45. Learning with kernels. volume 4. Citeseer.
  46. Parameter estimation of support vector machine with radial basis function kernel using grid search with leave-p-out cross validation for classification of motion patterns of subviral particles. Current Directions in Biomedical Engineering 7, 121–124.
  47. kNN and SVM classification for EEG: a review. InECCE2019 , 555–565doi:https://doi.org/10.1007/978-981-15-2317-5˙47.
  48. A comparative analysis of logistic regression, random forest and knn models for the text classification. Augmented Human Research 5, 1–16.
  49. Easy kernel width choice. https://blog.smola.org/post/940859888/easy-kernel-width-choice. Accessed: 10-06-2021.
  50. A meta-learning method to select the kernel width in support vector regression. Machine learning 54, 195–209. doi:https://doi.org/10.1023/B:MACH.0000015879.28004.9b.
  51. Unsupervised anomaly detection based on clustering and multiple one-class SVM. IEICE transactions on communications 92, 1981–1990. doi:https://doi.org/10.1587/transcom.E92.B.1981.
  52. Breast mass classification based on cytological patterns using RBFNN and SVM. Expert Systems with Applications 36, 5284–5290. doi:https://doi.org/10.1016/j.eswa.2008.06.127.
  53. Recognition of patterns with help of generalized portraits. Avtomat. i Telemekh 24, 774–780.
  54. A practical approach to model selection for support vector machines with a gaussian kernel. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41, 330–340. doi:https://doi.org/10.1109/TSMCB.2010.2053026.
  55. How to tune the RBF SVM hyperparameters? an empirical evaluation of 18 search algorithms. Artificial Intelligence Review , 1–27doi:https://doi.org/10.1007/s10462-021-10011-5.
  56. Predicting criminal recidivism with support vector machine, in: 2010 International Conference on Management and Service Science, IEEE. pp. 1–9. doi:https://doi.org/10.1109/ICMSS.2010.5575352.
  57. Super-parameter selection for gaussian-kernel SVM based on outlier-resisting. Measurement 58, 147–153. doi:https://doi.org/10.1016/j.measurement.2014.08.019.
  58. Biometrics for internet-of-things security: A review. Sensors 21, 6163.
  59. A fast leave-one-out cross-validation for SVM-like family. Neural Computing and Applications 27, 1717–1730. doi:https://doi.org/10.1007/s00521-015-1970-4.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now