Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HawkEye: Advancing Robust Regression with Bounded, Smooth, and Insensitive Loss Function (2401.16785v2)

Published 30 Jan 2024 in cs.LG

Abstract: Support vector regression (SVR) has garnered significant popularity over the past two decades owing to its wide range of applications across various fields. Despite its versatility, SVR encounters challenges when confronted with outliers and noise, primarily due to the use of the $\varepsilon$-insensitive loss function. To address this limitation, SVR with bounded loss functions has emerged as an appealing alternative, offering enhanced generalization performance and robustness. Notably, recent developments focus on designing bounded loss functions with smooth characteristics, facilitating the adoption of gradient-based optimization algorithms. However, it's crucial to highlight that these bounded and smooth loss functions do not possess an insensitive zone. In this paper, we address the aforementioned constraints by introducing a novel symmetric loss function named the HawkEye loss function. It is worth noting that the HawkEye loss function stands out as the first loss function in SVR literature to be bounded, smooth, and simultaneously possess an insensitive zone. Leveraging this breakthrough, we integrate the HawkEye loss function into the least squares framework of SVR and yield a new fast and robust model termed HE-LSSVR. The optimization problem inherent to HE-LSSVR is addressed by harnessing the adaptive moment estimation (Adam) algorithm, known for its adaptive learning rate and efficacy in handling large-scale problems. To our knowledge, this is the first time Adam has been employed to solve an SVR problem. To empirically validate the proposed HE-LSSVR model, we evaluate it on UCI, synthetic, and time series datasets. The experimental outcomes unequivocally reveal the superiority of the HE-LSSVR model both in terms of its remarkable generalization performance and its efficiency in training time.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. A new era: artificial intelligence and machine learning in prostate cancer. Nature Reviews Urology, 16(7):391–403, 2019.
  2. Overview of supervised learning. The elements of statistical learning: Data mining, inference, and prediction, pages 9–41, 2009.
  3. A taxonomy of traffic forecasting regression problems from a supervised learning perspective. IEEE Access, 7:68185–68205, 2019.
  4. Support vector regression machines. Advances in Neural Information Processing Systems, 9, 1996.
  5. Support-vector networks. Machine Learning, 20(3):273–297, 1995.
  6. Support vector machine embedding discriminative dictionary pair learning for pattern classification. Neural Networks, 155:498–511, 2022.
  7. Weighted incremental–decremental support vector machines for concept drift with shifting window. Neural Networks, 152:528–541, 2022.
  8. Robust cost-sensitive kernel method with Blinex loss and its applications in credit risk evaluation. Neural Networks, 143:327–344, 2021. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2021.06.016.
  9. Forecasting short-term electricity load using hybrid support vector regression with grey catastrophe and random forest modeling. Utilities Policy, 73:101294, 2021.
  10. Research on water temperature prediction based on improved support vector regression. Neural Computing and Applications, pages 1–10, 2022.
  11. Fine-tuned support vector regression model for stock predictions. Neural Computing and Applications, 35(32):23295–23309, 2023.
  12. Metaheuristic-based support vector regression for landslide displacement prediction: A comparative study. Landslides, 19(10):2489–2511, 2022.
  13. A novel method for petroleum and natural gas resource potential evaluation and prediction by support vector machines (svm). Applied Energy, 351:121836, 2023.
  14. State of health estimation of lithium-ion battery based on improved ant lion optimization and support vector regression. Journal of Energy Storage, 50:104215, 2022.
  15. Estimation of bank profitability using vector error correction model and support vector regression. Economic Alternatives, 28(2):157–170, 2022.
  16. Adaptive support vector regression for UAV flight control. Neural Networks, 24(1):109–120, 2011.
  17. Least squares support vector machine classifiers. Neural Processing Letters, 9:293–300, 1999.
  18. Predicting time series with support vector machines. In International Conference on Artificial Neural Networks, pages 999–1004. Springer, 1997.
  19. Douglas M Hawkins. Identification of outliers, volume 11. Springer, 1980.
  20. Support vector machines. Springer Science & Business Media, 2008.
  21. Trading convexity for scalability. In Proceedings of the 23rd International Conference on Machine Learning, pages 201–208, 2006.
  22. Robust non-convex least squares loss function for regression with outliers. Knowledge-Based Systems, 71:290–302, 2014.
  23. Ramp loss nonparallel support vector machine for pattern classification. Knowledge-Based Systems, 85:224–233, 2015.
  24. Robust support vector machine with generalized quantile loss for classification and regression. Applied Soft Computing, 81:105483, 2019.
  25. Robust support vector regression with generic quadratic nonconvex ε𝜀\varepsilonitalic_ε-insensitive loss. Applied Mathematical Modelling, 82:235–251, 2020.
  26. Kernel-based online regression with canal loss. European Journal of Operational Research, 297(1):268–279, 2022.
  27. Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4331–4339, 2019.
  28. Learning with a wasserstein loss. Advances in Neural Information Processing Systems, 28, 2015.
  29. Multi-sensor fusion methodology for enhanced land vehicle positioning. Information Fusion, 46:51–62, 2019.
  30. General multi-view semi-supervised least squares support vector machines with multi-manifold regularization. Information Fusion, 62:63–72, 2020.
  31. Ping Zhong. Training robust support vector regression with smooth non-convex loss function. Optimization Methods and Software, 27(6):1039–1058, 2012.
  32. The minimum sum of absolute errors regression: A state of the art survey. International Statistical Review/Revue Internationale de Statistique, pages 317–326, 1982.
  33. A new asymmetric ε𝜀\varepsilonitalic_ε-insensitive pinball loss function based support vector quantile regression model. Applied Soft Computing, 94:106473, 2020.
  34. S Balasundaram and Yogendra Meena. Robust support vector regression in primal with asymmetric huber loss. Neural Processing Letters, 49:1399–1431, 2019.
  35. On regularization based twin support vector regression with huber loss. Neural Processing Letters, 53(1):459–515, 2021.
  36. Ramp-loss nonparallel support vector regression: robust, sparse and scalable approximation. Knowledge-Based Systems, 147:55–67, 2018.
  37. Ramp loss least squares support vector machine. Journal of Computational Science, 14:61–68, 2016.
  38. The representer theorem for hilbert spaces: a necessary and sufficient condition. Advances in Neural Information Processing Systems, 25, 2012.
  39. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223–311, 2018.
  40. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  41. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7), 2011.
  42. Lecture 6.5-RMSprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4(2):26–31, 2012.
  43. Robust regression under the general framework of bounded loss functions. European Journal of Operational Research, 310(3):1325–1339, 2023.
  44. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):1–27, 2011.
  45. Kernel methods for pattern analysis. Cambridge University Press, 2004.
  46. UCI machine learning repository. 2017.
  47. Milton Friedman. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86–92, 1940.
  48. Approximations of the critical region of the fbietkan statistic. Communications in Statistics-Theory and Methods, 9(6):571–595, 1980.
  49. Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7:1–30, 2006.
  50. KEEL data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. J. Mult. Valued Logic Soft Comput, 17, 2015.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets