Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Condensed Gradient Boosting (2211.14599v2)

Published 26 Nov 2022 in cs.LG

Abstract: This paper presents a computationally efficient variant of gradient boosting for multi-class classification and multi-output regression tasks. Standard gradient boosting uses a 1-vs-all strategy for classifications tasks with more than two classes. This strategy translates in that one tree per class and iteration has to be trained. In this work, we propose the use of multi-output regressors as base models to handle the multi-class problem as a single task. In addition, the proposed modification allows the model to learn multi-output regression problems. An extensive comparison with other multi-ouptut based gradient boosting methods is carried out in terms of generalization and computational efficiency. The proposed method showed the best trade-off between generalization ability and training and predictions speeds.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. A gradient boosting approach for training convolutional and deep neural networks. IEEE Open Journal of Signal Processing, 4:313–321, 2023.
  2. Bmri-net: A deep stacked ensemble model for multi-class brain tumor classification from mri images. Interdisciplinary Sciences: Computational Life Sciences, 15(3):499–514, September 2023.
  3. An intelligent multi-output regression model for soil moisture prediction. In Intelligent and Fuzzy Techniques for Emerging Conditions and Digital Transformation, pages 474–481, Cham, 2022. Springer International Publishing.
  4. Multioutput regression neural network training via gradient boosting. In 30th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2022, Bruges, Belgium, October 5-7, 2022, 2022.
  5. Deep learning for multi-output regression using gradient boosting. IEEE Access, 12:17760–17772, 2024.
  6. Multiobjective semisupervised classifier ensemble. IEEE transactions on cybernetics, 49(6):2280–2293, 2018.
  7. Multi-task gradient boosting. In Hybrid Artificial Intelligent Systems, pages 97–107, Cham, 2023. Springer Nature Switzerland.
  8. Ensemble of multi-task learning networks for facial expression recognition in-the-wild with learning from synthetic data. In Computer Vision – ECCV 2022 Workshops, pages 60–75, Cham, 2023. Springer Nature Switzerland.
  9. Analysis and visualization of accidents severity based on lightgbm-tpe. Chaos, Solitons & Fractals, 157:111987, 2022.
  10. Predicting occurrence of liquefaction-induced lateral spreading using gradient boosting algorithms integrated with particle swarm optimization: Pso-xgboost, pso-lightgbm, and pso-catboost. Acta Geotechnica, 18(6):3403–3419, June 2023.
  11. Effective ensemble learning approach for large-scale medical data analytics. International Journal of System Assurance Engineering and Management, 15(1):13–20, 01 2024.
  12. An up-to-date comparison of state-of-the-art classification algorithms. Expert Systems with Applications, 82:128–150, 2017.
  13. A comparative analysis of gradient boosting algorithms. Artificial Intelligence Review, 54:1937–1967, 2021.
  14. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, 2022.
  15. Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189–1232, 2001.
  16. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA, 2016. Association for Computing Machinery.
  17. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  18. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3146–3154, 2017.
  19. Multi-output ls-svr machine in extended feature space. In 2012 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications (CIMSA) Proceedings, pages 130–134, 2012.
  20. Cost efficient gradient boosting. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1551–1561, 2017.
  21. Accelerating gradient boosting machines. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 516–526. PMLR, 2020.
  22. Ezgi Zorarpaci. A fast intrusion detection system based on swift wrapper feature selection and speedy ensemble classifier. Engineering Applications of Artificial Intelligence, 133:108162, 2024.
  23. Fasten: Fast ensemble learning for improved adversarial robustness. IEEE Transactions on Information Forensics and Security, 19:2565–2580, 2024.
  24. A survey on multi-output regression. WIREs Data Mining and Knowledge Discovery, 5(5):216–233, 2015.
  25. Multi-target regression via input space expansion: treating targets as inputs. Machine Learning, 104(1):55–98, Jul 2016.
  26. Multi-target regression via output space quantization. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–9, 2020.
  27. Compact multi-class boosted trees. In 2017 IEEE International Conference on Big Data (Big Data), pages 47–56, 2017.
  28. Gbdt-mo: Gradient-boosted decision trees for multiple outputs. IEEE Transactions on Neural Networks and Learning Systems, 32(7):3156–3167, 2021.
  29. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119–139, 1997.
  30. Direct optimization of margins improves generalization in combined classifiers. In Advances in Neural Information Processing Systems 11, [NIPS Conference, Denver, Colorado, USA, November 30 - December 5, 1998], pages 288–294. The MIT Press, 1998.
  31. Functional gradient techniques for combining hypotheses. Advances in Neural Information Processing Systems, pages 221–246, 1999.
  32. Multiple additive regression trees with application in epidemiology. Statistics in Medicine, 22(9):1365–1381, 2003.
  33. Multiclass boosting for weak classifiers. Journal of Machine Learning Research, 6:189–210, 2005.
  34. Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors). The Annals of Statistics, 28(2):337 – 407, 2000.
  35. Ping Li. Abc-boost: adaptive base class boost for multi-class classification. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages 625–632, New York, NY, USA, 2009. Association for Computing Machinery.
  36. Multi-step-ahead traffic speed forecasting using multi-output gradient boosting regression tree. Journal of Intelligent Transportation Systems, 24(2):125–141, 2020.
  37. Top-down induction of clustering trees. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998, pages 55–63. Morgan Kaufmann, 1998.
  38. Constraint based induction of multi-objective regression trees. In Knowledge Discovery in Inductive Databases, pages 222–233, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
  39. Classification and Regression Trees. Wadsworth, Oxfordshire, UK, 1984.
  40. Ensembles of multi-objective decision trees. In Machine Learning: ECML 2007, pages 624–631, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
  41. Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140, Aug 1996.
  42. Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 10 2001.
  43. Deep tree-ensembles for multi-output prediction. Pattern Recognition, 121:108211, 2022.
  44. Jerome H. Friedman. Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4):367–378, 2002. Nonlinear Methods and Data Mining.
  45. Isabelle M Guyon. Design of experiments for the nips 2003 variable selection benchmark. 2003. Last visited on April 28, 2024.
  46. Cifar-10 (canadian institute for advanced research). 2009. Last visited on April 28, 2024.
  47. The uci machine learning repository. https://archive.ics.uci.edu. Last accessed on April 28, 2024.
  48. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  49. Vehicle classification in distributed sensor networks. Journal of Parallel and Distributed Computing, 64(7):826–838, 2004. Computing and Communication in Distributed Sensor Networks.
  50. Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7:1–30, 2006.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - GAA-UAM/C-GB: Condensed-Gradient Boosting (3 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets