Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Federated Learning Benchmark on Tabular Data: Comparing Tree-Based Models and Neural Networks (2405.02074v1)

Published 3 May 2024 in cs.LG

Abstract: Federated Learning (FL) has lately gained traction as it addresses how machine learning models train on distributed datasets. FL was designed for parametric models, namely Deep Neural Networks (DNNs).Thus, it has shown promise on image and text tasks. However, FL for tabular data has received little attention. Tree-Based Models (TBMs) have been considered to perform better on tabular data and they are starting to see FL integrations. In this study, we benchmark federated TBMs and DNNs for horizontal FL, with varying data partitions, on 10 well-known tabular datasets. Our novel benchmark results indicates that current federated boosted TBMs perform better than federated DNNs in different data partitions. Furthermore, a federated XGBoost outperforms all other models. Lastly, we find that federated TBMs perform better than federated parametric models, even when increasing the number of clients significantly.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. J. P. Albrecht, “How the gdpr will change the world,” Eur. Data Prot. L. Rev., vol. 2, p. 287, 2016.
  2. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492, 2016.
  3. S. Banabilah, M. Aloqaily, E. Alsayed, N. Malik, and Y. Jararweh, “Federated learning review: Fundamentals, enabling technologies, and future applications,” Information Processing & Management, vol. 59, no. 6, p. 103061, 2022.
  4. K. Doshi and Y. Yilmaz, “Federated learning-based driver activity recognition for edge devices,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3338–3346, 2022.
  5. C. Kern, T. Klausch, and F. Kreuter, “Tree-based machine learning methods for survey research,” in Survey research methods, vol. 13, p. 73, NIH Public Access, 2019.
  6. Y. Liu, Y. Liu, Z. Liu, Y. Liang, C. Meng, J. Zhang, and Y. Zheng, “Federated forest,” IEEE Transactions on Big Data, 2020.
  7. Q. Li, Y. Cai, Y. Han, C. M. Yung, T. Fu, and B. He, “Fedtree: A fast, effective, and secure tree-based federated learning system.” https://github.com/Xtra-Computing/FedTree/blob/main/FedTree_draft_paper.pdf, 2022.
  8. Z. Tian, R. Zhang, X. Hou, J. Liu, and K. Ren, “Federboost: Private federated learning for gbdt,” arXiv preprint arXiv:2011.02796, 2020.
  9. Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965–978, IEEE, 2022.
  10. T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, K. Chen, et al., “Xgboost: extreme gradient boosting,” R package version 0.4-2, vol. 1, no. 4, pp. 1–4, 2015.
  11. S. Popov, S. Morozov, and A. Babenko, “Neural oblivious decision ensembles for deep learning on tabular data,” arXiv preprint arXiv:1909.06312, 2019.
  12. P. Kontschieder, M. Fiterau, A. Criminisi, and S. R. Bulo, “Deep neural decision forests,” in Proceedings of the IEEE international conference on computer vision, pp. 1467–1475, 2015.
  13. Y. Yang, I. G. Morillo, and T. M. Hospedales, “Deep neural decision trees,” arXiv preprint arXiv:1806.06988, 2018.
  14. B. Peters, V. Niculae, and A. F. Martins, “Sparse sequence-to-sequence models,” arXiv preprint arXiv:1905.05702, 2019.
  15. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015.
  16. MIT press, 2016.
  17. Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko, “Revisiting deep learning models for tabular data,” Advances in Neural Information Processing Systems, vol. 34, pp. 18932–18943, 2021.
  18. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  19. A. Kadra, M. Lindauer, F. Hutter, and J. Grabocka, “Well-tuned simple nets excel on tabular datasets,” Advances in neural information processing systems, vol. 34, pp. 23928–23941, 2021.
  20. R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,” Information Fusion, vol. 81, pp. 84–90, 2022.
  21. L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on tabular data?,” arXiv preprint arXiv:2207.08815, 2022.
  22. M. Iman, A. Giuntini, H. R. Arabnia, and K. Rasheed, “A comparative study of machine learning models for tabular data through challenge of monitoring parkinson’s disease progression using voice recordings,” in Advances in Computer Vision and Computational Biology, pp. 485–496, Springer, 2021.
  23. P. Shankar, A. A. Modi, and M. Liwicki, “Tree-based ensembles vs neuron-based methods for tabular data-a case study in crop disease forecasting,” Artificial intelligence in agriculture, 2022.
  24. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  25. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  26. S. Ö. Arik and T. Pfister, “Tabnet: Attentive interpretable tabular learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 6679–6687, 2021.
  27. X. Huang, A. Khetan, M. Cvitkovic, and Z. Karnin, “Tabtransformer: Tabular data modeling using contextual embeddings,” arXiv preprint arXiv:2012.06678, 2020.
  28. W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang, “Autoint: Automatic feature interaction learning via self-attentive neural networks,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1161–1170, 2019.
  29. Y. Liu, T. Fan, T. Chen, Q. Xu, and Q. Yang, “Fate: An industrial grade platform for collaborative learning with data protection,” J. Mach. Learn. Res., vol. 22, jan 2021.
  30. D. J. Beutel, T. Topal, A. Mathur, X. Qiu, T. Parcollet, P. P. de Gusmão, and N. D. Lane, “Flower: A friendly federated learning research framework,” arXiv preprint arXiv:2007.14390, 2020.
  31. A. Ziller, A. Trask, A. Lopardo, B. Szymkow, B. Wagner, E. Bluemke, J.-M. Nounahon, J. Passerat-Palmbach, K. Prakash, N. Rose, et al., “Pysyft: A library for easy federated learning,” in Federated Learning Systems, pp. 111–139, Springer, 2021.
  32. D. Dimitriadis, M. H. Garcia, D. M. Diaz, A. Manoel, and R. Sim, “Flute: A scalable, extensible framework for high-performance federated learning simulations,” arXiv preprint arXiv:2203.13789, 2022.
  33. C. He, S. Li, J. So, M. Zhang, H. Wang, X. Wang, P. Vepakomma, A. Singh, H. Qiu, L. Shen, P. Zhao, Y. Kang, Y. Liu, R. Raskar, Q. Yang, M. Annavaram, and S. Avestimehr, “Fedml: A research library and benchmark for federated machine learning,” Advances in Neural Information Processing Systems, Best Paper Award at Federate Learning Workshop, 2020.
  34. M. Yang, L. Song, J. Xu, C. Li, and G. Tan, “The tradeoff between privacy and accuracy in anomaly detection using federated xgboost,” arXiv preprint arXiv:1907.07157, 2019.
  35. IBM, “IBM federated learning xgboost tutorial for ui,” 2022.
  36. Q. Li, B. He, and D. Song, “Practical one-shot federated learning for cross-silo setting,” arXiv preprint arXiv:2010.01017, 2020.
  37. X. Liu, T. Shi, C. Xie, Q. Li, K. Hu, H. Kim, X. Xu, B. Li, and D. Song, “Unifed: A benchmark for federated learning frameworks,” arXiv preprint arXiv:2207.10308, 2022.
  38. Y. Wang, Z. Pan, J. Zheng, L. Qian, and M. Li, “A hybrid ensemble method for pulsar candidate classification,” Astrophysics and Space Science, vol. 364, pp. 1–13, 2019.
  39. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020.
  40. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450, 2020.
  41. J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “A novel framework for the analysis and design of heterogeneous federated learning,” IEEE Transactions on Signal Processing, vol. 69, pp. 5234–5249, 2021.
  42. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning, pp. 5132–5143, PMLR, 2020.
  43. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017.
  44. H. Zhu, J. Xu, S. Liu, and Y. Jin, “Federated learning on non-iid data: A survey,” Neurocomputing, vol. 465, pp. 371–390, 2021.
  45. W. Lindskog and C. Prehofer, “Federated learning for tabular data using tabnet: A vehicular use-case,” in IEEE 18th International Conference on Intelligent Computer Communication and Processing, IEEE Institute of Electrical and Electronic Engineers, 2022.
  46. V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci, “Deep neural networks and tabular data: A survey,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  47. K. Cheng, T. Fan, Y. Jin, Y. Liu, T. Chen, D. Papadopoulos, and Q. Yang, “Secureboost: A lossless federated learning framework,” IEEE Intelligent Systems, vol. 36, no. 6, pp. 87–98, 2021.
  48. H. R. Roth, Y. Cheng, Y. Wen, I. Yang, Z. Xu, Y.-T. Hsieh, K. Kersten, A. Harouni, C. Zhao, K. Lu, et al., “Nvidia flare: Federated learning from simulation to real-world,” arXiv preprint arXiv:2210.13291, 2022.
  49. J. Huang, “Maximum likelihood estimation of dirichlet distribution parameters,” CMU Technique report, vol. 18, 2005.
  50. S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” arXiv preprint arXiv:1812.01097, 2018.
  51. Kaggle, “Kaggle insurance premium prediction,” 2020.
  52. D. Dua and C. Graff, “UCI machine learning repository,” 2017.
  53. R. K. Pace and R. Barry, “Sparse spatial autoregressions,” Statistics & Probability Letters, vol. 33, no. 3, pp. 291–297, 1997.
  54. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  55. L. A. C. de Souza, G. A. F. Rebello, G. F. Camilo, L. C. Guimarães, and O. C. M. Duarte, “Dfedforest: Decentralized federated forest,” in 2020 IEEE International conference on blockchain (blockchain), pp. 90–97, IEEE, 2020.
  56. A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets