Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration (2101.07077v8)

Published 18 Jan 2021 in cs.LG

Abstract: A decision tree looks like a simple directed acyclic computational graph, where only the leaf nodes specify the output values and the non-terminals specify their tests or split conditions. From the numerical perspective, we express decision trees in the language of computational graph. We explicitly parameterize the test phase, traversal phase and prediction phase of decision trees based on the bitvectors of non-terminal nodes. As shown, the decision tree is a shallow binary network in some sense. Especially, we introduce the bitvector matrix to implement the tree traversal in numerical approach, where the core is to convert the logical `AND' operation to arithmetic operations. And we apply this numerical representation to extend and unify diverse decision trees in concept.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Learning certifiably optimal rule lists for categorical data. The Journal of Machine Learning Research, 18(1):8753–8830, 2017.
  2. Leo Breiman. Classification and regression trees. Routledge, 2017.
  3. A hybrid decision tree/genetic algorithm method for data mining. Information Sciences, 163(1):13–35, 2004.
  4. Adaptively sparse transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2174–2184, Hong Kong, China, November 2019. Association for Computational Linguistics.
  5. Decision forests for computer vision and medical image analysis. Springer Science & Business Media, 2013.
  6. Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning. Foundations and Trends® in Computer Graphics and Vision, 7(2–3):81–227, 2012.
  7. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems (TOIS), 35(2):1–31, 2016.
  8. Convergence of a relaxed variable splitting coarse gradient descent method for learning sparse weight binarized activation neural networks. arXiv: Optimization and Control, 2019.
  9. JH Fridedman. Multivariate adaptive regression splines (with discussion). Ann. Statist, 19(1):79–141, 1991.
  10. Entity embeddings of categorical variables. arXiv: Learning, 2016.
  11. The tree ensemble layer: Differentiability meets conditional computation. 2020.
  12. End-to-end learning of decision trees and forests. International Journal of Computer Vision, pages 1–15, 2019.
  13. Andreas Holzinger. Data mining with decision trees: Theory and applications. Online Information Review, 39(3):437–438, 2015.
  14. Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical Statistics, 15(3):651–674, 2006.
  15. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
  16. Sethi I K. Entropy nets: from decision trees to neural networks. Proceedings of the IEEE, 78(10), 1990.
  17. Sigsoftmax: Reanalysis of the softmax bottleneck. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 286–296. Curran Associates, Inc., 2018.
  18. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, pages 1467–1475, 2015.
  19. Sotiris B. Kotsiantis. A hybrid decision tree classifier. Journal of Intelligent and Fuzzy Systems, 26(1):327–336, 2014.
  20. On controllable sparse alternatives to softmax. In Advances in Neural Information Processing Systems, pages 6422–6432, 2018.
  21. GPU-based parallelization of QuickScorer to speed-up document ranking with tree ensembles. In 7th Italian Information Retrieval Workshop, IIR 2016, volume 1653. CEUR-WS, 2016.
  22. Weiyin Loh. Fifty years of classification and regression trees. International Statistical Review, 82(3):329–348, 2014.
  23. QuickScorer: A fast algorithm to rank documents with additive ensembles of regression trees. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 73–82, 2015.
  24. QuickScorer: Efficient traversal of large ensembles of decision trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 383–387. Springer, 2017.
  25. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623, 2016.
  26. A regularized framework for sparse and structured neural attention. In Advances in Neural Information Processing Systems, pages 3338–3348, 2017.
  27. Efficient non-greedy optimization of decision trees. In Advances in neural information processing systems, pages 1729–1737, 2015.
  28. Finbarr O’Sullivan. Discussion: Multivariate adaptive regression splines. Ann. Statist, 19(1):99–102, 1991.
  29. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019.
  30. Catboost: unbiased boosting with categorical features. arXiv: Learning, 2017.
  31. The power of unbiased recursive partitioning: A unifying view of ctree, mob, and guide. arXiv: Methodology, 2019.
  32. Maximum likelihood regression trees. Journal of Computational and Graphical Statistics, 13(3):586–598, 2004.
  33. NBDT: Neural-backed decision trees, 2020.
  34. Explaining the success of adaboost and random forests as interpolating classifiers. The Journal of Machine Learning Research, 18(1):1558–1590, 2017.
  35. Deep neural decision trees. arXiv: Learning, 2018.
  36. RapidScorer: fast tree ensemble evaluation by maximizing compactness in data level parallelization. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 941–950, 2018.
  37. Recursive partitioning and applications. Springer Science & Business Media, 2010.
  38. Hybrid decision tree. Knowledge Based Systems, 15(8):515–528, 2002.
  39. Zhihua Zhou and Ji Feng. Deep forest: Towards an alternative to deep neural networks. arXiv: Learning, 2017.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now