Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Properly Learning Decision Trees with Queries Is NP-Hard (2307.04093v1)

Published 9 Jul 2023 in cs.CC, cs.DS, and cs.LG

Abstract: We prove that it is NP-hard to properly PAC learn decision trees with queries, resolving a longstanding open problem in learning theory (Bshouty 1993; Guijarro-Lavin-Raghavan 1999; Mehta-Raghavan 2002; Feldman 2016). While there has been a long line of work, dating back to (Pitt-Valiant 1988), establishing the hardness of properly learning decision trees from random examples, the more challenging setting of query learners necessitates different techniques and there were no previous lower bounds. En route to our main result, we simplify and strengthen the best known lower bounds for a different problem of Decision Tree Minimization (Zantema-Bodlaender 2000; Sieling 2003). On a technical level, we introduce the notion of hardness distillation, which we study for decision tree complexity but can be considered for any complexity measure: for a function that requires large decision trees, we give a general method for identifying a small set of inputs that is responsible for its complexity. Our technique even rules out query learners that are allowed constant error. This contrasts with existing lower bounds for the setting of random examples which only hold for inverse-polynomial error. Our result, taken together with a recent almost-polynomial time query algorithm for properly learning decision trees under the uniform distribution (Blanc-Lange-Qiao-Tan 2022), demonstrates the dramatic impact of distributional assumptions on the problem.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. The complexity of properly learning simple concept classes. Journal of Computer & System Sciences, 74(1):16–34, 2009. Preliminary version in FOCS 2004.
  2. Approximating optimal binary decision trees. Algorithmica, 62(3-4):1112–1121, 2012.
  3. Proof verification and the hardness of approximation problems. J. ACM, 45(3):501–555, may 1998.
  4. Dana Angluin. Remarks on the difficulty of finding a minimal disjunctive normal form for boolean functions. Unpublished Manuscript.
  5. Dana Angluin. Queries and concept learning. Machine learning, 2:319–342, 1988.
  6. Probabilistic checking of proofs: A new characterization of NP. J. ACM, 45(1):70–122, jan 1998.
  7. On the proper learning of axis-parallel concepts. The Journal of Machine Learning Research, 4:157–176, 2003.
  8. Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM (JACM), 36(4):929–965, 1989.
  9. Weakly learning DNF and characterizing statistical query learning using Fourier analysis. In Proceedings of the 26th Annual ACM Symposium on Theory of Computing (STOC), pages 253–262, 1994.
  10. Interpretability via model extraction. In Proceedings of the 4th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML), 2017.
  11. Decision tree heuristics can fail, even in the smoothed setting. In Proceedings of the 25th International Conference on Randomization and Computation (RANDOM), volume 207, pages 45:1–45:16, 2021.
  12. Properly learning decision trees in almost polynomial time. Journal of the ACM (JACM), 69(6):39:1–39:19, 2022.
  13. Leo Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 16(3):199–231, 2001.
  14. Born again trees. Technical report, University of California, Berkeley, 1996.
  15. Nader Bshouty. Exact learning via the monotone theory. In Proceedings of 34th Annual Symposium on Foundations of Computer Science (FOCS), pages 302–311, 1993.
  16. Nader H. Bshouty. Superpolynomial lower bounds for learning monotone classes. Electron. Colloquium Comput. Complex., TR23-006, 2023.
  17. Short PCPs with polylog query complexity. SIAM Journal on Computing, 38(2):551–607, 2008.
  18. Decision trees for entity identification: Approximation algorithms and hardness results. In Proceedings of the 26th ACM Symposium on Principles of Database Systems (PODS), pages 53–62, 2007.
  19. Extracting tree-structured representations of trained networks. Proceedings of the 8th Conference on Advances in Neural Information Processing Systems (NeurIPS), 8:24–30, 1995.
  20. Irit Dinur. The PCP theorem by gap amplification. J. ACM, 54(3):12–es, jun 2007.
  21. Learning decision trees from random examples. Information and Computation, 82(3):231–246, 1989.
  22. Vitaly Feldman. Hardness of approximate two-level logic minimization and pac learning with membership queries. In Proceedings of the 38th Annual ACM Symposium on Theory of Computing (STOC), pages 363–372, 2006.
  23. Vitaly Feldman. Hardness of proper learning. In Encyclopedia of Algorithms, pages 897–900. 2016.
  24. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784, 2017.
  25. Property testing and its connection to learning and approximation. Journal of the ACM, 45:653–750, 1998.
  26. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, 1979.
  27. Exact learning when irrelevant variables abound. Information Processing Letters, 70(5):233–239, 1999.
  28. David Haussler. Quantifying inductive bias: AI learning algorithms and valiant’s learning framework. Artificial Intelligence, 36(2):177–221, 1988.
  29. Lower bounds on learning decision lists and trees. Information and Computation, 126(2):114–122, 1996.
  30. Constructing optimal binary decision trees is NP-complete. Information processing letters, 5(1):15–17, 1976.
  31. Learning decision trees using the Fourier spectrum. SIAM Journal on Computing, 22(6):1331–1348, December 1993.
  32. On an optimal split tree problem. In Workshop on Algorithms and Data Structures, pages 157–168. Springer, 1999.
  33. Superpolynomial lower bounds for decision tree learning and testing. In Proceedings of the 2023 ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1962–1994, 2023.
  34. Leonid A Levin. Universal sorting problem. Problemy Predaci Informacii, 9:265–266, 1973.
  35. On the hardness of the minimum height decision tree problem. Discrete Applied Mathematics, 144(1-2):209–212, 2004.
  36. Decision tree approximations of boolean functions. Theoretical Computer Science, 270(1-2):609–623, 2002.
  37. Computational limitations on learning from examples. Journal of the ACM (JACM), 35(4):965–984, 1988.
  38. Optimization, approximation, and complexity classes. Journal of Computer and System Sciences, 43(3):425–440, 1991.
  39. Netanel Raviv. Truth table minimization of computational models. CoRR, abs/1306.3766, 2013.
  40. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1 – 85, 2022.
  41. The hardness of the expected decision depth problem. Information processing letters, 101(3):112–118, 2007.
  42. Detlef Sieling. Minimization of decision trees is hard to approximate. Journal of Computer and System Sciences, 74(3):394–403, 2008.
  43. Learning sparse multivariate polynomials over a field with queries and counterexamples. In Proceedings of the 6th Annual Conference on Computational Learning Theory (COLT), pages 17–26, 1993.
  44. Luca Trevisan. Inapproximability of Combinatorial Optimization Problems, chapter 13, pages 381–434. John Wiley & Sons, Ltd, 2014.
  45. Seeing the forest through the trees: Learning a comprehensible model from an ensemble. In European Conference on Machine Learning (ECML), pages 418–429, 2007.
  46. Leslie Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
  47. Leslie G Valiant. Learning disjunction of conjunctions. In Proceedings of the 9th International Joint Conference on Artificial Intelligence (IJCAI), pages 560–566, 1985.
  48. A genetic algorithm for interpretable model extraction from decision tree ensembles. In Trends and Applications in Knowledge Discovery and Data Mining, pages 104–115, 2017.
  49. Born-again tree ensembles. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 9743–9753, 2020.
  50. Finding small equivalent decision trees is hard. International Journal of Foundations of Computer Science, 11(2):343–354, 2000.
  51. Interpreting models via single tree approximation, 2016.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now