Dice Question Streamline Icon: https://streamlinehq.com

Statistical Learning Theory vs. Deep Learning and the Role of Benchmarks

Determine whether statistical learning theory adequately accounts for the progress and methodology of deep learning; ascertain whether the emerging science of benchmarks proposed by Moritz Hardt complements statistical learning theory or competes with it as an epistemological foundation for machine learning.

Information Square Streamline Icon: https://streamlinehq.com

Background

The paper contrasts statistical learning theory (SLT)—a frequentist foundation emphasizing learnability, consistency, and risk control—with recent claims that SLT has limited relevance to the empirical successes of deep learning. In particular, Moritz Hardt has argued for an alternative or additional epistemological framework centered on benchmark-driven scientific practice.

Resolving the relationship between SLT and benchmark science would clarify the theoretical underpinnings of modern machine learning and guide future foundational work.

References

Open Question II (from Machine Learning). In the section on classification, I discussed the development of SLT (statistical learning theory) as an instance of achievabilist thinking—an approach that has long aspired to serve as the (frequentist) epistemological foundation of machine learning. However, it has recently been argued—most notably by computer scientists such as Moritz Hardt—that SLT has little to do with the actual advancement of one of the most fascinating and successful branches of modern machine learning: deep learning. In his keynote speech at the 2024 ICLR (International Conference on Learning Representations), Hardt proposed that deep learning requires a new epistemological foundation, which he dubbed the emerging science of benchmarks (https://iclr.cc/virtual/2024/invited-talk/21799). Is statistical learning theory mistaken? Does the emerging science of benchmarks complement, or compete with, statistical learning theory?

— A Plea for History and Philosophy of Statistics and Machine Learning (2506.22236 - Lin, 27 Jun 2025) in Section 9 (Closing): Open Question II (from Machine Learning)