Deep Learning Works in Practice. But Does it Work in Theory? (1801.10437v1)

Abstract: Deep learning relies on a very specific kind of neural networks: those superposing several neural layers. In the last few years, deep learning achieved major breakthroughs in many tasks such as image analysis, speech recognition, natural language processing, and so on. Yet, there is no theoretical explanation of this success. In particular, it is not clear why the deeper the network, the better it actually performs. We argue that the explanation is intimately connected to a key feature of the data collected from our surrounding universe to feed the machine learning algorithms: large non-parallelizable logical depth. Roughly speaking, we conjecture that the shortest computational descriptions of the universe are algorithms with inherently large computation times, even when a large number of computers are available for parallelization. Interestingly, this conjecture, combined with the folklore conjecture in theoretical computer science that $ P \neq NC$, explains the success of deep learning.

• The paper investigates the theoretical reasons behind deep learning's practical success, proposing that specific properties of data, such as high Kolmogorov complexity and non-parallelizable logical depth, are key.
• The authors conjecture that relevant data often exhibits high non-parallelizable logical depth, which measures the computational effort required to generate it from a compact representation.
• It is proposed that deeper neural networks are more effective at handling this logical depth than shallower models, suggesting an inherent computational advantage tied to network depth.

Analyzing the Theoretical Foundations of Deep Learning's Success

The paper "Deep Learning Works in Practice. But Does it Work in Theory?" by Lê Nguy^en Hoang and Rachid Guerraoui provides a thought-provoking investigation into the theoretical underpinnings of deep learning's empirical success. Despite deep learning's substantial achievements in diverse fields such as image analysis, speech recognition, and natural language processing, the absence of a comprehensive theoretical explanation for these successes remains a compelling topic of inquiry.

The Conjectures

The authors introduce several conjectures to explore why deep learning is effective, focusing on the properties of data the machine learning models utilize:

1. Complexity of Data: The paper posits that most data relevant for machine learning from our universe has a Kolmogorov complexity exceeding 10^9 bits. This implies that traditional hand-coded algorithms may be inadequate because the potential solution space is vast and complex.
2. Non-Parallelizable Logical Depth: The authors propose the conjecture that considerable portions of data exhibit large non-parallelizable logical depth—a measure capturing the computational effort required to generate a given dataset from its most compact representation. They argue that the observed phenomena in our universe inherently possess this depth, necessitating algorithms capable of handling such complexity.
3. Depth in Neural Networks: The final conjecture is that deeper neural networks can accommodate this logical depth more effectively than shallower models. This stems from the assumption that deeper networks can handle more complex, non-parallelizable operations over a larger number of computational steps than can be parallelized in shallower networks.

Implications for Deep Learning

The paper's conjectures build a theoretical framework that correlates with practical findings—deep learning utilizes depth to capture complex features in data that shallower methods may overlook. The authors argue that deep networks' ability to compute functions with significant logical depth is an essential factor in their success. This property aligns with the broader belief in the theoretical computer science community that $P \neq NC$, suggesting certain computational tasks inherently resist parallelization.

Future Directions

The paper acknowledges the nascent stage of formalizing these theoretical insights and invites further exploration in several areas:

• Formal Definitions: Developing formal definitions of non-parallelizable logical depth that can be applied rigorously to neural networks and real-world data.
• Mathematical Proofs: Establishing proofs that demonstrate the necessity of deep networks for computing tasks with high logical depth, thereby differentiating them from shallower structures.
• Practical Evaluations: Quantifying the logical depth of datasets and mapping such metrics to the required neural network structures needed for effective processing.

Conclusion

The paper by Hoang and Guerraoui provides valuable conjectures and insights into the theoretical mechanisms that could explain deep learning's effectiveness. By bridging empirical successes with theoretical constructs, the authors offer a foundation for understanding why deeper network structures succeed where other computational approaches may falter. They posit that deep learning's ability to navigate complexities in data with high Kolmogorov complexity and logical depth is key to its capability, stimulating future research at the intersection of theoretical computer science and machine learning.