Why do CNNs excel at feature extraction? A mathematical explanation (2307.00919v1)
Abstract: Over the past decade deep learning has revolutionized the field of computer vision, with convolutional neural network models proving to be very effective for image classification benchmarks. However, a fundamental theoretical questions remain answered: why can they solve discrete image classification tasks that involve feature extraction? We address this question in this paper by introducing a novel mathematical model for image classification, based on feature extraction, that can be used to generate images resembling real-world datasets. We show that convolutional neural network classifiers can solve these image classification tasks with zero error. In our proof, we construct piecewise linear functions that detect the presence of features, and show that they can be realized by a convolutional network.
- Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics 7(10), 2019.
- Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, 2019.
- Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. In Advances in Neural Information Processing Systems, 2019.
- Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710.11278, 2017.
- Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks. 4(2), 1991.
- Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
- Matus Telgarsky. Representation benefits of deep feedforward networks. ArXiv preprint arXiv:1509.08101, 2015.