Topological obstructions in neural networks learning (2012.15834v2)

Abstract: We apply topological data analysis methods to loss functions to gain insights into learning of deep neural networks and deep neural networks generalization properties. We use the Morse complex of the loss function to relate the local behavior of gradient descent trajectories with global properties of the loss surface. We define the neural network Topological Obstructions score, "TO-score", with the help of robust topological invariants, barcodes of the loss function, that quantify the "badness" of local minima for gradient-based optimization. We have made experiments for computing these invariants for fully-connected, convolutional and ResNet-like neural networks on different datasets: MNIST, Fashion MNIST, CIFAR10, CIFAR100 and SVHN. Our two principal observations are as follows. Firstly, the neural network barcode and TO score decrease with the increase of the neural network depth and width, thus the topological obstructions to learning diminish. Secondly, in certain situations there is an intriguing connection between the lengths of minima segments in the barcode and the minima generalization errors.