On the effect of pooling on the geometry of representations (1703.06726v1)

Abstract: In machine learning and neuroscience, certain computational structures and algorithms are known to yield disentangled representations without us understanding why, the most striking examples being perhaps convolutional neural networks and the ventral stream of the visual cortex in humans and primates. As for the latter, it was conjectured that representations may be disentangled by being flattened progressively and at a local scale. An attempt at a formalization of the role of invariance in learning representations was made recently, being referred to as I-theory. In this framework and using the language of differential geometry, we show that pooling over a group of transformations of the input contracts the metric and reduces its curvature, and provide quantitative bounds, in the aim of moving towards a theoretical understanding on how to disentangle representations.