The minimal width of neural networks and immersions of real projective space

Abstract: We investigate the minimal width of deep neural networks that can approximate all continuous functions from $\mathbb{R}<sup>n$ to $\mathbb{R}<sup>m$ in the topology of uniform convergence on compact subsets. We show that whenever the real projective space $\mathbb{RP}<sup>{n-1}$ admits a smooth immersion into $\mathbb{R}<sup>n$, the minimal width is at least $n+\min{n,m}$, assuming that the activation function is continuous and monotone. As a consequence, the minimal width of ReLU networks is $n+m$ whenever $n$ is equal to 2, 3, 4, 7 or 8 and $1\le m \le n$, extending results of Hanin-Sellke and Hwang. This suggests an intriguing connection between the approximation theory of neural networks and differential topology.