Algebraic Networks and Architectural Degenerations

Abstract: We study the geometry of polynomial neural networks with monomial activation functions and no bias. We introduce a general framework of algebraic networks, together with their realization maps and associated affine neurovarieties. In this setting we define morphisms, subnetworks, symmetry groups and quotient parameter spaces and we discuss geometric notions of identifiability and reducibility. Our main goal is to relate the singularities of neurovarieties to degenerations of the underlying architecture. For fully connected networks, we define the architectural degeneracy locus as the locus of functions admitting a representation by parameters with a rank-deficient layer or an inactive hidden neuron. We prove that, for fully connected networks with non-increasing widths and scalar output, full parameters give smooth points of the corresponding neurovariety under explicit layerwise regularity assumptions. In particular, for these architectures, the singular locus is contained in the architectural degeneracy locus.