On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory (2412.11521v2)

Abstract: Symmetries (transformations by group actions) are present in many datasets, and leveraging them holds considerable promise for improving predictions in machine learning. In this work, we aim to understand when and how deep networks -- with standard architectures trained in a standard, supervised way -- learn symmetries from data. Inspired by real-world scenarios, we study a classification paradigm where data symmetries are only partially observed during training: some classes include all transformations of a cyclic group, while others -- only a subset. In the infinite-width limit, where kernel analogies apply, we derive a neural kernel theory of symmetry learning. The group-cyclic nature of the dataset allows us to analyze the Gram matrix of neural kernels in the Fourier domain; here we find a simple characterization of the generalization error as a function of class separation (signal) and class-orbit density (noise). This characterization reveals that generalization can only be successful when the local structure of the data prevails over its non-local, symmetry-induced structure, in the kernel space defined by the architecture. We extend our theoretical treatment to any finite group, including non-abelian groups. Our framework also applies to equivariant architectures (e.g., CNNs), and recovers their success in the special case where the architecture matches the inherent symmetry of the data. Empirically, our theory reproduces the generalization failure of finite-width networks (MLP, CNN, ViT) trained on partially observed versions of rotated-MNIST. We conclude that conventional deep networks lack a mechanism to learn symmetries that have not been explicitly embedded in their architecture a priori. Our framework could be extended to guide the design of architectures and training procedures able to learn symmetries from data.

• The paper introduces a neural kernel theory to demonstrate how deep networks learn symmetry from data with partially observed transformations.
• It empirically shows that conventional networks struggle to generalize symmetry invariance in settings like rotated-MNIST unless symmetry is built into the architecture.
• It highlights that effective symmetry generalization depends on class separation and orbit density, informing future architectural designs.

On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory

The paper explores a significant question in the domain of machine learning: under what conditions can deep networks learn symmetries from data, specifically in a supervised classification context where symmetries in the data are only partially observed during training? The analysis is developed using a neural kernel theory, leveraging the infinite-width limit of neural networks akin to kernel machines. The authors investigate how these networks can generalize symmetry invariance to unseen classes, particularly when certain classes in training data provide exhaustive samples of transformations within a cyclic group, while others present only partial rotations.

Core Contributions

1. Neural Kernel Theory for Symmetry Learning: The authors present an in-depth analysis using a neural kernel theory framework to establish how symmetries can be learned by deep networks. The focus remains on their ability to generalize from cyclic group transformations that are partially observed. The work emphasizes understanding when the local structure of data overpowers non-local symmetric structures in the kernel space defined by the network's architecture.
2. Empirical and Theoretical Investigations: The authors apply their theoretical framework to empirical data using neural networks with varying architectures (e.g., MLPs, CNNs, ViTs) trained on the rotated-MNIST dataset. The theory is corroborated by experiments which align with theoretical predictions, showing that conventional neural networks fail to generalize symmetry invariances that are not explicitly incorporated into the architecture a priori.
3. Impact of Class Separation and Orbit Density: A central aspect of their findings is the relationship between the successful generalization of symmetry invariance and two key factors: the separation between classes and the density of class orbits. The authors derive a simple characterization where successful generalization occurs only when data exhibit distinguished local structures within these parameters.
4. Application to Equivariant Architectures: The framework is also applicable to architectures explicitly designed to leverage symmetry, such as Convolutional Neural Networks (CNNs) with invariances matched to the dataset symmetry. The paper highlights that these networks can sometimes achieve better generalization through architectural alignment with inherent data symmetries.

Implications and Future Directions

The implications of this paper are two-fold, encompassing both immediate applications and longer-term theoretical impacts. Practically, the paper suggests that for deep networks to aptly leverage symmetries not hardcoded into their structure, new architectural and training strategy designs are necessary. In future research, extending this approach could lead to novel methods that allow neural networks to internally adjust their kernel representations based on the symmetry structure of unseen data.

Theoretically, the insights garnered from this paper extend our understanding of kernel methods in neural networks, especially regarding how these networks can or cannot adapt to inherent data symmetry in absence of explicit mechanisms. This can fuel future work exploring adaptive mechanisms in network architectures that automatically align with data symmetries during learning.

In conclusion, this paper delineates the limitations of conventional deep networks in learning unobserved data symmetries and introduces a kernel-theoretic approach to enhancing our understanding of symmetry learning. Further research could illuminate paths toward architectures capable of autonomously inferring and applying symmetries, thus advancing both the theory and application of machine learning to complex, real-world datasets.