Neural Fourier Transform: A General Approach to Equivariant Representation Learning

Abstract: Symmetry learning has proven to be an effective approach for extracting the hidden structure of data, with the concept of equivariance relation playing the central role. However, most of the current studies are built on architectural theory and corresponding assumptions on the form of data. We propose Neural Fourier Transform (NFT), a general framework of learning the latent linear action of the group without assuming explicit knowledge of how the group acts on data. We present the theoretical foundations of NFT and show that the existence of a linear equivariant feature, which has been assumed ubiquitously in equivariance learning, is equivalent to the existence of a group invariant kernel on the dataspace. We also provide experimental results to demonstrate the application of NFT in typical scenarios with varying levels of knowledge about the acting group.

• The paper introduces NFT as a nonlinear extension of the Fourier Transform to learn latent equivariant representations without explicit group actions.
• It employs an encoder-decoder architecture to identify and reconstruct symmetry-driven subspaces, enabling finite-dimensional approximations.
• Experimental results on 1D time series, images, and 3D scenes demonstrate NFT’s superior ability to handle complex, nonlinear symmetry distortions.

Neural Fourier Transform: A General Approach to Equivariant Representation Learning

This paper introduces the Neural Fourier Transform (NFT) framework, which seeks to advance the understanding and application of symmetries in data by developing a method for learning equivariant representations of data without prior explicit knowledge of group actions on the data. The NFT aims to learn latent linear actions of groups on data using only tuples of observations. This approach diverges from traditional Fourier Transform (FT) and Group Convolution (GC), which require known group actions and often assume linearity on inputs.

Key Contributions

The core contribution of this paper is the introduction of NFT, which extends FT by enabling a nonlinear characterization of equivariant representations. The authors present NFT as a solution to overcome the limitations of conventional FT and GC that necessitate pre-defined and tractable knowledge of how symmetries act on data. The NFT framework comprises two main components: an encoder that maps data tuples to a latent space with linearizing actions, and a decoder that reconstructs data from this latent space, similar to how FT operates on linear spaces.

Theoretical Foundations:

1. Existence and Uniqueness: The authors establish the existence of a latent linear equivariant space underpinned by the presence of a group-invariant kernel on data space. This existence proof implies that data can be decomposed into direct sums of action-equivariant subspaces.
2. Finite-dimensional Approximation: When faced with limited latent dimension, NFT acts as a nonlinear spectral analysis tool that selectively emphasizes dominant modes of symmetry. This is especially critical in contexts where only a truncated or constrained latent representation is feasible.
3. Rich Theory Interconnected with Kernel Methods: The framework provides a clear connection between invariant kernels and equivariant mappings, aligning with the tradition of kernel-based learning but extending it to encompass group symmetries and actions.

Experimental Results

The NFT was evaluated on multiple datasets encompassing different data types and symmetry scenarios:

• 1D Time Series: NFT surpassed classical DFT in deciphering dominant frequencies in time-warped series where classical methods fall short due to nonlinear distortions.
• Image Shifts and Rotations: NFT proficiently handled image transitions under fisheye distortion and learned meaningful symmetries without prior assumptions about the group's nature.
• 3D Object and Scene Tasks: Application in novel view prediction of 3D scenes demonstrated NFT’s capability to synthesize new views despite occlusions and non-invertibility in the observation space of pixels.

Implications and Future Work

The introduction of NFT challenges extant assumptions in equivariant learning, especially the dependence on known tractable group actions. The theoretical results suggest that latent equivariant spaces can be identified and utilized without explicit symmetry assumptions, enhancing the potential for unsupervised and semi-supervised learning paradigms. This paper posits future research directions in refining NFT's application to more varied group symmetries, integrating it with real-world datasets that exhibit complex, often unknown, symmetries, and constructing architectures that further exploit the learned equivariant features for downstream tasks such as anomaly detection or unsupervised clustering in high-dimensional data streams.

The work pushes for an exploration of NFT's efficacy in domains where nonlinear environmental interactions affect observable phenomena, urging the community to consider broader aspects of symmetry learning through data-driven latent actions. The NFT framework thus opens a path towards more adaptive, symmetry-aware machine learning models, balancing theoretical robustness with empirical richness.