Spectral Representations for Convolutional Neural Networks (1506.03767v1)

Abstract: Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs). We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.

• The paper introduces spectral pooling and spectral parametrization to enhance CNN efficiency and training by operating in the frequency domain.
• It demonstrates that frequency domain representations reduce redundancy and accelerate convergence by 2 to 5 times compared to spatial methods.
• Empirical results on benchmarks like CIFAR-10/100 show improved information retention and competitive classification performance over traditional pooling techniques.

Spectral Representations for Convolutional Neural Networks

The paper entitled "Spectral Representations for Convolutional Neural Networks" presents significant advancements in convolutional neural network (CNN) architecture through the innovative use of spectral representations. This work explores the frequency domain as more than a tool for computational efficiency, proposing it as a powerful framework for both representing and training CNNs. Specifically, the paper introduces two novel concepts: spectral pooling and spectral parametrization of CNN filters, each demonstrating distinct computational and informational benefits.

The cornerstone of this research is the utilization of the Discrete Fourier Transform (DFT) to transition traditional CNN operations to the spectral domain. The paper leverages the operator duality inherent in the DFT—where convolution in the spatial domain equates to element-wise multiplication in the frequency domain—to achieve significant speed-ups in convolutional operations. Alongside computational efficiency, the frequency domain representation inherently aligns with typical filter structures, offering reduced redundancy and improved optimization trajectories.

Spectral Parametrization

The approach of spectral parametrization involves learning CNN filters directly in the frequency domain. This reparametrization maintains model equivalence in the spatial domain but yields numerous optimization benefits. The sparsity of filters in the spectral domain reduces redundant dimensions, guiding the optimization process through more meaningful paths and significantly accelerating convergence. The empirical results indicate a convergence speedup of approximately 2 to 5 times when compared to the spatial domain parameterization. This suggests that frequency domain representations can more effectively capture the salient structure of CNN filters, leading to better utilization of standard stochastic optimization techniques.

Spectral Pooling

Spectral pooling, as introduced by this paper, redefines the pooling operation by projecting input representations onto the frequency basis and truncating the frequency map for dimensionality reduction. This method addresses critical shortcomings of traditional stride-based pooling methods, such as max pooling, which often suffer from poor information retention due to their aggressive dimensionality reduction. In contrast, spectral pooling preserves more information per parameter by exploiting the frequency domain's characteristic power distribution. This approach also allows for arbitrary output map dimensionality, enabling smoother reductions across network depth and permitting innovative regularization strategies such as randomized resolution.

Experimental Results and Implications

Empirical evaluations underscore the efficacy of these spectral methods. The spectral pooling approach consistently outperforms standard pooling approaches in terms of information preservation, as demonstrated by lower reconstruction error rates for equivalent dimensionality reductions. The network architectures employing spectral pooling achieved competitive classification rates on benchmark datasets like CIFAR-10 and CIFAR-100, rivaling state-of-the-art methods even without data augmentation and dropout.

The implications of these findings are both practical and theoretical. Spectral representations open new avenues for more efficient network training and can potentially extend to entire architectures being embedded in the frequency domain. This approach could negate the need for repeated transformations between spatial and frequency domains, which are computationally expensive when nonlinearities are applied in the spatial domain.

Future work might explore embedding the entire network architecture in the frequency domain, leveraging wavelets for a balanced representation between spatial and spectral locality, or developing sensible nonlinearities for the frequency domain to minimize computational costs.

In conclusion, the exploration of spectral representations for CNNs introduces substantial advancements in the design and training efficiency of neural networks. The adoption of spectral pooling and spectral filter parametrization offers compelling computational and representational advantages, marking a pivotal contribution to the ongoing development of deep learning methodologies.