Learning strides in convolutional neural networks

Abstract: Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance while reducing the computational complexity of the whole architecture. A critical hyperparameter of such layers is their stride: the integer factor of downsampling. As strides are not differentiable, finding the best configuration either requires cross-validation or discrete optimization (e.g. architecture search), which rapidly become prohibitive as the search space grows exponentially with the number of downsampling layers. Hence, exploring this search space by gradient descent would allow finding better configurations at a lower computational cost. This work introduces DiffStride, the first downsampling layer with learnable strides. Our layer learns the size of a cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way. Experiments on audio and image classification show the generality and effectiveness of our solution: we use DiffStride as a drop-in replacement to standard downsampling layers and outperform them. In particular, we show that introducing our layer into a ResNet-18 architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride configurations. Moreover, formulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture. We show how this regularization allows trading off accuracy for efficiency on ImageNet.

• The paper introduces DiffStride, a learnable downsampling layer that optimizes stride configurations via backpropagation and DFT-based spectral pooling.
• It adapts masking functions from transformers to dynamically learn optimal cropping regions, reducing the need for manual tuning of hyperparameters.
• Empirical results on datasets like CIFAR and ImageNet demonstrate that DiffStride consistently outperforms traditional strided convolutions in both audio and image classification tasks.

An Examination of DiffStride: A Learnable Downsampling Layer for Convolutional Neural Networks

The paper "Learning strides in convolutional neural networks" introduces DiffStride, a novel downsampling layer designed to enhance convolutional neural networks (CNNs) by learning the optimal strides through backpropagation. This approach seeks to improve upon traditional methods that require either manually fixed or cross-validated striding configurations, both of which present significant computational barriers as the network depth increases.

Overview of Methodology

DiffStride draws inspiration from spectral pooling to offer a differentiable alternative to traditional downsampling mechanisms. It utilizes the Discrete Fourier Transform (DFT) to downsample input representations, which allows it to crop inputs in the frequency domain rather than spatially. The key innovation lies in its ability to learn the cropping region's size, effectively making strides differentiable. A key component of DiffStride's operation is the adaptation of masking functions initially utilized in adaptive attention spans for transformers, which are reformulated here to dictate the cropping size in both the horizontal and vertical dimensions.

Experiments and Results

The efficacy of DiffStride is evaluated across eight audio and image classification tasks, incorporating datasets such as CIFAR10, CIFAR100, and ImageNet. The empirical results demonstrate that DiffStride consistently outperforms traditional strided convolutions and spectral pooling methods, particularly when the initial stride configurations are suboptimal. This showcases DiffStride's ability to reach or exceed baseline performance through learned adjustments.

1. Audio Classification: DiffStride improved classification accuracy across five audio tasks when compared to both traditional strided convolutions and spectral pooling. Notably, the learned strides aligned with known auditory processing features, providing interpretability to the results.
2. Image Classification: In experiments with CIFAR and ImageNet datasets, DiffStride was shown to be robust against varied initial stride configurations. It was able to maintain high-performance levels without necessitating exhaustive cross-validation of stride parameters.

Theoretical and Practical Implications

The capability of DiffStride to learn strides dynamically has implications for both the theoretical understanding and practical applications of CNNs. It shifts the paradigm where stride configurations must be pre-determined and potentially enhances the adaptability of CNN architectures across varied tasks. The diffusion of learnable parameters into aspects of model architecture traditionally viewed as static hyperparameters may drive future research and development in neural architecture optimization.

Moreover, the paper introduces a regularization framework within DiffStride that aims to balance computational complexity with performance, further underscoring the strategy's flexibility. This regularization is crucial for efficiently deploying CNNs in resource-constrained environments, such as mobile or embedded systems.

Future Directions

While DiffStride presents a significant step towards adaptive CNN architectures, there remain opportunities for future work. Extending DiffStride to handle 1D or 3D CNNs could open new avenues for its application in contexts such as time-series analysis and video processing. Additionally, exploring the integration of DiffStride with other learnable architectural components could lead to a more holistic approach to CNN design.

As neural network architectures become increasingly intricate, the ability to learn more about the model's structure dynamically will likely be a critical focus area. DiffStride represents one of the initial innovations in this direction, suggesting a potential shift toward more adaptable and efficient neural network architectures.