Omni-Dimensional Dynamic Convolution (2209.07947v1)

Abstract: Learning a single static convolutional kernel in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs). Instead, recent research in dynamic convolution shows that learning a linear combination of $n$ convolutional kernels weighted with their input-dependent attentions can significantly improve the accuracy of light-weight CNNs, while maintaining efficient inference. However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. Inspired by this, we present Omni-dimensional Dynamic Convolution (ODConv), a more generalized yet elegant dynamic convolution design, to advance this line of research. ODConv leverages a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer. As a drop-in replacement of regular convolutions, ODConv can be plugged into many CNN architectures. Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various prevailing CNN backbones including both light-weight and large ones, e.g., 3.77%~5.71%|1.86%~3.72% absolute top-1 improvements to MobivleNetV2|ResNet family on the ImageNet dataset. Intriguingly, thanks to its improved feature learning ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters. Furthermore, ODConv is also superior to other attention modules for modulating the output features or the convolutional weights.

• The paper introduces a multi-dimensional attention mechanism that modulates convolutional kernels across spatial, input, output, and kernel number dimensions.
• The paper demonstrates parameter efficiency by achieving significant top-1 accuracy improvements on ImageNet for architectures like MobileNetV2 and ResNet.
• The paper offers integration flexibility as ODConv serves as a drop-in replacement for regular convolution layers across various CNN architectures.

Omni-Dimensional Dynamic Convolution

This paper introduces Omni-Dimensional Dynamic Convolution (ODConv), a novel approach to enhance convolutional neural networks (CNNs) by leveraging a multi-dimensional attention mechanism. Traditional CNN architectures rely on static convolutional kernels. Recent advances like dynamic convolution propose the use of multiple kernels with attentions conditioned on input features to improve accuracy while maintaining efficiency. However, such methods have only utilized the convolutional kernel number dimension for dynamic properties. ODConv expands this concept across all dimensions of the kernel space — spatial size, input channels, output channels, and kernel number — to more comprehensively modulate the convolution operations.

Key Contributions

• Multi-dimensional Attention Mechanism: ODConv introduces attentions along all four dimensions of the kernel space. This rich, multi-faceted modulation allows ODConv to capture contextual cues that previous designs overlooked.
• Parameter Efficiency: The ability of ODConv to effectively use a single convolutional kernel to equal or surpass the performance of existing methods using multiple kernels demonstrates its parameter efficiency. This reduces the model's computational complexity substantially.
• Integration Flexibility: Serving as a drop-in replacement for regular convolutional layers, ODConv can be incorporated into various CNN architectures without necessitating structural modifications.

Experimental Validation

Experiments conducted on the ImageNet and MS-COCO datasets with several CNN architectures such as MobileNetV2 and ResNet illustrate significant accuracy improvements. Specifically, ODConv offers absolute top-1 accuracy boosts of 3.77% to 5.71% for MobileNetV2 and 1.86% to 3.72% for ResNet on ImageNet. These gains demonstrate its ability to achieve a beneficial trade-off between model size and accuracy, outperforming similar dynamic convolution and attention-based methodologies.

Implications and Future Work

Theoretically, ODConv enhances the feature extraction capabilities by distributing attention processing across multiple dimensions within the kernel space, addressing previous limitations in dynamic convolution's design. Practically, the diverse application scope of ODConv promises efficiency in model deployment, especially for modern applications requiring real-time performance or operating under resource constraints.

The promising results of ODConv suggest further exploration in several areas:

• Integrating ODConv into deeper and more complex architectures: Further deployment in architectures beyond ResNet101 would illuminate its scalability and adaptability capabilities.
• Exploring hyperparameter optimization: ODConv shows sensitivity to hyperparameters like reduction ratio and kernel number, indicating potential gains through careful tuning across varied architectures.
• Investigating attention sharing and activation functions: Experimentation may lead to new paradigms that refine attention mechanisms or identify optimal configurations tailored to specific tasks or datasets.

ODConv's demonstrated strengths in both theoretical conceptualization and experimental performance mark it as a noteworthy advancement in convolutional designs and highlight potential pathways for evolving intelligent systems towards enhanced feature learning capacities.