A guide to convolution arithmetic for deep learning

Abstract: We introduce a guide to help deep learning practitioners understand and manipulate convolutional neural network architectures. The guide clarifies the relationship between various properties (input shape, kernel shape, zero padding, strides and output shape) of convolutional, pooling and transposed convolutional layers, as well as the relationship between convolutional and transposed convolutional layers. Relationships are derived for various cases, and are illustrated in order to make them intuitive.

• The paper clarifies convolution arithmetic by detailing formulas linking input size, kernel, stride, and padding.
• It provides visual examples and reverse-engineering of transposed convolutions to compute output dimensions accurately.
• The guide supports precise CNN design, benefiting applications in image recognition, autoencoding, and generative modeling.

A Guide to Convolution Arithmetic for Deep Learning

Vincent Dumoulin and Francesco Visin provide a valuable tutorial on convolution arithmetic, specifically aimed at deep learning practitioners who utilize Convolutional Neural Networks (CNNs). Given the non-trivial nature of designing CNN architectures, this paper elucidates the complex relationships that govern convolutional, pooling, and transposed convolutional layers.

Introduction and Relevance

CNNs have revolutionized various machine learning tasks, particularly in fields such as image and speech recognition. Unlike fully connected layers, the functionality and design of convolutional layers are influenced by several parameters including input shape, kernel size, zero padding, and stride. Importantly, the paper also explores transposed convolutions, which are critical for architectures like autoencoders and generative models.

Discrete Convolutions

The paper begins with a thorough examination of discrete convolutions, highlighting their necessity for leveraging the inherent structure in data such as images and audio signals. Convolutions are sparse and parameter-efficient, attributes pivotal for processing high-dimensional data. Utilizing visual examples, the authors dissect how the kernel interacts with the input feature map, illustrating the resulting output feature maps.

They define key properties influencing output size:

• Input size ($i_j$)
• Kernel size ($k_j$)
• Stride ($s_j$)
• Zero padding ($p_j$)

The relationships between these properties are methodically described, providing formulas to calculate the output size accurately under various combinations of these parameters.

Pooling Layers

Pooling layers serve to reduce feature map dimensions by summarizing the input regions, commonly through max or average pooling. The output size of pooling layers is dictated by the same properties as convolutional layers, minus zero padding. The paper extends previously discussed relationships to these pooling operations, offering clarity on their arithmetic.

Transposed Convolutions

Transposed convolutions, or fractionally strided convolutions, facilitate the transformation from a lower to higher-dimensional space, maintaining a connectivity pattern analogous to the original convolution. These layers are essential for tasks including image generation and feature map upsampling.

For clarity, the authors reverse-engineer transposed convolutions by linking them to equivalent direct convolutions, thereby presenting relationships for calculating output sizes. They discuss:

• No zero padding, unit strides transposed convolutions
• Zero padding, unit strides transposed convolutions
• Non-unit strides transposed convolutions

This structured approach demystifies the seemingly complex arithmetic involved, making it more accessible for those designing or employing such layers in their models.

Practical Implications

This work simplifies the initial learning curve for practitioners new to CNNs and supports experienced researchers by providing a precise reference for convolution arithmetic. By enabling precise control over architecture design, the principles discussed can significantly aid in optimizing models for specific tasks and constraints.

Theoretical Significance and Future Directions

From a theoretical standpoint, understanding the explicit arithmetic behind convolutional operations supports the development of more efficient algorithms and architectures. The relationships and principles elucidated have broad applications across various neural network frameworks, such as TensorFlow and PyTorch.

In terms of future developments, the exploration of even more intricate convolution types, like dilated convolutions discussed briefly in the paper, could offer new avenues for innovation, especially in handling data with varying levels of granularity.

Conclusion

Dumoulin and Visin's paper meticulously articulates the arithmetic of convolutions within CNNs, equipping machine learning practitioners with the knowledge to design and deploy effective neural network architectures. The methodological rigor and clarity of this guide make it an indispensable resource for both novice and seasoned researchers aiming to leverage the full potential of CNNs in their projects.