Video Classification with Channel-Separated Convolutional Networks (1904.02811v4)

Abstract: Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M, Kinetics, and Something-Something, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.

• The paper proposes channel-separated networks that factorize 3D convolutions into pointwise and depthwise operations to enhance efficiency.
• It employs group convolutions to achieve 2-3 times computational savings while maintaining or improving classification accuracy.
• Evaluations on Sports1M, Kinetics, and Something-Something highlight CSN’s potential for scalable and effective video analysis.

Video Classification with Channel-Separated Convolutional Networks

The paper "Video Classification with Channel-Separated Convolutional Networks" presents a paper on the application of group convolutions to video classification tasks, proposing architectures that leverage three-dimensional (3D) channel-separated networks (CSN) to improve both computational efficiency and accuracy in processing video data.

Key Insights and Methodology

The research primarily explores group convolutions in 3D networks, investigating their potential to reduce the high computational costs associated with traditional video classification architectures, which often rely on 3D spatiotemporal convolutions. These traditional models operate with a complexity of $\mathcal{O}(CTHW)$, compared to the $\mathcal{O}(CHW)$ complexity of 2D CNNs, where $T$ represents the number of frames and $C$, $H$, $W$ denote channels, height, and width, respectively. By separating channel interactions from spatiotemporal interactions, the proposed CSN architecture aims to optimize these high-dimensional convolutions.

Channel-separated networks utilize group or depthwise convolutions to factorize 3D convolutions into pointwise and depthwise operations, preserving channel interactions while significantly reducing FLOPs and parameters. Two variations of the proposed networks, interaction-reduced (ir-CSN) and interaction-preserved (ip-CSN), were evaluated across several datasets, namely Sports1M, Kinetics, and Something-Something, demonstrating significant computational savings and competitive accuracy compared to state-of-the-art methods.

Numerical Results

The paper highlights several numerical results:

• In the empirical evaluation, CSN architectures achieved about 2-3 times more efficient computations compared to baseline models while preserving or improving classification accuracy.
• ip-CSN-152 showed substantial accuracy gains over models like I3D, R(2+1)D, and S3D-G, achieving state-of-the-art performance with noticeable computational efficiency.

Implications and Future Directions

The findings have important implications for both theoretical advancements in deep neural network architectures and practical video processing applications. The use of channel interactions as a form of regularization provides a fresh perspective on balancing model complexity and generalization, suggesting that deeper networks can benefit from regularization without compromising accuracy.

Future research may delve into the exploration of channel interaction mechanisms and their integration into more complex neural architectures or investigations into the potential of CSNs in other video-based tasks, such as video generation or enhancement. Extending the paper to include multimodal inputs like audio-visual representations could further enrich the landscape of video understanding.

The paper represents a meaningful step towards efficient and effective video classification, proving that strategic convolutional design can lead to substantial improvements in computational demands and output performance, thus opening avenues for more scalable and sustainable video processing solutions in AI.