No Time to Waste: Squeeze Time into Channel for Mobile Video Understanding (2405.08344v1)

Abstract: Current architectures for video understanding mainly build upon 3D convolutional blocks or 2D convolutions with additional operations for temporal modeling. However, these methods all regard the temporal axis as a separate dimension of the video sequence, which requires large computation and memory budgets and thus limits their usage on mobile devices. In this paper, we propose to squeeze the time axis of a video sequence into the channel dimension and present a lightweight video recognition network, term as \textit{SqueezeTime}, for mobile video understanding. To enhance the temporal modeling capability of the proposed network, we design a Channel-Time Learning (CTL) Block to capture temporal dynamics of the sequence. This module has two complementary branches, in which one branch is for temporal importance learning and another branch with temporal position restoring capability is to enhance inter-temporal object modeling ability. The proposed SqueezeTime is much lightweight and fast with high accuracies for mobile video understanding. Extensive experiments on various video recognition and action detection benchmarks, i.e., Kinetics400, Kinetics600, HMDB51, AVA2.1 and THUMOS14, demonstrate the superiority of our model. For example, our SqueezeTime achieves $+1.2\%$ accuracy and $+80\%$ GPU throughput gain on Kinetics400 than prior methods. Codes are publicly available at https://github.com/xinghaochen/SqueezeTime and https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SqueezeTime.

• The paper presents an innovative CTL block that integrates temporal dynamics into channels, reducing computational load and memory usage for mobile video understanding.
• It achieves higher accuracy and faster processing speeds, with notable improvements on Kinetics400, Kinetics600, and HMDB51 benchmarks.
• The approach opens opportunities for mobile applications in security, autonomous driving, and streaming by enabling efficient, high-performance video analysis.

SqueezeTime: Efficient Video Understanding on Mobile Devices

Introduction

Handling video data efficiently on mobile devices is a challenging problem. Most traditional video processing models use 3D convolutional networks or add separate temporal processing operations to 2D Convolutional Networks (CNNs). While effective, these methods are computationally heavy and demand significant memory, making them impractical for mobile applications.

Enter SqueezeTime—a lightweight video recognition network designed for mobile video understanding. SqueezeTime introduces an innovative approach by squeezing the temporal axis of a video sequence into the channel dimension. This shift reduces the computational and memory load, making it suitable for edge devices.

SqueezeTime Overview

SqueezeTime employs a novel Channel-Time Learning (CTL) Block to ensure efficient temporal modeling within the squeezed architecture. The CTL Block has two branches:

1. Temporal Focus Convolution (TFC): Emphasizes the significance of different temporal channels.
2. Inter-temporal Object Interaction (IOI): Restores temporal positions and enhances object interaction modeling.

Key Contributions

1. Efficient Temporal Squeezing: By integrating the temporal dimension into channels, SqueezeTime minimizes memory and computational demands.
2. Innovative CTL Block: Designed to model temporal importance and interactions effectively, the CTL block boosts accuracy.
3. Superior Performance: SqueezeTime outperforms existing methods in multiple benchmarks, offering higher accuracy and faster processing speeds on both GPUs and CPUs.

Numerical Results and Benchmarks

Let’s dive into the numbers. Extensive experiments demonstrate SqueezeTime's prowess against state-of-the-art methods:

• Kinetics400 (K400) Dataset:
  • SqueezeTime achieves a 1.2% improvement in Top1 accuracy and an 80% increase in GPU throughput compared to leading methods.
• Kinetics600 (K600) Dataset:
  • SqueezeTime delivers a 76% Top1 accuracy, outperforming the nearest competitor by 0.5%.
• HMDB51 Dataset:
  • On HMDB51, SqueezeTime scores a 65.6% Top1 accuracy, ahead of multiple advanced models.
• Action Detection on AVA2.1:
  • Achieving a commendable 15.1% mAP, SqueezeTime processes video frames in only 3.4 ms.
• Temporal Action Localization on THUMOS14:
  • SqueezeTime leads with a 32.7 average mAP and completes its tasks 14% faster than the next best method.

Practical and Theoretical Implications

From an application standpoint, SqueezeTime's efficiency opens up new possibilities for mobile video analysis, be it in security, autonomous driving, or video streaming services. The reduced computational and memory footprint marks significant progress in deploying high-performance video models on edge devices.

Theoretically, this work challenges the conventional wisdom of treating time as a separate dimension and demonstrates the efficacy of compact temporal encoding. It paves the way for further exploration of hybrid models that blend spatial and temporal information seamlessly.

Future Developments

Looking forward, SqueezeTime's success may inspire:

• Further Optimization: Enhancing the temporal recovery and interaction mechanisms could yield even lighter and faster models.
• Broader Applications: Expanding the methodology to other time-series tasks, such as anomaly detection in sensor data or real-time video feedback systems.
• New Architectures: Combining the squeezing approach with emerging technologies like Vision Transformers could create hybrid models with unparalleled efficiency.

SqueezeTime has opened new doors for mobile video understanding, making impressive strides in balancing performance with efficiency. This innovative approach could signal a shift in how we process temporal data in constrained environments. We're excited to see where this leads next!