Tensor-Train Recurrent Neural Networks for Video Classification (1707.01786v1)

Abstract: The Recurrent Neural Networks and their variants have shown promising performances in sequence modeling tasks such as Natural Language Processing. These models, however, turn out to be impractical and difficult to train when exposed to very high-dimensional inputs due to the large input-to-hidden weight matrix. This may have prevented RNNs' large-scale application in tasks that involve very high input dimensions such as video modeling; current approaches reduce the input dimensions using various feature extractors. To address this challenge, we propose a new, more general and efficient approach by factorizing the input-to-hidden weight matrix using Tensor-Train decomposition which is trained simultaneously with the weights themselves. We test our model on classification tasks using multiple real-world video datasets and achieve competitive performances with state-of-the-art models, even though our model architecture is orders of magnitude less complex. We believe that the proposed approach provides a novel and fundamental building block for modeling high-dimensional sequential data with RNN architectures and opens up many possibilities to transfer the expressive and advanced architectures from other domains such as NLP to modeling high-dimensional sequential data.

• The paper introduces a novel Tensor-Train decomposition approach that drastically reduces RNN parameter counts for video classification.
• It integrates Tensor-Train layers into RNN architectures, eliminating the need for separate CNN pre-processing through end-to-end training.
• Empirical results on multiple video datasets demonstrate that TT-RNNs achieve competitive accuracy with reduced computational resources and faster training.

Overview of Tensor-Train Recurrent Neural Networks for Video Classification

The paper addresses a significant limitation in Recurrent Neural Networks (RNNs) and their variants such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), when applied to high-dimensional input data such as video sequences. Traditional RNN architectures encounter practical difficulties due to the massive size of the input-to-hidden weight matrices required to handle high-dimensional video inputs, rendering them computationally expensive and complex to train in large-scale applications. Current methodologies often mitigate this by pre-processing video frames using Convolutional Neural Networks (CNNs) to extract features, which are then passed to RNNs. However, such approaches disconnect the training of CNNs and RNNs, leading to suboptimal performance.

The authors introduce a novel approach by incorporating Tensor-Train decomposition, a form of tensor factorization, to the weight matrices of RNNs. This innovative use of the Tensor-Train format allows for a substantial reduction in the number of parameters, enabling RNN architectures to manage high-dimensional video inputs more efficiently. Specifically, Tensor-Train RNNs (TT-RNNs) leverage factorized representations, reducing the parameter space by several orders of magnitude compared to traditional fully-connected layers while maintaining competitive performance against state-of-the-art models in video classification tasks.

Contributions and Methodology

The paper proposes replacing the input-to-hidden mapping in RNNs with Tensor-Train Layers (TTL), facilitating an end-to-end training sequence without the need for CNN-based pre-processing. The Tensor-Train decomposition is used to factorize the weight matrix into multiple smaller cores, which interact to represent the original high-dimensional space more compactly. This approach directly addresses the weight matrix issue by providing a scalable and computationally efficient solution that retains the ability to capture complex temporal patterns in sequential data.

Experiments conducted on several real-world video datasets (e.g., UCF11, Hollywood2, and Youtube Celebrities Face Data) demonstrate that TT-RNNs, including TT-GRUs and TT-LSTMs, achieve comparable or superior performance to existing methods that rely heavily on pre-trained CNNs for feature extraction. Remarkably, TT-RNNs require fewer parameters and computation resources, highlighting their suitability for deployment in resource-constrained environments such as mobile devices.

Empirical Results

The empirical results underscore the efficacy of TT-RNNs, with performance metrics often rivalling those of substantially more complex models. Across multiple datasets, TT-GRUs and TT-LSTMs consistently outperformed their non-tensorized counterparts (plain GRUs and LSTMs), some of which performed poorly due to the high dimensionality of direct input sequences. Additionally, TT-RNNs demonstrated reduced training times and resource requirements, leveraging the compact Tensor-Train structure effectively during both training and inference.

Implications and Future Directions

The development of TT-RNNs marks a pivotal step towards integrating sophisticated RNN architectures directly with high-dimensional video inputs, broadening the applicability of RNNs to various domains beyond natural language processing. By eliminating the pre-processing bottleneck, TT-RNNs can potentially enable more unified and harmonious end-to-end architectures that capitalize on the strengths of RNNs in sequence modeling.

The implications are substantial for applications requiring robust video analysis, including video summarization, caption generation, and real-time action recognition. Future work may explore more complex RNN-based architectures, utilizing the TT-RNN framework to transfer methodologies successful in other domains to high-dimensional video data. Additional research may also focus on optimizing tensor factorization itself or investigating other tensor decomposition techniques to further enhance the flexibility and performance of RNNs in handling diverse data modalities.