Unsupervised Learning of Video Representations using LSTMs (1502.04681v3)

Abstract: We use multilayer Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We try to visualize and interpret the learned features. We stress test the model by running it on longer time scales and on out-of-domain data. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only a few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.

Unsupervised Learning of Video Representations using LSTMs

This paper presents a method for learning representations of video sequences through unsupervised learning using Long Short Term Memory (LSTM) networks. The authors propose the utilization of multilayer LSTM networks to encode input video sequences into fixed-length representations and subsequently decode these representations for various tasks such as sequence reconstruction and future frame prediction.

Model Architecture and Learning Task

The system employs an encoder-decoder framework facilitated by LSTMs. Specifically, an encoder LSTM processes input video frames and maps them into a fixed-length representation. This representation is then decoded by one or more decoder LSTMs for the following tasks:

1. Input Sequence Reconstruction: Recapitulating the input video sequence.
2. Future Sequence Prediction: Predicting future frames in the sequence.

The paper investigates the efficacy or ramifications of two types of input representations:

• Raw image patches.
• High-level features (termed "percepts") extracted using a pretrained convolutional neural network (CNN).

The authors further explore model variations including:

• Conditional versus unconditioned decoders (where the former conditions the output on previously generated frames).
• Using both image reconstruction and future prediction tasks simultaneously.

Experimental Setup

The evaluation involves both qualitative and quantitative measures:

• Qualitative assessments include visualizations to determine the model’s ability to extrapolate learned video representations.
• Quantitative assessments involve stress-testing the model on extended time scales and out-of-domain data.
• Testing also involves evaluating the learned representations on supervised learning tasks, such as human action recognition, specifically on the UCF-101 and HMDB-51 datasets.

Results

Qualitative Analysis

The models were first tested on a simplified dataset of moving MNIST digits. The results demonstrated that LSTM-based models could effectively disentangle and predict the motion of overlapping digits, even when inputs were extended beyond the model’s training period.

When extending this approach to natural image patches, the LSTM models were able to learn meaningful representations, though the resulting predictions were initially blurry. Increasing the model capacity (number of LSTM units) improved the sharpness of predictions.

Quantitative Analysis

Key findings from action recognition tasks reinforced the model's applicability:

• Pre-trained LSTM models on unsupervised video data improved classification accuracy on action recognition tasks, notably achieving notable improvements when labeled data was limited.
• Comparing different model variants, the Composite Model (combining reconstruction and future prediction tasks) with conditional future prediction provided the best performance both on predicting future sequences and on supervised classification tasks.

Implications and Future Directions

The paper underscores the importance of unsupervised learning paradigms in extracting structured and temporally coherent representations from video data. The proposed methods circumvent the need for extensive labeled datasets, thus presenting significant practical advantages.

Potential future extensions of this work could include:

• Applying the model convolutionally across video patches and integrating these representations at multiple levels of a CNN architecture.
• Expansion of the model to leverage attention mechanisms for dynamic focusing on relevant sequence sections, thereby enhancing performance on more complex video sequences.

Conclusion

This paper demonstrates that LSTM-based encoder-decoder architectures can effectively learn useful video representations through unsupervised learning. The success of these models in improving action recognition performance suggests promising directions for further applications and refinements of unsupervised video learning methodologies.