Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers (1906.02792v1)

Abstract: Video Captioning and Summarization have become very popular in the recent years due to advancements in Sequence Modelling, with the resurgence of Long-Short Term Memory networks (LSTMs) and introduction of Gated Recurrent Units (GRUs). Existing architectures extract spatio-temporal features using CNNs and utilize either GRUs or LSTMs to model dependencies with soft attention layers. These attention layers do help in attending to the most prominent features and improve upon the recurrent units, however, these models suffer from the inherent drawbacks of the recurrent units themselves. The introduction of the Transformer model has driven the Sequence Modelling field into a new direction. In this project, we implement a Transformer-based model for Video captioning, utilizing 3D CNN architectures like C3D and Two-stream I3D for video extraction. We also apply certain dimensionality reduction techniques so as to keep the overall size of the model within limits. We finally present our results on the MSVD and ActivityNet datasets for Single and Dense video captioning tasks respectively.

Self-Attention Based Video Summarization: An Analysis of Universal Transformers

This paper, "Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers," presents a novel application of Transformer architecture, specifically Universal Transformers, for video captioning tasks. Traditional video captioning methods have heavily relied on models such as LSTMs and GRUs to extract spatio-temporal features using convolutional neural networks (CNNs) followed by recurrent neural network (RNN) architectures with attention mechanisms. These models often face limitations due to their inherent sequential nature and difficulty in capturing long-term dependencies. The introduction of the Transformer model has been largely successful in text-based sequence modeling and is now being extended to video processing tasks.

Overview of the Model Architecture

The authors propose a Transformer-based approach that utilizes C3D and Two-stream I3D architectures for video feature extraction. These models employ 3D convolutions to preserve both spatial and temporal information, offering a comprehensive representation of video data. The Transformer-based model then processes these extracted features without involving recurrence, instead relying entirely on self-attention mechanisms, which allow for parallelization and potentially more efficient computation compared to RNN-based systems.

Universal Transformers, an extension of the Transformer model, incorporate weight sharing across layers and Adaptive Computation Time (ACT) to address issues with sequence-to-sequence tasks. However, whilst ACT was implemented, it was set aside after the realization that it caused premature halting, hindering the model's learning capacity.

Experimental Results and Evaluation

The proposed methodology was evaluated on two prominent datasets: MSVD for single video captioning, and ActivityNet for dense video captioning. The results demonstrated competitive performance with state-of-the-art methods, particularly noted in BLEU scores. Universal Transformers outperformed other models in generating more diverse and meaningful captions, showcasing their potential for tasks with varying complexity and input lengths.

• MSVD (Microsoft Video Description Dataset): For single-caption generation, the model showed appreciable performance at word and phrase levels, surpassing prior methods in BLEU-1 and BLEU-2 scores. This indicates a notable improvement in capturing the salient features of videos and accurately describing them in text form.
• ActivityNet: Here, the emphasis was on generating multi-sentence paragraphs describing long videos. The challenge lay in synthesizing a coherent sequence of sentences to form a complete summary. Despite the inherent complexity, the model achieved notable paragraph-wise BLEU scores, underscoring the benefits of a self-attention mechanism in understanding prolonged sequential data.

Key Findings and Contributions

1. Feature Extraction with Video Focus: This paper highlights the efficacy of using specialized 3D CNNs like C3D and I3D for capturing detailed spatio-temporal features, providing a rich input to the Transformer model.
2. Self-attention in Video Captioning: The extension of self-attention mechanisms to video data exemplifies a shift from traditional sequential models, demonstrating how such architectures can leverage parallel processing to enhance performance, especially in detecting essential features across video frames.
3. Universal Transformers for Generalization: Despite some operational challenges like early halting, Universal Transformers show promise in handling variable-length inputs and improving model adaptability.

Future Prospects and Research Directions

The research opens several avenues for further exploration. Fine-tuning the feature extraction networks and integrating adaptive computation mechanisms effectively could refine performance. Additionally, leveraging richer semantic information through pre-trained image attributes could allow models to capture nuanced video content. Continued investigation in these areas could significantly advance the efficiency and accuracy of video summarization tasks.

This insightful exploration into applying Universal Transformers to video captioning presents meaningful implications for the field, potentially influencing future research directions toward more adaptive, self-attentive models in diverse sequence processing applications.