Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling (2210.03941v1)

Abstract: While recent large-scale video-language pre-training made great progress in video question answering, the design of spatial modeling of video-LLMs is less fine-grained than that of image-LLMs; existing practices of temporal modeling also suffer from weak and noisy alignment between modalities. To learn fine-grained visual understanding, we decouple spatial-temporal modeling and propose a hybrid pipeline, Decoupled Spatial-Temporal Encoders, integrating an image- and a video-language encoder. The former encodes spatial semantics from larger but sparsely sampled frames independently of time, while the latter models temporal dynamics at lower spatial but higher temporal resolution. To help the video-LLM learn temporal relations for video QA, we propose a novel pre-training objective, Temporal Referring Modeling, which requires the model to identify temporal positions of events in video sequences. Extensive experiments demonstrate that our model outperforms previous work pre-trained on orders of magnitude larger datasets.

• The paper introduces a novel decoupled modeling approach that separately leverages image-language and video-language encoders for enhanced visual understanding.
• The methodology integrates Temporal Referring Modeling to identify event positions, effectively reinforcing temporal relations with sparse frame sampling.
• Experiments on ActivityNet-QA and AGQA 2.0 benchmarks show superior performance over conventional models despite significantly lower pre-training data.

The paper "Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling" addresses the limitations of existing video-LLMs in video question answering (VQA) by proposing a novel decoupled spatial-temporal modeling approach. This approach enhances fine-grained visual understanding by separating spatial and temporal modeling, significantly improving upon the traditional methods that suffer from weak and noisy temporal alignment and less sophisticated spatial modeling compared to image-LLMs.

Methodology:

1. Decoupled Spatial-Temporal Encoders (DeST): The proposed framework integrates an image-language encoder and a video-language encoder.
   • The image-language encoder processes high-resolution spatial semantics from sparsely sampled video frames independently of temporal information.
   • The video-language encoder captures temporal dynamics at a lower spatial but higher temporal resolution.
2. Temporal Referring Modeling (TRM): A novel pre-training objective is introduced wherein models learn to identify the temporal positions of events within video sequences. This involves querying both absolute and relative temporal positions, aimed at reinforcing the understanding of temporal relations in video QA.
3. Data Utilization: The approach leverages sparsely sampled frames to enhance spatial understanding and synthesized video concatenations for modeling temporal transitions and relations.

Results:

• Performance: The DeST model outperforms existing state-of-the-art VQA models, even those pre-trained on substantially larger datasets. This superiority is validated through extensive experiments that show significant improvements in both spatial and temporal question types.
• Benchmark Evaluations: The model is tested on ActivityNet-QA and AGQA 2.0 benchmarks, showing that it not only excels in questions demanding spatial understanding but also markedly improves performance in questions requiring a nuanced grasp of temporal sequences and events.
• Ablation Studies: These studies demonstrate the complementary advantages of the image-language and video-language encoders. When each is tested independently, there is a notable performance drop, reinforcing the necessity of both encoders working in tandem for effective video QA.

Contributions:

• Hybrid Pipeline: By decoupling the spatial and temporal aspects of video processing into distinct streams, the paper mitigates the inefficiencies of conventional video-LLMs and makes full use of the strengths of existing image-language technology.
• Pre-Training Strategy: The development of TRM as a pre-training objective for video-language encoders shows the potential to learn effective temporal relations with limited data.
• Efficient Learning: The model demonstrates efficiency in learning robust video-language representations utilizing significantly less pre-training data, setting an example for data-efficient training paradigms in video QA.

In summary, the paper presents an innovative and efficient framework for improving the understanding of complex visual content in video QA, offering a solution that leverages the complementary strengths of both image and video-LLMs to achieve fine-grained visual representations and superior performance on benchmark tasks.