VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (2501.00599v3)

Abstract: Video LLMs (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. Besides, the lack of high-quality object-level video instruction data and a comprehensive benchmark further hinders their advancements. To tackle these challenges, we introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding, i.e., enabling perception and reasoning on any objects throughout the video. Specially, we thoroughly develop VideoRefer Suite across three essential aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent data engine to meticulously curate a large-scale, high-quality object-level video instruction dataset, termed VideoRefer-700K. Next, we present the VideoRefer model, which equips a versatile spatial-temporal object encoder to capture precise regional and sequential representations. Finally, we meticulously create a VideoRefer-Bench to comprehensively assess the spatial-temporal understanding capability of a Video LLM, evaluating it across various aspects. Extensive experiments and analyses demonstrate that our VideoRefer model not only achieves promising performance on video referring benchmarks but also facilitates general video understanding capabilities.

• The paper introduces the VideoRefer Suite, including the VideoRefer-700K dataset, a novel VideoRefer model, and the VideoRefer-Bench benchmark, to advance fine-grained spatial-temporal object understanding in Video LLMs.
• The VideoRefer model features a spatial-temporal object encoder that uses a Spatial Token Extractor and Temporal Token Merge Module to capture detailed regional and sequential object representations.
• Experimental results demonstrate that the VideoRefer model achieves superior performance on the VideoRefer-Bench, showing enhanced capabilities in generating object-focused descriptions and performing complex video reasoning tasks.

The paper "VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM" introduces an innovative framework designed to enhance the capabilities of Video LLMs (Video LLMs) for fine-grained spatial-temporal video understanding and object reasoning. The primary motivation for this work is the existing gap in Video LLMs, which typically excel at holistic scene comprehension but struggle to capture the intricacies of spatial-temporal details on an object level.

Key Contributions:

1. VideoRefer Suite:
   • The suite encompasses three pivotal components aimed at advancing video object understanding:
     • Dataset: The authors introduce VideoRefer-700K, a meticulously curated large-scale dataset containing object-level video instruction data. This dataset aims to facilitate precise regional and sequential representation learning.
     • Model: A novel model, termed the VideoRefer model, has been developed. It includes a spatial-temporal object encoder capable of capturing detailed regional and sequential representations. This is achieved using a Spatial Token Extractor for spatial encoding and a Temporal Token Merge Module for temporal information aggregation across video frames.
     • Benchmark: A comprehensive evaluation tool, VideoRefer-Bench, is proposed to assess the spatial-temporal comprehension capabilities of Video LLMs. This benchmark includes multiple metrics focusing on both description generation (VideoRefer-Bench$^{\texttt{D}$) and multiple-choice question answering (VideoRefer-Bench$^{\texttt{Q}$).
2. Data Engine:
   • A multi-agent data engine is developed to facilitate the creation of the VideoRefer-700K dataset. This engine involves several components:
     • Analyzer, Annotator, Segmentor, Reviewer, and Refiner. Each component plays a role in ensuring high-quality data generation with object-level instruction pairs, incorporating strategies like mask generation and textual correspondence verification.
3. Model Architecture:
   • The VideoRefer model leverages both single-frame and multi-frame input modes. The Spatial Token Extractor pools features within masks representing objects, while the Temporal Token Merge Module provides continuous object-level token representations to adapt to the temporal dynamics of videos.
4. Evaluation:
   • Extensive experiments demonstrate the model's superior performance on VideoRefer-Bench compared to contemporary methods, showing enhanced capabilities in tasks like fine-grained video object referring, complex video relationship analysis, and object retrieval. The model also exhibits advancements in general video understanding tasks, positioning itself as a robust tool for spatial-temporal video analysis.

Experimental Outcomes:

• In VideoRefer-Bench$^{\texttt{D}$, the model performed exceptionally well in Subject Correspondence (SC) and Appearance Description (AD) metrics, indicating its proficiency in generating accurate, object-focused narratives.
• On the VideoRefer-Bench$^{\texttt{Q}$, the model showed a marked improvement in relationship and reasoning-based understanding tasks, further asserting its robust capability in handling intricate video content.

Limitations:

Despite its strengths, the paper acknowledges a limitation regarding the inability of the model to handle grounding tasks effectively. Future work is suggested to integrate these capabilities, aiming to broaden the framework's applicability in real-world scenarios requiring precise object identification within dynamic contexts.

Overall, the VideoRefer Suite represents a significant step forward in enhancing the spatial-temporal comprehension abilities of Video LLMs, offering tools and methods that are likely to benefit a wide range of applications in video analysis and understanding.