PG-Video-LLaVA: Pixel Grounding Large Video-Language Models

Abstract: Extending image-based Large Multimodal Models (LMMs) to videos is challenging due to the inherent complexity of video data. The recent approaches extending image-based LMMs to videos either lack the grounding capabilities (e.g., VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we propose PG-Video-LLaVA, the first LMM with pixel-level grounding capability, integrating audio cues by transcribing them into text to enrich video-context understanding. Our framework uses an off-the-shelf tracker and a novel grounding module, enabling it to spatially localize objects in videos following user instructions. We evaluate PG-Video-LLaVA using video-based generative and question-answering benchmarks and introduce new benchmarks specifically designed to measure prompt-based object grounding performance in videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in Video-ChatGPT, for video-based conversation benchmarking, ensuring reproducibility of results which is a concern with the proprietary nature of GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its advantages to the video domain, delivering promising gains on video-based conversation and grounding tasks. Project Page: https://github.com/mbzuai-oryx/Video-LLaVA

• The paper introduces a new method that integrates pixel-level grounding and audio cues to spatially localize objects in video content.
• It builds on LLaVA-1.5 with CLIP-based spatio-temporal encodings and a learnable MLP to align video and language representations.
• Performance benchmarks demonstrate improved video conversation and grounding tasks, outperforming previous models on tested datasets.

PG-Video-LLaVA: A Novel Approach to Video-LLMs with Pixel-Level Grounding

The paper "PG-Video-LLaVA: Pixel Grounding Large Video-LLMs" introduces a new methodology, PG-Video-LLaVA, expanding the capabilities of image-based Large Multimodal Models (LMMs) to video data. This innovation focuses on integrating pixel-level grounding abilities into video-LLMs, alongside incorporating audio cues to enhance overall video comprehension. Unlike existing models such as VideoChat, Video-LLaMA, or Video-ChatGPT, which either lack grounding capabilities or do not fully exploit audio signals, PG-Video-LLaVA stands out by spatially localizing objects in videos as per user instructions.

The proposed model builds on the foundation of the LLaVA-1.5 framework, complemented by spatio-temporal encodings derived from a CLIP-based visual encoder specifically adapted for video interpretation. This adaptation facilitates a thorough understanding of video content by averaging frame-level features over temporal and spatial dimensions and aligning these features with a LLM through a learnable Multi-Layer Perceptron (MLP). These enhancements offer the model notable gains in performance on video-based conversation and grounding tasks.

An important aspect of PG-Video-LLaVA is its modular design, enabling seamless integration with future iterations of visual grounding technologies. The effective incorporation of audio context is achieved by transcribing audio into text format using advanced transcription techniques akin to Whisper. This multimodal approach ensures the model can process and leverage auditory information, particularly useful in scenarios involving rich audio content such as dialogues or video conferences.

In terms of evaluation, the authors adopt a comprehensive benchmarking approach, focusing on video-based generative and question-answering performances. Notably, they introduce new benchmarks, specifically targeting the evaluation of prompt-based object grounding in videos. The adoption of Vicuna over proprietary models like GPT-3.5 ensures reproducibility and transparency, crucial for ongoing research integrity.

The experimental results highlight PG-Video-LLaVA's strong performance in both qualitative and quantitative assessments. On video conversation benchmarks, the model consistently outperforms earlier iterations, particularly excelling in contextual and temporal understanding, as demonstrated by improved scores in metrics like correctness, detail orientation, and consistency.

In addition to conversation capabilities, PG-Video-LLaVA demonstrates robust performance in spatial grounding tasks. Utilizing datasets such as Vid-STG and HC-STVG for benchmarking, the model shows superior pixel-level grounding abilities. This is crucial as it aligns video elements more closely with semantic descriptions, enabling the model to generate more accurate and contextually relevant outputs, which is particularly evident when tested in zero-shot settings on established QA datasets like MSRVTT-QA and MSVD-QA.

The implications of PG-Video-LLaVA are multifaceted. Practically, the incorporation of pixel-level grounding and audio inputs can greatly enhance interactive and intelligent video applications, from advanced virtual assistants to automated video editing tools. Theoretically, this advancement sets a precedent for future developments in video-LLMs, encouraging the exploration of even richer multimodal datasets and grounding techniques.

Overall, PG-Video-LLaVA represents a sophisticated advancement in the field of LMMs for videos, striking a balance between innovative feature integration and practical applicability. Its techniques and results not only push the envelope in video language processing but also pave the way for subsequent research to build on a foundation that emphasizes comprehensive, multimodal understanding of complex video data.