DreamFrame: Enhancing Video Understanding via Automatically Generated QA and Style-Consistent Keyframes

Abstract: Recent large vision-LLMs (LVLMs) for video understanding are primarily fine-tuned with various videos scraped from online platforms. Existing datasets, such as ActivityNet, require considerable human labor for structuring and annotation before effectively utilized for tuning LVLMs. While current LVLMs are primarily trained on existing datasets in broad, general-purpose settings, adapting them to specific downstream scenarios remains challenging, as collecting and annotating task-specific videos is highly labor-intensive and time-consuming. To address this issue, we propose a three-stage framework named DreamFrame for automatically generating style-consistent keyframes and corresponding question-answer (QA) pairs to support LVLM instruction tuning. DreamFrame generates datasets in a movie-like manner. First, we utilize an LLM to generate structured movie plots including movie prior information (like overview and style), frame descriptions and plot-related QA pairs, with a story expansion strategy to mitigate context length limitations.Then, to ensure visual consistency across generated frames, we design a Style Immobilization Process which maintains consistent style through an embedding learning strategy. Finally, frame descriptions and style embeddings are integrated to produce coherent keyframes. Using DreamFrame, we construct a dataset comprising approximately 1k stylized keyframe-like videos and 100k diverse QA pairs. Extensive fine-tuned experiments on various LVLM architectures demonstrate the effectiveness of the proposed dataset. Furthermore, based on the proposed dataset, we fine-tune a new LVLM named DreamFrame-7B, which significantly surpasses the previous similar-sized LVLMs across different benchmarks.

• The paper introduces MovieLLM, a framework that uses AI-generated synthetic data to improve comprehension of long video formats by generating coherent movie plots and visuals.
• It employs a multi-stage pipeline using GPT-4 for plot generation, style immobilization via textual inversion, and keyframe production for robust data creation.
• Experimental results show that MovieLLM-enhanced models outperform baselines on zero-shot video QA tasks and long video comprehension benchmarks across 15 movie genres.

Overview of "MovieLLM: Enhancing Long Video Understanding with AI-Generated Movies"

This paper introduces MovieLLM, a novel framework designed to improve the understanding of long video formats, such as full-length movies, by leveraging synthetic data. The framework addresses the limitations of existing multimodal LLMs that struggle with long-duration content due to a scarcity of high-quality, diverse datasets. MovieLLM seeks to overcome these challenges by utilizing GPT-4 and text-to-image (T2I) models to generate detailed, consistent scripts and visuals, thereby creating a flexible and scalable data generation pipeline.

Methodology

The methodology of MovieLLM involves three main stages:

1. Movie Plot Generation:
   • Utilizes GPT-4 to generate diverse and coherent movie plots by defining specific elements such as themes, overviews, styles, characters, and key frame descriptions.
   • Implements a story expansion strategy to mitigate LLM forgetting issues, dividing the plot into epoch chapters, narrative threads, and frame descriptions.
2. Style Immobilization Process:
   • Employs textual inversion to convert style descriptions into stable diffusion model embeddings, guiding it to generate scenes with consistent styles.
3. Video Instruction Data Generation:
   • Generates key frames using style embeddings and celebrity-wise character representations along with enriching QA pairs derived from the movie plots.

Experimental Validation

The paper reports strong experimental results, asserting that MovieLLM significantly enhances the performance of multimodal models in understanding complex video narratives. The dataset generated using MovieLLM demonstrates robustness in addressing issues of scarcity and bias prevalent in existing data.

Results and Implications

MovieLLM's resulting dataset supports 15 different movie genres, highlighting its capability to foster diverse video understanding scenarios. In comparative evaluations, models fine-tuned with data from MovieLLM outperform baselines on zero-shot video QA tasks and long video comprehension benchmarks, suggesting improvements in temporal, plot, and overview understanding.

Future Directions

By automating the generation of long video datasets, this approach reduces dependence on costly manual annotation processes, thus broadening scalability for training robust multimodal models. The paper speculates that integrating advanced diffusion models for enhanced video data generation could further expand MovieLLM's applicability, facilitating future developments in video understanding via AI.

Conclusion

MovieLLM presents a significant advancement in the automatic generation of high-quality data for long video comprehension by synthesizing intricate sequences of visual and textual data. Its methodology not only reduces manual labor but also enables the creation of large, diverse datasets necessary for training sophisticated multimodal models, potentially reshaping approaches to AI-driven video understanding in the future.