AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (2410.03051v4)

Abstract: Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.

• The paper introduces a token merging strategy that reduces visual tokens to 10–20% without sacrificing performance.
• The proposed architecture integrates vision encoders with language models using an MLP projection, eliminating extra temporal parameters.
• Establishing the VDC benchmark and VDCscore metric, the study sets new standards for detailed video captioning evaluation.

Overview of "AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark"

This paper introduces AuroraCap, a novel approach to video detailed captioning, built upon a large multimodal model. The research addresses the inefficiencies associated with handling lengthy video sequences through a token merging strategy, demonstrating minimal performance loss while significantly enhancing processing efficiency. AuroraCap's architecture forgoes additional temporal modeling parameters, instead leveraging existing pre-trained components for seamless integration. This approach yields impressive results across established benchmarks, including a CIDEr score of 88.9 on Flickr30k, outperforming contemporary models such as GPT-4V and Gemini-1.5 Pro.

Technical Contributions

1. Token Merging Strategy: A central innovation in AuroraCap revolves around the selective merging of visual tokens using a bipartite soft matching algorithm within transformer layers. This technique significantly reduces token redundancy, enabling the model to maintain high resolution and performance with just 10-20% of original visual tokens.
2. Efficient Model Architecture: AuroraCap aligns visual features from vision encoders with LLMs through an MLP-based projection without additional parameters. This design aligns with prior successful strategies utilized in visual LLMs, including LLaVA and its enhancements.
3. Dataset and Benchmark Creation: Recognizing the limitations of existing video captioning datasets, the authors developed the Video Detailed Captions (VDC) benchmark. VDC comprises varied and richly annotated video-caption pairs extending beyond simple object descriptions to include intricate temporal and contextual details. A novel metric, VDCscore, was also introduced to evaluate detailed captions through decomposed question-answer pairs enhancing semantic understanding.
4. Training and Evaluation Framework: AuroraCap employs an adaptive three-stage training strategy—pretraining, vision, and language stages—optimizing both visual and linguistic modalities. The model is evaluated on various paradigms including image captioning and video question answering, showcasing competitive or superior results in challenging conditions.

Implications and Future Directions

The AuroraCap framework underscores a shift towards computationally efficient video understanding methods without sacrificing descriptive richness. Its token merging approach offers a pathway to scalable implementations, crucial for applications in resource-constrained environments or scenarios requiring rapid processing.

In terms of practical implications, AuroraCap could enhance content generation, automatic scene understanding, and real-time captioning in multimedia applications. The development of the VDC benchmark also sets the stage for more comprehensive evaluation of models tasked with capturing intricacies within video data.

Looking forward, there exists potential for extending AuroraCap's methodologies to other modalities or even real-time systems. The research community may also delve into optimizing token merging algorithms further, potentially incorporating dynamic or content-aware strategies to improve the adaptability and accuracy of detailed caption systems.

In conclusion, the paper presents AuroraCap as an efficient and effective advancement in video captioning technology, with potential widespread benefits for machine understanding of dynamic visual media. The introduction of a new benchmark and evaluation metric encourages increased richness and precision in video description capabilities, challenging future systems to rise to a higher standard of performance.