VideoLLaMA2: Open-Source Video LLM

• VideoLLaMA2 is an open-source video large language model that integrates dedicated vision and audio branches with spatial-temporal convolution to enhance video analysis tasks.
• The model architecture employs frozen encoders and an STC connector to transform visual and audio inputs into tokens for processing by a LLaMA2-based decoder, enabling versatile video-language applications.
• Efficiency optimizations like PureKV and ST-SpAttn reduce memory usage and processing time while achieving competitive benchmarks in video QA, engagement prediction, and action recognition.

VideoLLaMA2 is a family of open-source Video LLMs (Video-LLMs) designed to advance spatial-temporal reasoning and audio-visual-language understanding for a broad range of video analysis tasks. Building on the LLaMA2 LLM architecture, VideoLLaMA2 incorporates dedicated vision and audio branches with a specialized connector for spatial-temporal convolution, enabling strong performance across video question answering, captioning, engagement prediction, complaint generation, surveillance action recognition, and other multimodal video tasks (Cheng et al., 2024). The model supports both video–text and audio–text paradigms and has catalyzed a number of efficiency and task-specific adaptations in subsequent research.

1. Architecture and Model Design

VideoLLaMA2 consists of two main branches: a vision–language branch and an audio–language branch, both interfacing with a LLM decoder (e.g., LLaMA2 or Mistral-Instruct) (Cheng et al., 2024).

• Vision Encoder: Uses CLIP ViT-L/14 (frozen), generating frame-level visual features from resized video frames (e.g., $336\times 336$). These are processed via a Spatial–Temporal Convolution (STC) connector, transforming stacked frame features into tokens representing spatial-temporal patches.
• Audio Encoder: Employs BEATs (frozen) to encode log-mel spectrograms from raw audio, then applies a linear projection to produce LLM-compatible tokens.
• STC (Spatial–Temporal Convolution) Connector: Transforms sequential visual features with a pipeline of RegStage blocks and 3D convolution for effective spatio-temporal aggregation before projecting to the LLM embedding space.

  $$\mathrm{STC}(X) = \mathrm{MLP}(\mathrm{RegStage}_2(\mathrm{Conv3D}(\mathrm{RegStage}_1(X))))$$

• Fusion and Decoding: Visual, audio, and/or text tokens are concatenated with instructional prompts and passed autoregressively through the LLM decoder, allowing flexible handling of pure visual, pure audio, or mixed audio-visual queries.

Variants including VideoLLaMA2.1-7B-AV have 7B parameters and support joint processing of multiple modalities (Sun et al., 4 Aug 2025).

2. Training Methodology

The training strategy for VideoLLaMA2 follows a staged paradigm with modality-specific pretraining and large-scale multi-task instruction fine-tuning (Cheng et al., 2024). Key aspects include:

• Video–Language Pretraining: Conducted on 12M+ video–text pairs aggregated from Panda-70M, WebVid-10M, and similar sources. The CLIP encoder and LLM are frozen; the STC connector is optimized on cross-entropy for captioning/question answering tasks.
• Video–Language Multi-task Fine-tuning: Utilizes ~570K samples spanning video captioning, action classification, VQA, and generic instruction following. Both the STC connector and LLM weights are trained; the CLIP encoder remains frozen.
• Audio–Language Training: Consists of pretraining on WavCaps, followed by multi-task fine-tuning on datasets such as AudioCaps, Clotho, and VGGSound, with separate fine-tuning on audio–text and audio–video tasks.
• Loss Function: Aggregates sub-task cross-entropy losses, typically with equal weighting at $λ=1$ for all tasks.

Optimizations such as freezing specific encoders and focusing on connector adaptation allow efficient multi-modal integration.

3. Task-Specific Adaptations and Benchmarks

Video Understanding and QA

VideoLLaMA2 demonstrates competitive or superior performance among open-source models on multiple-choice and open-ended video QA, as well as captioning. For example, accuracy on MV-Bench exceeds 54 with 16-frame inputs, and open-ended QA scores on MSVD can reach 71.7% (Cheng et al., 2024).

Short Video Engagement Prediction

In engagement continuation rate (ECR) prediction on the SnapUGC dataset, VideoLLaMA2 (with audio) achieved SROCC=0.691 and PLCC=0.701, outperforming models using only visual/text features. Its architecture for ECR prediction fuses frame-wise visual embeddings, full-length audio, and text metadata via cross-attention, with regression realized by a pooled MLP head (Sun et al., 4 Aug 2025).

Embodied Complaint Generation

For complaint description from user-uploaded videos (CoD-V), VideoLLaMA2-7B is combined with retrieval-augmented generation (RAG): averaged CLIP embeddings represent video and product aspect, which are then used to retrieve similar complaints from a large text-image corpus. Retrieved examples, along with an emotion label, are prepended to the LLaMA2 prompt and the model generates coherent complaint text. On the ComVID dataset, this setup surpasses BLIP-VQA-Base, Qwen2-VL-7b, and Gemma3-12b by 3–5 percentage points on ROUGE and BLEU (Das et al., 24 Sep 2025).

Surveillance Action Recognition

As a zero-shot action recognizer on the UCF-Crime dataset, VideoLLaMA2 attains 42.6% accuracy (vs 24.3% for CLIP), demonstrating strong LLM-based reasoning over visual content. Self-Reflective Sampling (Self-ReS) further enhances performance by pruning irrelevant visual segments according to model-internal attention scores, yielding up to 44.6% top-1 accuracy without any task-specific fine-tuning (Pereira et al., 2024).

Safety-Critical Driving Scenarios

In the SAVeD dataset for ADAS crash and near-miss event analysis, VideoLLaMA2 is used as a zero-shot baseline (no modifications, retraining, or explicit adaptation). It achieves a 67.8% macro score for crash videos and 27.0% for near-miss scenes (exact match/cosine), outperforming zero-shot InternVL2.5 HiCo R16. However, fine-tuning simpler models with frame-level SAVeD annotations closes and often surpasses the gap for subtle scenarios, demonstrating the importance of domain-adapted annotations (Zhai et al., 19 Dec 2025).

4. Efficiency: KV Cache and Attention Optimization

A major scalability challenge for Video-LLMs is the quadratic complexity in attention for long video sequences. PureKV is a plug-and-play framework enabling VideoLLaMA2 to efficiently process high-resolution, long-form video by:

• Cross-Layer Token Importance: Uses lower-layer attention scores, combined with value vector norms, to estimate and prune high-layer KV cache entries, enabling up to 80% memory reduction.
• Spatial-Temporal Sparse Attention (ST-SpAttn): Applies structured sparsity, letting each token attend to local and anchor tokens within and across frames, yielding near-linear scaling with input length.
• Empirical speedup: Achieves 3× prefill acceleration (from 0.1190 ms/token to 0.0376 ms/token) on MVBench, with only ∼6.7% relative drop in average ROUGE accuracy at a 20% cache budget (Jiang et al., 29 Oct 2025).

This approach preserves compatibility with FlashAttention and sparse attention backends, making real-time, high-resolution video understanding feasible.

5. Evaluation Results and Comparative Performance

VideoLLaMA2 is systematically benchmarked across domains. Key results:

Task/Domain	Metric	Frames	VideoLLaMA2 (7B)	Top Comparator
MC-VQA (MV-Bench)	Accuracy	16	54.6	LLaVA-NeXT-Video: 46.5
OE-VQA (MSVD)	Accuracy	8	71.7	LLaVA-NeXT-Video: 67.8
Engagement Prediction	FinalScore	8	0.695	Qwen2.5-VL: 0.664
Complaint Generation (CoD-V)	BLEU-1	4	0.69	Qwen2-VL-7b: 0.66
Surveillance Action (UCF-Crime)	Acc.	16	42.6%	CLIP: 24.3%
ADAS Crash Recognition	Macro (%)	—	67.8	InternVL2.5 HiCo R16: 55.97 (FT)

Benchmarks reveal that the inclusion of audio (either direct or via joint training) consistently improves performance on metrics sensitive to narrative and affective information, particularly for engagement prediction (Sun et al., 4 Aug 2025, Cheng et al., 2024). However, limitations remain for nuanced, highly domain-specific video phenomena, especially in zero-shot scenarios.

6. Limitations, Challenges, and Future Directions

VideoLLaMA2 currently operates on a limited number of frames (typically 8–16), constraining understanding of long-horizon sequences and context-dependent events (Cheng et al., 2024). Encoders are generally frozen, restricting the model’s capacity for low-level adaptation. Efficiency improvements from PureKV mitigate, but do not fully overcome, input-length quadratic scaling (Jiang et al., 29 Oct 2025).

Other observed limitations include:

• Occasional hallucination or omission of subtle details, especially in complaint generation and complex driving scenes (Das et al., 24 Sep 2025, Zhai et al., 19 Dec 2025).
• Incomplete temporal synchronization between audio and visual modalities in models employing global spectrogram representations (Sun et al., 4 Aug 2025).
• Semantic confusion among similar action classes in surveillance settings (Pereira et al., 2024).

Plausible future work involves expanding frame budgets, integrating hierarchical memory, incorporating dynamic audio–visual alignment, scaling backbone size, and broader domain adaptation to tasks such as robotics or high-stakes safety analysis (Cheng et al., 2024).

7. Impact and Research Landscape

VideoLLaMA2 is among the leading open-source Video-LLMs, providing a reproducible platform that closes the performance gap with proprietary models like Gemini 1.5 Pro and GPT-4V on several benchmarks. It is cited as a strong zero-shot baseline in downstream tasks (e.g., SAVeD, CoD-V, UCF-Crime), serving as a reference point for future adaptation and evaluation pipelines (Cheng et al., 2024, Zhai et al., 19 Dec 2025, Das et al., 24 Sep 2025, Pereira et al., 2024).

The model has driven the development of specialized efficiency mechanisms (e.g., PureKV), retrieval-augmented and emotion-grounded generation, and attention-guided frame pruning strategies, influencing both methodological research and real-world video understanding systems. VideoLLaMA2’s architecture and codebase underpin several top-performing entries in competitive academic challenges and are likely to remain foundational in video-LLM research.