Video Large Language Models (Vid-LLMs)

• Video Large Language Models (Vid-LLMs) are multimodal foundation models that integrate video, language, and audio to enable open-ended reasoning and complex video analysis.
• They leverage hybrid architectures, combining explicit video analysis and latent embeddings with adaptive token compression for precise temporal and spatial grounding.
• Efficient token compression and advanced temporal modeling techniques boost inference speeds while sustaining high performance on benchmarks like captioning, retrieval, and Q&A.

Video LLMs (Vid-LLMs) are multimodal foundation models that integrate video, language, and often additional modalities (e.g., audio) within a generative or discriminative LLM architecture. Vid-LLMs are distinguished by their open-ended reasoning capabilities, ability to process long temporal sequences, perform temporal and spatial grounding, and unify diverse video-based tasks including captioning, retrieval, recognition, and question answering. They rely on learned video–text representations, adaptive token compression, and hybrid visual–language pipelines to operate over vast and heterogeneous video datasets.

1. Taxonomy and Architectural Principles

Vid-LLMs can be categorized according to their modality integration and architectural design (Tang et al., 2023):

Category	Principle	Example Modules
Analyzer × LLM	Explicit video analysis via pretrained models; outputs are text or dense tags; LLM performs end tasks	Caption generator + LLM
Embedder × LLM	Video encoder produces latent video embedding, projected into the LLM input space	Vision Transformer + linear/cross-attention adapters
(Analyzer + Embedder) × LLM	Fusion of analytic (textual/semantic) and embedder (latent feature) outputs prior to or within LLM	Hybrid adapters and fusion blocks

Adapter designs vary: “connective adapters” align video features to the LLM token space (e.g., via MLP, cross-attention, or “Q-Former”), while “insertive adapters” insert visual representations into intermediate LLM layers for deeper temporal–spatial fusion. Temporal modeling, spatiotemporal reasoning, and multi-granularity summarization are typically realized via architectural extensions (e.g., Time Gating, hierarchical token merging) (Hu et al., 8 Oct 2024, Weng et al., 4 Apr 2024).

2. Video–Language Pretraining and Automatic Narration

A central innovation in Vid-LLMs is the use of pre-trained LLMs as automatic video narrators, producing dense, diverse, and temporally synchronized textual descriptions for every region of long videos. LaViLa exemplifies this paradigm (Zhao et al., 2022):

• Narrator Module: A visually conditioned LLM (e.g., GPT-2 with added cross-attention) auto-generates natural language captions for video segments, learning:

$$p_\text{narrator}(y'|x') = \prod_{l=1}^L p(s'_l|s'_{1..l-1}, x')$$

• Rephraser Module: Text-to-text LLMs (e.g., T5) diversify supervision by paraphrasing captions.
• Contrastive Dual-Encoder: Learns aligned video–text embeddings, optimized by an InfoNCE loss over all narrated, annotated, and paraphrased pairs.

This approach offers dense coverage, better temporal synchronization, and higher linguistic diversity, which directly enhance transferability and robustness on downstream benchmarks such as Epic-Kitchens-100 (+5.9% absolute gain in retrieval) and EGTEA (+10.1% accuracy).

3. Efficient Video Representation and Token Compression

The scale and redundancy of video data mandate efficient tokenization and compression strategies. Recent advances adopt both dynamic and parameter-free compression techniques:

• Frame- and Token-Level Adaptivity: Video Compression Commander (VidCom²) preserves essential content by quantifying frame and token uniqueness:

$$\sigma_t = \frac{\exp((u_t-\max_k u_k)/\tau)}{\sum_k \exp((u_k-\max_i u_i)/\tau) + \epsilon}$$

with token retention ratios $r_t$ adjusted to maximize informativeness under a global token budget (Liu et al., 20 May 2025).

• Attention-Debiased Pruning (AdaTP): Addresses global and local biases in attention-derived token importance. Cosine similarity between visual and text tokens guides selection, and spatial deduplication ensures diversity among retained tokens (Sun et al., 26 May 2025).
• Hierarchical and Streaming Encodings: Models like LongVLM and VideoStreaming process videos in short-term segments, applying hierarchical token merging and propagating condensed segment-wise memory to the LLM, further selecting only question-relevant memories (Weng et al., 4 Apr 2024, Qian et al., 25 May 2024).

These methods enable Vid-LLMs to process long videos with reduced inference cost, achieving up to 3.9× acceleration with 75% token reduction while maintaining over 99.5% of original performance (Ma et al., 28 Aug 2025).

4. Temporal, Grounding, and Multimodal Reasoning

Temporal modeling remains a bottleneck. Baselines frequently overlook temporal order or spatial–temporal dependencies, sometimes performing similarly to image LLMs (“single-frame bias”) (Hu et al., 8 Oct 2024, Xiao et al., 8 Aug 2024).

• Time Gating: The TG-Vid model employs gating on spatial attention, temporal attention, and MLP sub-modules. Each sub-module’s output is modulated as:

$$Y^l = \sigma(\text{Cat}(V^l, \hat{Y}^l) W) \odot \hat{Y}^l + V^l$$

with module-specific learnable gates, offering fine-grained temporal control (Hu et al., 8 Oct 2024).

• Video Grounding Benchmarks (LLM4VG): Systematic evaluation shows that direct VidLLMs struggle with precise localization of temporal moments, often underperforming random baselines. Augmenting LLMs with per-second caption/VQA descriptions and prompt-based confidence judgment improves temporal recall but reveals that fine-grained, time-aware supervision remains unsolved (Feng et al., 2023).

Integration of audio, as targeted by newer AV-LLMs, and explicit geometric cues for 3D reasoning are emerging priorities (Chen et al., 29 Sep 2025).

5. Evaluation Protocols and Application Domains

Unified benchmarks such as VLM-Eval now assess Vid-LLMs across captioning, QA, retrieval, and action recognition, using both n-gram-based and GPT-based metrics. The latter offer high correlation with human judgment for correctness, precision (no hallucinations), and coverage (Li et al., 2023).

Novel tasks include:

• Video Editing Understanding (VEU-Bench): 19 tasks spanning recognition, reasoning, and judging for editing elements (e.g., shot size, cut type). Current Vid-LLMs frequently perform below random in dynamic editing reasoning (Li et al., 24 Apr 2025).
• Multi-grained Video Reasoning: SurgVidLM’s StageFocus mechanism cascades holistic video summarization and fine-grained functional step analysis, integrating both low- and high-frequency temporal features via multi-frequency attention. Large-scale domain data (SVU-31K) and hierarchical supervision are decisive for procedural comprehension (Wang et al., 22 Jun 2025).
• 3D Scene Perception: Vid-LLM fuses geometry and language via cross-task adapters and metric depth heads, validating directly on 3D QA, dense captioning, and visual grounding tasks without explicit 3D inputs (Chen et al., 29 Sep 2025).

Applications now span creative video production (Qian et al., 8 Apr 2025), driver assist, clinical and surgical analysis, and video editing recommendation.

6. Limitations and Future Research Directions

Major limitations and open questions in Vid-LLMs include:

• Temporal Reasoning Deficiency: Empirical probes show that Vid-LLMs are not robust to adversarial temporal swaps or language-only perturbations, highlighting a reliance on contextual cues rather than true chronological reasoning (Xiao et al., 8 Aug 2024).
• Token Efficiency and Scaling: The computational requirements of LLM inference on long video remain challenging. Plug-and-play token pruning, hierarchical memory, and speculative decoding accelerate generation, but joint optimization of pruning and semantic preservation remains open (Ji et al., 22 Aug 2025, Ma et al., 28 Aug 2025).
• Grounding and Hallucination: Current architectures often fail in localization and generate unsupported responses when queries are outside video content; alignment for answerability and negative-sample training are being pursued (Yoon et al., 7 Jul 2025).
• Interpretability and Rationales: Most models lack frame- or segment-level rationales, and produce little transparent evidence tracking. Visual rationale generation and explicit reasoning chains are gaining attention as post-hoc or in-model modules.

Future work will likely focus on advanced architectural integration of temporal, visual, audio, and geometric cues, dataset curation for dense supervision, plug-and-play efficiency modules, and interpretable, trustable output generation for safety-critical contexts.

7. Representative Mathematical Formulations

Below is a summary table of canonical Vid-LLM formulations, as found in the referenced works:

Component / Task	Formula	Paper Reference
Auto-narration LM	$$p_\text{narrator}(y'|x') = \prod_{l=1}^L p(s'_l|s'_{1..l-1}, x')$$	(Zhao et al., 2022)
Dual-encoder contrastive loss	$L = \frac{1}{|B|}\sum_{(x,y)\in B}\big($InfoNCE$(v, u) +$InfoNCE$(u, v)\big)$	(Zhao et al., 2022)
Temporal token merging (LongVLM)	$$a^{(p_i, q_i)} = \frac{1}{C}\sum_{c=1}^C \cos(f_c^{(p_i)}, f_c^{(q_i)})$$	(Weng et al., 4 Apr 2024)
Frame uniqueness for adaptive compression	$$\sigma_t = \frac{\exp((u_t-\max_k u_k)/\tau)}{\sum_k \exp((u_k-\max_i u_i)/\tau)+\epsilon}$$	(Liu et al., 20 May 2025)
StageFocus multi-frequency fusion (SurgVidLM)	$$E(X_C, X_F) = \text{softmax}(X_C X_F^T / \sqrt{d}) X_C$$	(Wang et al., 22 Jun 2025)
Align-for-answerability scoring	$$s(v, x, y) = \begin{cases} 1, & \text{if } k(v, x) \cdot t(y) = 1 \ 0, & \text{otherwise} \end{cases}$$	(Yoon et al., 7 Jul 2025)

These mathematical underpinnings are central to the training, inference, and evaluation of modern Vid-LLMs.

---

Vid-LLMs continue to evolve rapidly, with advances in modality fusion, temporal modeling, token efficiency, benchmark task coverage, and explanation. They are foundational to the next generation of multi-granularity, open-domain video understanding systems.