---
title: Video Multimodal LLMs
url: https://www.emergentmind.com/topics/video-multimodal-large-language-models-vllms
type: topic
---

# Video Multimodal LLMs

Video Multimodal Large Language Models (VLLMs) are architectures that extend large language models (LLMs) to process and reason over video content, often incorporating complementary modalities such as audio, object tracks, and sometimes external sensor data. The integration of LLMs with video understanding represents a convergence between natural language processing, computer vision, and multimodal representation learning, addressing the highly structured spatiotemporal nature of video.

## 1. Foundational Architectures and Cross-Modal Fusion

VLLMs depart from image-LMs by addressing the temporal dimension in addition to spatial and multimodal (e.g., auditory) information. Early and representative VLLMs typically rely on established visual encoders—frozen or trained end-to-end—that extract per-frame features, which are then aligned or adapted to the LLM token embedding space. Prominent architectures fall broadly into:

- **Two-Stage/Plug-and-Play Adapters:** Models such as Video-LLaMA [2306.02858] and RED-VILLM [2404.11865] use frozen image/audio encoders (ViTs, CNNs, CLIP, ImageBind) and introduce lightweight temporal modules (e.g., Q-Formers, temporal adapters) that pool and aggregate temporal features from frames or audio segments. The outputs are linearly projected and provided as a soft prompt or prefix to a frozen LLM (e.g., LLaMA, Vicuna) for downstream text generation.

- **Joint End-to-End Models:** More tightly coupled systems like ResNetVLLM [2504.14432] train a backbone ResNet visual encoder from scratch, concatenated at the token level with textual prompts, and use a shared transformer with optional cross-attention between visual and text tokens. This enables end-to-end learning of visual and linguistic features for seamless cross-modal fusion.

- **Unified Representations:** Video-LLaVA [2311.10122] introduces "alignment before projection," unifying both image and video features into a shared text-feature space via pre-aligned encoders (e.g., LanguageBind), then projecting into the LLM embedding space, substantially improving performance and convergence in joint image/video training.

- **Spatio-Temporal Query Architectures:** Models such as SpaceVLLM [2503.13983] use interleaved spatio-temporal queries—specialized tokens distributed amid per-frame patch embeddings—plus a query-guided spatial decoder that allows explicit mapping between queries and coordinates/bounding boxes at each frame, enabling explicit spatio-temporal grounding.

## 2. Temporal Reasoning and Video Token Representation

Capturing temporal correlations is essential for video understanding. Several architectural innovations have been introduced:

- **Temporal Pooling and Query Mechanisms:** Video Q-Formers [2306.02858] cross-attend to per-frame features using a fixed set of learnable temporal queries, aggregating temporally distributed salient elements across the video. RED-VILLM [2404.11865] computes both spatial and temporal pooled representations, passing these through plug-and-play temporal modules to adapt off-the-shelf image LLMs to the video domain.

- **Transformer-Based Approaches:** Certain models implement full-sequence transformers over concatenated visual and textual tokens, using position embeddings to encode frame/time index, implicitly capturing temporal dependencies through self-attention [2504.14432, 2406.13763].

- **Memory-Based Streaming and Selection:** VideoStreaming [2405.16009] employs a memory-propagated streaming encoder, segmenting long videos into clips, encoding each with context from the previous, and maintaining a fixed-sized memory. At inference, an adaptive Gumbel-TopK selection identifies the question-relevant memory slices, which are concatenated with the user prompt for autoregressive answer generation.

- **Token Compression and Efficiency:** DyCoke [2411.15024] introduces a training-free dynamic compression pipeline that reduces temporal redundancy via windowed merging and prunes spatially redundant tokens in the decoder key-value caches, achieving substantial speedup and memory reduction without impacting accuracy.

## 3. Multimodal Integration: Audio, Gaze, and Sensor Streams

While initially focused on RGB frames, top VLLMs now handle additional modalities:

- **Audio-Visual Integration:** Audio-Visual LLM [2312.06720] and Video-LLaMA [2306.02858] both leverage modality-specific encoders (e.g., CLAP for audio; CLIP for frames) plus cross-modal projection and gating. These models support visual-only, audio-only, or joint AV inference by applying explicit gating tokens that activate the necessary encoders and mask irrelevant channels during both training and inference. Modality-augmented training regimens yield strong performance on both video QA and audio captioning tasks.

- **Sensor and Egocentric Inputs:** GazeLLM [2504.00221] incorporates synchronized egocentric video–gaze data, mimicking foveal vision via eye-tracker-guided cropping. Processing high-res crops near the gaze point and discarding periphery achieves competitive comprehension at one-tenth the pixel/memory budget, pointing to an efficient path for scaling VLLMs to long or high-resolution inputs.

- **Object Trajectories and Scene Graphs:** MVU [2403.16998] extracts object-centric modalities—object presence, spatial location, and motion trajectories—using vision tools like OWL-ViT and converts them into natural-language tokens for prompt-based fusion, thus providing the pretrained VLM with modular world knowledge for robust long-video reasoning.

## 4. Training Protocols, Datasets, and Instruction Tuning

Progress in VLLMs has been catalyzed by careful design of instruction-tuning corpora, cross-modal alignment strategies, and efficient transfer from vision or image LLMs:

- **Pretraining Regimes:** Most models pretrain their visual encoders and LLMs on separate large-scale datasets: e.g., WebVid-2M for video–text, CC595k for images, and multi-turn dialogue datasets from MiniGPT-4, LLaVA, or in-house instruction data [2306.02858, 2311.11865, 2401.05702].

- **Instruction Tuning:** Multi-stage protocols first pretrain on raw caption data, then fine-tune cross-modal adapters on high-quality dialogue or instruction pairs. RED-VILLM [2404.11865] demonstrates that fine-tuning <1% of parameters (\(\theta_t, \theta_z\)) on only ~100 K video-text instructions yields SOTA performance, exploiting the frozen alignment of the backbone image-LLM.

- **Temporal and Spatial Supervision:** For temporally and spatially localized tasks, datasets such as Charades-STA (temporal intervals), RefCOCO+/HCSTVG (object boxes), and the large synthetic Uni-STG [2503.13983], provide explicit span or bounding box supervision. Joint optimization over language and spatial losses enables direct temporal or spatio-temporal grounding.

- **Streaming and Dialogue:** LIVE (Learning-In-Video-Stream) [2406.11816] converts offline temporally-annotated datasets into streaming dialogue format for supervision. Training losses couple next-token LM objectives with explicit EOS prediction at non-response frames to support long-context streaming inference.

| Model              | Frozen Backbones      | Temporal Module        | Multimodal Capability    | Key Training Corpus                        |
|--------------------|----------------------|-----------------------|-------------------------|--------------------------------------------|
| Video-LLaMA        | ViT-G/14, ImageBind  | Q-Formers             | Audio-Visual            | WebVid-2M, MiniGPT-4 instruct              |
| RED-VILLM          | ViT/CLIP, LLaVA      | Plug-in Temporal MLP  | Visual                  | ActivityNet-QA instruct                    |
| ResNetVLLM         | ResNet (learned)     | Transformer           | Visual                  | Video-ChatGPT-100K                         |
| SpaceVLLM          | SigLIP, Qwen2        | Spatio-Temporal Query | Spatio-Temporal         | Unified Spatio-Temporal Grounding (Uni-STG)|
| Audio-Visual LLM   | CLIP, CLAP/HTSAT     | Linear + Gating       | Audio-Visual            | WavCaps, WebVid2M, VGGSound                |
| VideoStreaming     | CLIP+Phi2            | Streaming Encoder     | Visual                  | MovieNet-QA, NextQA                        |

## 5. Task Coverage, Capabilities, and Performance

VLLMs now reflect a broadening coverage of tasks beyond classical video QA:

- **Dense Video Captioning and Summarization:** VideoNarrator [2507.17050] generates temporally aligned, high-quality dense captions for segments, using only off-the-shelf VLLMs as modular generators, context enrichers, and verifiers, reducing hallucinations and supporting downstream summarization and retrieval applications.

- **Spatio-Temporal Grounding:** SpaceVLLM [2503.13983] achieves SOTA on benchmarks requiring both interval and spatial object localization (e.g., HCSTVG, Charades-STA), enabled by its spatio-temporal queries and query-guided spatial decoders.

- **Temporal Grounding and Activity Localization:** Two-stage schemes [2407.06157] using image-based LLMs for per-frame description followed by text-LM temporal reasoning often outperform monolithic video-LLMs even at similar parameter scale.

- **Audio-Visual Reasoning:** Audio-Visual LLM [2312.06720] and Video-LLaMA [2306.02858] exhibit strong performance on video QA and audio captioning, showing the importance of modality-augmented training and explicit modality selection.

- **Long-Video and Streaming Understanding:** VideoLLM-online [2406.11816] implements streaming dialogue over videos exceeding thousands of frames, with real-time inference and explicit temporal alignment via streaming EOS loss, outperforming prior clip-based systems on streaming benchmarks.

- **Hallucination Diagnosis and Decoding:** Compositional hallucinations—errors emerging from entangled spatiotemporal concepts—are systematically analyzed via OmniVCHall and mitigated by TriCD [2602.00559], which enhances robustness to both isolated and compositional hallucinations via reinforcement-learned, contrastive triple-pathway decoding.

## 6. Efficiency, Scalability, and Memory Optimization

Scalability remains a central challenge due to the quadratic complexity in token length for transformers:

- **Token Pruning and Compression:** DyCoke [2411.15024] fuses window-based temporal merging with dynamic KV cache reduction, yielding up to 1.5× latency improvement and 1.4× lower memory while retaining or improving task accuracy, demonstrating that effective pruning does not necessarily degrade model performance.

- **Streaming Memory Mechanisms:** Methods such as memory-propagated streaming encoders (VideoStreaming [2405.16009]) enable streaming understanding of arbitrarily long videos by propagating and distilling condensed memory representations, achieving efficient per-question inference without full re-encoding.

- **Gaze-Guided Cropping:** GazeLLM [2504.00221] achieves major token savings and quadratic computational reductions by cropping only the gaze-relevant foveal region, empirically matched in semantic coverage to full-frame processing on natural instruction tasks.

## 7. Limitations, Open Problems, and Future Directions

Several persistent limitations and future opportunities are noted:

- Challenges in handling long-duration videos remain, especially with limited context windows and the computational expense of dense frame sampling [2311.11865, 2306.02858]. Memory-efficient attention, sparse transformers, and hierarchical token selection are under active investigation [2405.16009, 2411.15024].
- Robust spatio-temporal grounding, motion tracking, and causal/event understanding in unconstrained video remain open problems, with performance lags on datasets requiring explanation or compositional reasoning [2503.13983, 2602.00559].
- Multimodal fusion at arbitrary granularity (object, region, trajectory), efficient audio-visual co-representation, and scaling to additional sensor modalities (infrastructure video, egocentric, action streams) are actively researched [2312.06720, 2403.16998].
- Mitigating hallucinations, especially compositional errors, is a major focus, with plug-and-play contrastive decoding and reinforcement learning-based calibration emerging as promising directions [2602.00559].
- Future work is expected to exploit unified backbone encoders, plug-in or memory-augmented temporal modules, and systematic self-supervised pretraining, facilitated by large-scale, diverse, and high-quality instruction and grounding datasets [2503.13983].

In summary, modern VLLMs integrate vision, audio, and text at scale through unified representational spaces, adapter-augmented cross-modal fusion, and instruction optimization, achieving strong performance on a growing spectrum of multimodal video understanding tasks while addressing efficiency, scalability, and new failure modes [2306.02858, 2404.11865, 2504.14432, 2503.13983, 2312.06720, 2411.15024, 2405.16009, 2403.16998, 2311.10122, 2507.17050].

Source: https://www.emergentmind.com/topics/video-multimodal-large-language-models-vllms