--- title: 'LongVLM: Hierarchical Video Language Model' url: https://www.emergentmind.com/topics/longvlm type: topic --- # LongVLM: Hierarchical Video Language Model LongVLM is a Video-based Large Language Model (VideoLLM) architecture designed for efficient understanding of long videos, which typically consist of thousands of frames and thus tens of thousands of visual tokens. Unlike previous VideoLLMs that aggregate visual features into a single global representation, LongVLM decomposes videos into sequential short-term segments, applies hierarchical token merging to retain fine-grained local information, and integrates global semantic cues to enhance context comprehension. This approach balances computational efficiency with detailed temporal and spatial analysis, enabling significant gains over prior methods in both quantitative and qualitative evaluation benchmarks [2404.03384]. ## 1. Motivation and Problem Formulation Long video understanding poses challenges due to the high token count arising from many frames. For example, sampling 100 frames with 256 patch tokens per frame yields 25,600 tokens for a CLIP-ViT-L/14 backbone, exceeding the practical input constraints of off-the-shelf LLMs. Most existing VideoLLMs address this by compressing all frame features into a single vector, typically via pooling or learned queries, which captures overall context but fails to model fine-grained, temporally localized events. Further, there is often inadequate modeling of sequential sub-events and insufficient propagation of global semantics into localized representations. The formal goal is, given a video $I\in\mathbb{R}^{T\times H\times W\times 3}$ and user query $q$, to generate a comprehensive answer $a$ such that \[ a = \mathrm{LLM}\left(\Phi_V (\mathrm{VideoRep}(I)),\, q\right), \] where $\mathrm{VideoRep}(I)$ must preserve temporal order, retain both local and global information, and limit the total number of tokens [2404.03384]. ## 2. Model Architecture LongVLM’s pipeline comprises several distinct steps: 1. **Frame-level Feature Extraction:** Uniformly sample $T$ video frames $\{x^t\}_{t=1}^T$, extract per-frame patch features $v^t \in \mathbb{R}^{N \times d}$ from a frozen vision encoder (e.g., CLIP-ViT-L/14), as well as $E$ [CLS] tokens $c^t_e$ from the last $E$ layers. 2. **Segmentation and Hierarchical Token Merging:** Divide frames into $S$ contiguous segments of length $K=T/S$. Within each segment $s$, collect all patch tokens to form $V^s$, then reduce $V^s$ to $M$ tokens using a hierarchical bipartite matching mechanism inspired by ToMe. 3. **Global Semantic Integration:** Average [CLS] tokens temporally for each selected layer to obtain global feature vectors $\bar c_e$, then stack them as $F_{\text{global}}$ and concatenate them with the temporally ordered local segment tokens $F_{\text{local}}$. This ensures every local token can attend to the global context through self-attention. 4. **Projection and Generation:** Project the concatenated video token sequence $F_{\text{vid}} = [F_{\text{global}}; F_{\text{local}}]$ using a learned linear mapping $\Phi_V$ to the LLM’s input space. The frozen LLM (Vicuna-7B v1.1) receives both the visually encoded tokens and the text query, autoregressively generating the answer [2404.03384]. ## 3. Mathematical Formulation Key notation and computational steps are as follows: - $T$: number of sampled frames; $N$: patch tokens per frame; $d$: channel dimension; $E$: [CLS] global feature layers. - Hierarchical merging reduces segment patch tokens from $K N$ to $M$ via: 1. Randomly split tokens into $\mathcal{A}_i$ and $\mathcal{B}_i$, 2. Compute multi-head cosine similarities across all pairs, 3. Pool and merge top-scoring pairs iteratively until $M$ remain. - Concatenated token sequence: $F_{\text{vid}} \in \mathbb{R}^{(E + M S) \times d}$. - Only the projection layer $\Phi_V$ is fine-tuned, optimizing the autoregressive token cross-entropy loss \[ \mathcal{L} = -\sum_{t=1}^L \log P(a_t \mid a_{