--- title: 'ViLaMP: Hierarchical Video-Language Model' url: https://www.emergentmind.com/topics/vilamp type: topic --- # ViLaMP: Hierarchical Video-Language Model ViLaMP is a hierarchical video-language model developed to address the complexity and efficiency barriers of long-form video processing for vision-language understanding. Leveraging the principle of differential distillation, ViLaMP systematically assigns higher representational "precision" to video regions that are most relevant to a given textual query and least redundant within their temporal context. It combines query-driven keyframe selection with salient feature pooling to efficiently encode ultra-long videos (up to 10,000 frames) for downstream video-language tasks while optimizing computational and memory efficiency. ViLaMP has demonstrated state-of-the-art performance across multiple long-form video understanding benchmarks, supporting practical hour-scale inference on a single NVIDIA A100 GPU [2504.02438]. ## 1. Differential Distillation Principle ViLaMP operationalizes differential distillation, which prioritizes the retention of information in proportion to its utility for a downstream query $Q$ and its novelty within the temporal context. Given a video component $v$ (such as a frame or patch), the *differential saliency* score is: \[ D(v) = R(v, Q) - T(v, \mathcal{C}(v)) \] where $R(v, Q) \in [-1, 1]$ quantifies the query relevance (e.g., as cosine similarity between the component- and query-embeddings), and $T(v, \mathcal{C}(v)) \in [0,1]$ quantifies its redundancy with respect to a set of context features $\mathcal{C}(v)$. The global selection and compression objective under computational budget $B$ seeks to maximize the sum of differential information among selected keyframes $\mathcal{K}$ and merged non-keyframe features: \[ \max_{\mathcal{K},\,\{w\}} \Bigg[ \sum_{f_n \in \mathcal{K}} D_f(f_n) + \sum_{f_n \notin \mathcal{K}} \sum_{m=1}^M w_n^m D_p(p_n^m) \Bigg] \] with constraints $|\mathcal{K}| \le K, \sum_m w_n^m=1, w_n^m \ge 0$, and $\text{Cost}(\mathcal{K},\{w\}) \le B$. This principle enables ViLaMP to allocate more tokens to salient, query-critical content while aggressively compressing redundant video segments [2504.02438]. ## 2. Hierarchical Model Architecture ViLaMP employs a two-tier hierarchy to compress and encode long videos efficiently: - **Frame Level (Mixed Precision):** A subset of $K \ll N$ keyframes preserves full tokenized patch information ($M$ tokens per keyframe). The remaining $N-K$ non-keyframes are reduced to a single token each, reflecting mixed-precision treatment. - **Patch Level:** Each non-keyframe undergoes a learnable softmax pooling over its $M$ patch embeddings to retain those features that are both maximally relevant to the query and minimally redundant with the temporally neighboring keyframes. The processing pipeline is as follows: 1. Video $\rightarrow$ CLIP-based encodings (via SigLIP-so400m). 2. Differential Keyframe Selection (dks). 3. Differential Feature Merging (dfm) for non-keyframes. 4. Vision–Language Connector (multi-layer perceptrons). 5. LLM prompt construction and answer generation (using Qwen2-7B) [2504.02438]. ## 3. Differential Keyframe Selection Mechanism Given a sequence of frames $V = \{f_1, ..., f_N\}$ and a query $Q$, each frame embedding $\boldsymbol{f}_n$ and the query embedding $\boldsymbol{q}$ are obtained via a CLIP encoder: \[ \boldsymbol{f}_n = E_f(f_n),\quad \boldsymbol{q}=E_f(Q) \] \[ R_f(f_n, Q) = \cos(\boldsymbol{f}_n, \boldsymbol{q}) \] Frame redundancy relative to context $\mathcal{C}$ is defined as: \[ T_f(f_n, \mathcal{C}) = \max_{f \in \mathcal{C}} \cos(\boldsymbol{f}_n, E_f(f)) \] A greedy selection procedure sorts frames by $R_f$, iteratively admitting frames whose maximum similarity with already selected keyframes does not exceed a threshold $\tau$, until the quota $K$ is reached. This produces a set $\mathcal{K}$ of query-relevant, temporally distinctive keyframes. The computational complexity is $O(N \log N + NK)$ [2504.02438]. ## 4. Differential Feature Merging for Non-Keyframes For each non-keyframe $f_n \notin \mathcal{K}$, with $M$ spatial patches $\{p_n^m\}_{m=1}^M$ and nearest preceding keyframe $f_k$, per-patch relevance and redundancy are computed as: \[ R_p(p_n^m, Q) = \cos(\boldsymbol{p}_n^m, \boldsymbol{q}), \quad T_p(p_n^m, p_k^m) = \cos(\boldsymbol{p}_n^m, \boldsymbol{p}_k^m) \] \[ D_p(p_n^m) = R_p(p_n^m, Q) - \lambda T_p(p_n^m, p_k^m) \] Softmax pooling with sharpness $\alpha$ yields weights $w_n^m$: \[ w_n^m = \frac{\exp(D_p(p_n^m)/\alpha)}{\sum_{j=1}^M \exp(D_p(p_n^j)/\alpha)} \] Aggregated token $\boldsymbol{t}_n$ for $f_n$: \[ \boldsymbol{t}_n = \sum_{m=1}^M w_n^m\,\boldsymbol{p}_n^m \] This mechanism ensures retention of spatially and temporally novel features for each compressed non-keyframe [2504.02438]. ## 5. Training Regime, Optimization, and Inference The vision–language connector consists of two separate two-layer MLPs: MLP$_k$ for keyframe patch embeddings $\boldsymbol{p}_k^m \to \boldsymbol{h}_k^m$; MLP$_n$ for non-keyframe tokens $\boldsymbol{t}_n \to \boldsymbol{h}_n$. Inputs are interleaved in temporal order with the query text appended, then passed to a large language model (e.g., Qwen2-7B). Training minimizes cross-entropy loss: \[ \mathcal{L} = -\ln P\bigl(A \mid \{\boldsymbol{h}_k^m\} \cup \{\boldsymbol{h}_n\}, Q\bigr) \] The data schedule spans three phases (approx. 9.2 million samples): 1. Pretraining on WebVid, InternVid (7.4M video–caption pairs) 2. Short-video QA tuning (1.3M MC & OE samples) 3. Long-video fine-tuning (0.5M, e.g., FineVideo, CinePile) Optimization uses AdamW with cosine decay learning rates (vision: $2 \times 10^{-6}$, rest: $1 \times 10^{-5}$), batch size 1 ($\times 4$ with gradient accumulation), and mixed precision (FP16/FP8) for 32× A100 GPUs, completing one epoch in approximately two weeks. ViLaMP reduces token complexity from $O((MN)^2)$ in naive encodings to $O(KM + N)$, utilizing approximately 16.3K tokens for $N = 10$\,K, $K = 32$, $M = 196$. Empirical measurements show ≈50% lower GPU memory and ≈18% FLOPs relative to VideoChat-Flash at 10K frames [2504.02438]. ## 6. Empirical Performance and Benchmarking A summary of ViLaMP’s comparative evaluation (7B parameters) against contemporaneous open-source video-language models (7–9B scale) is presented below. | Model | LVBench | EgoSchema | LongVideoBench | MLVU | Video-MME Overall / Long | |------------------------|---------|-----------|----------------|------|-------------------------| | LLaVA-Video (7B) | – | 65.6 | 58.2 | 70.8 | 63.3 / 69.7 | | NVILA (7B) | – | – | – | 70.1 | 64.2 / 70.0 | | ViLaMP (7B, 1 FPS) | 45.2 | 70.2 | 61.2 | 72.6 | 67.5 / 73.5 | On the VideoNIAH “needle-in-a-haystack” benchmark (2K → 10K frames), ViLaMP sustains ≈58% accuracy at 10K frames, while VideoChat-Flash drops to ≈47%. At 8K frames, ViLaMP achieves FLOPs = 2.56T, memory = 45GB, compared to VideoChat-Flash's 13.9T FLOPs and 92GB memory usage [2504.02438]. ## 7. Implementation Details and Usage ViLaMP is implemented in PyTorch with Accelerate for mixed precision. Major components include: - **Vision Encoder:** SigLIP-so400m-patch14-384 (HuggingFace) - **Embedding Backbone:** CLIP-ViT-B-32 for frames and queries - **LLM:** Qwen2-7B - **Image Resolution:** 384×384 px - **Default Hyperparameters:** $K=32$ keyframes, temporal threshold $\tau=0.85$, patch redundancy weight $\lambda=1$, pooling sharpness $\alpha=10^{-2}$ - **Tokenization:** Keyframes retain $M$ patch tokens; non-keyframes produce a single pooled token each - **Memory/Throughput:** Designed for efficient batch-1 inference on single A100 GPUs (10K-frame videos) - **Code & Weights:** Public release at https://github.com/steven-ccq/ViLAMP ViLaMP extends the differential distillation approach across a lightweight hierarchical scheme, enabling end-to-end modeling on ultra-long video sequences while maintaining competitive accuracy and efficiency within its parameter regime [2504.02438]. Source: https://www.emergentmind.com/topics/vilamp