--- title: Frozen Large Video Language Models url: https://www.emergentmind.com/topics/frozen-large-video-language-models-lvlms type: topic --- # Frozen Large Video Language Models Frozen Large Video Language Models (LVLMs) are Large Vision-Language Models designed for video tasks where the bulk of the model parameters—typically the multimodal backbone, vision encoder, and language components—remain unmodified ("frozen") during adaptation or downstream use. Instead of full or partial fine-tuning, only lightweight modules (such as adapters, prompts, or fusion heads) are trained, or in some regimes, no training is used at all. This paradigm enables parameter-efficient adaptation, strong out-of-domain generalization, maximal reuse of pre-trained capabilities, and scalable deployment. Frozen LVLMs are foundational in contemporary systems for micro-video recommendation, video question answering, long video understanding, fine-grained action recognition, and other scenarios where training efficiency, plug-and-play extensibility, or data scarcity are principal constraints. ## 1. Architectural Principles and Freezing Strategies Frozen LVLM architectures retain the parameters of the core video-language backbone, leveraging pre-training or instruction-tuning performed on massive video/image-text corpora. The primary strategies to utilize these models in a frozen state include: - **Plug-and-Play Feature Extraction:** The LVLM generates rich clip-level or event-level embeddings from visual input, which downstream systems consume with minimal adaptation. - **Adapter- or Prompt-Based Tuning:** All LVLM weights remain fixed; adaptation is restricted to shallow modules, such as soft prompts with parameter-efficient tuning (e.g., P-Tuning v2), or small alignment MLPs for token space projection [2508.15903][2504.06835]. - **Training-Free Regimes:** The LVLM weights are untouched; adaptation relies on architectural mediation (e.g., attention-based selection or token pooling) and procedures like retrieval augmentation, with no gradient-based model updates [2503.11205][2306.11732]. - **Hybrid Compression and Selection:** Query-guided or attention-driven token compression reduces a long video to a manageable sequence of visual "pseudo-tokens," enabling existing (short-context) LVLMs to handle long videos without retraining [2504.06835][2503.11205]. Freezing is motivated by efficiency, risk of catastrophic forgetting, desire for broad generalization, and the high cost of video-scale fine-tuning. ## 2. Feature Extraction and Compression Paradigms Multiple paradigms are adopted for representing video input with frozen LVLMs: - **Intermediate Hidden State Extraction:** Dense, per-frame or per-segment hidden states from the LVLM decoder are preferred over caption-based summarization, since they preserve fine-grained visual semantics essential for recommendation and reasoning [2512.21863]. - **Caption-Based Representations:** Generated captions can be embedded for downstream use, but this approach discards temporal and local visual cues [2512.21863]. - **Event Sequence Mapping:** Video-to-event mappers transform raw video into discrete, temporally aligned event sequences using spatio-temporal feature extraction, adaptive pooling, and codebook-based quantization—yielding semantically coherent inputs for frozen language backbones [2508.15903]. - **Query/Prompt-Guided Compression:** Lightweight compression modules distill dense frame tokens into compact, query-specific sequences using attention with frozen encoders. This enables efficient encoding of long videos into the token limits of existing LVLMs [2504.06835][2503.11205]. A systematic empirical study demonstrated that hidden-state-centric features outperform captions by 1–2% absolute in Hit@10 metrics for recommendation, and that multi-layer aggregation (e.g., uniform averaging or learnable weights) captures complementary granularity, further boosting performance [2512.21863]. ## 3. Integration and Fusion Mechanisms Frozen LVLM-derived features are integrated into larger systems via multiple fusion strategies: - **ID Embedding Fusion:** In recommendation, LVLM features are adaptively gated and fused with item ID embeddings to capture collaborative filtering (CF) signals. Fusion is strictly superior to replacement; replacing ID embeddings with pure content features degrades performance below that of using IDs alone [2512.21863]. ```math g_i = \sigma(\mathrm{MLP}([e_i^\mathrm{id}; e_i^\mathrm{v}])) \quad e_i = g_i \odot e_i^\mathrm{id} + (1-g_i) \odot e_i^\mathrm{v} ``` - **Prompt Fusion:** In action recognition, soft-prompt tokens are prepended and only their embeddings are updated during training, providing efficient parameter tuning with the LVLM backbone remaining entirely frozen [2508.15903]. - **Alignment Layers:** Small multi-layer projections align compressed visual tokens with the LLM's text embedding space, enabling text generation or classification with pre-trained backbones [2504.06835]. - **Training-Free Token Selection:** Attention-based selection and pooling assemble a set of informative visual tokens for inference in video QA and understanding, without any parameter updates [2503.11205]. Integration strategies emphasize efficiency: all trainable weights reside in the fusion or adapter module, with LVLM computation amortized across tasks and datasets. ## 4. Representative Applications Frozen LVLMs have been central to advances in the following domains: - **Micro-Video Recommendation:** Leveraging frozen LVLMs with Dual Feature Fusion (DFF), state-of-the-art performance is achieved—Hit@10 reaches 0.1020, +6.9% over baselines—with trainable parameters limited to a lightweight integrator and sequential backbone. Hidden-state features, adaptive ID fusion, and uniform/multi-layer aggregation are all essential [2512.21863]. - **Long Video Question Answering (VideoQA) and Captioning:** Methods like LVC and LLaVA-MLB compress long sequences of patch tokens using query-guided or attention-guided pooling, transforming sequences spanning 64 frames into a handful of pseudo-image tokens. This enables direct feeding of long inputs into frozen VLMs such as InternVL2 or Phi-3.5-Vision, yielding +14.6% accuracy improvement on MLVU benchmarks [2504.06835][2503.11205]. - **Zero-Shot VideoQA via Retrieval Augmentation:** The R2A (Retrieving-to-Answer) framework orchestrates frozen CLIP encoders, large text retrieval corpora, and LLMs such as DeBERTa. It retrieves semantically aligned captions, formats them into temporally-aware prompts, and predicts masked answers with zero trainable parameters. R2A outperforms Flamingo-80B (parameter- and compute-heavy) on NextQA and similar datasets despite being fully frozen [2306.11732]. - **Fine-Grained Action Recognition:** VT-LVLM-AR applies a frozen LLaVA-1.5 (7B) with a compact video-to-event mapping and prompt tuning of only ~1.2M parameters, achieving 94.1% accuracy on NTU RGB+D X-Sub and providing highly interpretable intermediate representations [2508.15903]. ## 5. Efficiency, Generalization, and Empirical Insights Frozen LVLMs exhibit consistently strong empirical properties: - **Inference and Training Efficiency:** Since no gradient updates are applied to backbone LVLM parameters, both training cost and inference latency are minimized. For example, LVC's alignment layer (∼10^8 parameters) trains in under 7 hours, with full backbone fine-tuning requiring >100 GPU-hours [2504.06835]. - **Parameter and Compute Cost:** The majority of the parameter budget remains untrained; trainable adapters constitute less than 2% of the total parameters in DFF and <0.02% in VT-LVLM-AR [2512.21863][2508.15903]. In R2A, all modules, from retrieval model to LLM, remain entirely frozen [2306.11732]. - **Generalization and Robustness:** Performance on out-of-distribution splits and novel word/location benchmarks indicates that frozen LVLMs, when appropriately adapted via fusion or prompt mechanisms, retain strong generalization—sometimes surpassing much larger, fine-tuned multi-modal models [2306.11732][2508.15903]. - **Token and Latency Trade-Offs:** LLaVA-MLB compresses the number of tokens fed to the LLM by up to 40–50%, with concurrent speedups and 0.5–4 ppt accuracy advances, as measured on ANet-QA, NExTQA, EgoSchema, and VCGBench [2503.11205]. | Model/Method | Frozen Params | Trainable Params | Application Domain | |-------------------|:------------:|:----------------:|------------------------------| | DFF (2512.21863) | >99% LVLM | <2% fusion/backb.| Micro-video recommendation | | LVC (2504.06835) | 100% backbone| 10^8 alignment | Long video understanding | | VT-LVLM-AR (2508.15903)| 7B LVLM | 1.2M prompt | Action recognition | | LLaVA-MLB (2503.11205) | 100% | 0% | Training-free video QA | | R2A (2306.11732) | 100% | 0% | Zero-shot VideoQA | ## 6. Methodological Considerations and Empirical Guidelines The following principles have been substantiated across systematic evaluations: - **Hidden states from intermediate LVLM layers consistently outperform caption-based features** for all high-resolution, temporally-dense retrieval, ranking, or classification tasks [2512.21863]. - **Fusion with discrete ID or collaborative signals** is essential in recommendation; naive replacement with content-only embeddings degrades both accuracy and robustness [2512.21863]. - **Multi-granularity layer aggregation** (e.g., averaging or learnable global weights) captures complementary visual semantics [2512.21863]. - **Training or designing adapters only at the compression or fusion interface** preserves the LVLM’s catastrophic forgetting resistance and open-domain generalization [2508.15903][2504.06835]. - **Fully training-free and parameter-efficient methods** (e.g., LLaVA-MLB, R2A) suffice to outperform many baseline and even large fine-tuned models—provided plug-and-play retrieval, pooling, or prompt conversion is used intelligently [2503.11205][2306.11732]. Recommended practices include extracting dense hidden states, leveraging gating-based fusion, utilizing multi-layer signals, feeding raw video frames (not merely metadata), and keeping all core LVLM parameters frozen [2512.21863]. ## 7. Limitations, Extensions, and Outlook Frozen LVLMs rely fundamentally on the breadth and alignment of pre-trained visual and language priors. Identified limitations include: - **Temporal reasoning may be compromised by excessive compression** or limited by the fixed capacity of base LVLMs [2504.06835][2503.11205]. - **Coverage of retrieval and alignment models,** as in R2A, is limited by corpus and encoder flexibility; rare or highly specialized concepts may be missed [2306.11732]. - **No direct visual reasoning beyond caption-aligned content** in retrieval-based approaches; improvements may require integration of multi-modal or audio cues. - **Potential extension to larger and more diverse corpora,** additional modalities, and adaptive plug-in adapters—without sacrificing the frozen model’s efficiency and generalization properties. A plausible implication is that, as video-language pre-training becomes more comprehensive and compression/adapter schemes become more sophisticated, frozen LVLMs will remain integral to scalable, efficient, and robust video understanding systems. This paradigm strongly aligns with trends in modular foundation model deployment and parameter-efficient adaptation. --- **References:** - [2512.21863]: "Frozen LVLMs for Micro-Video Recommendation: A Systematic Study of Feature Extraction and Fusion" - [2504.06835]: "LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding" - [2508.15903]: "VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videos" - [2503.11205]: "LLaVA-MLB: Mitigating and Leveraging Attention Bias for Training-Free Video LLMs" - [2306.11732]: "Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models" Source: https://www.emergentmind.com/topics/frozen-large-video-language-models-lvlms