Frozen Large Video Language Models

• Frozen LVLMs are large vision-language models with fixed core parameters, enabling reuse of pre-trained features and robust generalization.
• They employ adapter tuning, prompt-based approaches, and query-guided compression to efficiently extract, compress, and fuse video embeddings.
• These models power applications like micro-video recommendation, video QA, and fine-grained action recognition with minimal trainable parameters.

Frozen Large Video LLMs (LVLMs) are Large Vision-LLMs designed for video tasks where the bulk of the model parameters—typically the multimodal backbone, vision encoder, and language components—remain unmodified ("frozen") during adaptation or downstream use. Instead of full or partial fine-tuning, only lightweight modules (such as adapters, prompts, or fusion heads) are trained, or in some regimes, no training is used at all. This paradigm enables parameter-efficient adaptation, strong out-of-domain generalization, maximal reuse of pre-trained capabilities, and scalable deployment. Frozen LVLMs are foundational in contemporary systems for micro-video recommendation, video question answering, long video understanding, fine-grained action recognition, and other scenarios where training efficiency, plug-and-play extensibility, or data scarcity are principal constraints.

1. Architectural Principles and Freezing Strategies

Frozen LVLM architectures retain the parameters of the core video-language backbone, leveraging pre-training or instruction-tuning performed on massive video/image-text corpora. The primary strategies to utilize these models in a frozen state include:

• Plug-and-Play Feature Extraction: The LVLM generates rich clip-level or event-level embeddings from visual input, which downstream systems consume with minimal adaptation.
• Adapter- or Prompt-Based Tuning: All LVLM weights remain fixed; adaptation is restricted to shallow modules, such as soft prompts with parameter-efficient tuning (e.g., P-Tuning v2), or small alignment MLPs for token space projection (Li et al., 21 Aug 2025, Wang et al., 9 Apr 2025).
• Training-Free Regimes: The LVLM weights are untouched; adaptation relies on architectural mediation (e.g., attention-based selection or token pooling) and procedures like retrieval augmentation, with no gradient-based model updates (Shen et al., 14 Mar 2025, Pan et al., 2023).
• Hybrid Compression and Selection: Query-guided or attention-driven token compression reduces a long video to a manageable sequence of visual "pseudo-tokens," enabling existing (short-context) LVLMs to handle long videos without retraining (Wang et al., 9 Apr 2025, Shen et al., 14 Mar 2025).

Freezing is motivated by efficiency, risk of catastrophic forgetting, desire for broad generalization, and the high cost of video-scale fine-tuning.

2. Feature Extraction and Compression Paradigms

Multiple paradigms are adopted for representing video input with frozen LVLMs:

• Intermediate Hidden State Extraction: Dense, per-frame or per-segment hidden states from the LVLM decoder are preferred over caption-based summarization, since they preserve fine-grained visual semantics essential for recommendation and reasoning (Sun et al., 26 Dec 2025).
• Caption-Based Representations: Generated captions can be embedded for downstream use, but this approach discards temporal and local visual cues (Sun et al., 26 Dec 2025).
• Event Sequence Mapping: Video-to-event mappers transform raw video into discrete, temporally aligned event sequences using spatio-temporal feature extraction, adaptive pooling, and codebook-based quantization—yielding semantically coherent inputs for frozen language backbones (Li et al., 21 Aug 2025).
• Query/Prompt-Guided Compression: Lightweight compression modules distill dense frame tokens into compact, query-specific sequences using attention with frozen encoders. This enables efficient encoding of long videos into the token limits of existing LVLMs (Wang et al., 9 Apr 2025, Shen et al., 14 Mar 2025).

A systematic empirical study demonstrated that hidden-state-centric features outperform captions by 1–2% absolute in Hit@10 metrics for recommendation, and that multi-layer aggregation (e.g., uniform averaging or learnable weights) captures complementary granularity, further boosting performance (Sun et al., 26 Dec 2025).

3. Integration and Fusion Mechanisms

Frozen LVLM-derived features are integrated into larger systems via multiple fusion strategies:

• ID Embedding Fusion: In recommendation, LVLM features are adaptively gated and fused with item ID embeddings to capture collaborative filtering (CF) signals. Fusion is strictly superior to replacement; replacing ID embeddings with pure content features degrades performance below that of using IDs alone (Sun et al., 26 Dec 2025).

g_i = \sigma(\mathrm{MLP}([e_i^\mathrm{id}; e_i^\mathrm{v}])) \quad e_i = g_i \odot e_i^\mathrm{id} + (1-g_i) \odot e_i^\mathrm{v}

• Prompt Fusion: In action recognition, soft-prompt tokens are prepended and only their embeddings are updated during training, providing efficient parameter tuning with the LVLM backbone remaining entirely frozen (Li et al., 21 Aug 2025).
• Alignment Layers: Small multi-layer projections align compressed visual tokens with the LLM's text embedding space, enabling text generation or classification with pre-trained backbones (Wang et al., 9 Apr 2025).
• Training-Free Token Selection: Attention-based selection and pooling assemble a set of informative visual tokens for inference in video QA and understanding, without any parameter updates (Shen et al., 14 Mar 2025).

Integration strategies emphasize efficiency: all trainable weights reside in the fusion or adapter module, with LVLM computation amortized across tasks and datasets.

4. Representative Applications

Frozen LVLMs have been central to advances in the following domains:

• Micro-Video Recommendation: Leveraging frozen LVLMs with Dual Feature Fusion (DFF), state-of-the-art performance is achieved—Hit@10 reaches 0.1020, +6.9% over baselines—with trainable parameters limited to a lightweight integrator and sequential backbone. Hidden-state features, adaptive ID fusion, and uniform/multi-layer aggregation are all essential (Sun et al., 26 Dec 2025).
• Long Video Question Answering (VideoQA) and Captioning: Methods like LVC and LLaVA-MLB compress long sequences of patch tokens using query-guided or attention-guided pooling, transforming sequences spanning 64 frames into a handful of pseudo-image tokens. This enables direct feeding of long inputs into frozen VLMs such as InternVL2 or Phi-3.5-Vision, yielding +14.6% accuracy improvement on MLVU benchmarks (Wang et al., 9 Apr 2025, Shen et al., 14 Mar 2025).
• Zero-Shot VideoQA via Retrieval Augmentation: The R2A (Retrieving-to-Answer) framework orchestrates frozen CLIP encoders, large text retrieval corpora, and LLMs such as DeBERTa. It retrieves semantically aligned captions, formats them into temporally-aware prompts, and predicts masked answers with zero trainable parameters. R2A outperforms Flamingo-80B (parameter- and compute-heavy) on NextQA and similar datasets despite being fully frozen (Pan et al., 2023).
• Fine-Grained Action Recognition: VT-LVLM-AR applies a frozen LLaVA-1.5 (7B) with a compact video-to-event mapping and prompt tuning of only ~1.2M parameters, achieving 94.1% accuracy on NTU RGB+D X-Sub and providing highly interpretable intermediate representations (Li et al., 21 Aug 2025).

5. Efficiency, Generalization, and Empirical Insights

Frozen LVLMs exhibit consistently strong empirical properties:

• Inference and Training Efficiency: Since no gradient updates are applied to backbone LVLM parameters, both training cost and inference latency are minimized. For example, LVC's alignment layer (∼10^8 parameters) trains in under 7 hours, with full backbone fine-tuning requiring >100 GPU-hours (Wang et al., 9 Apr 2025).
• Parameter and Compute Cost: The majority of the parameter budget remains untrained; trainable adapters constitute less than 2% of the total parameters in DFF and <0.02% in VT-LVLM-AR (Sun et al., 26 Dec 2025, Li et al., 21 Aug 2025). In R2A, all modules, from retrieval model to LLM, remain entirely frozen (Pan et al., 2023).
• Generalization and Robustness: Performance on out-of-distribution splits and novel word/location benchmarks indicates that frozen LVLMs, when appropriately adapted via fusion or prompt mechanisms, retain strong generalization—sometimes surpassing much larger, fine-tuned multi-modal models (Pan et al., 2023, Li et al., 21 Aug 2025).
• Token and Latency Trade-Offs: LLaVA-MLB compresses the number of tokens fed to the LLM by up to 40–50%, with concurrent speedups and 0.5–4 ppt accuracy advances, as measured on ANet-QA, NExTQA, EgoSchema, and VCGBench (Shen et al., 14 Mar 2025).

Model/Method	Frozen Params	Trainable Params	Application Domain
DFF (Sun et al., 26 Dec 2025)	>99% LVLM	<2% fusion/backb.	Micro-video recommendation
LVC (Wang et al., 9 Apr 2025)	100% backbone	10^8 alignment	Long video understanding
VT-LVLM-AR (Li et al., 21 Aug 2025)	7B LVLM	1.2M prompt	Action recognition
LLaVA-MLB (Shen et al., 14 Mar 2025)	100%	0%	Training-free video QA
R2A (Pan et al., 2023)	100%	0%	Zero-shot VideoQA

6. Methodological Considerations and Empirical Guidelines

The following principles have been substantiated across systematic evaluations:

• Hidden states from intermediate LVLM layers consistently outperform caption-based features for all high-resolution, temporally-dense retrieval, ranking, or classification tasks (Sun et al., 26 Dec 2025).
• Fusion with discrete ID or collaborative signals is essential in recommendation; naive replacement with content-only embeddings degrades both accuracy and robustness (Sun et al., 26 Dec 2025).
• Multi-granularity layer aggregation (e.g., averaging or learnable global weights) captures complementary visual semantics (Sun et al., 26 Dec 2025).
• Training or designing adapters only at the compression or fusion interface preserves the LVLM’s catastrophic forgetting resistance and open-domain generalization (Li et al., 21 Aug 2025, Wang et al., 9 Apr 2025).
• Fully training-free and parameter-efficient methods (e.g., LLaVA-MLB, R2A) suffice to outperform many baseline and even large fine-tuned models—provided plug-and-play retrieval, pooling, or prompt conversion is used intelligently (Shen et al., 14 Mar 2025, Pan et al., 2023).

Recommended practices include extracting dense hidden states, leveraging gating-based fusion, utilizing multi-layer signals, feeding raw video frames (not merely metadata), and keeping all core LVLM parameters frozen (Sun et al., 26 Dec 2025).

7. Limitations, Extensions, and Outlook

Frozen LVLMs rely fundamentally on the breadth and alignment of pre-trained visual and language priors. Identified limitations include:

• Temporal reasoning may be compromised by excessive compression or limited by the fixed capacity of base LVLMs (Wang et al., 9 Apr 2025, Shen et al., 14 Mar 2025).
• Coverage of retrieval and alignment models, as in R2A, is limited by corpus and encoder flexibility; rare or highly specialized concepts may be missed (Pan et al., 2023).
• No direct visual reasoning beyond caption-aligned content in retrieval-based approaches; improvements may require integration of multi-modal or audio cues.
• Potential extension to larger and more diverse corpora, additional modalities, and adaptive plug-in adapters—without sacrificing the frozen model’s efficiency and generalization properties.

A plausible implication is that, as video-language pre-training becomes more comprehensive and compression/adapter schemes become more sophisticated, frozen LVLMs will remain integral to scalable, efficient, and robust video understanding systems. This paradigm strongly aligns with trends in modular foundation model deployment and parameter-efficient adaptation.

---

References:

• (Sun et al., 26 Dec 2025): "Frozen LVLMs for Micro-Video Recommendation: A Systematic Study of Feature Extraction and Fusion"
• (Wang et al., 9 Apr 2025): "LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding"
• (Li et al., 21 Aug 2025): "VT-LVLM-AR: A Video-Temporal Large Vision-LLM Adapter for Fine-Grained Action Recognition in Long-Term Videos"
• (Shen et al., 14 Mar 2025): "LLaVA-MLB: Mitigating and Leveraging Attention Bias for Training-Free Video LLMs"
• (Pan et al., 2023): "Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen LLMs"