MVEB: Massive Video Embedding Benchmark

Abstract: We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

• The paper introduces MVEB, a unified benchmark evaluating video embeddings across 23 curated tasks spanning six modalities.
• It employs principled task filtering and zero-shot evaluation to reveal trade-offs among architectures and modality-specific performance.
• Experimental results highlight that audio can both enhance and detract from performance depending on annotation provenance, and model scale is secondary to data alignment.

Massive Video Embedding Benchmark: A Comprehensive Evaluation Framework for Video Representations

Motivation and Context

The development of robust video representation models is hampered by fragmented benchmarking practices. Existing video benchmarks typically focus on a narrow slice of functionality, such as action recognition or retrieval, without capturing the breadth of downstream tasks relevant to general-purpose video understanding. This results in a landscape where models can excel in isolated domains yet exhibit poor generalization, making cross-model and cross-task comparison unreliable.

The Massive Video Embedding Benchmark (MVEB) addresses this gap by providing a unified, multi-task, multi-modal, and systematically curated benchmark suite. As part of the broader MTEB family, MVEB enables standardized zero-shot evaluation of video, audio, and text embedding models, emphasizing modality coverage, task diversity, and reproducibility.

Figure 1: Overview of task types and example subtypes in MVEB+.

Benchmark Construction and Modalities

MVEB is built from a rigorously pruned pool of 184 video-centric tasks (MVEB+), resulting in a curated benchmark of 23 tasks that span six major categories: classification, zero-shot classification, clustering, pair classification, retrieval (across eight modality directions), and video-centric question answering. These tasks draw upon publicly available datasets with diverse domains (e.g., action recognition, emotion, instructional content) and modalities (video, audio, text).

MVEB uniquely supports paired evaluations for every audio-bearing dataset, allowing both video-only and video+audio variants on the same clips. This design enables direct quantification of audio's contribution to embedding quality, an aspect missing from prior benchmarks.

Task selection followed principled filtering based on domain/task diversity, modality coverage, linguistic breadth, redundancy reduction (via rank-correlation pruning), and runtime efficiency. To accommodate models with restricted modality support, MVEB provides modality-specific leaderboards, including text-video and video-only tracks in addition to the full audio/text/video scope.

Evaluation Protocol and Model Coverage

Evaluation is strictly zero-shot: models generate embeddings without any benchmark-specific or task-specific finetuning. The protocol aligns with each model's declared input configuration, such as number of video frames sampled, audio preprocessing, and input prompt templating.

The candidate set includes 33 public checkpoints spanning six paradigms:

• Self-supervised (video-only)
• Video-text contrastive
• Audio-visual contrastive
• Multimodal binding
• MLLM-based embedding (adapted from large multimodal LLMs)
• Generative MLLMs (used as embedders with hidden-state pooling)

Models are compared using Borda-style rank aggregation across all tasks and ensemble mean scores, with reproducibility ensured by versioned artifacts and public code releases.

Benchmark Results and Key Findings

No Single Model Dominates

LCO-Embedding-Omni-7B achieves the highest overall mean score (57.6) and excels in QA and clustering, but loses classification to BidirLM-Omni-2.5B-Embedding, retrieval to eBind, and pair classification to LCO-Embedding-Omni-3B. The absence of a clear all-category winner implies that current architectures involve trade-offs, and generalist video embedders are still an open research problem.

Figure 2: Per-category mean scores for five representative models spanning the four paradigms, illustrating the absence of a single dominant architecture.

Paradigm-Specific Insights

• Contrastive adaptation is essential: Generative MLLMs used as embedders, without contrastive adaptation, show significant performance collapse on cross-modal tasks (e.g., Qwen2.5-Omni-7B achieves only 12.8 mean, compared to 55.0 for its contrastive counterpart e5-omni-7B).
• Parameter scale is not the decisive factor: Smaller multimodal binding models (eBind-full at 1.8B and eBind-audio-vision at 764M) outperform much larger MLLM embedders in retrieval and zero-shot classification, indicating that training data alignment and objective are more important than raw backbone size.
• Audio+video evaluators vs. text-video specialists: On the video+audio+btext leaderboard, omni models dominate, but when evaluation is restricted to text-video, models optimized for this pairing (e.g., Qwen3-VL-Embedding-8B) outperform all audio-inclusive competitors.

  Figure 3: Mean score vs. parameter count for the three leaderboards, highlighting Pareto-efficient models and showing negligible gain from scale beyond modest parameter counts, particularly when audio is excluded.

Data and Annotation Provenance Effects

Paired evaluation of video-only and audio+video tasks, stratified by label provenance, establishes a six-point consistent gap: audio improves results when dataset labels are based on both visual and audio cues (AV-grounded), but harms performance when labels derive from visuals only (V-grounded). This is a reproducible, cross-family finding and overturns the naive expectation that audio universally enhances video understanding.

Temporal Context and Task Difficulty

Test-time scaling of frame count demonstrates rapid log-scale improvement from 1 to 8 frames and diminishing returns beyond 32 frames.

Figure 4: Per-model performance as a function of sampled frame count, with gains saturating at 32 frames.

Clustering and zero-shot classification remain the hardest categories, with absolute scores lagging well behind other tasks, primarily because of metric bounding, label noise, and the unsupervised character of clustering relative to supervised categories.

Retrieval: Cross-Modal Directional Structure

Analysis of retrieval directions shows that the eight possible query-target pairs cluster into three major capability groups according to model rank correlations, reducing the effective number of independent axes in cross-modal evaluation.

Figure 5: Pairwise Spearman correlation across the eight retrieval directions, showing strongest decorrelation between $T \to VA$ and $A \to V$.

Theoretical and Practical Implications

MVEB's findings have immediate ramifications for embedding model design and evaluation. The pronounced paradigm-specific strengths and weaknesses reinforce the necessity for explicit cross-modal adaptation stages. The clear demonstration that audio can harm embedding utility in visually grounded annotation settings challenges simplistic fusion strategies and calls for dynamic modality handling mechanisms—potentially with attention to metadata describing annotation provenance.

Moreover, the decoupling of model scale from performance, in light of the advances by small, well-aligned contrastive binders, compels a reevaluation of resource investment allocation, especially in the context of sustainable and maintainable model deployment.

MVEB's integration into the MTEB evaluation ecosystem establishes infrastructure for continuous benchmarking, community-driven expansion, and reproducible, versioned evaluation across modalities. This should facilitate more rapid progress in general-purpose representation learning and highlight the need for task- and annotation-aware assessment of embedding quality.

Conclusion

MVEB supplies a unified, modality- and task-diverse benchmark for video representation models, offering both breadth and depth via its 23-task curated suite derived from 184 tasks. The landscape revealed by MVEB is one of sharp trade-offs, modality- and annotation-sensitive effects, and a lack of overall dominance by any one architectural family or parameter scaling regime. These results set a new standard for evaluation rigor in multimodal representation learning and should influence both the construction of future backbone architectures and the development of benchmarking methodology itself (2606.14958).