MVEB: Massive Video Embedding Benchmark

• MVEB is a unified benchmark that assesses fixed-dimensional video embeddings across six diverse task families without per-task fine-tuning.
• It employs a rigorous five-stage curation process to select 23 tasks from a larger pool, ensuring task diversity, rank fidelity, and improved runtime efficiency.
• Empirical findings reveal no single model dominates all tasks, emphasizing contrastive adaptation as key for robust cross-modal performance.

Searching arXiv for MVEB and related embedding benchmark papers. Massive Video Embedding Benchmark (MVEB) is a video-embedding benchmark introduced to evaluate the zero-shot transfer quality of general-purpose video representations across six task families: classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric multiple-choice question answering. It is positioned as the video counterpart within the MTEB family of embedding benchmarks and is implemented inside the MTEB framework, with unified evaluation across text, image, audio, and video modalities. MVEB is a curated 23-task subset drawn from a larger 184-task pool called MVEB+, and its central design goal is to preserve task diversity and rank fidelity while substantially reducing evaluation cost (Assadi et al., 12 Jun 2026).

1. Definition and scope

MVEB evaluates a video embedding as a fixed-dimensional representation $z$ produced from a video input, optionally joined with its audio track and/or paired text depending on the task. The benchmark evaluates embeddings directly, without per-task fine-tuning, and uses either simple heads or similarity-based rules to solve downstream problems. In this setting, temporal aggregation follows each model’s native design; when needed, a simple uniform sampler and mean-pooling are used:

$z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$

where $x_t$ are frames sampled at model-declared fps and $T$ is the number of frames passed to encoder $f$ (Assadi et al., 12 Jun 2026).

The benchmark was created to address a gap in prior video evaluation practice: earlier benchmarks often isolate a single capability, such as retrieval or action recognition, which makes it difficult to assess whether embeddings transfer across heterogeneous downstream tasks. MVEB instead unifies diverse tasks, modalities, and input-output directions under one maintained protocol. This suggests that the benchmark is intended less as a narrow leaderboard for a single problem class than as an evaluation substrate for comparing representation learning paradigms under a common zero-shot regime (Assadi et al., 12 Jun 2026).

MVEB spans classification, zero-shot classification, clustering, pair classification, retrieval with eight cross-modal directions, and video-centric QA. The retrieval directions are explicitly $T \rightarrow V$, $A \rightarrow V$, $AT \rightarrow V$, $V \rightarrow T$, $VA \rightarrow T$, $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$0, $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$1, and $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$2. Video-centric QA is implemented as retrieval among answer options rather than as unrestricted generation (Assadi et al., 12 Jun 2026).

2. Benchmark composition, task design, and metrics

MVEB is selected from MVEB+ through a five-stage curation procedure: validity, unique coverage, linguistic breadth, redundancy removal via Spearman rank correlation, and runtime efficiency. If two tasks correlate at $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$3 over models, the broader or lower-runtime one is retained. According to the benchmark report, this pruning preserves rank fidelity relative to MVEB+, with Pearson $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$4 and Spearman $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$5, while accelerating evaluation by $z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$6–$z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$7 on a single H100 GPU (Assadi et al., 12 Jun 2026).

The benchmark’s six task families differ in both supervision structure and scoring rule. Classification uses a linear probe over frozen embeddings with 8 examples per class and reports accuracy as the main metric:

$z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$8

Macro-F1 can be optionally computed:

$z = \frac{1}{T} \sum_{t=1}^{T} f(x_t)$9

Zero-shot classification uses text embeddings of label prompts such as “a video of {label}” and predicts by similarity argmax, again with accuracy as the main metric. Only models that declare a joint video-text space participate in that task family (Assadi et al., 12 Jun 2026).

Clustering uses MiniBatchKMeans with $x_t$0 equal to the number of gold labels and reports V-measure:

$x_t$1

with

$x_t$2

The benchmark explicitly states that it uses V-measure rather than NMI, though it also records the standard NMI definition for reference. Pair classification takes two embeddings and predicts whether they satisfy a criterion such as same activity or same speaker; the primary score is maximum average precision over cosine-similarity thresholds, using

$x_t$3

Retrieval uses ranked target lists and adopts nDCG@10 as the primary metric:

$x_t$4

For video-centric QA, the score is accuracy over multiple-choice answer retrieval (Assadi et al., 12 Jun 2026).

A recurrent design principle is that embedding similarity is cosine unless the model declares an alternative. Cross-modal similarity is computed in the models’ stated embedding spaces, and the benchmark notes that models trained with contrastive losses typically align modalities through InfoNCE-style objectives:

$x_t$5

This suggests that the benchmark is partly organized around testing whether such alignment survives transfer beyond the objectives on which the models were trained (Assadi et al., 12 Jun 2026).

3. Datasets and modality coverage

MVEB covers classic action datasets, instruction and conversational video, audiovisual events, music QA, multi-shot narratives, and large-scale captioned video. The benchmark’s classification and zero-shot classification coverage includes Breakfast, HMDB51, Kinetics-400/600/700-2020, Something-Something v2, UCF101, an audio-bearing UCF101-51VA subset, VGGSound, RAVDESS, MELD, AVE, and WorldSense short-clip slices. Clustering includes HMDB51, UCF101, RAVDESS, AVE, MELD, MUSIC-AVQA, and WorldSense-1min domain clusters. Pair classification includes HumanAnimalCartoon, MELD, RAVDESS, AVE, MUSIC-AVQA, and Vinoground. Retrieval spans MSVD, MSR-VTT, DiDeMo, ActivityNet Captions, VATEX, PANDA-70M, Shot2Story-20K, YouCook2, AudioCaps-AV, VGGSound-AV, VALOR-32K, and AVMemeExam. Video-centric QA includes NExT-QA, EgoSchema, Perception Test, Daily-Omni, Video-MME, OmniVideoBench, WorldQA, AVQA, AVMeme Exam, and WorldSense-1min (Assadi et al., 12 Jun 2026).

A notable methodological feature is the systematic pairing of audio-bearing sources into two variants on the same clips: video-only ($x_t$6) and audio+video ($x_t$7). This permits apples-to-apples comparisons of the audio track’s contribution. Licensing and redistribution follow upstream releases, while MVEB hosts evaluation wrappers and, where allowed, decoded frame samples and 16 kHz mono audio excerpts under the mteb/ namespace on Hugging Face rather than redistributing full-resolution videos (Assadi et al., 12 Jun 2026).

The benchmark therefore mixes uni- and multimodal evaluation regimes while controlling clip identity. A plausible implication is that MVEB is designed not merely to test representation strength, but to separate gains due to modality access from gains due to dataset choice or task formulation. That distinction becomes central in the benchmark’s audio analysis (Assadi et al., 12 Jun 2026).

4. Model families and evaluation protocol

MVEB evaluates 33 open checkpoints spanning six paradigms. These are self-supervised video encoders from the V-JEPA-2 family; video-text contrastive encoders such as X-CLIP; audio-visual contrastive encoders with a shared AVT space, namely Perception Encoder (PE-AV) small, base, and large; multimodal binding models in the eBind family; MLLM-based embedding models adapted to produce embeddings, including LCO-Embedding-Omni, e5-omni, BidirLM-Omni-2.5B-Embedding, UME-R1, OmniEmbed-Nemotron-3B, VLM2Vec-V2.0, Qwen3-VL-Embedding, and Jina-Embeddings-v5-omni variants; and generative MLLMs used as embedders without an adaptation stage, specifically Qwen2.5-Omni models (Assadi et al., 12 Jun 2026).

The report distinguishes “multimodal binding” from MLLM-based embedding approaches. In the benchmark’s terminology, multimodal binding aligns modality-specific encoders to a single shared embedding space without an MLLM backbone. By contrast, “contrastive adaptation” refers to a dedicated embedding training stage, such as contrastive or retrieval objectives, layered on top of a generative MLLM backbone. That distinction becomes important in the interpretation of cross-modal results (Assadi et al., 12 Jun 2026).

The evaluation pipeline is integrated into the MTEB framework. It uses unified interfaces and metrics across modalities, versioned tasks and models, named experiment scopes, and richer model metadata for reproducibility. Frame sampling respects model-declared policies to avoid out-of-distribution evaluation. Variable-length encoders, including the Omni MLLM-embed family, Qwen3-VL-Embedding, and Perception Encoder variable-length checkpoints, use fps $x_t$8 with max_frames = 64 by default; fixed-length encoders such as X-CLIP, eBind, and V-JEPA-2 use their native frame counts. Audio is resampled to each model’s target sampling rate, converted to mono, and subject to per-model duration caps where applicable. Embedding production is strictly zero-shot, except that classification uses logistic regression trained on frozen embeddings with 8 examples per class (Assadi et al., 12 Jun 2026).

Model ranking combines per-task scores through a Borda count and also reports arithmetic means. Contamination audits cross-reference declared training data and are reported where available. The benchmark also provides modality-restricted leaderboards: MVEB with 23 tasks, MVEB(text, video) with 19 tasks, and MVEB(video) with 9 tasks (Assadi et al., 12 Jun 2026).

5. Empirical findings

The principal empirical result is that no single model dominates across all 23 tasks. LCO-Embedding-Omni-7B ranks first overall by Borda count with mean 57.6 and leads QA and clustering, while remaining close to leaders in other categories. Category-specific leaders differ by task family: eBind leads retrieval at approximately 62.3 and zero-shot classification at approximately 61.1 despite being in the 0.76–1.8B parameter range; BidirLM-Omni-2.5B-Embedding leads classification at approximately 61.2; LCO-Embedding-Omni-3B leads pair classification at approximately 80.7 on MVEB(video); and LCO-Embedding-Omni-7B leads QA at approximately 57.0 (Assadi et al., 12 Jun 2026).

The text-video-only leaderboard changes the ranking structure. On MVEB(text, video), Qwen3-VL-Embedding-8B and Qwen3-VL-Embedding-2B take ranks 1 and 2 with means 60.9 and 58.1, ahead of all omni AVT models, and win five of six categories, with QA remaining led by LCO. This indicates that leaderboard position depends materially on modality budget and benchmark slice rather than only on model family or parameter count (Assadi et al., 12 Jun 2026).

A central result concerns contrastive adaptation. Generative MLLMs used as embedders without such adaptation collapse on cross-modal tasks: Qwen2.5-Omni-7B and Qwen2.5-Omni-3B score means of 12.8 and 11.4, compared with 55.0 for e5-omni-7B from the same backbone family after contrastive adaptation. The benchmark characterizes a dedicated contrastive or retrieval stage as a near-prerequisite for cross-modal tasks. It also reports that smaller specialized contrastive models, including eBind and PE-AV, outperform or rival larger MLLM-based embedders on retrieval, indicating that training-data alignment can exceed raw scale as a determinant of cross-modal transfer (Assadi et al., 12 Jun 2026).

The benchmark further analyzes retrieval-direction structure through Spearman correlations across models. The eight directions cluster into three capability groups: text-target retrieval ($x_t$9, $T$0) with $T$1; video-target retrieval with audio in the query ($T$2, $T$3, $T$4) with $T$5 among them; and text-as-query directions ($T$6, $T$7, $T$8) with $T$9. The most decoupled pair is $f$0 versus $f$1 at $f$2. The benchmark interprets this as evidence that its treatment of audio+video as a joint target exposes a distinct capability axis (Assadi et al., 12 Jun 2026).

6. Audio contribution and annotation provenance

One of the most specific findings in MVEB concerns the effect of adding audio. Across 48 paired task groups and 14 audio-capable models, the benchmark defines

$f$3

for each model-dataset pair. It then groups datasets by annotation provenance: AV-grounded if labels were produced from both audio and visual content, and V-grounded if labels were produced from visuals alone (Assadi et al., 12 Jun 2026).

The sign of $f$4 depends on that provenance. For AV-grounded datasets, the mean effect is $f$5 with $f$6 and $f$7; for V-grounded datasets, the mean effect is $f$8 with $f$9 and $T \rightarrow V$0. The report describes this as an approximately six-point gap consistent across task types and model families. It further notes that the negative penalty on V-grounded datasets is about three times the positive benefit on AV-grounded datasets, suggesting that audio becomes a confound when labels do not reward it (Assadi et al., 12 Jun 2026).

The effect is not reducible to a single model paradigm. Paradigm-level means lie near zero, with $T \rightarrow V$1, while within-paradigm spread is larger than cross-paradigm differences. eBind variants consistently lose from adding audio at approximately $T \rightarrow V$2, whereas MLLM-based embedders range from $T \rightarrow V$3 for e5-omni-3B to $T \rightarrow V$4 for BidirLM-Omni-2.5B. The benchmark therefore concludes that the outcome is dataset- and implementation-dependent rather than paradigmatic (Assadi et al., 12 Jun 2026).

The v-versus-va comparison is tightly controlled: clips are held constant, metrics and evaluation code are identical, and low-scoring outliers with near-random video-only baselines were excluded from the $T \rightarrow V$5 summary to avoid noise-dominated differences. A plausible implication is that MVEB’s audio findings are intended as an evaluation-specific statement about annotation alignment rather than as a general claim that audio is intrinsically beneficial or harmful (Assadi et al., 12 Jun 2026).

7. Runtime, practical use, and limitations

MVEB’s curation is motivated partly by computational tractability. On a single NVIDIA H100, MVEB requires approximately 32 hours for a strong omni model, compared with approximately 300 hours for MVEB+. Representative runtimes are reported as follows: LCO-Embedding-Omni-7B, 31.9 hours on MVEB versus 302.7 on MVEB+; Qwen2.5-Omni-7B, 29.4 versus 267.3; PE-AV-small, 16.7 versus 159.7; and eBind-audio-vision, 16.8 versus 122.7 (Assadi et al., 12 Jun 2026).

The benchmark also studies test-time temporal budget. Frame-scaling experiments over $T \rightarrow V$6 show logarithmic gains, with most improvement arriving by $T \rightarrow V$7 and the change from 32 to 64 frames adding approximately 2.2 points on average. The report states that 32 frames is a reasonable ceiling for general-purpose evaluation. This suggests that MVEB is designed not only to compare models, but also to provide operational guidance for evaluation under compute constraints (Assadi et al., 12 Jun 2026).

Practical recommendations in the report follow directly from the measured category leaders. For cross-modal retrieval or zero-shot classification, multimodal binding models, especially eBind, are presented as the strongest option and highly parameter-efficient. For classification, clustering, pair classification, or video-centric QA, MLLM-based embedders with contrastive adaptation, such as LCO-Embedding-Omni and BidirLM-Omni-Embedding, are described as leading. The benchmark advises against using generative MLLMs “as-is” for embeddings because they collapse on cross-modal tasks without a contrastive or retrieval adaptation stage. It also recommends enabling audio for AV-grounded labels and disabling it for V-grounded labels to avoid an average penalty of approximately four to five points, while noting per-model variation in audio handling (Assadi et al., 12 Jun 2026).

The reported limitations are also specific. Model coverage is characterized as a snapshot, with additional open checkpoints to be added as clean inference APIs stabilize, including ImageBind, LanguageBind, InternVideo, VideoMAE, and V-JEPA-2.1. Absolute scores partly reflect training-time frame and audio budgets because those differ by model. Dataset gaps remain in long-form content, low-resource languages, fine-grained sports, medical and scientific domains, and sign language. Training-data provenance is incomplete for some models, so contamination is audited where possible and marked NA otherwise. Some emotion and social-media-derived labels contain noise and ambiguity, and such splits are flagged for possible future deprecation (Assadi et al., 12 Jun 2026).

In summary, MVEB defines a unified benchmark for evaluating video embeddings across six task families and eight retrieval directions, with explicit support for joint audio-video inputs and modality-restricted leaderboard slices. Its principal findings are that no single architecture dominates, contrastive adaptation is nearly mandatory for cross-modal performance, training-data alignment can outweigh model size for retrieval, and the value of audio depends strongly on whether dataset labels are AV-grounded or V-grounded (Assadi et al., 12 Jun 2026).