ViLLA-MMBench: A Multimodal Movie Recommender Benchmark

• ViLLA-MMBench is a unified, reproducible benchmark suite for LLM-augmented multimodal movie recommendation that integrates audio, visual, and textual modalities.
• It offers a configurable experimental pipeline with interchangeable fusion strategies and recommender backbones for systematic ablation of modality and augmentation effects.
• Empirical findings reveal modality-specific impacts on cold-start, coverage, and beyond-accuracy metrics, providing actionable insights for multimodal recommendation research.

ViLLA-MMBench is a reproducible, extensible benchmark for LLM-augmented multimodal movie recommendation. It is built on MovieLens and MMTF-14K, aligns dense item embeddings from three modalities—audio, visual, and text—and supports configurable text augmentation, interchangeable fusion strategies, and multiple recommendation backbones for systematic ablation. Its stated purpose is to benchmark joint modeling of visual, audio, and textual modalities for long-form video recommendation under both accuracy and beyond-accuracy criteria, including cold-start behavior and coverage (Nazary et al., 6 Aug 2025).

1. Scope and benchmark definition

ViLLA-MMBench is defined as a unified benchmark suite for multimodal movie recommendation in which textual, auditory, and visual side information are brought into a single experimental framework. The benchmark is centered on recommendation rather than on generic multimodal representation learning: it combines side-information encoders, fusion operators, and collaborative-filtering backbones within a single evaluation pipeline. The benchmark is also explicitly designed for reproducibility. Experiments are fully declarative via a single YAML file, and the released artifacts include code, precomputed embeddings, YAML configs, Colab notebooks, versioned embeddings, and logs (Nazary et al., 6 Aug 2025).

The suite addresses a specific gap identified in the underlying work: most benchmarks address only raw features or narrow fusion, whereas recommending long-form video content demands joint modeling of visual, audio, and textual modalities. In this setting, ViLLA-MMBench is not merely a dataset wrapper; it is a benchmarked pipeline with configurable augmentation, modality selection, fusion, model choice, split strategy, and hyperparameter search. Systematic benchmarking is used to identify universal versus backbone- or metric-specific combinations, rather than assuming that one multimodal recipe dominates across all recommenders and all evaluation targets (Nazary et al., 6 Aug 2025).

2. Data alignment and modality representations

The benchmark aligns MovieLens-1M with MMTF-14K. After intersection of raw logs from MovieLens-1M, which contain 1,000,209 ratings, with the 14,000+ trailer-based items of MMTF-14K, and after k-core filtering and metadata alignment, the resulting benchmark contains $|U| = 6{,}040$ users, $|I| = 992$ items, and $|R| = 632{,}397$ interactions, with sparsity $\simeq 0.1055\%$ (Nazary et al., 6 Aug 2025).

The side information is organized into three modalities.

Modality	Representation	Dimensionality
Audio	block-level features (blf)	$d_a \approx 128$
Audio	i-vectors (i_vec)	$20$
Visual	CNN features (cnn)	$2048$
Visual	aesthetic visual features (avf)	$d_v \approx 512$
Text	$\Phi_{mdl}(T_{NA/A}(i))$	encoder-dependent

For audio, the benchmark includes block-level features and i-vectors. The block-level features are low-level descriptors such as MFCC, spectral, and energy features extracted via openSMILE and pooled per 6 s block, yielding vectors $$e_i^{(\mathrm{aud},\mathrm{blf})} \in \mathbb{R}^{d_a}$$ with $|I| = 992$0. The i-vector representation is a 20-dimensional global audio fingerprint extracted per trailer and denoted $|I| = 992$1 (Nazary et al., 6 Aug 2025).

For vision, trailer frames are decoded at 1 FPS. CNN features are obtained by passing each frame through a pre-trained ResNet-50 or InceptionV3 and averaging the 2048-dimensional outputs over $|I| = 992$2 frames:

$|I| = 992$3

A separate aesthetic prediction network, for example a fine-tuned VGG-16, provides aesthetic visual features with $|I| = 992$4 dimensions per frame, likewise mean-pooled (Nazary et al., 6 Aug 2025).

3. Text augmentation and embedding encoders

ViLLA-MMBench distinguishes raw textual metadata from LLM-augmented text. In the raw setting, denoted NA mode, the text string is constructed as

$|I| = 992$5

In the augmented setting, denoted A mode, each movie is prompted using the template:

“Role: You are a helpful assistant. Task: Write a vivid, engaging 100–150-word synopsis for a movie. Inputs: Title: […], Genre List: […], Tags: […]”

The generated output $|I| = 992$6 is stored verbatim. Missing or sparse metadata is thereby automatically enriched using state-of-the-art LLMs, with the benchmark abstract naming OpenAI Ada as an example (Nazary et al., 6 Aug 2025).

Text embeddings are then produced by a configurable encoder $|I| = 992$7:

$|I| = 992$8

The benchmark provides three encoder families. OpenAI Ada uses $|I| = 992$9 with no additional projection. Sentence-T5, abbreviated ST, uses $|R| = 632{,}397$0. LLaMA-2 uses $|R| = 632{,}397$1 and may be projected to 1,536 dimensions via a learned linear layer when used with MF-style models. All modality vectors are $|R| = 632{,}397$2-normalized before fusion (Nazary et al., 6 Aug 2025).

The benchmark therefore treats text not only as metadata but as a controlled experimental variable. The central comparison is between raw metadata text and LLM-generated synopsis text, with the encoder choice itself being another axis of variation. This design makes it possible to isolate the effect of augmentation, the effect of embedding quality, and the interaction between text and the audio-visual channels.

4. Fusion operators and recommendation backbones

ViLLA-MMBench supports early-, mid-, and late-fusion.

In early fusion, the benchmark includes concatenation, PCA, and CCA. Concatenation is defined as

$|R| = 632{,}397$3

For $|R| = 632{,}397$4, the concatenated vector is z-score standardized and projected onto the smallest $|R| = 632{,}397$5 principal components satisfying

$|R| = 632{,}397$6

For $|R| = 632{,}397$7, the concatenated vector is split into two halves, projection matrices $|R| = 632{,}397$8 are learned to maximize $|R| = 632{,}397$9, and the first $\simeq 0.1055\%$0 canonical dimensions are retained:

$\simeq 0.1055\%$1

In mid-fusion, the AMR model uses an attention-gating network $\simeq 0.1055\%$2 that attends separately to each modality vector. In late fusion, multiple recommenders can be fused by rank aggregation, including Borda count, weighted Borda, average rank, and reciprocal-rank fusion (RRF), with RRF defined by

$\simeq 0.1055\%$3

(Nazary et al., 6 Aug 2025)

The benchmark evaluates multiple recommendation algorithms. Matrix Factorization (MF) uses

$\simeq 0.1055\%$4

with optimization objective

$\simeq 0.1055\%$5

VAECF uses the ELBO

$\simeq 0.1055\%$6

VBPR defines

$\simeq 0.1055\%$7

where $\simeq 0.1055\%$8 is fused side information and optimization uses BPR pairwise hinge loss. VMF sets

$\simeq 0.1055\%$9

and

$d_a \approx 128$0

AMR uses

$d_a \approx 128$1

where $d_a \approx 128$2 is a small attention-gating MLP that learns modality weights. All models are trained via SGD or Adam, with GPU-aware hyperparameter grid search (Nazary et al., 6 Aug 2025).

This combination of interchangeable fusion and interchangeable backbone is a defining structural feature of the benchmark. It allows ablation over representation, fusion geometry, and recommendation model within one controlled environment.

5. Experimental protocol and evaluation criteria

The full experimental workflow is controlled by a single config.yml. The configurable fields include dataset choice (movielens_100k|1m), split strategy (random|temporal|per_user), k_core, simulate_cold_start, LLM choice (openai|st|llama), augmentation (true|false), audio variant (blf|i_vec), visual variant (cnn|avf), fusion (concat|pca_ρ|cca_k), model (cf|vbpr|amr|vmf|vaecf), seed, epochs, use_gpu, fast_prototype, and parallel_hpo. The reported compute requirement is 2–6 h on a single Colab GPU (Tesla T4), or local multi-GPU via configuration (Nazary et al., 6 Aug 2025).

The benchmark includes three split strategies. The random split is a uniform train/test hold-out, for example 80/20. The temporal split leaves the latest interactions for test. The per_user split holds out $d_a \approx 128$3 interactions per user. Example hyperparameter ranges include latent dimension $d_a \approx 128$4, $d_a \approx 128$5, and $d_a \approx 128$6 (Nazary et al., 6 Aug 2025).

Evaluation spans both accuracy and beyond-accuracy metrics. Accuracy uses Recall@K and nDCG@K:

$d_a \approx 128$7

and

$d_a \approx 128$8

with

$d_a \approx 128$9

Beyond-accuracy metrics are defined on the top-$20$0 recommendation list $20$1 for each user. Cold-start rate@K is

$20$2

coverage@K is

$20$3

novelty@K is

$20$4

where $20$5, intra-list diversity@K is

$20$6

and calibration bias is

$20$7

The abstract summarizes these beyond-accuracy criteria as cold-start rate, coverage, novelty, diversity, and fairness; the detailed formulation specifies calibration bias through the genre-distribution discrepancy $20$8 (Nazary et al., 6 Aug 2025).

6. Empirical findings

The reported results emphasize both augmentation effects and interaction effects between modality combinations, fusion schemes, and backbones. Under RQ1, Sentence-T5 yields universal gains: Recall@10 increases by $20$9 for VBPR and $2048$0 for AMR, while nDCG@10 increases by $2048$1 for VBPR and $2048$2 for AMR. OpenAI-Ada gives $2048$3 Recall and $2048$4 nDCG in VBPR, but is mixed in AMR with a slight nDCG decrease. LLaMA-2 is neutral on VBPR, but gives large gains on AMR, with $2048$5 Recall and $2048$6 nDCG (Nazary et al., 6 Aug 2025).

Modality ablations show that the best combination depends on both the target metric and the backbone. For AMR cold-start, the best configuration is [Raw OpenAI text + CNN + blf] (CCA), giving ColdRate@10 AUC=0.8576 versus pure text AUC=0.7330. For AMR coverage, the best configuration is [Raw OpenAI + AVF + i-vec] (CCA), giving Coverage@10 AUC=0.7452. For VBPR cold-start, Augmented ST only is best, with AUC=0.8620, and adding vision/audio hurts cold performance. For VBPR coverage, [Aug OpenAI + AVF + i-vec] (CCA) is best, with AUC=0.7051 (Nazary et al., 6 Aug 2025).

The benchmark also reports cross-setting regularities. Strong LLM text embeddings, specifically OpenAI and ST, are described as universally beneficial across models and metrics. AVF plus i-vec is described as a nearly universal booster for coverage. CNN plus blf helps AMR cold-start but harms VBPR. In projection comparison, CCA-40 consistently outperforms PCA-95 for AMR in both cold-start, with ΔAUC +0.37, and coverage, with ΔAUC +0.0096. For VBPR, PCA wins for cold-start, whereas CCA wins for coverage. Late model-based fusion by rank aggregation, specifically RRF and Borda over VAECF+AMR variants, improves Recall@10 and nDCG@10 over VAECF alone by approximately +0.3–0.5 pts (Nazary et al., 6 Aug 2025).

These findings support the benchmark’s emphasis on systematic benchmarking rather than single-model claims. The reported results do not treat multimodality as uniformly helpful; instead, they separate universal gains, nearly universal boosters, and backbone-specific or metric-specific effects.

7. Relation to MMBench and benchmark significance

MMBench is a benchmark suite for end-to-end multi-modal DNN workloads with system and architecture level analysis, whereas ViLLA-MMBench is a benchmark suite for LLM-augmented multimodal movie recommendation with recommendation-centric metrics and recommender backbones. MMBench emphasizes clear multi-stage execution, frequent synchronization, high heterogeneity, and hardware/software implications; ViLLA-MMBench emphasizes dataset alignment, modality embeddings, fusion ablations, cold-start behavior, coverage, novelty, diversity, and calibration-oriented evaluation (Xu et al., 2022).

The shared naming indicates a common benchmark orientation but not a common evaluation layer. A plausible implication is that ViLLA-MMBench carries the benchmark-suite logic of controlled multimodal comparison into recommender systems, where the central experimental questions concern side-information quality, fusion choice, and backbone dependence rather than synchronization overhead or microarchitectural counters. This distinction is important because the name can invite confusion with the earlier MMBench line of system-and-architecture benchmarking; in published form, the two suites target different problem formulations and different measurement regimes (Nazary et al., 6 Aug 2025).

The benchmark’s released implementation reflects that orientation. The codebase is distributed at https://recsys-lab.github.io/ViLLA-MMBench; installation is available via pip (villa-mmbench); execution uses $2048$7 and the default configs reproduce all RQ1–RQ3 results. In that sense, ViLLA-MMBench functions both as an empirical benchmark and as a standardized experimental substrate for fair multimodal recommendation research (Nazary et al., 6 Aug 2025).