M3L-10M: Multimodal Enrichment of MovieLens-10M

• The paper introduces M3L-10M, a reproducible dataset that enriches MovieLens-10M with text, image, audio, and video modalities to enhance recommender systems.
• It details a fully documented pipeline that maps movies to TMDB, retrieves diverse media, and applies state-of-the-art encoders with modality-specific preprocessing.
• It supports research in cross-modal fusion, cold-start recommendations, and modality attention while balancing comprehensive coverage with practical retrieval constraints.

Searching arXiv for the primary paper and closely related multimodal recommender-system context. M3L-10M is a large-scale, reproducible, multimodal dataset for movie recommendation obtained by enriching MovieLens-10M with textual, visual, acoustic, and video side information (Spillo et al., 17 Feb 2026). It is described as the first fully reproducible, large-scale multimodal enrichment of MovieLens-10M, combining user–item interactions with four orthogonal side-information streams: text, image, audio, and video. The dataset is built through a fully documented pipeline that collects movie plots, posters, and trailers, extracts features with several state-of-the-art encoders, and releases mappings to download the original raw data, the extracted features, and the complete datasets in multiple formats. Its stated purpose is to support reproducibility and replicability in large-scale multimodal movie recommendation research.

1. Origin and scope

M3L-10M starts from the original ML-10M CSV, which contains 69,878 users, 10,677 movies, and 10M+ ratings (Spillo et al., 17 Feb 2026). The enrichment process links each movie to a TMDB identifier and uses the TMDB REST API to retrieve plot overviews, high-resolution poster URLs, and YouTube trailer URLs. When trailers had become private or geo-restricted on YouTube, the pipeline falls back to the original GroupLens-provided YouTube link; movies for which no trailer could be downloaded by either route were dropped.

The dataset is positioned within the literature on Multimodal Recommender Systems, where the paper identifies a gap between growing interest in MRSs and the scarcity of large-scale datasets built through documented, publicly reproducible procedures. In that context, M3L-10M is intended as a reproducible multimodal enrichment rather than a newly collected interaction corpus. This suggests that its primary contribution lies in systematic augmentation and standardization of side information around an already established collaborative filtering benchmark.

2. Data acquisition and preprocessing workflow

The documented mapping strategy proceeds from movies.csv in ML-10M to TMDB metadata. For each movieId, the corresponding TMDB ID is looked up from the MovieLens metadata, after which the pipeline queries the TMDB /movie/{tmdb_id} and /movie/{tmdb_id}/videos endpoints to harvest plot, poster_path, and trailer video IDs (Spillo et al., 17 Feb 2026). Posters are fetched via HTTP GET and stored temporarily as JPEG or PNG files, while trailers are downloaded with yt_dlp at up to 480p and stored temporarily as MP4.

Preprocessing is modality-specific. TMDB plot passages are stripped of HTML entities, lower-cased, and normalized to Unicode NFKC. No aggressive sub-sampling or core-filtering is applied. Poster images are resized to $224 \times 224$ pixels, center-cropped, and normalized using ImageNet mean/std. Audio is extracted from the first 30 s of each trailer, converted to mono, and resampled to 16 kHz; for AST, log-mel spectrograms are computed with 128 mel bins and 25 ms frames. Video trailers are uniformly sampled at 8 fps and center-cropped to $224 \times 224$ pixels.

A recurrent misconception would be to treat M3L-10M as a redistribution of movie media. It is not. After encoding, the raw files are discarded, and only non-invertible embeddings are released, explicitly in compliance with non-consumptive “fair use” research guidelines. This constraint is central to the dataset’s design because it shapes both reproducibility and the limits of downstream reprocessing.

3. Multimodal representation design

To produce a stable, comparable feature set, each modality is encoded with widely adopted, pre-trained backbones (Spillo et al., 17 Feb 2026). The paper emphasizes fixed-length embeddings for all modalities.

Modality	Encoders	Output dimensionality
Text	MiniLM, MPNet, CLIP-Text	384, 768, 512
Image	VGG16, ViT-base-patch16-224, CLIP-Image	4,096, 768, 512
Video	SlowFast R-50, R(2+1)D, MViT	$\approx 2{,}304$, 512, 768
Audio	VGGish, Whisper-base encoder, AST	128, 512, 768

For textual streams, the encoders are all-MiniLM-L6-v2, all-mpnet-base-v2, and the ViT-B/32 text encoder from CLIP. Transformer attention is given as

$$A = \mathrm{softmax}(QK^\top / \sqrt{d_k}), \quad Z = AV,$$

where $Q, K, V \in \mathbb{R}^{n \times d}$ are the query, key, and value matrices obtained from the last hidden layer. The final embedding is the [CLS] token,

$h = z_0 \in \mathbb{R}^d.$

For posters, the visual encoders are VGG16, ViT-base-patch16-224, and CLIP-Image. In the VGG16 case, after five conv–pool blocks, two 4,096-unit fully connected layers produce the final feature:

$$h = \mathrm{ReLU}(W_2(\mathrm{ReLU}(W_1 f))) \in \mathbb{R}^{4096}.$$

For trailers, video features are extracted with SlowFast R-50, R(2+1)D, and MViT. The SlowFast representation is summarized by a two-pathway formulation: if $x_{\text{slow}}$ and $x_{\text{fast}}$ are pathway outputs, the joint embedding is

$$h = \mathrm{Pool}(W_s x_s + W_f x_f) \in \mathbb{R}^{2304}.$$

For trailer audio, the encoders are VGGish, the Whisper-base encoder, and AST. The AST formulation treats each mel patch $224 \times 224$0 as a token and applies self-attention to yield

$224 \times 224$1

The coexistence of multiple encoders per modality is methodologically important. A plausible implication is that M3L-10M supports not only multimodal fusion across streams, but also intra-modality comparison across representational families such as CNNs, ViTs, CLIP-based encoders, and transformer-based audio models.

4. Composition, coverage, and storage format

After filtering to movies for which all four modalities were successfully embedded, M3L-10M comprises 69,878 users, 9,031 items, and 9,409,884 ratings, while preserving the original temporal span from January 1995 to November 2009 (Spillo et al., 17 Feb 2026). These figures correspond to 100% of ML-10M users, 84.6% of ML-10M movies, and 94.1% of original ML-10M interactions. Sparsity remains approximately 98.5%.

This coverage profile is a direct consequence of the requirement that all four modalities be available. The retained interaction mass is high relative to the removed catalog portion, but the filtering does exclude about 1.6K movies because all-modality completeness is enforced. This suggests a deliberate trade-off between modality consistency and item coverage.

On disk, the dataset is organized into an interaction file, four modality-specific embedding archives, and a modality mapping file:

File	Contents	Notes
interactions.csv	userId, movieId, rating, timestamp	Interaction data
text_embeddings.npz, image_embeddings.npz, audio_embeddings.npz, video_embeddings.npz	movieIds[9031] and embedding matrices of shape [9031 \times d] for each encoder	Feature archives
modality_map.json	Links encoder names to matrix keys	Lookup metadata

The paper provides an explicit example of feature retrieval: to retrieve the MPNet text embedding for movieId = 356, one loads text_embeddings.npz['mpnet'] [ index_of(356), :]. This file structure makes the alignment between item identifiers and encoder-specific matrices explicit, which is essential for reproducible benchmarking and ablation studies.

5. Recommended benchmark protocol and metrics

The recommended protocol is an 80/10/10 train/validation/test split over interactions, stratified by user so that each user appears in all splits (Spillo et al., 17 Feb 2026). A leave-one-out temporal variant is also supported, in which each user’s last rating is held out for test and the penultimate rating for validation.

The standard ranking metrics are defined as

$224 \times 224$2

$224 \times 224$3

$224 \times 224$4

$224 \times 224$5

where $224 \times 224$6 indicates whether the $224 \times 224$7-th recommended item is relevant, $224 \times 224$8 is the total relevant items for a user, and $224 \times 224$9 is a user-normalization constant for NDCG.

These evaluation choices situate M3L-10M within standard top-$\approx 2{,}304$0 ranking practice while also accommodating temporally ordered testing. The availability of both a stratified random split and a leave-one-out temporal variant is relevant because the former emphasizes balanced evaluation across users, whereas the latter is more aligned with sequential recommendation settings.

6. Research affordances and constraints

The paper identifies several research uses enabled by M3L-10M (Spillo et al., 17 Feb 2026). It supports cross-modal fusion models, including graph neural nets ingesting audio, video, and text. It supports cold-start and long-tail investigations because movies missing user feedback can still be profiled via embeddings. It supports hybrid CF + content pipelines that dynamically weight semantic versus stylistic signals. It also supports research on modality attention, including analysis of the circumstances under which acoustic cues from VGGish or Whisper outweigh plot semantics.

These affordances derive from the dataset’s combination of collaborative interactions with four complementary embedding types. The paper summarizes these modalities as reflecting “what” the story is through text, “how” it looks through image and video, and “how” it sounds through audio. That formulation is descriptive rather than theoretical, but it captures the intended partition of semantic and stylistic signals across modalities.

The limitations are equally explicit. Approximately 15% of original trailers were unrecoverable, so around 1.6K movies were filtered out. TMDB plots occasionally contain automated translations or laconic overviews, introducing noise. Poster aspect ratios vary widely, and although posters are resized, some stylistic context may be lost. Raw media are not redistributed; only embeddings are provided, which precludes re-encoding with future models.

A common misunderstanding would be to view these limitations as incidental implementation details. In practice, they shape the benchmark’s epistemic boundaries: item retention depends on trailer recoverability, text quality depends on TMDB plot quality, and future representation updates are constrained by the absence of raw media. Conversely, the paper frames the dataset as a foundational step toward reproducibility and replicability in large-scale multimodal movie recommendation, rather than as a final or exhaustive solution.