MovieNet: Multimodal Movie Dataset

• MovieNet dataset is a large-scale, multimodal resource for comprehensive movie understanding, including full-length films and trailers with detailed annotations.
• It integrates visual, audio, text, and metadata modalities, enabling advanced pretraining and transformer-based fusion for effective genre classification.
• The resource supports diverse tasks such as scene segmentation, character detection, and narrative analysis, driving robust research in video understanding.

The MovieNet dataset is a large-scale, multi-modal resource constructed for comprehensive movie understanding, encompassing both full-length feature films and a vast collection of trailers, with extensive annotations spanning vision, audio, language, and structured metadata. In its extended trailer subset, MovieNet supports state-of-the-art multimodal pretraining and fusion approaches for downstream genre classification and more. This entry details the dataset's structure, multimodal pipelines, feature bank, annotated tasks, model architectures, and benchmark outcomes, as well as access guidelines and research implications, as described by Huang et al. and Sulun et al. (Huang et al., 2020, Sulun et al., 2024).

1. Scope and Composition

MovieNet, as introduced by Huang et al., comprises:

• Full-Length Movies: 1,100 films (≥ 1 hour each), ≈ 3,000 hours of content across 28 genres and 40+ countries.
• Trailers: ≈ 33,000 unique movies with matching YouTube trailer URLs; 26,412 trailers after filtering (height ≥ 300 px, duration 19.6–214.4 s; details below).
• Metadata: 375,000 entries (title, release, country, runtime, rating, cast, plot summaries).
• Photos and Stills: 3.9 million images (posters, production art, etc.).
• Textual Content: English subtitles (manually aligned), scripts (≈ 2,000 aligned), synopses, Wikipedia plot descriptions.
• Audio: Raw 16 kHz waveforms, STFT features for speech/background.
• Annotations: 1.1M+ character bounding boxes/identities, 42,000 scene boundaries, 65,000 place/action tags, 92,000 cinematic style tags, and 4,208 aligned paragraph–segment pairs.
• Dataset Splits: Provided for all main annotations, with standard train/val/test partitions (no overlap at movie level).

The trailer collection, as specifically preprocessed and released by Sulun et al., forms a second, modality-rich subset optimized for pretrained multimodal feature extraction (Sulun et al., 2024).

2. Modality Coverage and Preprocessing

The MovieNet trailer pipeline (26,412 trailers, multi-label on 21 frequent genres) supports the extraction of five principal modalities using automated and reproducible procedures:

• Visual: Keyframes per detected scene (via FFmpeg); mid-scene 224×224 crops; 1–216 shots per trailer.
• Audio Waveform: Re-encoded mono Opus @ 48 kHz, chunked for event/music embedding.
• Speech Transcripts: Continuous ASR by OpenAI Whisper, language-filtered (English or “{Language} language” label).
• On-Screen Text: OCR by PaddleOCR (post-processed via spell correction).
• Music and Audio Event Tags: Music genre CNN embeddings (22 s, no overlap), AudioSet CNN features (3 s windows).

Preprocessing for each modality enforces consistent frame sizes, chunk durations, language filtering, and maximum temporal lengths. The data is aligned across modalities via indexed sequences, ensuring downstream temporal correspondence.

3. Released Pretrained Feature Bank

Sulun et al. provide, for each trailer in the filtered set, a complete suite of frozen features as stored NumPy arrays (≈ 200 GB):

Modality	Encoder (Frozen)	Input/Window	Dimensionality	Typical Temporal Length (99th %-ile)
Visual	CLIP (e.g., ViT-B/32)	224×224 keyframe	$ℝ^{512}$	216 frames/trailer
Audio Events	AudioSet CNN	3 s chunk	$ℝ^{128}$	140 chunks/trailer
Music Genre	MusicNet CNN	22 s chunk	$ℝ^{64}$	18 chunks/trailer
OCR→DistilBERT	PaddleOCR + DistilBERT-base	Text/frame (max 64)	$ℝ^{768}$	up to 64 tokens/frame
ASR→DistilBERT	Whisper + DistilBERT-base	Utterance (segmented)	$ℝ^{768}$	up to 86 segments/trailer

Sequences are stored as truncated or padded .npy arrays indexed by YouTubeID and modality, facilitating efficient batch retrieval for model training or inference. Loader code and documentation are provided for direct PyTorch integration.

4. Transformer-Based Multimodal Fusion for Genre Classification

Sulun et al. implement and benchmark three genre classifier architectures for this multimodal setup:

• MLP (baseline pooling): Temporal average over each modality, then concatenate; a single dense layer maps the 256-dim pooled feature to 21 genre logits (parameters: 57K).
• Single-Transformer (early fusion):
  • Each modality mapped via FC to 256-dim.
  • Learnable <SEP> token and positional embeddings (per-modality).
  • Modalities concatenated time-wise; a <CLS> token prepended.
  • Transformer encoder: $L=2$ layers, $H=8$ heads, $d_{model}=256$, feed-forward inner dim 512, with residual+layer-norm.
  • Output at <CLS> predicts sigmoided genre scores via FC (parameters: ≈8.56M).
• Multi-Transformer (late fusion):
  • Each (non-text) modality: mapped to $d_{model}=128$, prepend <CLS>, 1-layer transformer (8 heads, modality-specific position emb.).
  • Text features: optionally average over time.
  • <CLS> representations/averaged text concatenated; FC to 21 logits (parameters: ≈6.98M).

Loss is weighted multi-label binary cross-entropy with positive-class weight 0.25. Optimizer is Adam (lr=$1 \times 10^{-5}$, batch=32, dropout=0.5, gradient clip norm=1). Hyperparameters are tuned via grid-search on validation mAP. Metrics are macro-averaged [email protected], [email protected], and mean Average Precision (mAP).

5. Benchmarks, Results, and Comparative Analysis

Model performance on the MovieNet trailer test set, as reported in (Sulun et al., 2024):

Model	Precision	Recall	mAP
MovieNet SOTA (8-clip)	0.7974	0.2497	0.4688
Baseline MLP (pretrained avg)	0.7378	0.3217	0.5973
MLP (all pretrained features)	0.8205	0.3351	0.6316
Single-Transformer	0.8100	0.3711	0.6509
Multi-Transformer (best)	0.8200	0.3833	0.6602

The multi-transformer setting achieves a new state-of-the-art for movie trailer genre classification (mAP=66.02%), outperforming prior benchmarks by considerable margin. Per-genre analysis shows superiority in 20 of 21 genres.

Across the full MovieNet benchmark, additional tasks evaluated include cinematic style classification, character detection/identification, scene segmentation, action/place tagging, and segment retrieval. Notable results: Cascade R-CNN achieves 95.17 mAP for character detection; SlowFast achieves 23.52 mAP on action tagging; and graph-based visual-semantic embedding achieves Recall@1 of 21.98% on movie segment retrieval (Huang et al., 2020).

6. Access, Codebase, and Reproducibility

• Pretrained Feature Bank: Publicly available via Zenodo (https://zenodo.org/records/13909366), arranged as modality-wise .npy files per trailer.
• Codebase: Hosted at https://github.com/serkansulun/trailer-genre-classification. Includes feature loaders, model definitions/checkpoints, scripts for training/inference, and package requirements.
• API Usage (inference):

$ℝ^{128}$0

• A plug-and-play script (inference.py) allows users to process arbitrary local video files, automatically extracting features and predicting top-K genres.

Original MovieNet assets—including the 1,100 full-length movies and extended multimodal annotations—remain available (contingent on licensing) at https://movienet.github.io (Huang et al., 2020).

7. Applications and Research Significance

MovieNet's extensive scale, diversity, and multi-level annotations underpin research in:

• Genre classification, style analysis, and semantic retrieval over both images (posters/stills) and long-form video content.
• Mid-level entity tasks: character detection/ID, scene segmentation, action/place recognition, critical for narrative understanding.
• Multimodal/temporal fusion: The pretrained, aligned feature bank enables rapid exploration of cross-modal transformer architectures, late/early fusion, and efficient small-model training, with minimal compute.
• Story-based and high-level video analysis: Multi-level supervision and segment description alignment facilitate research in narrative modeling, synopsis-driven retrieval, and social situation graph construction.

The design demonstrates the critical advantage of large-scale, temporally-aligned, and richly annotated resources for bridging the gap from low-level perception to high-level, story-based long video understanding. The full release of features and code supports rigorous, reproducible experimentation and extension by the research community (Sulun et al., 2024, Huang et al., 2020).