MovieSeq: Multimodal Narrative Video Model

• MovieSeq is a multimodal language model for narrative video understanding that represents videos as interleaved modality blocks including images, plots, subtitles, and history.
• It leverages a decoder-only LLaMA-2 architecture with LoRA tuning and a CLIP ViT-B/16 vision encoder to fuse heterogeneous contextual inputs for free-form query answering.
• Empirical results across datasets demonstrate enhanced character identification, audio description, and plot-based reasoning, underscoring the benefits of contextual interleaving.

MovieSeq is a multimodal LLM for narrative video understanding that represents videos as interleaved multimodal sequences combining images, plots, videos, and subtitles, with optional history and character identity information, and is instruction-tuned to answer free-form queries across heterogeneous video tasks (Lin et al., 2024). It is designed for movies and TV episodes, where understanding depends not only on raw frames but also on characters, dialogues, storylines, relationships, and reasons. In this formulation, a clip is not treated as a single visual stream; instead, it is embedded into a prompt-level context block containing modality-tagged components such as img, vid, sub, plot, and cap, which are then consumed by a decoder-only LLM through causal attention (Lin et al., 2024).

1. Problem setting and conceptual scope

Narrative videos pose demands that differ from conventional short-form action recognition or captioning benchmarks. The central difficulties identified for MovieSeq are character identity resolution across shots, dialogue grounding, long-range storyline dependencies, social relationship inference, and “why/how” reasoning that may require external knowledge rather than frame-local evidence alone (Lin et al., 2024). The model targets these demands by treating contextual sources as first-class inputs rather than auxiliary metadata.

The core idea is to represent a video as an interleaved sequence of modality blocks. These blocks can include uniformly sampled video frames, character photos optionally paired with names, subtitles obtained from Whisper or WhisperX or from dataset annotations, IMDb plot synopsis paragraphs selected by retrieval or temporal heuristics, and prior clips with narrations for history-aware reasoning (Lin et al., 2024). This design generalizes beyond the common pattern in prior multimodal and video LLMs that either accept only a single visual stream or use fixed text-image interleaving mainly for few-shot exemplars.

The prompt schema is standardized as USER: {ctx} {q} followed by MovieSeq: {a} <EOS>, where ctx is the multimodal context block, q is a free-form question, and a is the target answer (Lin et al., 2024). This means the same generative interface is used for video classification, audio description, video-text retrieval, captioning, and question answering, with task differences expressed through context construction and the query itself rather than through task-specific output heads.

A plausible implication is that MovieSeq reframes narrative video understanding as context-conditioned autoregressive generation over heterogeneous evidence sources. The paper’s formulation suggests that the decisive modeling choice is not a new visual backbone, but the explicit packaging of multiple contextual channels into a single instruction-tuned sequence (Lin et al., 2024).

2. Interleaved multimodal representation

MovieSeq defines several canonical context templates. In the “Video with Images” setting, character photos are inserted as img blocks, optionally accompanied by textual names such as “img1 is name1,” enabling queries like “Who can be found in this video? If not, output None” or free-form requests for video description (Lin et al., 2024). In the “Video with Plots” setting, the prompt explicitly conditions on a plot paragraph and the video. In the “Video with Subtitles” setting, the model receives subtitle text aligned to the clip. In the “Video with History” setting, the context contains chronologically ordered past clips and narrations, written as vid_{t−n}, cap_{t−n}, …, vid_t (Lin et al., 2024).

Plot paragraphs are selected by one of two mechanisms. The retrieval-augmented variant encodes sentences with all-mpnet-base-v2 from SentenceTransformers, computes cosine similarity with the query, and returns the top-1 paragraph. The temporal heuristic computes $r=\lfloor t/T \rfloor$ for a clip at timestamp $t$ in a movie of duration $T$, selects the $r$-th sentence, and then extracts a paragraph using a sentence window of size $w$ (Lin et al., 2024). Subtitles are aligned using Whisper or WhisperX, or taken directly from datasets such as CMD and TVC when timestamps are already available.

Temporal structure is handled explicitly at input construction time. Frame sampling differs by dataset: 8 frames per clip for MAD, 32 for CMD, TVC, and MovieQA, and 64 for LVU; MovieNet uses all available frames (Lin et al., 2024). History is ordered from past to present, and temporal relationships in video tokens are modeled by a single-layer transformer with temporal positional encodings.

This interleaving mechanism is intended to let the model associate names with faces, speakers with scenes, and background plot information with current visual evidence. The paper’s examples emphasize identity grounding, dialogue reasoning, plot-based rationale generation, and history-conditioned event continuation, rather than purely perceptual captioning (Lin et al., 2024).

3. Architecture and learning objective

MovieSeq uses LLaMA-2 as a decoder-only LLM and integrates LoRA into all linear layers for efficient fine-tuning; all parameters are frozen except input embedding layers, and LoRA incurs only approximately $0.12\%$ extra trainable parameters (Lin et al., 2024). The vision encoder is CLIP ViT-B/16. For efficiency under multiple inputs and variable-length videos, the model uses frame-wise [CLS] tokens rather than patch tokens. Images are treated as one-frame videos.

The visual projector is a single-layer transformer with temporal positional encoding. It maps per-frame CLIP [CLS] embeddings into a temporally aware token sequence that is fed into the LLM together with text tokens from subtitles, plots, names, and captions (Lin et al., 2024). There is no explicit gated fusion module or Q-Former; cross-modal grounding is performed implicitly by the causal attention mechanism over the concatenated sequence.

Character identity linking is supported by external resources from MovieNet and AutoAD-II. For identity recognition, the paper compares CLIP-based similarity retrieval with a dedicated identification module trained on MovieNet, and reports that the latter is stronger; it is also reused for MAD audio description (Lin et al., 2024). Plot retrieval uses all-mpnet-base-v2 sentence embeddings and cosine similarity.

The training objective is standard auto-regressive causal language modeling over the answer tokens. Let $a=(x_1,\dots,x_L)$ denote the answer sequence, ctx the multimodal context, and $q$ the query. The model maximizes

$$\max \sum_{i=1}^{L} \log p_\theta(x_i \mid ctx, q, a_{<i}),$$

equivalently minimizing

$$L_{CLM}=-\sum_{i=1}^{L}\log p_\theta(x_i \mid ctx, q, a_{<i}).$$

No contrastive or retrieval-specific loss is added, including for retrieval evaluation, which is cast as generation-first (Lin et al., 2024).

The reported training setup is per-dataset: 20 epochs per dataset, learning rate $t$0, LoRA rank 16 with $t$1, subtitles and plots truncated to 512 tokens, and generated responses capped at 64 tokens (Lin et al., 2024).

4. Datasets, tasks, and empirical profile

MovieSeq is evaluated on six datasets spanning five settings: LVU for video classification, Movienet for character identification, MADv2 for audio description generation, CMD for video-text retrieval, TVC for video captioning, and MovieQA for video question-answering (Lin et al., 2024). The evaluation protocol uses task-standard metrics, including top-1 accuracy for LVU, Precision/Recall/F1 for Movienet, ROUGE-L/CIDEr/SPICE and retrieval-style scores for MADv2, Geometric Mean and Recall@$t$2 for CMD, BLEU-4/METEOR/ROUGE-L/CIDEr for TVC, and multiple-choice accuracy for MovieQA.

On LVU, MovieSeq shows strong dependence on modality choice. MovieSeq (vid) reports Relation 61.0, Speak 37.6, Scene 76.8, Avg 58.5; MovieSeq (sub) reports Relation 61.0, Speak 61.0, Scene 62.4, Avg 61.5; and MovieSeq (vid+sub) reports Relation 75.6, Speak 63.0, Scene 65.9, Avg 68.1 (Lin et al., 2024). The result pattern indicates that subtitles substantially improve relation and speaking-style understanding, while they can reduce performance on purely visual scene classification.

On Movienet actor identification, the reported scores are 39.6/71.8/51.0 for CLIP-cosine, 75.9/82.7/79.1 for the TFM Decoder from AutoAD-II, and 88.5/75.5/81.4 for MovieSeq in Precision/Recall/F1 order (Lin et al., 2024). This shows that generative decoding of names can match or exceed specialized discriminative approaches on F1.

On MADv2 audio description, AutoAD-II with AV-AD and WebVid reports ROUGE-L 13.4, CIDEr 19.5, and $t$3 of 50.8, whereas MovieSeq (vid,img,plot,hist) reports ROUGE-L 15.5, CIDEr 24.4, SPICE 7.0, and $t$4 of 51.6 (Lin et al., 2024). On CMD retrieval, MovieSeq (vid) yields Geometric Mean 9.2 with $t$5, MovieSeq (sub) yields 33.1 with 21.2/38.0/44.8, and MovieSeq (vid+sub) yields 38.9 with 25.8/45.3/50.3 (Lin et al., 2024). On TVC, MovieSeq (vid+sub+img) reports BLEU-4 17.9, ROUGE-L 38.1, METEOR 19.9, and CIDEr 64.8, exceeding MovieSeq (vid) and MovieSeq (sub) (Lin et al., 2024).

MovieQA provides the clearest demonstration of external knowledge dependence. MovieSeq (vid) reports 27.32 accuracy, MovieSeq (subs) 27.10, MovieSeq (plot) 63.54, MovieSeq (vid+subs) 48.32, and MovieSeq (vid+plot) 66.41 (Lin et al., 2024). The data indicate that plot snippets are central for “how/why” reasoning in narrative video question answering.

5. Ablations and interpretive findings

The ablation studies isolate which context types matter most. On MAD, video-only description reports CIDEr 11.4 and Rep@4 1.33; adding plot raises CIDEr to 16.9 and reduces Rep@4 to 0.48; adding oracle history produces CIDEr 11.8 and Rep@4 0.11; adding characters via CLIP similarity yields CIDEr 19.7 and Rep@4 1.30; adding characters via a MovieNet-trained identification module yields CIDEr 23.6 and Rep@4 0.97; and the full MovieSeq (vid,img,plot,hist) reaches CIDEr 24.4 with Rep@4 0.12 (Lin et al., 2024). These numbers indicate that different context sources improve different aspects of output quality: plots improve content relevance, history reduces repetition, and stronger identity resolution improves named descriptions.

History length is also explicitly studied. With more past clips, the vid+cap template consistently improves CIDEr from 11.4 to 28.5 as the number of clips increases from 0 to 8, whereas cap alone improves moderately from 11.4 to 18.9 and vid alone changes little (Lin et al., 2024). This suggests that past narrations are a primary carrier of long-range event dependency, while visual history without narration is comparatively weak for causal continuity.

For plot augmentation, the estimated temporal window outperforms naive clip-similarity filtering, and increasing plot context up to 512 tokens steadily improves CIDEr to 17.6 on the reported MAD analysis (Lin et al., 2024). Multi-task co-training over balanced development sets from MAD, Movienet, and TVC improves the average from 53.8 under individual training to 55.2 under co-training, a gain of 1.5 (Lin et al., 2024).

Qualitative analysis in the paper aligns with these quantitative trends. Identity grounding is shown by inserting top-10 character photos and generating descriptions that mention names; dialogue reasoning is illustrated by using subtitles to infer relationships such as friends versus couple; plot reasoning is exemplified through a MovieQA case where the retrieved paragraph “Miley attends the birthday party as Hannah Montana” supports the rationale; and history-driven inference is demonstrated by conditioning on the earlier narration “He opens the trunk and takes out a flashlight” and predicting the subsequent action “he shines the light” (Lin et al., 2024).

6. Limitations, related work, and terminological distinctions

The stated limitations are primarily architectural and contextual. MovieSeq introduces no novel visual architecture and uses [CLS] tokens instead of patch tokens to save memory, which constrains spatial detail (Lin et al., 2024). Long-context modeling is limited by LLM context length, making full-length movies difficult to process without sacrificing fidelity. The method also depends on external annotation quality and preprocessing quality: Whisper errors, face-identification errors, or weakly derivable supervision can degrade grounding and increase hallucination (Lin et al., 2024).

The model is positioned against several neighboring lines of work. Relative to single-visual-input multimodal models such as BLIP-2, LLaVA, MiniGPT-4, Video-Chat, and Video-ChatGPT, MovieSeq generalizes the input to flexible interleaved multi-source contexts (Lin et al., 2024). Relative to interleaved in-context models such as Flamingo, Otter, and COSMO, it focuses on narrative video and on converting public datasets into multimodal instructions spanning images, subtitles, plots, and history. Relative to long-video models such as MovieChat, LLaMA-VID, and MA-LMM, its emphasis is rich contextual interleaving rather than only memory-efficient long-video processing (Lin et al., 2024).

The term “MovieSeq” also has nearby but distinct usages in the provided literature. The 2018 paper “Convolutional Collaborative Filter Network for Video Based Recommendation Systems” does not use the term “MovieSeq,” but it operationalizes sequence-centric trailer analysis by treating a trailer as a multi-channel time series and learning temporal convolutional filters that detect object and scene sequences predictive of attendance (Hsieh et al., 2018). By contrast, the 2025 paper “MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second” is relevant only if “MovieSeq” is interpreted broadly as movie or video sequences; it explicitly states that it does not cite or use any dataset or method named “MovieSeq” (Lin et al., 14 Jul 2025). A further source of ambiguity is Motion Sequencing, abbreviated MoSeq, a framework for decomposing spontaneous mouse behavior into behavioral “syllables”; the data note that it is sometimes referred to as “MovieSeq” in informal contexts, but it is a separate 3D depth-video and unsupervised modeling framework rather than the narrative-video multimodal LLM (Lin et al., 2022).

In that broader landscape, MovieSeq denotes a specific approach to narrative video understanding: instruction-tuned generation over interleaved multimodal context blocks, with empirical gains concentrated in settings where subtitles, identity cues, plot metadata, and temporal history are essential for disambiguation and reasoning (Lin et al., 2024).