---
title: 'MovieSeq: Multimodal Narrative Video Model'
url: https://www.emergentmind.com/topics/movieseq
type: topic
---

# MovieSeq: Multimodal Narrative Video Model

MovieSeq is a multimodal language model for narrative video understanding that represents videos as interleaved multimodal sequences combining images, plots, videos, and subtitles, with optional history and character identity information, and is instruction-tuned to answer free-form queries across heterogeneous video tasks [2407.21757]. It is designed for movies and TV episodes, where understanding depends not only on raw frames but also on characters, dialogues, storylines, relationships, and reasons. In this formulation, a clip is not treated as a single visual stream; instead, it is embedded into a prompt-level context block containing modality-tagged components such as `img`, `vid`, `sub`, `plot`, and `cap`, which are then consumed by a decoder-only language model through causal attention [2407.21757].

## 1. Problem setting and conceptual scope

Narrative videos pose demands that differ from conventional short-form action recognition or captioning benchmarks. The central difficulties identified for MovieSeq are character identity resolution across shots, dialogue grounding, long-range storyline dependencies, social relationship inference, and “why/how” reasoning that may require external knowledge rather than frame-local evidence alone [2407.21757]. The model targets these demands by treating contextual sources as first-class inputs rather than auxiliary metadata.

The core idea is to represent a video as an interleaved sequence of modality blocks. These blocks can include uniformly sampled video frames, character photos optionally paired with names, subtitles obtained from Whisper or WhisperX or from dataset annotations, IMDb plot synopsis paragraphs selected by retrieval or temporal heuristics, and prior clips with narrations for history-aware reasoning [2407.21757]. This design generalizes beyond the common pattern in prior multimodal and video large language models that either accept only a single visual stream or use fixed text-image interleaving mainly for few-shot exemplars.

The prompt schema is standardized as `USER: {ctx} {q}` followed by `MovieSeq: {a} <EOS>`, where `ctx` is the multimodal context block, `q` is a free-form question, and `a` is the target answer [2407.21757]. This means the same generative interface is used for video classification, audio description, video-text retrieval, captioning, and question answering, with task differences expressed through context construction and the query itself rather than through task-specific output heads.

A plausible implication is that MovieSeq reframes narrative video understanding as context-conditioned autoregressive generation over heterogeneous evidence sources. The paper’s formulation suggests that the decisive modeling choice is not a new visual backbone, but the explicit packaging of multiple contextual channels into a single instruction-tuned sequence [2407.21757].

## 2. Interleaved multimodal representation

MovieSeq defines several canonical context templates. In the “Video with Images” setting, character photos are inserted as `img` blocks, optionally accompanied by textual names such as “img1 is name1,” enabling queries like “Who can be found in this video? If not, output None” or free-form requests for video description [2407.21757]. In the “Video with Plots” setting, the prompt explicitly conditions on a plot paragraph and the video. In the “Video with Subtitles” setting, the model receives subtitle text aligned to the clip. In the “Video with History” setting, the context contains chronologically ordered past clips and narrations, written as `vid_{t−n}, cap_{t−n}, …, vid_t` [2407.21757].

Plot paragraphs are selected by one of two mechanisms. The retrieval-augmented variant encodes sentences with `all-mpnet-base-v2` from SentenceTransformers, computes cosine similarity with the query, and returns the top-1 paragraph. The temporal heuristic computes $r=\lfloor t/T \rfloor$ for a clip at timestamp $t$ in a movie of duration $T$, selects the $r$-th sentence, and then extracts a paragraph using a sentence window of size $w$ [2407.21757]. Subtitles are aligned using Whisper or WhisperX, or taken directly from datasets such as CMD and TVC when timestamps are already available.

Temporal structure is handled explicitly at input construction time. Frame sampling differs by dataset: 8 frames per clip for MAD, 32 for CMD, TVC, and MovieQA, and 64 for LVU; MovieNet uses all available frames [2407.21757]. History is ordered from past to present, and temporal relationships in video tokens are modeled by a single-layer transformer with temporal positional encodings.

This interleaving mechanism is intended to let the model associate names with faces, speakers with scenes, and background plot information with current visual evidence. The paper’s examples emphasize identity grounding, dialogue reasoning, plot-based rationale generation, and history-conditioned event continuation, rather than purely perceptual captioning [2407.21757].

## 3. Architecture and learning objective

MovieSeq uses LLaMA-2 as a decoder-only language model and integrates LoRA into all linear layers for efficient fine-tuning; all parameters are frozen except input embedding layers, and LoRA incurs only approximately $0.12\%$ extra trainable parameters [2407.21757]. The vision encoder is CLIP ViT-B/16. For efficiency under multiple inputs and variable-length videos, the model uses frame-wise `[CLS]` tokens rather than patch tokens. Images are treated as one-frame videos.

The visual projector is a single-layer transformer with temporal positional encoding. It maps per-frame CLIP `[CLS]` embeddings into a temporally aware token sequence that is fed into the language model together with text tokens from subtitles, plots, names, and captions [2407.21757]. There is no explicit gated fusion module or Q-Former; cross-modal grounding is performed implicitly by the causal attention mechanism over the concatenated sequence.

Character identity linking is supported by external resources from MovieNet and AutoAD-II. For identity recognition, the paper compares CLIP-based similarity retrieval with a dedicated identification module trained on MovieNet, and reports that the latter is stronger; it is also reused for MAD audio description [2407.21757]. Plot retrieval uses `all-mpnet-base-v2` sentence embeddings and cosine similarity.

The training objective is standard auto-regressive causal language modeling over the answer tokens. Let $a=(x_1,\dots,x_L)$ denote the answer sequence, `ctx` the multimodal context, and $q$ the query. The model maximizes
$$
\max \sum_{i=1}^{L} \log p_\theta(x_i \mid ctx, q, a_{<i}),
$$
equivalently minimizing
$$
L_{CLM}=-\sum_{i=1}^{L}\log p_\theta(x_i \mid ctx, q, a_{<i}).
$$
No contrastive or retrieval-specific loss is added, including for retrieval evaluation, which is cast as generation-first [2407.21757].

The reported training setup is per-dataset: 20 epochs per dataset, learning rate $3\times10^{-5}$, LoRA rank 16 with $\alpha=16$, subtitles and plots truncated to 512 tokens, and generated responses capped at 64 tokens [2407.21757].

## 4. Datasets, tasks, and empirical profile

MovieSeq is evaluated on six datasets spanning five settings: LVU for video classification, Movienet for character identification, MADv2 for audio description generation, CMD for video-text retrieval, TVC for video captioning, and MovieQA for video question-answering [2407.21757]. The evaluation protocol uses task-standard metrics, including top-1 accuracy for LVU, Precision/Recall/F1 for Movienet, ROUGE-L/CIDEr/SPICE and retrieval-style scores for MADv2, Geometric Mean and Recall@$k$ for CMD, BLEU-4/METEOR/ROUGE-L/CIDEr for TVC, and multiple-choice accuracy for MovieQA.

On LVU, MovieSeq shows strong dependence on modality choice. `MovieSeq (vid)` reports Relation 61.0, Speak 37.6, Scene 76.8, Avg 58.5; `MovieSeq (sub)` reports Relation 61.0, Speak 61.0, Scene 62.4, Avg 61.5; and `MovieSeq (vid+sub)` reports Relation 75.6, Speak 63.0, Scene 65.9, Avg 68.1 [2407.21757]. The result pattern indicates that subtitles substantially improve relation and speaking-style understanding, while they can reduce performance on purely visual scene classification.

On Movienet actor identification, the reported scores are 39.6/71.8/51.0 for CLIP-cosine, 75.9/82.7/79.1 for the TFM Decoder from AutoAD-II, and 88.5/75.5/81.4 for MovieSeq in Precision/Recall/F1 order [2407.21757]. This shows that generative decoding of names can match or exceed specialized discriminative approaches on F1.

On MADv2 audio description, AutoAD-II with AV-AD and WebVid reports ROUGE-L 13.4, CIDEr 19.5, and $R@5/16$ of 50.8, whereas `MovieSeq (vid,img,plot,hist)` reports ROUGE-L 15.5, CIDEr 24.4, SPICE 7.0, and $R@5/16$ of 51.6 [2407.21757]. On CMD retrieval, `MovieSeq (vid)` yields Geometric Mean 9.2 with $R@1/5/10=4.3/11.8/14.9$, `MovieSeq (sub)` yields 33.1 with 21.2/38.0/44.8, and `MovieSeq (vid+sub)` yields 38.9 with 25.8/45.3/50.3 [2407.21757]. On TVC, `MovieSeq (vid+sub+img)` reports BLEU-4 17.9, ROUGE-L 38.1, METEOR 19.9, and CIDEr 64.8, exceeding `MovieSeq (vid)` and `MovieSeq (sub)` [2407.21757].

MovieQA provides the clearest demonstration of external knowledge dependence. `MovieSeq (vid)` reports 27.32 accuracy, `MovieSeq (subs)` 27.10, `MovieSeq (plot)` 63.54, `MovieSeq (vid+subs)` 48.32, and `MovieSeq (vid+plot)` 66.41 [2407.21757]. The data indicate that plot snippets are central for “how/why” reasoning in narrative video question answering.

## 5. Ablations and interpretive findings

The ablation studies isolate which context types matter most. On MAD, video-only description reports CIDEr 11.4 and Rep@4 1.33; adding plot raises CIDEr to 16.9 and reduces Rep@4 to 0.48; adding oracle history produces CIDEr 11.8 and Rep@4 0.11; adding characters via CLIP similarity yields CIDEr 19.7 and Rep@4 1.30; adding characters via a MovieNet-trained identification module yields CIDEr 23.6 and Rep@4 0.97; and the full `MovieSeq (vid,img,plot,hist)` reaches CIDEr 24.4 with Rep@4 0.12 [2407.21757]. These numbers indicate that different context sources improve different aspects of output quality: plots improve content relevance, history reduces repetition, and stronger identity resolution improves named descriptions.

History length is also explicitly studied. With more past clips, the `vid+cap` template consistently improves CIDEr from 11.4 to 28.5 as the number of clips increases from 0 to 8, whereas `cap` alone improves moderately from 11.4 to 18.9 and `vid` alone changes little [2407.21757]. This suggests that past narrations are a primary carrier of long-range event dependency, while visual history without narration is comparatively weak for causal continuity.

For plot augmentation, the estimated temporal window outperforms naive clip-similarity filtering, and increasing plot context up to 512 tokens steadily improves CIDEr to 17.6 on the reported MAD analysis [2407.21757]. Multi-task co-training over balanced development sets from MAD, Movienet, and TVC improves the average from 53.8 under individual training to 55.2 under co-training, a gain of 1.5 [2407.21757].

Qualitative analysis in the paper aligns with these quantitative trends. Identity grounding is shown by inserting top-10 character photos and generating descriptions that mention names; dialogue reasoning is illustrated by using subtitles to infer relationships such as friends versus couple; plot reasoning is exemplified through a MovieQA case where the retrieved paragraph “Miley attends the birthday party as Hannah Montana” supports the rationale; and history-driven inference is demonstrated by conditioning on the earlier narration “He opens the trunk and takes out a flashlight” and predicting the subsequent action “he shines the light” [2407.21757].

## 6. Limitations, related work, and terminological distinctions

The stated limitations are primarily architectural and contextual. MovieSeq introduces no novel visual architecture and uses `[CLS]` tokens instead of patch tokens to save memory, which constrains spatial detail [2407.21757]. Long-context modeling is limited by language model context length, making full-length movies difficult to process without sacrificing fidelity. The method also depends on external annotation quality and preprocessing quality: Whisper errors, face-identification errors, or weakly derivable supervision can degrade grounding and increase hallucination [2407.21757].

The model is positioned against several neighboring lines of work. Relative to single-visual-input multimodal models such as BLIP-2, LLaVA, MiniGPT-4, Video-Chat, and Video-ChatGPT, MovieSeq generalizes the input to flexible interleaved multi-source contexts [2407.21757]. Relative to interleaved in-context models such as Flamingo, Otter, and COSMO, it focuses on narrative video and on converting public datasets into multimodal instructions spanning images, subtitles, plots, and history. Relative to long-video models such as MovieChat, LLaMA-VID, and MA-LMM, its emphasis is rich contextual interleaving rather than only memory-efficient long-video processing [2407.21757].

The term “MovieSeq” also has nearby but distinct usages in the provided literature. The 2018 paper “Convolutional Collaborative Filter Network for Video Based Recommendation Systems” does not use the term “MovieSeq,” but it operationalizes sequence-centric trailer analysis by treating a trailer as a multi-channel time series and learning temporal convolutional filters that detect object and scene sequences predictive of attendance [1810.08189]. By contrast, the 2025 paper “MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second” is relevant only if “MovieSeq” is interpreted broadly as movie or video sequences; it explicitly states that it does not cite or use any dataset or method named “MovieSeq” [2507.10065]. A further source of ambiguity is Motion Sequencing, abbreviated MoSeq, a framework for decomposing spontaneous mouse behavior into behavioral “syllables”; the data note that it is sometimes referred to as “MovieSeq” in informal contexts, but it is a separate 3D depth-video and unsupervised modeling framework rather than the narrative-video multimodal language model [2211.08497].

In that broader landscape, MovieSeq denotes a specific approach to narrative video understanding: instruction-tuned generation over interleaved multimodal context blocks, with empirical gains concentrated in settings where subtitles, identity cues, plot metadata, and temporal history are essential for disambiguation and reasoning [2407.21757].

Source: https://www.emergentmind.com/topics/movieseq