MovieCORE: Deep Cinematic QA Benchmark

• MovieCORE is a benchmark that shifts from surface-level video QA to deep, evidence-aware analysis of cinematic narratives.
• It uses 986 movie clips and over 4,900 QA pairs, emphasizing interpretive questions on motives, causality, and symbolism.
• The design incorporates an agentic brainstorming pipeline and an ACE post-training mechanism to enhance answer quality with grounded reasoning.

MovieCORE is a movie/video question answering benchmark designed to test whether vision-LLMs can move beyond recognizing obvious events and instead perform deeper, more deliberate interpretation of cinematic content. It frames this transition as a shift from shallow, “System-1”-like video understanding toward “System-2” reasoning: slow, analytic, evidence-aware reasoning about motives, causality, symbolism, psychology, emotion, and narrative dynamics. The benchmark consists of movie clips, captions, and question-answer pairs intended not merely to ask what is happening, but why it is happening, how characters’ states evolve, what underlying motivations or tensions are present, and what visual evidence supports such interpretations (Faure et al., 26 Aug 2025).

1. Conceptual orientation

MovieCORE is explicitly organized around a distinction between surface comprehension and grounded depth. Existing movie and long-video benchmarks are described as often focusing on factual recall, event description, temporal context, or relatively direct causal questions. By contrast, MovieCORE emphasizes questions involving character motivations, causality, psychological complexity, emotional resonance, subtext, symbolism, narrative progression, cinematographic evidence, and social or relational dynamics (Faure et al., 26 Aug 2025).

A central design principle is the distinction between generic abstraction and grounded depth. The benchmark does not seek merely philosophical or generic commentary about a movie. Instead, it targets higher-order reasoning that remains specific to the actual clip. In this formulation, a question is intended to demand analysis while still being answerable from scene-level evidence. This directly addresses a common misconception about difficult video QA: that harder questions are necessarily broader or more speculative. MovieCORE instead treats grounded interpretive reasoning as the relevant difficulty axis.

The benchmark’s motivating claim is that current models that perform well on surface-level movie QA still fail badly when asked to reason deeply about the same clips. This claim is operationalized by reusing the clip pool of MovieChat-1k while changing the type of questions being asked, thereby attempting to isolate question difficulty from raw video content (Faure et al., 26 Aug 2025).

2. Corpus composition and benchmark design

MovieCORE is built on 986 movie clips drawn from MovieChat-1k, with 4,930 question-answer pairs and 986 captions. It uses 986 of the original 1,000 clips, omitting 14 that were unavailable or lacked necessary annotations. The split follows MovieChat-1k: 816 videos and 4,080 QA pairs for training, and 170 videos and 850 QA pairs for testing (Faure et al., 26 Aug 2025).

Split	Videos	QA pairs
Train	816	4,080
Test	170	850

The reuse of the MovieChat-1k clip pool is methodologically important. Because the videos are largely the same as a prior benchmark, the benchmark design is intended to isolate the effect of question type. This suggests that performance differences relative to prior movie QA settings can be interpreted less as a consequence of different footage and more as a consequence of deeper reasoning demands.

MovieCORE intentionally does not use MovieChat captions because the authors found them inaccurate or unbalanced. Instead, it constructs a richer textual prior for each clip. MiniCPM-v2.6 is prompted with eight structured questions covering step-by-step events, main subject, emotional tone, key actions, main characters or entities, settings or locations, genre, and intended audience. The output is a “Data Info” package used as context for downstream annotation agents (Faure et al., 26 Aug 2025).

The benchmark does not present a formal taxonomy of answer types, but the recurring reasoning styles described in the text and figures include motives, visual evidence, social tension, symbolic interpretation, and narrative dynamics. A word cloud is also reported as a rough diversity analysis, highlighting terms such as “emotional,” “character,” “influence,” “tension,” “psychological,” “cultural,” “underscore,” “depth,” and “critical,” which the authors take as evidence that the dataset emphasizes interpretive and analytical content rather than literal description (Faure et al., 26 Aug 2025).

3. Agentic brainstorming annotation pipeline

One of MovieCORE’s main methodological contributions is an agentic brainstorming annotation pipeline implemented with the AutoGen framework. The process is orchestrated by a Critic Agent acting as a master of ceremonies. Different roles are assigned to different models: GPT-4o is used for the more reasoning-intensive VQA Expert and Meta Reviewer roles, while GPT-4o-mini powers the other experts (Faure et al., 26 Aug 2025).

The workflow proceeds in five stages. First, the Critic Agent receives task instructions and the extracted video priors, then forwards them to the System-2 Video Question Answering Assistant, also called the System II VQA Expert, which generates up to five initial System-2-style QA pairs. Second, the Skeptical Researcher reviews those QAs for contextual relevance and accuracy, challenging assumptions and demanding stronger grounding. Third, the Detective proposes additional questions or angles that might expose hidden motivations, biases, causal drivers, or latent narrative structure. Fourth, the Meta Reviewer aggregates critiques and suggestions while filtering speculative proposals and retaining only refinements that remain truthful to the clip. Fifth, the Critic Agent compiles the feedback and sends it back to the System-2 VQA Expert for revision (Faure et al., 26 Aug 2025).

The authors contrast this multi-agent process with “single-pass” automatic annotation. Their qualitative comparisons emphasize that single-pass outputs often produce broad commentary, whereas the agentic version more often asks for specific scenes, motivations, or visual techniques supporting an interpretation. The intended effect is not only greater difficulty, but greater fidelity to visible evidence. This suggests that the annotation pipeline is designed as a quality-control mechanism for grounded interpretive complexity rather than as a volume-generation mechanism alone.

Human involvement is limited but nontrivial. Seven graduate students evaluated a subset of 30 videos, 30 captions, and 150 QA pairs using standardized 1–5 scales. Average human scores were reported as follows: captions scored 3.9 in accuracy, 4.0 in clarity, and 4.1 in depth; questions scored 4.3 in clarity, 4.5 in depth, 4.0 in relevance, and 3.8 in answerability; answers scored 4.3 in clarity, 4.2 in depth, 3.8 in relevance, and 4.1 in answerability (Faure et al., 26 Aug 2025). The paper interprets these scores as evidence that the generated questions are generally understandable, deep, and grounded enough to be answerable from the clip, while acknowledging that some difficult or borderline cases remained.

4. Cognitive diagnostics and quality assessment

MovieCORE introduces a set of “cognitive tests” intended as dataset diagnostics rather than direct model evaluations. The three main dimensions are syntactic complexity, readability or comprehension level, and cognitive level (Faure et al., 26 Aug 2025).

Syntactic complexity is measured through parse tree depth using spaCy parse trees. If $C(t)$ is the set of children of token $t$, the recursive definition is

$$d(t) = \begin{cases} 0 & \text{if } C(t)=\emptyset \ 1 + \max_{c \in C(t)} d(c) & \text{if } C(t)\neq\emptyset \end{cases}$$

and for a sentence rooted at token $r$, the parse tree depth is $D=d(r)$. MovieCORE has average parse tree depth 5.88 across questions and answers, compared with 5.47 for EgoSchema, 2.84 for MVBench, 2.45 for MovieChat-1k, and 2.26 for ActivityNetQA. On the question side alone, MovieCORE is slightly lower than EgoSchema at 5.38 versus 6.56, but its answers are much more complex at 6.39 versus 4.38, giving it the highest overall average (Faure et al., 26 Aug 2025).

Readability is measured with the Flesch–Kincaid grade score,

$$\text{F-K Grade Score} = 0.39 \left( \frac{W}{S} \right) + 11.8 \left( \frac{Y}{W} \right) - 15.59,$$

where $W$ is the number of words, $S$ the number of sentences, and $Y$ the number of syllables. MovieCORE’s average F-K score is 14.03, compared with 8.30 for EgoSchema, 3.11 for MVBench, 1.84 for ActivityNetQA, and 1.4 for MovieChat-1k (Faure et al., 26 Aug 2025). In the paper’s interpretation, this indicates that the benchmark requires more advanced comprehension and supports more nuanced reasoning.

Cognitive level is estimated through Bloom’s Taxonomy classification using GPT-4o-mini, with levels Remember (1), Understand (2), Apply (3), Analyze (4), Evaluate (5), and Create (6). MovieCORE scores an average Bloom Taxonomy level of 4.9, compared with 3.1 for EgoSchema, 2.2 for MVBench, 1.9 for ActivityNetQA, and 1.8 for MovieChat-1k. Moreover, 99.2% of MovieCORE questions and answers are classified as higher-order, defined as levels 4–6, compared with 33.1% for EgoSchema and near zero for the others (Faure et al., 26 Aug 2025).

These diagnostics support the benchmark’s stated objective: not simply to be longer or more open-ended than existing movie QA datasets, but to shift the evaluative focus toward analysis, evaluation, and synthesis.

5. Evaluation framework and empirical findings

Because exact-match accuracy is poorly suited to open-ended, interpretive answers, MovieCORE proposes an LLM-assisted multi-dimensional rubric in which GPT-4o-mini assigns 0–5 scores along five axes: Accuracy, Comprehensiveness, Depth, Evidence, and Coherence (Faure et al., 26 Aug 2025). In the supplementary rubric, Accuracy measures semantic correctness relative to the reference answer, Comprehensiveness measures coverage of key points, Depth measures analytical depth and interpretive insight, Coherence measures clarity and logical organization, and Evidence measures whether the answer uses strong, relevant support from the video. Evidence is especially central to the benchmark’s notion of grounded reasoning.

The benchmark compares proprietary models, open-source zero-shot models, and fully supervised open-source models. In the zero-shot setting, InternVL2.5 obtains the best open-source average score at 3.62, followed by Qwen2.5-VL at 3.52, InternVL2 at 3.44, mPlug-Owl3 at 2.86, LongVU at 2.22, HERMES at 1.41, MA-LMM at 0.79, and InstructBLIP at 0.61. Proprietary systems score higher: Gemini 2.5-flash reaches 4.13, GPT-4o 4.02, and Gemini-1.5-pro 3.86 (Faure et al., 26 Aug 2025). The paper highlights particularly large deficits for open models in Depth and Evidence.

Fine-tuning on MovieCORE yields substantial gains for open models. HERMES improves from 1.41 to 2.93, MA-LMM from 0.79 to 2.79, and InstructBLIP from 0.61 to 2.63 (Faure et al., 26 Aug 2025). Even with supervision, however, these systems remain below the reported proprietary scores, which the paper treats as evidence that deeper cognitive movie understanding remains unsolved.

A particularly strong ablation uses the same underlying clips to compare “System-2” MovieCORE with “System-1” MovieChat-1k. Using HERMES, the zero-shot score on MovieCORE is 1.14 average, while MovieChat-1k reports 78.6% accuracy, mapped by the authors to approximately 3.93 on a 0–5 scale. In the fully supervised setting, HERMES reaches 3.52 on MovieCORE but 84.9% accuracy on MovieChat-1k, roughly 4.25 on the same 0–5 scale (Faure et al., 26 Aug 2025). Since the clips are the same, this comparison is presented as evidence that question type, rather than video content, is the principal source of difficulty.

Traditional text-generation metrics are also reported for comparability. For example, HERMES + ACE reaches BLEU-4 0.0654, CIDEr 0.1622, and METEOR 0.2138, improving over plain HERMES at 0.0308, 0.1230, and 0.0983 (Faure et al., 26 Aug 2025). The paper nonetheless argues that such n-gram metrics are poorly aligned with the benchmark’s actual target: semantically rich, evidence-grounded reasoning.

6. Agentic Choice Enhancement, limitations, and place in the literature

MovieCORE also introduces Agentic Choice Enhancement (ACE), a lightweight post-training answer selection mechanism. ACE does not retrain the base VLM. It generates multiple candidate answers with beam search and then uses a separate small LLM as a judge to select the best one:

$$C \gets \text{VLM.generate}(V, Q, \text{beam\_width}=k)$$

$t$0

$t$1

with $t$2 in the provided pseudocode (Faure et al., 26 Aug 2025). In the paper’s formulation, $t$3 is the video, $t$4 the question, $t$5 the candidate set, $t$6 the scalar quality score assigned by Llama-3.2 (1B), and $t$7 the selected answer. ACE is described as a “second pair of eyes” that re-ranks candidate answers without additional training.

ACE improves all supervised models. InstructBLIP rises from 2.63 to 3.29, an absolute gain of 0.66 and a reported relative improvement of about 25%; MA-LMM rises from 2.79 to 3.35, a gain of 0.56 and about 20%; HERMES rises from 2.93 to 3.41, a gain of 0.48 and about 16% (Faure et al., 26 Aug 2025). For HERMES, the improvements are broad across the five rubric dimensions: Accuracy increases from 3.52 to 3.81, Comprehensiveness from 2.72 to 3.30, Depth from 2.83 to 3.12, Evidence from 2.98 to 3.38, and Coherence from 2.62 to 3.42. An ablation on HERMES with ACE reports similar average scores for beam sizes 3, 5, and 7—3.45, 3.41, and 3.37 respectively—supporting the claim that selection, rather than aggressive search, is the main factor.

The paper identifies the dominant failure modes of existing VLMs as shallow answers, weak evidence, and insufficient depth. Models often produce plausible but generic interpretations, fail to refer to particular scenes, do not explain the causal chain linking events to motivations, or remain at the level of “what happened” instead of “why,” “how,” or “why not” (Faure et al., 26 Aug 2025). This characterization aligns with the benchmark’s broader critique of surface-level movie QA.

Several limitations are explicitly acknowledged. Human verification covers only 30 videos and 150 QA pairs, so most annotations remain automatically generated. Because the clips come from MovieChat-1k, genre diversity may be constrained by the source dataset. Evaluation is partly LLM-assisted, raising concerns about judge-model bias, calibration, and reproducibility. The benchmark also embraces some subjectivity in “why” questions, which means that a single ground truth can be inherently fuzzy (Faure et al., 26 Aug 2025).

Within the broader literature, MovieCORE occupies a distinct position. Moviescope studies movie-level multimodal analysis from trailers, plots, posters, audio, and metadata, with tasks such as genre prediction and budget estimation rather than clip-grounded interpretive QA (Cascante-Bonilla et al., 2019). MPGN addresses subtitle-rich narrative video corpus moment retrieval through subtitle-based moment sampling and modal-specific pseudo query generation, emphasizing retrieval and localization rather than System-2 movie QA (Jung et al., 2022). MV-CoRe augments image-based complex VQA with object features and scene graph representations, but remains fundamentally single-image and does not model temporal or narrative movie structure (Peng et al., 9 Aug 2025). Against this background, MovieCORE is best understood as a benchmark for clip-grounded, cognitively demanding movie understanding, together with an annotation methodology, an evaluation rubric, and a lightweight inference-time enhancement method (Faure et al., 26 Aug 2025).