CausalStep: Causal Video Reasoning Benchmark

• CausalStep is a benchmark that evaluates explicit stepwise causal reasoning by segmenting videos into causally linked events.
• It restricts models from using full-video context by requiring reasoning based solely on local segments and previously validated answers.
• The scoring system rewards sustained chain reasoning, highlighting practical challenges in long-range causal integration and stepwise dependency tracking.

CausalStep is a video benchmark for explicit stepwise causal reasoning in videos, introduced to evaluate whether a model can reason through long videos as sequences of causally linked events rather than answer isolated questions from unrestricted global context (Li et al., 22 Jul 2025). Its defining premise is that many existing video benchmarks permit shortcut solutions because they expose the full video and all questions at once, allowing models to exploit future evidence, whole-video summaries, or scene-level priors instead of demonstrating a causally grounded, sequential reasoning process. CausalStep therefore operationalizes video understanding as an incremental chain: first establish what is happening in a segment, then explain why the next event occurs, without peeking ahead.

1. Conceptual basis and benchmark objective

CausalStep was introduced against the claim that prior video reasoning benchmarks often remain too permissive for testing true causal reasoning. In the benchmark’s framing, robust video understanding is not exhausted by action recognition or temporal ordering; it requires following event chains over time, identifying what changed, why a later event occurred, and how each event depends on prior ones. The target capability is what the paper calls explicit stepwise causal reasoning in videos (Li et al., 22 Jul 2025).

The benchmark is designed to block what the paper describes as “global-context shortcuts.” In conventional full-video question answering, a model may answer correctly by using broad contextual cues that bypass the local causal dependency under evaluation. CausalStep instead restricts access so that each reasoning step depends only on the relevant local segment pair and the already validated answer history. This makes the reasoning process more process-sensitive and diagnostically interpretable.

A useful way to characterize the benchmark is that it distinguishes between two abilities that are often conflated in video QA: descriptive understanding of visible content and causal transition reasoning across adjacent events. This suggests that CausalStep is not merely a harder multiple-choice dataset, but a benchmark whose protocol is itself part of the evaluation target.

2. Formal task structure and sequential protocol

Each video is manually segmented into a sequence of causally linked units,

$\{S_1, S_2, \ldots, S_N\},$

where each $S_i$ corresponds to a distinct causal event or state. The segmentation is based on underlying narrative and event structure, using MGIT’s action-level annotations as the foundation (Li et al., 22 Jul 2025).

For every segment $S_i$, the benchmark includes a descriptive question $Q^{desc}_i$ that tests whether the model understands what is visibly happening in that segment. For every segment except the first, it also includes a causal reasoning question $Q^{causal}_i$, which asks about the causal relationship between $S_i$ and its immediate predecessor $S_{i-1}$. The first segment therefore has only a descriptive question, while later segments are associated with both descriptive understanding and causal dependency.

The core protocol is strictly sequential. The chain begins with $Q^{desc}_1$. If the current descriptive question is answered correctly, the model proceeds. At a later causal step $i$, the model is given the current segment $S_i$, the preceding segment $S_i$0, and the history of previous correct answers, but not future segments or future questions. The paper specifies the model calls as

$S_i$1

for descriptive questions and

$S_i$2

The restart mechanism is central to the benchmark. If a model misses a causal question at segment $S_i$3, the current chain is broken and the model must restart from the descriptive question of that same segment $S_i$4. If it misses a descriptive question, the chain is broken and the restart begins at the next segment’s descriptive question, $S_i$5. This prevents unvalidated advancement through the chain and makes each causal answer contingent on previously established understanding.

The benchmark’s figure-level intuition is a short event chain. Segment $S_i$6 may show a preparatory action, with $S_i$7 asking what is happening. Segment $S_i$8 may depict a consequence such as loss of balance, and $S_i$9 asks why that occurred relative to $S_i$0. Segment $S_i$1 may depict a subsequent collision or score change, with $S_i$2 asking for the causal connection to $S_i$3. Under the protocol, an incorrect causal explanation at one step interrupts the chain and forces restart rather than allowing the model to continue as though its prior state were valid.

3. Scoring system and diagnostic metrics

CausalStep uses a nonuniform scoring system that rewards sustained correct reasoning over isolated successes (Li et al., 22 Jul 2025). Each correct descriptive question yields 1 point. For causal questions, the value depends on the position of the question within the current uninterrupted chain: the first causal question in a chain yields 1 point, the second yields 2, and so on. The pseudocode updates are

$S_i$4

after a correct descriptive answer, and

$S_i$5

after a correct causal answer. If the chain breaks, the causal score counter resets. This design privileges long uninterrupted reasoning trajectories.

The evaluation uses seven diagnostic metrics, each targeting a different failure mode or competence profile.

Metric	Meaning
CSR	Proportion of videos completed with no error
AMCL	Average longest uninterrupted chain per video
MCL	Single longest uninterrupted chain on any video
RF	Average number of restarts per video
WS	Total chain-weighted score
DUA	Accuracy on descriptive questions in isolation
ICRA	Accuracy on causal questions in an isolated setting

Chain Success Rate (CSR) measures end-to-end success: a video counts only if the entire reasoning chain is completed without error. Average Maximum Chain Length (AMCL) captures typical sustained depth before restart, while Maximum Chain Length (MCL) captures peak capability on any individual video. Restart Frequency (RF) measures brittleness by counting how often chains are interrupted. Weighted Score (WS) summarizes cumulative success under the chain-weighted scoring rule. Descriptive Understanding Accuracy (DUA) isolates perceptual or descriptive competence. Isolated Causal Reasoning Accuracy (ICRA) targets causal inference in a reduced-context setting.

Two interpretive ambiguities are explicitly noted in the source text. First, the paper does not provide symbolic formulas for all seven metrics in the main text. Second, the wording for ICRA is not fully aligned with the main task description: ICRA is described as accuracy on causal questions “when only the current segment is provided,” whereas the main causal QA protocol uses the pair $S_i$6. The main text therefore leaves the exact implementation of the isolated causal setting somewhat under-specified.

4. Dataset construction, annotation pipeline, and distractor design

CausalStep is built from a filtered subset of MGIT videos. The filtering criteria were that a video must support explicit stepwise causal reasoning with interconnected events, resist shortcuts where answers can be inferred from a single scene, and distribute key events across different times and/or locations (Li et al., 22 Jul 2025).

The benchmark statistics reported in the paper are as follows.

Item	Value
Videos	100
Mean duration	430.5 s
Duration range	149 s / 994.4 s
QA pairs	1,852
Question format	Multiple-choice
Options per question	8
Categories	6
Average segments per video	8.76
Segment range	2 / 51
Annotation	AI-assisted + Manual
Descriptive QA pairs	926
Reasoning QA pairs	926

The six categories are listed as Cartoons, Movies & TV Shows, Outdoor Sports, Regular Sports, Performances, and Documentaries. The paper also notes a wording variation elsewhere, where examples are summarized as “cartoons, movies, sports, performances, documentaries, and TV shows.” The benchmark table is the more explicit category listing.

Question construction combines automated generation with human review. Detailed segment descriptions are provided to GPT-4o to generate candidate descriptive and causal questions. Human annotators then refine these outputs for factual correctness, clarity, grounding, and alignment with the intended task. The main text does not report annotator counts or inter-annotator agreement statistics; it refers to appendices for prompts and review principles.

A major design feature is the distractor strategy. Instead of arbitrary incorrect options, CausalStep uses a taxonomy-based distractor generation process. The paper explicitly mentions error types including temporal confusion, causal misattribution, and object misrecognition. GPT-4o generates plausible, incorrect, contextually relevant, and semantically similar distractors, after which humans review and edit them to ensure that they remain nontrivial and diagnostically meaningful. Option order is randomized during evaluation.

The paper also contains a small unresolved inconsistency. Because the first segment of each video should have only a descriptive question, one might expect fewer causal than descriptive items. However, both the abstract and the dataset table report an even split of 926 descriptive QA pairs and 926 reasoning QA pairs. The main text does not explicitly reconcile this.

5. Experimental evaluation and empirical findings

The evaluation includes proprietary multimodal models, open-source multimodal models, and human baselines (Li et al., 22 Jul 2025). The proprietary models are GPT-4o-2024-11-20, GPT-4.1-2025-04-14, o4-mini-2025-04-16, Gemini-2.0-Flash, Gemini-2.0-Flash-thinking, Gemini-2.5-Flash, and Claude-3.5-Sonnet-20241022. The open-source models are Qwen2.5-VL-7B, Qwen2.5-VL-72B, Gemma3-12b-it, Gemma3-27b-it, InternVL3-8B, InternVL3-38B, LLaVA-OneVision, Video-LLaVA, and Phi4-multimodal-instruct. The paper states that all models use consistent video frame sampling strategies and input formats, including the same number of frames, though the exact frame count is not specified in the main text.

Among all tested systems, o4-mini-2025-04-16 is the best overall model, with:

• CSR = 51%
• AMCL = 7.19
• MCL = 30
• RF = 1.69
• WS = 55.06
• DUA = 85.2%
• ICRA = 39.8%

The strongest open-source model is Gemma3-27b-it, with:

• CSR = 29%
• AMCL = 5.94
• MCL = 20
• RF = 2.42
• WS = 37.64
• DUA = 77.7%
• ICRA = 26.3%

Human baselines are substantially stronger:

• CSR = 79%
• AMCL = 8.03
• MCL = 46
• RF = 0.74
• WS = 62.39
• DUA = 92.0%
• ICRA = 76.8%

The theoretical maximum is:

• CSR = 100
• AMCL = 8.76
• MCL = 51
• RF = 0
• WS = 68.76
• DUA = 100.0%
• ICRA = 100.0%

These results support three central conclusions. First, there is a large human–model gap, and it is especially large on causal reasoning. The gap in descriptive understanding is modest compared with the gap in isolated causal reasoning: 92.0% vs 85.2% on DUA, but 76.8% vs 39.8% on ICRA. Second, current models struggle to maintain long uninterrupted reasoning chains. Even the best model reaches only 51% CSR, compared with 79% for humans, and its MCL = 30 remains below human 46 and the benchmark maximum 51. Third, proprietary models consistently outperform open-source ones across metrics, but none reaches human-level stepwise reasoning.

The joint pattern of high-ish DUA with substantially lower CSR and ICRA is particularly diagnostic. It indicates that many systems can describe what is happening in a segment yet fail at inferring why one segment follows another or at preserving correctness as the sequence deepens. The paper interprets these failures as weaknesses in long-range causal integration, stepwise dependency tracking, and robustness to confusable distractors.

6. Significance, interpretability, and limitations

CausalStep’s significance lies in its claim to combine three properties that prior benchmarks generally do not enforce simultaneously: spatio-temporal understanding, causal understanding, and explicit stepwise evaluation (Li et al., 22 Jul 2025). Its contribution is therefore not only the collection of videos and questions, but the evaluation protocol itself. Because access is restricted to the current local context and the validated history, the benchmark is designed to be more resistant to shortcut exploitation than full-video QA settings.

This protocol also makes failures more interpretable. Error analysis can, in principle, be localized to descriptive grounding, local causal transition reasoning, chain maintenance, restart behavior, or distractor susceptibility. The taxonomy-based distractor design strengthens this diagnostic ambition by making wrong answers more informative about how a system failed, not just that it failed.

The benchmark also has evident limitations. It contains only 100 videos, even if those videos are long and densely annotated. It is derived from a filtered subset of MGIT, so its distribution inherits MGIT’s source characteristics. The main text does not provide extensive discussion of annotation bias, demographic bias, or inter-annotator agreement. It also does not give fully formal symbolic definitions for all seven metrics. In addition, the paper leaves unresolved the discrepancy between the claimed stepwise structure and the even split of 926 descriptive and 926 reasoning QA pairs.

Several future directions are implied rather than extensively developed. The source text suggests that progress on CausalStep-like evaluation will likely require stronger context memory, better explicit causal reasoning mechanisms, and training methods aimed at multi-step, context-dependent, and error-recovering reasoning. A plausible implication is that the benchmark is intended not merely to rank models, but to shift evaluation toward process-constrained video reasoning in which causal validity and sequential consistency are treated as first-class capabilities rather than by-products of full-context prediction.