ViTIB: Interleaved Video-Text QA Benchmark

• The paper introduces ViTIB, a benchmark interleaving key-video frames with textual reasoning, demonstrating consistent performance gains of 3–10 points across models.
• It employs an automatic key-frame selection using Gemini-2.0-Flash and a rigorous human recheck process to ensure that only highly relevant frames support each reasoning step.
• ViTIB is designed for multiple-choice video QA with structured reasoning, offering insights into efficient interleaved evidence usage compared to full-video inputs.

Video-Text Interleaved Benchmark (ViTIB) denotes a benchmark formulation for video question answering in which visual evidence and textual reasoning are explicitly coupled rather than separated into a full-video input followed by text-only deliberation. In "ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in LLMs" (Zhang et al., 14 Jul 2025), ViTIB is the benchmark constructed to enable and evaluate Video-Text Interleaved Chain-of-Thought (ViTCoT): each sample pairs the original video with a short “key-video,” annotated step-by-step reasoning, and a final multiple-choice answer. The benchmark is motivated by the claim that human video reasoning is inherently interleaved, because salient visual content is re-examined during inference rather than discarded after initial encoding. Related work extends the same interleaving principle to long-context subtitled video-language inputs and retrieval-oriented reasoning, most notably LongVideoBench (Wu et al., 2024).

1. Motivation and formal definition

ViTIB is defined by the attempt to align benchmark design with the structure of human video reasoning. The motivating premise is that, as a question about a video is processed, key frames are naturally revisited to support intermediate reasoning steps. On this view, prior step-by-step prompting paradigms remain incomplete when they prompt reasoning in text but do not preserve the most salient visual evidence within the chain-of-thought itself. In the ViTCoT formulation, ViTIB is therefore not merely a video QA dataset; it is a benchmark explicitly designed to evaluate “Video-Text Interleaved” Chain-of-Thought reasoning (Zhang et al., 14 Jul 2025).

The benchmark is described as pairing each video-QA sample with a short “key-video,” defined as a handful of frames that directly supports the reasoning steps. This differs from full-video-plus-text benchmarks in which the entire visual stream is supplied but the reasoning trace is represented only textually. The core claim is that interleaving visual snippets with reasoning steps better mirrors human practice.

The same paper situates ViTIB against prior work on Video-of-Thought and VideoCoT. Those approaches prompt LLMs step-by-step, but the reasoning process remains text only. ViTIB is constructed to test whether retaining concise, question-relevant visual evidence inside the reasoning pipeline improves video understanding.

2. Benchmark construction and verification

The construction pipeline comprises three main stages plus a human recheck stage. The first stage is Automatic Sample Removal: any raw data entry missing video, question, or answer is discarded. This establishes the basic validity of the source pool before interleaved annotation begins (Zhang et al., 14 Jul 2025).

The second stage is Key-Frames Recognition (via MLLM). Gemini-2.0-Flash is used to select the frame indices most relevant to answering the question. Its inputs are the full video $V$, question $Q$, candidate reasoning steps $R$, and answer $A$, and its output is a ranked list of frame indices $F_k$. This stage makes the benchmark explicitly question-conditioned: key frames are not generic summaries of the video, but evidence selected with respect to the reasoning path required by a specific QA instance.

The third stage is Assemble Key-Video. Frames at $F_k$ are extracted from the original video and packaged at $\mathrm{FPS}=1$. The resulting key-video is therefore a compact visual artifact rather than a full temporal clip, and is intended to capture the frames judged most relevant to the target reasoning.

The final stage is Human Recheck. Each assembled key-video is scored independently by three annotators according to a 0–100 rubric: 0–60 denotes largely irrelevant content; 60–70 denotes only a few useful frames; 70–80 denotes rough support when combined with full video; 80–90 denotes complete reasoning with full video; and 90–100 denotes self-sufficiency for the correct answer. A sample is kept only when all three scores are at least 80. If any score is below 80, reviewers revise the frame selection. The final average human score is 83.6/100. This verification protocol makes the key-video a curated reasoning aid rather than an unfiltered MLLM output.

3. Dataset schema and task taxonomy

ViTIB is built on VideoEspresso, whose base set contains 14 scene categories. The resulting benchmark contains 1 382 total video-QA samples and 5 051 frames in its key-videos, with an average of 3.7 frames per sample. The “Narrative Analysis” category has the highest average frame count at 5.0 frames (Zhang et al., 14 Jul 2025).

Each sample contains five elements:

• original video $V$
• multiple-choice question $Q$ with options $O$
• key-video $Q$0 consisting of 3–5 frames
• annotated step-by-step reasoning $Q$1
• final answer $Q$2

The task suite is multiple-choice Video Question Answering across 14 reasoning categories: Narrative Analysis, Event Recognition, Ingredient Identification, Causal Reasoning, Theme Detection, Context Understanding, Influence Inference, Role Identification, Interactions, Behavior Recognition, Emotion Recognition, Cooking Procedures, Traffic Scene Analysis, and Situational Reasoning. The input format is either the video $Q$3 alone or the pair $Q$4, together with $Q$5 and $Q$6; for ViTCoT specifically, the input additionally includes preliminary text reasoning $Q$7 plus interleaved $Q$8. The output format is the chain-of-thought $Q$9 and the final answer $R$0.

The paper does not explicitly define train, validation, and test splits. It states instead that all reported results are on the held-out ViTIB test set. This omission is important for reproducibility, because standardized split definitions are typically part of benchmark operationalization.

4. Evaluation methodology and reported performance

The primary evaluation metric is Accuracy (Acc.). For predicted answers $R$1 over $R$2 samples with gold answers $R$3, the benchmark uses

$R$4

where $R$5 is the indicator function. The paper does not report BLEU, METEOR, CIDEr, or ROUGE-L scores (Zhang et al., 14 Jul 2025).

Experiments are reported on Qwen2.5-VL-3B, Qwen2.5-VL-7B, VideoLLaMA3-7B, Intern2.5-VL-8B, and Gemini-2.0-Flash. The text-only baselines are Vanilla CoT, Desp-CoT, and Plan-and-Solve. The central empirical result is that interleaving key-videos yields consistent gains across all five models, with average improvements of +3–10 points.

The most detailed figures are reported for Qwen2.5-VL-7B. Its Vanilla CoT average is 42.8%; ViT CoT reaches 51.4% for a gain of +8.6 pts; ViT Desp-CoT reaches 52.9% for +10.1 pts; and ViT Plan-and-Solve reaches 52.2% for +9.4 pts. On Qwen2.5-VL-3B, ViT Plan-and-Solve improves performance from 33.3% to 50.5%, a gain of +17.2 pts. The paper also reports that ViTCoT effectively activates more neuron values in MLLMs.

Two additional ablations are central to the benchmark’s interpretation. First, even when vanilla methods are given both the original video and the key-video, ViTCoT still outperforms them by approximately 2.8 pts. This suggests that the gain is not reducible to supplying extra visual evidence alone, but is tied to the interleaved reasoning paradigm itself. Second, automatically retrieved non-Oracle key-videos via CLIP still yield +1.7 pts, indicating that the framework is not restricted to Oracle frame selection.

5. Position within interleaved video-language benchmarking

The paper compares ViTIB with MSR-VTT, ActivityNet QA, and VideoCoT. MSR-VTT has 10 000 videos and supports open-ended captioning and QA; ActivityNet QA has approximately 5 700 QA pairs; VideoCoT has approximately 6 000. By scale, ViTIB is smaller, with 1 382 samples, but each sample is enriched with interleaved key-videos and human-verified chains of thought. The comparison emphasizes annotation structure rather than raw size: prior datasets attach the full video plus text QA, whereas ViTIB explicitly pairs each QA instance with a concise set of frames that directly supports each reasoning step, thereby reducing irrelevant context (Zhang et al., 14 Jul 2025).

Benchmark	Scale	Interleaving structure
ViTIB	1 382 video-QA samples	Key-video of 3–5 frames plus annotated reasoning
MSR-VTT	10 000 videos	Full video with open-ended captioning/QA
ActivityNet QA	≈5 700 QA pairs	Full video with text QA
VideoCoT	≈6 000	Full video with text reasoning prompts

A broader interleaved benchmark perspective appears in LongVideoBench, which is described as a large-scale Video-Text Interleaved Benchmark designed for long-context multimodal understanding (Wu et al., 2024). It contains 3,763 web-collected videos and 6,678 human-annotated multiple-choice questions, with videos ranging from 8–15 s through 15–60 min, and all videos include English subtitles interleaved at mid-timestamps between adjacent frames. Its formulation uses a long interleaved input

$R$6

a referring query $R$7 that locates a context window $R$8, and a reasoning step that selects the correct answer from a multiple-choice set. In contrast to ViTIB’s curated key-video support, LongVideoBench emphasizes retrieval and reasoning over long interleaved sequences. This suggests a useful distinction inside the interleaved-benchmark landscape: ViTIB concentrates on concise, question-conditioned visual evidence for reasoning traces, whereas LongVideoBench concentrates on long-context retrieval and referring reasoning.

LongVideoBench also sharpens the interpretation of interleaving as a research direction. Its results show that input length matters, that longer videos are harder, that performance degrades with greater referring distance, that subtitles help but are insufficient, and that relation tasks are harder than perception tasks. These findings generalize the motivation behind ViTIB: interleaving is not only a matter of adding modality tokens, but of structuring retrieval and reasoning over the right multimodal evidence.

6. Limitations, open problems, and future directions

Several limitations are explicit. The first is scale: at 1 382 samples, ViTIB is relatively small compared with larger video datasets. The paper notes that this may limit generalization. The second is split specification: train, validation, and test splits are not explicitly defined, which makes reproducibility and standardized comparison more challenging. The third is human cost: manual re-check with a threshold of at least 80/100 from all three annotators is labor-intensive. The fourth is Oracle dependence: the current ViTCoT setup uses Oracle key-videos, and fully automated key-frame selection remains to be optimized (Zhang et al., 14 Jul 2025).

The future directions named for ViTIB are concrete. They include semi-automated or active-learning-driven validation to expand scale; improvement of automated key-frame selection through methods such as CLIP or shot summaries; extension to open-ended generation metrics such as BLEU, METEOR, CIDEr, and ROUGE-L; support for multi-turn dialogue and longer videos; and end-to-end MLLM finetuning on interleaved inputs.

The broader interleaved-benchmark literature adds architectural directions for future ViTIB development. LongVideoBench proposes retrieval-augmented architectures that explicitly index frame-text chunks and retrieve top-$R$9 windows, hierarchical attention that first attends broadly and then refines locally, cross-modal compression to reduce per-frame token load, curriculum-style pretraining over increasingly long interleaved sequences, and explicit temporal encodings and span-based retrieval heads to reduce depth forgetting (Wu et al., 2024). Taken together, these proposals indicate that ViTIB is not only a dataset artifact but part of a larger research program in which benchmark design, retrieval, temporal modeling, and multimodal reasoning are co-developed.