---
title: 'ViTIB: Interleaved Video-Text QA Benchmark'
url: https://www.emergentmind.com/topics/video-text-interleaved-benchmark-vitib
type: topic
---

# ViTIB: Interleaved Video-Text QA Benchmark

Video-Text Interleaved Benchmark (ViTIB) denotes a benchmark formulation for video question answering in which visual evidence and textual reasoning are explicitly coupled rather than separated into a full-video input followed by text-only deliberation. In "ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models" [2507.09876], ViTIB is the benchmark constructed to enable and evaluate Video-Text Interleaved Chain-of-Thought (ViTCoT): each sample pairs the original video with a short “key-video,” annotated step-by-step reasoning, and a final multiple-choice answer. The benchmark is motivated by the claim that human video reasoning is inherently interleaved, because salient visual content is re-examined during inference rather than discarded after initial encoding. Related work extends the same interleaving principle to long-context subtitled video-language inputs and retrieval-oriented reasoning, most notably LongVideoBench [2407.15754].

## 1. Motivation and formal definition

ViTIB is defined by the attempt to align benchmark design with the structure of human video reasoning. The motivating premise is that, as a question about a video is processed, key frames are naturally revisited to support intermediate reasoning steps. On this view, prior step-by-step prompting paradigms remain incomplete when they prompt reasoning in text but do not preserve the most salient visual evidence within the chain-of-thought itself. In the ViTCoT formulation, ViTIB is therefore not merely a video QA dataset; it is a benchmark explicitly designed to evaluate “Video-Text Interleaved” Chain-of-Thought reasoning [2507.09876].

The benchmark is described as pairing each video-QA sample with a short “key-video,” defined as a handful of frames that directly supports the reasoning steps. This differs from full-video-plus-text benchmarks in which the entire visual stream is supplied but the reasoning trace is represented only textually. The core claim is that interleaving visual snippets with reasoning steps better mirrors human practice.

The same paper situates ViTIB against prior work on Video-of-Thought and VideoCoT. Those approaches prompt large language models step-by-step, but the reasoning process remains text only. ViTIB is constructed to test whether retaining concise, question-relevant visual evidence inside the reasoning pipeline improves video understanding.

## 2. Benchmark construction and verification

The construction pipeline comprises three main stages plus a human recheck stage. The first stage is **Automatic Sample Removal**: any raw data entry missing video, question, or answer is discarded. This establishes the basic validity of the source pool before interleaved annotation begins [2507.09876].

The second stage is **Key-Frames Recognition (via MLLM)**. Gemini-2.0-Flash is used to select the frame indices most relevant to answering the question. Its inputs are the full video $V$, question $Q$, candidate reasoning steps $R$, and answer $A$, and its output is a ranked list of frame indices $F_k$. This stage makes the benchmark explicitly question-conditioned: key frames are not generic summaries of the video, but evidence selected with respect to the reasoning path required by a specific QA instance.

The third stage is **Assemble Key-Video**. Frames at $F_k$ are extracted from the original video and packaged at $\mathrm{FPS}=1$. The resulting key-video is therefore a compact visual artifact rather than a full temporal clip, and is intended to capture the frames judged most relevant to the target reasoning.

The final stage is **Human Recheck**. Each assembled key-video is scored independently by three annotators according to a 0–100 rubric: 0–60 denotes largely irrelevant content; 60–70 denotes only a few useful frames; 70–80 denotes rough support when combined with full video; 80–90 denotes complete reasoning with full video; and 90–100 denotes self-sufficiency for the correct answer. A sample is kept only when all three scores are at least 80. If any score is below 80, reviewers revise the frame selection. The final average human score is 83.6/100. This verification protocol makes the key-video a curated reasoning aid rather than an unfiltered MLLM output.

## 3. Dataset schema and task taxonomy

ViTIB is built on **VideoEspresso**, whose base set contains 14 scene categories. The resulting benchmark contains **1 382 total video-QA samples** and **5 051 frames** in its key-videos, with an average of **3.7 frames per sample**. The **“Narrative Analysis”** category has the highest average frame count at **5.0 frames** [2507.09876].

Each sample contains five elements:

- original video $V$
- multiple-choice question $Q$ with options $O$
- key-video $V_k$ consisting of **3–5 frames**
- annotated step-by-step reasoning $R$
- final answer $A$

The task suite is multiple-choice Video Question Answering across 14 reasoning categories: **Narrative Analysis, Event Recognition, Ingredient Identification, Causal Reasoning, Theme Detection, Context Understanding, Influence Inference, Role Identification, Interactions, Behavior Recognition, Emotion Recognition, Cooking Procedures, Traffic Scene Analysis,** and **Situational Reasoning**. The input format is either the video $V$ alone or the pair $(V, V_k)$, together with $Q$ and $O$; for ViTCoT specifically, the input additionally includes preliminary text reasoning $R'$ plus interleaved $V_k$. The output format is the chain-of-thought $R$ and the final answer $A$.

The paper does not explicitly define train, validation, and test splits. It states instead that all reported results are on the held-out ViTIB test set. This omission is important for reproducibility, because standardized split definitions are typically part of benchmark operationalization.

## 4. Evaluation methodology and reported performance

The primary evaluation metric is **Accuracy (Acc.)**. For predicted answers $\hat A_i$ over $N$ samples with gold answers $A_i$, the benchmark uses

$$
\mathrm{Acc}(\%) \;=\; \frac{1}{N}\,\sum_{i=1}^{N} \mathbf{1}\bigl(\hat A_i = A_i\bigr)\;\times 100\%
$$

where $\mathbf{1}(\cdot)$ is the indicator function. The paper does **not** report BLEU, METEOR, CIDEr, or ROUGE-L scores [2507.09876].

Experiments are reported on **Qwen2.5-VL-3B, Qwen2.5-VL-7B, VideoLLaMA3-7B, Intern2.5-VL-8B,** and **Gemini-2.0-Flash**. The text-only baselines are **Vanilla CoT, Desp-CoT,** and **Plan-and-Solve**. The central empirical result is that interleaving key-videos yields consistent gains across all five models, with average improvements of **+3–10 points**.

The most detailed figures are reported for **Qwen2.5-VL-7B**. Its **Vanilla CoT** average is **42.8%**; **ViT CoT** reaches **51.4%** for a gain of **+8.6 pts**; **ViT Desp-CoT** reaches **52.9%** for **+10.1 pts**; and **ViT Plan-and-Solve** reaches **52.2%** for **+9.4 pts**. On **Qwen2.5-VL-3B**, **ViT Plan-and-Solve** improves performance from **33.3%** to **50.5%**, a gain of **+17.2 pts**. The paper also reports that ViTCoT effectively activates more neuron values in MLLMs.

Two additional ablations are central to the benchmark’s interpretation. First, even when vanilla methods are given both the original video and the key-video, ViTCoT still outperforms them by approximately **2.8 pts**. This suggests that the gain is not reducible to supplying extra visual evidence alone, but is tied to the interleaved reasoning paradigm itself. Second, automatically retrieved non-Oracle key-videos via **CLIP** still yield **+1.7 pts**, indicating that the framework is not restricted to Oracle frame selection.

## 5. Position within interleaved video-language benchmarking

The paper compares ViTIB with **MSR-VTT, ActivityNet QA,** and **VideoCoT**. **MSR-VTT** has **10 000 videos** and supports open-ended captioning and QA; **ActivityNet QA** has approximately **5 700 QA pairs**; **VideoCoT** has approximately **6 000**. By scale, ViTIB is smaller, with **1 382** samples, but each sample is enriched with interleaved key-videos and human-verified chains of thought. The comparison emphasizes annotation structure rather than raw size: prior datasets attach the full video plus text QA, whereas ViTIB explicitly pairs each QA instance with a concise set of frames that directly supports each reasoning step, thereby reducing irrelevant context [2507.09876].

| Benchmark | Scale | Interleaving structure |
|---|---:|---|
| ViTIB | 1 382 video-QA samples | Key-video of 3–5 frames plus annotated reasoning |
| MSR-VTT | 10 000 videos | Full video with open-ended captioning/QA |
| ActivityNet QA | ≈5 700 QA pairs | Full video with text QA |
| VideoCoT | ≈6 000 | Full video with text reasoning prompts |

A broader interleaved benchmark perspective appears in **LongVideoBench**, which is described as a large-scale Video-Text Interleaved Benchmark designed for long-context multimodal understanding [2407.15754]. It contains **3,763 web-collected videos** and **6,678 human-annotated multiple-choice questions**, with videos ranging from **8–15 s** through **15–60 min**, and all videos include English subtitles interleaved at mid-timestamps between adjacent frames. Its formulation uses a long interleaved input
$$
I = \{(f_1,s_1),(f_2,s_2),\ldots,(f_T,s_T)\},
$$
a referring query $q$ that locates a context window $C \subset I$, and a reasoning step that selects the correct answer from a multiple-choice set. In contrast to ViTIB’s curated key-video support, LongVideoBench emphasizes retrieval and reasoning over long interleaved sequences. This suggests a useful distinction inside the interleaved-benchmark landscape: ViTIB concentrates on concise, question-conditioned visual evidence for reasoning traces, whereas LongVideoBench concentrates on long-context retrieval and referring reasoning.

LongVideoBench also sharpens the interpretation of interleaving as a research direction. Its results show that input length matters, that longer videos are harder, that performance degrades with greater referring distance, that subtitles help but are insufficient, and that relation tasks are harder than perception tasks. These findings generalize the motivation behind ViTIB: interleaving is not only a matter of adding modality tokens, but of structuring retrieval and reasoning over the right multimodal evidence.

## 6. Limitations, open problems, and future directions

Several limitations are explicit. The first is **scale**: at **1 382 samples**, ViTIB is relatively small compared with larger video datasets. The paper notes that this may limit generalization. The second is **split specification**: train, validation, and test splits are not explicitly defined, which makes reproducibility and standardized comparison more challenging. The third is **human cost**: manual re-check with a threshold of at least **80/100** from all three annotators is labor-intensive. The fourth is **Oracle dependence**: the current ViTCoT setup uses Oracle key-videos, and fully automated key-frame selection remains to be optimized [2507.09876].

The future directions named for ViTIB are concrete. They include semi-automated or active-learning-driven validation to expand scale; improvement of automated key-frame selection through methods such as **CLIP** or shot summaries; extension to open-ended generation metrics such as **BLEU, METEOR, CIDEr,** and **ROUGE-L**; support for multi-turn dialogue and longer videos; and end-to-end MLLM finetuning on interleaved inputs.

The broader interleaved-benchmark literature adds architectural directions for future ViTIB development. LongVideoBench proposes **retrieval-augmented architectures** that explicitly index frame-text chunks and retrieve top-$K$ windows, **hierarchical attention** that first attends broadly and then refines locally, **cross-modal compression** to reduce per-frame token load, **curriculum-style pretraining** over increasingly long interleaved sequences, and **explicit temporal encodings and span-based retrieval heads** to reduce depth forgetting [2407.15754]. Taken together, these proposals indicate that ViTIB is not only a dataset artifact but part of a larger research program in which benchmark design, retrieval, temporal modeling, and multimodal reasoning are co-developed.

Source: https://www.emergentmind.com/topics/video-text-interleaved-benchmark-vitib