VideoKR-Eval: Benchmark for Video Reasoning
- The paper introduces VideoKR-Eval as a benchmark for continuous video reasoning, ensuring that models require temporally extended evidence rather than relying on single-frame shortcuts.
- Its construction pipeline filters examples using multi-model single-frame probing followed by expert reannotation to guarantee genuine domain-specific question grounding.
- Empirical results demonstrate significant improvements in model performance on knowledge-intensive tasks, validating the benchmark's design to enhance true video comprehension.
VideoKR-Eval is a 2,000-example benchmark for knowledge- and reasoning-intensive video understanding. It was introduced with VideoKR to address a specific weakness in earlier “expert-level” video benchmarks: many examples can be answered from a single frame or from weak textual cues, so they do not reliably require continuous video comprehension. VideoKR-Eval is therefore designed to test whether a model truly understands a video over time and can combine that visual understanding with domain knowledge and multi-step reasoning, rather than exploiting shortcut signals (Fu et al., 3 Jun 2026).
1. Problem setting and motivation
VideoKR-Eval was created for a setting in which models are expected to perform genuine video-level reasoning on expert-domain material. The motivating claim is that prior benchmark suites for knowledge-intensive video reasoning contain a substantial fraction of examples that are too easy in the sense that they can be solved without watching the video carefully. In the paper’s formulation, many earlier examples can actually be answered from a single frame or even weak textual cues, which makes them unreliable as tests of continuous video understanding (Fu et al., 3 Jun 2026).
This concern is narrower than the general problem of benchmarking video-LLMs. Broad evaluation frameworks such as VLM-Eval cover Video Question Answering, Video Captioning, Video-Text Retrieval, and Action Recognition, and were introduced to supply a unified evaluation pipeline for video LLMs (Li et al., 2023). VideoKR-Eval instead targets one particular failure mode inside “expert-level” video QA: the mismatch between nominal task difficulty and the actual dependence of the question on temporally extended visual evidence.
A common misconception addressed by VideoKR-Eval is that a benchmark drawn from expert or domain-aware video sources automatically measures continuous video reasoning. The benchmark’s construction rejects that assumption. Its central premise is that expert-domain questions must be filtered for shortcut resistance before they can serve as a reliable test of knowledge-intensive video understanding.
2. Construction pipeline
VideoKR-Eval is constructed from three existing knowledge-intensive video benchmarks: VideoMMMU, MMVU, and SciVideoBench (Fu et al., 3 Jun 2026). The construction procedure has two stages.
First, existing examples are filtered through multi-model single-frame probing. For each candidate example from the source benchmarks, the authors test three frontier models—Qwen3-VL-235B-A22B, Claude-4.5-Sonnet, and GPT-5.2. Each model receives only the question, the answer choices, and one randomly sampled frame, and each item is evaluated in three independent trials. An item is kept only if all three models fail this single-frame shortcut test. The paper describes this as a strict intersection criterion.
Second, videos whose original QA pairs were judged too easy are not retained with their old questions. Instead, domain experts create new questions grounded in the video content, with answers that are uniquely determined and require relevant knowledge. This stage preserves benchmark scale while improving video dependence and annotation quality.
The resulting benchmark is not merely a subset of prior data. The appendix breakdown shows that the final set combines retained hard examples with expert-reannotated replacements:
| Source benchmark | Candidate count | Final count |
|---|---|---|
| MMVU | 1,000 | 759 |
| VideoMMMU | 900 | 581 |
| SciVideoBench | 1,000 | 660 |
Across the full pool, 2,900 candidate examples yield 2,000 final examples. The paper further specifies 1,254 retained original examples and 746 expert-reannotated examples, after filtering out 1,646 items from the initial pool.
3. Benchmark composition and task profile
VideoKR-Eval is defined as a 2,000-example evaluation set designed to measure whether a model truly understands a video over time and can combine that visual understanding with domain knowledge and multi-step reasoning (Fu et al., 3 Jun 2026). The tasks are mainly in the familiar video QA format, including both multiple-choice and open-ended questions.
The annotation instructions emphasize three requirements. Questions should be grounded in clearly observable video evidence, require relevant domain knowledge, and have uniquely determined answers. The benchmark is therefore intended to probe a conjunction of capabilities rather than a single skill.
The paper characterizes the required reasoning in terms of continuous video understanding, domain knowledge, and multi-step inference. It is meant to test reasoning such as interpreting domain-specific procedures, identifying scientific or medical processes, making causal or inferential judgments from video evidence, and combining temporal understanding with factual knowledge. The examples are associated with scientific experiments, medical procedures, engineering processes, lecture-like or professional-domain video content, and other expert-level scenarios.
This task profile places VideoKR-Eval away from simple action recognition. A plausible implication is that success on the benchmark depends not only on recognizing visual entities and actions, but also on integrating temporally distributed evidence with external knowledge and inference over procedure, causality, or explanation.
4. Single-frame solvability and shortcut resistance
The formal element that defines VideoKR-Eval most sharply is its single-frame solvability criterion. An example is classified as single-frame-solvable for a model only if the model answers it correctly in all three independent trials using only the question, the answer options, and one randomly sampled frame. VideoKR-Eval keeps only examples that are not single-frame-solvable for all three probing models (Fu et al., 3 Jun 2026).
The paper reports single-frame answerability rates for the three probing models across the source benchmarks and the final benchmark:
| Model | VidMMMU / MMVU / SciVidBench | VideoKR-Eval |
|---|---|---|
| Claude-4.5-Sonnet | 35.3 / 41.3 / 21.8 | 9.5 |
| Qwen3-VL-235B-A22B | 39.3 / 45.2 / 13.2 | 10.1 |
| GPT-5.2 | 38.3 / 49.7 / 23.0 | 10.7 |
These numbers are the empirical basis for the claim that prior benchmarks are vulnerable to shortcut reasoning. On earlier suites, frontier models can solve more than 35% of examples from a single frame in some cases. On VideoKR-Eval, the rate is reduced to around 10%. The benchmark was designed to make that reduction possible by enforcing intersection-based failure across multiple probing models before an item is retained.
This does not imply that VideoKR-Eval eliminates all shortcut behavior. The reported rates remain nonzero. The paper’s claim is narrower: the benchmark is much harder to bypass with single-frame reasoning than the source benchmarks from which it was derived.
5. Evaluation protocol and empirical role
VideoKR-Eval is used inside a standardized evaluation protocol with LMMs-Eval. Each model is run three times with independent sampling, and the reported score is the mean. For each model, the official prompt is used if available; otherwise the default LMMs-Eval prompt is used. The paper also standardizes inference settings, including temperature and maximum response length tokens. For input frames, each model’s recommended inference configuration is followed when specified (Fu et al., 3 Jun 2026).
In the experimental suite, VideoKR-Eval is evaluated alongside Video-MME, MVBench, LongVideoBench, VideoMMMU, MMVU, and SciVideoBench. Within that broader suite, it functions as the most targeted benchmark for knowledge-intensive video reasoning.
The benchmark plays a central role in the paper’s claim that data design is a key bottleneck in advanced video reasoning. In the main results table, the post-trained models improve especially on knowledge-intensive benchmarks, including VideoKR-Eval. For Qwen2.5-VL-7B-Instruct, the baseline knowledge-intensive average is 41.9, and after VideoKR SFT + RL it rises to 46.6, an improvement of . For Qwen3-VL-8B-Instruct, the corresponding figures are 48.5 and 51.5, an improvement of . The paper highlights that the largest improvements occur on MMVU and VideoKR-Eval, with VideoKR-Eval showing a particularly strong gain of points for Qwen2.5-VL-7B.
Because VideoKR-Eval is designed to be resistant to single-frame shortcuts, improvements on it are presented as evidence of improved true video reasoning rather than better exploitation of benchmark-specific heuristics. This interpretation is explicit in the paper’s discussion of the benchmark’s role.
6. Relation to the VideoKR corpus and benchmark significance
VideoKR-Eval is closely related to the VideoKR training corpus, but the two serve different purposes. VideoKR is the training dataset: 315K video reasoning examples over 145K newly collected, CC-licensed videos, spanning 82 professional subjects, and split into VideoKR-SFT-201K for supervised fine-tuning and VideoKR-RL-114K for reinforcement learning with verifiable rewards. VideoKR-Eval is the held-out benchmark used to assess whether post-training on VideoKR improves real knowledge-intensive video reasoning (Fu et al., 3 Jun 2026).
The paper also reports decontamination efforts intended to preserve the credibility of the evaluation set as distinct from training data: YouTube-ID matching removed 131 videos, and near-duplicate filtering removed 877 videos. This suggests deliberate separation between corpus construction and held-out assessment.
In the paper’s broader argument, VideoKR-Eval is the evaluation counterpart to the design philosophy used for the training corpus. That philosophy is supported by an ablation on training corpora in which zero-shot accuracy of strong base models on 3,000 sampled QA examples is relatively high on prior corpora—57.1 for Video-R1, 51.1 for VideoRFT, 49.1 for OneThinker, and 54.5 for VideoAuto-R1 with Qwen3-VL-8B-Inst—but lower on VideoKR at 42.3. The authors interpret lower zero-shot accuracy on VideoKR as evidence that the training set is harder and provides a stronger learning signal. VideoKR-Eval extends the same logic to evaluation: it is intentionally challenging and demands actual reasoning.
The significance of VideoKR-Eval therefore lies less in sheer scale than in curation strategy. It is a carefully curated, expert-validated benchmark intended to measure whether models can perform knowledge-intensive video reasoning under conditions designed to suppress single-frame shortcuts. Within the contemporary benchmark landscape, its distinguishing feature is not broad task coverage but stringent filtering for continuous video dependence.