SciVideoBench: Scientific Video Reasoning Benchmark

• SciVideoBench is a comprehensive benchmarking framework designed to evaluate scientific video reasoning in multimodal models using complex experimental data across 25+ academic subjects.
• It integrates synchronized video, ASR-generated transcripts, and peer-reviewed text to assess conceptual, hypothetical, and quantitative reasoning with precise spatiotemporal grounding.
• Empirical findings reveal that current LMMs struggle, particularly in quantitative tasks, underscoring the need for domain-specific adaptations and advanced chain-of-thought prompting.

SciVideoBench is a comprehensive benchmarking framework developed to rigorously assess scientific video reasoning in large multimodal models (LMMs). Its principal aim is to address the limitations of prior video understanding benchmarks, which are restricted to general scenarios and fail to test higher-order cognitive skills essential for scientific applications. SciVideoBench focuses on complex experimental videos drawn from research-grade sources, representing 25+ specialized academic subjects and challenging models with tasks that require precise spatiotemporal grounding, domain-specific expertise, and multi-step logical inference (Deng et al., 9 Oct 2025).

1. Purpose and Context

SciVideoBench was conceived to fill the critical gap between existing video benchmarks—where tasks often rely on surface-level perception and saturated recognition accuracy—and the demands of advanced scientific reasoning. Scientific experimental videos present unique challenges:

• Rich spatiotemporal cues (e.g., annotation overlays, timestamped procedures)
• Multi-modal inputs: synchronized video, transcript (from ASR models like Whisper), and paired peer-reviewed text
• Deep reliance on subject-matter expertise in physics, chemistry, biology, and medicine

Unlike perception-focused datasets, SciVideoBench is intended as a high-fidelity testbed to evaluate whether LMMs can perform three core types of reasoning that are prevalent in academic research:

• Conceptual: understanding scientific principles and protocols
• Hypothetical: analyzing alternative experimental designs or outcomes
• Quantitative: extracting and manipulating experimental values for numeric reasoning

This design attempts to close the gap between AI capabilities and those needed of a genuine scientific co-investigator.

2. Benchmark Structure and Dataset Design

SciVideoBench is derived from 241 experimental videos published in professional platforms (e.g., JoVE), spanning fields from Fluid Mechanics to Oncology. Each video is carefully annotated using a hybrid pipeline:

• Temporal synchronization uses ASR (Whisper) to provide frame-accurate transcripts
• Each question demands an overview of visual observation and textual understanding
• Disciplines are covered evenly: Physics, Chemistry, Biology, Medicine, subdivided into over 25 specialized topics

A set of 1,000 multiple-choice questions was generated semi-automatically through staged annotation:

• LLM agents propose candidate questions, visual comparers validate temporal and spatial grounding, and human domain experts refine for plausibility and scientific coverage
• Distractor options are verified for plausibility, eliminating bias and language-only shortcuts

Each question is classified as conceptual, hypothetical, or quantitative. The answer is visually grounded—requiring observation of specific on-screen events, measurements, or procedural steps.

3. Evaluation Protocols

SciVideoBench introduces rigorous evaluation methodologies:

• Models tested: both proprietary (Gemini-2.5-Pro, GPT-4o, Qwen2.5-VL) and open-source (spanning 0.5B to >70B parameters)
• Metrics: Overall accuracy, broken down by question category and scientific discipline, is the primary evaluation metric
• Vision-blind baseline: scores computed when video input is withheld; random-chance accuracy confirms that questions require multimodal understanding
• Chain-of-Thought (CoT) prompting: models are evaluated in both direct and “reasoning” modes, with performance improvements especially pronounced in multi-step quantitative tasks

Models are tested using consistent frame sampling parameters (e.g., 256 frames for GPT-4o), fixed sampling temperature, and synchronized multi-modal inputs to ensure reproducibility.

Performance metric formula: $$\text{Accuracy} = \frac{\text{Number of Correct Answers}}{\text{Total Number of Questions}} \times 100\%$$ Additional stratified scores are reported for conceptual, hypothetical, and quantitative reasoning per discipline.

4. Key Empirical Findings

Analysis of evaluation results reveals substantial deficiencies of current LMMs in scientific video reasoning:

• Proprietary model advantage: Gemini-2.5-Pro achieves ~64.3% overall accuracy, nearly doubling the best open-source scores (mid-to-high 30%); the performance gap is even larger in quantitative tasks
• Visual grounding indispensability: vision-blind accuracy hovers near chance (10–20%), confirming the necessity for robust cross-modal integration
• CoT prompting: notable improvements in quantitative tasks (up to +21% absolute accuracy on Gemini-1.5-Pro), but mixed impact on conceptual and hypothetical questions
• Discipline sensitivity: Strong performance in Medicine does not generalize to Chemistry, suggesting domain adaptation remains an open problem
• Primary failure modes:

1. Incorrect visual parsing (e.g., missing timestamps or experimental annotations)
2. Faulty logical reasoning steps
3. Insufficient scientific knowledge

These findings indicate a need for more advanced model architectures and training regimes to achieve truly expert-level video reasoning.

5. Methodological and Technical Details

SciVideoBench leverages advanced annotation and multimodal data integration:

• Videos, transcripts (ASR), and peer-reviewed text are time-aligned to create rich, multi-channel inputs for difficult questions
• Annotation pipeline incorporates LLM agents for question generation, evaluation, and refinement
• Distractor generation and answer selection involve careful screening by both automated systems and human experts to enforce scientific plausibility and visual grounding

LaTeX macros such as

$\DeclareMathOperator*{\argmax}{arg\,max}$

are predefined for expressing mathematical operations relevant to quantitative reasoning, though not directly used in test items.

Frame sampling parameters and temperature are standardized across model comparisons. The methodology ensures that superficial text-based cueing cannot substitute for genuine visual understanding.

6. Future Research Directions

The SciVideoBench results illuminate several key avenues for continued research:

• Enhanced spatiotemporal grounding: Model architectures must improve localized temporal reasoning and the ability to parse fine-grained events and annotations
• Domain-specific expert modeling: Infusing LMMs with specialized scientific knowledge—possibly through targeted training or retrieval mechanisms—can help bridge accuracy deficits
• Multi-modal integration: Further exploitation of synchrony between video, transcript, and text is warranted; audio modalities provide modest accuracy gains and could be extended
• Advanced reasoning techniques: Chain-of-thought and step-by-step prompting substantially improve certain categories of question-answering; however, reliability remains an open challenge
• Data and model scaling: Larger language backbones improve conceptual/hypothetical reasoning, but do not guarantee gains in quantitative performance, pointing to the need for targeted architectural innovation

A plausible implication is that future benchmarks, training procedures, and model designs will increasingly focus on domain-adaptive, multi-modal, and step-wise reasoning capabilities to meet the rigorous standards set by SciVideoBench.

7. Comparative Perspective and Impact

SciVideoBench establishes the frontier for scientific video reasoning benchmarks, contrasting sharply with prior work in general-purpose video comprehension or captioning (e.g., VideoMCC (Tran et al., 2016)) and synthetic “needle-in-a-haystack” skill isolation frameworks (Zhao et al., 13 Jun 2024). Its multidisciplinary, multi-modal, and reasoning-intensive design raises the expectations for LMMs beyond surface-level perception and simple recognition tasks. Extensive evaluation protocols and analysis components situate SciVideoBench as the authoritative reference for progress in multimodal scientific reasoning.

This approach is complemented by ongoing developments in benchmarks for intrinsic faithfulness (Zheng et al., 27 Mar 2025), granular evaluation dimensions (Huang et al., 20 Nov 2024), and scientific cross-domain model transfer (Hasson et al., 4 Jul 2025). Collectively, these efforts are driving a new generation of multimodal AI systems capable of operating at the complexity level required by border science and research applications.