ExpInstruct: Scientific Commentary Dataset

• ExpInstruct is a multimodal dataset that generates precise step-level commentaries by aligning experiment video clips with procedure descriptions, scientific principles, and safety guidelines.
• The dataset uses automated ASR with GPT-4o corrections and human validation to ensure accurate temporal alignment and scientific fidelity across 21 disciplines.
• It serves as a benchmark for multimodal models, with evaluations using BLEU, METEOR, ROUGE-L, CIDEr, and BERTScore to highlight challenges in concise, context-aware narration.

Searching arXiv for the specified topic to ground the article in the relevant paper. ExpInstruct is a dataset introduced for the task of automatic commentary generation for multi-discipline scientific experiments. It is designed for step-level generation from experiment videos, where each commentary is expected to do more than narrate visible actions: it should describe the procedure and, when relevant, articulate the underlying scientific principle and include step-specific safety guidance. In the associated formulation, an experiment video $V=\{v_1,v_2,\dots,v_n\}$ and title $T$ are mapped to sequential step commentaries $y_i$ via $y_i=\arg\max_{y_i} P(y_i \mid v_i, T, y_{<i})$, making temporal conditioning explicit at the clip level (Chen et al., 13 Jul 2025).

1. Definition and research setting

ExpInstruct was introduced because existing video captioning and instructional-video corpora do not target the pedagogical demands of scientific experiments. Prior resources, as characterized in the paper, emphasize high-level summarization, daily activity narration, or generic step descriptions, whereas scientific experiment commentary requires synchronized procedural description, principled scientific explanation, and selective safety reminders. The dataset therefore targets a distinct generation problem: commentary that is instructional rather than merely descriptive, and scientifically grounded rather than visually approximate (Chen et al., 13 Jul 2025).

The paper positions this as a multimodal generation task in which the core unit is the experiment step rather than the full video. This design choice is consequential. Step alignment constrains commentary to local procedural context, while the conditioning on earlier commentaries preserves experiment-level coherence. A plausible implication is that ExpInstruct sits between video captioning, instructional narration, and domain-specific scientific explanation, but is not reducible to any of them.

2. Annotation target and commentary schema

Each ExpInstruct sample is a video clip–commentary pair derived from a longer experiment video segmented into sequential procedural steps. The commentary schema contains a mandatory procedure component and two optional components: a scientific principle field and a safety guideline field. The paper states that not every step requires principle or safety content; procedure is always central, whereas principle and safety are included only when relevant (Chen et al., 13 Jul 2025).

This selective structure differentiates ExpInstruct from uniformly verbose annotation schemes. The goal is not to append generic explanations to every step, but to generate context-sensitive commentary. The annotation prompts therefore instruct annotators to avoid redundant warnings, avoid overemphasizing basic common-sense precautions, and preserve the original structure when no principle or safety content is warranted. The result is a supervision format that couples procedural grounding with sparse, high-value scientific augmentation.

The paper also specifies intermediate JSON-like annotation structures containing fields such as summary, steps, procedure, ASR_id, and optional safety and principle. The ASR_id constraint requires each step’s transcript fragments to form a continuous sequence of positive integers without jumps or repetition across steps. This makes temporal alignment an explicit part of the annotation design, not merely a post hoc property.

3. Construction pipeline

The dataset construction pipeline combines automated processing with human validation. Source videos were collected from YouTube and Bilibili, restricted to Creative Commons-licensed content. Ten college students from different scientific majors reviewed candidate videos and filtered out material that was not a real experiment or was only theoretical. Retained videos were downloaded at 360p to 720p and 30 FPS (Chen et al., 13 Jul 2025).

Transcription was performed with WhisperX to obtain timestamp-aligned ASR with second-level boundaries. Because domain terminology is often corrupted by ASR, GPT-4o was used to correct scientific transcription errors, and multilingual transcripts were translated into English for linguistic consistency. GPT-4o was then used again to segment videos into distinct procedural steps using the aligned transcripts. This produced clip-level units corresponding to concrete experimental actions such as rinsing apparatus or adding reagents.

A further annotation phase addressed a gap in the raw source material. The paper reports that over 46% of procedures require scientific principles and over 35% require safety guidance, while the original source material contained such information in less than 5% and 3% of cases, respectively. GPT-4o was therefore prompted to add principle and safety fields when relevant. Manual verification by 10 college students from relevant fields, including chemistry and biomedical engineering, then checked temporal alignment, filtered irrelevant segments, and validated the correctness of procedures, principles, and safety guidance. Incorrect scientific annotations were revised or the sample or video was discarded, and overly cautious or generic safety advice was removed.

This pipeline means ExpInstruct is neither a purely manual corpus nor a purely synthetic one. It is AI-assisted and human-verified, with expert screening concentrated on the points most likely to affect scientific fidelity and instructional value.

4. Corpus composition and statistics

ExpInstruct spans 21 scientific subjects across three core disciplines: science, healthcare, and engineering. The paper reports 7,714 video clips from 1,011 experiment videos, split by distinct videos into 6,991 training samples and 723 testing samples. This split design is intended to support generalization evaluation without train–test overlap at the source-video level (Chen et al., 13 Jul 2025).

Statistic	Total	Discipline breakdown
Video clips	7,714	Science 3,565; Healthcare 2,285; Engineering 1,864
Experiment videos	1,011	—
Train / test split	6,991 / 723	Split by distinct videos
Average duration	33.80 s	Science 20.40 s; Healthcare 46.60 s; Engineering 44.02 s
Steps per video	7.63	Science 5.89; Healthcare 10.07; Engineering 10.41
Average text length	53.99 words	Science 34.05; Healthcare 90.77; Engineering 46.05
Principle rate	52.04%	Science 41.65%; Healthcare 60.79%; Engineering 62.35%
Safety guideline rate	36.42%	Science 25.83%; Healthcare 52.91%; Engineering 36.02%

The paper further reports average text lengths by content type: 36.78 words for procedure, 19.34 for principle, and 16.89 for safety guideline. Video clips range from 3 to 120 seconds, average video duration is around 33 seconds, and the corpus contains 25,078 unique words. These statistics indicate that ExpInstruct is not only multimodal and step-aligned but also lexically richer than generic video captioning datasets, which is consistent with its scientific domain specificity.

The disciplinary distribution is also informative. Science accounts for 46.2% of clips, healthcare 29.6%, and engineering 24.2%. Healthcare and engineering clips are longer on average and have higher principle rates than science clips, suggesting denser verbalization of explanatory content in those subdomains.

5. Benchmark role and empirical findings

ExpInstruct functions both as a training corpus and as a benchmark. In the paper, it is used to train and evaluate ExpStar, while also serving as the evaluation substrate for 14 leading LMMs, including proprietary systems such as GPT-4o, GPT-4o-mini, Gemini-2.0-Flash, and GLM-4V-Plus, and open-source models such as Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen2-VL-7B, InternVL-2.5-4B, InternVL-2.5-8B, LLaVA-Next-Video-7B, VideoNarrator-7B, VideoLLaMA2-7B, Video-LLaVA-7B, and CogVLM2-Video-8B (Chen et al., 13 Jul 2025).

Automatic evaluation uses BLEU-1 through BLEU-4, METEOR, ROUGE-L, CIDEr, and BERTScore. Human evaluation is conducted on 200 test samples with 5 annotators using Fluency, Instructional clarity, and Scientific appropriateness, each on a 2-point scale. On this benchmark, ExpStar obtains the best reported automatic scores, including BLEU-1 46.49, BLEU-2 23.92, BLEU-3 13.53, BLEU-4 6.37, METEOR 29.28, ROUGE-L 28.89, CIDEr 40.96, and BERTScore 62.80. It also achieves the best human scores: Fluency 1.63, Instructional clarity 1.55, and Scientific appropriateness 1.51.

For ExpInstruct itself, these benchmark results are significant because they operationalize the dataset’s difficulty profile. The paper emphasizes that general LMMs can often narrate visible procedure, but they remain weak at concise scientific explanation and step-specific safety guidance. The benchmark therefore measures more than caption quality; it measures pedagogically relevant scientific commentary generation.

6. Distinctive challenges, misconceptions, and limitations

A common misconception would be to treat ExpInstruct as a standard video-captioning dataset with extra metadata. The paper explicitly argues otherwise. The dataset is challenging because it requires concurrent visual-step understanding, temporal alignment, scientific reasoning, safety reasoning, and concise synchronization with the pace of the experiment. General models are described as tending toward outputs that are too long and generic, especially in safety commentary, while some models omit safety entirely or over-generate principles and warnings at nearly every step (Chen et al., 13 Jul 2025).

The dataset’s qualitative examples reinforce this point. In a chemical reaction experiment involving sodium hydroxide and copper sulfate, appropriate commentary must jointly describe the reaction step, articulate the chemistry, and add relevant safety information. The appendix further includes cases from optics, mechanics, biology, and material engineering, together with at least one failure case, indicating that the difficulty is not confined to chemistry.

ExpInstruct also has limitations implicit in its construction. It depends on ASR quality, GPT-4o-assisted correction and annotation, and student-based manual verification rather than subject-specialist faculty review. This suggests that some residual annotation noise may be difficult to eliminate completely, although the paper’s filtering and revision procedures are intended to reduce it. Another plausible implication is that the benchmark may reward models that are good at selective explanation and selective safety generation, not merely models that maximize descriptive verbosity.

7. Position within scientific multimodal instruction

Within the broader literature, ExpInstruct defines a new multimodal instruction regime centered on scientific experiments rather than everyday tasks. Its principal contribution is to make experiment commentary generation a concrete, benchmarkable problem with structured supervision: procedure as the default channel, principle and safety as optional but scientifically meaningful augmentations. The paper characterizes it as the first dataset tailored to experiment commentary generation and reports over 7K step-level commentaries across 21 subjects and 3 disciplines (Chen et al., 13 Jul 2025).

Its broader significance lies in the type of supervision it introduces. The dataset does not ask a model merely to say what is visible; it asks the model to explain why a step matters scientifically and when it is risky. This suggests a shift from generic multimodal narration toward domain-aware instructional generation. In that sense, ExpInstruct is best understood as a dataset for pedagogically grounded scientific video commentary, with benchmark design and annotation structure intended to support models that can narrate, explain, and warn in a coordinated way.