M3-Med: Medical Multi-hop Video Benchmark

• M3-Med is a benchmark for multi-lingual, multi-modal, and multi-hop reasoning in medical instructional videos that combines subtitles, visual evidence, and structured medical knowledge.
• It defines two temporal grounding tasks (TAGSV and TAGVC) to evaluate AI's ability to localize answers through multi-step inference rather than mere text matching.
• The dataset features bilingual annotations, expert-curated questions, and per-video knowledge graphs, exposing a significant human–model performance gap on complex queries.

M$^3$-Med is a benchmark for Multi-lingual, Multi-modal, and Multi-hop reasoning in medical instructional video understanding, designed to evaluate whether AI systems can localize temporally grounded answers in videos while integrating subtitles, visual evidence, and structured medical knowledge rather than relying on surface-level text matching alone (Liu et al., 6 Jul 2025). It is organized around medical instructional videos such as first-aid demonstrations, rehabilitation exercises, and dental procedures, and it pairs these videos with natural-language questions, temporal answer spans, subtitles, and per-video knowledge graphs. The benchmark is bilingual in Chinese and English, defines two temporal grounding tasks—Temporal Answer Grounding in a Single Video (TAGSV) and Temporal Answer Grounding in a Video Corpus (TAGVC)—and was constructed to expose a substantial performance gap between current models and human experts, especially on complex multi-hop questions.

1. Motivation and problem setting

M$^3$-Med belongs to the broader line of work on video-language understanding, but it is specialized for medical instructional videos, a format that functions as a primary medium for medical education and clinical skills training, patient self-care guidance and health literacy, and procedural review for explanation to non-experts (Liu et al., 6 Jul 2025). In this setting, video understanding requires more than object recognition or keyword retrieval. The target capability is procedural understanding over time: linking spoken or subtitled explanations to visual actions, tracking anatomical structures and tools, and resolving questions whose answers emerge only after multiple inferential steps.

The benchmark is motivated by two limitations that the authors identify in existing medical video question-answering resources such as MedVidQA and HealthVidQA. The first is Linguistic Singularity: prior datasets are largely English-only, despite the fact that medical education and patient information are deeply multilingual and real deployments may require cross-lingual generalization. The second is Shallow Reasoning: many earlier questions can be solved through subtitle matching without genuine inspection of the visual stream. In medicine, that shortcut is particularly problematic because instructions are frequently distributed across narration and demonstration, and correct interpretation may depend on visually observing hand placement, instrument manipulation, body posture, or temporal sequencing. M$^3$-Med is therefore constructed to stress deeper, cross-modal, multi-step reasoning in both Chinese and English rather than simple transcript retrieval.

2. Corpus composition and annotation pipeline

M$^3$-Med consists of thousands of medical instructional videos, natural-language questions in Chinese or English, temporal answer spans, high-quality subtitles, and a dedicated knowledge graph for each video (Liu et al., 6 Jul 2025). The dataset was collected through keyword-based search on YouTube using medical and health-related terms taken from wikiHow and their Chinese translations, supplemented with medically relevant categories sampled from HowTo100M, followed by manual screening for instructional relevance, duplicate removal, and topic diversity.

The benchmark scale reported in Table 2 is as follows:

Subset	Videos	Questions
Chinese	1,628	3,208 simple + 2,746 complex
English	2,120	3,496 simple + 3,297 complex
Total	3,748	12,747

The total question set is approximately balanced between simple and complex instances: 6,704 simple questions and 6,043 complex questions. The Chinese and English portions are not described as full parallel annotations of the same videos; rather, they form two subsets of a multilingual benchmark.

Annotation uses a three-tier team: professional doctors write questions, medical students mark temporal answer spans, and senior experts supervise quality control and arbitration. Subtitles are produced automatically with Whisper and then normalized through English lemmatization with NLTK and Chinese tokenization with Jieba. These subtitles serve simultaneously as model input, as material for lexical and semantic analyses, and as input to knowledge graph construction. For temporal span quality control, 10% of samples are double-labeled and evaluated with Cohen’s Kappa; annotations with $\kappa < 0.25$ are flagged and corrected by supervisors.

3. Knowledge graphs and multi-hop question design

A central design element of M$^3$-Med is the construction of a per-video Knowledge Graph (KG) that captures medically relevant entities and relations and then guides multi-hop question writing (Liu et al., 6 Jul 2025). Candidate entities are extracted from subtitles using Qwen2.5 for Chinese and GPT-4o for English. These entities include body parts, tools, diseases, and actions. Their visual presence is then grounded in nearby frames with Grounding DINO, after which annotators filter irrelevant items, merge duplicates or synonyms, and manually specify relations such as tool–body-part or exercise–muscle relations.

This KG-centered pipeline is used not only as optional model input but also as a structural constraint on question creation. For each video, expert annotators write an approximately balanced set of simple and complex questions. Simple questions are usually answerable from one textual span or a local visual cue. Complex questions must involve multiple KG nodes and cross-modal evidence. The paper characterizes multi-hop reasoning as a chain of evidence $e_1, e_2, \ldots, e_k$, where each $e_j$ is a textual or visual snippet, or a KG node or edge, and complex questions require $k \ge 2$ hops. In practical terms, a model may need to identify a key entity in text, locate the corresponding action or object in frames, and then synthesize the textual and visual cues to infer the answer.

The question inventory spans factual or descriptive prompts, procedural or instructional prompts, temporal or sequential prompts, causal or functional prompts, and visually grounded prompts. The important distinction, however, is the simple/complex split. The paper reports that complex questions have much lower word-level IoU with the corresponding subtitle spans than simple questions or prior datasets, while still showing high semantic and visual relevance through S-BERT and AltCLIP analyses. This indicates that complex items are grounded in the underlying content but are resistant to lexical shortcuts.

4. Task definitions and evaluation protocol

M$^3$-Med frames answering as temporal grounding, not text generation (Liu et al., 6 Jul 2025). The output is a temporal interval rather than a free-form answer or multiple-choice label.

In TAGSV, the input is a question $^3$0 and a single video $^3$1. The target is a ground-truth temporal segment

$^3$2

Given a predicted segment

$^3$3

the per-question score is temporal Intersection-over-Union:

$^3$4

Reported metrics are thresholded accuracies at $^3$5,

$^3$6

and mean IoU,

$^3$7

In TAGVC, the input is a question and a corpus of videos. The model must retrieve the correct video and localize the answer segment within it. For each question, the system returns up to 50 ranked candidates

$^3$8

A candidate receives score

$^3$9

and corpus-level evaluation uses

$^3$0

The overall TAGVC score is the arithmetic mean of Top-1, Top-10, and Top-50 IoU.

Because the answer is a grounded interval, there are no BLEU or ROUGE metrics. When LLMs are evaluated, they are prompted to output timestamps in a specified format such as seconds or hh:mm:ss.

5. Baselines, ablations, and empirical findings

The paper evaluates both specialized temporal grounding systems and general large models on M$^3$1-Med (Liu et al., 6 Jul 2025). The specialized baselines include MutualSL, PBL, CCGS, and FMALG. The large-model baselines are GPT-4o in text-only usage with subtitles and KG, and Qwen2.5-VL as a multimodal model with video frames, text, and KG. Human challengers not involved in dataset construction provide an upper bound.

Ablation settings are organized around modality access: Video + Subtitles + KG (“Full Input”), Video + Subtitles, Subtitles + KG, and Subtitles Only. The reported qualitative hierarchy is consistent across tasks: Full Input > Video + Subtitles > Subtitles + KG > Subtitles Only. This shows that video is critical for localization and visually grounded reasoning, while KGs provide a moderate boost over text-only input but do not compensate for missing visual evidence.

The principal empirical conclusion is a substantial human–model gap. Human performance reaches approximately TAGSV mIoU $^3$2–$^3$3 across languages and difficulties, and TAGVC Average top-k IoU $^3$4–$^3$5. All tested models fall far below these levels. The gap widens on complex questions, where performance drops sharply across all architectures and input settings. This confirms that the multi-hop design is not merely nominal; it produces a materially harder evaluation regime.

The paper also reports that zero-shot GPT-4o and Qwen2.5-VL often outperform specialized temporal grounding models in average IoU, suggesting strong emergent reasoning and generalization abilities, but these models exhibit distinctive failure modes: wrong time format, single timestamps instead of intervals, non-integer seconds, extra text, occasional refusals such as “Null” or “Cannot answer,” and hallucinated timestamps not aligned with the video. Performance on Chinese and English is roughly comparable for modern multilingual models, which the authors attribute to multilingual pretraining, but the absolute gap to human experts remains large in both languages.

6. Significance, limitations, and later developments

M$^3$6-Med functions as a medically curated stress test for video-language systems in a high-stakes domain (Liu et al., 6 Jul 2025). It contributes bilingual coverage, per-video KGs, explicit multi-hop question construction, and two temporal grounding tasks that approximate realistic workflows such as navigating educational video repositories and retrieving visually grounded procedural answers. The benchmark is publicly available and also serves as the basis of the NLPCC 2023–2025 shared tasks, including CMIVQA, MMIVQA, and M4IVQA.

The paper also identifies several limitations. The videos come primarily from YouTube and HowTo100M, which creates potential copyright constraints for downstream commercial use. Manual screening cannot guarantee factual correctness of every instructional video, so hallucination or misinformation risks remain. Subtitle generation scales readily with Whisper, but high-quality KG construction and question writing remain semi-automated rather than fully automatic. The domain scope is restricted to instructional videos rather than in-the-wild clinical video streams such as surgical endoscopy or ICU monitoring.

Subsequent work extends the benchmark family rather than replacing it. Med-CRAFT introduces M$^3$7-Med-Auto, described as a fully automatically regenerated version of M$^3$8-Med that reuses the same TAGSV/TAGVC task definitions and the same Simple vs. Complex reasoning taxonomy while generating query–answer pairs through deterministic traversal of a Spatiotemporal Knowledge Graph over real surgical videos (Liu et al., 30 Nov 2025). That later work treats the original M$^3$9-Med as the expert-curated reference point and repositions it as the manually annotated member of a broader benchmark family. A plausible implication is that M$^3$0-Med has become not only a benchmark in its own right but also a template for scalable, KG-mediated evaluation design in medical video reasoning.