- The paper introduces a benchmark of 340 full-length clinical videos, 759 hours, and 1,253 evidence-grounded questions spanning diagnostic examinations and surgery, with evidence just 0.166% of relevant frames on average.
- The paper finds that current multimodal language models struggle with sparse evidence retrieval and interpretation: Gemini-3.1-Pro-Preview achieves the best overall accuracy at 41.1%, while oracle evidence frames still leave models below 40% on semantic tasks.
- The paper shows that more frames and generic sampling do not reliably improve performance because attention drifts across redundant visual content, highlighting the need for clinically informed retrieval that preserves both local evidence and global procedural context.
Overview
"MedHorizon: Towards Long-context Medical Video Understanding in the Wild" (2605.06537) introduces an evaluation benchmark designed to test multimodal LLMs (MLLMs) on full-length clinical procedure videos rather than pre-localized images, clips, or trimmed segments. The central premise is that real clinical review—assessing colonoscopy withdrawal quality, verifying ultrasound standard views, or tracking surgical phases—requires reasoning over entire procedures in which decisive evidence is temporally sparse, spatially subtle, and context dependent. The authors argue that existing medical video benchmarks largely assume evidence has already been localized, leaving the retrieval-before-reasoning capability under-tested.
The benchmark comprises 340 public full-length videos from 8 source datasets, spanning 759 decoded hours (with the longest study at 37.2 hours), across 7 organs and two clinical scenarios (diagnostic examination and surgery). It provides 1,253 evidence-grounded four-way multiple-choice questions. The defining statistic is evidence sparsity: on the localizable subset of 1,035 questions, the mean question-level evidence ratio is 0.166%—roughly 1.7 evidence frames per 1,000 decoded frames, which the authors state is 169× sparser than general benchmarks such as QVHighlights.
Benchmark construction
Construction follows an evidence-first pipeline. Each question is generated from an explicit source annotation—phase intervals, lesion records, instrument onsets, study-level labels—rather than from templates alone. Candidate items are filtered automatically for temporal validity and ambiguity (81 invalid candidates removed from a pool of 1,334), rewritten into natural language using GPT-5.4, and given strengthened distractors drawn from clinically plausible, temporally nearby, or visually confusable labels in the same task family. All 1,253 retained items then pass human audit by three trained (non-physician) reviewers checking video usability, question well-formedness, evidence support, and label consistency, with senior adjudication for disputes. Answer letters are balanced via option shuffling (A: 309, B: 303, C: 319, D: 322).
The capability taxonomy spans two dimensions. Fine-grained understanding tasks include phase recognition (PR), step discrimination (SD), region localization (RL), bowel preparation quality (BPQ), largest-lesion selection (LLS), and lesion binding to site, histology, or size. Multi-hop semantic reasoning tasks include instrument recognition (IR), workload ranking (WR), ranked step retrieval (RSR), count-proportion reasoning (CPR), and lesion analysis (LA), stratified by reasoning depth up to three-hop aggregation. The authors are explicit that per-task denominators are highly uneven (PR has 517 items; WR, RSR, and CPR have 26 each), so small-task accuracies should be read as directional diagnostics rather than stable rankings.
Compared with recent medical video benchmarks—PitVQA, OphNet, SurgPub-Video, MedVidBench, SUREON—MedHorizon is the only one in its comparison that jointly covers surgery and examination, preserves full-procedure context, exhibits extreme evidence sparsity, and requires multi-hop reasoning, while exceeding the total duration of all listed benchmarks by a wide margin (759 hours versus at most 31.5 hours).
Main results
The evaluation covers closed-source MLLMs, open-source video MLLMs, specialized long-video methods (AKS, ViLAMP, WFS-SB, VideoLLaMA3, LongVA, VideoChat-Flash), and medical-domain MLLMs (Lingshu, Hulu-Med, HuatuoGPT-Vision, MedGRPO), each under its closest reproducible official inference setting with uniform-over-full-video subsampling (never prefix truncation) when caps apply. The strongest model, Gemini-3.1-Pro-Preview, reaches only 41.1% overall accuracy; most open-source and medical models cluster in the mid-20s. Because random guessing yields 25%, several models are barely above chance.
| Model category |
Representative result (overall) |
| Gemini-3.1-Pro-Preview |
41.1% (best) |
| GPT-5.4 |
32.6% |
| LLaVA-Video-72B |
29.4% |
| WFS-SB (specialized sampler) |
28.1% |
| Lingshu-32B (medical) |
27.9% |
| MedGRPO (medical) |
20.1% |
Per-task patterns are informative: tasks benefiting from procedural regularities or salient cues (BPQ, RL, IR, WR) show higher peaks, while lesion-centric binding tasks (LLS, Site, Hist, Size) are unstable across models, and aggregation tasks (CPR, LA) remain hardest. The implication drawn is that the binding constraint is the coupling of sparse evidence localization with cross-temporal reasoning, not raw context capacity.
Four diagnostic findings
Frame scaling is unreliable (RQ1). Accuracy does not increase monotonically with more sampled frames. Qwen3-VL-8B oscillates within a narrow band, improving at 128–256 frames but dropping at 512; Lingshu-7B peaks at 32 frames and then declines; MedGRPO degrades monotonically from 4 to 512 frames. The interpretation is that added frames introduce near-duplicate anatomy and low-information segments that dilute useful signal—so the bottleneck is evidence-aware selection and compression, not temporal coverage.
Retrieval and interpretation are dual bottlenecks (RQ2). A four-setting diagnostic (random, text-only, uniform frames, oracle GT-images) shows text-only accuracy mirrors chance, ruling out language shortcuts, while uniform video input yields limited gains—MedGRPO actually falls to 13.6% on semantic tasks under uniform sampling. The uniform-to-oracle gap isolates a retrieval deficit (MedGRPO: 13.6%→33.9%; Lingshu-7B: 22.9%→38.0% on semantic tasks). Critically, even with oracle evidence frames, no model exceeds 40%, demonstrating that retrieval is only the first failure mode; post-retrieval clinical interpretation remains a substantial, independent bottleneck.
Weak procedural reasoning and attention drift (RQ3). Temporal-order perturbations (reverse, block shuffle, local shuffle) reduce Qwen3-VL-8B only from 26.5% to 24.2–24.5%, indicating models exploit chronology only weakly. Attention analysis—aggregating query-to-visual attention over 128 frames at frame and patch level—shows attention is diffuse rather than concentrated: Qwen3-VL-8B and Lingshu-7B require 76.7% and 85.6% of temporal units to accumulate 90% of attention, while maintaining frame similarity near 0.90. The paper names this phenomenon attention drift: under strong spatiotemporal redundancy, attention spreads over visually repetitive but clinically non-decisive content instead of isolating sparse answer-relevant outliers. Notably, medical-domain training (MedGRPO) slightly improves frame discriminability but not evidence-selective attention.
Generic sampling only partially helps (RQ4). Uniform sampling is stable but evidence-agnostic; change-focused samplers improve short-window tasks (RL, Size) but hurt workflow-sensitive tasks (PR), showing that visual dynamism does not imply clinical relevance. Specialized methods at 128-frame budgets outperform general MLLMs (AKS 27.3%, ViLAMP 27.0%, WFS-SB 28.1%), with localized gains such as WFS-SB reaching 32.8% on Site, but they underperform on tasks requiring global procedural continuity. The conclusion is that sampling can reduce redundancy but cannot substitute for clinically evidence-aware retrieval that preserves global structure.
Limitations and open questions
The authors state several constraints plainly. The benchmark is bounded by public-video diversity and upstream annotation quality; question construction is an audit of existing professional annotations by non-physician reviewers rather than independent clinical labeling, and study-level items (e.g., ultrasound contractions, bowel preparation) are excluded from sparsity estimation because their support is intentionally global. The multiple-choice format cannot capture free-form clinical reporting. The main results table is deliberately not a matched-budget leaderboard, since models run under their official interfaces with differing frame budgets, so cross-model comparisons conflate architecture with input protocol. Task-level denominators are small for several reasoning tasks. Open questions include whether evidence-aware temporal retrieval can be learned without dense medical supervision, and whether attention concentration on sparse clinical outliers can be induced in architectures trained on redundant natural video.
Conclusion
MedHorizon establishes a full-procedure evaluation setting for medical video understanding in which evidence is 169× sparser than in general long-video benchmarks. The consistent result—41.1% best accuracy, chance-level text-only performance, sub-40% accuracy even with oracle evidence, non-monotonic frame scaling, and diffuse attention—indicates that current MLLMs lack both reliable sparse-evidence retrieval and grounded clinical interpretation over complete procedures. The benchmark, released with evidence metadata, evaluation scripts, and provenance-preserving source pointers, provides a concrete testbed for methods that must locate weak evidence within redundant clinical streams and aggregate it across temporal and anatomical context.