MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding

Abstract: Existing medical video benchmarks primarily evaluate whether a model produces the correct answer, but rarely assess whether it answers at the right time. In real clinical settings, AI systems must decide not only what to predict, but also when to answer, defer judgment, or proactively raise alerts. This creates a critical gap between benchmark evaluation and deployment requirements. We present MedStreamBench, a benchmark for time-aware medical video understanding. MedStreamBench integrates 22 medical datasets and 5,419 QA instances across four temporal settings: retrospective, present, future, and proactive. Unlike conventional benchmarks that assume full-video access, MedStreamBench restricts models to temporally bounded evidence windows and supports both single-turn and streaming evaluation. We further introduce a proactive monitoring setting that requires models to determine whether and when clinically relevant alerts should be triggered. Beyond answer correctness, MedStreamBench evaluates temporal behavior through responsiveness and post-evidence stability. Experiments on leading general-purpose and medical vision-LLMs reveal a substantial gap between offline recognition and temporally grounded decision-making, with performance dropping markedly in streaming and proactive settings. Our benchmark is available at https://huggingface.co/datasets/Venn2024/MedStreamBench.

• The paper's main contribution is the introduction of a temporally grounded benchmark that enforces evidence window constraints for realistic streaming medical video QA.
• The methodology integrates 22 diverse medical datasets and employs four temporal reasoning modes, including retrospective, present, future, and proactive monitoring.
• Empirical results reveal significant performance drops in streaming and proactive tasks, emphasizing the necessity for temporally disciplined AI models in clinical applications.

MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding

Motivation and Problem Formulation

There is a persistent gap between current evaluation protocols for medical video understanding and the requirements imposed by real-world clinical deployment. Traditional benchmarks overwhelmingly assess whether a model produces a correct answer given offline or full-video access, neglecting the critical aspect of temporal answerability: recognizing not only what is answerable but also determining the correct timing for answering, deferring, or proactively raising alerts based on evolving visual evidence. This temporal dimension is indispensable in applications such as intraoperative guidance, screening alerts, workflow monitoring, and anomaly detection. Without this, models risk exhibiting temporally inconsistent or unsafe behavior in streaming clinical settings.

MedStreamBench directly targets this deficit by introducing a comprehensive, temporally grounded medical video question answering (QA) benchmark. Its design enforces evidence window constraints and supports single-turn and streaming protocols, incorporating retrospective, present, future, and—crucially—proactive monitoring modes. This temporal diversity forms the backbone of realistic model evaluation for medical AI intended for clinical deployment.

Figure 1: Overview of MedStreamBench, integrating heterogeneous medical video/image-sequence datasets into a unified, evidence-window-constrained, temporally diversified QA format.

Resource Construction and Benchmark Design

Dataset Integration and Pipeline

MedStreamBench harmonizes 22 medical datasets spanning diverse diagnostic and procedural contexts: laparoscopic, endoscopic, robotic, open, screening, and capsule endoscopy. The construction pipeline is structured into three main phases: temporal data preparation, label-grounded QA generation, and benchmark validation. Temporal alignment standardizes visual sources and synchronizes them with annotations (including labels, directories, or mined signals). QA item construction leverages structured labels, mined findings, and hybrid strategies to maximize coverage, while rigorous validation combines AI-assisted review and expert adjudication.

Figure 2: MedStreamBench pipeline: temporal processing, QA pair construction, and expert-aided validation form the complete benchmark generation workflow.

Every item in MedStreamBench is tightly bound to an explicit evidence window and temporal query, encoded in a unified schema capturing the visual input, question, answer, temporal specification, mode, rubric, and source metadata. This format supports reproducibility, extensibility, and precise temporal control.

Temporal Reasoning Settings

MedStreamBench formalizes four key temporal reasoning settings:

1. Retrospective QA: Assessing events in the preceding window; requires memory of recent history.
2. Present QA: Assessing the current surgical or visual state at a specific query time.
3. Future QA: Predicting upcoming events, requiring reasoning from incomplete evidence.
4. Proactive Streaming: Continuous monitoring for event detection, demanding real-time temporal discipline for alerts and abstentions.

Figure 3: Illustration of the four temporal reasoning modes: retrospective, present, future, and proactive, all tied to delineated evidence windows along the surgical video timeline.

Each mode is instantiated in both single-turn and streaming paradigms with mode-specific constraints and expectations. Proactive monitoring, a novel addition, enforces strict bounds on premature anticipation or delayed detection—key safety aspects in clinical contexts.

Evaluation Metrics and Protocol

MedStreamBench employs multi-dimensional evaluation metrics:

• Content Correctness ($C$): Semantic and evidence-grounded accuracy as per the evidence window.
• Responsiveness ($R$): Temporal accuracy of the first positive answer or alert, relative to the expected moment.
• Stability ($S$): Post-evidence consistency, penalizing reversions to abstention after an event is observable.
• Overall ($O$): Weighted aggregation, emphasizing correctness but rewarding temporally precise response, particularly in streaming and proactive settings.

Metrics are operationalized through rigorously defined rulesets (e.g., normalized exact match, rubric-based human/LLM judgment, and penalty for early or late answers). Evaluation is performed per-organ and per-mode to uncover fine-grained failure modes.

Empirical Results and Analysis

Experimental results with state-of-the-art general-purpose and medical vision-LLMs spotlight a marked drop in performance when transitioning from offline (retrospective/present) to streaming and proactive tasks. For example, Gemini-2.5-Pro achieves strong overall scores (retrospective: 0.34, present: 0.41), but performance attenuates in future (resp: 0.36) and proactive (resp: 0.45) modes. Open-source models notably lag behind, especially in responsiveness and stability for streaming tasks.

Figure 4: Organ-level aggregated model performance heatmap; dominant models differ per procedural group, highlighting heterogeneity across clinical tasks.

These findings surface fundamental limitations in current multi-modal LLM architectures: while semantic recognition in static or full-context inputs is feasible, robust temporal reasoning—wherein the model must withhold, defer, or proactively alert according to constrained evidence—is poorly realized. Proactive and future-streaming scenarios consistently expose premature answering, late/delayed detection, and instability across rounds, posing significant safety and reliability concerns for real-world integration.

Figure 5: Human spot-check agreement rates for AI-generated labels, showing 60% concordance for high-confidence items.

The organ-level stratification further identifies that best-in-class models differ across specialties—e.g., InternVL3.5-8B excels in GI capsule contexts, while HuluMed-4B is stronger on robotic surgical QA—highlighting persistent model specialization and the necessity for comprehensive temporal benchmarks.

Implications and Future Directions

Pragmatically, MedStreamBench prompts the field to reassess the notion of "answerability" from a static to a dynamic temporal discipline. This is especially pertinent in safety-critical and high-stakes applications, where the cost of premature or delayed action is nontrivial. Only with evidence-window enforced, temporally aware benchmarks can model selection be valid for downstream deployment in clinical settings.

Theoretically, these results suggest that existing vision-language architectures—fine-tuned predominantly for static or offline contexts—lack explicit mechanisms for stateful memory, temporal abstention, and evidence-driven change detection. Architectures incorporating temporal gating, dynamic abstention modules, evidence-scoring, and uncertainty quantification may be required to bridge this gap. Moreover, MedStreamBench provides a rigorous substrate for the development and diagnostic evaluation of such context-aware, temporally adaptive models.

In the longer term, MedStreamBench's extensible infrastructure (dataset expansion, schema generalization, organ/domain scalability) sets a template for future benchmarks in other domains necessitating streaming decision-making under uncertainty.

Conclusion

MedStreamBench establishes a new paradigm in medical video QA benchmarking by enforcing strict temporal constraints and comprehensive task diversity, bridging the disconnect between traditional model evaluation and real-world clinical deployment requirements. Initial results demonstrate a substantial gap between current large multimodal models’ proficiency in static recognition and their temporal-decision capabilities in streaming and proactive monitoring settings. Addressing this gap will necessitate a new class of temporally disciplined models, and MedStreamBench provides a rigorous platform to drive their development and assessment.