Extent of LVLMs’ capability to meet diverse clinical demands

Determine the extent to which large vision-language models (including general-purpose models such as DeepSeek-VL, GPT-4V/GPT-4o, Claude3-Opus, Gemini, and Qwen-VL, as well as medical-specific models such as MedDr, LLaVA-Med, Med-Flamingo, RadFM, and Qilin-Med-VL-Chat) can accommodate the diverse demands encountered in real-world clinical scenarios across modalities, tasks, departments, and perceptual granularities.

Background

Large Vision-LLMs (LVLMs) have shown promising performance on selected medical vision–language tasks, and both general-purpose and medical-specific systems are being actively evaluated and deployed. However, clinical environments present highly diverse demands across institutions, departments, imaging modalities, and task types, creating uncertainty about how broadly these models can satisfy real-world needs.

The paper introduces GMAI-MMBench to provide a comprehensive evaluation across 39 modalities, 18 clinical VQA tasks, 18 departments, and multiple perceptual granularities. This benchmark is motivated by the need to clarify the true breadth of LVLM capabilities with respect to clinical demands and to quantify how well current systems generalize to practical clinical settings.

References

However, it remains unclear to what extent these LVLMs can accommodate the diverse demands in real clinical scenarios.

— GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI  (2408.03361 - Chen et al., 2024) in Abstract; Introduction

A preregistered, multisite field study comparing unaided-first vs. immediate guidance will test whether this design scales across scanners, protocols, and training populations.

— Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration  (2608.30352 - Li et al., 31 Aug 2026) in Section 5, Conclusion

The obvious extension to this study is to evaluate the same protocol on the top-tier flagship models, which we leave to future work.

— Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images  (2609.24190 - Dolphin et al., 21 Sep 2026) in Discussion and Conclusions, final paragraph

Our benchmarking of state-of-the-art 3D medical VLMs shows that existing models remain close to chance on fine-grained anatomical recognition and struggle to couple CT-derived morphology with PET-derived metabolic signals, identifying joint structural-metabolic reasoning as a critical open problem for the pattern recognition community.

— MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT  (2609.03690 - Zheng et al., 3 Sep 2026) in Section Discussion and Conclusion, concluding paragraph