Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery (2503.23130v3)

Abstract: The DeepSeek models have shown exceptional performance in general scene understanding, question-answering (QA), and text generation tasks, owing to their efficient training paradigm and strong reasoning capabilities. In this study, we investigate the dialogue capabilities of the DeepSeek model in robotic surgery scenarios, focusing on tasks such as Single Phrase QA, Visual QA, and Detailed Description. The Single Phrase QA tasks further include sub-tasks such as surgical instrument recognition, action understanding, and spatial position analysis. We conduct extensive evaluations using publicly available datasets, including EndoVis18 and CholecT50, along with their corresponding dialogue data. Our empirical study shows that, compared to existing general-purpose multimodal LLMs, DeepSeek-VL2 performs better on complex understanding tasks in surgical scenes. Additionally, although DeepSeek-V3 is purely a LLM, we find that when image tokens are directly inputted, the model demonstrates better performance on single-sentence QA tasks. However, overall, the DeepSeek models still fall short of meeting the clinical requirements for understanding surgical scenes. Under general prompts, DeepSeek models lack the ability to effectively analyze global surgical concepts and fail to provide detailed insights into surgical scenarios. Based on our observations, we argue that the DeepSeek models are not ready for vision-language tasks in surgical contexts without fine-tuning on surgery-specific datasets.

• The paper demonstrates DeepSeek-V3's efficiency in handling vision-language tasks in robotic-assisted surgery, outperforming GPT-4o in simple recognition tests.
• The paper's methodology employs Single Phrase QA, Visual QA, and Detailed Description to comprehensively assess AI performance in surgical contexts.
• The paper highlights the need for improved prompt engineering and domain-specific fine-tuning to enhance spatial reasoning and detailed context understanding.

Can DeepSeek-V3 Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery

Introduction

The paper explores the potential of DeepSeek-V3, a Multimodal LLM (MLLM), for understanding and reasoning in robotic-assisted surgery (RAS) contexts. DeepSeek-V3 was benchmarked against GPT-4o, focusing on its ability to manage Single Phrase QA, Visual QA, and Detailed Description tasks. The research aims to assess the applicability of DeepSeek-V3 to the RAS domain and whether it can function effectively as an assisting AI in surgical environments.

Methodology

Evaluation Paradigms

The analysis was divided into three paradigms:

1. Single Phrase QA: Focused on simple, concise queries related to surgical instruments, organs, and their positions. DeepSeek-V3 needed to provide quick and accurate responses based on visual prompts.
2. Visual QA: Evaluated the ability to carry out natural-sounding conversations about surgical images, requiring the generation of short and contextually relevant answers.
3. Detailed Description: Assessed the model's capability to provide comprehensive summaries of the entire surgical scene, demanding an extensive understanding of the surgical environment.

   Figure 1: Qualitative examples of Single Phrase QA, Visual QA, and Detailed Description on the EndoVis18 dataset.

Results and Analysis

Performance on Single Phrase QA

DeepSeek-V3 outperformed GPT-4o for simple recognition tasks within surgical contexts. However, both models struggled with tasks involving spatial analysis and surgical motions, highlighting limitations in reasoning and spatial understanding within static images. This limitation suggests a need for further domain-specific training to improve precision in real-world applications.

Figure 2: Qualitative examples of Single Phrase QA, Visual QA, and Detailed Description on the EndoVis18 dataset.

Performance on Visual QA

DeepSeek-V3 excelled in Visual QA tasks, particularly in metrics that measured understanding and lexical coverage, like CIDEr and ROUGE-1. Despite being trained on less computational resources, it managed to closely rival GPT-4o in performance, shedding light on its potential efficiency in efficiently capturing context-focused insights in the RAS domain.

Performance on Detailed Description

Both DeepSeek-V3 and GPT-4o faced challenges due to the length and complexity of responses required, evidenced by low BLEU and CIDEr scores. However, GPT-4o produced more contextually grounded descriptions. DeepSeek-V3 often generated abstract results due to its reliance on clear prompts, a limitation that constrains its effectiveness in providing detailed context comprehension without explicit guiding instructions.

Conclusion

DeepSeek-V3 demonstrates promising results in certain surgical AI tasks, excelling especially in Visual QA scenarios due to its ability to maintain computational efficiency while achieving competence in narrative synthesis. However, its effectiveness in handling complex surgical scenarios is restricted by its current limitations in spatial reasoning and motion interpretation. Future efforts should focus on enhancing prompt engineering, harnessing domain-specific dataset fine-tuning, and reducing computational costs through strategies like Mixture of Experts and model distillation. These improvements are vital for making MLLMs like DeepSeek-V3 valuable tools in surgical contexts, potentially aiding in the development of efficient AI companions in robotic-assisted surgeries.