Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine (2401.08396v4)

Abstract: Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows.

Understanding Skin Manifestations: The Case of Gingival Melanoma

Background

Skin changes and lesions can be signs of various health issues, and sometimes they can be the symptomatic presentation of rare and serious diseases. A recent clinical case highlights the importance of early detection and treatment of gingival melanoma, an uncommon and aggressive form of malignancy originating in the oral cavity's melanocytes.

Case Presentation

In the noted case, a 61-year-old woman presented to a medical clinic with concerns about the rapidly expanding discoloration along her gums—initially noticed over a year ago. Upon examination, the discoloration was dark and concentrated in irregular patches near the base of the gums.

Diagnosis and Treatment

A diagnosis of gingival melanoma was made after clinical evaluation and led to further imaging, which thankfully revealed no lymph-node involvement or distant metastases. The patient underwent successful surgical resection of the lesion. Four months post-surgery, she showed no signs of recurrence.

Differentiating from Other Conditions

The condition shared similarities with other potential diagnoses such as amalgam tattoo, Kaposi's sarcoma, oral melanoacanthoma, and physiological pigmentation—each with its distinct characteristics. However, the rapid expansion of the discoloration and its specific appearance were indicative of gingival melanoma. Its dark pigmented lesions and aggressive growth pattern distinguished it as the leading diagnosis.

Conclusion

This case underscores the critical nature of medical vigilance and patient awareness, particularly when it comes to changes in skin appearance. For healthcare professionals, it highlights the intricacy of skin lesion diagnosis and the importance of considering a broad spectrum of potential conditions. As for patients, it reaffirms the significance of promptly seeking medical advice when noticing changes in one’s body, as early detection can be lifesaving, especially with conditions like gingival melanoma.