Quantitative evaluation of visual referring prompting (VRP) on GPT-4V for biomedical images

Conduct a thorough, quantitative assessment of visual referring prompting (VRP) on GPT-4V(ision)’s understanding of biomedical images to determine its impact on multimodal image comprehension.

Background

Visual referring prompting (VRP) augments textual prompts by editing images to include visual pointers, potentially guiding large multimodal models like GPT-4V to attend to relevant regions.

Although initial case studies and a benchmark (VRPTEST) exist, a rigorous and quantitative evaluation specific to biomedical imaging tasks has not yet been carried out, leaving the practical benefit of VRP in this domain uncertain.