LocateBench: Evaluating the Locating Ability of Vision Language Models

Abstract: The ability to locate an object in an image according to natural language instructions is crucial for many real-world applications. In this work we propose LocateBench, a high-quality benchmark dedicated to evaluating this ability. We experiment with multiple prompting approaches, and measure the accuracy of several large vision LLMs. We find that even the accuracy of the strongest model, GPT-4o, lags behind human accuracy by more than 10%.