GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation

Abstract: Visual grounding, localizing objects from natural language descriptions, represents a critical bridge between language and vision understanding. While multimodal LLMs (MLLMs) achieve impressive scores on existing benchmarks, a fundamental question remains: can MLLMs truly ground language in vision with human-like sophistication, or are they merely pattern-matching on simplified datasets? Current benchmarks fail to capture real-world complexity where humans effortlessly navigate ambiguous references and recognize when grounding is impossible. To rigorously assess MLLMs' true capabilities, we introduce GroundingME, a benchmark that systematically challenges models across four critical dimensions: (1) Discriminative, distinguishing highly similar objects, (2) Spatial, understanding complex relational descriptions, (3) Limited, handling occlusions or tiny objects, and (4) Rejection, recognizing ungroundable queries. Through careful curation combining automated generation with human verification, we create 1,005 challenging examples mirroring real-world complexity. Evaluating 25 state-of-the-art MLLMs reveals a profound capability gap: the best model achieves only 45.1% accuracy, while most score 0% on rejection tasks, reflexively hallucinating objects rather than acknowledging their absence, raising critical safety concerns for deployment. We explore two strategies for improvements: (1) test-time scaling selects optimal response by thinking trajectory to improve complex grounding by up to 2.9%, and (2) data-mixture training teaches models to recognize ungroundable queries, boosting rejection accuracy from 0% to 27.9%. GroundingME thus serves as both a diagnostic tool revealing current limitations in MLLMs and a roadmap toward human-level visual grounding.

• The paper introduces GroundingME, a high-fidelity evaluation framework that reveals MLLMs’ deficiencies in visual grounding, especially in negative case rejection.
• It details a multi-stage curation process combining automated and manual annotation to rigorously test fine-grained discrimination, spatial reasoning, and small-object detection.
• The study shows that even state-of-the-art MLLMs struggle with rejection tasks, with methods like test-time trajectory selection and negative-sample augmentation offering modest improvements.

GroundingME: A High-Fidelity Evaluation Framework for Visual Grounding in MLLMs

Motivation and Problem Statement

Visual grounding, or Referring Expression Comprehension (REC), constitutes a cornerstone for linking vision and language, serving real-world applications where robots or interactive systems must localize entities based on natural language. While Multimodal LLMs (MLLMs) have achieved strong performance on canonical visual grounding datasets, there exists an unresolved question: do these models truly exhibit human-like visual grounding, or do they exploit dataset biases and syntactic shortcuts? Existing benchmarks, such as RefCOCO(g)/RefCOCO+ and CLEVR-Ref+, are saturated and frequently limited to unambiguous or simplistic cases.

The GroundingME benchmark addresses these deficiencies by introducing a challenging, multi-dimensional testbed targeting fine-grained discrimination, complex spatial reasoning, robust small-object detection, and—crucially—rejection when queries are visually ungroundable. This diagnostic perspective is currently absent in the literature and critical for real-world deployment, where hallucinated grounding can induce severe safety and trust issues.

Figure 1: Comparative illustration of existing benchmarks (top), which are either simplistic or shortcut-prone, versus GroundingME (bottom), which systematically stresses four axes of grounding difficulty.

Construction of GroundingME

The curation process for GroundingME is engineered to maximize complexity and diversity. The dataset construction proceeds in three strictly-controlled stages:

1. Bounding Box Annotation: For SA-1B imagery, semi-automated pipelines employing RAM++ and GroundingDINO with custom NMS generate bounding boxes, emphasizing high intra-class distractor density. For ultra-high-resolution images from HR-Bench, manual annotation is applied to guarantee integrity in small-object cases.
2. Description Generation: Gemini-2.5-Flash generates initial textual queries, leveraging both visual prompting and cropping strategies, then human annotators refine for specificity, uniqueness, and factual accuracy.
3. Manual Selection and Refinement: Annotators filter out simplistic examples, enforce a minimum class-instance threshold, and then further refine descriptions for alignment to subcategory challenge criteria. Rejection cases are fabricated by intentional attribute mismatches.

   Figure 2: Overview of the data construction pipeline, demonstrating the interplay between automation and expert manual curation.

This yields a dataset of 1,005 instances, organized into a strict two-level taxonomy: four primary axes (Discriminative, Spatial, Limited, Rejection), comprising 12 L-2 subcategories. GroundingME consists of 241 object classes, high scene clutter (Intra-Class Count Q3 = 12), and description complexity far surpassing earlier datasets (median 40 words versus 8.4 for RefCOCOg).

Figure 3: Proportional breakdown of the dataset’s four main challenge dimensions, with L-2 subcategory granularity.

Benchmarking MLLMs: A Critical Gap Exposed

GroundingME systematically evaluates 25 SOTA MLLMs, covering both dense and Mixture-of-Experts models (2B–235B), including commercial and open-source variants. All models are measured with strict input/output constraints and standard [email protected] for localization.

Key findings:

• Severe Performance Deficit: The leading model (Qwen3-VL-235B-A22B) attains only 45.1% accuracy, with most models between 10–40%. On the Rejection subcategory, nearly all models display 0% accuracy, habitually hallucinating objects rather than abstaining.
• Scaling Trends: Larger models consistently perform better, but even 235B models do not cross the 50% barrier, signifying that scaling alone is insufficient for robust grounding.
• No Commercial Advantage: Models such as Gemini-2.5 (Pro/Flash) and Seed-1.6-Vision do not outperform the best open-source systems, indicating the generality of the observed limitations.

Subcategory analysis reveals that Discriminative tasks (fine-grained attribute matching, text recognition, state discrimination) fare best, with substantial drops in performance for spatial reasoning (especially counting) and dramatically low results for negative (Rejection) cases.

Analytical Insights: Reasoning, Test-Time Strategies, and Training Interventions

The Role of "Thinking" in Improving Grounding

Activating explicit stepwise reasoning ("thinking mode") at inference time yields consistent gains (4.7%–7.4% increase in total accuracy), especially on spatial and rejection challenges. However, improvements remain modest, and hallucination on ungroundable queries persists, establishing that stepwise reasoning is a necessary but insufficient mechanism for human-aligned grounding.

Figure 4: Case study contrasting successful versus failed thinking trajectories on a rejection instance, demonstrating the susceptibility to over-commitment to plausible distractors.

Test-Time Scaling via Reasoning-Trajectory Judgment

A judge-based Test-Time Scaling (TTS) framework is proposed: for each prompt, 16 thinking trajectories (temperature = 0.7) are generated, and a text-only LLM (DeepSeek-R1, MiMo-7B) adjudicates the best based on reasoning quality—without image access—yielding an absolute improvement of up to 2.9%, primarily on the Rejection and Spatial axes. Using a multimodal judge as baseline recapitulates these findings, indicating that fine-grained logic, not image-level cues, drives the marginal gains.

Figure 5: Quantified performance uplift attributable to "thinking mode" compared to standard greedy evaluation.

Data-Mixture Training for Negative Case Robustification

To remedy rejection failure, data-mixture supervised fine-tuning is conducted: RefCOCOg is augmented with synthetically-generated negative examples at various negative:positive ratios (from 1:8 up to 2:1). On in-domain validation (negative and positive splits), rejection accuracy scales monotonically from 30.5% (baseline) to 97.3% (2:1 mix), while positive-case performance degrades, confirming the transfer trade-off.

Out-of-domain (OOD) evaluation on GroundingME demonstrates substantial but attenuated OOD generalization, with rejection accuracy rising to 27.9% at worst cost to positive-class performance (e.g., Limited subcategory). Thus, simple negative augmentation is highly effective in-domain, but non-trivial distribution gaps in harder benchmarks like GroundingME remain.

Figure 6: Out-of-domain rejection accuracy progression versus negative/positive SFT mix ratio, underscoring differential OOD generalization.

Implications and Future Directions

The results underscore that current SOTA MLLMs lack both fine-grained discrimination and reliable negative-class abstention—properties crucial for deployment-readiness in safety-critical settings (robotics, autonomous perception, assistive technology). The tendency to hallucinate under ambiguous or non-existent queries is a systemic risk unmitigated by naive scaling or current prompting paradigms.

Test-time trajectory selection and negative-sample augmentation present concrete, scalable interventions. However, the limited cross-benchmark generalization implies a need for curriculum-based or adversarial grounding challenges during pretraining, refined multimodal feedback alignment, and diagnostic error-case tracking protocols.

An emergent implication is that genuine multi-stage visual reasoning remains unsolved: stepwise thinking only partially bridges the gap, and models frequently succumb to overgeneralization even when reasoning is logically structured.

Conclusion

GroundingME provides compelling evidence that modern MLLMs, irrespective of training dataset scale or architecture size, do not yet master human-caliber visual grounding, particularly when tasked with rejection and reasoning-intensive cases. The dual emphasis on diagnostic substructure and robust negative-case generation establishes a template for the next generation of grounding benchmarks and evaluation protocols. Future MLLM research must explicitly address fine-grained negative conditioning and multi-step spatial discrimination to close the current deployment gap.