GroundingME Benchmark

• GroundingME Benchmark is a multidimensional framework that assesses visual grounding performance across discriminative, spatial, limited, and rejection tasks.
• It employs a hybrid automated–manual annotation protocol alongside metrics like IoU and rejection rate to capture nuanced language–vision reasoning.
• Experimental results reveal significant gaps in current MLLMs, underscoring the need for tailored training strategies and uncertainty-aware predictions.

GroundingME is a multidimensional benchmark designed to rigorously expose the gap between current multimodal LLMs (MLLMs) and human-level performance in visual grounding—the task of localizing objects described by natural language within images. Unlike prior datasets that emphasize simplified or synthetic tasks, GroundingME targets real-world complexity by systematically structuring evaluation across fine-grained discrimination, intricate spatial reasoning, robustness to occlusion and resolution, and explicit rejection of ungroundable queries. Experimental results across 25 contemporary MLLMs demonstrate that high performance on legacy benchmarks fails to translate to authentic visual-language understanding, as evidenced by sub-50% accuracy for the most advanced models and persistent failure to handle “rejection” cases. GroundingME further investigates improvement avenues via test-time reasoning selection and tailored data-mixture training strategies, delineating the challenges and future directions for progressing toward genuinely grounded AI systems (Li et al., 19 Dec 2025).

1. Multidimensional Structure of Visual Grounding

GroundingME partitions referring expression comprehension into four orthogonal evaluation dimensions, each calibrated to stress distinct aspects of grounded vision-language reasoning:

1. Discriminative: Focuses on selecting a single object from a set of visually similar candidates based on subtle attribute differentials (color, texture, embedded text, or state). Formalized as $$b^* = \arg\max_{b_i\in B} \mathbf{1}(\text{Attr}(b_i) = a_t),$$ where $B$ is the set of candidate regions, and $a_t$ the target attribute vector. Success requires fine-grained visual parsing—keyword-based shortcuts are insufficient.
2. Spatial: Involves interpreting relational or ordinal cues in language, often with chained or multi-object constraints. Tasks include identifying entities based on spatial relationships (“the hut immediately to the left of a light grey hut...”) or explicit counting (“the flag that is followed on its right by five more flags”). Distilling the correct $b^*$ requires reasoning over object placement, scene geometry, and viewer-centric reference frames.
3. Limited: Tests model robustness when objects are occluded or extremely small ($$\alpha = \frac{\text{area}(b)}{\text{area}(I)} < 0.5\%$$). These cases demand high-resolution feature extraction and inference under partial evidence, as objects may lack canonical context or clarity.
4. Rejection: Challenges models with ungroundable queries—object descriptions that match no true region in the image, often due to subtly falsified facts. The correct output is the empty set ($b^* = \text{null}$). Safe reasoning here requires the model to withhold unwarranted speculation even when faced with plausible but incorrect cues.

This structure ensures comprehensive coverage of the linguistic and perceptual spectrum encountered in practical human-MLLM grounding tasks (Li et al., 19 Dec 2025).

2. Dataset Construction and Annotation Protocol

GroundingME comprises 1,005 human-verified referring-expression tasks carefully distributed across the four L-1 dimensions: 20.3% Discriminative, 29.9% Spatial, 29.9% Limited, and 20.0% Rejection. Curation follows a hybrid automated–manual pipeline optimized for realism, diversity, and task difficulty:

• Bounding Box Annotation:
  • Standard SA-1B source images are processed with RAM++ (class enumeration), GroundingDINO (candidate proposal), and a custom non-maximum suppression biased toward classes with higher instance counts.
  • HR-Bench images, targeting ultra-small objects, employ manual annotation to ensure reliable ground-truth coverage.
• Description Generation:
  • Descriptions are generated with Gemini-2.5-Flash using visual prompting on both full images and (for “Limited” cases) cropped regions, maximizing both linguistic detail and response specificity.
• Human Verification and Refinement:
  • Trivial or ambiguous samples are removed (e.g., class with $<$3 instances or bounding box $>$50% image area).
  • Diversity constraints enforce intra-class instance counts (min 5), select “Counting” queries from scenes with at least 8 instances, and ensure textual tasks have visible numbers/strings.
  • Every description receives expert revision for uniqueness, clarity, task type, and in the case of “Rejection,” deliberate factual inversion.
• Dataset Statistics:
  • 241 distinct object classes
  • Instance $\sqrt{\text{area}}$ spans 21–946 pixels; image $\sqrt{\text{area}}$ 1,500–7,680 pixels (cf. RefCOCO 83–610)
  • Instance area ratio quartiles: 0.16%, 1.0%, 2.7%
  • Description lengths: quartile range 18–58 words
  • Intra-class count quartiles: 5, 7, 12

This methodology ensures that GroundingME encapsulates the breadth and ambiguity typical of real semantic-visual reference tasks (Li et al., 19 Dec 2025).

3. Evaluation Metrics and Protocols

GroundingME employs a set of rigorous metrics to capture both spatial accuracy and model caution:

• Intersection over Union (IoU) for bounding box prediction:

$B$0

• Accuracy@$B$1:

$B$2

• $B$3 is primary; also reported are $B$4, $B$5, and mean-IoU accuracy (averaged over $B$6).
  • Rejection Rate: On “Rejection” tasks, the fraction where a model correctly outputs null (i.e., abstains from hallucinating a bounding box).

These criteria penalize both false positives (over-confident hallucination) and false negatives (failure to ground true objects), offering a fine-grained breakdown by reference type and linguistic challenge (Li et al., 19 Dec 2025).

4. Experimental Results and Model Diagnostics

Extensive evaluation over 25 state-of-the-art MLLMs highlights a persistent gap in visual grounding performance:

• Overall Accuracy: The leading model (Qwen3-VL-A22B) achieves only 45.1% $B$7 across the full test suite.
• Result Distribution: Most models cluster between 10–40%, with several below 10%.
• Dimension-wise Breakdown for Qwen3-VL-A22B:

| Subtask | [email protected] (%) | |-----------------|------------| | Discriminative | 69.6 | | Spatial | 49.7 | | Limited | 54.0 | | Rejection | 0.0 |

• Rejection Deficit: Under greedy decoding, almost all models fail to output null on “Rejection” tasks. Instead, they reflexively hallucinate bounding boxes, ignoring critical cues for “no match”—raising significant deployment and safety concerns.
• Scaling Observations: Family-wise scaling (e.g., Qwen3-VL 2B to 32B) boosts overall $B$8 from ~21% to ~39.5%, primarily improving “Discriminative” subtasks; “Rejection” remains unaffected without tailored intervention.

This data indicates that even advanced MLLMs do not generalize from conventional benchmarks to the real-world complexity captured in GroundingME (Li et al., 19 Dec 2025).

5. Strategies for Addressing the Visual Grounding Gap

GroundingME investigates two specific methodologies for model improvement:

• Test-Time Trajectory Scaling:
  • Each query is answered with $B$9 chain-of-thought trajectories (Qwen3-VL-A22B-Thinking, $a_t$0). An external LLM judge (DeepSeek-R1) performs pairwise comparison, selecting the best “thinking trajectory.”
  • This best-of-16 selection yields an overall improvement from 49.8% to 52.7% (i.e., +2.9%), with greatest gains in “Spatial” (+2.8pp) and “Rejection” (+9.7pp) categories.
• Data-Mixture Training for Rejection:
  • Qwen3-VL-8B-Instruct is fine-tuned on a blend of positive (groundable) and negative (ungroundable) samples, minimizing cross-entropy over class labels (box vs. null). Varying the negative-to-positive ratio (1:8$a_t$12:1) allows calibration of rejection sensitivity.
  • This approach achieves up to 27.9% accuracy on “Rejection” (at 2:1 ratio) but modestly degrades non-rejection task performance, reflecting a precision-recall trade-off that must be systematically managed.

These findings demonstrate that neither brute-force scaling nor standard chain-of-thought prompting alone bridge the observed gap; targeted training and selective inference provide partial remedies, primarily for categories that had previously exhibited near-zero performance (Li et al., 19 Dec 2025).

6. Implications and Future Research Directions

GroundingME conclusively demonstrates that existing MLLMs, despite achieving high marks on standard referring grounding datasets, remain far from human-level performance on nuanced visual-linguistic alignment. The inability to reject ungroundable queries introduces a critical safety risk, as current systems will hallucinate rather than acknowledge uncertainty. This suggests a need for architectural changes that allow explicit uncertainty modeling and for evaluation setups that penalize unwarranted assertions.

A plausible implication is that further human-level grounding will require not only continued model scaling and curriculum construction, but also research into reasoning-aware training, balanced positive/negative modalities, more sophisticated relational reasoning architectures, and uncertainty-aware prediction protocols. GroundingME, with its four-dimensional structure and diagnostic reporting, provides a rigorous platform for benchmarking such future advances and calibrating genuine progress toward robust vision-language integration (Li et al., 19 Dec 2025).