Beyond Referring Expressions: Scenario Comprehension Visual Grounding

Abstract: Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming. We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.

• The paper introduces a new RSC benchmark and ScenGround method that shift visual grounding from short referring expressions to detailed, scenario-based narratives.
• It employs a multi-phase dataset curation and a difficulty-aware curriculum RL, achieving 55.7 mIoU on in-distribution and 38.4 mIoU on out-of-distribution data.
• The work integrates relational, spatial, and pragmatic cues, paving the way for practical applications in context-aware robotics, dialogue systems, and visual reasoning.

Towards Scenario-Based Visual Grounding: The Referring Scenario Comprehension (RSC) Benchmark and ScenGround Reasoning Method

Motivation and Problem Formulation

The paper "Beyond Referring Expressions: Scenario Comprehension Visual Grounding" (2604.02323) addresses the limitations of contemporary visual grounding benchmarks and methods, which are primarily tuned for direct lexical matches between short referring expressions and image regions. While traditional Referring Expression Comprehension (REC) datasets (e.g., RefCOCO, RefCOCOg, RefCOCO+) facilitate grounding via short, literal, and category-centric queries, real-world user needs frequently involve referencing objects through scenario-driven, context-embedded narratives, omitting explicit category names and requiring considerable pragmatic and relational reasoning.

To bridge this gap, the authors propose the Referring Scenario Comprehension (RSC) task, which formalizes visual grounding based on paragraph-length queries emphasizing user roles, goals, intentional context, explicit distractor mentions, and diverse disambiguation cues. The challenge fundamentally differs from REC: success demands integrating relational, spatial, and pragmatic cues spread across varied scenario narratives, not simply mapping explicit entity names to image regions.

Figure 1: Illustration of the contrast between traditional REC (where category names and short descriptions enable lexical matching) and RSC, which demands scenario-based, paragraph-level reasoning to resolve distractors and ground the target object.

The RSC Benchmark: Dataset Design and Characteristics

RSC comprises approximately 31k training, 4k in-domain (ID) test, and 3k out-of-distribution (OOD) test examples. The core dataset construction pipeline consists of three phases:

1. Phase 1: Source Filtering and Balancing – Instances are curated from MS-COCO and LVIS (for OOD), enforcing category disjointness, and stratified according to interpretable difficulty tags: Uniqueness (U), Clutter (C), Size (S), Overlap (O), and Position (P).
2. Phase 2: Annotation via LLMs – Scenarios and reasoning traces are generated in a two-stage model-in-the-loop pipeline, with validation on schema compliance, box alignment, and alias consistency.
3. Phase 3: Automatic and Human Quality Control – Instances pass through strict LLM-based judging (tasking for uniqueness, box accuracy, and scenario coherence) and human audit ensuring scenario non-leakage and semantic validity.

Figure 2: RSC data curation pipeline including tag balancing, LLM-driven scenario/trace annotation, and multi-stage quality filtering.

Each datum offers: the scenario narrative, a reasoning trace (mapping scenario cues to visual evidence), acceptable aliases, ground-truth bounding box, a canonical referring expression, and difficulty tags. Benchmark splits ensure strict in-distribution and out-of-distribution evaluation, with the OOD set sampled from LVIS categories disjoint with COCO.

Linguistic and Structural Analysis

RSC scenario queries exhibit dramatically higher linguistic and contextual complexity compared to existing datasets. Scenario queries average 52.7 words, with a vocabulary nearly twice as large as REC baselines. Disambiguation is achieved predominantly through relational, attribute-based, and pragmatic cues, tightly coupled with explicit distractor mentions.

Figure 3: Distribution of query lengths across RSC, RefCOCO+, and RefCOCOg. RSC peaks at 50–60 words, reflecting scenario-based, paragraph-level queries.

Difficulty tags are equitably distributed to facilitate controlled studies on reasoning challenges. The OOD split is substantially more difficult, with smaller objects and a pronounced long-tail category distribution, as confirmed by both frequency analysis and performance drop under cross-category generalization.

Figure 4: Marginal distributions of difficulty tags for RSC in-distribution and OOD splits, showing balanced difficulty axes and pronounced OOD complexity.

The ScenGround Curriculum Reasoning Baseline

The authors introduce ScenGround, a reference architecture combining supervised schema alignment and difficulty-aware RL for robust scenario grounding. The training regime consists of two core stages:

Thought-Primed Supervised Fine-Tuning (TP-SFT)

TP-SFT aligns the system to the required JSON output schema, enforces the emission of explicit reasoning chains, and anchors the policy with a stable base on the easier slices of RSC. This stage teaches the model to generate self-contained > traces, and prepares for the reward-driven RL stage.

Figure 5: ScenGround prompt and output schema: the model is required to fill <think> with reasoning and <answer> with structured JSON for object and bbox.

Incentive-Curriculum GRPO (IC-GRPO)

Building on DeepSeek-R1's group-relative policy optimization paradigm, this stage incorporates multifaceted, shaped rewards (IoU-geometric consistency, alias-aware semantic matching, output format robustness), and exposes the model progressively to harder slices (by difficulty score and tags). A prompt-template ensemble augments surface-form robustness.

Ablation demonstrates the efficacy of this curriculum-driven RL: single-stage RL or schema alignment alone are insufficient to close the OOD gap, especially for hard cases (e.g., high-overlap, small, or ambiguous targets).

Empirical Results

Across RSC in-distribution and out-of-distribution, ScenGround establishes strong performance gains:

• On RSC-ID, ScenGround achieves 55.7 mIoU and 94.2% category accuracy, well above the best off-the-shelf VLMs like Qwen2.5-VL (30.3 mIoU).
• On RSC-OOD, robustness persists with 38.4 mIoU, indicating non-trivial scenario generalization; however, category accuracy (21.1%) reveals the persisting challenge of open-set semantic mapping.
• Closed-source LLMs (GPT-4o, Claude 3.7) show high category classification rates but fail on precise localization, highlighting the challenge of joint scenario-to-pixel reasoning.

(Table 1 and Table 2, summarized verbally)

Ablations confirm curriculum RL as key to OOD robustness and improved reasoning/localization coupling. Transfer to standard REC benchmarks, while not matching task-specific pipelines, demonstrates gains over SFT and baseline settings, especially on longer-expression tasks (RefCOCOg), indicating the emergence of transferable grounding skills.

Qualitative inspections reveal ScenGround's capacity to resolve ambiguous targets in the presence of distractor cues, leveraging scenario role and context rather than keyword matching.

Figure 6: ScenGround predictions (blue) accurately localize targets prescribed by scenario even in cluttered or ambiguous settings; ground truth in red.

Theoretical and Practical Implications

The transition from REC to RSC exposes systematic blind spots in current VLMs and grounding architectures: high scores on direct cues cannot be extrapolated to real-world, user-centric referential tasks. Curriculum RL for reasoning and grounding, as instantiated in ScenGround, effectively closes the ID and OOD localization gap via structured, difficulty-aware supervision, providing a scalable foundation for future visual scenario reasoning tasks.

From a theoretical standpoint, the explicit coupling of scenario-based reasoning (involving multi-sentence intent, relational logic, and pragmatic distractor manipulation) with structured reward architectures in RL fine-tuning presents a demonstrably superior strategy for general-purpose VLMs aiming at robust agentic vision-language reasoning.

Practical extensions include scenario-centric dialogue systems, context-aware robot perception, and accessibility tools requiring nuanced, non-category-centric grounding.

Future Directions

Key areas warrant further investigation:

• Semantic Naming in OOD: While localization is improved on OOD categories, open-vocabulary semantic prediction remains an open challenge.
• Human-in-the-Loop and Multimodal Curation: RSC, while shown to echo natural scenario structure, is based on LLM-generated narratives. Incorporating broader human annotation, multi-object/multi-turn setup, and temporal context will be critical for ecological validity.
• Transfer to Embodied and Interactive Agents: Extending the scenario grounding paradigm to multi-object, temporal, or multi-modal (video, 3D) scenarios would further stress and clarify the limitations of current VLMs.

Conclusion

The RSC benchmark and ScenGround pipeline define a new standard for evaluating and improving multimodal models under naturalistic, scenario-driven referential demands. The paradigm exposes sharply the limitations of category-centric and phrase-matching approaches, and demonstrates that difficulty-aware curriculum RL significantly improves both localization and the integration of complex reasoning traces. The persistent challenge of cross-category semantic transfer in OOD generalization emphasizes the need for advances in linguistic and perceptual open-vocabulary learning.

Figure 7: Additional ScenGround qualitative examples, illustrating robust scenario-driven grounding even under high clutter, occlusion, or ambiguous distractor context.