$\textit{Don't Guess, Just Ask}$: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification

Abstract: Referring segmentation aims to segment the target objects in images or videos based on the textual query. Despite remarkable progress over the past years, existing works always assume that the user-provided queries are already precise and clear. However, this assumption is impractical. In real-world scenarios, it is unrealistic to expect all users to thoroughly review their visual content and carefully ensure their queries are unique and unambiguous. When encountering such cases, existing segmentation models tend to arbitrarily guess the user preferences, often resulting in undesired outcomes. To address this limitation, we propose \textbf{IC-Seg}, a novel agentic framework that proactively clarifies user intent through multi-turn conversation before segmentation. To effectively incentivize this capability, we further introduce \textbf{Hi-GRPO}, a new hierarchical optimization strategy that injects dense and informative supervision signals at the trajectory, turn, and step levels. This strategy encourages efficient intent clarification, effectively eliminating redundant interactions and improving overall dialogue quality. For evaluation, we establish \textbf{Ambi-RVOS}, a referring video object segmentation benchmark with ambiguous user queries. Extensive experiments demonstrate that IC-Seg not only outperforms existing methods by a large margin in resolving ambiguous queries, but also maintains state-of-the-art performance on standard reasoning segmentation benchmarks. Code and data will be released at \url{https://github.com/iSEE-Laboratory/IC-Seg}.

• The paper presents IC-Seg, a framework that addresses ambiguity in referring segmentation through multi-turn user clarification, improving object localization.
• Hi-GRPO, the hierarchical reward system, enhances dialog efficiency and segmentation accuracy by providing dense supervision at trajectory, turn, and step levels.
• Benchmarked on Ambi-RVOS, IC-Seg achieves up to 20% improvement in $f{content}fj{content}\mathbf{F}$ over top methods, excelling in both ambiguous and standard scenarios.

Resolving Ambiguity in Referring Segmentation with Multi-turn Clarification

Motivation and Problem Statement

Referring segmentation, the task of segmenting target objects in visual data guided by textual queries, underpins numerous real-world applications including video editing, autonomous driving, and human-robot interaction. Recent advances in Multimodal LLMs (MLLMs) have led to substantial progress in this domain, enabling accurate object localization given user prompts. However, prevailing approaches fundamentally assume that user queries are always precise and sufficient for unique target identification. This assumption is often violated in real scenarios, where queries tend to be ambiguous, incomplete, or imprecise. In such cases, traditional segmentation models resort to arbitrary guessing, leading to undesirable outcomes and unreliable system behavior.

Figure 1: Ambiguous referring segmentation example—existing models arbitrarily select a target, while IC-Seg uses proactive intent clarification.

IC-Seg Framework and Hierarchical Optimization

To address query ambiguity, the paper proposes IC-Seg, an agentic framework designed for multi-turn intent clarification prior to segmentation. Upon receiving an ambiguous query, IC-Seg actively engages in dialogue with the user to refine intent, rather than prematurely producing a segmentation output. This interactive loop is realized by a trainable policy MLLM $\pi_\theta$ and an MLLM-based User Simulator, which together simulate human-in-the-loop clarification and feedback cycles.

Figure 2: IC-Seg resolves query ambiguity via multi-turn dialogues, leveraging hierarchical reward supervision from the Hi-GRPO algorithm.

Training such a capability requires dense and informative supervision across the entire trajectory of interaction. Vanilla RL approaches with sparse outcome rewards are inadequate, often producing repetitive or redundant dialogue. To overcome this, the authors introduce Hi-GRPO (Hierarchical Group Relative Policy Optimization), which injects rewards at three granularities:

• Trajectory-level: Rewards maximize object localization accuracy, encompassing metrics like IoU and spatial precision.
• Turn-level: Rewards optimize dialogue quality by encouraging ambiguity reduction and interaction efficiency, quantified via entropy decrease and discriminative questioning.
• Step-level: Dense token-wise rewards are assigned using expert-guided feedback, enabling precise credit assignment for reasoning steps and dialog turns.

This hierarchy ensures that supervision is both globally aligned with final outcomes and locally sensitive to reasoning actions.

Benchmarking: Ambi-RVOS for Ambiguous Queries

To rigorously evaluate intent clarification capabilities, the authors introduce Ambi-RVOS, a new benchmark tailored for ambiguous referring video object segmentation. This dataset contains video sequences and queries with multiple plausible candidates per query, reflecting the complexity of real user instructions. Ambi-RVOS is structured into varying difficulty levels, corresponding to the number of candidate objects present per ambiguous query.

Empirical Results and Ablations

IC-Seg substantially outperforms specialist and reasoning-based segmentation baselines on Ambi-RVOS, achieving up to $+20\%$ improvement in $\mathcal{J}{content}\mathcal{F}$ over strong Qwen3-VL baselines. Notably, IC-Seg achieves SOTA metrics not only in ambiguous interaction but also in standard reasoning segmentation datasets, demonstrating robustness and generalization. Ablation studies confirm the necessity of each Hi-GRPO reward granularity—removing turn-level or step-level guidance noticeably degrades performance, increases redundant interaction turns, and reduces segmentation efficiency.

IC-Seg's multi-turn dialog mechanism is visualized qualitatively: baseline models either guess or enter repetitive loops, while IC-Seg strategically isolates and verifies candidates, leveraging user feedback to actively resolve ambiguity.

Figure 3: Qualitative comparison—IC-Seg eliminates ambiguity and avoids guessing by efficiently engaging the user.

Training dynamics further illustrate stability and efficiency: localization and process rewards consistently improve, the number of queries stabilizes, and the model converges to concise yet informative interaction behavior.

Figure 4: IC-Seg-8B training dynamics demonstrate progressive improvement in both localization and clarification metrics.

Practical and Theoretical Implications

IC-Seg exemplifies a paradigm shift from passive grounding to interactive, intent-aware visual reasoning systems. Practically, this framework mitigates the risks associated with misinterpreted user instructions and enhances the reliability of AI in safety-critical domains. By lowering the burden on users to craft unambiguous prompts, IC-Seg enables more natural, intuitive collaboration with visual AI systems.

Theoretically, the hierarchical RL structure provides a blueprint for efficient credit assignment in long-horizon, multi-turn agentic environments. Step-level guidance, informed by expert diagnostic signals, enables granular supervision that scales to complex dialog and reasoning trajectories.

Future Perspectives

While IC-Seg currently operates in a text-based interactive setting, future directions may include the integration of multimodal user input (e.g., gesture, speech, and visual cues). Extending the agentic clarification paradigm beyond segmentation, to other open-world perception tasks, poses promising challenges and avenues for research. The hierarchical reward structure introduced here may serve as a foundation for general agentic RL protocols in multimodal interaction domains.

Conclusion

The paper advances referring segmentation by introducing IC-Seg, a multi-turn, intent-aware agentic model trained with hierarchical optimization. It quantitatively and qualitatively demonstrates the necessity and effectiveness of interactive clarification for resolving ambiguity in real-world scenarios. Ambi-RVOS establishes a rigorous benchmark for this capability. The hierarchical supervision framework yields significant improvements and offers a principled approach to training agentic AI. These innovations are poised to inform future developments in robust, transparent, and user-centric visual systems.