Ambi-RVOS: Ambiguous RVOS Benchmark

• Ambi-RVOS is a referring video object segmentation benchmark that evaluates systems under ambiguous queries requiring clarification.
• It generates ambiguity by preserving natural query vagueness or stripping discriminative attributes, creating multiple candidate objects with a designated target.
• The benchmark assesses both segmentation accuracy and interaction efficiency using metrics like IoU, boundary F-score, and entropy reduction for ambiguity resolution.

Ambi-RVOS is a referring video object segmentation benchmark with intentionally ambiguous user queries, introduced to evaluate RVOS under the realistic condition that a single natural-language query may be compatible with multiple objects in a scene, while the annotation still specifies one intended target (Yang et al., 17 May 2026). It departs from the conventional RVOS assumption that the query is already precise and unique, and it reframes evaluation from pure one-shot segmentation toward ambiguity recognition and clarification-driven target resolution (Yang et al., 17 May 2026).

1. Conceptual departure from standard RVOS

Standard RVOS is typically formulated as a mapping from a video and a referring expression to a sequence of masks for a single referred object, as in representative formulations used by MTTR, ReferDINO, SVAC, and Tenet (Botach et al., 2021, Liang et al., 24 Jan 2025, Zhang et al., 28 Sep 2025, Lin et al., 8 Oct 2025). In these settings, the input sentence is treated as sufficient to identify the target instance, and the technical problem is centered on multimodal alignment, temporal reasoning, and dense segmentation.

Ambi-RVOS changes the premise. It is defined around ambiguous user queries: a single query may fit several objects in the video, but only one object is designated as the ground-truth target (Yang et al., 17 May 2026). The benchmark is explicitly motivated by the observation that existing referring segmentation systems generally assume that user-provided queries are already unique and clear, whereas in practical use they often are not (Yang et al., 17 May 2026).

This benchmark therefore tests a different failure mode from standard RVOS. Rather than asking only whether a model can segment the object named by a well-specified sentence, it asks whether the model can detect underspecification, obtain the missing information, and then segment the intended instance (Yang et al., 17 May 2026). The paper presents Ambi-RVOS as the first benchmark the authors are aware of that systematically assumes ambiguous queries and requires interaction to determine the target (Yang et al., 17 May 2026).

2. Construction and annotation protocol

Ambi-RVOS is curated from two existing RVOS benchmarks: ReVOS and MeViS (Yang et al., 17 May 2026). The benchmark does not require new pixel-level segmentation annotation, because the target masks are inherited from those source datasets (Yang et al., 17 May 2026).

The ambiguous queries are created in two ways (Yang et al., 17 May 2026):

1. Direct preservation of naturally ambiguous source queries: if an original query in ReVOS or MeViS already matches multiple objects in the scene, it is retained as an ambiguous query.
2. Attribute-stripping from uniquely identifying descriptions: if a source query uniquely identifies an object because it contains multiple discriminative attributes, those attributes are progressively removed until the simplified query refers to more than one candidate.

The resulting sample still has a single designated target, but the text no longer uniquely determines it (Yang et al., 17 May 2026). This creates supervised ambiguity: a model that simply guesses among compatible objects may output a valid-looking mask for the wrong instance.

The benchmark is organized around a candidate set for each ambiguous query. If the initial number of compatible objects is $M$, and the number remaining after clarification turn $k$ is $N_k$, ambiguity is quantified through Shannon entropy under a uniform prior (Yang et al., 17 May 2026):

$$H_k = -\sum_{i=1}^{N_k} p_i \log_2 p_i = \log_2 N_k \quad \text{with } p_i = 1 / N_k,\, N_k>0.$$

The paper notes that this candidate set is used by the judge model for ambiguity-reduction assessment rather than as a standard public segmentation label (Yang et al., 17 May 2026).

3. Dataset composition and ambiguity structure

Ambi-RVOS contains 1500 curated test samples (Yang et al., 17 May 2026). Each sample includes a video from ReVOS or MeViS, an ambiguous natural-language query, and the ground-truth target object with per-frame masks inherited from the source dataset (Yang et al., 17 May 2026).

The test set is stratified by the number of initial candidate objects (Yang et al., 17 May 2026):

Split	Candidates	Samples
Simple	exactly 2	400
Medium	3–5	600
Difficult	6 or more	500

This partition is intended to provide a controlled progression in ambiguity severity (Yang et al., 17 May 2026). The benchmark is deliberately small but focused, emphasizing scenes with a high density of visually similar objects and queries whose identifying details have been removed or were ambiguous from the outset (Yang et al., 17 May 2026).

The paper does not define a formal ambiguity taxonomy, but the construction protocol and examples imply several recurrent forms of ambiguity: attribute ambiguity, spatial ambiguity, temporal or state ambiguity, and identity ambiguity within groups (Yang et al., 17 May 2026). This suggests that Ambi-RVOS is not merely a benchmark for noisy phrasing; it is a benchmark for disambiguation under multimodal competition.

4. Evaluation protocol and ambiguity-aware interaction

Ambi-RVOS evaluates the final segmentation with standard RVOS metrics (Yang et al., 17 May 2026):

• Region similarity $\mathcal{J}$: average IoU between predicted and ground-truth masks
• Contour accuracy $\mathcal{F}$: mean boundary F-score
• Overall score:

$$\mathcal{J\!F} = \frac{\mathcal{J} + \mathcal{F}}{2}$$

The task remains single-target: although multiple objects may satisfy the text, only one object is counted as correct (Yang et al., 17 May 2026). There is no separate multi-referent metric.

In the paper’s evaluation setup, methods are allowed to interact with a user simulator, instantiated as Qwen3-VL-Instruct-32B, for up to 5 turns (Yang et al., 17 May 2026). A method may ask clarifying questions; after interaction it must commit to a target and produce the final segmentation (Yang et al., 17 May 2026). In the IC-Seg pipeline specifically, this commitment is implemented by predicting a keyframe, a 2D bounding box $[x_1,y_1,x_2,y_2]$, and a point $[x,y]$ on that keyframe, which are then passed to SAM2 to obtain per-frame masks (Yang et al., 17 May 2026).

Ambiguity reduction is explicitly quantified in the associated Hi-GRPO training scheme. The entropy reduction reward is defined as (Yang et al., 17 May 2026):

$$\mathcal{R}^{\text{ent}} = \frac{H_0 - H_K}{H_0}, \quad H_0 > 0,$$

where $k$0 and $k$1. An additional efficiency reward penalizes redundant questioning (Yang et al., 17 May 2026):

$k$2

These quantities make the benchmark suitable not only for evaluating segmentation accuracy, but also for evaluating whether a system asks questions that actually reduce ambiguity (Yang et al., 17 May 2026).

A user study on 150 samples $k$3 5 humans is reported to show that the simulator’s answers are close to human responses, supporting the simulated-user evaluation protocol (Yang et al., 17 May 2026).

5. Reported benchmark behavior

Ambi-RVOS exposes a sharp degradation of standard RVOS systems under ambiguous language. On the overall split, the reported $k$4 scores include the following (Yang et al., 17 May 2026):

Model	Setting	$k$5
SAMWISE	specialist RVOS	19.4
ReferDINO	specialist RVOS	21.3
SAM3	specialist RVOS	18.6
GLUS-7B	reasoning segmentation	27.1
RGA3-7B	reasoning segmentation	24.7
UniPixel-7B	non-agentic baseline	29.2
Qwen3-VL-8B*	same backbone, no Hi-GRPO interaction training	36.5
IC-Seg-8B	interactive	55.1

The split-wise numbers reported for Qwen3-VL-8B* and IC-Seg-8B make the role of candidate count explicit (Yang et al., 17 May 2026). For Qwen3-VL-8B*, $k$6 drops from 43.8 on the Simple split to 38.6 on Medium and 28.2 on Difficult. For IC-Seg-8B, the corresponding values are 63.7, 57.4, and 45.5 (Yang et al., 17 May 2026). The benchmark therefore scales difficulty in a way that remains visible even for interactive models.

The paper also reports that using only the trajectory-level reward in Hi-GRPO yields 46.4 $k$7 on Ambi-RVOS, whereas adding the ambiguity-reduction terms and step-level guidance raises performance to 52.0 $k$8 for IC-Seg-4B (Yang et al., 17 May 2026). This indicates that the benchmark is sensitive not just to final localization quality, but also to the structure of the clarification policy.

A further reported finding is that standard supervised fine-tuning on a small number of ambiguous samples is insufficient: the paper notes that SFT baselines trained on the same 120 Ambi-RVOS training samples underperform, and can even degrade relative to their original one-shot models (Yang et al., 17 May 2026). This reinforces the claim that ambiguity resolution in this setting is not reducible to ordinary supervised adaptation.

6. Position within RVOS research

Ambi-RVOS is best understood against the background of recent RVOS research. MTTR formulates RVOS as multimodal sequence prediction with a single referred object (Botach et al., 2021). ReferDINO couples GroundingDINO-based region grounding, temporal enhancement, and a deformable mask decoder, again under the assumption that the sentence identifies the target (Liang et al., 24 Jan 2025). Tenet factorizes RVOS into referring, video, and segmentation factors, but still presumes that the referring sentence can be resolved into a single temporal prompt (Lin et al., 8 Oct 2025). SVAC scales frame coverage and segmentation tokens in an MLLM+SAM-2 pipeline without introducing explicit ambiguity handling (Zhang et al., 28 Sep 2025). FlowRVS recasts RVOS as a language-conditioned continuous deformation from video latent to mask latent, yet still operates with one textual query and one target mask sequence (Wang et al., 7 Oct 2025).

Ambi-RVOS does not replace these formulations; it stress-tests an assumption they largely share. The benchmark shows that strong temporal modeling, stronger segmentation backbones, or larger multimodal models do not by themselves solve ambiguous reference (Yang et al., 17 May 2026). In the reported experiments, models that are competitive on standard RVOS still perform poorly when the query is compatible with multiple objects (Yang et al., 17 May 2026).

This suggests a methodological shift. Future RVOS systems evaluated on Ambi-RVOS-like settings may need explicit mechanisms for ambiguity detection, candidate management, clarification planning, and post-clarification commitment, rather than only better one-shot video-language segmentation. Ambi-RVOS therefore functions both as a benchmark and as a critique of the standard “perfect query” assumption that has structured most prior RVOS evaluation (Yang et al., 17 May 2026).