REF-VC: Zero-Shot Verification for REC

• REF-VC is a zero-shot referring expression comprehension method that reformulates object grounding into independent True/False verification of candidate boxes.
• It leverages a generic detector to propose regions and a vision-language model to verify each box with a binary decision process.
• The verification-first pipeline minimizes cross-box interference, supports tie-breaking and abstention, and boosts accuracy on benchmarks such as RefCOCO.

Searching arXiv for the specified paper and closely related REC references. {"query":"arXiv (Liu et al., 12 Sep 2025) Zero-Shot Referring Expression Comprehension via Visual-Language True/False Verification", "max_results": 5} {"query":"GroundingDINO arXiv referring expression comprehension YOLO-World CogVLM", "max_results": 10} {"query":"YOLO-World Cheng 2024 arXiv", "max_results": 5} {"query":"GroundingDINO arXiv", "max_results": 5} {"query":"CogVLM arXiv", "max_results": 5} REF-VC denotes a zero-shot referring expression comprehension workflow that reformulates object grounding as box-wise visual-language verification rather than direct global selection. In this formulation, a generic detector first produces candidate regions, and a vision-LLM then answers a binary True/False question for each region independently; the final prediction is derived from the verified subset, with support for tie-breaking and abstention. The method is presented as a verification-first REC pipeline requiring no REC-specific training, and it is evaluated on RefCOCO, RefCOCO+, and RefCOCOg, where it is reported to outperform both selection-based prompting built on the same proposal set and several REC-oriented baselines (Liu et al., 12 Sep 2025).

1. Problem formulation and task definition

Referring Expression Comprehension (REC) takes an image $I$ and a natural-language description $s$, and returns a bounding box $\hat{b}$ for the referred object, or abstains if no candidate fits. In the zero-shot setting considered here, the system consists of a detector $D$, a vision-LLM $F$, and a proposal set $\mathcal{B}=\{b_i\}$. For each candidate $b_i$, the verifier produces $y_i \in \{\mathrm{True}, \mathrm{False}\}$. The operational task is

$$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$

Evaluation follows standard REC benchmarks. A prediction is counted as correct when

$\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$

and performance is reported as [email protected], namely the percentage of expressions satisfying that criterion. The reported benchmarks are RefCOCO, RefCOCO+, and RefCOCOg, with the usual Val/TestA/TestB or Val/Test splits as applicable (Liu et al., 12 Sep 2025).

Within this setting, REF-VC treats REC not as direct box selection among all proposals, but as a sequence of independent verification decisions. This changes the inductive structure of the problem: instead of asking a VLM to resolve a single multi-way competition over many boxes, it asks the model to assess each candidate region in isolation against the referring expression. The paper explicitly positions this as a workflow change rather than a new trained architecture or loss (Liu et al., 12 Sep 2025).

2. Verification-first workflow

The core workflow begins with coarse class inference from the referring expression. The VLM $s$0 is first asked to infer a relevant object class $s$1 from $s$2, after which YOLO-World is run in a class-conditioned manner to generate proposals: $s$3 YOLO-World is described as a COCO-clean generic detector and as class-conditioned but non-grounding. The paper notes that this COCO-clean choice costs about $s$4 [email protected] relative to similar COCO-trained YOLO variants, but is used to preserve system-level zero-shot cleanliness with respect to REC-style supervision (Liu et al., 12 Sep 2025).

For each proposal $s$5, the system constructs an overlay image $s$6 containing only that single highlighted box and queries the VLM with a binary prompt of the form “Does this box match the description?”. The response is constrained to

$s$7

The method does not use confidence scores from the VLM; it uses only the discrete True/False output. This step is the defining REF-VC operation: per-box visual-language verification (Liu et al., 12 Sep 2025).

After verification, the set of positively verified proposals is

$s$8

The decision rule has three cases. If $s$9, the single verified box is returned directly. If $\hat{b}$0, only those verified boxes are overlaid and indexed, and the VLM performs a smaller tie-breaking selection over that reduced subset. If $\hat{b}$1, all proposals are indexed and presented for fallback selection; if the VLM answers “none” or equivalent, the system abstains, otherwise it outputs the chosen box. This architecture therefore supports both multiple matches and abstention, which the paper identifies as advantages over direct choose-one prompting (Liu et al., 12 Sep 2025).

3. Verification versus selection

The comparison baseline uses the same proposals $\hat{b}$2 and the same VLM, but changes only the prompting regime. In the selection baseline, the full image is shown once with all candidate boxes overlaid and indexed, and the VLM is asked to output the single best index. The paper evaluates both single-shot selection and majority-voting selection with three calls at temperature $\hat{b}$3 (Liu et al., 12 Sep 2025).

A simplified two-box analysis is used to formalize the difference. Let box $\hat{b}$4 be correct and box $\hat{b}$5 be a distractor. Define $\hat{b}$6 as the probability that selection chooses box $\hat{b}$7, $\hat{b}$8 as the probability that the verifier labels the correct box True, and $\hat{b}$9 as the probability that the verifier labels the distractor True. Then selection accuracy is

$D$0

Verification accuracy, with fallback to selection when both boxes are True or both are False, is

$D$1

The derived condition for selection to outperform verification is

$D$2

The paper emphasizes that for fixed $D$3, the threshold is increasing and concave in $D$4, and is strictly above the identity line $D$5 for $D$6. In the symmetric simplification $D$7 and $D$8, the threshold exceeds $D$9 for $F$0, and the gap $F$1 peaks around $F$2 with a gap of about $F$3 (Liu et al., 12 Sep 2025).

This analysis is used to motivate why verification can outperform direct selection even when the verifier is imperfect. The controlled empirical comparison aligns with that claim. With identical YOLO-World proposals and identical VLMs, verification-first consistently outperforms selection. For LLaVA on validation splits, selection yields $F$4 on RefCOCO, $F$5 on RefCOCO+, and $F$6 on RefCOCOg, whereas verification-first yields $F$7, $F$8, and $F$9, respectively. With GPT-4o, single-shot selection gives $\mathcal{B}=\{b_i\}$0, $\mathcal{B}=\{b_i\}$1, and $\mathcal{B}=\{b_i\}$2 on the same validation splits; majority-vote selection gives $\mathcal{B}=\{b_i\}$3, $\mathcal{B}=\{b_i\}$4, and $\mathcal{B}=\{b_i\}$5; verification-first gives $\mathcal{B}=\{b_i\}$6, $\mathcal{B}=\{b_i\}$7, and $\mathcal{B}=\{b_i\}$8 (Liu et al., 12 Sep 2025).

4. Pipeline composition and benchmark results

The paper evaluates three classes of systems: REC-trained baselines, strict zero-shot baselines, and the proposed zero-shot workflows. Among supervised baselines, CogVLM is reported at $\mathcal{B}=\{b_i\}$9 Val / $b_i$0 TestA / $b_i$1 TestB on RefCOCO, $b_i$2 / $b_i$3 / $b_i$4 on RefCOCO+, and $b_i$5 Val / $b_i$6 Test on RefCOCOg. GroundingDINO trained on REC reports $b_i$7 / $b_i$8 on RefCOCO TestA/TestB, $b_i$9 / $y_i \in \{\mathrm{True}, \mathrm{False}\}$0 on RefCOCO+ TestA/TestB, and $y_i \in \{\mathrm{True}, \mathrm{False}\}$1 on RefCOCOg Test. GroundingDINO + CRG reports $y_i \in \{\mathrm{True}, \mathrm{False}\}$2 / $y_i \in \{\mathrm{True}, \mathrm{False}\}$3, $y_i \in \{\mathrm{True}, \mathrm{False}\}$4 / $y_i \in \{\mathrm{True}, \mathrm{False}\}$5, and $y_i \in \{\mathrm{True}, \mathrm{False}\}$6 on those test splits (Liu et al., 12 Sep 2025).

In the strict zero-shot regime, GroundingDINO used without REC fine-tuning reports $y_i \in \{\mathrm{True}, \mathrm{False}\}$7 Val / $y_i \in \{\mathrm{True}, \mathrm{False}\}$8 TestA / $y_i \in \{\mathrm{True}, \mathrm{False}\}$9 TestB on RefCOCO, $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$0 / $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$1 / $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$2 on RefCOCO+, and $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$3 Val / $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$4 Test on RefCOCOg. GPT-4o used in an organic or vanilla direct-box-output manner obtains $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$5 on RefCOCO Val, $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$6 on RefCOCO+ Val, and $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$7 on RefCOCOg Val, which the paper uses to argue that GPT-4o is not simply recovering REC behavior without a proposal-based workflow (Liu et al., 12 Sep 2025).

Against these baselines, the main REF-VC configuration—verification-first with GPT-4o and YOLO-World proposals—reports $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$8 Val / $$(I,s)\xrightarrow{\text{zero-shot workflow}} \hat{b}\ \text{or abstain}.$$9 TestA / $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$0 TestB on RefCOCO, $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$1 / $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$2 / $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$3 on RefCOCO+, and $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$4 Val / $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$5 Test on RefCOCOg. Relative to zero-shot GroundingDINO on validation, the gains are $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$6 on RefCOCO, $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$7 on RefCOCO+, and $\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$8 on RefCOCOg. Relative to GroundingDINO + CRG on test splits, REF-VC is higher on RefCOCO TestA ($\mathrm{IoU}(\hat{b}, b^{\mathrm{gt}}) > 0.5,$9 vs $s$00), RefCOCO TestB ($s$01 vs $s$02), RefCOCO+ TestA ($s$03 vs $s$04), RefCOCO+ TestB ($s$05 vs $s$06), and RefCOCOg Test ($s$07 vs $s$08) (Liu et al., 12 Sep 2025).

The paper also reports that the largest gains occur on RefCOCO and RefCOCO+, which it associates with multi-object scenes and fine-grained attribute language. It further notes that verification improves performance with both proprietary and open VLMs: GPT-4o substantially exceeds LLaVA in absolute accuracy, but for both models the verification-first prompting strategy is superior to selection-based prompting (Liu et al., 12 Sep 2025).

5. Methodological interpretation and relation to REC pipelines

The central methodological claim is that workflow design, rather than task-specific pretraining, can drive strong zero-shot REC performance. The paper motivates this claim through controlled comparisons in which the proposal set and VLM are held fixed while only the prompting paradigm changes. Under that control, verification-first exceeds selection by roughly $s$09–$s$10 absolute points across datasets and VLM choices, implying that the gain is not reducible to detector quality alone or to an incidental difference in model backbone (Liu et al., 12 Sep 2025).

This argument also positions REF-VC relative to proposal-ranking REC pipelines. Classical two-stage approaches generate candidate regions and then score or rank them jointly with the expression. REF-VC preserves the two-stage structure but changes the second stage from global competition to atomic binary verification. The paper attributes the resulting advantage to several factors: minimized cross-box interference, a stronger evidential signal from True/False outputs, and search-space reduction before any residual multi-way decision. It also states that this design supports attributes such as “the small red mug,” spatial relations such as “on the left” or “in front of the man,” and ordinal language such as “the third person from the right,” because each candidate is assessed in isolation while still being grounded in the full image context (Liu et al., 12 Sep 2025).

A further methodological point concerns ambiguity handling. Because the verifier can return multiple True boxes, the workflow does not force premature collapse to a single candidate. Mild ambiguity can be postponed to a second-stage selection over the pruned set, while complete failure of verification triggers either fallback selection or abstention. This differs materially from single-shot selection, which is structurally forced to choose among all boxes at once. A plausible implication is that verification-first decomposes REC into a high-recall filtering stage followed by a lower-entropy disambiguation stage, which is consistent with the behavior observed in the reported experiments (Liu et al., 12 Sep 2025).

6. Limitations, scope, and prospective extensions

The paper identifies detector recall as the principal hard failure mode. Because REF-VC is proposal-based, if YOLO-World fails to produce a candidate box covering the true referent, the downstream verifier cannot recover the error. This is characterized as an inherent limitation of any two-stage proposal-plus-ranking design. The method also incurs a higher computational cost than single-shot selection because it requires one VLM call per proposal, with additional calls for tie-breaking or fallback. In crowded scenes with many tightly packed similar objects, verification may also degrade because candidate boxes become noisy, overlapping, or visually near-duplicate (Liu et al., 12 Sep 2025).

Another stated limitation is the absence of joint optimization. The detector and VLM are off-the-shelf modules, and the method is purely workflow-level; there is no learned alignment between proposal generation and verification. The paper therefore suggests several future directions: joint optimization or co-training, improved prompting for class extraction and True/False verification, use of soft probabilities or calibrated confidence in place of strict binary outputs, compute-aware cascades that reduce proposal count or stage VLM usage, and extension of the verification-first pattern to other localization or segmentation tasks (Liu et al., 12 Sep 2025).

Within its stated scope, REF-VC is not presented as a replacement for supervised REC under all regimes. Supervised CogVLM remains substantially higher on all listed benchmarks. The paper’s stronger claim is narrower: a zero-shot workflow built from a COCO-clean generic detector and a general-purpose VLM can exceed both zero-shot grounding baselines and some REC-trained pipelines, and the decisive factor is the verification-first decomposition of the task. That conclusion is directly tied to the controlled studies in which proposals and VLMs are fixed while prompting varies (Liu et al., 12 Sep 2025).