OV-RefOD: Open Vocabulary Referring Detection

• OV-RefOD is an object detection framework that localizes arbitrary concepts from free-form text queries, going beyond fixed detector labels.
• It employs pseudo-labeling and CLIP-based region-text alignment alongside prompt-conditioned prediction to improve zero-shot localization performance.
• Inference techniques like Reverse Contrast Attention reweight cross-modal attention, enhancing interpretability and precision without retraining.

Searching arXiv for the two specified papers to ground the article. arXiv search: (Kang et al., 2023) arXiv search: (Juanico et al., 26 Jul 2025) Open Vocabulary Referring Object Detection (OV-RefOD) denotes object localization conditioned on text queries that are not restricted to a fixed detector label set. In the reported literature, this setting appears both as zero-shot localization from rich referring expressions and as prompt-based extraction of object boxes from open-source vision-LLMs (VLMs). The supplied sources describe two distinct mechanisms for this problem: learning region-text alignment for arbitrary concepts through pseudo-labeling in a CLIP-based detector, and modifying final-layer cross-modal attention at inference time to improve localization and interpretability without retraining (Kang et al., 2023, Juanico et al., 26 Jul 2025).

1. Scope and task formulation

Open-vocabulary object detection (OVOD) is described as extending target vocabulary from pre-defined categories to open-world by transferring knowledge of arbitrary concepts from vision-language pre-training models to detectors. Within that framing, prior OVOD methods are characterized as suffering from indirect supervision or limited transferable concepts, and the PLAC formulation is introduced to directly learn region-text alignment for arbitrary concepts (Kang et al., 2023).

For referring-object localization, the reported RefCOCOg protocol uses the validation split with approximately $47\text{K}$ expressions, where queries describe individual objects with rich attributes, including color, relationships, and text. Evaluation is zero-shot: no fine-tuning on referring-expression-comprehension data is performed. Given each image-query pair, the OVOD model scores regions against a single text embedding and returns the top-$k$ regions, with $k \in \{1,5,10\}$ (Kang et al., 2023).

A second formulation treats OV-RefOD as prompt-conditioned box prediction in general-purpose VLMs. The reported benchmark uses the COCO val 2017 novel split with 2,064 $(\text{image}, \text{query})$ pairs. Queries take the form: “Give the normalized bounding box coordinates in $[x1,y1,x2,y2]$ of all instances of $\{cls\}$ in the image.” Ground truth is drawn from the COCO base-vocabulary subset (Juanico et al., 26 Jul 2025).

These protocols indicate that OV-RefOD, as represented here, covers both arbitrary-concept grounding from free-form language and open-vocabulary localization from explicit detection prompts.

2. PLAC: pseudo-labeling for arbitrary concepts

The PLAC method uses Deformable DETR with a Swin-Transformer backbone, either Tiny or Base. The detector outputs $N$ object queries, each decoded into a region embedding $o_i \in \mathbb{R}^D$, a bounding box $b_i$, and a classification score. For any text concept $c$, whether a base class name or a pseudo-label, a frozen CLIP text embedding $k$0 is obtained via $k$1. Region-text alignment is scored by

$k$2

with fixed $k$3 and $k$4. At inference, a free-form query text $k$5 is encoded as $k$6, each region embedding $k$7 is scored by $k$8, and the top-$k$9 regions are selected (Kang et al., 2023).

The pseudo-label generator itself is trained on CC3M, consisting of $k \in \{1,5,10\}$0 million web image-alt-text pairs, while CLIP encoders remain frozen. For each pair $k \in \{1,5,10\}$1,

$k \in \{1,5,10\}$2

The PLAC module is a 3-layer MLP with GeLU mapping $k \in \{1,5,10\}$3: $k \in \{1,5,10\}$4 Training combines mean-squared error,

$k \in \{1,5,10\}$5

with Relational KD (RKD),

$k \in \{1,5,10\}$6

where

$k \in \{1,5,10\}$7

and $k \in \{1,5,10\}$8 is the average distance over the batch. The overall objective is

$k \in \{1,5,10\}$9

with $(\text{image}, \text{query})$0. Optimization uses AdamW with learning rate $(\text{image}, \text{query})$1, batch size $(\text{image}, \text{query})$2, for $(\text{image}, \text{query})$3 epochs (Kang et al., 2023).

Pseudo-label extraction on detection images proceeds by training a Region Proposal Network using Deformable DETR as RPN on LVIS-base annotations only. From each training image, up to approximately $(\text{image}, \text{query})$4 proposals are extracted. Proposals are discarded if they have IoU $(\text{image}, \text{query})$5 against any base annotation or objectness $(\text{image}, \text{query})$6. For each remaining proposal $(\text{image}, \text{query})$7, the image crop is encoded with $(\text{image}, \text{query})$8, projected through PLAC, and the resulting $(\text{image}, \text{query})$9 is stored as the region’s pseudo-text embedding (Kang et al., 2023).

3. Detector training and zero-shot OV-RefOD with PLAC

Full OVOD training uses two-stage bipartite matching. In Stage 1, object queries are matched to annotated base-class text embeddings $[x1,y1,x2,y2]$0. In Stage 2, leftover queries are matched to PLAC pseudo-labels $[x1,y1,x2,y2]$1 on proposals. Matching uses the Hungarian algorithm with a cost combining classification and, for base annotations, box regression (Kang et al., 2023).

For each matched pair $[x1,y1,x2,y2]$2, the binary target is $[x1,y1,x2,y2]$3, and for all other $[x1,y1,x2,y2]$4, $[x1,y1,x2,y2]$5. The classification loss is binary cross entropy,

$[x1,y1,x2,y2]$6

The box loss, applied only for Stage 1 base matches, is

$[x1,y1,x2,y2]$7

The total OVOD loss is

$[x1,y1,x2,y2]$8

with $[x1,y1,x2,y2]$9 in Deformable DETR (Kang et al., 2023).

On LVIS rare classes, the reported AP$\{cls\}$0 values are as follows. In the controlled setting with fewer than $\{cls\}$1 million backbone parameters: ViLD (RN50) achieves $\{cls\}$2, RegionCLIP $\{cls\}$3, VLDet (Swin-T) $\{cls\}$4, and PLAC (Swin-T) $\{cls\}$5, which is reported as $\{cls\}$6 over base-only. In the large-scale setting with more than $\{cls\}$7 million parameters: OWL-ViT (ViT-L) achieves $\{cls\}$8, VLDet (Swin-B) $\{cls\}$9, and PLAC (Swin-B) $N$0 (Kang et al., 2023).

For RefCOCOg zero-shot Precision@$N$1, the reported values are:

• VLDet T: $N$2 at @1, $N$3 at @5, $N$4 at @10
• PLAC T: $N$5 at @1, $N$6 at @5, $N$7 at @10, with increases of $N$8, $N$9, and $o_i \in \mathbb{R}^D$0
• VLDet B: $o_i \in \mathbb{R}^D$1 at @1, $o_i \in \mathbb{R}^D$2 at @5, $o_i \in \mathbb{R}^D$3 at @10
• PLAC B: $o_i \in \mathbb{R}^D$4 at @1, $o_i \in \mathbb{R}^D$5 at @5, $o_i \in \mathbb{R}^D$6 at @10, with increases of $o_i \in \mathbb{R}^D$7, $o_i \in \mathbb{R}^D$8, and $o_i \in \mathbb{R}^D$9 (Kang et al., 2023)

A common misconception is that open-vocabulary referring localization is confined to noun concepts. The RefCOCOg results are reported specifically for arbitrary concepts, and the source explicitly distinguishes noun-concept benchmarks from referring-expression evaluation with richer linguistic content (Kang et al., 2023).

4. Reverse Contrast Attention as an inference-time intervention

Reverse Contrast Attention (RCA) is formulated for vision-language transformers at the final transformer layer. Hidden states $b_i$0 are produced from value vectors $b_i$1 via attention distribution $b_i$2: $b_i$3 RCA reweights each $b_i$4 so that mid-range activations near a central value $b_i$5 are amplified while extreme high or low activations are suppressed. Two equivalent options are given: $b_i$6 or

$b_i$7

The parameter $b_i$8 has default value $b_i$9, and $c$0 is chosen as the mean of the column-wise max attention across heads (Juanico et al., 26 Jul 2025).

The modified weights are renormalized by

$c$1

The method can also be viewed as an element-wise flooring of hidden states,

$c$2

with threshold $c$3 tied to $c$4. The RCA-modified hidden state is

$c$5

For each dimension $c$6, using

$c$7

the decomposition

$c$8

leads to the flooring inequality

$c$9

where $k$00 is the minimal subthreshold component. When $k$01 is small, $k$02, enforcing the flooring operation (Juanico et al., 26 Jul 2025).

Algorithmically, RCA is inserted after the final-layer unnormalized attention scores $k$03 are scaled by $k$04 and softmaxed. Per-query $k$05 is computed by taking cross-head maxima, $k$06 is computed as $k$07, $k$08 is formed by one of the two reweighting rules, the result is renormalized, and final hidden states are recomputed. No fine-tuning is required (Juanico et al., 26 Jul 2025).

5. Metrics and experimental protocols

The PLAC-based RefCOCOg evaluation uses Precision@$k$09, defined as the fraction of queries whose ground-truth box appears in the top-$k$10. The protocol is explicitly zero-shot, with the model run directly on image-query pairs and regions ranked by the single query text embedding (Kang et al., 2023).

The RCA study introduces FitAP because many VLMs do not supply usable confidence scores for conventional object-detection AP. FitAP is defined as

$k$11

where $k$12. For each threshold $k$13, detections are ranked by

$k$14

where $k$15 is normalized box area and $k$16 is computed against the closest unmatched ground-truth box at threshold $k$17. Precision-recall curves are then formed in the standard way over this ranking, and the area under the curve gives $k$18 (Juanico et al., 26 Jul 2025).

The RCA evaluation covers fifteen open-source VLMs. Reported late-fusion or modular models are Ovis2-34B, SAIL-VL-1.6-8B, Qwen2.5-VL-7B, MiniCPM-o-2.6, Kimi-VL-A3B, Gemma3-27B, VARCO-VISION-14B, PaliGemma2-3B-mix-448, and Moondream2. Reported early-fusion or tightly integrated models are DeepSeek-VL2, Valley-Eagle, WeThink-Qwen2.5VL-7B, POINTS1.5-Qwen2.5-7B, valley2_dpo, and Ristretto-3B. All checkpoints come from HuggingFace, LLM size is below $k$19B, model outputs are parsed with regex to extract boxes normalized to $k$20, and attention is extracted at the final cross-modal layer (Juanico et al., 26 Jul 2025).

A second common misconception is that confidence-based AP is the only valid way to evaluate detection quality. The FitAP construction is an explicit counterexample: it replaces detector confidence with the product of normalized box area and IoU when model outputs do not expose a conventional confidence channel (Juanico et al., 26 Jul 2025).

6. Empirical patterns, interpretability, and reported directions

For RCA, the reported FitAP table shows that $k$21 of $k$22 models improved. Qwen2.5-VL-7B increases from $k$23 to $k$24, a reported $k$25, and MiniCPM-o-2.6 increases from $k$26 to $k$27, a reported $k$28. Other positive changes include SAIL-VL-1.6-8B from $k$29 to $k$30 $k$31, DeepSeek-VL2 from $k$32 to $k$33 $k$34, PaliGemma2-3B-mix from $k$35 to $k$36 $k$37, and Moondream2 from $k$38 to $k$39 $k$40. Negative cases include WeThink-Qwen2.5VL-7B $k$41, Ristretto-3B $k$42, POINTS1.5-Qwen2.5-7B $k$43, and Valley-Eagle $k$44 (Juanico et al., 26 Jul 2025).

The interpretability analysis ties performance to attention sharpness and fusion timing. The sharpness measure $k$45 is the mean column-max of multi-head attention, and $k$46 is the number of hidden-state dimensions below $k$47. Pearson correlations between $k$48 and $k$49 are reported as $k$50 for Qwen2.5-VL-7B, $k$51 for DeepSeek-VL2, and $k$52 (not significant) for WeThink. In the two benefit cases, increasing $k$53 correlates with decreasing $k$54, which is reported as supporting the flooring inequality. The same analysis states that late-fusion models benefit most from RCA, while DeepSeek-VL2, despite being an early-fusion Mixture-of-Experts model, also improves, suggesting that internal disentanglement can arise under early fusion if capacity and gating permit (Juanico et al., 26 Jul 2025).

Qualitative visualizations are consistent with the quantitative results. Reported examples include recovery of missed boxes for bus, elephant, and small objects in Qwen2.5-VL-7B, sharper localization of elephants, cups, umbrellas, and airplanes in PaliGemma2-3B-mix and valley2_dpo, and a “kite” case in which previously subdued token patches become prominent after RCA, leading to correct detection (Juanico et al., 26 Jul 2025).

For PLAC, practical recommendations emphasize training the PLAC MLP on large-scale image-text datasets such as CC3M, COYO-700M, and LAION-400M; keeping CLIP encoders frozen; and using two-stage Hungarian matching so that true base annotations are matched first and noisy pseudo-labels do not overwhelm learning. Suggested extensions include adding a cross-modal contrastive loss, hard-negative mining, end-to-end PLAC integration into DETR’s decoder, and leveraging relation phrases such as verbs and adjectives through soft templates or multi-sentence descriptions (Kang et al., 2023).

Taken together, the reported literature indicates two non-exclusive trajectories for OV-RefOD. One trajectory expands detector supervision beyond noun concepts by generating pseudo-text embeddings for arbitrary regions and training a CLIP-based detector against them. The other trajectory modifies cross-modal attention at inference time to expose semantically relevant but previously subdued evidence in multimodal transformers. This suggests that OV-RefOD can be advanced either by strengthening the training signal for arbitrary concepts or by rebalancing inference-time attention in already trained VLMs (Kang et al., 2023, Juanico et al., 26 Jul 2025).