PropVG: Proposal-Driven Visual Grounding

• The paper introduces PropVG, an end-to-end, detector-free framework that revisits the classic proposals plus grounding paradigm using transformers.
• It integrates foreground proposal generation with contrastive refer scoring and multi-granularity target discrimination to enhance both classic and generalized visual grounding.
• PropVG demonstrates improved accuracy and efficiency, achieving higher F1 scores and reduced inference times compared to traditional and end-to-end methods.

PropVG is an end-to-end, proposal-driven visual grounding framework that revisits the classic two-stage “proposals + grounding” paradigm in a fully transformer-based, detector-free form. It is designed for both classic visual grounding and generalized visual grounding, and integrates foreground object proposal generation with referential object comprehension without requiring additional detectors. The framework further introduces a Contrastive-based Refer Scoring module for sentence- and word-level discrimination, and a Multi-granularity Target Discrimination module for target-existence reasoning in generalized settings (Dai et al., 5 Sep 2025).

1. Scope, task setting, and motivation

PropVG addresses four task formulations. In classic visual grounding, it supports Referring Expression Comprehension, which localizes the referred object with a bounding box, and Referring Expression Segmentation, which segments the referred object at pixel level. In generalized visual grounding, it supports Generalized REC and Generalized RES, which must detect or segment zero, one, or multiple referred targets and explicitly determine whether the target exists (Dai et al., 5 Sep 2025).

The framework is motivated by two limitations identified in prior lines of work. Traditional proposal-based two-stage methods such as MAttNet, NMTree, and Ref-NMS depend on pre-trained detectors, are heavy and slow, and are not optimized end-to-end for referential grounding. End-to-end direct referring methods such as TransVG, SeqTR, SimVG, OneRef, and ReLA predict grounding outputs directly from image-text features, but they rely on supervision from the referred target only and therefore under-utilize foreground structure. The paper also argues that existing generalized methods often perform existence prediction from a single scalar or branch, rather than from object-level and semantic-level cues jointly (Dai et al., 5 Sep 2025).

Within this framing, PropVG asks whether a proposal-driven design can be revived without external detectors and whether multi-granularity information can improve both referential scoring and absent-target recognition. This suggests a deliberate synthesis of classical proposal-based grounding with contemporary transformer-based multimodal encoding, rather than a simple return to earlier two-stage pipelines.

2. End-to-end architecture

The input to PropVG is an image $$\mathcal{I} \in \mathbb{R}^{H_v \times W_v \times 3}$$ and a text referring expression $\mathcal{T}$. The model uses BEiT-3 as a large multimodality transformer pre-trained on vision and vision-language tasks. BEiT-3 jointly encodes the image and text through early multimodal fusion, producing visual features $f_v$ of shape $N_i \times C$ and textual token features $f_t \in \mathbb{R}^{N_t \times C}$ (Dai et al., 5 Sep 2025).

A SimFPN module converts the single-scale ViT features into multi-scale maps $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$ for $j \in \{0,1,2,3\}$. Each multi-scale feature map is then split along channels into a segmentation-branch subset and a detection-branch subset. The segmentation pathway uses a UNet decoder to fuse multi-scale features into a high-resolution representation and a SegHead to predict a global segmentation mask $M_{seg} \in [0,1]^{H \times W}$. For RES, this mask corresponds to the referent mask; for GRES, it may represent all regions matching the text or all zeros if the referent is absent (Dai et al., 5 Sep 2025).

The detection pathway initializes $N$ learnable object queries $Q_{init} \in \mathbb{R}^{N \times C}$ and refines them with a multi-scale deformable transformer decoder using the detection-branch features. The decoder outputs proposal queries $\mathcal{T}$0. A DetHead predicts proposal boxes $\mathcal{T}$1 and foreground/background scores $\mathcal{T}$2. A Query Projection module maps $\mathcal{T}$3 into a referring space, after which the referential branch computes refer scores for each proposal (Dai et al., 5 Sep 2025).

The resulting organization is explicitly bifurcated: one branch performs global referring segmentation, while the other performs foreground proposal generation and referential scoring. A plausible implication is that PropVG treats segmentation and proposal reasoning as complementary but not redundant pathways, with the later existence module consuming signals from both.

3. Referential scoring and existence reasoning

A central component of PropVG is the Contrastive-based Refer Scoring module. After Query Projection, the projected proposal queries interact with the text features by Multi-Head Cross-Attention, where the queries attend to textual keys and values, yielding text-aware query features $\mathcal{T}$4. CRS then computes sentence-level and word-level similarities separately (Dai et al., 5 Sep 2025).

For sentence-level scoring, a global text feature $\mathcal{T}$5 is obtained via masked max pooling over token features. For each query, cosine similarity with $\mathcal{T}$6 is computed using a learnable temperature $\mathcal{T}$7 initialized to $\mathcal{T}$8, producing $\mathcal{T}$9. For word-level scoring, cosine similarities between each query and each token yield $f_v$0, and a MaxPool over tokens gives a single word-level score per query. The final refer score is an adaptive fusion:

$f_v$1

where $f_v$2. The paper characterizes this as contrastive scoring because it aligns positive query-text pairs and separates negatives through cosine similarities and BCE supervision (Dai et al., 5 Sep 2025).

The second core component is the Multi-granularity Target Discrimination module, which addresses target-existence prediction in generalized grounding. MTD introduces a learnable existence query $f_v$3 and combines object-level granularity, represented by proposal refer scores $f_v$4, with semantic-level granularity, represented by the segmentation mask $f_v$5. This fusion is implemented with Score Prior Cross Attention blocks:

$f_v$6

Here the prior scores $f_v$7 bias attention toward proposals or pixels with stronger evidence (Dai et al., 5 Sep 2025).

After two SPCA interactions, the model predicts an existence probability $f_v$8. PropVG then defines the final existence score as

$f_v$9

where $N_i \times C$0 is the TopK Average Score over segmentation, obtained by sorting pixel scores, selecting the top $N_i \times C$1, and averaging them. The reported best performance uses $N_i \times C$2 (Dai et al., 5 Sep 2025).

These modules instantiate the paper’s multi-granularity view in two distinct senses: CRS combines sentence-level and word-level language discrimination, while MTD combines proposal-level and pixel-level visual evidence. This suggests that PropVG treats referential identification and existence reasoning as separate but structurally parallel discrimination problems.

4. Training objectives, supervision, and implementation

The total loss in PropVG is

$N_i \times C$3

with default weights $N_i \times C$4, $N_i \times C$5, and $N_i \times C$6 (Dai et al., 5 Sep 2025).

The detection loss is DETR-style. For matched queries, it includes cross-entropy for foreground/background classification and L1 plus GIoU for box regression; unmatched queries receive background classification loss. Referring loss is BCE over proposal refer scores, where positive labels correspond to queries matched to referred targets and negatives include both foreground non-referent objects and background queries. The segmentation loss combines BCE and Dice loss on the global segmentation mask. The existence loss is BCE on $N_i \times C$7, with label $N_i \times C$8 for non-empty expressions and $N_i \times C$9 for empty-target expressions (Dai et al., 5 Sep 2025).

Foreground supervision is an explicit design choice. Training annotations include not only referents but all salient foreground COCO instances filtered by is_crowd = 0, absolute area greater than 100 pixels, and relative area in $f_t \in \mathbb{R}^{N_t \times C}$0 of the image. Hungarian matching assigns queries to foreground objects or to no-object, and the assignment is also used to define positives and negatives for referential supervision (Dai et al., 5 Sep 2025).

The implementation uses BEiT-3 ViT-B as the multimodal encoder, SimFPN for multi-scale features, a UNet decoder for segmentation, and a multi-scale deformable decoder for detection. For generalized visual grounding, the input image size is $f_t \in \mathbb{R}^{N_t \times C}$1; for classic visual grounding, it is $f_t \in \mathbb{R}^{N_t \times C}$2; for ablations, it is $f_t \in \mathbb{R}^{N_t \times C}$3. Training uses Adam with initial learning rates of $f_t \in \mathbb{R}^{N_t \times C}$4 for the BEiT-3 encoder and $f_t \in \mathbb{R}^{N_t \times C}$5 for other modules. The batch size is 16 over 4 NVIDIA 4090 GPUs. The training schedule is 12 epochs for generalized tasks, 30 epochs for classic tasks, and 10 epochs for ablations, with learning-rate decay by a factor of $f_t \in \mathbb{R}^{N_t \times C}$6 at epochs 7 and 11 for the 12-epoch regime (Dai et al., 5 Sep 2025).

At inference time, PropVG outputs proposal boxes, refer scores, a global segmentation mask, and an existence score. The appendix reports that using refer score alone yields the best F1 and gIoU among tested proposal-scoring combinations, that the best proposal threshold is $f_t \in \mathbb{R}^{N_t \times C}$7, and that NMS is optional because DETR outputs are relatively non-redundant (Dai et al., 5 Sep 2025).

5. Benchmarks and empirical performance

PropVG is evaluated on 10 datasets covering REC, RES, GREC, GRES, robustness-oriented datasets, and beyond-one-to-one segmentation. For classic visual grounding, the paper reports results on RefCOCO, RefCOCO+, and RefCOCOg. For generalized grounding and robustness, it reports results on gRefCOCO, Ref-ZOM, and R-RefCOCO, R-RefCOCO+, and R-RefCOCOg (Dai et al., 5 Sep 2025).

On RefCOCO REC, with BEiT-3 and pretraining, PropVG achieves $f_t \in \mathbb{R}^{N_t \times C}$8 on val/testA/testB. On RefCOCO+ REC, it achieves $f_t \in \mathbb{R}^{N_t \times C}$9, and on RefCOCOg REC it achieves $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$0 on val-u/test-u. For RES, the paper states that PropVG beats EEVG by $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$1 mIoU on RefCOCO, $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$2 on RefCOCO+, and $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$3 on RefCOCOg. Compared with MAttNet, PropVG improves REC accuracy by approximately $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$4 on average across splits and reduces inference time to approximately $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$5 ms versus $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$6 ms (Dai et al., 5 Sep 2025).

On gRefCOCO GRES, PropVG reports val gIoU $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$7, cIoU $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$8, and N-acc $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$9, with testA gIoU $j \in \{0,1,2,3\}$0 and testB gIoU $j \in \{0,1,2,3\}$1. On Ref-ZOM, it reports oIoU $j \in \{0,1,2,3\}$2, mIoU $j \in \{0,1,2,3\}$3, and Acc $j \in \{0,1,2,3\}$4. On R-RefCOCO, R-RefCOCO+, and R-RefCOCOg, it reports rIoU values of $j \in \{0,1,2,3\}$5, exceeding the HDC baseline by $j \in \{0,1,2,3\}$6, $j \in \{0,1,2,3\}$7, and $j \in \{0,1,2,3\}$8, respectively. On gRefCOCO GREC, PropVG outperforms SimVG in F1score with $j \in \{0,1,2,3\}$9 versus $M_{seg} \in [0,1]^{H \times W}$0 on val, $M_{seg} \in [0,1]^{H \times W}$1 versus $M_{seg} \in [0,1]^{H \times W}$2 on testA, and $M_{seg} \in [0,1]^{H \times W}$3 versus $M_{seg} \in [0,1]^{H \times W}$4 on testB (Dai et al., 5 Sep 2025).

The paper also emphasizes that PropVG is competitive with or better than several MLLM-based systems such as LISA, GSVA, and GLaMM while using far fewer parameters, citing approximately $M_{seg} \in [0,1]^{H \times W}$5B parameters versus $M_{seg} \in [0,1]^{H \times W}$6B or more for those baselines. This suggests that the framework’s gains are not attributable solely to scale, but to the proposal-driven and multi-granularity design.

6. Ablations, interpretation, and relation to similarly named models

The ablation studies isolate the effects of PropVG’s components. Starting from a baseline, the addition of SimFPN, a UNet decoder, a multi-scale deformable decoder, channel splitting, and Query Projection improves F1score from $M_{seg} \in [0,1]^{H \times W}$7 to $M_{seg} \in [0,1]^{H \times W}$8 and gIoU from $M_{seg} \in [0,1]^{H \times W}$9 to $N$0. Removing foreground supervision and reverting to a direct-referring formulation reduces F1 by approximately $N$1, which the paper uses to support the value of proposal-based decomposition and extra supervision (Dai et al., 5 Sep 2025).

Further ablations show that CRS and MTD each provide distinct gains. The baseline yields F1 $N$2, N-acc $N$3, and gIoU $N$4. Adding CRS only gives F1 $N$5 and gIoU $N$6. Adding MTD only gives F1 $N$7, gIoU $N$8, and N-acc $N$9. Using both modules together yields F1 $Q_{init} \in \mathbb{R}^{N \times C}$0, N-acc $Q_{init} \in \mathbb{R}^{N \times C}$1, and gIoU $Q_{init} \in \mathbb{R}^{N \times C}$2. Within CRS, sentence-only and word-only contrastive scoring both help, but adaptive fusion performs best. Within MTD, SPCA over queries alone or masks alone improves performance, while combining both together with multiplicative fusion using $Q_{init} \in \mathbb{R}^{N \times C}$3 and TAS produces the strongest existence and segmentation results (Dai et al., 5 Sep 2025).

The qualitative analysis associates proposal-driven supervision with reduced false positives on non-referent foreground objects. It also reports that different expressions over the same image can emphasize different foreground proposals because BEiT-3 performs multimodal encoding. Examples cited in the paper include different proposal emphases for “a guy in an orange tie” versus “all human beings,” and strong handling of crowded scenes, multi-target descriptions, and empty targets (Dai et al., 5 Sep 2025).

A recurring source of confusion is nomenclature. The literature also contains “PVG: Progressive Vision Graph for Vision Recognition,” which is a graph-based vision backbone for image recognition rather than a grounding system (Wu et al., 2023), and “ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery,” which is a remote sensing visual grounding framework based on language decoupling and a survey-locate-verify scheme (Li et al., 2 Apr 2026). PropVG is distinct from both: it is proposal-driven, detector-free, and centered on end-to-end foreground proposal learning, contrastive refer scoring, and multi-granularity existence reasoning (Dai et al., 5 Sep 2025).

The paper does not foreground limitations extensively, but it does imply several. Proposal quality remains consequential, particularly for tiny objects or rare categories. Segmentation is produced as a global mask rather than as per-instance masks, so multi-target segmentation is not instance-aware. The architecture also remains computationally substantial because it uses BEiT-3 and deformable transformer decoders, even if it is more efficient than classic two-stage pipelines (Dai et al., 5 Sep 2025). Future directions mentioned in context include instance-level segmentation per proposal, parameter-efficient finetuning for BEiT-3, and integration with larger multimodal LLMs.