---
title: 'PropVG: Proposal-Driven Visual Grounding'
url: https://www.emergentmind.com/topics/propvg
type: topic
---

# PropVG: Proposal-Driven Visual Grounding

PropVG is an end-to-end, proposal-driven visual grounding framework that revisits the classic two-stage “proposals + grounding” paradigm in a fully transformer-based, detector-free form. It is designed for both classic visual grounding and generalized visual grounding, and integrates foreground object proposal generation with referential object comprehension without requiring additional detectors. The framework further introduces a Contrastive-based Refer Scoring module for sentence- and word-level discrimination, and a Multi-granularity Target Discrimination module for target-existence reasoning in generalized settings [2509.04833].

## 1. Scope, task setting, and motivation

PropVG addresses four task formulations. In classic visual grounding, it supports Referring Expression Comprehension, which localizes the referred object with a bounding box, and Referring Expression Segmentation, which segments the referred object at pixel level. In generalized visual grounding, it supports Generalized REC and Generalized RES, which must detect or segment zero, one, or multiple referred targets and explicitly determine whether the target exists [2509.04833].

The framework is motivated by two limitations identified in prior lines of work. Traditional proposal-based two-stage methods such as MAttNet, NMTree, and Ref-NMS depend on pre-trained detectors, are heavy and slow, and are not optimized end-to-end for referential grounding. End-to-end direct referring methods such as TransVG, SeqTR, SimVG, OneRef, and ReLA predict grounding outputs directly from image-text features, but they rely on supervision from the referred target only and therefore under-utilize foreground structure. The paper also argues that existing generalized methods often perform existence prediction from a single scalar or branch, rather than from object-level and semantic-level cues jointly [2509.04833].

Within this framing, PropVG asks whether a proposal-driven design can be revived without external detectors and whether multi-granularity information can improve both referential scoring and absent-target recognition. This suggests a deliberate synthesis of classical proposal-based grounding with contemporary transformer-based multimodal encoding, rather than a simple return to earlier two-stage pipelines.

## 2. End-to-end architecture

The input to PropVG is an image $\mathcal{I} \in \mathbb{R}^{H_v \times W_v \times 3}$ and a text referring expression $\mathcal{T}$. The model uses BEiT-3 as a large multimodality transformer pre-trained on vision and vision-language tasks. BEiT-3 jointly encodes the image and text through early multimodal fusion, producing visual features $f_v$ of shape $N_i \times C$ and textual token features $f_t \in \mathbb{R}^{N_t \times C}$ [2509.04833].

A SimFPN module converts the single-scale ViT features into multi-scale maps $f_v^j \in \mathbb{R}^{H_j \times W_j \times C_j}$ for $j \in \{0,1,2,3\}$. Each multi-scale feature map is then split along channels into a segmentation-branch subset and a detection-branch subset. The segmentation pathway uses a UNet decoder to fuse multi-scale features into a high-resolution representation and a SegHead to predict a global segmentation mask $M_{seg} \in [0,1]^{H \times W}$. For RES, this mask corresponds to the referent mask; for GRES, it may represent all regions matching the text or all zeros if the referent is absent [2509.04833].

The detection pathway initializes $N$ learnable object queries $Q_{init} \in \mathbb{R}^{N \times C}$ and refines them with a multi-scale deformable transformer decoder using the detection-branch features. The decoder outputs proposal queries $Q_{prop} \in \mathbb{R}^{N \times C}$. A DetHead predicts proposal boxes $P_{bbox} \in \mathbb{R}^{N \times 4}$ and foreground/background scores $P_{score} \in \mathbb{R}^{N \times 2}$. A Query Projection module maps $Q_{prop}$ into a referring space, after which the referential branch computes refer scores for each proposal [2509.04833].

The resulting organization is explicitly bifurcated: one branch performs global referring segmentation, while the other performs foreground proposal generation and referential scoring. A plausible implication is that PropVG treats segmentation and proposal reasoning as complementary but not redundant pathways, with the later existence module consuming signals from both.

## 3. Referential scoring and existence reasoning

A central component of PropVG is the Contrastive-based Refer Scoring module. After Query Projection, the projected proposal queries interact with the text features by Multi-Head Cross-Attention, where the queries attend to textual keys and values, yielding text-aware query features $Q_p \in \mathbb{R}^{N \times C}$. CRS then computes sentence-level and word-level similarities separately [2509.04833].

For sentence-level scoring, a global text feature $f_s$ is obtained via masked max pooling over token features. For each query, cosine similarity with $f_s$ is computed using a learnable temperature $T$ initialized to $0.07$, producing $S_{sent} \in \mathbb{R}^{N \times 1}$. For word-level scoring, cosine similarities between each query and each token yield $S_{word} \in \mathbb{R}^{N \times N_t}$, and a MaxPool over tokens gives a single word-level score per query. The final refer score is an adaptive fusion:
$$
S_{ref} = w_s \cdot S_{sent} + (1 - w_s) \cdot \text{MaxPool}(S_{word}),
$$
where $w_s = \sigma(\text{MLP}(f_s)) \in (0,1)$. The paper characterizes this as contrastive scoring because it aligns positive query-text pairs and separates negatives through cosine similarities and BCE supervision [2509.04833].

The second core component is the Multi-granularity Target Discrimination module, which addresses target-existence prediction in generalized grounding. MTD introduces a learnable existence query $Q_{exist} \in \mathbb{R}^{1 \times C}$ and combines object-level granularity, represented by proposal refer scores $S_{ref}$, with semantic-level granularity, represented by the segmentation mask $M_{seg}$. This fusion is implemented with Score Prior Cross Attention blocks:
$$
O = \text{Softmax}\left(QK^{T} + \text{MLP}(S)\right) V.
$$
Here the prior scores $S$ bias attention toward proposals or pixels with stronger evidence [2509.04833].

After two SPCA interactions, the model predicts an existence probability $\varepsilon_{exist} \in [0,1]$. PropVG then defines the final existence score as
$$
S_{exist} = \text{Max}(S_{ref}) \times \text{TAS}(M_{seg}) \times \varepsilon_{exist},
$$
where $\text{TAS}(M_{seg})$ is the TopK Average Score over segmentation, obtained by sorting pixel scores, selecting the top $K$, and averaging them. The reported best performance uses $K=250$ [2509.04833].

These modules instantiate the paper’s multi-granularity view in two distinct senses: CRS combines sentence-level and word-level language discrimination, while MTD combines proposal-level and pixel-level visual evidence. This suggests that PropVG treats referential identification and existence reasoning as separate but structurally parallel discrimination problems.

## 4. Training objectives, supervision, and implementation

The total loss in PropVG is
$$
\mathcal{L}_{total} = \mathcal{L}_{seg} + \lambda_{det} \cdot \mathcal{L}_{det} + \lambda_{exist} \cdot \mathcal{L}_{exist} + \lambda_{ref} \cdot \mathcal{L}_{ref},
$$
with default weights $\lambda_{det}=0.1$, $\lambda_{exist}=0.2$, and $\lambda_{ref}=1.0$ [2509.04833].

The detection loss is DETR-style. For matched queries, it includes cross-entropy for foreground/background classification and L1 plus GIoU for box regression; unmatched queries receive background classification loss. Referring loss is BCE over proposal refer scores, where positive labels correspond to queries matched to referred targets and negatives include both foreground non-referent objects and background queries. The segmentation loss combines BCE and Dice loss on the global segmentation mask. The existence loss is BCE on $S_{exist}$, with label $1$ for non-empty expressions and $0$ for empty-target expressions [2509.04833].

Foreground supervision is an explicit design choice. Training annotations include not only referents but all salient foreground COCO instances filtered by `is_crowd = 0`, absolute area greater than 100 pixels, and relative area in $[0.05, 0.8]$ of the image. Hungarian matching assigns queries to foreground objects or to no-object, and the assignment is also used to define positives and negatives for referential supervision [2509.04833].

The implementation uses BEiT-3 ViT-B as the multimodal encoder, SimFPN for multi-scale features, a UNet decoder for segmentation, and a multi-scale deformable decoder for detection. For generalized visual grounding, the input image size is $320 \times 320$; for classic visual grounding, it is $384 \times 384$; for ablations, it is $224 \times 224$. Training uses Adam with initial learning rates of $5\times10^{-5}$ for the BEiT-3 encoder and $5\times10^{-4}$ for other modules. The batch size is 16 over 4 NVIDIA 4090 GPUs. The training schedule is 12 epochs for generalized tasks, 30 epochs for classic tasks, and 10 epochs for ablations, with learning-rate decay by a factor of $0.1$ at epochs 7 and 11 for the 12-epoch regime [2509.04833].

At inference time, PropVG outputs proposal boxes, refer scores, a global segmentation mask, and an existence score. The appendix reports that using refer score alone yields the best F1 and gIoU among tested proposal-scoring combinations, that the best proposal threshold is $Thr_p = 0.9$, and that NMS is optional because DETR outputs are relatively non-redundant [2509.04833].

## 5. Benchmarks and empirical performance

PropVG is evaluated on 10 datasets covering REC, RES, GREC, GRES, robustness-oriented datasets, and beyond-one-to-one segmentation. For classic visual grounding, the paper reports results on RefCOCO, RefCOCO+, and RefCOCOg. For generalized grounding and robustness, it reports results on gRefCOCO, Ref-ZOM, and R-RefCOCO, R-RefCOCO+, and R-RefCOCOg [2509.04833].

On RefCOCO REC, with BEiT-3 and pretraining, PropVG achieves $92.70 / 95.07 / 89.58$ on val/testA/testB. On RefCOCO+ REC, it achieves $87.27 / 90.87 / 81.26$, and on RefCOCOg REC it achieves $88.15 / 88.30$ on val-u/test-u. For RES, the paper states that PropVG beats EEVG by $+1.4\%$ mIoU on RefCOCO, $+4.0\%$ on RefCOCO+, and $+2.4\%$ on RefCOCOg. Compared with MAttNet, PropVG improves REC accuracy by approximately $+14\%$ on average across splits and reduces inference time to approximately $76$ ms versus $320$ ms [2509.04833].

On gRefCOCO GRES, PropVG reports val gIoU $73.29$, cIoU $69.23$, and N-acc $72.83$, with testA gIoU $74.43$ and testB gIoU $65.87$. On Ref-ZOM, it reports oIoU $71.95$, mIoU $71.15$, and Acc $98.11$. On R-RefCOCO, R-RefCOCO+, and R-RefCOCOg, it reports rIoU values of $62.34 / 59.04 / 55.09$, exceeding the HDC baseline by $+9.5$, $+10.0$, and $+11.2$, respectively. On gRefCOCO GREC, PropVG outperforms SimVG in F1score with $72.2$ versus $62.1$ on val, $68.8$ versus $64.6$ on testA, and $59.0$ versus $54.8$ on testB [2509.04833].

The paper also emphasizes that PropVG is competitive with or better than several MLLM-based systems such as LISA, GSVA, and GLaMM while using far fewer parameters, citing approximately $0.2$B parameters versus $7$B or more for those baselines. This suggests that the framework’s gains are not attributable solely to scale, but to the proposal-driven and multi-granularity design.

## 6. Ablations, interpretation, and relation to similarly named models

The ablation studies isolate the effects of PropVG’s components. Starting from a baseline, the addition of SimFPN, a UNet decoder, a multi-scale deformable decoder, channel splitting, and Query Projection improves F1score from $63.41$ to $68.81$ and gIoU from $65.98$ to $69.85$. Removing foreground supervision and reverting to a direct-referring formulation reduces F1 by approximately $2\%$, which the paper uses to support the value of proposal-based decomposition and extra supervision [2509.04833].

Further ablations show that CRS and MTD each provide distinct gains. The baseline yields F1 $68.81$, N-acc $70.39$, and gIoU $69.85$. Adding CRS only gives F1 $70.61$ and gIoU $71.30$. Adding MTD only gives F1 $70.28$, gIoU $71.26$, and N-acc $73.17$. Using both modules together yields F1 $71.00$, N-acc $75.81$, and gIoU $71.99$. Within CRS, sentence-only and word-only contrastive scoring both help, but adaptive fusion performs best. Within MTD, SPCA over queries alone or masks alone improves performance, while combining both together with multiplicative fusion using $S_{ref}$ and TAS produces the strongest existence and segmentation results [2509.04833].

The qualitative analysis associates proposal-driven supervision with reduced false positives on non-referent foreground objects. It also reports that different expressions over the same image can emphasize different foreground proposals because BEiT-3 performs multimodal encoding. Examples cited in the paper include different proposal emphases for “a guy in an orange tie” versus “all human beings,” and strong handling of crowded scenes, multi-target descriptions, and empty targets [2509.04833].

A recurring source of confusion is nomenclature. The literature also contains “PVG: Progressive Vision Graph for Vision Recognition,” which is a graph-based vision backbone for image recognition rather than a grounding system [2308.00574], and “ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery,” which is a remote sensing visual grounding framework based on language decoupling and a survey-locate-verify scheme [2604.01893]. PropVG is distinct from both: it is proposal-driven, detector-free, and centered on end-to-end foreground proposal learning, contrastive refer scoring, and multi-granularity existence reasoning [2509.04833].

The paper does not foreground limitations extensively, but it does imply several. Proposal quality remains consequential, particularly for tiny objects or rare categories. Segmentation is produced as a global mask rather than as per-instance masks, so multi-target segmentation is not instance-aware. The architecture also remains computationally substantial because it uses BEiT-3 and deformable transformer decoders, even if it is more efficient than classic two-stage pipelines [2509.04833]. Future directions mentioned in context include instance-level segmentation per proposal, parameter-efficient finetuning for BEiT-3, and integration with larger multimodal language models.

Source: https://www.emergentmind.com/topics/propvg