Generalized Referring Expression (GREx)

Updated 3 July 2026

Generalized Referring Expression (GREx) is a unifying framework that expands classical referring expression tasks to scenarios where free-form text may denote zero, one, or many visual objects across images, videos, and 3D scenes.
It leverages benchmark datasets like gRefCOCO and metrics such as cIoU, gIoU, and N-acc to rigorously evaluate performance in segmentation, comprehension, and generation tasks.
The framework employs cross-modal transformers, hierarchical decoding, and adaptive binding prototypes to align visual and textual information, driving state-of-the-art advances in grounded vision–language research.

Generalized Referring Expression (GREx) is a unifying framework that extends classical referring expression tasks to real-world conditions in which a free-form expression may refer to any number of visual objects—including none, one, or many—across images, videos, and even 3D scenes. GREx encompasses segmentation, comprehension (box or region detection), and generation, and has become foundational for grounded vision-language research that targets the full complexity of human–AI communication (Nguyen et al., 2024, Liu et al., 2023, Ding et al., 8 Jan 2026, Ding et al., 1 Aug 2025). The following sections detail formal definitions, task variants, meta-architectures, benchmark datasets and metrics, leading model designs, and empirical findings.

1. Formalization and Taxonomy

GREx generalizes classical Referring Expression Segmentation (RES), Referring Expression Comprehension (REC), and Referring Expression Generation (REG) under a paradigm where expressions may denote an arbitrary number of entities (possibly zero) in a visual domain. This expansion yields three principal tasks (Ding et al., 8 Jan 2026, Nguyen et al., 2024, Liu et al., 2023):

Generalized Referring Expression Segmentation (GRES): Given an image $I$ (or visual scene $V$ ) and text expression $T$ , output a segmentation mask $M \in [0,1]^{H \times W}$ (or set of masks), which covers the union of all targets referred by $T$ ; for zero-target, $M$ is empty.
Generalized Referring Expression Comprehension (GREC): Given $(I, T)$ , output a set $B = \{b_i\}_{i=1}^{N}$ of bounding boxes (or regions), $N \geq 0$ .
Generalized Referring Expression Generation (GREG): Given $I$ and a user-designated subset of regions or objects (as boxes or a segmentation mask), generate $V$ 0 such that it uniquely identifies exactly that (possibly multi-object) set.

Zero-, single-, and multi-target cases are treated uniformly, making GREx suitable for naturalistic instructions such as “all birds on the left except the small one,” “the man with no hat,” or “the unicorn in the image” (where the referent may not exist).

2. Benchmark Datasets and Evaluation Metrics

The gRefCOCO benchmark, derived from COCO, is the de facto evaluation corpus for GREx tasks (Liu et al., 2023, He et al., 2023, Ding et al., 8 Jan 2026). It comprises 19,994 images and ≈278,000 natural-language expressions annotated for multi-object, single-object, and no-object (zero-target) references:

Expression Type	#Instances	% of Dataset
Single-target	~135,258	≈52%
Multi-target	~90,064	≈35%
No-target	~34,537	≈13%

Each instance provides per-object segmentation masks, bounding boxes, and expression-level tags.

Key metrics:

cIoU (Cumulative IoU): $V$ 1; overall intersection over union.
gIoU (Global IoU): $V$ 2, treating perfectly correct no-target predictions as 1, false positives as 0.
N-acc (No-target Accuracy): Fraction of no-target samples where the model outputs an empty result.
Precision@ $V$ 3: Fraction of examples where all and only the correct instances (boxes or masks) are identified at IoU ≥ 0.5 (He et al., 2023, Ding et al., 8 Jan 2026).

Specialized datasets such as Ref-ZOM (object absence scenarios), Multi3DRes (3D point clouds; (Wu et al., 2024)), and dialogue-based GREC corpora (for discourse context; (Shao et al., 2 Dec 2025)) extend these protocols.

3. Unified Model Meta-Architecture

Leading GREx models employ a multi-stage architecture centered on cross-modal transformers and region-/instance-aware decoding (Nguyen et al., 2024, Luo et al., 2024, Li et al., 2024, Ding et al., 1 Aug 2025). The common pipeline consists of:

Visual Encoder: Backbone (typically Swin Transformer, ViT, 3D U-Net) extracts feature maps ( $V$ 4) at multiple scales.
Text Encoder: A language backbone (BERT, RoBERTa, LLMs) encodes the expression ( $V$ 5).
Cross-Modal Interaction: Via transformer blocks (e.g., Phrase-Object Transformer (POT), cross-attention, region-language fusion), N learnable or dynamically generated queries represent candidate instances or regions.
Instance/Phrase Alignment: For each query $V$ 6, joint alignment to text phrases is enforced; e.g., cosine similarity loss between object query and associated phrase embedding (Nguyen et al., 2024).
Mask or Box Prediction: Each query yields a candidate mask ( $V$ 7) or box; instance matching is performed (e.g. via Hungarian algorithm).
No-Target Prediction: Combines global instance and sentence features to predict absence of referents, trained with binary cross-entropy.
Hierarchical/Adaptive Aggregation: Merging predicted masks (weighted softmax or PReLU), possibly accompanied by hierarchical or multi-granular decoding (Luo et al., 2024).

Recent approaches incorporate instance-aware matching, explicit counting heads, adaptive region proposals, and deep multimodal semantic alignment to address multi-target and compositionality challenges.

4. Leading Methods and Innovations

Instance-Level and Phrase-Object Alignment

InstAlign (Nguyen et al., 2024) pioneers explicit object-level reasoning by generating a set of $V$ 8 object tokens $V$ 9 aligned via a cross-modal transformer and supervised with a phrase-object alignment loss: $T$ 0 where $T$ 1 is the sub-phrase embedding most relevant to query $T$ 2. Joint optimization of instance matching, mask prediction, phrase binding, global aggregation, and no-target scoring yields state-of-the-art gIoU and N-acc (gIoU 74.34%, N-acc 79.72 on gRefCOCO).

Hierarchical Decoding and Counting

CoHD (Luo et al., 2024) introduces a hierarchical decoding pipeline supported by an adaptive object counting module. Fine-to-coarse semantic maps are cross-modally decoded and aggregated via intra-/inter-level weighting: $T$ 3 where $T$ 4 is the channel-attended semantic map at level $T$ 5 and $T$ 6 is its saliency. This design enhances granularity disambiguation and explicit counting (SmoothL1 loss on predicted object counts), outperforming previous methods on multiple benchmarks.

Adaptive Binding Prototypes

MABP (Li et al., 2024) disperses instance matching across a grid of region-specific queries, each serving as an adaptive prototype. This strategy distributes learning pressure in binary-supervised GRES and improves handling of compositional queries. The resulting model surpasses earlier SOTA (gIoU 68.9%, N-acc 62.2%).

Multimodal LLM and Foundation Model Integration

GSVA (Xia et al., 2023) leverages LLMs extended with explicit SEG and REJ tokens to prompt a segmentation foundation model (SAM). Each [SEG] token outputs a mask; [REJ] signals absence. Learned prompt structure enables robust multi-target and no-target segmentation, producing gIoU competitive with SOTA (66.47% GSVA-7B finetuned) and substantially boosting no-target rejection.

5. Empirical Performance and Ablation Insights

Recent models achieve substantial gains via (i) explicit instance-level reasoning, (ii) phrase-object binding, (iii) hierarchical or region-adaptive aggregation, and (iv) robust no-target classification (Nguyen et al., 2024, Luo et al., 2024, Li et al., 2024). Table: (gRefCOCO, val split, cIoU/gIoU/N-acc):

Method	cIoU	gIoU	N-acc
ReLA	65.5	63.98	57.5
MABP	65.7	68.9	62.2
CoHD	66.4	68.4	62.8
InstAlign	68.94	74.34	79.72
GSVA-7B	63.29	66.47	62.43
Latent-VG	72.45	68.23	70.42

Ablations demonstrate that removing explicit instance matching, hierarchical aggregation weights, or phrase-level alignment degrades both segmentation and no-target accuracy by 2–10 percentage points (Nguyen et al., 2024, Luo et al., 2024).

6. GREx Extensions: Weak Supervision, 3D, and Dialogue

Weak Supervision: LIHE (Shi et al., 15 Nov 2025) addresses GREx in the absence of ground-truth boxes by (i) using large vision-LLMs for decoupling and referent count prediction, and (ii) introducing a hybrid Euclidean-hyperbolic metric (HEMix) to prevent semantic collapse of related concepts.

3D GREx: Generalized 3D-GRES (Wu et al., 2024) extends GREx to point clouds, formulating mask prediction over $T$ 7 points or superpoints. MDIN proposes distributed sparse text-driven queries and multi-object decoupling losses, enabling strong performance on Multi3DRes.

Dialogue: Three-tier data synthesis protocols (Shao et al., 2 Dec 2025) enable in-domain, co-referential supervision for dialogue-conditioned GREC. Progressive gains are obtained via staged template, LLM, and dialogue simulation data for fine-tuning transformer-based grounding models.

7. Theoretical Perspectives and Surface Realization

The theoretical foundations of GREx are formalized in logic-based frameworks parameterized by the expressiveness $T$ 8 of the allowed formula fragment (Areces et al., 2010). The $T$ 9-GRE problem defines the set of all $M \in [0,1]^{H \times W}$ 0-referring expressions that refer to a target set, revealing complexity tradeoffs:

For rich $M \in [0,1]^{H \times W}$ 1 (e.g., $M \in [0,1]^{H \times W}$ 2 in DL), GRE has linear-length solutions and polynomial-time algorithms.
For weak $M \in [0,1]^{H \times W}$ 3 (e.g., conjunctive existential logic), GRE may exhibit exponential-length solutions and intractability for some instances.

These considerations guide content determination versus surface realization (generation) modules, affecting the succinctness and practicality of NLG in GREx pipelines.

References: