Referring Grasp Synthesis in Robotics

• Referring Grasp Synthesis (RGS) is a language-conditioned robotic grasping problem that maps natural language and visual inputs to generate precise grasp representations.
• It employs multimodal fusion techniques and depth cue integration to extract grounding masks and dense grasp maps for handling cluttered and ambiguous scenes.
• The approach evolves from planar parallel-jaw grasping to dexterous hand synthesis, emphasizing efficiency, accurate semantic part alignment, and improved grasp metrics.

Searching arXiv for papers on Referring Grasp Synthesis and related language-guided grasping. Referring Grasp Synthesis (RGS) is a language-conditioned robotic grasping problem in which a system maps a referring expression and perceptual observations to both the localization of the referred object and a grasp representation suitable for execution. Across the literature summarized here, the formulation spans planar grasping in cluttered RGB-D tabletop scenes and dexterous full-DOF hand synthesis on 3D objects. In the planar setting, RGS typically predicts a grounding mask together with dense grasp maps for quality, orientation, and gripper width, from which a grasp rectangle is extracted (Yu et al., 2024, Tziafas et al., 2023). In dexterous settings, the output is a hand configuration whose contacts satisfy the language-specified object part while remaining stable and collision-aware (Jian et al., 25 Mar 2025, Chang et al., 2024). The common technical core is multimodal grounding: language must resolve the intended target or target part, and grasp synthesis must remain geometrically valid under clutter, ambiguity, or task-specific contact constraints.

1. Problem formulation and scope

RGS is defined in multiple but related ways. In the OCID-VLG tabletop benchmark introduced with CROG, the task predicts a grasp pose for an object referred through natural language in cluttered scenes (Tziafas et al., 2023). The formal input is an RGB image and a variable-length referring expression; the output includes a binary object mask or equivalent grounding, followed by a grasp pose (Tziafas et al., 2023). In the parameter-efficient CLIP-based formulation, the inputs are explicitly an RGB image $I \in \mathbb{R}^{H \times W \times 3}$, a depth map $D \in \mathbb{R}^{H \times W}$, and a tokenized referring expression $T \in \mathbb{R}^L$, while the outputs are an instance segmentation mask $M \in \{0,1\}^{H \times W}$ and three image-plane maps: grasp quality $Q \in \mathbb{R}^{H \times W}$, grasp orientation $\Theta \in \mathbb{R}^{H \times W}$, and gripper opening width $P \in \mathbb{R}^{H \times W}$ (Yu et al., 2024).

For planar RGS, a final grasp is extracted by choosing the pixel of maximal predicted quality and reading the corresponding angle and width. One formulation gives a final 4-DoF grasp rectangle $G=(x,y,\theta,l)$ with

$$x,y = \arg\max_{u,v} Q(u,v), \quad \theta = \Theta(x,y), \quad l = P(x,y)$$

(Yu et al., 2024). A closely related formulation in OGRG augments this with depth sampling to form a 5-DoF grasp

$$G=\{x,y,z,\theta,l\}, \quad (x,y)=\arg\max_{(i,j)}Q_{ij}, \quad z=D_{x,y}$$

(Yu et al., 9 Sep 2025). This suggests a practical distinction between 2.5D grasp synthesis from image-plane predictions and explicit Cartesian grasp recovery at inference.

Dexterous RGS uses a different output space. G-DexGrasp defines the input as a natural-language referring expression $D \in \mathbb{R}^{H \times W}$0, a 3D object point cloud $D \in \mathbb{R}^{H \times W}$1, and an optional part segmentation $D \in \mathbb{R}^{H \times W}$2, and produces a full-DOF dexterous-hand grasp $D \in \mathbb{R}^{H \times W}$3, where $D \in \mathbb{R}^{H \times W}$4, $D \in \mathbb{R}^{H \times W}$5, and $D \in \mathbb{R}^{H \times W}$6 with $D \in \mathbb{R}^{H \times W}$7 for MANO (Jian et al., 25 Mar 2025). The synthesized grasp is required to be collision-free and stable, and to place fingertip contacts on the sub-region specified by language (Jian et al., 25 Mar 2025). Text2Grasp similarly conditions on an object point cloud and a part-level text prompt, but outputs a 66-dimensional hand grasp vector including MANO pose, shape, centroid offset, and finger mask (Chang et al., 2024).

Across these formulations, RGS therefore encompasses at least three technical subproblems: language grounding, grasp parameter prediction, and geometric validity. The exact balance depends on whether the target is a planar parallel-jaw grasp in clutter or a dexterous hand configuration aligned to a semantically specified part.

2. Benchmark construction and data regimes

The principal benchmark for planar RGS in the cited works is OCID-VLG, derived from cluttered indoor tabletop scenes and designed to couple referring expressions, segmentation masks, and grasp supervision (Tziafas et al., 2023, Yu et al., 2024, Yu et al., 9 Sep 2025). CROG describes its construction from 1,763 indoor tabletop RGB-D scenes with high object clutter, drawn from the OCID dataset, with 31 semantic categories and 58 distinct instances (Tziafas et al., 2023). It reuses OCID-Grasp 4-DoF annotations, with approximately 75 k hand-labeled rectangular grasps per scene, and generates referring expressions through scene-graph annotations over colors, instance attributes, pairwise spatial relations, and absolute locations (Tziafas et al., 2023). The resulting benchmark contains 89,639 tuples of image, expression, mask, and grasp, split by scene into 1,176 / 176 / 411 train/val/test scenes (Tziafas et al., 2023).

The parameter-efficient tuning framework also uses OCID-VLG and summarizes it as 1,763 cluttered tabletop scenes, 58 object categories, and approximately 89.6 K referring expressions paired with segmentation masks and ground-truth grasp rectangles (Yu et al., 2024). OGRG describes the same dataset as approximately 90 K expressions with 75 K annotated grasp rectangles (Yu et al., 9 Sep 2025). The discrepancy between “31 semantic categories” and “58 object categories” reflects the fact that one source distinguishes semantic categories from distinct instances, whereas another compresses the description into a single object-category count; no reconciliation beyond the provided text is stated.

The linguistic structure of OCID-VLG is central to RGS difficulty. CROG reports five expression families: name only, attribute only, relation only, location only, and mixed (Tziafas et al., 2023). Expressions are explicitly designed so that categories may be ambiguous, forcing reliance on attributes or relations rather than object names alone (Tziafas et al., 2023). This is important because several later methods claim improved handling of complex spatial reasoning or duplicated objects, and those claims are meaningful only relative to such ambiguity-rich supervision (Yu et al., 2024, Yu et al., 9 Sep 2025).

Outside planar tabletop RGS, dexterous methods use different datasets. Text2Grasp trains on OakInk and evaluates additionally on AffordPose, with Oak-Shape split into 1308 train and 183 test models, and uses task-to-part prompt construction as a control signal (Chang et al., 2024). G-DexGrasp tests on 299 novel objects from OakInk, PartNet-Mobility and 3D-Net under 5 grasp types, emphasizing unseen object categories and semantic part alignment rather than cluttered RGB-D tabletop grasp rectangles (Jian et al., 25 Mar 2025). This suggests that “RGS” has already diversified into benchmark families with distinct output spaces and generalization criteria.

3. Core architectural patterns

A recurrent design pattern in planar RGS is end-to-end multimodal dense prediction: language and vision are fused before or during decoding, and the fused representation supports both grounding and grasp-map estimation (Tziafas et al., 2023, Yu et al., 2024, Yu et al., 9 Sep 2025). CROG extends the CRIS referring-segmentation model by combining a CLIP ResNet-50 visual trunk, a CLIP text encoder, an FPN-style multimodal neck, a Transformer cross-attention decoder, a contrastive segmentation head, and three parallel grasp heads for quality, angle, and width (Tziafas et al., 2023). The grasp heads operate on token-aligned visual features $D \in \mathbb{R}^{H \times W}$8 upsampled to $D \in \mathbb{R}^{H \times W}$9 resolution (Tziafas et al., 2023).

The parameter-efficient framework in "A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping" introduces a frozen CLIP backbone with inserted bi-directional Vision–Language adapters and a lightweight multimodal decoder (Yu et al., 2024). Early-stage intermediate features $T \in \mathbb{R}^L$0 and $T \in \mathbb{R}^L$1 are fused by a bi-directional VL-adapter at each CLIP block $T \in \mathbb{R}^L$2 (Yu et al., 2024). Each adapter stage projects visual tokens by a $T \in \mathbb{R}^L$3 convolution and language tokens by a linear layer, adds previous-stage adapter outputs, optionally includes projected depth features for RGS, concatenates the streams, applies layer normalization and one Transformer layer, then splits and reinjects the fused signals back into the CLIP branches (Yu et al., 2024). The total extra parameters per stage are two small convolution layers, two small linear layers, and one tiny Transformer block, amounting to approximately $T \in \mathbb{R}^L$4–$T \in \mathbb{R}^L$5 of CLIP’s weights in total (Yu et al., 2024).

The decoder in that framework collects multi-scale visual features, a global sentence embedding $T \in \mathbb{R}^L$6, the final word-token sequence $T \in \mathbb{R}^L$7, and depth fusion features $T \in \mathbb{R}^L$8 (Yu et al., 2024). It then performs pixel–sentence fusion through element-wise multiplication and addition, followed by pixel–words fusion using Transformer-decoder layers with self-attention over fused pixel–sentence features and cross-attention from pixel features to word tokens (Yu et al., 2024). Three parallel heads predict $T \in \mathbb{R}^L$9, $M \in \{0,1\}^{H \times W}$0, and $M \in \{0,1\}^{H \times W}$1 over the $M \in \{0,1\}^{H \times W}$2 grid (Yu et al., 2024).

OGRG retains the same broad objective but changes the backbone and fusion mechanism. It uses a 4-stage Swin Transformer for vision, BERT for language, a small ResNet-18 depth branch, and four bidirectional aligners, one per stage (Yu et al., 9 Sep 2025). At each stage, language-to-vision and vision-to-language attention are computed, reshaped by a $M \in \{0,1\}^{H \times W}$3 convolution plus ReLU, gated, and added back to the original streams (Yu et al., 9 Sep 2025). The fused visual features from all stages are then passed to a light FCN decoder producing $M \in \{0,1\}^{H \times W}$4, $M \in \{0,1\}^{H \times W}$5, $M \in \{0,1\}^{H \times W}$6, and $M \in \{0,1\}^{H \times W}$7 (Yu et al., 9 Sep 2025). Runtime is reported as 17.59 FPS on a single NVIDIA RTX 2080 Ti (Yu et al., 9 Sep 2025).

HiFi-CS represents a different decomposition. It treats RGS as a two-stage pipeline consisting of visual grounding followed by grasp pose estimation from the grounded object point cloud (Bhat et al., 2024). A frozen CLIP ViT-B/16 and CLIP text encoder provide representations, and a lightweight decoder with hierarchical FiLM fusion predicts a mask $M \in \{0,1\}^{H \times W}$8 (Bhat et al., 2024). The projected mask and depth are converted into an object point cloud, which is then passed to AnyGrasp SDK to obtain candidate 6-DoF grasps (Bhat et al., 2024). This is not end-to-end grasp-map prediction; rather, it couples language grounding with a downstream grasp planner.

Dexterous RGS methods diverge even further from planar RGS architectures. Text2Grasp uses a two-stage pipeline in which TextGraspDiff, a text-conditioned diffusion network, produces a coarse grasp vector, and a hand-object contact optimizer refines it (Chang et al., 2024). G-DexGrasp instead uses part-aware prior retrieval, a conditional VAE for coarse extrinsic grasp generation, and a prior-assisted refinement optimization over hand parameters (Jian et al., 25 Mar 2025). Both methods depend on explicit part-level semantics, either from prompts of the form “Grasp the [Object_Part] of the [Object_Category]” or from a retrieved affordance-part intent inferred from the referring instruction (Chang et al., 2024, Jian et al., 25 Mar 2025).

4. Multimodal fusion, depth, and supervision

The main technical differentiators in recent planar RGS systems are the form of multimodal alignment and the use of depth. In the parameter-efficient CLIP framework, the bi-directional VL-adapter is explicitly designed to align multimodal inputs for pixel-level language understanding, while the depth fusion branch incorporates geometric cues to facilitate robot grasping predictions (Yu et al., 2024). Depth is optionally projected into the adapter as a third stream and is also preserved as a depth fusion feature for the final decoder (Yu et al., 2024). An ablation reports that removing the depth branch drops grasp accuracy by approximately 8 percentage points, which is presented as confirmation of the geometric cue’s benefit (Yu et al., 2024).

OGRG similarly emphasizes bi-directional fusion and depth, but with a simpler depth usage. Depth is encoded by ResNet-18 into $M \in \{0,1\}^{H \times W}$9 and element-wise added to the first-stage vision feature, with no additional geometric projection beyond sampling $Q \in \mathbb{R}^{H \times W}$0 at the chosen pixel (Yu et al., 9 Sep 2025). The paper states that this simple fusion proved sufficient to boost both segmentation and angle regression (Yu et al., 9 Sep 2025). Its ablation reports that depth fusion improves mIoU by $Q \in \mathbb{R}^{H \times W}$1 and J@1 by $Q \in \mathbb{R}^{H \times W}$2 relative to OGRG-nodepth (Yu et al., 9 Sep 2025).

CROG does not use an explicit depth fusion branch in its multimodal backbone; instead, it learns grounding and grasp synthesis jointly from image-text pairs and rectangular grasp supervision (Tziafas et al., 2023). Its multimodal coupling is achieved through sentence-conditioned fusion in an FPN-style neck and token-level cross-attention with text features (Tziafas et al., 2023). This places more burden on learned cross-modal alignment than on geometry-specific feature design.

HiFi-CS uses hierarchical FiLM fusion rather than cross-attention between dense language and vision features. Feature maps are modulated by per-channel scaling and bias derived from the text embedding,

$Q \in \mathbb{R}^{H \times W}$3

at multiple levels of the decoder (Bhat et al., 2024). Because the CLIP encoders remain frozen and no auxiliary alignment losses are used, semantic alignment to CLIP embedding space is described as implicit (Bhat et al., 2024). The method is primarily a grounding architecture, with the final grasp pose delegated to AnyGrasp (Bhat et al., 2024).

Supervision also differs across methods. Planar RGS in CROG, the PET framework, and OGRG is trained with full pixel-wise mask supervision and ground-truth grasp rectangles (Tziafas et al., 2023, Yu et al., 2024, Yu et al., 9 Sep 2025). OGRG further investigates a weakly supervised Referring Grasp Affordance setting using only single-pixel grasp annotations, but that is adjacent to rather than identical with RGS (Yu et al., 9 Sep 2025). Dexterous methods such as Text2Grasp and G-DexGrasp rely on part labels, part segmentation, contact maps, or language-derived affordance-part inference rather than pixel-wise clutter masks (Chang et al., 2024, Jian et al., 25 Mar 2025).

5. Optimization objectives and inference procedures

Planar RGS methods generally optimize a grounding term together with grasp-map regression terms. In the parameter-efficient CLIP framework, the text-to-pixel contrastive loss for mask and quality is

$Q \in \mathbb{R}^{H \times W}$4

where $Q \in \mathbb{R}^{H \times W}$5 is the ground-truth label and $Q \in \mathbb{R}^{H \times W}$6 is the logistic sigmoid (Yu et al., 2024). Angle and width are trained with Smooth-L1 losses,

$Q \in \mathbb{R}^{H \times W}$7

and the total RGS loss is the sum of $Q \in \mathbb{R}^{H \times W}$8, $Q \in \mathbb{R}^{H \times W}$9, and $\Theta \in \mathbb{R}^{H \times W}$0 (Yu et al., 2024).

CROG uses an analogous multitask structure. Its grounding term is a contrastive alignment loss computed as binary cross-entropy over pixel logits $\Theta \in \mathbb{R}^{H \times W}$1, and its grasp loss is Smooth-L1 over the predicted quality, angle, and width maps (Tziafas et al., 2023). The total objective is

$\Theta \in \mathbb{R}^{H \times W}$2

(Tziafas et al., 2023).

OGRG decomposes the loss into grounding and grasp terms,

$\Theta \in \mathbb{R}^{H \times W}$3

with binary cross-entropy for the grounding mask and

$\Theta \in \mathbb{R}^{H \times W}$4

where $\Theta \in \mathbb{R}^{H \times W}$5, $\Theta \in \mathbb{R}^{H \times W}$6, and $\Theta \in \mathbb{R}^{H \times W}$7 (Yu et al., 9 Sep 2025). In practice, $\Theta \in \mathbb{R}^{H \times W}$8 (Yu et al., 9 Sep 2025).

Training schedules are also reported. The parameter-efficient framework freezes all original CLIP weights, inserts VL-adapters at each stage in both vision and text streams, and trains only the adapter parameters and task decoder, corresponding to approximately 1–2 M trainable parameters and a tunable fraction of $\Theta \in \mathbb{R}^{H \times W}$9–$P \in \mathbb{R}^{H \times W}$0 of CLIP’s approximately 350 M weights (Yu et al., 2024). It uses AdamW with initial learning rate $P \in \mathbb{R}^{H \times W}$1, polynomial decay over 40 epochs, batch size 13, and preprocessing to $P \in \mathbb{R}^{H \times W}$2 images with maximum sentence length 20 tokens (Yu et al., 2024). OGRG uses AdamW with initial learning rate $P \in \mathbb{R}^{H \times W}$3, polynomial decay, batch size 4 per GPU across 8 GPUs, 26 epochs, and image size $P \in \mathbb{R}^{H \times W}$4 (Yu et al., 9 Sep 2025). CROG fine-tunes all CLIP layers rather than freezing them, using Adam, initial learning rate $P \in \mathbb{R}^{H \times W}$5 decayed by $P \in \mathbb{R}^{H \times W}$6 over 35 epochs, and batch size 16 per GPU on 4 GPUs (Tziafas et al., 2023).

Inference metrics in planar RGS are centered on grounding and grasp correctness. The principal grounding metric is mIoU between the predicted mask and ground truth (Yu et al., 2024, Yu et al., 9 Sep 2025). The principal grasp metric is the Jacquard Index J@N, defined as the percentage of top-$P \in \mathbb{R}^{H \times W}$7 grasp proposals whose rectangle satisfies both $P \in \mathbb{R}^{H \times W}$8 and $P \in \mathbb{R}^{H \times W}$9 relative to ground truth (Yu et al., 2024); OGRG reports J@1 and J@Any under the same grasp-overlap and angle-error conditions (Yu et al., 9 Sep 2025). CROG also reports J@1 and J@Any (Tziafas et al., 2023).

Dexterous RGS optimization is more elaborate. Text2Grasp follows DDPM for the diffusion stage and then minimizes a composite energy containing part-weighted contact loss, collision, joint-angle, and self-collision terms with Adamax for 200 steps (Chang et al., 2024). G-DexGrasp retrieves a generalizable grasp prior, generates coarse extrinsics with a conditional VAE, and then refines all parameters by minimizing a composite energy containing contact, contact-map, force-closure, penetration, reverse-penetration, joint-limit, and self-penetration terms (Jian et al., 25 Mar 2025). It runs two Adam-based optimizations: first over $G=(x,y,\theta,l)$0 for 700 steps at learning rate $G=(x,y,\theta,l)$1, then over $G=(x,y,\theta,l)$2 for another 700 steps at learning rate $G=(x,y,\theta,l)$3 (Jian et al., 25 Mar 2025).

6. Empirical performance, capabilities, and limitations

The empirical record in OCID-VLG shows a clear progression from early end-to-end CLIP integration to stronger grounding-grasping fusion and parameter-efficient multimodal tuning. CROG reports on the OCID-VLG test split an RIS-IoU of 81.1, J@1 of 77.2, and J@Any of 87.7, outperforming multi-stage baselines such as SAM+CLIP, GLIP+SAM, Det-Seg+CLIP, and SSG+CLIP by large margins (Tziafas et al., 2023). The accompanying ablation indicates that removing CLIP initialization reduces IoU by 7.2 and J@1 by 6.2, removing contrastive loss reduces J@1 by 3.8, removing grasp loss reduces IoU by 1.8, and removing the decoder reduces IoU by 2.9 and J@1 by 4.9 (Tziafas et al., 2023).

The parameter-efficient CLIP-based framework reports, on OCID-VLG with a ResNet-50 backbone, that CROG full fine-tuning reaches 81.10 % mIoU and 77.20 % J@1, ETRG-A without depth fusion reaches 79.06 % mIoU and 81.16 % J@1, and ETRG-A with depth fusion reaches 79.82 % mIoU and 89.06 % J@1 (Yu et al., 2024). ResNet-101 yields similar gains of approximately +12 percentage points J@1 over CROG (Yu et al., 2024). The result is notable because the model freezes CLIP and tunes only a small adapter and decoder budget (Yu et al., 2024). The paper’s summary states that by freezing CLIP and adding approximately 2 M adapter parameters plus a lightweight decoder, the RGS model outperforms prior full-fine-tuned methods by more than 10 percentage points in Jacquard grasp metrics (Yu et al., 2024).

OGRG advances the same RGS subtask further. On OCID-VLG it reports the following RGS results: CROG with 81.10 mIoU, 77.20 J@1, and 87.70 J@Any; ETRG with 80.11 mIoU, 89.38 J@1, and 93.49 J@Any; LAVT with 92.52 mIoU, 87.55 J@1, and 91.77 J@Any; OGRG-nodepth with 94.87 mIoU, 88.49 J@1, and 93.70 J@Any; and full OGRG with 95.60 mIoU, 90.81 J@1, and 94.70 J@Any (Yu et al., 9 Sep 2025). It further reports that adding the Bi-Aligner to LAVT yields +2.35% mIoU and +1.0 J@1, and that depth fusion yields +0.73% mIoU and +2.32 J@1 over OGRG-nodepth (Yu et al., 9 Sep 2025). The paper emphasizes that this is achieved at 17.59 FPS on a single NVIDIA RTX 2080 Ti (Yu et al., 9 Sep 2025).

HiFi-CS reports a different profile because its strongest contributions are in grounding rather than end-to-end grasp-map prediction. On OCID-VLG closed-vocabulary grounding, HiFi-CS achieves IoU 88.3 with notably strong precision at high IoU thresholds, while remaining a lightweight 6 M-parameter decoder on top of frozen CLIP encoders (Bhat et al., 2024). In real-world RGS experiments over 15 tabletop scenes, 5 object classes, and 3 difficulty levels per class, HiFi-CS attains overall segmentation accuracy 85.0% and grasp accuracy 42.7%, while the hybrid GrSAM+HiFi-CS achieves 90.3% segmentation accuracy and 60.3% grasp accuracy (Bhat et al., 2024). Because the final grasp pose comes from AnyGrasp, these results reflect the sensitivity of multi-stage RGS pipelines to grounding quality (Bhat et al., 2024).

Robot results for CROG indicate the remaining execution gap. In Gazebo, CROG attains grounding accuracy 76% and success rate 62% in isolated scenes, dropping to 60% and 42% in cluttered scenes (Tziafas et al., 2023). On the real robot, isolated and cluttered success rates are 23.9% and 20.0%, respectively (Tziafas et al., 2023). The reported failure modes include grounding errors under heavy occlusion or novel colors and textures, and execution failures from collisions despite correct grasp prediction (Tziafas et al., 2023). A plausible implication is that benchmark-level improvements in J@N do not automatically translate into proportionate real-robot gains once perception, collision checking, and control are coupled.

Complex language scenarios receive uneven treatment across methods. The parameter-efficient CLIP framework states that RGS assumes a unique referring expression and that scenes with multiple identical objects require an unambiguous description such as “the upper-right mustard bottle,” though in practice the depth-fusion branch and pixel–words fusion improve separation in tightly packed clutter (Yu et al., 2024). OGRG explicitly targets open-form language expressions and spatial reasoning, including scenes containing duplicated object instances (Yu et al., 9 Sep 2025). HiFi-CS evaluates attribute-rich text queries and reports that attribute-wise IoU degrades minimally as query complexity grows, whereas baselines drop sharply (Bhat et al., 2024). These claims together suggest that recent progress has shifted the bottleneck from simple noun grounding toward compositional attribute and spatial disambiguation.

Dexterous RGS methods report strong semantic-part alignment under different metrics. Text2Grasp achieves part accuracy 87.8% with template prompts and 82.3% with personalized prompts, while maintaining comparable grasp quality to baselines (Chang et al., 2024). G-DexGrasp, evaluated on unseen categories and language-based task instructions, reduces Solid.Intsec.Vol to 2.94 cm³, Penet.Depth to 0.29 cm, Sim.Dis to 4.27 cm, boosts Part Acc. to 71.6%, and reaches 96.3% part accuracy when given perfect segmentations (Jian et al., 25 Mar 2025). Both methods indicate that part-conditioned control can be effective, but G-DexGrasp also identifies reliance on open-vocabulary part segmentation for unseen objects as a current limitation (Jian et al., 25 Mar 2025).

7. Relationship to adjacent problems and emerging directions

RGS sits at the intersection of referring expression grounding, robot grasp detection, affordance learning, and language-conditioned manipulation. The cited works make this especially clear by alternating between end-to-end and modular decompositions. CROG and the PET framework couple grounding and grasp-map prediction within a single network (Tziafas et al., 2023, Yu et al., 2024). HiFi-CS separates grounding from downstream 6-DoF grasp estimation (Bhat et al., 2024). OGRG studies both RGS under full supervision and Referring Grasp Affordance under weak supervision (Yu et al., 9 Sep 2025). The dexterous methods shift the problem from object selection to contact-region compliance and hand configuration generation (Chang et al., 2024, Jian et al., 25 Mar 2025).

One recurring misconception is that RGS is simply visual grounding followed by standard grasp planning. The literature only partly supports that view. HiFi-CS is explicitly two-stage, but CROG, ETRG-style models, and OGRG show that direct joint prediction of grounding and grasp maps can substantially improve Jacquard-style grasp metrics in clutter (Tziafas et al., 2023, Yu et al., 2024, Yu et al., 9 Sep 2025). Another misconception is that language in grasping merely names object categories. OCID-VLG was designed to require attributes, relations, and locations when categories are ambiguous (Tziafas et al., 2023), and later models report specific gains from architectures intended to preserve word-level and spatial information (Yu et al., 2024, Yu et al., 9 Sep 2025, Bhat et al., 2024).

Several trajectories are visible. One is parameter efficiency: freezing large vision-language backbones and training only adapters or lightweight decoders can match or surpass full fine-tuning for RGS (Yu et al., 2024, Bhat et al., 2024). A second is geometric augmentation through depth, which consistently improves planar grasp performance in the cited ablations (Yu et al., 2024, Yu et al., 9 Sep 2025). A third is the move from unique-object referring expressions toward duplicate-heavy scenes and spatial language, which OGRG and the RGA companion setting make explicit (Yu et al., 9 Sep 2025). A fourth is the expansion from planar parallel-jaw grasps to dexterous hand synthesis grounded in semantic parts or affordance types (Chang et al., 2024, Jian et al., 25 Mar 2025).

The literature also identifies unresolved constraints. Open-vocabulary part segmentation remains a limiting factor for dexterous generalization to unseen objects (Jian et al., 25 Mar 2025). Multi-stage pipelines can accumulate grounding errors into grasp failures (Bhat et al., 2024). Real-robot execution remains more fragile than benchmark metrics imply, especially under occlusion, novel appearance, and collision-prone clutter (Tziafas et al., 2023). Taken together, these results suggest that RGS is evolving from a narrowly defined RGB-D tabletop task into a broader family of multimodal grasping problems, but that robust deployment still depends on closing the gap between semantic grounding accuracy and physically reliable execution.