GenSeg-R1-G: Fine-Grained Referring Segmentation

• The paper introduces GenSeg-R1-G as a variant that decouples reasoning from segmentation by using a vision-language model to emit structured spatial prompts for SAM 2.
• It leverages a reinforcement learning framework with SAM-2-in-the-loop rewards to directly optimize mask IoU, negative-point validity, and accurate abstention on no-target prompts.
• Trained on GRefCOCO, GenSeg-R1-G demonstrates superior target mIoU and robust no-target detection, reducing false negatives compared to baseline models.

GenSeg-R1-G is a specialized variant of the GenSeg-R1 framework for fine-grained referring segmentation, introduced in “GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation” (Hegde et al., 10 Feb 2026). It operates in a decoupled reason-then-segment pipeline in which a vision-LLM reasons over an image and a natural-language query, emits structured spatial prompts, and delegates mask generation to a frozen promptable segmenter, SAM 2. Its defining distinction from the base GenSeg-R1 models is that it is trained on GRefCOCO with a SAM-2-in-the-loop reward that directly optimizes downstream mask quality and explicitly handles negative or no-target prompts through abstention. The name should not be conflated with the earlier medical-imaging method GenSeg from “Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimes,” where the exact string “GenSeg-R1-G” does not appear (Zhang et al., 2024).

1. Definition and task setting

GenSeg-R1-G addresses fine-grained referring segmentation in a setting where the target may be present, absent, single-instance, or multi-instance. The paper formulates the system as a decoupled reason-then-segment pipeline: a finetuned VLM receives an image $I$ and a natural-language query $q$, reasons about the scene, and outputs structured spatial prompts rather than a mask (Hegde et al., 10 Feb 2026). A frozen SAM 2 model then converts those prompts into a segmentation mask.

For target-present cases, the output prompt set is defined as

$$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$

where $b_i = [x_1, y_1, x_2, y_2]$ is a bounding box and

$p_i^{(1)}, p_i^{(2)} \in [0,1000]^2.$

The coordinate system is normalized to $[0,1000]$, matching Qwen3-VL’s native coordinate space (Hegde et al., 10 Feb 2026). The use of two interior points is explicitly motivated as providing SAM 2 richer localization cues than a single point, especially for elongated, articulated, or multi-part objects.

For negative prompts, the model emits $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2.$1 This signals that no prompt is sent to SAM 2 and no mask should be produced (Hegde et al., 10 Feb 2026). This no-target pathway is central to the model’s design and differentiates it from compared systems that hallucinate localizations on negative queries.

The model therefore does not predict masks directly. A precise statement in the paper is that GenSeg-R1-G is a grounding model, not a direct segmentation model: it predicts boxes, points, or a no-target flag, while SAM 2 performs the actual mask generation (Hegde et al., 10 Feb 2026).

2. Architectural composition

GenSeg-R1-G uses the same overall framework as GenSeg-R1, but with a GRefCOCO-specialized training recipe (Hegde et al., 10 Feb 2026). The reasoning backbone is a finetuned Qwen3-VL Instruct model. The paper’s hardware table groups “GenSeg-R1-4B / -G” together on $2\times$ H200, which strongly indicates that GenSeg-R1-G is the 4B-scale Qwen3-VL variant, although the paper does not separately state an 8B version (Hegde et al., 10 Feb 2026).

The frozen segmenter is SAM 2 with checkpoint facebook/sam2-hiera-large (Hegde et al., 10 Feb 2026). At inference and reward-computation time, SAM 2 is prompted with the predicted box $b_i$, positive point $p_i^{(1)}$, and optionally both points; the highest-confidence mask is selected. For multi-object queries, instance masks are combined by pixel-wise union (Hegde et al., 10 Feb 2026).

The VLM response follows a tagged text format. It may contain optional reasoning inside > ... and a structured answer in <answer> ... </answer> (Hegde et al., 10 Feb 2026). For positive cases, the answer contains one or more instances represented by a box and two keypoints. For no-target cases, it emits {"no_target": true}.

This architecture places GenSeg-R1-G in a broader family of systems that separate textual or spatial grounding from promptable mask decoding. In Seg-R1, for example, a large multimodal model generates boxes and points that are then consumed by frozen SAM2, but the system is trained on foreground segmentation with a different RL formulation and without the GRefCOCO-specific no-target regime (You et al., 27 Jun 2025). A plausible implication is that GenSeg-R1-G’s novelty lies less in the promptable-segmenter interface itself than in the way the RL signal is specialized to final mask quality and abstention behavior.

3. Reinforcement-learning formulation

The paper distinguishes two reward configurations within GenSeg-R1 (Hegde et al., 10 Feb 2026). GenSeg-R1-4B and GenSeg-R1-8B use a distance-based reward based on format correctness, bounding-box accuracy, point accuracy, and a non-repetition penalty. GenSeg-R1-G extends this with a SAM-2-in-the-loop outcome-based reward.

Training uses Group Relative Policy Optimization (GRPO). For each training example $(I,q)$, the policy can be written as

$q$0

where $q$1 is the text response containing reasoning and structured prompts (Hegde et al., 10 Feb 2026). The system samples $q$2 candidate responses, scores them with a composite reward, normalizes rewards within each group, and updates the policy toward higher-reward samples without training a separate value function (Hegde et al., 10 Feb 2026).

The reward additions specific to GenSeg-R1-G are explicitly described as three extra components (Hegde et al., 10 Feb 2026):

1. SAM 2 IoU reward $q$3, weight $q$4
2. Negative-point validity reward $q$5, weight $q$6
3. No-target reward $q$7, reward $q$8

The core mask-quality term is defined through the mask $q$9 produced by frozen SAM 2: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$0 with

$$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$1

The paper states that this is “the primary signal that directly ties the VLM output to segmentation quality” (Hegde et al., 10 Feb 2026).

The negative-point validity reward is described geometrically rather than formulaically. It validates that predicted negative points lie inside the bounding box but outside the ground-truth mask and at sufficient distance from the mask boundary; the paper identifies it as the highest-weighted added component and says it drives boundary refinement (Hegde et al., 10 Feb 2026).

The no-target reward applies when the ground-truth mask is empty. In that case, the model is rewarded for predicting no_target and penalized for hallucinating boxes (Hegde et al., 10 Feb 2026). The exact penalty magnitude is not given in the text provided.

The total reward is described as an extension of the base distance reward: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$2 This is a faithful reconstruction from the weighting scheme described in the paper, while the internal decomposition of $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$3 is only given qualitatively (Hegde et al., 10 Feb 2026).

The motivation for this reward design is explicit. The authors argue that optimizing only coordinate proximity creates a train-test mismatch because a box can be numerically close to the annotation and still be a poor prompt for SAM 2. GenSeg-R1-G is therefore introduced to optimize the actual downstream mask and to handle negative prompts properly, since prior methods hallucinate masks on all negative queries in GRefCOCO evaluation (Hegde et al., 10 Feb 2026).

4. Training data and implementation

GenSeg-R1-G is trained on GRefCOCO with negative points, totaling 15,285 samples (Hegde et al., 10 Feb 2026). The dataset includes positive referring expressions, negative or no-target expressions, two positive interior points, two negative background points, polygon masks for mask-quality computation, and empty masks for negative prompts (Hegde et al., 10 Feb 2026). The paper states that these richer annotations allow the model to learn precise prompt generation, reliable no-target rejection, and boundary-sensitive prompting behavior.

The training configuration is given explicitly in the paper (Hegde et al., 10 Feb 2026):

• learning rate: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$4
• weight decay: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$5
• max gradient norm: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$6
• KL coefficient: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$7
• global batch size: 8
• micro-batch: 2 / device
• rollout batch size: 8
• samples per prompt: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$8
• max prompt length: 2048 tokens
• max response length: 1500 tokens
• hardware for GenSeg-R1-4B / -G: $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$9 H200 (141 GB)

Rollout generation uses vLLM, and distributed training uses FSDP / ZeRO-3 (Hegde et al., 10 Feb 2026). The coordinate range for all annotations is rescaled to $b_i = [x_1, y_1, x_2, y_2]$0 (Hegde et al., 10 Feb 2026).

This training recipe differs sharply from the original GenSeg in medical imaging, which is a conditional-GAN-based framework for ultra low-data medical segmentation and not a referring-segmentation RL system (Zhang et al., 2024). In that earlier work, “GenSeg” denotes a mask-to-image generative augmentation method with multilevel optimization over generator weights $b_i = [x_1, y_1, x_2, y_2]$1, architecture $b_i = [x_1, y_1, x_2, y_2]$2, discriminator weights $b_i = [x_1, y_1, x_2, y_2]$3, and a downstream segmenter $b_i = [x_1, y_1, x_2, y_2]$4, whereas GenSeg-R1-G denotes a vision-language grounding model optimized through GRPO and SAM-2-in-the-loop reward (Zhang et al., 2024). The terminological overlap has caused ambiguity, but the systems address different tasks, modalities, and optimization paradigms.

5. Empirical performance

The key evaluation for GenSeg-R1-G is GRefCOCO validation, which contains 1,000 samples: 642 target-present and 358 no-target (Hegde et al., 10 Feb 2026). Models are evaluated by having the VLM predict prompts, frozen SAM 2 generate masks, and metrics be computed against ground-truth masks (Hegde et al., 10 Feb 2026). The reported metrics are Target mIoU, Target cIoU, and No-Target Accuracy.

On GRefCOCO validation, the paper reports the following results (Hegde et al., 10 Feb 2026):

Model	Target mIoU	No-Target Accuracy
GenSeg-R1-G	76.69%	82.40%
GenSeg-R1-4B	75.58%	79.05%
Seg-R1-7B	69.51%	0.00%
Seg-Zero-7B	66.11%	0.00%

The corresponding Target cIoU values are 76.83% for GenSeg-R1-G, 75.54% for GenSeg-R1-4B, 69.02% for Seg-R1-7B, and 65.90% for Seg-Zero-7B (Hegde et al., 10 Feb 2026). GenSeg-R1-G therefore improves over Seg-R1-7B by +7.18 points target mIoU and over Seg-Zero-7B by +10.58 points target mIoU (Hegde et al., 10 Feb 2026).

Its no-target behavior is particularly distinctive. The paper reports that GenSeg-R1-G correctly abstains on

$b_i = [x_1, y_1, x_2, y_2]$5

no-target prompts (Hegde et al., 10 Feb 2026). At the same time, its false-negative rate on target-present examples is

$b_i = [x_1, y_1, x_2, y_2]$6

The corresponding figure for GenSeg-R1-4B is $b_i = [x_1, y_1, x_2, y_2]$7, while Seg-R1 and Seg-Zero register $b_i = [x_1, y_1, x_2, y_2]$8 because they never predict no-target (Hegde et al., 10 Feb 2026). This is important because it shows that GenSeg-R1-G’s abstention is not obtained by overusing rejection.

The paper also evaluates GenSeg-R1-G on ReasonSeg test. There it achieves 65.93% mIoU and 57.83% cIoU, compared with 61.41 / 55.63 for Seg-Zero-7B and 57.75 / 48.93 for Seg-R1-7B (Hegde et al., 10 Feb 2026). This suggests that a model specialized for GRefCOCO and no-target robustness still transfers beyond its training set.

The base GenSeg-R1 family also reports strong RefCOCOg and ReasonSeg performance under distance-based reward. On RefCOCOg validation, the best model GenSeg-R1-8B achieves 0.7127 cIoU and 0.7382 mIoU, outperforming the corresponding Qwen3-VL Instruct baselines by +15.3 and +21.9 points and surpassing Seg-Zero-7B by +3.3 cIoU under identical evaluation (Hegde et al., 10 Feb 2026). These numbers contextualize GenSeg-R1-G as a specialized branch of a broader RL-grounded segmentation framework rather than an isolated model.

6. Relation to adjacent work and naming ambiguity

The label “GenSeg-R1-G” is specific to the 2026 GenSeg-R1 paper (Hegde et al., 10 Feb 2026). It should not be confused with the medical-image segmentation method GenSeg from 2024, where the exact string “GenSeg-R1-G” is absent (Zhang et al., 2024). In that earlier paper, the named variants are GenSeg-UNet, GenSeg-DeepLab, and GenSeg-SwinUnet, along with ablations such as Separate and swaps of the underlying conditional GAN, including Pix2Pix, SPADE, and ASAPNet (Zhang et al., 2024). The closest correspondence there is the GenSeg mask-to-image generator with weights $b_i = [x_1, y_1, x_2, y_2]$9 and architecture $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2.$0, but that is not a model named GenSeg-R1-G (Zhang et al., 2024).

GenSeg-R1-G also occupies a distinct position relative to several nearby post-2025 lines of work. Seg-R1 treats segmentation as autoregressive prompt generation for frozen SAM2, optimized with GRPO from image-mask pairs, and reports zero-shot transfer to referring and reasoning segmentation (You et al., 27 Jun 2025). The GenSeg-R1 paper explicitly compares against Seg-R1-7B on GRefCOCO and shows that Seg-R1-7B lacks no-target detection capability in that benchmark, scoring 0.00% no-target accuracy (Hegde et al., 10 Feb 2026). A plausible implication is that GenSeg-R1-G can be understood as a task-specialized response to precisely those failure modes: train-test mismatch between grounding metrics and mask quality, and hallucination on negative prompts.

GREx introduces a generalized referring-expression setting with single-target, multi-target, and no-target expressions through gRefCOCO, GRES, GREC, and GREG (Ding et al., 8 Jan 2026). GenSeg-R1-G’s training on GRefCOCO with negative points places it in direct contact with that generalized regime, especially the no-target case. However, the two works differ in method: GREx proposes the ReLA architecture for generalized segmentation and comprehension, whereas GenSeg-R1-G uses RL-driven vision-language grounding plus frozen SAM 2 (Ding et al., 8 Jan 2026).

ConceptSeg-R1, despite the similar “R1” naming, is a different framework that formalizes generalized concept segmentation through a CI/CD/CR taxonomy and uses Meta-GRPO with a concept translation module into SAM 3 (Zhao et al., 19 May 2026). The ConceptSeg-R1 paper never mentions GenSeg-R1-G. This suggests that “R1” has become a naming motif for RL-augmented reasoning-grounding systems, but not a single unified model family.

7. Limitations and caveats

The GenSeg-R1 paper lists several limitations that apply to the framework, including GenSeg-R1-G (Hegde et al., 10 Feb 2026). These include domain shift, since training is on COCO-family data and out-of-domain generalization is not characterized; long-tail attributes, where rare attribute combinations may still fail; SAM 2 sensitivity, because slight coordinate errors can cause discontinuous changes in SAM 2 output; and data overlap, since GRefCOCO shares COCO images with RefCOCOg, although the paper argues this does not inflate RefCOCOg results (Hegde et al., 10 Feb 2026).

There are also specification limits in the published method description. The paper gives the high-level GRPO setup and the weighted reward components, but it does not print a full GRPO surrogate-loss equation specific to GenSeg-R1-G, nor the exact scalarization formulas for the negative-point validity reward or the no-target hallucination penalty (Hegde et al., 10 Feb 2026). Accordingly, those aspects are best understood as partially specified.

The paper’s qualitative figures show that GenSeg-R1-G and GenSeg-R1-4B often produce masks closely aligned with ground truth, while Seg-R1-7B and Seg-Zero-7B often produce poorer, misaligned masks; it also shows negative examples in which GenSeg-R1-G correctly outputs no_target while the baselines hallucinate masks (Hegde et al., 10 Feb 2026). At the same time, the authors do not claim perfection. The ReasonSeg qualitative figure includes failure cases where all models fail (Hegde et al., 10 Feb 2026).

In summary, GenSeg-R1-G is best understood as a GRefCOCO-specialized, SAM-2-in-the-loop RL variant of GenSeg-R1, designed to optimize actual downstream mask quality and explicit abstention for no-target prompts (Hegde et al., 10 Feb 2026). Its core output remains a set of boxes and interior points, or {"no_target": true}, with frozen SAM 2 responsible for the final mask. The model’s significance lies in aligning vision-language grounding with the metric that matters at inference time—the mask produced by the downstream segmenter—while introducing a practically important no-target detection capability absent from the compared baselines (Hegde et al., 10 Feb 2026).