GenSeg-R1: RL-Based Referring Segmentation

• The paper presents a ‘reason-then-segment’ pipeline, using RL to optimize spatial prompts for improved downstream mask quality.
• It leverages finetuned Qwen3-VL models with GRPO, integrating bounding boxes with dual interior keypoints for detailed localization.
• The framework achieves strong performance on RefCOCOg, GRefCOCO, and ReasonSeg benchmarks while effectively handling negative prompts.

GenSeg-R1 is a reinforcement-learning-based framework for fine-grained referring image segmentation that decomposes the task into a trainable reasoning-and-grounding stage and a frozen promptable segmentation stage. In the formulation used by the paper, a vision-LLM receives an image $I$ and a natural-language query $q$, reasons about the referred content, and emits structured spatial prompts; a frozen SAM 2 backend then converts those prompts into binary masks. The method is instantiated with finetuned Qwen3-VL models at 4B and 8B scale and optimized with Group Relative Policy Optimization (GRPO), without supervised reasoning-chain annotations. The central claim is that prompt quality should be optimized for downstream mask quality rather than for localization alone, and the reported results show strong performance on RefCOCOg, GRefCOCO, and ReasonSeg under a unified SAM 2-based evaluation protocol (Hegde et al., 10 Feb 2026).

1. Task definition and conceptual position

GenSeg-R1 addresses referring image segmentation: given an image and a natural-language expression, the system must identify the referred instance or instances and return a precise segmentation mask. The paper frames this as a fine-grained grounding problem in which text-conditioned localization and mask generation are separated. Rather than asking a vision-LLM to generate masks directly, GenSeg-R1 first predicts structured spatial prompts and then delegates mask extraction to SAM 2, which remains frozen throughout training and inference (Hegde et al., 10 Feb 2026).

This decoupling is the method’s defining design choice. The paper presents it as a “reason-then-segment” pipeline: the front end performs multimodal reasoning and grounding, while the back end performs high-quality mask generation. A plausible implication is that the division of labor allows the trainable model to focus on language-conditioned spatial decisions while preserving the segmentation quality associated with SAM-family models.

Within the broader R1 line of work, GenSeg-R1 can be understood as a specialization of reinforcement-learning-based prompt generation for segmentation. Seg-R1 had already shown that a large multimodal model can learn to emit point and box prompts for foreground segmentation tasks such as camouflaged object detection and salient object detection, again using SAM 2 as the mask generator (You et al., 27 Jun 2025). GenSeg-R1 extends that idea from foreground object discovery to natural-language-conditioned referring segmentation and adds explicit handling of negative queries, where no valid target exists (Hegde et al., 10 Feb 2026).

2. Architecture and structured prompt representation

The trainable front end in GenSeg-R1 is a finetuned Qwen3-VL model. Its output space is not dense masks but a structured prompt set

$$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$

where each referred instance $i$ is represented by a bounding box $b_i=[x_1,y_1,x_2,y_2]$ and two interior keypoints $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$ (Hegde et al., 10 Feb 2026). Qwen3-VL uses a normalized coordinate system over $[0,1000]$, and the annotations are rescaled into that coordinate space.

The system can optionally produce free-form reasoning in a $\langle think \rangle$ region and then emits structured outputs inside $\langle answer \rangle$ tags. For negative queries, where the expression matches no object in the image, the model is expected to output a no-target flag such as $\{\text{"no\_target": true}\}$ instead of hallucinating a box (Hegde et al., 10 Feb 2026).

The box-plus-two-points representation is central to the method. According to the paper, the bounding box supplies a coarse extent, while the two interior points provide finer localization cues than a single centroid, especially for elongated objects or objects with multiple parts. At inference time, SAM 2 receives the predicted box as a box prompt and $q$0 as a positive-point prompt, with the option to use both points; the highest-confidence mask is selected, and multi-object predictions are merged by pixel-wise union (Hegde et al., 10 Feb 2026).

The prompt ablation reported in the paper shows a monotonic improvement from increasingly informative prompts. On RefCOCOg validation with GenSeg-R1-4B, box-only prompting yields cIoU $q$1 and mIoU $q$2; box plus one interior point yields cIoU $q$3 and mIoU $q$4; box plus two interior points yields cIoU $q$5 and mIoU $q$6. The paper further notes that $q$7 benefits noticeably, which it interprets as evidence that the extra points improve fine-grained mask quality rather than merely coarse overlap (Hegde et al., 10 Feb 2026).

3. Reinforcement learning with GRPO

GenSeg-R1 trains the grounding model with Group Relative Policy Optimization, following a DeepSeek-R1-style recipe. For each training example $q$8, the policy samples $q$9 candidate responses, each candidate is scored with a reward, and the policy is updated toward the higher-reward outputs. The paper emphasizes that GRPO is attractive in this setting because it avoids a separate value network and instead normalizes rewards within each sampled group, making optimization practical and stable for large vision-LLMs (Hegde et al., 10 Feb 2026).

An important methodological consequence is the removal of supervised reasoning-chain annotation requirements. The paper states that the model is not trained on human-written chain-of-thought labels. Instead, reasoning traces in $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$0 regions emerge from the RL objective itself, and such traces are present in more than 95% of outputs (Hegde et al., 10 Feb 2026). This suggests that the intermediate reasoning behavior is induced instrumentally by reward design rather than by direct reasoning supervision.

Two reward regimes are used. For GenSeg-R1-4B and GenSeg-R1-8B, training uses a lightweight distance-based reward. This combines format checking for valid $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$1/$$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$2 tags and valid JSON, bounding-box correctness using IoU and L1 distance thresholds, point accuracy based on coordinate distance and whether the point lies inside the predicted box, and a non-repetition penalty (Hegde et al., 10 Feb 2026). The practical motivation is speed: these rewards do not require SAM 2 inference during rollout.

The stronger variant, GenSeg-R1-G, adds outcome-based reward terms by placing SAM 2 in the loop during training. On top of the format and distance-based checks, it uses three additional components: an SAM 2 IoU reward with weight $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$3, in which the predicted box and points are fed into SAM 2 and the resulting mask $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$4 is compared with $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$5; a negative-point validity reward with weight $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$6, requiring predicted negative points to lie inside the box but outside the ground-truth mask and sufficiently far from the boundary; and a no-target reward with value $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$7, rewarding correct abstention on empty-mask queries and penalizing hallucinated boxes (Hegde et al., 10 Feb 2026).

The paper argues that this SAM 2-in-the-loop reward reduces train-test mismatch. That claim is specific: the model is optimized not just for coordinate proximity under box metrics, but for the quality of masks that SAM 2 can actually produce from the predicted prompts (Hegde et al., 10 Feb 2026).

4. Training data, evaluation protocol, and quantitative results

The standard GenSeg-R1-4B and GenSeg-R1-8B models are trained on VisionReasoner-MultiObjects-7K, which contains 7,099 samples with bounding-box and single-point annotations. Images are resized to $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$8, and coordinates are rescaled into the $$\mathcal{P} = \bigl\{(b_i,\; p_i^{(1)},\; p_i^{(2)})\bigr\}_{i=1}^{N},$$9 system. GenSeg-R1-G is trained on GRefCOCO, totaling 15,285 samples, with two positive interior points, two negative background points, and ground-truth polygon masks for SAM 2-in-the-loop reward computation (Hegde et al., 10 Feb 2026).

Evaluation is reported on three principal benchmarks: RefCOCOg validation with 2,573 referring expressions over 1,300 images; GRefCOCO validation with 1,000 samples, comprising 642 target and 358 no-target cases; and ReasonSeg test with 779 samples requiring compositional reasoning. Across models, the evaluation protocol is consistent: the vision-LLM predicts spatial prompts, SAM 2 converts them into masks, and the resulting masks are compared with ground truth (Hegde et al., 10 Feb 2026).

The metrics include cumulative IoU,

$i$0

mean IoU, precision-at-threshold metrics $i$1, $i$2, and $i$3, detection metrics AP, [email protected], and [email protected], and, on GRefCOCO, no-target accuracy, defined as the fraction of no-target samples for which the model predicts no_target instead of hallucinating a box (Hegde et al., 10 Feb 2026).

Benchmark	Model	Reported result
RefCOCOg validation	GenSeg-R1-8B	cIoU $i$4, mIoU $i$5
RefCOCOg validation	GenSeg-R1-4B	cIoU $i$6, mIoU $i$7
GRefCOCO validation	GenSeg-R1-G	target mIoU $i$8, target cIoU $i$9, no-target accuracy $b_i=[x_1,y_1,x_2,y_2]$0
ReasonSeg test	GenSeg-R1-4B	mIoU $b_i=[x_1,y_1,x_2,y_2]$1, cIoU $b_i=[x_1,y_1,x_2,y_2]$2

On RefCOCOg validation, GenSeg-R1-8B improves over Qwen3-VL-8B-Instruct from cIoU $b_i=[x_1,y_1,x_2,y_2]$3 and mIoU $b_i=[x_1,y_1,x_2,y_2]$4 to cIoU $b_i=[x_1,y_1,x_2,y_2]$5 and mIoU $b_i=[x_1,y_1,x_2,y_2]$6, corresponding to gains of $b_i=[x_1,y_1,x_2,y_2]$7 cIoU points and $b_i=[x_1,y_1,x_2,y_2]$8 mIoU points. GenSeg-R1-4B similarly improves over Qwen3-VL-4B-Instruct by $b_i=[x_1,y_1,x_2,y_2]$9 cIoU points and $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$0 mIoU points (Hegde et al., 10 Feb 2026).

Under the same SAM 2 evaluation protocol, GenSeg-R1-8B surpasses Seg-Zero-7B by $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$1 cIoU and $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$2 mIoU, and surpasses Seg-R1-7B by $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$3 cIoU and $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$4 mIoU (Hegde et al., 10 Feb 2026). On ReasonSeg test, GenSeg-R1-4B reaches $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$5 mIoU and $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$6 cIoU, exceeding Seg-Zero-7B by $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$7 mIoU and Seg-R1-7B by $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$8 mIoU (Hegde et al., 10 Feb 2026).

5. Negative prompts, ablations, and emergent reasoning

A distinctive feature of GenSeg-R1 is explicit treatment of negative prompts. The GRefCOCO validation results are used to demonstrate this capability. GenSeg-R1-G achieves $p_i^{(1)}, p_i^{(2)} \in [0,1000]^2$9 no-target accuracy, while GenSeg-R1-4B reaches $[0,1000]$0. By contrast, Seg-R1-7B and Seg-Zero-7B both achieve $[0,1000]$1 no-target accuracy because they always hallucinate masks on negative prompts (Hegde et al., 10 Feb 2026).

The paper further reports that GenSeg-R1-G correctly abstains on 295 of 358 no-target prompts and has only a $[0,1000]$2 false-negative rate, whereas GenSeg-R1-4B has a $[0,1000]$3 false-negative rate (Hegde et al., 10 Feb 2026). In the paper’s discussion, this is presented as a practical contribution for interactive settings, where confidently segmenting a non-existent target is worse than abstaining.

A gain-attribution analysis isolates the effect of GRPO finetuning relative to the instruction-tuned baselines. For the 8B model, cIoU rises from $[0,1000]$4 to $[0,1000]$5 and mIoU from $[0,1000]$6 to $[0,1000]$7; for the 4B model, cIoU rises from $[0,1000]$8 to $[0,1000]$9 and mIoU from $\langle think \rangle$0 to $\langle think \rangle$1 (Hegde et al., 10 Feb 2026). These figures support the claim that RL training changes the grounding behavior substantially, not merely incrementally.

The paper also examines overlap between GRefCOCO and RefCOCOg. Although many images overlap, only 10 of the 2,573 RefCOCOg validation expressions appear verbatim in GRefCOCO training, and the clean-subset cIoU is reported as equal or higher than full-set cIoU. The authors use this to argue against simple score inflation through leakage (Hegde et al., 10 Feb 2026). This does not eliminate all concerns about overlap, but it narrows the memorization-based explanation for the reported gains.

Finally, the paper interprets the prevalence of $\langle think \rangle$2 traces and the strong ReasonSeg performance as evidence of emergent reasoning useful for compositional grounding. That interpretation is consistent with the reported results, though it remains tied to the specific benchmarks and reward structure used in the study (Hegde et al., 10 Feb 2026).

6. Relation to adjacent work and stated limitations

GenSeg-R1 sits within a broader family of RL-driven multimodal segmentation and reasoning systems. Seg-R1 had already reformulated segmentation as autoregressive prompt generation with GRPO, using Qwen-2.5-VL to emit bounding boxes, points, and labels that guide SAM 2; it reported strong results on COD10K and surprising zero-shot transfer to referring and reasoning segmentation, despite training only on foreground segmentation image-mask pairs (You et al., 27 Jun 2025). GenSeg-R1 differs in task definition, prompt schema, and reward design: it is explicitly built for referring segmentation, uses per-instance boxes plus two interior keypoints, and introduces no-target handling and SAM 2-in-the-loop optimization for GenSeg-R1-G (Hegde et al., 10 Feb 2026).

ConceptSeg-R1 generalizes the R1 direction further by formalizing concept segmentation across context-independent, context-dependent, and context-reasoning categories, introducing Meta-GRPO, a Concept Translation Module, and a Shortcut Router on top of SAM 3 (Zhao et al., 19 May 2026). Compared with that system, GenSeg-R1 is narrower in scope but more tightly engineered for fine-grained referring segmentation with natural-language queries and explicit no-target abstention. Omni-R1 is broader still: it proposes unified generative multimodal reasoning through interleaved visual actions such as $\langle think \rangle$3, $\langle think \rangle$4, $\langle think \rangle$5, $\langle think \rangle$6, and $\langle think \rangle$7, using perception-aligned SFT and perception-calibrated RL (Cheng et al., 14 Jan 2026). A plausible implication is that GenSeg-R1 occupies the grounded, segmentation-specific end of a larger trend toward policy learning over structured visual actions.

The limitations stated by the GenSeg-R1 paper are concrete. Training data comes from COCO-family images, so performance is not characterized on medical, satellite, or industrial imagery. Long-tail attribute combinations remain challenging. SAM 2 is sensitive to prompt format, and minor coordinate noise can cause discontinuous changes in output masks. The paper also notes that, although overlap analysis argues against inflated results, fully disjoint training and evaluation would make the zero-shot claim stronger (Hegde et al., 10 Feb 2026).

Taken together, these constraints define the current scope of the method. GenSeg-R1 demonstrates that fine-grained referring segmentation can be formulated as policy learning over structured spatial prompts, with the downstream segmenter kept frozen and the reward aligned to actual mask quality. The reported evidence indicates that this recipe is effective on RefCOCOg, GRefCOCO, and ReasonSeg, especially when prompt optimization includes explicit no-target handling and SAM 2-in-the-loop reward (Hegde et al., 10 Feb 2026).