B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

Abstract: Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis. For complex referring segmentation, recent methods pair large vision-LLMs with segmentation decoders: the former analyzes the image and prompt, while the latter predicts the target mask. Although reinforcement learning improves reasoning-intensive vision-language systems, trainable tools such as segmentation decoders are typically optimized separately with differentiable objectives, and the principled integration of such objectives into reinforcement learning remains underexplored. Thus, we introduce group relative tool optimization (GRTO), a mathematically grounded framework for jointly optimizing a policy with differentiable tool use. GRTO reuses group relative policy optimization (GRPO) rollouts to optimize the auxiliary tool objective, letting decoder gradients complement policy rewards. Further, we derive Bootstrapped-GRTO (B-GRTO), a pre-training method that cheaply bootstraps the tool, leading to faster convergence and superior performance. Across three challenging referring segmentation settings, B-GRTO results in substantial improvements over plain GRPO, matching or surpassing domain-specific state-of-the-art methods. This demonstrates the value of unifying reinforcement learning with differentiable auxiliary objectives for reasoning-intensive segmentation.

• The paper introduces a unified RL framework that jointly optimizes segmentation policies and tools using a bootstrapping approach.
• It integrates reward signals with dense differentiable losses to enhance mask quality under ambiguous and challenging conditions.
• Experimental results show improved convergence and robust performance across camouflaged, remote sensing, and reasoning segmentation tasks.

Bootstrapped Group Relative Tool Optimization for Referring Segmentation

Overview

"B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation" (2605.23500) introduces a mathematically rigorous regime for jointly optimizing a reasoning policy and a trainable segmentation tool in complex vision-language referring segmentation tasks. The paper builds on Group Relative Policy Optimization (GRPO) and extends it into Group Relative Tool Optimization (GRTO), enabling unified reinforcement learning for vision-LLMs (VLMs) and segmentation decoders. It further proposes Bootstrapped-GRTO (B-GRTO), which leverages off-policy experience replay to pre-align the segmentation tool efficiently, improving convergence and robustness in downstream joint optimization.

Motivation and Problem Formulation

Referring segmentation tasks go beyond static semantic segmentation by requiring a model to ground open-ended prompts in visual input, producing a mask for the object(s) referenced in natural language. These settings pose compounded challenges: prompts may be ambiguous, while visual domains can be out-of-distribution or feature low-contrast, fine-grained targets. Existing systems typically treat segmentation decoders (e.g., SAM3) as fixed, training only the VLM policy via RL, or alternatively, fine-tune the decoder with supervised losses under perfectly specified prompts. Both fail to address the scenario where both prompt ambiguity and domain adaptation are required.

The key insight of this paper is that trainable tools—segmentation decoders—should be jointly optimized with the policy, using both RL feedback and dense differentiable objectives, thereby supporting open-domain reasoning with adaptable mask prediction.

Figure 1: Challenging examples requiring both prompt reasoning and a tuned segmentation tool; B-GRTO enables robust optimization when neither the tool nor prompt is optimal.

Methodology

GRTO: Joint Policy and Tool Optimization

The GRTO objective integrates policy optimization and tool fine-tuning in a single framework. Policy updates are driven by reward signals from mask quality, implemented via GRPO, while tool gradients are estimated from the same sampled groups using auxiliary differentiable losses (BCE + soft-IoU). The joint objective aligns the tool to the direction of policy improvement, enabling synergy between prompt interpretation and mask quality.

Figure 2: Schematic comparison—standard fine-tuning needs optimal prompts or tools; most RL approaches fix the tool. GRTO/B-GRTO enables optimization when both are imperfect, with BTO allowing cheap tool pretraining and bootstrapping.

B-GRTO: Bootstrapped Tool Optimization

Given the asymmetry in training cost between the policy (expensive RL rollouts) and the tool (cheap differentiable updates), B-GRTO decouples the tool pretraining stage. Bootstrapped Tool Optimization (BTO) leverages reward-weighted experience replay from a reference policy, pre-aligning the segmentation decoder to anticipated post-policy improvements at negligible additional computational cost. Subsequent GRTO joint training continues to refine both components.

Figure 3: The B-GRTO pipeline—tool bootstrapping (BTO) via replay buffer from reference policy, followed by joint GRTO training with policy and tool.

Mathematically, the BTO stage uses importance sampling from reference rollouts to optimize the tool, and GRTO supplements this with online RL and dense loss gradients, maintaining theoretical consistency with the KL-regularized RL objective.

Experimental Evaluation

Domains and Setup

Experiments are conducted on three orthogonal referring segmentation settings:

• Camouflaged Object Detection: low-contrast targets (COD10K, CAMO, NC4K).
• Remote Sensing Segmentation: heterogeneous geospatial scenes (EarthReason).
• Challenging Reasoning Segmentation: compositional queries requiring deep contextual grounding (ReasonSeg-X, ReasonSeg-R).

InternVL3.5-8B is used as the VLM backbone, with SAM3 as the trainable mask decoder. The policy is RL-optimized with mask IoU rewards, while the decoder is trained with BCE + soft-IoU.

Main Results

B-GRTO matches or improves over domain-specific state-of-the-art methods in camouflage and remote sensing segmentation, consistently outperforming policy-only GRPO and basic GRTO (joint optimization without bootstrapping). In challenging reasoning segmentation, gains are less pronounced relative to specialized architectures but substantial compared to baseline GRPO.

Figure 4: Camouflage detection—weighted F-measure validation curves; B-GRTO attains superior convergence and final performance, with negligible bootstrapping cost.

Figure 5: Remote sensing segmentation—mean validation (gIoU/cIoU); B-GRTO achieves stronger robustness and rapid convergence.

Figure 6: Reasoning segmentation—mean validation (gIoU/cIoU); bootstrapping improves robustness, especially in zero-shot settings.

Qualitative results reveal improved prompt generation and mask delineation for B-GRTO, particularly in ambiguous or domain-shifted scenarios. Error analyses further show that class-dependent improvements from bootstrapped tool optimization are consistent across settings.

Figure 7: COD10K qualitative results—B-GRTO produces sharper masks for camouflaged objects.

Figure 8: ReasonSeg-X qualitative results—robust grounding for compositional queries with B-GRTO.

Figure 9: EarthReason qualitative results—B-GRTO adapts to annotation semantics in remote sensing.

Ablations and Study

Ablation regimes demonstrate the necessity of joint objective alignment—reverse sequential tool fine-tuning fails to match GRTO/B-GRTO performance, and bounding-box post-filtering is necessary for optimal results. Memory-bound settings where only the tool is bootstrapped (B-GRPO) still yield gains, confirming the value of BTO.

Figure 10: COD10K error study—class-wise gains for bootstrapped tool versus policy-only optimization.

Figure 11: EarthReason error study—bootstrapping reduces error across diverse geospatial classes.

Implications and Future Directions

The findings illustrate the practical importance of integrating RL feedback with continuous tool learning. Trainable segmentation decoders, when aligned to policy optimization, unlock substantial improvements for reasoning-driven pixel-level tasks. The bootstrapping approach enables cost-efficient initialization, generalizes across domains, and mitigates reward signal drift from policy evolution.

Practical implications include enhanced robustness for real-world referring segmentation pipelines in robotics, medical imaging, and geospatial analysis, where both prompt ambiguity and domain shift are prevalent. Theoretically, B-GRTO demonstrates that unified policy-tool optimization expands the space of tractable RL for vision-language systems.

Future research may investigate finer granularity in bootstrapping—dynamic replay buffers, curricula for under-represented classes, and hybrid RL/SFT objectives—potentially yielding further gains. Extensions to dense generation tasks, such as autoregressive multimodal outputs, may benefit from analogous unified objectives.

Conclusion

Bootstrapped Group Relative Tool Optimization (B-GRTO) rigorously combines RL-based policy learning with replay-buffer-driven tool fine-tuning for complex referring segmentation. Across camouflaged object detection, remote sensing, and compositional reasoning, fine-tuning the decoder within the joint objective yields significant gains over policy-only RL. The results imply that the segmentation tool should not be treated as a fixed reward model but continuously aligned with evolving policy strategies. B-GRTO thus provides a general-purpose regime for vision-language segmentation tasks, balancing robustness, convergence, and cross-domain adaptability.