ConverSeg: Conversational Image Segmentation

• ConverSeg is a large-scale benchmark and framework for conversational image segmentation that integrates free-form language prompts with precise pixel-level masks.
• It employs a scalable, automated data engine to synthesize tens of thousands of aligned prompt–mask pairs, enabling robust and diverse model training.
• ConverSeg-Net, its adapter-augmented model, effectively fuses visual and language priors to achieve state-of-the-art performance across multiple reasoning categories.

ConverSeg is a large-scale benchmark and framework for Conversational Image Segmentation (CIS), a task in which models must ground free-form, intent-driven natural language prompts into high-precision pixel-level masks. CIS expands the classic scope of natural-language image segmentation—traditionally limited to object naming or spatial descriptors—by including joint reasoning over high-level concepts such as physical functions, affordances, safety, and dynamic events. The ConverSeg ecosystem comprises (1) the ConverSeg benchmark featuring seven distinct reasoning categories and (2) ConverSeg-Net, a single-pass, adapter-augmented model leveraging both segmentation and language priors. The platform is enabled by a scalable, automated data engine that synthesizes tens of thousands of aligned prompt–mask pairs without the need for manual annotation (Sahoo et al., 13 Feb 2026).

1. Scope and Motivation for Conversational Image Segmentation

Conversational Image Segmentation (CIS) is defined by the task: given an image $I$ and a free-form linguistic prompt $p$, predict a binary mask $M_p \in \{0,1\}^{H \times W}$ such that $M_p(x, y) = 1$ for those pixels $(x, y)$ that satisfy the intent of $p$. Prompts often reference implicit or higher-order attributes—functional affordances (“segment surfaces safe for hot cookware”), transient events (“the player about to catch the ball”), safety considerations (“objects likely to tip over”), or counterfactual uses (“things you might use to prop open a door”).

Unlike Referring Image Segmentation (RIS), which predominantly tests entity recognition or spatial localization (e.g., “the red car,” “leftmost person”), CIS challenges models to integrate abstract reasoning (functionality, intent, physics, safety) with open-vocabulary perceptual grounding. This broadening of scope is critical for robotics, human–machine collaboration, and augmented reality, where effective interaction often requires mapping nuanced, intent-rich instructions to actionable visual regions.

2. Benchmark Structure and Dataset Characteristics

The ConverSeg benchmark systematically categorizes instructions and their corresponding masks into seven concept domains:

Concept	Representative Prompt Example	Reasoning Type
Entities	"the weathered wooden furniture"	Attribute-based
Spatial	"three leftmost cups", "items inside containers"	Geometric/Layout
Intent/Events	"the door being opened"	Event/Relation
Affordances	"surfaces you could cut on"	Functional
Functions	"props to hold open a door"	Use/Causality
Safety	"sharp objects posing a hazard"	Hazard/ Stability
Physics	"surfaces stable enough to stack books"	Intuitive Physics

ConverSeg includes 1,687 human-verified prompt–mask pairs on COCO validation images, with roughly 20% in both Entities and Spatial, and the remainder distributed among higher-level categories. Approximately 493 instances are annotated manually (average prompt length 7.8 words), with the remaining 1,194 masks “SAM-seeded” by combining detection boxes with segmentation masks from SAM2. Training data is scaled up synthetically to 106,000 positive and 106,000 negative (empty mask) prompt–mask pairs, enabling robust model learning without human-in-the-loop supervision.

Negative prompts are constructed to improve robustness, including adversarial queries with no valid mask (e.g., “segment the wine glass” when absent).

3. AI-Powered Data Generation Pipeline

The data engine underpinning ConverSeg operates through a five-stage vision-language modeling (VLM) driven pipeline:

1. Scene Understanding: Given image $I$, a VLM describes 5–7 salient regions, each with category, attribute, and location information.
2. Mask Generation: For each region description, bounding boxes are generated (Moondream3), and coarse masks are extracted (SAM2).
3. Mask Verification and Refinement: VLM checks mask–text consistency; subpar masks are refined using dense point sampling and selected for sharpness via IoU maximization and VLM evaluation.
4. Prompt Generation: For each of the seven concept categories, meta-prompts steer the VLM to generate up to three nontrivial, in-scope prompts per region, linked to the corresponding mask.
5. Prompt–Mask Alignment Verification: VLM ensures that the mask strictly matches the semantics of the generated prompt.

This automated engine allows scalable, diverse prompt–mask creation beyond manually feasible annotation rates.

4. ConverSeg-Net Model Architecture and Training

ConverSeg-Net is a compact, one-pass model that fuses visual segmentation priors with language understanding via lightweight adaptation:

• Image Encoder: SAM2 ViT-MAE (Vision Transformer pre-trained with Masked Autoencoding) processes RGB images, outputting a spatial feature map ($Z_\text{img}$), kept frozen during fine-tuning.
• Prompt Encoder: Qwen2.5-VL (3B parameters) jointly encodes image and text, producing sparse ($e_s$) and dense ($e_d$) prompt embeddings via linear and MLP adapters (with LoRA).
• Adapters and Fine-tuning: Only adapters, LoRA on cross-attention, and the SAM2 mask decoder are tuned.
• Mask Decoder: Combines $p$0, $p$1, and $p$2 via cross-attention blocks and an MLP head, producing the per-pixel mask probability $p$3.

Formalization of the fusion process:

$p$4

for $p$5.

Training Curriculum:

• Phase 1: Pretraining on 440K segmentation masks (COCONut, PACO), 321K referring-expression (RefCOCO/+/g) pairs, 48K open-vocabulary region masks.
• Phase 2: Post-training with 106K synthetic positive, 106K negative pairs, and an equal mix of Phase 1 data.
• Loss function:

$p$6

5. Empirical Results and Performance Analysis

ConverSeg-Net achieves state-of-the-art results on the ConverSeg benchmark and strong performance transfer to classical RIS and zero-shot reasoning benchmarks.

ConverSeg gIoU Performance:

Category	SAM-seeded	Human-annotated
All	70.8	67.4
Entities	74.0	71.6
Spatial	70.9	68.7
Intent/Events	74.1	67.0
Affordances	68.7	64.4
Functions	--	63.5 (7B)
Safety/Physics	64.2	63.8

ConverSeg-Net surpasses previous feed-forward models such as Seg-Zero, LISA-Llama2-13B, and EVF-SAM (BEIT-3) in both SAM-seeded and human-annotated splits.

On classical RIS datasets:

• RefCOCO (Qwen3B): 78.4/80.8/75.8 (val/testA/testB)
• RefCOCO+ (Qwen3B): 72.5/77.7/66.4
• RefCOCOg (Qwen3B): 75.1/74.7/74.7

On ReasonSeg (zero-shot):

• ConverSeg-Net (3B): 56.4 (val)/52.2 (test)
• ConverSeg-Net (7B): 61.9 (val)/57.0 (test)

Ablation studies indicate substantial drops in performance when omitting LoRA fine-tuning, text+image joint encoding, or dense prompt embeddings. Curriculum mixing Phase 1 and Phase 2 data is critical for balancing generalization and task specificity.

6. Applications, Limitations, and Future Directions

Applications include:

• Assistive robotics: mapping conversational queries (“Which cup can I pick up without smashing the others?”) to actionable visual regions.
• Human–robot collaboration: following verbal instructions concerning safety, assembly/disassembly, or spatial manipulations.
• Augmented reality: overlaying guidance for safe and effective interactions (“highlight all stable surfaces”).

Known Limitations:

• Prompt ambiguity: Some queries (“segment reflections”) exhibit high inter-human and model variance.
• Precision–recall tradeoff: The model may mask only a subset of valid instances, reflecting high precision but limited recall.
• Limited real-world physics: Queries involving hidden or out-of-domain attributes (“objects supporting >5kg”) are not reliably grounded by current vision-LLMs.

Future Improvements:

• Integration of lightweight physics simulation or stability predictors for advanced physical reasoning.
• Enhanced prompt engines for multi-turn clarifications and disambiguation.
• Extension to temporal tasks (video) for event grounding.
• Human-in-the-loop feedback for nuanced or subjective segmentation.

7. Significance within the Language-Guided Segmentation Landscape

ConverSeg establishes a new paradigm for evaluating and training segmentation systems on intent-driven, conversationally specified tasks. By leveraging automated, scalable data curation and a two-phase curriculum, ConverSeg-Net bridges the gap between entity-centric RIS and the full spectrum of real-world functional, event-based, and safety-critical segmentation demands. The demonstration of robust transferability to both benchmark and out-of-distribution datasets underscores its utility across numerous vision–language practical scenarios (Sahoo et al., 13 Feb 2026).