ConverSeg-Net: Conversational Image Segmentation

• The paper presents a novel single-pass, end-to-end framework, ConverSeg-Net, that grounds conversational prompts into pixel-accurate segmentation masks.
• It integrates a high-resolution segmentation vision backbone with a compact vision–language model using lightweight adapters for efficient cross-modal fusion.
• Training leverages an AI-powered data engine and a two-phase curriculum, achieving state-of-the-art performance on diverse segmentation benchmarks.

ConverSeg-Net is a single-pass, end-to-end framework for Conversational Image Segmentation (CIS), designed to ground open-ended, intent-driven natural language prompts into pixel-accurate segmentation masks. It addresses the challenge of segmenting abstract, functional, and physical reasoning queries that go beyond traditional categorical or spatial referring expression segmentation. ConverSeg-Net achieves this by fusing a high-resolution segmentation vision backbone with a compact vision–LLM, supported by an AI-powered data engine that synthesizes large-scale prompt–mask pairs without human supervision. The model demonstrates significant gains on the ConverSeg benchmark and competitive performance on standard language-guided segmentation tasks, excelling particularly in abstract, affordance-related, and safety-driven prompts (Sahoo et al., 13 Feb 2026).

1. Model Architecture

ConverSeg-Net consists of four major components that together ground conversational prompts into segmentation masks:

1. Image Encoder: Utilizes the Segment Anything Model v2 (SAM2) image encoder, a Masked Autoencoder (MAE)–pretrained Vision Transformer (ViT) fine-tuned for high-resolution segmentation. For an image $I\in\mathbb{R}^{H\times W\times 3}$, the encoder outputs feature maps $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$, where $H'=H/16$, $W'=W/16$, $D_{\mathrm{img}}=1024$. The encoder is frozen during training.
2. Prompt Encoder: Employs Qwen-2.5-VL-3B as a joint vision–language backbone. It processes both image and conversational text prompt $p=\{w_1,\dots,w_T\}$, producing text-token-aligned embeddings $$\{\mathbf{h}_1,\dots,\mathbf{h}_T,\mathbf{h}_{\mathrm{EOS}}\} \subset \mathbb{R}^{D_t}$$, with $D_t=4096$.
3. Lightweight Adapters for Cross-Modal Fusion: Cross-modal fusion is realized via two types of prompt embeddings: sparse (token-wise, $$\mathbf{e}_t = W_s\,\mathbf{h}_t \in \mathbb{R}^{D_{\mathrm{dec}}}$$ with $W_s \in \mathbb{R}^{D_{\mathrm{dec}} \times D_t}$, $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$0) and a dense embedding ($$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$1), with the latter used as a global spatial bias. LoRA is applied to Qwen’s self- and cross-attention modules (rank $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$2, scaling $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$3), enabling efficient fine-tuning.
4. Mask Decoder: Leverages SAM2’s four-layer masked Transformer decoder, operating with bidirectional cross-attention between image and prompt embeddings. The output is upsampled to the image resolution and mapped by an MLP to pixel-wise mask logits, followed by a sigmoid activation to yield the probability mask $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$4.

2. Training Objectives and Procedure

ConverSeg-Net is trained to minimize a combination of pixel-wise binary cross-entropy (BCE) and Dice loss between predicted masks $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$5 and ground-truth $$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$6:

$$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$7

$$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$8

$$\mathbf{Z}_{\mathrm{img}} \in \mathbb{R}^{H'\times W'\times D_{\mathrm{img}}}$$9

The optimization utilizes AdamW with a learning rate of $H'=H/16$0, weight decay 0.05, cosine schedule, and a batch size of 6. Training is staged in two phases: (1) pretraining on literal concept datasets for 100k steps, and (2) conversational post-training for 90k steps by mixing conversational and literal data on a single NVIDIA A100 80 GB GPU, resizing images to a max side of 1024 px.

3. AI-Powered Data Engine

A fully automated pipeline generates 106k conversational prompt–mask pairs for training:

1. Scene Understanding: A VLM (Gemini-2.5-Flash) analyzes each image, outputting 5–7 region descriptions ($H'=H/16$1) encompassing categories, attributes, and context.
2. Mask Generation: Region descriptions are mapped to bounding boxes by Moondream3, then to coarse masks $H'=H/16$2 by SAM2.
3. Mask Verification and Refinement: Mask–text consistency is checked by a VLM; refinement is performed using IoU-driven sampling and selection among candidate masks.
4. Concept-Driven Prompt Generation: Five meta-prompts (Entities, Spatial Layout, Relations/Events, Affordances/Functions, Physics/Safety) condition a VLM to generate up to three conversational queries per concept family, paired with aligned masks.
5. Prompt–Mask Alignment Verification: Final VLM checks ensure semantic alignment of (prompt, mask) pairs; only validated pairs are retained. Additional negative prompts are constructed to train empty-mask prediction on invalid queries.

This pipeline operates without human supervision, enabling large-scale, diverse training for complex conversational image segmentation.

4. Quantitative and Qualitative Evaluation

ConverSeg-Net achieves state-of-the-art performance on the ConverSeg benchmark, which covers five concept families: entities, spatial relations, relations/events, affordances/functions, and physics/safety. Generalized IoU (gIoU, %) on both SAM-seeded and human-annotated splits is summarized below:

Model	Prompt Encoder	All (SAM)	All (Human)
LISA (Llama2 13B)	Llama2 13B	55.2	53.8
Seg-Zero	Qwen2.5-VL 7B	69.2	61.1
ConverSeg-Net (3B)	Qwen2.5-VL 3B	70.8	67.4
ConverSeg-Net (7B)	Qwen2.5-VL 7B	72.4	67.9

ConverSeg-Net also maintains competitive results on standard referring expression datasets (RefCOCO/+/g), with gIoU of 78.4–79.4% (RefCOCO), 72.5–74.3% (RefCOCO+), 75.0–75.5% (RefCOCOg), and achieves strong zero-shot results on ReasonSeg (52.2–57.0%), matching or surpassing models trained explicitly on these domains.

Qualitative analysis reveals that ConverSeg-Net generates precise masks for prompts involving physical affordances and nuanced intent, such as isolating only “surfaces stable enough to stack books” or “objects likely to tip,” outperforming prior approaches that commonly miss such functional subtleties.

5. Ablation Analyses and Architectural Variants

Comprehensive ablation studies investigate the contribution of curriculum learning, adapter training, visual grounding, and embedding design:

• Curriculum learning: A two-phase curriculum (pretraining followed by conversational mixing) yields a superior balance between literal and conversational segmentation, with ConverSeg gIoU = 67.4%.
• Prompt encoder adaptation: Freezing the prompt encoder (disabling LoRA) results in a 19.1 pp drop in ConverSeg performance; text-only input to Qwen (no image context) causes a 17.9 pp drop.
• Embedding structure: Removing dense (global) embedding has minimal impact (–0.1 pp), indicating robustness to this architectural choice.
• Backbone flexibility: Swapping Qwen-2.5-VL-3B for Perception-LM-3B yields only a minor decrease (66.5% vs. 67.4%), suggesting generalizability across VLM backbones.

6. Data Diversity and Model Scalability

The effectiveness of ConverSeg-Net is closely linked to the scale and diversity of its training data, especially conversational prompt–mask pairs that span abstract, relational, and intent-driven segmentation challenges. The automated data pipeline enables broad generalization without relying on manual annotation. Scaling the prompt encoder model size from 3B to 7B parameters further improves segmentation accuracy across both abstract and literal concept types. This suggests that continued expansion of both data diversity and model capacity will likely benefit future iterations on this task.

7. Conclusion and Implications

ConverSeg-Net demonstrates that combining segmentation priors from SAM2 with a lightweight, LoRA-adapted vision–language backbone, and training on diverse automatically synthesized data, enables pixel-accurate grounding of conversational, abstract, and functionally specified queries. Its performance across ConverSeg, RefCOCO/+/g, and ReasonSeg benchmarks, along with qualitative advantages on affordance and safety segmentation tasks, underscore the importance of curriculum-based training and scalable data synthesis in advancing conversational image segmentation (Sahoo et al., 13 Feb 2026).