\textsc{CR-Seg}: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation

Abstract: Reasoning segmentation aims to segment target objects described by complex language through joint visual-textual reasoning. Existing methods typically rely on either learned semantic tokens to bridge Multimodal LLMs (MLLMs) and segmentation models, suffering from difficult cross-modal alignment, or explicit spatial prompts such as bounding boxes, which may lose holistic response semantics. To address these limitations, we propose Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation, termed CR-Seg, a two-stage framework for coarse-to-refined reasoning segmentation. Specifically, we design an Extract Attention Maps and Points (EAP) module to extract attention maps for coarse target localization and select informative points, both of which are fed into SAM for mask refinement. To alleviate reasoning--answer inconsistency, we further introduce Global-to-Local Chain-of-Thought (GLCoT), which guides the model to reason progressively from global scene context to local target details. Extensive experiments on reasoning segmentation benchmarks demonstrate the effectiveness of CR-Seg.

• The paper introduces CR-Seg, which uses raw MLLM attention maps, learned point prompts, and mask priors to connect Qwen3-VL with SAM3 for semantically aligned segmentation.
• The paper achieves 79.85% gIoU and 74.56% cIoU on FReasonSeg, while reaching 70.8% gIoU on ReasonSeg validation and outperforming prior single-step methods.
• The paper’s Global-to-Local Chain-of-Thought strategy reduces reasoning–answer inconsistency from 8.2% to 2.6% and improves ReasonSeg gIoU by 1.7 percentage points, though performance remains tied to SAM and model-specific attention patterns.

Motivation and problem statement

Reasoning segmentation requires a model to segment objects specified by implicit, reasoning-heavy language rather than explicit referring expressions. The dominant paradigm couples a Multimodal LLM (MLLM) with the Segment Anything family, but the interface between the two is the central design question. The paper identifies two established paradigms and their respective failures. Internal-Representation-Based (IRS) methods such as LISA, GSVA, PixeLM, and LENS use hidden states of special tokens (e.g., [SEG]) as the bridge; these states aggregate holistic response semantics but receive no explicit spatial supervision beyond the final mask loss, making cross-modal alignment costly. Explicit-Prompt-Based (ERS) methods such as Seg-Zero, VisionReasoner, and Dr. Seg train MLLMs to emit grounding cues (boxes or points); this avoids latent alignment training but compresses the entire chain-of-thought into sparse spatial prompts, so errors in grounding propagate directly to the mask.

The authors additionally observe a failure mode they term reasoning–answer inconsistency: when distractors appear in later reasoning steps, models may produce grounding predictions that contradict their own reasoning, yielding erroneous masks. CR-Seg addresses both problems by using MLLM attention maps—dense, directly mask-supervisable, yet semantically rich—as the spatial–semantic bridge between the MLLM and SAM3.

Method

CR-Seg uses Qwen3-VL-4B as the reasoning model and SAM3 as the segmenter, trained in two stages on 2 A6000 GPUs (~56 GPU hours total).

Attention extraction via learnable queries. After the MLLM generates a structured <thinking>…</thinking><answer>…</answer> response, $N$ learnable queries are appended to the sequence and passed through the backbone. Rather than reading out Softmax-normalized attention—which flattens distributions over long token sequences and weakens gradients—the method bypasses Softmax and uses the raw query–key affinity matrix $\frac{QK^\top}{\sqrt{d}}$ as the attention map. This design choice preserves dynamic range for cross-modal alignment.

EAP module. The Extract Attention Maps and Points module converts raw attention into SAM-compatible prompts: (1) visual-token entries are retained and averaged over layers and queries to produce an aggregated map $\operatorname{Att} \in \mathbb{R}^{H \times W}$; (2) foreground/background regions are defined by cumulative attention mass thresholds ($\alpha_{\text{fg}}, \alpha_{\text{bg}}$), from which five positive points (the argmax plus four attention-weighted samples) and five uniformly sampled negative points are drawn; (3) a four-layer convolutional connector projects $\operatorname{Att}$ into SAM's input space as a mask prior $M_P$. All three outputs condition SAM3's prompt encoder and mask decoder.

Stage 1 — pretraining alignment. Using RefCOCO series data, only LoRA adapters on the MLLM, the connector, and SAM's prompt encoder/mask decoder are unfrozen. The loss combines cosine similarity on the attention map with a segmentation loss (DICE + BCE + Sobel-based boundary loss). Training converges by ~3k steps, already reaching 70.2 gIoU on ReasonSeg Val at that point—an indication that the alignment stage alone carries most of the localization capability.

Stage 2 — SFT with GLCoT. Generation and segmentation are jointly optimized. To combat reasoning–answer inconsistency, the paper introduces Global-to-Local Chain-of-Thought (GLCoT): a three-step scheme (global scene survey → candidate identification → distractor exclusion → target localization). GLCoT annotations are distilled from Qwen3-VL-235B-A22B-Instruct, which is given ground-truth masks overlaid on images during data generation—a notable dependency, since teacher reasoning quality is conditioned on oracle spatial information. The total objective adds an SFT cross-entropy term to the alignment loss. For small targets in this stage, the positive point set is restricted to the single highest-attention point within the predicted box.

FReasonSeg benchmark

The paper argues that ReasonSeg mostly lacks visually similar same-category distractors, allowing textual reasoning to substitute for fine-grained visual discrimination. FReasonSeg fills this gap: built from RefCOCOm via Qwen3-VL-235B-A22B-Instruct, it contains three difficulty tiers (L1: no same-category ambiguity; L2: two same-category objects distinguishable by one feature; L3: three or more requiring multi-attribute composition), totaling 283 samples after multi-stage human filtering (L1 = 24, L2 = 125, L3 = 134). The small sample size—particularly the 24-sample L1 subset—is a caveat when interpreting per-tier results.

Main results

On FReasonSeg, CR-Seg achieves state-of-the-art overall performance:

Method	Overall gIoU	Overall cIoU
Seg-Zero-7B	76.00	69.78
Dr. Seg†	76.82	72.09
LENS	75.03	72.66
CR-Seg	79.85	74.56

(† denotes reproduction on Qwen3-VL-4B + SAM3.) Performance holds across L2/L3, suggesting the attention prior remains robust under same-category interference where text-guided multi-step agents (SAM3-Agent, Evol-SAM3) degrade substantially.

On ReasonSeg, CR-Seg attains 70.8/66.8 gIoU/cIoU on Val and 64.8/62.6 on Test, surpassing the previous single-step state of the art by 2.4% gIoU on Val and achieving the best average across splits—all with LoRA fine-tuning on limited data. The paper emphasizes reduced alignment overhead relative to IRS methods (fewer epochs × dataset size), though the comparison against RL-trained ERS baselines mixes training regimes and should be read accordingly.

Ablations and analysis

GLCoT. Introducing GLCoT raises gIoU from 69.1 to 70.8 (+1.7%) while reducing the newly proposed RAI metric (fraction of samples whose reasoning contradicts the final answer, verified manually after LLM-based flagging) from 8.2 to 2.6 (−5.6%). Without GLCoT, RAI is comparable to VisionReasoner (11.5) and Dr. Seg (9.7). Since CR-Seg's output length (139 words) is not the longest among compared methods, the lower RAI is attributed to the reasoning structure rather than verbosity.

Components and stages. Removing point prompts costs 2.3 gIoU; skipping Stage-1 unfreezing of the MLLM costs 3.4; skipping connector or SAM unfreezing costs more (64.9 without SAM tuning); skipping either stage entirely drops performance sharply (55.9 without Stage 1, 57.2 without Stage 2), confirming both stages are necessary.

Query count. Performance peaks at 64 learnable queries and degrades beyond, hypothesized to reflect inter-query competition for salient regions.

Case study. When fed VisionReasoner's erroneous reasoning as input alongside learnable queries, CR-Seg still attends to the correct region and produces an accurate mask. This is the paper's strongest claim regarding the attention bridge: mask generation appears resilient to upstream reasoning errors because attention aggregates semantics from the full response rather than a single discrete grounding output. Attention visualizations over "Step 1–3" tokens further show systematic narrowing from global scene regions to target boundaries, supporting the coarse-to-fine claim mechanistically rather than only through end metrics.

Scaling and backbones. Scaling Qwen3-VL from 4B to 8B improves FReasonSeg (79.85 → 81.42) though slightly reduces ReasonSeg Val. Swapping SAM3 for SAM2 changes results by less than 1%. Substituting Qwen3.5-4B (hybrid attention, restricted to eight full-attention layers) helps ReasonSeg but hurts FReasonSeg, indicating that concentrated attention favors textual-reasoning-dominated cases while broader attention supports instance discrimination—an interesting but unexplored observation.

Limitations and open questions

The paper concedes several dependencies. CR-Seg remains tied to SAM, and the convolutionally projected attention maps diverge from SAM's native mask-prompt format, which the authors acknowledge limits further gains. The framework depends on the MLLM's internal attention mechanism—making it sensitive to architectural changes, as the Qwen3.5 hybrid-attention experiment demonstrates—and still requires supervised training, unlike training-free agent pipelines. The GLCoT distillation pipeline relies on a large teacher given oracle masks, raising questions about how well the learned reasoning transfers when teacher supervision quality degrades. Finally, FReasonSeg is small (283 samples) and derived from RefCOCOm imagery, leaving open whether the reported robustness to same-category distractors generalizes beyond its distribution. Two directions the authors name explicitly: simpler refinement strategies that reduce dependence on SAM's prompt format, and attention pathways requiring less additional training.

Conclusion

CR-Seg positions MLLM attention maps as an explicitly supervisable, mask-aligned bridge that mitigates the alignment cost of IRS methods while preserving response semantics that ERS methods discard, and pairs this with GLCoT to reduce reasoning–answer inconsistency. The empirical evidence—state-of-the-art FReasonSeg results, competitive ReasonSeg performance under LoRA-scale training, and a substantial RAI reduction—supports the core design, while the residual coupling to SAM's prompt interface and to MLLM-specific attention structure defines the open problems the work leaves unresolved.