SegCompass: Interpretable Reasoning Segmentation

• SegCompass is a reasoning segmentation model that maps chain-of-thought tokens and visual features into a shared, sparse concept space.
• It employs a query codebook and slot mapper to convert sparse activations into multi-slot heatmaps that guide pixel-level mask predictions.
• The model combines reinforcement learning with supervised segmentation losses to achieve state-of-the-art performance and enhanced interpretability.

SegCompass is a reasoning-segmentation model that uses a Sparse Autoencoder (SAE) to build an explicit, interpretable, and differentiable alignment pathway between chain-of-thought reasoning and visual grounding. Given an image-instruction pair, it first generates a chain-of-thought trace, then maps both reasoning tokens and visual tokens into a shared high-dimensional sparse concept space, selects salient concepts with a query codebook, grounds them through a slot mapper into a multi-slot heatmap, and finally decodes the heatmap into a segmentation mask. It is designed for reasoning segmentation, where the target object must be resolved through compositional, relational, attribute-based, or commonsense reasoning rather than by text-conditioned segmentation alone, and it is reported to match or surpass state-of-the-art performance on RefCOCO, RefCOCO+, RefCOCOg, gRefCOCO, and ReasonSeg (Lu et al., 21 May 2026).

1. Task setting and motivation

SegCompass addresses reasoning segmentation: the input is an image together with a natural-language instruction, and the output is a pixel-level mask for the referred object or objects. The paper presents this setting as more demanding than standard referring segmentation because the instruction can require multi-step resolution of relations, attributes, and commonsense constraints. The stated example is selecting “the mug that matches the plate’s color and is closest to the sink” (Lu et al., 21 May 2026).

The motivating problem is an interpretability gap in earlier reasoning-segmentation pipelines. The paper distinguishes two prior tendencies. The first is latent query alignment, exemplified by LISA-style pipelines and related successors, where hidden states from an LLM or MLLM are mapped into latent queries for a mask decoder. This is end-to-end trainable, but the paper characterizes it as an opaque “black box.” The second is textual localization readout, where the model emits intermediate box coordinates, point prompts, or patch indices. This is readable, but the paper argues that it is not genuinely interpretable, because the chain-of-thought is often unconstrained and the textual localization stage often functions as an unconstrained post-hoc interface rather than a differentiable alignment mechanism (Lu et al., 21 May 2026).

SegCompass is proposed as a direct response to that gap. Its central claim is that reasoning segmentation requires a white-box alignment interface that exposes how language-side reasoning concepts become spatial evidence and then masks. In the paper’s formulation, the SAE provides that interface by turning hidden states into sparse concept activations whose support, aggregation, grounding, and downstream mask effects can all be inspected.

2. Core architecture and representation pipeline

At a high level, SegCompass consists of a multimodal LLM for reasoning generation, an SAE-based concept interface, a query codebook for concept selection, a slot mapper for spatial grounding, and a final mask decoder (Lu et al., 21 May 2026).

Module	Role	Key output
MLLM policy	Generate CoT and concentration tokens	$y_{1:T}$, $e_k$
SAE	Map hidden states into sparse concept space	$h(z)$
Query codebook	Decode active sparse concepts into dense concept vectors	$r_k$
Slot mapper	Ground concept-slot queries onto image features	$H_k$, $C_k$
Mask decoder	Produce final masks from keys and heatmaps	$M$

The visual branch uses a frozen image encoder, in practice SAM ViT-H, to extract image features: $$K = F_{\text{enc}}(x_{\text{img}}), \qquad K \in \mathbb{R}^{h \times w \times d_k}.$$ These keys provide the spatial substrate for later grounding.

The reasoning branch uses LLaVA-1.5 or Qwen2.5-VL as the MLLM backbone. For an image-instruction pair, the policy generates a token sequence containing a > ... </think> chain-of-thought trace together with $K_s$ concentration tokens. The generation process is written as

$$y_t \sim \pi_\theta(\cdot \mid y_{0:t-1}, x_{\text{img}}, x_{\text{txt}}), \quad t=1,\ldots,T.$$

The hidden states at the concentration-token positions are extracted as $e_k$0 and later serve as compact reasoning-conditioned query embeddings (Lu et al., 21 May 2026).

The SAE is the central interface. It receives hidden states

$e_k$1

from a selected intermediate MLLM layer and maps them into a sparse space

$e_k$2

with decoder reconstruction

$e_k$3

The paper gives the example

$e_k$4

The support of active sparse features is

$e_k$5

This support set is important because it makes the intermediate representation inspectable at the level of activated sparse concepts (Lu et al., 21 May 2026).

The query codebook is a learned matrix

$e_k$6

described in one passage as $e_k$7, whose role is to convert active sparse dimensions into dense concept vectors. The nonzero activations $e_k$8 are decoded through this codebook, and Transformer encoder blocks aggregate them into $e_k$9 concept representations $h(z)$0.

Each concept representation is fused with its corresponding concentration-token embedding: $h(z)$1 These slot queries are then grounded onto image keys by the slot mapper. Multi-head attention scores are computed as

$h(z)$2

Two heads produce, respectively, a multi-slot heatmap and per-slot confidences: $h(z)$3 The heatmaps are the explicit spatial grounding objects in SegCompass; the confidences indicate the reliability of each slot (Lu et al., 21 May 2026).

Finally, the decoder consumes image keys together with slot heatmaps and confidences: $h(z)$4 The decoder is described as a lightweight module with three stacked 2D convolutional blocks followed by a SAM-style Two-Way Transformer. The paper emphasizes that the heatmap is not merely explanatory: it is the actual spatial bottleneck through which reasoning-guided grounding affects mask prediction.

3. Sparse Autoencoder and joint optimization

The SAE is pretrained before downstream task training and then frozen. Its objective is

$h(z)$5

The first term is a reconstruction loss, and the second is an $h(z)$6 sparsity penalty. The paper’s argument is that this overcomplete sparse representation reduces feature superposition and promotes more disentangled concept dimensions, making the interface more interpretable (Lu et al., 21 May 2026).

The supplementary implementation details specify that the SAE follows SAE-V, is trained separately for each MLLM backbone using 200K OBELICS samples, and is attached to middle layers rather than late layers. The reported layer choices are layer 16 of 32 for LLaVA-1.5-7B, layer 16 of 40 for LLaVA-1.5-13B, and layer 13 of 28 for Qwen2.5-VL-7B. SAE training uses learning rate $h(z)$7, batch size 64, and 10 epochs, with warmup in the initial stage (Lu et al., 21 May 2026).

After SAE pretraining, SegCompass is trained jointly with reinforcement learning on the reasoning path and supervised segmentation losses on the grounding and mask path. The total loss is

$h(z)$8

The implementation sets

$h(z)$9

and within the segmentation term the Dice coefficient weight is

$r_k$0

The supervised segmentation loss is

$r_k$1

and the confidence loss is

$r_k$2

where the slot labels $r_k$3 are produced by bipartite matching between predictions and ground-truth masks (Lu et al., 21 May 2026).

The reinforcement-learning component uses GRPO. For each query $r_k$4, the old policy samples a group of outputs $r_k$5, and the optimization objective is given as

$r_k$6

The KL estimator is

$r_k$7

Each RL rollout receives a segmentation reward and a format reward. For single-object tasks, the segmentation reward is soft IoU between predicted probability mask and ground truth. For multi-object tasks, the model retains slot predictions above a confidence threshold, computes pairwise soft overlaps with ground-truth masks, uses Hungarian matching to maximize total overlap, and returns the mean overlap over matched pairs; if one side is empty and the other is non-empty, the reward is 0, while if both are empty it is 1. The format reward checks whether the output contains exactly one <think> ... block, non-empty content, and no extra text before or after. In the reported implementation, the segmentation reward weight is 0.7 and the format reward weight is 0.3 (Lu et al., 21 May 2026).

The downstream system is trained on 8 × NVIDIA A100 80GB, with AdamW, weight decay 0.01, base MLLM learning rate $r_k$8, query-codebook multiplier 25×, slot-mapper multiplier 80×, decoder multiplier 10×, OneCycleLR scheduling, gradient clipping 1.0, and global batch size 128. The maximum number of slots is $r_k$9, the maximum prompt length is 1400 tokens, and the maximum response length is 2000 tokens (Lu et al., 21 May 2026).

4. Interpretability mechanism and empirical evidence

The paper’s interpretability claim is not that the entire model is transparent in a symbolic sense, but that the alignment pathway is explicit and inspectable. Specifically, SegCompass exposes the sparse concept activations $H_k$0, the codebook-decoded concept vectors, the slot queries, the spatial heatmaps $H_k$1, and the slot confidences $H_k$2. This is the basis for the description of the method as a “white-box” connection that is more traceable than latent queries and more coherent than textual readouts (Lu et al., 21 May 2026).

The paper provides both visual and quantitative evidence for that claim. In its activation analysis, instance tokens and CoT tokens activate the most sparse features, question tokens activate fewer, and background tokens activate the fewest. The reported trend remains stable for thresholds below 0.1, which the authors interpret as evidence that the activations are not dominated by negligible noise. Qwen2.5-VL-7B shows higher activation counts and stronger separations than LLaVA-1.5-7B, matching its stronger downstream segmentation performance (Lu et al., 21 May 2026).

A second analysis measures whether the union of the top-$H_k$3 most activated tokens covers the referred instance better than random selection. The reported result is that top-activated tokens cover referred instances far better than a random baseline, even though the SAE itself is not trained directly with segmentation supervision. This indicates that sparse activations are aligned with object-relevant visual evidence rather than being purely abstract language features.

The paper’s strongest quantitative interpretability evidence is a pair of correlations. First, on Qwen2.5-VL-7B for gRefCOCO, the correlation between SAE reconstruction error and training Dice loss is reported as

$H_k$4

Second, the correlation between heatmap cIoU and final mask cIoU during inference is

$H_k$5

The first result ties concept-space quality to segmentation learning quality; the second ties the quality of the explicit grounded heatmap to the quality of the final predicted mask. The paper presents these results as evidence that the interpretability interface is not merely descriptive but functionally predictive of downstream segmentation accuracy (Lu et al., 21 May 2026).

The visualizations reported in the paper show that activated sparse features and heatmap peaks concentrate on instruction-relevant regions in ReasonSeg examples. A plausible implication is that SegCompass turns interpretability from a readout convenience into an operational intermediate representation: the grounded sparse concepts are the route by which reasoning affects segmentation.

5. Benchmarks, metrics, and reported performance

SegCompass is evaluated on five benchmarks: RefCOCO, RefCOCO+, RefCOCOg, gRefCOCO, and ReasonSeg. The reported metrics are cIoU on RefCOCO, RefCOCO+, and RefCOCOg, and both cIoU and gIoU on gRefCOCO and ReasonSeg. ReasonSeg is used in a zero-shot reasoning-segmentation setting (Lu et al., 21 May 2026).

The paper compares against 27 prior works grouped into three families: non-LLM segmentation methods, latent-query alignment methods, and textual localization readout methods. The strongest direct comparisons are drawn against methods such as LISA, RAS, LIRA, HiMTok, X-SAM, Text4Seg, VisionReasoner, Seg-Zero, and SAM-R1, although the central quantitative focus is on SegCompass itself (Lu et al., 21 May 2026).

Configuration	Reported benchmark results	Setting
SegCompass-7B (Qwen2.5-VL)	RefCOCO 85.3 / 87.5 / 83.6; RefCOCO+ 78.3 / 84.7 / 76.6; RefCOCOg 82.8 / 81.0; gRefCOCO val 76.1 gIoU / 72.2 cIoU; ReasonSeg test 64.0 gIoU / 64.8 cIoU	7B
SegCompass-13B (LLaVA-1.5)	RefCOCO 86.3 / 87.3 / 84.1; RefCOCO+ 80.5 / 84.6 / 76.9; RefCOCOg 84.0 / 81.3; gRefCOCO val 76.8 / 72.2; ReasonSeg test 64.2 / 66.5	13B

On gRefCOCO, the paper reports for SegCompass-13B: val 76.8 gIoU / 72.2 cIoU, testA 77.7 / 77.3, and testB 71.9 / 72.4. For SegCompass-7B with Qwen2.5-VL, the corresponding numbers are val 76.1 / 72.2, testA 76.8 / 76.5, and testB 70.0 / 72.8 (Lu et al., 21 May 2026).

On ReasonSeg in the reported zero-shot setting, SegCompass-13B reaches val 67.4 gIoU / 67.2 cIoU and test 64.2 / 66.5. SegCompass-7B with Qwen2.5-VL reaches val 66.6 / 66.3 and test 64.0 / 64.8. SegCompass-7B with LLaVA-1.5 is also reported, at val 66.3 / 65.6 and test 61.0 / 63.0 (Lu et al., 21 May 2026).

The paper’s summary claim is that SegCompass matches or surpasses state of the art on these benchmarks, particularly under comparable 7B and 13B backbone settings. It also provides public code at https://github.com/ZhenyuLU-Heliodore/SegCompass (Lu et al., 21 May 2026).

6. Ablations, limitations, and position within the literature

The ablation studies emphasize that SegCompass depends on the interaction between reasoning RL and supervised grounding. On SegCompass-13B, the training-mode comparison reports:

• RL only: RefCOCOg 65.9, gRefCOCO 63.0, ReasonSeg 40.1
• Supervision only: 77.9, 74.0, 59.3
• Both: 81.3, 77.3, 66.5

These results indicate that the reinforcement-learning signal and the supervised segmentation objectives are complementary rather than interchangeable (Lu et al., 21 May 2026).

The visual backbone ablation reports:

• ViT-B (0.09B): 78.8 / 75.0 / 58.8
• ViT-L (0.31B): 80.1 / 76.2 / 63.9
• ViT-H (0.64B): 81.3 / 77.3 / 66.5

Here the three numbers correspond to RefCOCOg, gRefCOCO, and ReasonSeg. The paper therefore attributes part of the final performance to the strength of the frozen vision encoder (Lu et al., 21 May 2026).

The reward-design ablation compares format and segmentation reward weighting:

• 0.0 / 1.0: 79.1 / 75.0 / 63.9
• 0.5 / 0.5: 81.0 / 76.8 / 66.2
• 0.3 / 0.7: 81.3 / 77.3 / 66.5

This shows that segmentation reward is more important than format reward, but that some format regularization improves results. The GRPO group-size study over $H_k$6 reports that larger group size improves performance while the total sampled responses to convergence remain similar (Lu et al., 21 May 2026).

Within the broader literature, the method differs from latent-query alignment by not decoding opaque MLLM hidden states directly into mask queries. It differs from textual-localization pipelines by not relying on post-hoc coordinates or prompt-like localization tokens as the main grounding mechanism. Instead, it inserts an explicit concept bottleneck: $H_k$7 That sequence is the paper’s most concise description of SegCompass as a model class.

The reported limitations are partly explicit and partly structural. The paper notes or implies dependence on CoT quality, the need for separate SAE pretraining for each backbone, substantial engineering complexity from joint RL and segmentation training, and nontrivial compute overhead involving FSDP, vLLM sampling, rollout workers, and frozen reference models. It also notes that interpretability remains partial rather than total, because the codebook aggregation and Transformer concept consolidation are still learned neural modules rather than symbolic operators. In addition, the provided material does not include extensive sensitivity studies over SAE latent dimension, sparsity coefficient, or codebook architecture (Lu et al., 21 May 2026).

Taken together, these results position SegCompass as an interpretable reasoning-segmentation architecture whose distinctive contribution is not merely higher benchmark scores, but a specific alignment design: sparse concepts as the interface between multimodal reasoning and pixel-level grounding. The paper’s own evidence suggests that the quality of this interface is tightly coupled to segmentation quality, which is the central empirical basis for its claim that enhanced interpretability and enhanced performance coincide rather than compete (Lu et al., 21 May 2026).