Text4Seg++: Segmentation as Text Generation

• The paper introduces a framework that reformulates image segmentation as structured text generation using box-wise semantic descriptors and semantic bricks.
• It replaces global patch descriptors with localized region prediction, significantly reducing token length while preserving spatial accuracy.
• The unified model, trained under an autoregressive language modeling objective, achieves competitive performance on various segmentation benchmarks.

Text4Seg++ is a multimodal LLM (MLLM) segmentation framework that reformulates image segmentation as structured text generation. It extends Text4Seg’s text-as-mask paradigm—where segmentation masks are serialized as semantic descriptors—by replacing global image-wise patch descriptions with box-wise semantic descriptors and structured mask tokens called semantic bricks, so that segmentation is learned as a next-brick prediction task under the standard autoregressive language modeling objective rather than by attaching a segmentation decoder. In the reported formulation, the framework remains compatible with existing MLLM backbones, supports referring expression segmentation, generalized referring expression segmentation, reasoning segmentation, grounding, and remote sensing segmentation, and is trained as one unified model without task-specific fine-tuning (Lan et al., 8 Sep 2025, Lan et al., 2024).

1. Conceptual basis and research context

Text4Seg++ is motivated by a structural mismatch between autoregressive MLLMs and dense prediction. MLLMs are optimized for next-token generation over text, whereas segmentation is conventionally posed as dense spatial prediction. Prior MLLM segmentation systems often bridge this gap by adding an external mask decoder conditioned on a special token such as <seg> plus visual features, or by generating polygon coordinates directly. Text4Seg++ is defined against both alternatives: it does not add a task-specific segmentation decoder, and it does not use polygon coordinates as the primary representation (Lan et al., 8 Sep 2025).

The predecessor framework, Text4Seg, established the underlying text-as-mask view. Its central claim was that dense masks can be represented as semantically meaningful text tokens and optimized with the original autoregressive training pipeline of an MLLM. Text4Seg++ preserves that claim, but changes the representation so that generation is more compact and spatially targeted. The progression is from image-wise semantic descriptors over a full patch grid to region-localized box-wise semantic descriptors plus a symbolic mask vocabulary (Lan et al., 2024).

Within the broader literature, the phrase “text-driven segmentation” is used in several distinct senses. In scene-text vision, it can denote text-specific mask refinement or recognition-derived pseudo-mask generation for text instances (Xu et al., 2020, Zu et al., 2023). In medical imaging, it can denote text-conditioned feature enhancement for semi-supervised 3D segmentation (Huang et al., 16 Jul 2025). Text4Seg++ belongs to a different line: decoder-free image segmentation inside MLLMs via generative language modeling (Lan et al., 8 Sep 2025).

2. Text4Seg foundations: image-wise semantic descriptors

The original Text4Seg represents an image by a $16 \times 16$ grid of semantic descriptors, giving $16 \times 16 = 256$ non-overlapping patches. The ground-truth segmentation mask is resized to $16 \times 16$, flattened in row-major order, and each patch index is replaced by a text label, phrase, or referring expression. A descriptor may be a class name such as “sky” or “sand,” a phrase such as “brown dog,” or a longer expression such as “a dog in the left” (Lan et al., 2024).

Formally, if the resized mask is

$M \in \mathcal{V}^{H \times W},$

with default $H=W=16$, and each patch label is $m_{i,j} \in \mathcal{V}$, then the serialized descriptor sequence is

$$S = [m_{1,1}, m_{1,2}, \dots, m_{1,W}, m_{2,1}, \dots, m_{H,W}].$$

Reconstruction reverses the process: the generated sequence is parsed, reshaped into a $16 \times 16$ grid, converted to a coarse mask, and upsampled back to image resolution. For a target label $c$, the reconstructed coarse mask is

$\hat{Y}_{i,j}^{(c)} = \mathbf{1}[m_{i,j} = c].$

This formulation makes the segmentation mask itself a language sequence rather than an auxiliary embedding. The training sample format uses standard instruction tuning, with segmentation content bounded by <seg> and </seg>. The training objective is the standard autoregressive language modeling loss $16 \times 16 = 256$0, so segmentation supervision is absorbed into ordinary next-token prediction rather than handled by a segmentation-specific decoder loss (Lan et al., 2024).

A practical bottleneck appears immediately: sequence length. On refCOCO, the full $16 \times 16 = 256$1 descriptor sequence averages about 583 tokens and requires roughly 19 seconds on a V100 GPU for one round of referring expression segmentation. To reduce redundancy, Text4Seg introduces Row-wise Run-Length Encoding (R-RLE), which compresses repeated descriptors within each row and preserves row boundaries with newline separators. On refCOCO, R-RLE reduces average token length from 583 to 154, a 74% reduction, and accelerates inference by about $16 \times 16 = 256$2, without compromising performance; by contrast, image-wise RLE reduces tokens but causes notable cIoU drops, including 74.2 to 70.4 on refCOCO and 68.0 to 64.7 on refCOCO+ (Lan et al., 2024).

The original Text4Seg therefore established two premises that Text4Seg++ retains: first, segmentation can be treated as native language generation; second, the main systems bottleneck is not only semantics but the combinatorics of output length. This suggests that improvements in representation compactness are as important as improvements in backbone scale.

3. Box-wise semantic descriptors and semantic bricks

Text4Seg++ replaces global image-wise semantic descriptors with box-wise semantic descriptors (B-SD). Each segmented instance is represented by a structured triplet containing a referent, a bounding box, and a localized mask serialization:

$M \in \mathcal{V}^{H \times W},$6

The paper explicitly defines the special tokens <ref>, </ref>, <box>, </box>, <seg>, and </seg>. The <ref> field may contain a category label, a referring expression, or an abstract identifier such as roi0, roi1, ... for reasoning tasks where no explicit class label is available. The <box> field localizes the region of interest, and each coordinate is quantized into 64 discrete bins. The <seg> field then encodes the mask only inside that box, rather than over the full image (Lan et al., 8 Sep 2025).

This representation changes the geometry of the generation problem. Instead of describing every patch of the image, including extensive background regions, the model first predicts where the relevant region is and then serializes only the binary occupancy inside that region. The paper characterizes this as a shift from a global dense patch-wise text description to a localized region-wise structured description (Lan et al., 8 Sep 2025).

To make the <seg> field compact, Text4Seg++ introduces semantic bricks, an extended vocabulary of 126 structured mask tokens: $16 \times 16 = 256$3 These encode runs of foreground or background cells in raster order on a binary mask canvas. The paper describes this as “symbolic plotting on a $16 \times 16 = 256$4 canvas.” Intuitively, fgk denotes a run of $16 \times 16 = 256$5 foreground cells and bgk denotes a run of $16 \times 16 = 256$6 background cells. Because each run becomes a single token, repeated textual strings such as others*16 are replaced by atomic structured vocabulary items (Lan et al., 8 Sep 2025).

The token-efficiency consequences are central. On RefCOCO, the paper reports the following average token lengths:

Representation	Average tokens
$16 \times 16 = 256$7 I-SD	767.6
$16 \times 16 = 256$8 B-SD without bricks	283.0
$16 \times 16 = 256$9 B-SD with bricks	150.4

These numbers are notable because Text4Seg++ attains higher mask resolution than the original $16 \times 16$0 Text4Seg formulation while still shortening the generated sequence. The paper also reports that increasing B-SD resolution from $16 \times 16$1 to $16 \times 16$2 to $16 \times 16$3 improves segmentation quality, and adopts $16 \times 16$4 as the best tradeoff (Lan et al., 8 Sep 2025).

4. Next-brick prediction, architecture, and training regime

Text4Seg++ formulates segmentation as next-brick prediction. The response sequence contains referent tokens, box coordinates, and semantic brick tokens, and training uses the pure generative language modeling loss: $16 \times 16$5 where $16 \times 16$6 is the image, $16 \times 16$7 is the query, and $16 \times 16$8 are response tokens. The paper does not introduce a segmentation-specific loss beyond this causal objective (Lan et al., 8 Sep 2025).

A task-level factorization is described as

$16 \times 16$9

where $M \in \mathcal{V}^{H \times W},$0 are semantic bricks. In inference, the model autoregressively generates the referent, then the box, then the brick sequence; the brick sequence is expanded in raster order into a binary $M \in \mathcal{V}^{H \times W},$1 region mask, which is then mapped back to image coordinates using the predicted box (Lan et al., 8 Sep 2025).

Architecturally, the framework leaves the underlying MLLM unchanged. The paper states compatibility with Qwen2-VL, Deepseek-VL2, and InternVL3. The actual Text4Seg++ experiments are mainly conducted with Qwen2-VL-7B, and larger-scale results also use Qwen2.5-14B. The standard MLLM decomposition remains intact: vision encoder, LLM, and modality connector. No extra segmentation decoder is used. The only optional external module is SAMRefiner, employed as post-processing in some comparisons rather than as a jointly trained component (Lan et al., 8 Sep 2025).

Training uses LoRA for post-training or fine-tuning. Reported settings for Text4Seg++ are 8 NVIDIA H100 GPUs, global batch size 128, AdamW, initial learning rate $M \in \mathcal{V}^{H \times W},$2, warm-up ratio 0.03, linear decay afterward, weight decay 0, gradient clipping 1.0, LoRA rank 128, and ZeRO-1 memory optimization. The default image input resolution is a minimum pixel count corresponding to about $M \in \mathcal{V}^{H \times W},$3 while preserving native aspect ratio (Lan et al., 8 Sep 2025).

The supervision mixture is intentionally broad. Training sources include COCO panoptic segmentation, refCOCO, refCOCO+, refCOCOg, grefCOCO, Pix2Cap, ReasonSeg, MUSE, RRSIS_D, Earthreason, and LLaVA-665k visual instruction data. The paper emphasizes that Text4Seg++ is trained as one unified model for 50k steps and then evaluated on multiple downstream tasks without task-specific fine-tuning (Lan et al., 8 Sep 2025).

5. Benchmarks, reported performance, and empirical profile

Text4Seg++ is evaluated on natural-image referring segmentation and grounding benchmarks, generalized referring segmentation, reasoning segmentation, and remote sensing. The reported results position the framework as strong at both segmentation and localization, especially at 7B and 14B scales (Lan et al., 8 Sep 2025).

Setting	Text4Seg++ result	Comparator from reported table
RefCOCO family RES, 7B	79.3 average cIoU	SAM4MLLM 76.4
RefCOCO family RES, 14B	79.9 average cIoU	Text4Seg 76.2
RefCOCO grounding, 7B	91.1 average [email protected]	Qwen2-VL 87.9
RefCOCO grounding, 14B	92.0 average [email protected]	InternVL3-14B 89.1
MUSE reasoning segmentation	63.8 average	POPEN 52.5
EarthReason	70.1 average	SegEarth-R1 68.0

Several task-specific nuances are also reported. On grefCOCO, Text4Seg++ obtains 70.0 average at 7B scale, while Text4Seg reaches 71.1 when task-specifically fine-tuned with refiner; the paper therefore describes Text4Seg++ as highly competitive in generalized referring expression segmentation, but not uniformly dominant there. On ReasonSeg, Text4Seg++ reports 54.5 average, exceeding LISA’s 50.9 and SegLLM’s 53.1, but remaining below Seg-Zero’s 56.2. On RRSIS-D, Text4Seg++ reaches 70.8 average, below RMSIN’s 72.4 and SegEarth-R1’s 74.4, while on EarthReason it reaches 70.1 average and is reported as state of the art in that benchmark (Lan et al., 8 Sep 2025).

The original Text4Seg remains relevant for open-vocabulary segmentation, which is reported mainly for the earlier representation rather than for Text4Seg++. Text4Seg achieves 16.5 mIoU on ADE20K-150, 52.5 on PASCAL Context-59, and 76.5 on PASCAL VOC-20. The paper describes this as competitive for an MLLM-based method but below specialized segmentation models such as SAN and OVSeg (Lan et al., 2024).

Ablation studies clarify where the gains come from. Increasing B-SD resolution improves segmentation quality. Semantic bricks produce only small direct accuracy changes on $M \in \mathcal{V}^{H \times W},$4 B-SD—79.7 vs 79.7 on refCOCO, 75.1 vs 74.9 on refCOCO+, and 78.3 vs 78.1 on refCOCOg—but are still important because they dramatically shorten sequences and enable higher-resolution B-SD. Performance also improves with higher input image resolution, and the best practical setting is reported as about $M \in \mathcal{V}^{H \times W},$5 minimum pixels while preserving aspect ratio (Lan et al., 8 Sep 2025).

6. Interpretation, misconceptions, and limitations

A frequent misconception is to treat Text4Seg++ as a decoder-assisted segmentation model whose textual output is merely an intermediate prompt. In the reported formulation, that is incorrect: the segmentation content is generated directly as language, and the core method is explicitly decoder-free. A second misconception is to equate it with polygon serialization. The paper positions semantic descriptors and semantic bricks as a semantically aligned alternative to long coordinate sequences, arguing that polygon outputs are a poor fit for LLMs because the model must infer geometry from numeric strings rather than from structured symbolic tokens (Lan et al., 8 Sep 2025).

The method’s practical strengths are tied to this representation choice. It predicts tokens rather than dense tensors from a special head, combines localization and mask generation in one serialization, and uses box prediction to narrow the spatial search space before mask generation. This suggests that a substantial part of its performance comes not only from backbone capacity but from aligning the output format with the MLLM’s native causal decoding regime.

The limitations are also explicit. Text4Seg++ still depends on sufficient image input resolution, and performance improves with higher-resolution visual inputs. In generalized referring segmentation, task-specific fine-tuning with the earlier Text4Seg plus refiner can still outperform the unified Text4Seg++. In reasoning segmentation, specialized methods such as Seg-Zero can still exceed it. Open-vocabulary semantic segmentation remains below specialized dense vision models, a limitation already visible in Text4Seg’s results (Lan et al., 8 Sep 2025, Lan et al., 2024).

In a wider segmentation taxonomy, Text4Seg++ occupies a distinct niche. It is not a text-instance segmentation system specialized for extracting text pixels from images, as in TextSeg/TexRNet (Xu et al., 2020). It is not a weakly supervised recognition-to-segmentation pipeline driven by text-recognition attention, as in weakly supervised text instance segmentation (Zu et al., 2023). It is not a 3D semi-supervised medical segmentation framework using category text to enhance voxel features, as in Text-SemiSeg (Huang et al., 16 Jul 2025). Rather, it is a general MLLM-native segmentation framework whose central claim is that dense prediction can be expressed as structured language without architectural surgery.

Taken together, the Text4Seg and Text4Seg++ line establishes a specific program for segmentation in MLLMs: represent masks as semantically meaningful sequences, compress those sequences aggressively enough for practical decoding, and let the LLM solve segmentation with the same next-token machinery it already uses for dialogue and reasoning. The move from image-wise semantic descriptors to box-wise semantic descriptors and semantic bricks is the key refinement that makes this program substantially more granular and efficient (Lan et al., 8 Sep 2025)