--- title: 'Text4Seg++: Segmentation as Text Generation' url: https://www.emergentmind.com/topics/text4seg-dfe8796a-eae6-4086-a9ec-1b0898c8ed18 type: topic --- # Text4Seg++: Segmentation as Text Generation Text4Seg++ is a multimodal large language model (MLLM) segmentation framework that reformulates image segmentation as structured text generation. It extends Text4Seg’s text-as-mask paradigm—where segmentation masks are serialized as semantic descriptors—by replacing global image-wise patch descriptions with box-wise semantic descriptors and structured mask tokens called semantic bricks, so that segmentation is learned as a next-brick prediction task under the standard autoregressive language modeling objective rather than by attaching a segmentation decoder. In the reported formulation, the framework remains compatible with existing MLLM backbones, supports referring expression segmentation, generalized referring expression segmentation, reasoning segmentation, grounding, and remote sensing segmentation, and is trained as one unified model without task-specific fine-tuning [2509.06321] [2410.09855]. ## 1. Conceptual basis and research context Text4Seg++ is motivated by a structural mismatch between autoregressive MLLMs and dense prediction. MLLMs are optimized for next-token generation over text, whereas segmentation is conventionally posed as dense spatial prediction. Prior MLLM segmentation systems often bridge this gap by adding an external mask decoder conditioned on a special token such as `` plus visual features, or by generating polygon coordinates directly. Text4Seg++ is defined against both alternatives: it does not add a task-specific segmentation decoder, and it does not use polygon coordinates as the primary representation [2509.06321]. The predecessor framework, Text4Seg, established the underlying text-as-mask view. Its central claim was that dense masks can be represented as semantically meaningful text tokens and optimized with the original autoregressive training pipeline of an MLLM. Text4Seg++ preserves that claim, but changes the representation so that generation is more compact and spatially targeted. The progression is from image-wise semantic descriptors over a full patch grid to region-localized box-wise semantic descriptors plus a symbolic mask vocabulary [2410.09855]. Within the broader literature, the phrase “text-driven segmentation” is used in several distinct senses. In scene-text vision, it can denote text-specific mask refinement or recognition-derived pseudo-mask generation for text instances [2011.14021] [2303.10848]. In medical imaging, it can denote text-conditioned feature enhancement for semi-supervised 3D segmentation [2507.12382]. Text4Seg++ belongs to a different line: decoder-free image segmentation inside MLLMs via generative language modeling [2509.06321]. ## 2. Text4Seg foundations: image-wise semantic descriptors The original Text4Seg represents an image by a \(16 \times 16\) grid of semantic descriptors, giving \(16 \times 16 = 256\) non-overlapping patches. The ground-truth segmentation mask is resized to \(16 \times 16\), flattened in row-major order, and each patch index is replaced by a text label, phrase, or referring expression. A descriptor may be a class name such as “sky” or “sand,” a phrase such as “brown dog,” or a longer expression such as “a dog in the left” [2410.09855]. Formally, if the resized mask is \[ M \in \mathcal{V}^{H \times W}, \] with default \(H=W=16\), and each patch label is \(m_{i,j} \in \mathcal{V}\), then the serialized descriptor sequence is \[ S = [m_{1,1}, m_{1,2}, \dots, m_{1,W}, m_{2,1}, \dots, m_{H,W}]. \] Reconstruction reverses the process: the generated sequence is parsed, reshaped into a \(16 \times 16\) grid, converted to a coarse mask, and upsampled back to image resolution. For a target label \(c\), the reconstructed coarse mask is \[ \hat{Y}_{i,j}^{(c)} = \mathbf{1}[m_{i,j} = c]. \] This formulation makes the segmentation mask itself a language sequence rather than an auxiliary embedding. The training sample format uses standard instruction tuning, with segmentation content bounded by `` and ``. The training objective is the standard autoregressive language modeling loss \(\mathcal{L}_{txt}\), so segmentation supervision is absorbed into ordinary next-token prediction rather than handled by a segmentation-specific decoder loss [2410.09855]. A practical bottleneck appears immediately: sequence length. On refCOCO, the full \(16 \times 16\) descriptor sequence averages about 583 tokens and requires roughly 19 seconds on a V100 GPU for one round of referring expression segmentation. To reduce redundancy, Text4Seg introduces Row-wise Run-Length Encoding (R-RLE), which compresses repeated descriptors within each row and preserves row boundaries with newline separators. On refCOCO, R-RLE reduces average token length from 583 to 154, a 74% reduction, and accelerates inference by about \(3\times\), without compromising performance; by contrast, image-wise RLE reduces tokens but causes notable cIoU drops, including 74.2 to 70.4 on refCOCO and 68.0 to 64.7 on refCOCO+ [2410.09855]. The original Text4Seg therefore established two premises that Text4Seg++ retains: first, segmentation can be treated as native language generation; second, the main systems bottleneck is not only semantics but the combinatorics of output length. This suggests that improvements in representation compactness are as important as improvements in backbone scale. ## 3. Box-wise semantic descriptors and semantic bricks Text4Seg++ replaces global image-wise semantic descriptors with box-wise semantic descriptors (B-SD). Each segmented instance is represented by a structured triplet containing a referent, a bounding box, and a localized mask serialization: ```text text labels [[x1 y1 x2 y2]] semantic descriptors ``` The paper explicitly defines the special tokens ``, ``, ``, ``, ``, and ``. The `` field may contain a category label, a referring expression, or an abstract identifier such as `roi0`, `roi1`, ... for reasoning tasks where no explicit class label is available. The `` field localizes the region of interest, and each coordinate is quantized into 64 discrete bins. The `` field then encodes the mask only inside that box, rather than over the full image [2509.06321]. This representation changes the geometry of the generation problem. Instead of describing every patch of the image, including extensive background regions, the model first predicts where the relevant region is and then serializes only the binary occupancy inside that region. The paper characterizes this as a shift from a global dense patch-wise text description to a localized region-wise structured description [2509.06321]. To make the `` field compact, Text4Seg++ introduces semantic bricks, an extended vocabulary of 126 structured mask tokens: \[ \{\texttt{fg1}, \texttt{fg2}, \ldots, \texttt{fg63}\} \cup \{\texttt{bg1}, \texttt{bg2}, \ldots, \texttt{bg63}\}. \] These encode runs of foreground or background cells in raster order on a binary mask canvas. The paper describes this as “symbolic plotting on a \(64 \times 64\) canvas.” Intuitively, `fgk` denotes a run of \(k\) foreground cells and `bgk` denotes a run of \(k\) background cells. Because each run becomes a single token, repeated textual strings such as `others*16` are replaced by atomic structured vocabulary items [2509.06321]. The token-efficiency consequences are central. On RefCOCO, the paper reports the following average token lengths: | Representation | Average tokens | |---|---:| | \(64 \times 64\) I-SD | 767.6 | | \(64 \times 64\) B-SD without bricks | 283.0 | | \(64 \times 64\) B-SD with bricks | 150.4 | These numbers are notable because Text4Seg++ attains higher mask resolution than the original \(16 \times 16\) Text4Seg formulation while still shortening the generated sequence. The paper also reports that increasing B-SD resolution from \(32^2\) to \(64^2\) to \(80^2\) improves segmentation quality, and adopts \(64 \times 64\) as the best tradeoff [2509.06321]. ## 4. Next-brick prediction, architecture, and training regime Text4Seg++ formulates segmentation as next-brick prediction. The response sequence contains referent tokens, box coordinates, and semantic brick tokens, and training uses the pure generative language modeling loss: \[ \mathcal{L}_{txt} = - \sum_{t=1}^{T} \log p(y_t \mid I, x, y_{} \mid I,q) \, p(\text{ref text} \mid I,q,\cdot) \, p(\texttt{} , x_1,y_1,x_2,y_2 \mid I,q,\cdot) \, p(\texttt{}, b_1,\ldots,b_N \mid I,q,\cdot), \] where \(b_i\) are semantic bricks. In inference, the model autoregressively generates the referent, then the box, then the brick sequence; the brick sequence is expanded in raster order into a binary \(64 \times 64\) region mask, which is then mapped back to image coordinates using the predicted box [2509.06321]. Architecturally, the framework leaves the underlying MLLM unchanged. The paper states compatibility with Qwen2-VL, Deepseek-VL2, and InternVL3. The actual Text4Seg++ experiments are mainly conducted with Qwen2-VL-7B, and larger-scale results also use Qwen2.5-14B. The standard MLLM decomposition remains intact: vision encoder, language model, and modality connector. No extra segmentation decoder is used. The only optional external module is SAMRefiner, employed as post-processing in some comparisons rather than as a jointly trained component [2509.06321]. Training uses LoRA for post-training or fine-tuning. Reported settings for Text4Seg++ are 8 NVIDIA H100 GPUs, global batch size 128, AdamW, initial learning rate \(2\times 10^{-4}\), warm-up ratio 0.03, linear decay afterward, weight decay 0, gradient clipping 1.0, LoRA rank 128, and ZeRO-1 memory optimization. The default image input resolution is a minimum pixel count corresponding to about \(784^2\) while preserving native aspect ratio [2509.06321]. The supervision mixture is intentionally broad. Training sources include COCO panoptic segmentation, refCOCO, refCOCO+, refCOCOg, grefCOCO, Pix2Cap, ReasonSeg, MUSE, RRSIS_D, Earthreason, and LLaVA-665k visual instruction data. The paper emphasizes that Text4Seg++ is trained as one unified model for 50k steps and then evaluated on multiple downstream tasks without task-specific fine-tuning [2509.06321]. ## 5. Benchmarks, reported performance, and empirical profile Text4Seg++ is evaluated on natural-image referring segmentation and grounding benchmarks, generalized referring segmentation, reasoning segmentation, and remote sensing. The reported results position the framework as strong at both segmentation and localization, especially at 7B and 14B scales [2509.06321]. | Setting | Text4Seg++ result | Comparator from reported table | |---|---:|---| | RefCOCO family RES, 7B | 79.3 average cIoU | SAM4MLLM 76.4 | | RefCOCO family RES, 14B | 79.9 average cIoU | Text4Seg 76.2 | | RefCOCO grounding, 7B | 91.1 average Acc@0.5 | Qwen2-VL 87.9 | | RefCOCO grounding, 14B | 92.0 average Acc@0.5 | InternVL3-14B 89.1 | | MUSE reasoning segmentation | 63.8 average | POPEN 52.5 | | EarthReason | 70.1 average | SegEarth-R1 68.0 | Several task-specific nuances are also reported. On grefCOCO, Text4Seg++ obtains 70.0 average at 7B scale, while Text4Seg reaches 71.1 when task-specifically fine-tuned with refiner; the paper therefore describes Text4Seg++ as highly competitive in generalized referring expression segmentation, but not uniformly dominant there. On ReasonSeg, Text4Seg++ reports 54.5 average, exceeding LISA’s 50.9 and SegLLM’s 53.1, but remaining below Seg-Zero’s 56.2. On RRSIS-D, Text4Seg++ reaches 70.8 average, below RMSIN’s 72.4 and SegEarth-R1’s 74.4, while on EarthReason it reaches 70.1 average and is reported as state of the art in that benchmark [2509.06321]. The original Text4Seg remains relevant for open-vocabulary segmentation, which is reported mainly for the earlier representation rather than for Text4Seg++. Text4Seg achieves 16.5 mIoU on ADE20K-150, 52.5 on PASCAL Context-59, and 76.5 on PASCAL VOC-20. The paper describes this as competitive for an MLLM-based method but below specialized segmentation models such as SAN and OVSeg [2410.09855]. Ablation studies clarify where the gains come from. Increasing B-SD resolution improves segmentation quality. Semantic bricks produce only small direct accuracy changes on \(32 \times 32\) B-SD—79.7 vs 79.7 on refCOCO, 75.1 vs 74.9 on refCOCO+, and 78.3 vs 78.1 on refCOCOg—but are still important because they dramatically shorten sequences and enable higher-resolution B-SD. Performance also improves with higher input image resolution, and the best practical setting is reported as about \(784^2\) minimum pixels while preserving aspect ratio [2509.06321]. ## 6. Interpretation, misconceptions, and limitations A frequent misconception is to treat Text4Seg++ as a decoder-assisted segmentation model whose textual output is merely an intermediate prompt. In the reported formulation, that is incorrect: the segmentation content is generated directly as language, and the core method is explicitly decoder-free. A second misconception is to equate it with polygon serialization. The paper positions semantic descriptors and semantic bricks as a semantically aligned alternative to long coordinate sequences, arguing that polygon outputs are a poor fit for LLMs because the model must infer geometry from numeric strings rather than from structured symbolic tokens [2509.06321]. The method’s practical strengths are tied to this representation choice. It predicts tokens rather than dense tensors from a special head, combines localization and mask generation in one serialization, and uses box prediction to narrow the spatial search space before mask generation. This suggests that a substantial part of its performance comes not only from backbone capacity but from aligning the output format with the MLLM’s native causal decoding regime. The limitations are also explicit. Text4Seg++ still depends on sufficient image input resolution, and performance improves with higher-resolution visual inputs. In generalized referring segmentation, task-specific fine-tuning with the earlier Text4Seg plus refiner can still outperform the unified Text4Seg++. In reasoning segmentation, specialized methods such as Seg-Zero can still exceed it. Open-vocabulary semantic segmentation remains below specialized dense vision models, a limitation already visible in Text4Seg’s results [2509.06321] [2410.09855]. In a wider segmentation taxonomy, Text4Seg++ occupies a distinct niche. It is not a text-instance segmentation system specialized for extracting text pixels from images, as in TextSeg/TexRNet [2011.14021]. It is not a weakly supervised recognition-to-segmentation pipeline driven by text-recognition attention, as in weakly supervised text instance segmentation [2303.10848]. It is not a 3D semi-supervised medical segmentation framework using category text to enhance voxel features, as in Text-SemiSeg [2507.12382]. Rather, it is a general MLLM-native segmentation framework whose central claim is that dense prediction can be expressed as structured language without architectural surgery. Taken together, the Text4Seg and Text4Seg++ line establishes a specific program for segmentation in MLLMs: represent masks as semantically meaningful sequences, compress those sequences aggressively enough for practical decoding, and let the language model solve segmentation with the same next-token machinery it already uses for dialogue and reasoning. The move from image-wise semantic descriptors to box-wise semantic descriptors and semantic bricks is the key refinement that makes this program substantially more granular and efficient [2509.06321] Source: https://www.emergentmind.com/topics/text4seg-dfe8796a-eae6-4086-a9ec-1b0898c8ed18