Text4Seg: Language-Centric Segmentation

• Text4Seg is a family of segmentation paradigms that uses linguistic structure as a key signal for supervision, conditioning, or direct mask generation across diverse domains.
• It includes supervised document segmentation with hierarchical models and transformer methods, enabling precise sentence-boundary detection and effective content partitioning.
• Text4Seg extends to vision and medical applications through weakly supervised, interactive, and diffusion-based approaches that deliver competitive performance on various benchmarks.

Text4Seg denotes a set of segmentation paradigms in which text is central to the segmentation process, but the term is used in more than one sense across the literature. In document NLP, it refers to text segmentation as sentence-level boundary prediction over contiguous topical units, often trained from large automatically derived corpora (Koshorek et al., 2018). In computer vision, it denotes text-guided segmentation, including weakly supervised referring expression segmentation, open-vocabulary instance segmentation with text and clicks, and decoder-free multimodal LLM generation of masks as text (Strudel et al., 2022). In medical imaging and diffusion-based systems, the same label is attached to report-guided or prompt-guided segmentation frameworks that use text during training, inference, or both (Ye et al., 2024).

1. Terminological scope and major formulations

In the supplied literature, “Text4Seg” is not a single architecture. It is used for supervised document segmentation, for text-guided image segmentation, for weakly supervised text instance segmentation, and for text-as-mask generation inside MLLMs. The common denominator is that linguistic structure is treated as segmentation signal: either the target is text itself, the supervision is derived from textual structure, the segmentation is conditioned on language, or the mask is serialized as language.

Usage	Canonical input/output	Representative source
Supervised document text segmentation	$x=(s_1,\dots,s_n)\rightarrow y=(y_1,\dots,y_{n-1})$ boundary labels	(Koshorek et al., 2018)
Weakly supervised or interactive text-guided image segmentation	$(I,t)$ or $(I,t,p)\rightarrow M$	(Strudel et al., 2022, Warner et al., 2023)
Text-as-mask MLLM segmentation	image and prompt $\rightarrow$ semantic descriptor sequence $\rightarrow$ mask	(Lan et al., 2024, Lan et al., 8 Sep 2025)
Medical and volumetric language-guided segmentation	X-ray or 3D volume with report or prompt guidance $\rightarrow$ mask	(Ye et al., 2024, Ma, 16 Apr 2025)

This multiplicity has methodological consequences. In some papers, “text” means the document being partitioned; in others it means a prompt, a report, a referring expression, or a generated semantic descriptor. A plausible implication is that the term is best treated as a family resemblance rather than a uniquely defined benchmark or model name.

2. Supervised document segmentation and the NLP lineage

A canonical document-segmentation formulation treats a document as a sequence of sentences $x=(s_1,\dots,s_n)$ and predicts binary boundary labels $y=(y_1,\dots,y_{n-1})$, where $y_i\in\{0,1\}$ indicates whether $s_i$ ends a segment. The supervised objective is sentence-wise cross-entropy,

$(I,t)$0

with greedy inference that starts a new segment whenever $(I,t)$1, where $(I,t)$2 is tuned on the development set. The associated Wiki-727k resource contains 727,746 English Wikipedia documents with hierarchical segmentations derived from the Table of Contents, split 80% train, 10% dev, and 10% test; for top-level segmentation it reports 3.48 $(I,t)$3 2.23 segments per document and segment length 13.6 $(I,t)$4 20.3 sentences (Koshorek et al., 2018).

The model reported for this setting is hierarchical: a two-layer bidirectional LSTM over words produces sentence embeddings via max-pooling, and a second two-layer bidirectional LSTM over sentence embeddings predicts boundary probabilities through a fully connected layer and softmax. Word embeddings are initialized with pre-trained GoogleNews word2vec. On $(I,t)$5, the model reports 22.13 on Wiki-727k test and 18.24 on Wiki-50, compared with a random baseline of 53.09 and 52.65 respectively; human performance on Wiki-50 is 14.97. Average CPU runtime is 1.6 s per document, versus 23.6 s for GraphSeg, and the model runtime is described as linear in the number of words and sentences (Koshorek et al., 2018).

This supervised framing sits between older unsupervised embedding-based segmentation and newer transformer segmentation. “Text Segmentation based on Semantic Word Embeddings” introduced a general optimization framework, contrasted greedy, exact dynamic programming, and iterative refinement, and reported that Content Vector Segmentation achieved $(I,t)$6 on the Choi 3–11 set under exact optimization, with iterative refinement approaching dynamic-programming performance at lower cost (Alemi et al., 2015). At the transformer end, CrossFormer formulates segmentation as sentence-level labeling with a Cross-Segment Fusion Module that computes $(I,t)$7, pools a global semantic vector $(I,t)$8, and fuses it with each $(I,t)$9 embedding before classification. It reports boundary F1 78.88 on WIKI-727k with Longformer-Large and 78.31 on WIKI-zh with RoBERTa-Base, and it is also used as a semantic chunker for RAG (Ni et al., 31 Mar 2025).

3. Referring expressions, clicks, and weak supervision in 2D vision

In vision, one major Text4Seg line treats segmentation as grounding language in image space. TSEG formulates weakly supervised segmentation of referring expressions from image-level supervision only. It computes a patch-text similarity matrix

$(I,t,p)\rightarrow M$0

then replaces single-label patch assignment with Multi-label Patch Assignment,

$(I,t,p)\rightarrow M$1

combined with Global Weighted Pooling and a size penalty. On PhraseCut validation, TSEG reports 28.77 mIoU and 30.12 with CRF, versus 21.12 for SPA; on RefCOCO it reports 25.44 and 25.95 with CRF; on zero-shot Pascal VOC it reports 48.5 mIoU, or 50.3 after label engineering for the person class (Strudel et al., 2022).

A second line augments language with interaction. “Text and Click inputs for unambiguous open vocabulary instance segmentation” defines the task as $(I,t,p)\rightarrow M$2, where $(I,t,p)\rightarrow M$3 is the image, $(I,t,p)\rightarrow M$4 is a text prompt such as “person” or “tie,” and $(I,t,p)\rightarrow M$5 is a single foreground click inside the target instance. The method uses CLIP and MaskCLIP to produce a dense saliency map $(I,t,p)\rightarrow M$6, then refines that saliency with a class-agnostic mask head conditioned on the click. On COCO val, when trained on VOC classes, it reports unseen mIoU 33.45 versus 30.33 for RITM; on RefCOCO it reports 72.89 mIoU with class name plus 3 clicks and 68.07 with class name plus 1 click. The appendix explicitly states that detailed backbone, decoder, click encoding, and loss functions are not provided (Warner et al., 2023).

A third variant uses recognition attention as weak supervision for text instances in images. “Weakly-Supervised Text Instance Segmentation” treats recognition attention maps $(I,t,p)\rightarrow M$7 as rough per-instance heatmaps, refines them with Text Adaptive Refinement into pseudo labels $(I,t,p)\rightarrow M$8, and trains a segmentation head with BCE against those pseudo labels, while adding mask-augmented contrastive learning. The total objective is

$(I,t,p)\rightarrow M$9

with $\rightarrow$0 and $\rightarrow$1. It reports 48.09 fIoU on ICDAR13-FST and 59.75 on TextSeg under weak supervision, described as improvements of 18.95% and 17.80% over previous weakly supervised methods; with larger-scale training it reports 60.55 on ICDAR13-FST and 71.24 on TextSeg. TAR is reported at 2.6 ms with 71.24 fIoU, compared with 5.6 ms and 46.21 for PRMs and 89.4 ms and 64.95 for FC-CRFs (Zu et al., 2023).

4. Text-as-mask generation in multimodal LLMs

The 2024 Text4Seg paper redefines segmentation as autoregressive text generation. An image is tessellated into a $\rightarrow$2 grid of patches, each patch is assigned a semantic descriptor, and the descriptor grid is serialized between <seg> and </seg> with row delimiters \n and cell delimiters |. The training objective is standard next-token negative log-likelihood,

$\rightarrow$3

where $\rightarrow$4 is the image and $\rightarrow$5 is optional instruction text. No segmentation decoder or auxiliary loss is added, and the paper reports LoRA fine-tuning on DeepseekVL-1.3B/7B, LLaVA-1.5-7B/13B, Qwen-VL-7B, and InternVL2-8B (Lan et al., 2024).

The key compression device is Row-wise Run-Length Encoding. Instead of emitting full descriptors for all $\rightarrow$6 cells, Text4Seg compresses repeated descriptors within each row. On RefCOCO, the paper reports that R-RLE reduces the length of semantic descriptors by 74% and accelerates inference by $\rightarrow$7, with average token length changing from approximately 583 to approximately 154. It also reports average RES cIoU of 75.4 for InternVL2-8B and 76.2 for LLaVA-1.5-13B, average GRES scores of 71.1 and 71.5 respectively, REC [email protected] up to 87.7, and open-vocabulary segmentation mIoU of 16.5 on ADE-150, 52.5 on PC-59, and 76.5 on PAS-20 (Lan et al., 2024).

Text4Seg++ extends this generative formulation by moving from image-wise semantic descriptors to box-wise semantic descriptors and semantic bricks. A region is represented as <ref> … </ref> <box> [[x1 y1 x2 y2]] </box> <seg> … </seg>, and the mask inside a $\rightarrow$8 box is raster-scanned into foreground and background run tokens $\rightarrow$9 and $\rightarrow$0. The next-token loss becomes a next-brick prediction loss,

$\rightarrow$1

and average token lengths on RefCOCO are reported as 767.6 for image-wise descriptors scaled to $\rightarrow$2, 283.0 for box-wise descriptors without bricks, and 150.4 for box-wise descriptors with bricks. Text4Seg++ reports 79.3 average cIoU on RefCOCO with a 7B model, 91.1 average REC [email protected], 63.8 on MUSE reasoning segmentation, 70.8 on RRSIS-D, and 70.1 on EarthReason, all without task-specific fine-tuning (Lan et al., 8 Sep 2025).

A recurrent misconception is that decoder-free generation necessarily implies weaker masks. The reported evidence is more specific: Text4Seg is competitive or state-of-the-art on several referring and comprehension benchmarks, but it also states that open-vocabulary segmentation remains below specialized OVS models and that patch granularity can hurt small or thin objects (Lan et al., 2024).

5. Medical and 3D extensions

In chest X-rays, SGSeg uses language during training but removes the requirement for external text at inference. Its language-guided U-Net combines a ConvNeXt-T image encoder, a BERT text encoder pretrained on MIMIC, and a GuidedDecoder with cross-modal attention; a Localization-Enhanced Report Generation module based on RT-DETR, a location-based attention aggregator, and weakly supervised pseudo-label extraction generates concise location reports internally at test time. On QaTa-COV19, which contains 9,258 chest X-ray images with the official split 5,716 train, 1,429 val, and 2,113 test, SGSeg reports Acc 0.971, Dice 0.874, and Jaccard 0.778, versus 0.950, 0.832, and 0.724 for Swin U-Net and 0.975, 0.898, and 0.815 for LanGuideMedSeg. The ablation “Self-Guidance (weakly-supervised LERG)” reports 0.971, 0.874, and 0.778, while “Full Text” reports 0.973, 0.890, and 0.797 (Ye et al., 2024).

TextDiffSeg generalizes text guidance to 3D medical image segmentation through latent diffusion. It uses a 3D image encoder, a BioBERT text encoder pretrained on MIMIC-III, a 3D label encoder that embeds masks into a continuous latent $\rightarrow$3, cross-modal attention that fuses image and text features into $\rightarrow$4, and a conditional denoiser trained with

$\rightarrow$5

where $\rightarrow$6 and $\rightarrow$7. The model operates on 96×96×96 patches, uses DDIM with 10 steps at test time, and reports DICE/NSD of 88.31/91.45 for kidney tumor, 71.88/89.91 for pancreas tumor, 84.47/93.79 for liver tumor, and 75.62/86.16 for colon cancer, outperforming Diff-UNet on all four tasks (Ma, 16 Apr 2025).

These medical variants make different deployment choices. SGSeg emphasizes text-free inference after multimodal training, whereas TextDiffSeg preserves explicit prompting at test time. This suggests two distinct design patterns inside medical Text4Seg: self-generated textual guidance and persistent prompt conditioning.

6. Diffusion-attention segmentation, vector-path decoding, and recurring limitations

Later work pushes Text4Seg in two orthogonal directions. Seg4Diff extracts open-vocabulary masks directly from joint self-attention in multimodal diffusion transformers. In SD3, the paper identifies the 9th MM-DiT block as a semantic grounding expert layer whose image-to-text attention maps align text tokens with coherent regions. Zero-shot Seg4Diff reports VOC20 89.2, Object 62.0, PC59 49.0, ADE 34.2, and City 26.5, and lightweight MAGNET fine-tuning on 10k COCO images raises these to 89.8, 62.9, 51.2, 35.2, and 26.0 respectively. The same study also reports unsupervised segmentation via <pad> tokens and modest gains in CLIPScore and T2I-CompBench++ after mask-aware fine-tuning (Kim et al., 22 Sep 2025).

Moondream Segmentation instead performs referring image segmentation by autoregressively decoding a vector path and a bounding box, rasterizing that path into a coarse mask, and iteratively refining it for $\rightarrow$8 steps with a SAM-style refiner conditioned on frozen Moondream 3 vision features. Its reinforcement-learning stage uses a curriculum reward that switches from box IoU to a Tversky index and then to Boundary IoU. The model reports cIoU of 80.2 on RefCOCO val, 72.5 on RefCOCO+ val, 73.7 on RefCOCOg val, 62.6 mIoU on LVIS val, and on the cleaned RefCOCO-M split 87.6 cIoU and 85.4 [email protected]. The paper explicitly states that it outperforms Text4Seg on the RefCOCO-family comparisons it reports (Reid, 3 Apr 2026).

Across these variants, several limitations recur. Supervised document segmentation depends on structural weak labels from Wikipedia headings and is sensitive to thresholding and domain shift (Koshorek et al., 2018). Weakly supervised referring segmentation remains substantially below fully supervised upper bounds, and attribute-rich or subtle expressions remain difficult (Strudel et al., 2022). Semantic-descriptor generation is constrained by patch granularity and sequence length even after R-RLE (Lan et al., 2024). SGSeg depends on report style and pseudo-label quality (Ye et al., 2024). TextDiffSeg assumes meaningful prompts and may still miss very long-range context under patch-based training (Ma, 16 Apr 2025). Seg4Diff notes failures on small or thin structures, ambiguous class names, occlusions, and clutter (Kim et al., 22 Sep 2025). Moondream notes ambiguity in polygon tokenizations, difficulty with disjoint or extremely thin structures, and evaluation noise in original polygon annotations, which motivated RefCOCO-M (Reid, 3 Apr 2026).

Taken together, the literature shows that Text4Seg has evolved from boundary prediction over sentence sequences to open-vocabulary, decoder-free, interactive, weakly supervised, diffusion-based, and medical segmentation systems. What remains stable is the central methodological claim: segmentation quality can be improved by treating language not as an auxiliary annotation channel but as a primary representational substrate, whether for supervision, conditioning, or direct mask generation.