Text-SemiSeg: Weak & Semi-Supervised Segmentation

• Text-SemiSeg is a heterogeneous research area that uses weak and semi-supervised strategies to perform annotation-efficient segmentation across document, visual, and medical modalities.
• It encompasses methods such as document boundary prediction via Wikipedia metadata, scene text segmentation using OCR attention, and text-guided 3D medical segmentation using CLIP-style prompts.
• Researchers integrate pseudo-labeling, consistency regularization, and specialized architectures to achieve competitive segmentation performance despite scarce dense annotations.

Searching arXiv for recent and foundational papers on “Text-SemiSeg” and closely related weakly/semi-supervised text segmentation.

I’m querying arXiv for “Text-SemiSeg”, “weakly supervised text segmentation”, “text instance segmentation”, and “text segmentation supervised learning”.

Text-SemiSeg is best understood as a heterogeneous research area rather than a single standardized task. In the literature, the label covers at least four distinct configurations: document text segmentation formulated as sentence-level boundary prediction; scene text segmentation and text instance segmentation learned with weak, self-supervised, or fully supervised signals; segmentation learned from image-level language such as referring expressions; and the specific 3D medical framework “Text-driven Multiplanar Visual Interaction for Semi-supervised Medical Image Segmentation” (Koshorek et al., 2018, Strudel et al., 2022, Zu et al., 2023, Huang et al., 16 Jul 2025). Across these settings, the unifying theme is segmentation under scarce dense annotation, with supervision transferred from auxiliary structure such as Wikipedia section hierarchies, OCR attention, predicted polygons, referring expressions, or category text embeddings.

1. Scope and task regimes

A plausible reading of the literature is that “Text-SemiSeg” names a family of annotation-efficient segmentation problems in which “text” can denote the segmented sequence itself, the segmented visual object, or the supervisory modality. The resulting task definitions are not interchangeable.

Setting	Supervision source	Representative paper
Document text segmentation	Wikipedia table-of-contents structure	(Koshorek et al., 2018)
Scene text instance segmentation	Text transcripts plus word-level boxes	(Zu et al., 2023)
Self-supervised scene text segmentation	Predicted polygon text regions	(Wang et al., 2023)
Referring expression segmentation	Image-level referring expressions	(Strudel et al., 2022)
3D medical segmentation with text guidance	Category text prompts plus labeled/unlabeled volumes	(Huang et al., 16 Jul 2025)

In document NLP, text segmentation means dividing a document into contiguous, topically coherent segments, with a binary boundary sequence over inter-sentence positions. In scene text vision, it means pixel-level separation of text foreground from background, or instance-wise masks for text elements. In language-grounded vision, segmentation is learned from text queries rather than mask annotations. In the medical paper that explicitly uses the name Text-SemiSeg, text is neither the target nor the image content; instead it is a semantic prior injected into semi-supervised volumetric segmentation (Koshorek et al., 2018, Strudel et al., 2022, Huang et al., 16 Jul 2025).

2. Supervision taxonomy and annotation economy

A central distinction in this area is the difference between weak supervision and semi-supervised learning. “Text Segmentation as a Supervised Learning Task” states that Wikipedia labels are inherited from editorial document structure and should therefore be viewed as weak supervision rather than manually annotated boundaries; the model is then trained in a standard supervised way on those noisy labels. The same source explicitly contrasts this with semi-supervised learning, in which one combines a smaller set of gold-labeled examples with a larger pool of unlabeled data, using objectives such as pseudo-labeling, consistency training, entropy minimization, graph propagation, or generative modeling (Koshorek et al., 2018).

The same distinction recurs in scene text. “Weakly-Supervised Text Instance Segmentation” removes pixel-level segmentation labels by using text transcripts $T$ and word-level bounding boxes $B_w$, with pseudo masks generated from recognition attention via a Text Adaptive Refinement module; it is therefore weakly supervised rather than semi-supervised in the classic segmentation sense (Zu et al., 2023). “Weakly-supervised segmentation of referring expressions” similarly replaces masks with image-level language supervision, while “Self-supervised Scene Text Segmentation with Object-centric Layered Representations Augmented by Text Regions” avoids pixel masks entirely but still relies on predicted polygon-level text regions as extra input information (Strudel et al., 2022, Wang et al., 2023).

By contrast, the 3D medical Text-SemiSeg paper adopts the standard low-label split $\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$ and $\mathcal D^U=\{x_i^u\}_{i=N_l+1}^{N_l+N_u}$, and combines supervised segmentation, decoder consistency, category-aware text-visual alignment, and mixed augmentation in a single semi-supervised objective (Huang et al., 16 Jul 2025). This suggests that the phrase “Text-SemiSeg” often denotes annotation-efficient segmentation more broadly, but only some works instantiate semi-supervision in the strict labeled–unlabeled sense.

3. Document segmentation as weakly labeled boundary prediction

The document branch is crystallized by “Text Segmentation as a Supervised Learning Task,” which reformulates text segmentation as sentence-level binary sequence labeling. For a document $x=(s_1,\ldots,s_n)$, the output is $y=(y_1,\dots,y_{n-1})$, where $y_i$ indicates whether sentence $s_i$ ends a segment. The model is a two-level hierarchical BiLSTM: a two-layer bidirectional LSTM encodes each sentence from tokens and max-pools to a sentence embedding, then another two-layer bidirectional LSTM contextualizes the sentence sequence, and a fully connected layer with softmax produces boundary probabilities $p_i$, decoded greedily by thresholding $p_i>\tau$ (Koshorek et al., 2018).

Its enabling contribution is Wiki-727k, a corpus of 727,746 English Wikipedia documents with hierarchical segmentations automatically derived from article table-of-contents structure. For experiments, training uses top-level segmentation only. The preprocessing removes photos, tables, template elements, single-sentence segments, documents with fewer than three segments, and documents where most segments were filtered; sentence splitting uses the NLTK Punkt tokenizer, and the first segment is removed during training because Wikipedia lead sections are often broad summaries (Koshorek et al., 2018).

The main quantitative results established that large-scale weak supervision can outperform older unsupervised methods on natural text. On the full Wiki-727k test set, the model achieves sentence-level $B_w$0 versus a random baseline of 53.09. On Wiki-50, it achieves $B_w$1, compared with GraphSeg at $B_w$2, random at $B_w$3, and human performance at $B_w$4. On Choi’s synthetic dataset, however, GraphSeg is markedly better, with $B_w$5–$B_w$6 $B_w$7 versus $B_w$8 for the proposed model, supporting the authors’ claim that synthetic benchmarks reward shallow lexical discontinuity detection more than natural within-document topic shifts (Koshorek et al., 2018).

For Text-SemiSeg, this paper is foundational less because it is itself semi-supervised than because it provides a reusable recipe: convert weakly structured document metadata into sentence-boundary labels at scale, train a supervised boundary detector, and then extend that baseline with pseudo-labeling, consistency regularization, domain-adaptive pretraining, or noise-aware learning. Those semi-supervised extensions are proposed explicitly in the paper’s discussion of future directions (Koshorek et al., 2018).

4. Scene text segmentation: fully supervised foundations, weak supervision, and self-supervision

The fully supervised reference point is “Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach,” which introduced TextSeg, a dataset of 4,024 images with a 2,646/340/1,038 train/val/test split and six types of annotations: word-wise and character-wise bounding polygons, masks, and transcriptions. The dataset mixes scene text and design text at roughly a 1:1 ratio and includes word-effect masks in addition to word and character masks. Its paired method, TexRNet, adds text-specific refinement through key features pooling, attention-based similarity checking, trimap loss, and a glyph discriminator loss; on TextSeg, DeepLabV3+ improves from 84.07 fgIoU to 86.06 fgIoU under TexRNet, which is the paper’s clearest “nearly 2%” gain (Xu et al., 2020).

Weakly supervised scene text instance segmentation is represented by “Weakly-Supervised Text Instance Segmentation.” Its central insight is that attention-based text recognition localizes character instances, so recognition attention can seed pseudo masks. The architecture couples an attention recognizer, a Text Adaptive Refinement module that performs two-stage iterative refinement on attention maps, and a text segmentation head trained with binary cross-entropy against pseudo labels. The reported gains over weakly supervised instance segmentation baselines are large: on ICDAR13-FST, the method improves weakly supervised performance by 18.95%, and on TextSeg by 17.80%, reaching 60.55 and 71.24 fIoU respectively in the strongest setup. The paper also emphasizes that the weakly supervised method reaches more than 82% of the fully supervised TexRNet performance without pixel-level mask labels (Zu et al., 2023).

A different annotation-efficient route is “Self-supervised Scene Text Segmentation with Object-centric Layered Representations Augmented by Text Regions.” That system uses no pixel-level masks and no synthetic pretraining, but conditions an object-centric layered autoencoder on polygon-level masks predicted by an external text localization model. Its Region Query Module uses a region image to encode “where to look” and “what to see,” while a background-replacement representation consistency constraint enforces that text masks remain stable when backgrounds are replaced. On TotalText, the detector-guided version reaches fgIoU 65.1 and F1 0.713, versus 47.8 and 0.621 for DeepUSPS; with ground-truth polygons, performance rises to fgIoU 76.6 and F1 0.838, which makes detector quality an explicit bottleneck (Wang et al., 2023).

Taken together, these papers show three different routes around dense annotation: build a high-quality supervised dataset and a text-specific refiner; derive pseudo masks from OCR attention; or use detector polygons as region priors in a self-supervised decomposition model. A plausible implication is that scene text segmentation depends as much on the quality and structure of auxiliary signals as on the final segmentation loss itself.

5. Text as supervision for segmentation

A second major branch uses text not as the segmented object but as the supervisory signal. “Weakly-supervised segmentation of referring expressions” introduces weakly supervised referring expression segmentation and proposes TSEG, a transformer-based model that learns segmentation masks directly from image-level referring expressions without pixel-level annotations. The method computes patch-text similarities and aggregates them with a multi-label patch assignment mechanism, replacing the single-label assumption that is inappropriate when expressions overlap spatially (Strudel et al., 2022).

The distinction between SPA and MPA is the paper’s key technical point. SPA uses a softmax over labels plus background, forcing each patch to choose one label. MPA instead performs binary patch-versus-background assignment for each expression independently, allowing overlap. On PhraseCut, this change raises validation mIoU from 21.12 for SPA to 28.77 for TSEG, and to 30.12 with CRF. On RefCOCO, RefCOCO+, and RefCOCOg, TSEG reaches 25.44, 22.01, and 22.05 mIoU respectively, far above GMP, GAP, and SPA baselines. In zero-shot Pascal VOC segmentation, TSEG reaches 48.5 mIoU, or 50.3 with label engineering, while GroupViT reports 51.2 after training on 30M image-text pairs (Strudel et al., 2022).

This literature demonstrates that image-level language can replace mask labels when the architecture preserves patch-level evidence and the pooling mechanism respects overlap and compositionality. It also clarifies a recurring misconception: language-grounded weak supervision is not equivalent to classical semi-supervised segmentation. The supervision is weak but spatially informative, and segmentation emerges as the latent explanation for image–text matching rather than from direct mask loss (Strudel et al., 2022).

6. The named framework “Text-SemiSeg” in 3D medical segmentation

The most literal use of the term appears in “Text-driven Multiplanar Visual Interaction for Semi-supervised Medical Image Segmentation,” which explicitly names its framework Text-SemiSeg. The method addresses standard semi-supervised 3D medical segmentation with a small labeled volumetric dataset and a larger unlabeled set, using a VNet-based MC-Net backbone with two decoders and a text encoder derived from CLIP-style prompt learning. Its three modules are Text-enhanced Multiplanar Representation (TMR), Category-aware Semantic Alignment (CSA), and Dynamic Cognitive Augmentation (DCA) (Huang et al., 16 Jul 2025).

TMR is designed around the paper’s claim that directly repeating text features through 3D volumes does not respect CLIP’s 2D pretraining paradigm. Encoder features $B_w$9 are pooled into coronal, sagittal, and axial planes, text-enhanced on each plane through attention, and then fused back into a residual 3D representation with learnable weights $\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$0. CSA then aligns prompt-learned text embeddings $\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$1 with category-specific visual features extracted from the final encoder representation and decoder predictions, using an MSE-style loss $\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$2. DCA reduces labeled–unlabeled discrepancy by mixing labeled foregrounds with unlabeled backgrounds and pseudo-labeled foregrounds with labeled backgrounds, updating the weaker decoder with a mixed segmentation loss $\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$3 (Huang et al., 16 Jul 2025).

The final training objective is

$\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$4

with $\mathcal D^L=\{(x_i^l,y_i^l)\}_{i=1}^{N_l}$5 implemented as a Gaussian warm-up coefficient. Experiments on Pancreas-CT, BraTS-2019, and MSD-Lung report consistent improvements over baselines. On Pancreas-CT, Text-SemiSeg reaches Dice 81.27 at 10% labeled and 83.50 at 20% labeled; on BraTS-2019 it reaches 85.27 and 86.69 Dice in the corresponding settings; and on MSD-Lung it reaches 65.02 and 68.21 Dice at 20% and 40% labeled. The ablation comparing simple repetition with multiplanar interaction improves Dice from 82.35 to 83.50 on Pancreas and from 85.57 to 86.69 on BraTS, while DCA provides the largest single ablation gain on both datasets (Huang et al., 16 Jul 2025).

Within the broader Text-SemiSeg literature, this paper is unusual because text is purely semantic prior rather than image content. It shows that textual category embeddings can function as class anchors in semi-supervised segmentation even when the segmentation target is anatomical rather than textual.

7. Limitations, misconceptions, and open directions

Several misconceptions are corrected repeatedly across this literature. First, weak supervision is not semi-supervision. Wikipedia section hierarchies, OCR attention, predicted polygons, and image-level referring expressions provide supervision, but they do not by themselves instantiate the classic labeled–unlabeled regime (Koshorek et al., 2018, Zu et al., 2023). Second, good results on synthetic or weak proxies do not guarantee performance on realistic segmentation targets: the document paper shows that Choi-style synthetic segmentation can favor shallow lexical discontinuity, while the scene text papers show strong dependence on detector quality or on the gap between weak and full supervision (Koshorek et al., 2018, Wang et al., 2023).

Third, annotation efficiency does not remove the need for task-specific inductive bias. In scene text, TexRNet argues that non-convex boundaries, diverse texture, arbitrary scale, and readability make text segmentation distinct from generic semantic segmentation; in referring expression segmentation, TSEG shows that one-label-per-patch assumptions are incompatible with overlapping language references; in medical Text-SemiSeg, the use of category text remains prompt-based and category-level rather than richly descriptive (Xu et al., 2020, Strudel et al., 2022, Huang et al., 16 Jul 2025).

The remaining performance gaps are substantial. TSEG’s weakly supervised PhraseCut result of 28.77 mIoU remains far below its fully supervised 49.6 mIoU upper bound, and detector-guided self-supervised scene text segmentation still trails supervised TexRNet on established benchmarks (Strudel et al., 2022, Xu et al., 2020, Wang et al., 2023). This suggests that future Text-SemiSeg systems will likely need to combine several of the signals already isolated in the literature: small amounts of dense annotation, large weakly labeled corpora, pseudo-labeling, consistency regularization, boundary-focused objectives, structure-aware refinement, and cross-task supervision transfer. In document segmentation, such extensions are already named explicitly—small manually segmented corpora combined with large weakly labeled Wikipedia corpora, pseudo-labeling on unlabeled target-domain text, consistency regularization, domain-adaptive pretraining, and noise-aware learning—making that branch a particularly clear blueprint for semi-supervised expansion (Koshorek et al., 2018).