Text-Swin-UMamba: Multimodal Segmentation

• The paper integrates short-form radiology reports with a Swin-UMamba backbone, leading to improved CT lesion segmentation performance (e.g., Dice 0.82) compared to prior methods.
• Text-Swin-UMamba employs a Mamba-based UNet architecture featuring SS2D selective scanning and multi-scale fusion, which efficiently captures both local detail and long-range dependencies.
• The approach demonstrates a significant 37% improvement over some baselines while also laying the groundwork for hybrid designs in scene text detection tasks.

Searching arXiv for the cited papers to ground the article in current records. Text-Swin-UMamba is a multimodal segmentation architecture that integrates short-form radiology report text with a Swin-UMamba image backbone for CT lesion segmentation (Cheng et al., 8 Aug 2025). The term is also entangled with two adjacent usages in the literature. “Swin-UMamba” denotes a Mamba-based UNet with ImageNet-based pretraining for medical image segmentation (Liu et al., 2024), and the overview of “TextMamba: Scene Text Detector with Mamba” uses “Text-Swin-UMamba” to describe a hybrid design that could combine Swin-style hierarchical features, SS2D-based selective scanning, and a U-shaped decoder for scene text detection (Zhao et al., 7 Dec 2025). As a result, the expression names an implemented text-conditioned medical segmentation model in one paper and a proposed architectural synthesis in another.

1. Terminology and lineage

The most direct usage of the name comes from “Text Embedded Swin-UMamba for DeepLesion Segmentation” (Cheng et al., 8 Aug 2025). In that work, Text-Swin-UMamba comprises a Swin-UMamba image encoder, a Swin-UMamba image decoder, and a Text Tower encoder. The medical-imaging formulation is explicitly multimodal: each 2D CT slice is paired with a structured sentence describing lesion location, type, and attributes.

A recurrent source of confusion is that the foundational “Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining” does not itself include a textual modality. The source material states that “Text-Swin-UMamba refers to the paper ‘Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining.’ The work does not involve textual modality; ‘Text-’ appears to be a prefix unrelated to the method” (Liu et al., 2024). In practice, the implemented text-conditioned system is the later DeepLesion paper, whereas the earlier Swin-UMamba paper provides the visual backbone and the broader Mamba-based encoder–decoder design.

The term also appears in a different domain in the TextMamba overview, where it denotes a practical hybrid architecture for scene text detection rather than medical lesion segmentation (Zhao et al., 7 Dec 2025). That usage is architectural and prospective rather than a separately benchmarked standalone paper.

Paper	Domain	Role of text
“Text Embedded Swin-UMamba for DeepLesion Segmentation” (Cheng et al., 8 Aug 2025)	CT lesion segmentation	Integrated as short-form radiology report descriptions
“Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining” (Liu et al., 2024)	Medical image segmentation	No textual modality
“TextMamba: Scene Text Detector with Mamba” (Zhao et al., 7 Dec 2025)	Scene text detection	“Text” refers to scene text, not report-conditioned fusion

2. Swin-UMamba as the visual substrate

Swin-UMamba is a UNet-style encoder–decoder with dense skip connections and deep supervision, integrating a Mamba-based visual backbone pretrained on ImageNet, specifically VMamba-Tiny, and a task-tailored decoder (Liu et al., 2024). Two variants are introduced: Swin-UMamba, which uses a CNN decoder, and Swin-UMamba†, which uses a Mamba-based decoder.

The encoder has five hierarchical stages with progressive down-sampling. The stem, or Stage-1, is a $2\times$ down-sampling convolution layer with $7\times7$ kernel, stride $2$, padding $3$, followed by 2D instance normalization. Stage-2 uses patch embedding with $2\times2$ patch size and keeps resolution at $1/4\times$. Stages 3–5 each begin with patch merging and then several Visual State Space blocks implementing SS2D selective scans. The depth per stage is fixed as $\{2,2,9,2\}$ VSS blocks for Stages 2–5, and the channel dimensions are $D=\{48,96,192,384,768\}$ for Stages 1–5. Positional embedding is not used.

The SS2D mechanism reconciles 2D spatial structure with 1D state space scans by unfolding a feature map $z$ along four directions $v \in \{1,2,3,4\}$, processing each resulting sequence through the selective scan operator $7\times7$0, and merging the outputs back into a 2D feature map:

$7\times7$1

$7\times7$2

$7\times7$3

The underlying state-space model is given in continuous time as

$7\times7$4

and in discretized form as

$7\times7$5

The principal motivation is the linear complexity of Mamba/SSM sequence modeling, $7\times7$6, in contrast to the quadratic complexity of attention, $7\times7$7.

The decoder differs by variant. Swin-UMamba uses transpose convolutions for up-sampling, residual convolution blocks for skip refinement, and deep supervision heads at each scale. Swin-UMamba† replaces CNN up-sample blocks with a patch expanding layer followed by two VSS blocks at each decoding stage. This replacement yields major efficiency reductions: on AbdomenMRI, parameters drop from $7\times7$8M to $7\times7$9M and FLOPs from $2$0G to $2$1G (Liu et al., 2024).

ImageNet-based pretraining is central in this design. The encoder’s VSS blocks and patch merging layers are initialized with ImageNet-pretrained weights from VMamba-Tiny; the stem, early patch embedding, decoder, and heads are trained from scratch. The transfer strategy freezes the pretrained encoder parameters for the first $2$2 epochs and then fine-tunes all layers end-to-end.

3. Text conditioning in the DeepLesion model

The defining contribution of Text-Swin-UMamba in the DeepLesion paper is the integration of short-form radiology report text into the Swin-UMamba decoder (Cheng et al., 8 Aug 2025). The text source consists of short-form structured descriptions tied to each 2D slice in ULS23 DeepLesion, including location such as “mediastinum” and “retroperitoneum,” lesion type such as “lymph node,” “nodule,” and “mass,” and attributes such as “hypoattenuation,” “enhancing,” and “irregularity.”

The text pathway is implemented through a Text Tower encoder. A custom Tokenizer preprocesses the text, and the pre-trained BioLord model encodes clinical sentences into contextual token embeddings. Mean pooling computes a fixed-size sentence embedding by averaging across token embeddings, yielding a vector of dimension $2$3.

Fusion is performed at five scales in the Swin-UMamba decoder. At each decoder stage $2$4, a dedicated linear projection, labeled “Lang Fusion,” maps the $2$5-dimensional text embedding to the channel dimension of the corresponding image feature map. Fusion is implemented by channel-wise concatenation followed by convolution. If $2$6 denotes the decoder’s image feature and $2$7 the sentence embedding, the stage-specific projection $2$8 produces

$2$9

and the fused feature is

$3$0

The paper indicates concatenation followed by convolution but does not provide a formal equation beyond the linear projection and fusion, and the exact broadcasting or tiling strategy is implementation-specific.

The visual architecture remains U-Net-like. The encoder employs patch embedding and hierarchical downsampling via patch merging across stages, while the decoder upsamples through “Up-Sample Block” and “Transpose Conv” components and aggregates features via skip connections and residual blocks. Vision state-space blocks, labeled “VSS Block,” are placed in both encoder and decoder stages, interleaved with patch embedding, patch merging, and upsampling operations. The paper does not provide explicit Mamba or SSM equations for this implementation; rather, it adopts the Swin-UMamba design as the backbone.

Additional text encoders and fusion variants are reported as implemented, including LLaVA-style Text Tower, BERT-based encoding, and tail encoding, with details in the GitHub repository. The primary reported results use the Text Tower with multi-scale decoder fusion (Cheng et al., 8 Aug 2025).

4. Data, optimization, and quantitative results

The DeepLesion study uses the publicly available ULS23 DeepLesion dataset, tailored from the original DeepLesion (Cheng et al., 8 Aug 2025). The reported dataset characteristics are $3$1 unique patients, $3$2 studies, and $3$3 CT slices, with age $3$4 years and sex distribution $3$5 male and $3$6 female. Each annotated slice has an associated structured sentence describing the finding. Patient-level splits are reported as train $3$7, validation $3$8, and test $3$9. The image format is 2D CT slices of size $2\times2$0, while the Discussion notes that lesions are centered on $2\times2$1 images to simplify segmentation relative to the original $2\times2$2 DeepLesion images.

Training uses five-fold cross-validation, AdamW, an initial learning rate of $2\times2$3, cosine annealing, and $2\times2$4 epochs. Hardware is a single NVIDIA V100x GPU with $2\times2$5 GB memory, and training takes approximately $2\times2$6–$2\times2$7 days per model. The loss is a combined Dice loss and cross-entropy loss optimized with AdamW; the exact weighting coefficients are not specified. The paper defines the Dice coefficient as

$2\times2$8

The paper reports Hausdorff95 in pixels and notes that a standard Hausdorff definition is

$2\times2$9

On the test set, Text-Swin-UMamba achieves Dice $1/4\times$0, Jaccard $1/4\times$1, Hausdorff95 $1/4\times$2 pixels, sensitivity $1/4\times$3, and specificity $1/4\times$4 (Cheng et al., 8 Aug 2025). Relative to prior methods, LanGuideMedSeg obtains Dice $1/4\times$5, so Text-Swin-UMamba yields a $1/4\times$6 improvement, $1/4\times$7. Relative to xLSTM-UNet, which reports Dice $1/4\times$8, the gain is $1/4\times$9, $\{2,2,9,2\}$0. Relative to nnUNet (2D), which reports Dice $\{2,2,9,2\}$1, the reported improvement is $\{2,2,9,2\}$2, $\{2,2,9,2\}$3, together with lower Hausdorff95, $\{2,2,9,2\}$4 versus $\{2,2,9,2\}$5 pixels.

The paper also provides qualitative cases. For case 002704_02_01_049, described as “chest, abdomen, lung, liver, fissure,” Text-Swin-UMamba achieved Dice $\{2,2,9,2\}$6. For case 001702_01_01_077, described as “lymph node, pelvis, enhancing,” it achieved Dice $\{2,2,9,2\}$7. The discussion attributes these gains to the clinical context supplied by short-form report descriptions, particularly location priors and attribute cues that help the decoder focus on relevant regions and interpret ambiguous boundaries.

5. Relation to TextMamba and the scene-text interpretation

In scene text detection, the TextMamba paper proposes a Mamba-based detector and the accompanying overview explicitly frames a possible “Text-Swin-UMamba” design as a hybrid of Swin-style hierarchy, Mamba selective scanning, and U-shaped multi-scale decoding (Zhao et al., 7 Dec 2025). This usage belongs to a different task family from DeepLesion segmentation: “text” here refers to text instances in natural images rather than textual conditioning from reports.

TextMamba itself is built on a DETR-style pipeline with a ResNet-50 backbone and a hybrid encoder called Mix-SSM. The stages are: ResNet-50 features $\{2,2,9,2\}$8, $\{2,2,9,2\}$9, are flattened to token sequences $D=\{48,96,192,384,768\}$0; the encoder integrates deformable attention, Top$D=\{48,96,192,384,768\}$1 selection, SS2D, and a dual-scale feed-forward network; an Embedding Pyramid Enhancement Module reconstructs pyramid maps and fuses multi-scale features; and a decoder predicts instance masks and polygon control points end-to-end.

The Mix-SSM block combines deformable attention, Top$D=\{48,96,192,384,768\}$2 masking on the query-key attention matrix $D=\{48,96,192,384,768\}$3, SS2D selective scanning, and DSFFN. The Top$D=\{48,96,192,384,768\}$4 operator retains only the strongest $D=\{48,96,192,384,768\}$5 affinities per query:

$D=\{48,96,192,384,768\}$6

SS2D then scans the 2D image grid in four canonical paths and feeds the resulting sequences to a selective state-space block; the SS2D depth is set to $D=\{48,96,192,384,768\}$7. DSFFN replaces the vanilla FFN with two parallel expansion-reduction paths, $D=\{48,96,192,384,768\}$8 and $D=\{48,96,192,384,768\}$9, aggregated with residual addition:

$z$0

EPEM reconstructs pyramid maps $z$1, fuses them with backbone features $z$2 by element-wise addition $z$3, applies a feature pyramid enhancement module, and retains the enhanced $z$4 to drive the mask head.

The scene-text results of TextMamba are reported as F-measures of $z$5, $z$6, and $z$7 on CTW1500, TotalText, and ICDAR19ArT, respectively (Zhao et al., 7 Dec 2025). Ablations show that the hybridization is important: the baseline with $z$8 attention layers has $z$9, replacing attention with SS2D only degrades to $v \in \{1,2,3,4\}$0, and Mix-SSM with $v \in \{1,2,3,4\}$1 attention layers plus SS2D reaches $v \in \{1,2,3,4\}$2. Deep attention with $v \in \{1,2,3,4\}$3 layers reaches $v \in \{1,2,3,4\}$4 but with a large parameter increase of $v \in \{1,2,3,4\}$5M relative units.

Within that overview, a “Text-Swin-UMamba” would use Swin for hierarchical local features, insert SS2D in encoder stages for long-range modeling, and apply EPEM and U-shaped fusion to propagate multi-scale cues into mask and polygon heads (Zhao et al., 7 Dec 2025). The proposed practical guidance includes SS2D depth of approximately $v \in \{1,2,3,4\}$6, tuning Top$v \in \{1,2,3,4\}$7 per stage, maintaining DSFFN’s dual branches, and using $v \in \{1,2,3,4\}$8 proposals with $v \in \{1,2,3,4\}$9 control points. The stated trade-offs are improved recall for long curved text, improved precision from Top$7\times7$00 and EPEM, and slight runtime overhead from multi-directional SS2D scans and pyramid reconstructions.

6. Limitations, misconceptions, and future directions

A central misconception is that Text-Swin-UMamba names a single, settled architecture across domains. The literature instead supports two distinct meanings. In medical imaging, it is an implemented text-conditioned Swin-UMamba for DeepLesion segmentation (Cheng et al., 8 Aug 2025). In scene text detection, it is a suggested hybrid extrapolated from TextMamba’s selective SSM design rather than a separately reported benchmarked system (Zhao et al., 7 Dec 2025). The earlier Swin-UMamba work, moreover, does not involve text at all (Liu et al., 2024).

The DeepLesion paper identifies several limitations. The dataset design uses lesion-centered $7\times7$01 images, simplifying segmentation relative to full-size $7\times7$02 CT slices and potentially constraining generalizability to more realistic scenarios with small lesions in large fields of view (Cheng et al., 8 Aug 2025). The approach is 2D; a 3D extension is described as feasible but was not attempted in this pilot. Architectural variants such as a text decoder for long-form generation and relational memory modules were not explored due to complexity.

The Swin-UMamba paper identifies a different set of limitations. It focuses on 2D medical image segmentation, so extension to volumetric data would require volumetric SS2D analogs or 3D state-space designs (Liu et al., 2024). It also notes potential domain shift between ImageNet pretraining and medical imagery, and it does not report wall-clock inference speed or memory footprints despite the favorable $7\times7$03 scaling of Mamba.

The TextMamba overview emphasizes that naively replacing attention with SS2D hurts performance, which indicates that hybrid design is necessary rather than optional (Zhao et al., 7 Dec 2025). In its ablations, “Replace attention with SS2D only” degrades from $7\times7$04 to $7\times7$05. It also warns that over-aggressive Top$7\times7$06 can drop essential context and hurt recall, while too shallow SS2D may underfit long-range dependencies. This suggests that selective SSM components are most effective when coupled with sparse attention, multi-scale fusion, and carefully tuned decoder pathways.

Taken together, these papers position Text-Swin-UMamba not as a single canonical model but as a family resemblance across Mamba-based encoder–decoder systems. In the implemented medical form, the defining idea is language-guided lesion segmentation through BioLord-based text embedding and multi-scale decoder fusion (Cheng et al., 8 Aug 2025). In the scene-text interpretation, the defining idea is the integration of Swin’s hierarchical locality, SS2D’s linear-time long-range modeling, and U-shaped fusion for arbitrarily shaped text detection (Zhao et al., 7 Dec 2025). The common thread is the use of state-space modeling to mediate between local detail, global context, and multi-scale decoding.