CausalCLIPSeg: Causal Medical Segmentation

• The paper demonstrates that combining CLIP-driven encoders with a causal intervention module enhances lesion segmentation by aligning text and pixel-level cues.
• It details a cross-correlation decoder that transforms global textual features into dense segmentation maps for precise lesion delineation.
• Experimental results on the QaTa-COV19 benchmark show state-of-the-art performance, outperforming previous methods with notable Dice gains.

CausalCLIPSeg is an end-to-end framework for referring medical image segmentation that leverages CLIP to delineate lesions indicated by textual descriptions. The method addresses two coupled difficulties: aligning visual and textual cues despite their distinct data properties, and reducing confounding bias that can drive the model toward spurious correlations rather than meaningful causal relationships. Its core design combines CLIP-driven image–text representations, a tailored cross-modal decoding method for text-to-pixel alignment, and a causal intervention module trained through an adversarial min–max game. On the QaTa-COV19 benchmark, it is reported to achieve state-of-the-art performance (Chen et al., 20 Mar 2025).

1. Problem setting and conceptual basis

Referring medical image segmentation targets delineating lesions indicated by textual descriptions. In this setting, the segmentation target is not specified solely by the image; it is conditioned on language that identifies the relevant lesion region. CausalCLIPSeg is motivated by the observation that large-scale pre-trained vision-LLMs provide a rich image-text embedding space, but direct transfer to medical segmentation is nontrivial because dense prediction requires pixel-level grounding rather than only global semantic matching (Chen et al., 20 Mar 2025).

A central premise of the framework is that CLIP’s semantic space can be enforced onto the medical domain even though CLIP was not trained on medical data. This is operationalized through a decoder that maps a global textual representation into a dense segmentation response map, thereby turning a global image–text prior into explicit text-to-pixel alignment. The second premise is causal: the image, the text, and the segmentation mask may all be affected by unobserved confounders, such as scanner artifacts, creating backdoor correlations that can degrade lesion delineation.

This combination of multimodal transfer and causal intervention situates CausalCLIPSeg at the intersection of referring segmentation, medical vision-language modeling, and causally informed representation learning. A plausible implication is that the framework is designed not merely to improve semantic compatibility between image and text, but to improve the evidential basis on which segmentation judgments are made.

2. CLIP-driven encoders and cross-modal decoding

The model uses CLIP-driven encoders for both text and image streams. The text encoder employs a byte-pair-encoding tokenizer with vocabulary size 49 152 and feeds tokens into a 12-layer Transformer. The final activation at the $[EOS]$ token is taken as a global text feature

$\tau \in \mathbb{R}^T.$

The vision encoder is a ResNet-101 pretrained on CLIP datasets and produces multi-scale feature maps

$$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$

These choices preserve CLIP’s pretrained multimodal structure while exposing multi-resolution visual representations suited to dense prediction (Chen et al., 20 Mar 2025).

The cross-modal decoder is the mechanism that translates the global text feature into a segmentation response. A linear projection maps $\tau$ into a textual kernel $W_\tau\in\mathbb R^{C\times K\times K}$ and a bias $b_\tau\in\mathbb R$. After upsampling the concatenated vision features $F$ to full resolution via interpolation and convolution, the model performs a spatial cross-correlation: $$S \;=\; D(\tau,F) \;=\; W_\tau \ast \mathrm{up}(F)\;+\;b_\tau\cdot\mathbf{1},$$ where $\ast$ denotes convolution and $\mathbf{1}$ is an all-ones map.

This decoder explicitly aligns each textual query with each pixel location. In the paper’s formulation, this is the mechanism by which CLIP’s image-text space is transferred to dense, pixel-wise medical segmentation. The architecture therefore differs from approaches that treat text only as a global conditioning vector; here, the text feature is converted into a spatially applied kernel that produces a dense response map.

3. Structural causal model and confounder disentanglement

The causal component is defined through a structural causal model

$\tau \in \mathbb{R}^T.$0

where $\tau \in \mathbb{R}^T.$1 is the image, $\tau \in \mathbb{R}^T.$2 the text, $\tau \in \mathbb{R}^T.$3 the mask, and $\tau \in \mathbb{R}^T.$4 unobserved confounders such as scanner artifacts. Under this model, the backdoor path $\tau \in \mathbb{R}^T.$5 induces spurious correlations. The purpose of causal intervention is therefore to separate lesion-relevant evidence from nuisance factors that co-vary with the target mask (Chen et al., 20 Mar 2025).

CausalCLIPSeg implements this via self-annotation of confounders. An adversarial masker network $\tau \in \mathbb{R}^T.$6 processes each stage feature $\tau \in \mathbb{R}^T.$7 and predicts a soft attention mask

$\tau \in \mathbb{R}^T.$8

The features are then disentangled into causal and spurious components: $\tau \in \mathbb{R}^T.$9 where $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$0 denotes element-wise multiplication.

Multi-scale causal and spurious features are resized to a common resolution via CARAFE and fused by $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$1 convolutions: $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$2

$$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$3

A common misconception is that the causal component requires external confounder labels. In CausalCLIPSeg, the confounders are self-annotated by the masker rather than manually supervised. The paper’s framing is that the module self-annotates confounders and excavates causal features from inputs for segmentation judgments, which distinguishes it from pipelines that depend on explicit nuisance annotations.

4. Adversarial min–max optimization and empirical performance

The optimization strategy introduces two identical decoders, $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$4 and $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$5, which segment from $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$6 and $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$7, respectively. With standard cross-entropy loss $$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$8, the objective is

$$\{F_i\}_{i=2}^4,\quad F_i\in\mathbb{R}^{C_i\times H_i\times W_i}.$$9

Here $\tau$0 trades off between encouraging strong causal predictions and penalizing spurious ones. The stated effect is an adversarial min–max game that encourages retention of lesion-relevant causal features while penalizing background spurious cues (Chen et al., 20 Mar 2025).

Training is reported on a single NVIDIA RTX 4090 in PyTorch. Images are resized to $\tau$1, the maximum text length is 20, both encoders are initialized with CLIP-pretrained weights, and optimization uses Adam with initial learning rate $\tau$2, cosine decay, and adversarial weight $\tau$3. Training runs for up to 2 000 epochs with early stopping after 100 epochs of no validation gain.

Evaluation is conducted on QaTa-COV19, consisting of 9 258 chest X-rays with masks labeled by radiologists and textual annotations such as “Bilateral pulmonary infection, four infected areas, upper lower left lung and …”. The split is 5 716 train, 1 429 validation, and 2 113 test. The metrics are Dice coefficient and mean IoU. Baselines include uni-modal methods—U-Net, UNet++, AttUNet, nnU-Net, TransUnet, Swin-Unet, and UCTransNet—and multi-modal methods—ConVIRT, TGANet, GLoRIA, ViLT, LAVT, and LViT.

Method	Dice	mIoU
LViT	83.66%	75.11%
CausalCLIPSeg	85.21%	76.90%

On the test split, CausalCLIPSeg achieves Dice $\tau$4 and mIoU $\tau$5. This outperforms the best prior multi-modal method, LViT, by $\tau$6 Dice and $\tau$7 mIoU. Compared to vision-only models, the method yields 5–7% absolute Dice gains, which the paper interprets as evidence for the benefit of textual guidance and causal intervention.

The ablation study isolates the role of CLIP initialization and the causal module:

Configuration	Dice	mIoU
No CLIP pre-training, no causal module	82.50%	73.24%
+ Causal module only	83.61%	74.86%
+ CLIP only	83.71%	74.69%
Full model	85.21%	76.90%

These results verify that both CLIP initialization and causal intervention are individually and jointly critical.

5. Contributions, scope, and interpretive significance

The paper identifies four key contributions. First, it demonstrates that CLIP’s image-text space can be successfully transferred to dense, pixel-wise medical segmentation via a simple cross-correlation decoder. Second, it introduces a causal intervention module that self-annotates confounders and disentangles causal versus spurious features in an end-to-end fashion. Third, it formulates an adversarial min–max game that optimizes causal features while penalizing confounding ones. Fourth, it reports a new state of the art on the QaTa-COV19 referring segmentation benchmark (Chen et al., 20 Mar 2025).

These claims are benchmark-specific and should be read with the scope of evaluation in mind. The current evaluation is limited to chest X-rays and COVID-19 lesions. Extending the framework to other modalities such as CT and MRI, and to other pathologies, is explicitly identified as an important direction. The paper also points to richer causal graphs, including modeling textual confounders, and to more efficient adversarial schemes as possible future developments.

A common misunderstanding would be to attribute the reported gains solely to CLIP pretraining. The ablation results do not support that interpretation: the CLIP-only and causal-only variants each improve on the baseline, but the full model performs better than either variant alone. Another misunderstanding would be to treat the causal mechanism as equivalent to generic attention masking. In the reported formulation, the masker, the structural causal model, and the adversarial objective are integrated specifically to address confounding bias rather than merely to sparsify features.

6. Relation to later causal multimodal segmentation frameworks

A related 2025 framework, Multimodal Causal-Driven Representation Learning (MCDRL), integrates causal inference with a vision-LLM to address domain generalization in medical image segmentation rather than referring segmentation. MCDRL is implemented in two steps: it first uses CLIP’s cross-modal capabilities to identify candidate lesion regions and construct a confounder dictionary through text prompts, and it then trains a causal intervention network that uses this dictionary to identify and eliminate the influence of domain-specific variations while preserving anatomical structural information critical for segmentation tasks (Liang et al., 7 Aug 2025).

Its pipeline differs materially from CausalCLIPSeg’s QaTa-COV19 setup. The inputs include a medical image and two sets of prompts: $\tau$8 class prompts of the form “A $\tau$9 in an endoscopic image” and $W_\tau\in\mathbb R^{C\times K\times K}$0 confounder prompts describing domain variations such as blurry lighting, narrow-band imaging, distant view, and mucus reflections. Multimodal Target Region Selection computes spatial cosine similarity between dense CLIP vision features and text embeddings, thresholds the top $W_\tau\in\mathbb R^{C\times K\times K}$1 pixels, and extracts region features $W_\tau\in\mathbb R^{C\times K\times K}$2. A confounder dictionary $W_\tau\in\mathbb R^{C\times K\times K}$3 is then passed through small MLPs to produce keys and values for a cross-attention intervention: $W_\tau\in\mathbb R^{C\times K\times K}$4 yielding domain-invariant features $W_\tau\in\mathbb R^{C\times K\times K}$5. The segmentation head is a Transformer-style decoder with multi-head self-attention and upsampling.

MCDRL evaluates domain generalization in a multi-domain, multi-center setting across bronchoscopy, laryngoscopy, and laparoscopy datasets, training on four sites and testing on the held-out fifth site. Reported average Dice for MCDRL is 78.6 with CLIP–ResNet50, 80.0 with CLIP–ViT-B/16, and 81.6 with CLIP–ViT-L/14, exceeding the listed baselines StyLIP and BiomedCoOp in the corresponding table. Its ablation reports Avg mDice of 69.37 for a baseline with no MTRS and no CDRL, 80.47 with only CDRL, 78.71 with only MTRS, and 88.46 for full MCDRL. This suggests that the causal-VLM design pattern exemplified by CausalCLIPSeg was subsequently extended toward explicit domain generalization, confounder dictionaries, and broader multi-center evaluation regimes.

Within this broader line of work, CausalCLIPSeg remains specifically defined by three features: CLIP-driven encoders, a cross-correlation decoder that converts global text into dense segmentation responses, and a self-annotating causal intervention module optimized through adversarial min–max training.