MTGNet: Multi-Text Guided Few-Shot Segmentation

• The paper demonstrates a dual-branch framework that leverages multi-textual priors and a frozen CLIP backbone to enhance few-shot semantic segmentation.
• MTGNet refines initial text-to-visual responses through key modules (MTPR, TAFF, FCWA) for precise semantic alignment and reduced intra-class variation.
• State-of-the-art results on benchmarks like PASCAL-5i and COCO-20i validate its effectiveness over traditional single-text approaches.

The Multi-Text Guided Few-Shot Semantic Segmentation Network (MTGNet) is a dual-branch framework that advances CLIP-based few-shot semantic segmentation by leveraging multiple textual prompts per class to construct expressive semantic priors. Unlike traditional approaches that rely on a single, generic text description for each novel class, MTGNet fuses a set of diverse text prompts and introduces explicit cross-modal fusion strategies to address semantic diversity and intra-class variation in complex object categories. Built on a frozen CLIP ViT-B/16 backbone, MTGNet employs three key modules—Multi-Textual Prior Refinement (MTPR), Text Anchor Feature Fusion (TAFF), and Foreground Confidence-Weighted Attention (FCWA)—and is optimized end-to-end using binary cross-entropy. MTGNet achieves state-of-the-art segmentation performance on challenging few-shot benchmarks, with systematic ablation analyses isolating the impact of each module and textual strategy (Jiao et al., 19 Nov 2025).

1. Architectural Overview

MTGNet operates on a standard few-shot segmentation episode, receiving support images with corresponding masks, a query image, and a set of $n$ class-specific textual prompts. The workflow branches into a textual and a visual pathway:

• Textual Branch: Each textual prompt is encoded via CLIP's text encoder, producing a matrix of $n$ embeddings per class. These are used to generate initial text-to-visual cosine similarity priors for the query image, further refined in the MTPR module.
• Visual Branch: Support and query images are processed through the CLIP image encoder to extract visual features. Textual and visual features from supports and queries, self-attention maps from the ViT, and initial priors, are fed through TAFF and FCWA to yield a robust visual prior.

The three resulting priors—two refined textual ($P_{qt}^{glb}$, $P_{qt}^{acc}$) and one refined visual ($P_{qs}^{ref}$)—are concatenated and decoded via an HDMNet-style lightweight decoder, yielding the final pixel-wise segmentation mask.

Key architectural parameters:

Component	Shape or Setting	Description
CLIP backbone	ViT-B/16, frozen, $d=512$	Processes all text and image features
# Textual Prompts	$n=5$ per class	E.g., "head", "body", part/attribute/generic descriptions
TSP threshold	$\tau=0.73$	Fixed for all similarity propagation, selected by ablation
Input Size	$473 \times 473$	All images resized for uniformity

This structure enables MTGNet to explicitly align multi-level textual semantics with support and query visual content, mitigating limitations of earlier CLIP-based methods (Jiao et al., 19 Nov 2025).

2. Multi-Textual Prior Refinement (MTPR)

The MTPR module addresses the semantic limitations of single-prompt CLIP approaches. It refines initial text-to-visual priors by propagating reliable, high-confidence activations across semantically similar regions and aggregating complementary information from multiple prompts.

• Threshold Similarity Propagation (TSP): Discards low-confidence activations per prompt, propagating confident responses spatially via self-attention affinities in the query image:

$$P_{qt}^{thr}(i,j) = P_{qt}^{init}(i,j)\,\mathbf{1}\{P_{qt}^{init}(i,j)\ge\tau\},\quad \tau=0.73$$

$P_{qt}^{ref} = P_{qt}^{thr} A_q$

• Multi-Text Aggregation: Collapses $n$ prompt-wise responses into two priors:
  • $P_{qt}^{glb}$ (global, per-pixel max across prompts), maximizing semantic coverage for structurally complex objects.
  • $P_{qt}^{acc}$ (accurate, per-pixel average), suppressing noise from low-quality or interfering prompts.

This dual approach ensures both generalization and precision, with ablations demonstrating incremental mIoU improvements over single-prompt and non-text approaches (+2.5–5% mIoU on PASCAL-5i depending on module combination) (Jiao et al., 19 Nov 2025).

3. Text Anchor Feature Fusion (TAFF)

TAFF performs explicit cross-modal alignment between local support prototypes and query features using the multi-text prompts as semantic anchors.

• Text-Guided Support Prototypes: For each text prompt, a refined support prior is used to pool masked support features, yielding $n$ part-level prototypes:

$$\mathcal P^s(i,:) = \bigl[\operatorname{softmax}_{j}(P_{st}^{ref}(i,j) M_s(j))\bigr]^\top F_s^\top$$

• Query Injection: Each query spatial location aggregates these prototypes according to TSP-refined prompt activations, subsequently fused with the original query feature:

$$F_q^s = (P_{qt}^{ref})^\top \mathcal P^s; \quad F_q' = (F_q^s)^\top + F_q$$

The result is a set of query representations grounded in both part-level textual semantics and local support structure, reducing intra-class visual drift and aiding classes with compositional variability.

4. Foreground Confidence-Weighted Attention (FCWA)

FCWA bolsters the robustness of visual priors by emphasizing consistent, reliable support regions and de-emphasizing outlier or noisy areas:

• Support-Foreground Self-Correlation: Computes self-similarity in the support foreground using attention maps, producing a consistency weighting.
• Cross-Image Correlation: Cross-correlates support and (TAFF-fused) query features, weighted by the internal support consistency, followed by spatial pooling to yield an initial visual prior:

$P_{qs}^{init}(j) = \max_i (A_{sq}'(i,j))$

• Further Refinement: Applies the TSP via query self-correlation, ensuring the final $P_{qs}^{ref}$ focuses on reliable, semantically aligned regions.

Combined with TAFF, FCWA produces major improvements in strong-intra-class-variation scenarios (e.g., "person," "table," "train"), with multi-text approaches yielding up to +9.7% mIoU over single-text (Jiao et al., 19 Nov 2025).

5. Loss Function and Optimization

MTGNet trains solely with a binary cross-entropy loss on the final decoder logits, with no auxiliary or contrastive losses:

$$\mathcal{L}_{BCE} = -\sum_{i} [M(i)\log Q_1(i) + (1-M(i))\log Q_0(i)]$$

The adoption of a pure segmentation endpoint simplifies optimization and validates the effectiveness of the cross-modal and multi-text design.

Key training strategies:

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$\sim$1e-4
Training epochs	150 (PASCAL-5i), 50 (COCO-20i)
Batch size	8 (1-shot), 4 (5-shot)
Inference sampling	1,000/query-support pairs (PASCAL-5i), 5,000 (COCO-20i)

6. Experimental Results and Ablations

MTGNet achieves leading performance on PASCAL-5i (76.8% mIoU, 1-shot) and COCO-20i (57.4% mIoU, 1-shot) under rigorous evaluation. Comparison with prior methods demonstrates substantial gains particularly in settings with significant intra-class variation.

Quantitative comparisons (PASCAL-5i, 1-shot mIoU):

Method	mIoU (%)
HDMNet	69.4
PI-CLIP	76.4
MTGNet	76.8

Ablation of modules on PASCAL-5i Fold-0 (1-shot):

Configuration	MTPR	TAFF	mIoU (%)
Baseline (HDMNet)	—	—	71.00
+ MTPR only	✓	—	74.77
+ MTPR+TAFF	✓	✓	75.87
+ All three (w/ FCWA)	✓	✓	77.71

Textual strategy ablation:

No-Text	Single-Text	Multi-Text	mIoU (%)	FB-IoU (%)
✓	—	—	72.71	84.58
—	✓	—	77.25	87.68
—	—	✓	77.71	87.93

Threshold sensitivity experiments show optimal performance for TSP at $\tau \approx 0.7$, balancing noise suppression against region coverage.

7. Related Work and Significance

MTGNet is positioned among a contemporary line of works integrating textual priors, notably CLIP-based cross-modal architectures such as PI-CLIP and LDAG (Wang et al., 20 Nov 2025). Unlike prior approaches that either rely solely on visual support or a single textual prompt, MTGNet demonstrates that multiple, targeted prompts combined with explicit cross-modal fusion (TAFF) and support region denoising (FCWA) substantially enhance few-shot segmentation.

A plausible implication is that further gains may be achievable via richer prompt engineering (e.g., leveraging LLM-generated attributes as in (Wang et al., 20 Nov 2025)) or joint optimization of text-vision backbones. MTGNet provides a modular foundation for such extensions and, given its ablation-driven insight, may serve as a baseline for future benchmarks in text-augmented few-shot segmentation research.