SELF1E: Decoder-Free Segmentation Method

• SELF1E Technique is a segmentation framework that leverages a single learnable [SEG] token to predict dense masks from preserved high-resolution features.
• It integrates pixel-unshuffle, residual refill, and a dual-perception attention mechanism to fuse visual embeddings and the segmentation token seamlessly.
• The method outperforms decoder-based models in accuracy and speed, achieving state-of-the-art results on segmentation benchmarks with scalable efficiency.

The SELF1E (Segmentation by External-Less Fusion with 1 Embedding) technique is a decoder-free image segmentation framework for multi-modal LLMs (MLLMs), formulated to predict dense segmentation masks using a single learnable segmentation embedding ([SEG] token). SELF1E operates by preserving and refining spatially high-resolution image features, applying innovative residual refilling and amplification, and enabling comprehensive feature interaction between image tokens and the segmentation token through a redesigned attention scheme. Unlike previous approaches, which typically rely on specialist mask decoders or an array of auxiliary tokens, SELF1E predicts masks directly by computing the similarity between high-resolution visual embeddings and the [SEG] token, demonstrating both high accuracy and computational efficiency (Zhang et al., 19 Mar 2026).

1. Architecture: Single-Token Paradigm and Feature Flow

SELF1E builds upon a standard MLLM architecture, such as InternVL3, consisting of a visual encoder $\mathcal{E}$ (CNN or ViT) and an autoregressive LLM $\mathcal{M}$. For input image $x$, the encoder yields uncompressed features $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$ with $N_0 = H_0 \cdot W_0$. A pixel-shuffling+MLP operation compresses these to $F_{V_1}\in\mathbb{R}^{N_1 \times d}$ with $N_1 = N_0/\alpha$ and $d \approx \alpha d_0$, incorporating a single [SEG] token into the sequence for LLM processing.

The LLM output consists of $F_{IMG} \in \mathbb{R}^{N_1 \times d}$ (visual tokens) and $F_{SEG}\in\mathbb{R}^{1\times d}$ ([SEG] token embedding). Unlike previous methods, SELF1E predicts the segmentation mask by directly computing the dot product between each (post-fused, high-resolution) per-pixel feature and $\mathcal{M}$0, normalized by $\mathcal{M}$1:

$\mathcal{M}$2

where $\mathcal{M}$3 and $\mathcal{M}$4 are the post-refined, high-resolution feature representations. No external decoder is used for this prediction pathway. Key architectural elements include the preservation and upsampling of uncompressed features, explicit computation and amplification of LLM-encoder feature residuals, and a dual-perception attention mechanism for pixel–[SEG] interaction (Zhang et al., 19 Mar 2026).

2. Mathematical Modules: Pixel-Unshuffle, Residual Refill, and Amplification

Pixel-Unshuffle Operation

Given $\mathcal{M}$5, pixel-unshuffle by a factor $\mathcal{M}$6 reorganizes it into $\mathcal{M}$7:

$\mathcal{M}$8

SELF1E wraps this in an MLP so $\mathcal{M}$9 yields $x$0 features.

Residual Feature Refilling (RFR)

The compressed LLM and encoder outputs produce a residual $x$1. After upsampling by $x$2, $x$3, and a self-replicated+MLP branch produces $x$4. The refilled feature is:

$x$5

Residual Feature Amplifier (RFA)

For enhanced alignment, both compressed streams are pixel-unshuffled via separate MLPs:

$x$6

Further fusion yields:

$x$7

and the [SEG] embedding is unshuffled and averaged across its $x$8 copies to yield $x$9.

Attention-Mask Redesign

SELF1E implements bidirectional attention among image tokens and [SEG], replacing the standard causal LLM mask. For $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$0 tokens, the attention mask $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$1 is

• $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$2 for all image tokens $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$3,
• $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$4 (image$F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$5[SEG]),
• $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$6 ([SEG]$F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$7image), and $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$8 elsewhere, ensuring full pixel–[SEG]–pixel communication per self-attention block.

3. Feature–Token Interaction in Self-Attention

The feature sequence $F_{V_0} \in \mathbb{R}^{N_0 \times d_0}$9 enables each transformer block to use queries, keys, and values from both pixel and [SEG] representations, with the dual-perception mask facilitating global feature interaction. Consequently, the [SEG] token aggregates a global summary over high-resolution features, while each pixel is modulated by the segmentation cue. The final segmentation mask emerges from the direct dot-product similarity of each pixel embedding with $N_0 = H_0 \cdot W_0$0.

This mechanism removes the need for explicit mask decoders or multi-token schemes. The [SEG] token serves as a single-point prompt for the network to edit all pixel embeddings, yielding dense masks seamlessly at the end.

4. Training Strategy and Segmentation Objectives

SELF1E is trained jointly on mixed visual question answering (VQA) and segmentation data with a multi-term loss:

$N_0 = H_0 \cdot W_0$1

• $N_0 = H_0 \cdot W_0$2: standard autoregressive cross-entropy on textual VQA responses,
• $N_0 = H_0 \cdot W_0$3: pixel-wise binary cross-entropy between predicted mask $N_0 = H_0 \cdot W_0$4 and ground truth $N_0 = H_0 \cdot W_0$5,

$N_0 = H_0 \cdot W_0$6

• $N_0 = H_0 \cdot W_0$7: Dice loss to maximize mask overlap,

$N_0 = H_0 \cdot W_0$8

No additional mask-IoU loss is explicitly used, since the Dice term closely approximates IoU alignment.

5. Quantitative Results and Benchmark Comparisons

SELF1E-SEG achieves state-of-the-art or near-parity results with decoder-based specialist models on referring expression segmentation benchmarks. On the cIoU metric:

Method	RefCOCO testA	RefCOCO+ testA	RefCOCOg test
HyperSeg (decoder+multi)	85.7%	83.5%	78.9%
HiMTok-8B(ft)	85.2%	82.7%	80.6%
UFO-8B(ft)	82.6%	80.4%	77.3%
SELF1E-SEG-8B	86.2%	84.2%	82.8%

SELF1E surpasses or equals the highest cIoU on all tasks, utilizing no external mask decoder and just a single [SEG] token. Inference efficiency is significant: SELF1E-8B (on a single RTX 4090 GPU) achieves 105 ms, 9.5 FPS, and 17.7 GB—approximately twice as fast as LISA-7B (SAM-decoder) and nearly ten times faster than UFO-7B (multi-token dot product), while being more parameter-efficient (Zhang et al., 19 Mar 2026).

6. Practical Trade-offs and Scalability

SELF1E’s focus on spatial grounding delivers precise segmentation, with a modest performance decrease in OCR-heavy or knowledge-driven VQA tasks (e.g., OKVQA, TextVQA), amounting to a 2–5% reduction relative to baseline. However, general VQA capability is preserved. The design scales across parameter regime, with consistent segmentation improvements for both SELF1E-2B and SELF1E-8B, without excessive memory or compute costs.

In summary, by aligning high-resolution visual features, residual LLM outputs, and single-token segmentation prompts within a unified attention mechanism, SELF1E demonstrates that decoder-free, highly efficient single-token segmentation is feasible without sacrificing accuracy or scalability (Zhang et al., 19 Mar 2026).