Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Abstract: Recent segmentation methods leveraging Multi-modal LLMs (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.
Summary
- The paper demonstrates that MLLMs can perform precise segmentation by employing a single segmentation token and eliminating specialist mask decoders.
- The SELF1E approach integrates Residual Features Refilling and Amplifier techniques to preserve high-resolution image details for improved segmentation accuracy.
- Experimental results on datasets like RefCOCO show competitive cIoU scores, highlighting SELF1Eโs efficiency and adaptability for both segmentation and VQA tasks.
MLLM-Based Segmentation with a Single Token Approach
Introduction
The prevalent approach to segmentation tasks leverages Multi-modal LLMs (MLLMs) by employing specialist mask decoders. These tactics, while effective, often rely heavily on intricate architectures and additional token manipulations, leading to inefficiencies and complexities. The paper "Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token" (2603.19026) explores the feasibility of streamlining the segmentation process directly through MLLMs without relying on external decoders or numerous tokens, suggesting the elimination of the specialist mask decoders by introducing the SELF1E technique.
Figure 1: MLLM with specialist encoder and decoder configurations are prominent in earlier methodologies.
Methodology
The SELF1E framework proposes a significant departure from traditional segmentation methods by retaining high-resolution image features directly from MLLMs and integrating Residual Features Refilling (RFR) and Residual Features Amplifier (RFA) techniques. These enhancements prioritize preserving spatial information and amplifying crucial residuals extracted during the processing phase within the MLLM.
Figure 2: The additional branch of pre-compressed image features utilizes self-replication to maintain precision.
Residual Features Integration
At the core of SELF1E is the strategy to maintain uncompressed feature resolutions. By performing a pixel-unshuffle on high-resolution features and utilizing residuals from compressed LLM features, SELF1E effectively captures more granular spatial details, essential for precise segmentation.
Figure 3: RFR and RFA operations enhance both the resolution and distinctiveness of image representations.
Experimental Results
Extensive benchmarks validate the SELF1E's efficacy across multiple datasets, including RefCOCO and ReasonSeg, showcasing performance on par with, or exceeding, models that utilize specialist decoders. The SELF1E models deliver competitive cIoU results, further substantiating the assertion that MLLMs can achieve robust segmentation outcomes devoid of complex external decoder setups.
Figure 4: Resolution improvements significantly enhance segmentation accuracy in various contexts.
Further comparisons illustrate prominent results on open-vocabulary segmentation tasks, distinctly reinforcing SELF1E's capability to adapt to diverse semantics using minimal supervision setupsโall executed without sacrificing the model's innate VQA functionality.
Figure 5: Representative visuals underscore SELF1E's nuanced VQA capabilities alongside its segmentation prowess.
Implications and Future Directions
The SELF1E model embodies a pivotal shift toward efficient, decoder-less segmentation. This method not only reduces reliance on bulky configurations but also emphasizes the potential of MLLMs as standalone solutions for complex visual tasks. The streamlined approach paves the path for future exploration into more lightweight and adaptable models that can further capitalize on intrinsic MLLM functionality.
Conclusion
The research presented in "Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token" underscores the potential of MLLMs as comprehensive segmenters, offering a framework that significantly simplifies the segmentation process. By eliminating specialist mask decoders and utilizing innovative feature amplification methods, SELF1E points toward a future where MLLMs function as independent yet powerful tools in both segmentation and broader vision-LLM applications. This paradigm shift invites continued exploration of optimizing these models for multifarious AI-driven tasks.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token โ Explained Simply
What is this paper about?
This paper shows how a big AI model that understands both pictures and text (an MLLM) can draw object masks in images by itself, using just one special โsegmentation token,โ and without any extra helper networks. The authors call their method SELF1E (read like โselfieโ), short for โunlocking segmentation from the MLLM itself with 1 Embedding.โ
Segmentation means coloring in exactly the pixels that belong to something in a picture, like โthe dogโ or โthe person in a red hat.โ
What questions are the researchers trying to answer?
They focus on three simple questions:
- Can a single special token be enough to segment objects, without using extra, specialized decoders?
- If yes, what stops current models from doing this well?
- How can we fix that problem while keeping the system simple and fast?
Their core idea: the main problem isnโt the single tokenโitโs that many MLLMs shrink image details to save space and speed, which hurts pixel-precise masks.
How did they do it? (Methods in everyday language)
First, a bit of background: Many vision-LLMs โcompressโ image features before sending them into the language part of the model. This is like shrinking a big, detailed photo to a smaller size so it fits in a notebook. Itโs faster, but you lose fine details that are important for drawing sharp boundaries in segmentation.
The paper introduces three key tricks:
- Keep a high-detail copy:
- Think of saving a high-resolution version of the image features before the model compresses them. This gives a clearer map of where pixels are in the original image.
- Refill missing details using the modelโs own leftovers (Residual Features Refilling, RFR):
- After the model processes the compressed image, you can measure what changed. This โdifferenceโ is like the extra knowledge the model learned.
- Upscale that difference and add it back into the high-resolution featuresโlike restoring small details that were lost when you shrank the photo.
- Unfold compressed details smartly (Residual Features Amplifier, RFA, with Pixel-Unshuffle):
- Pixel-shuffle is a way models pack nearby pixels together to compress space. Pixel-unshuffle is the reverseโunpacking those details.
- The authors run a small network that does โunfoldingโ (pixel-unshuffle) on both the compressed features and the model-processed features. Then they subtract them to get a cleaner, higher-resolution โdifferenceโ and fuse it with the high-res copy. This makes the details sharper and better aligned.
Finally, they redesign how the modelโs attention works so the image tokens and the single [SEG] token can โtalkโ to each other in both directions. You can think of the [SEG] token as the โpaintbrushโ that needs a full view of the picture, and the image tokens need to โhear backโ from the brush about what to paint.
How do they produce the mask? At the end, the model compares each pixelโs features with the [SEG] token (like checking how similar each pixel is to the target description), and colors in the pixels that match best.
What did they find, and why does it matter?
The authors tested their approach on several tough tasks:
- Referring expression segmentation (e.g., โsegment the man wearing a blue hatโ)
- Generalized referring segmentation (can handle multiple or missing targets)
- Reasoning segmentation (needs multi-step thinking, like โsegment the animal that is not a cat but is next to the bicycleโ)
- Open-vocabulary segmentation (segment categories the model was not directly trained to segment)
Key takeaways:
- With only one segmentation token and no extra decoder network, SELF1E matched or beat many strong methods that use extra, specialized decoders or multiple tokens.
- On reasoning-heavy tasks, their larger model set new top results, showing it can both understand language deeply and draw accurate masks.
- It also worked well when segmenting categories it hadnโt been specifically trained on, showing good generalization.
- Importantly, the model kept its normal question-answering abilities, meaning it still โtalksโ well while it โdraws.โ
Why this matters:
- The method is simpler: no extra, heavy โmask decoderโ is needed.
- Itโs efficient: one token instead of many, and it reuses the modelโs own features.
- It proves the real bottleneck was lost image detail from compression. Fixing the resolution lets a single token work surprisingly well.
Whatโs the bigger impact?
This work points toward more unified AI systems that can see, talk, and precisely select objectsโall inside one model. That can make future tools:
- Easier to build and maintain (fewer extra parts)
- Faster and cheaper to run (less computation)
- More flexible (one model that can chat, recognize, and segment)
Think of uses like smart photo editing, assistive tech that highlights objects for people with low vision, AR apps that stick effects on exact objects, and robots that need to pick up the right item. By showing that one token and no extra decoder can work, the paper opens the door to simpler, more powerful vision-language tools.
Knowledge Gaps
Below is a single, consolidated list of the paperโs unresolved knowledge gaps, limitations, and open questions that future work could address:
- Quantify efficiency: report end-to-end inference latency, GPU memory footprint, and throughput for SELF1E vs specialist decoder baselines and vs UFO (multi-token), including how costs scale with image resolution and the pixel-shuffle factor ฮฑ.
- Backbone generality: validate SELF1E across diverse MLLM/VLM backbones and visual encoders (e.g., Qwen-VL, LLaVA, InstructBLIP, BLIP-2) to test portability beyond InternVL3-2B/8B.
- Formal justification of self-replication: provide a theoretical analysis or empirical bounds showing when pixel self-replication approximates pre-shuffled features and how approximation error depends on ฮฑ, patch size, and encoder architecture.
- Sensitivity analyses: ablate the effects of ฮฑ, input image resolution, visual patch size, and the capacity/placement of PUS MLPs on accuracy, stability, and compute.
- Parameter overhead: quantify the additional parameters and training/inference compute introduced by RFR/RFA/PUS (despite claims of โminimalโ), and compare convergence speed and energy usage to decoder-based methods.
- Attention-mask side effects: systematically evaluate how segmentation-specific attention masks affect VQA/text generation (beyond the brief claim of preservation), including multi-turn dialogue, instruction-following, and hallucination rates.
- Multi-instance outputs: clarify how a single [SEG] token can produce multiple disjoint masks simultaneously (instance/panoptic segmentation), and benchmark on COCO instance/panoptic datasets with AP/PQ metrics.
- Boundary quality: measure boundary-sensitive metrics (Boundary IoU, F-score) and evaluate on datasets where fine edges matter (e.g., Cityscapes, BSDS) to substantiate claims about โfine-grainedโ masks.
- OVS efficiency and ambiguity: assess the cost and error modes of per-category querying in open-vocabulary segmentation (e.g., overlapping categories, long-tail labels), and explore joint or structured decoding strategies to avoid N queries per image.
- Generalized referring beyond gRefCOCO: analyze false positive/negative rates for nonexistent objects, calibration of mask confidence thresholds, and robustness to ambiguous or compositional referring expressions.
- Robustness: test under occlusion, clutter, small/thin objects, lighting/weather changes, compression artifacts, and out-of-domain datasets (medical, remote sensing) to characterize failure modes.
- Comparative fairness: control training data and backbone capacity across baselines (specialist decoders, UFO, HiMTok) to isolate the gains from SELF1Eโs design vs. dataset mixtures or model size.
- Alternative similarity heads: compare the current matrix product to cosine similarity with temperature, learned bilinear heads, or normalization schemes; evaluate effects on calibration and separability.
- LoRA configuration: ablate LoRA ranks, target modules (Q/K/V, FFN, embeddings), and adaptation scope to understand the finetuning trade-offs for segmentation vs. VQA retention.
- Mask-token design: investigate whether multiple lightweight [SEG] tokens (or hierarchical tokens) can preserve efficiency while improving multi-object or part-level segmentation, and compare to the single-token constraint.
- Dynamic masking schedules: explore task-conditioned attention masks (e.g., switching masks per token or step) and quantify gains vs. complexity across segmentation and VQA tasks.
- Multi-scale features: examine feature pyramid or cross-scale fusion to improve small-object segmentation without exploding token lengths or compute.
- Video segmentation: extend SELF1E to temporal settings, studying temporal consistency, memory usage, and streaming performance.
- Quantization/pruning: study INT8/INT4 quantization or sparse attention to deploy SELF1E on edge devices without severe accuracy loss.
- Security and safety: test robustness to adversarial or malicious prompts (prompt injection affecting [SEG]), and define safeguards against erroneous segmentation outputs in critical domains.
- Cross-backbone pixel-shuffle variants: evaluate compatibility with encoders that use different downsampling (e.g., patchify vs. conv+stride) and assess how PUS/RFR need to adapt.
- Training stability: report optimization stability (loss spikes, mode collapse), curriculum effects, and data sampling strategies beyond the brief SELF1E vs SELF1E-SEG description.
- Comprehensive VQA results: include full VQA benchmarks and breakdowns in the main paper (not only appendix), quantifying any trade-offs introduced by segmentation-oriented changes.
- Rigorous equation/spec clarity: fix typos and ambiguities in equations and shapes (e.g., dimensions of FIMGโฒ after RFA) and provide a clear derivation to aid reproducibility.
- Domain transfer: systematically test zero-shot transfer to specialized domains (medical, remote sensing, document layout) to validate the claimed generality of decoder-free segmentation.
Practical Applications
Immediate Applications
Below are actionable use cases that can be deployed now, based on the paperโs decoder-free, single-token MLLM segmentation (SELF1E), its residual refilling/amplification modules (RFR/RFA), and the redesigned imageโsegmentation token interaction.
- [Software/Creative tools] Text-guided image editing and redaction
- Use case: In photo/video editors, let users type โremove the logo,โ โblur the person in the red jacket,โ or โisolate the cakeโ to generate pixel-accurate masks without a specialist mask decoder.
- Products/workflows: Plugins for Photoshop, Premiere, DaVinci Resolve; web editors adding a โSegment by promptโ tool; a REST API that takes an image + text and returns a mask.
- Assumptions/Dependencies: MLLM with InternVL-style visual encoder or equivalent; enough GPU/CPU memory for high-res features; language coverage aligned with user base.
- [Data/AI tooling] Faster dataset labeling via language-guided masks
- Use case: Annotators quickly generate masks by typing phrases like โfront bumper,โ โperson holding the cup.โ The single-token pipeline simplifies deployment versus SAM-style decoders.
- Products/workflows: Label Studio/CVAT extensions; auto-labeling pipelines that create initial masks for QA; active learning loops using text prompts to refine ambiguous regions.
- Assumptions/Dependencies: Domain-relevant prompts; occasional human correction; fine-tuning may be needed for niche domains.
- [Robotics/Manufacturing] Natural-language pick-and-place segmentation
- Use case: On workcells, operators issue commands like โpick the blue screwdriverโ or โsegment the red gasket edgeโ to obtain masks for grasp planning.
- Products/workflows: Robot perception nodes that expose a โSEG by promptโ service; simplifies stacks that previously needed separate mask decoders.
- Assumptions/Dependencies: Real-time constraints depend on model size (2B/8B) and quantization; lighting/occlusions may require domain-specific fine-tuning.
- [AR/Accessibility] Visual assistance with object highlighting
- Use case: AR glasses or mobile apps highlight โthe nearest exit sign,โ โthe salt shaker,โ or โthe leftmost busโ for low-vision users.
- Products/workflows: On-device or edge-offloaded segmentation overlay; integration with VQA for interactive guidance.
- Assumptions/Dependencies: Latency and power budgets on mobile hardware; safety/usability testing for accessibility contexts.
- [E-commerce/Retail] Catalog and shelf analytics
- Use case: โSegment all red shirts,โ โmask the price tags,โ or โsegment products with damaged packagingโ for visual search, inventory auditing, and price/redaction tasks.
- Products/workflows: Back-office image processors; checkout camera analytics; product photography post-processing.
- Assumptions/Dependencies: Domain drift across brands and stores; privacy compliance for consumer images.
- [Security/Compliance] Automated redaction and privacy filters
- Use case: โBlur faces,โ โmask license plates,โ or โremove visible IDsโ in images/videos for GDPR/CCPA compliance.
- Products/workflows: Ingest-time or on-the-fly redaction services; newsroom publishing pipelines; body-worn camera processing.
- Assumptions/Dependencies: High recall requirements; policy thresholds; traceability/audit logs.
- [Insurance/Automotive] Damage assessment pre-processing
- Use case: โSegment the dented area,โ โmask shattered glass,โ โhighlight scratched panelsโ to streamline claims triage.
- Products/workflows: Claims intake portals; adjuster tools that pre-segment regions for cost estimation models.
- Assumptions/Dependencies: Robustness to varied capture conditions; occasional human-in-the-loop.
- [Remote sensing/Energy] Rapid open-vocabulary extraction
- Use case: โSegment solar panels,โ โmask rooftops with visible moss,โ โextract asphalt cracksโ from aerial imagery using prompts.
- Products/workflows: GIS pipelines; energy asset monitoring dashboards; change detection pre-processing.
- Assumptions/Dependencies: Domain shift (nadir views, spectral differences) often requires fine-tuning; very large images may need tiling.
- [Education/Academia] Lightweight research baselines and teaching demos
- Use case: Teach segmentation and tokenโimage interactions without maintaining specialist decoders; ablate components (RFR/RFA, attention masks) in class or lab settings.
- Products/workflows: Colab demos; course modules on multimodal attention and resolution bottlenecks.
- Assumptions/Dependencies: Availability of open-source checkpoints (e.g., InternVL3-2B/8B) and repo; moderate compute.
- [MLOps/Platform] A single-token segmentation microservice
- Use case: Offer a โ/segmentโ endpoint that takes image + prompt and returns a mask, minimizing service complexity (no external decoders).
- Products/workflows: Gateway that serves both VQA and segmentation; batch/stream processing for enterprise pipelines.
- Assumptions/Dependencies: Containerization with model quantization; GPU scheduling for bursts; monitoring for prompt abuse.
Long-Term Applications
The following opportunities are promising but need further research, scaling, domain adaptation, or regulatory clearance.
- [Healthcare] Interactive radiology and pathology segmentation
- Use case: โSegment the tumor core,โ โisolate edema,โ โmask necrotic tissueโ with language prompts for planning and research.
- Tools/workflows: PACS-integrated assistants; semi-automated contouring for radiotherapy.
- Assumptions/Dependencies: Rigorous validation, bias assessment, and regulatory approvals; domain-specific pretraining for modalities (CT/MRI/WSI).
- [Video analytics/Media] Prompt-to-mask video editing with temporal consistency
- Use case: โTrack and mask the person in the green hoodie across the clip,โ โremove brand logo throughout the scene.โ
- Tools/workflows: Video editors with temporal propagation; streaming inference for live broadcast redaction.
- Assumptions/Dependencies: Temporal models and memory; compute optimization for long sequences; stable prompt disambiguation over time.
- [Robotics/Autonomy] On-device segmentation for drones and mobile robots
- Use case: โSegment power lines,โ โmask puddles/ice patches,โ โisolate traversable terrainโ from aerial/ground views.
- Tools/workflows: Edge SOC deployment (INT8/FP8 quantization); low-latency single-token segmentation nodes.
- Assumptions/Dependencies: Hardware acceleration; robustness against motion blur and extreme lighting; safety certification.
- [3D/Spatial computing] Prompted segmentation in 3D point clouds and meshes
- Use case: โSegment handrails,โ โmask load-bearing beams,โ โisolate the sofaโ in 3D scans for BIM/AR.
- Tools/workflows: Multimodal extensions to point-cloud encoders; 2Dโ3D lifting workflows combining image masks and depth/SLAM.
- Assumptions/Dependencies: New encoders and token interaction designs for 3D; alignment between 2D and 3D representations.
- [Agriculture] Crop and disease segmentation at scale
- Use case: โSegment leaves with rust,โ โisolate fruit clusters,โ โmask weed patches.โ
- Tools/workflows: Drone/tractor vision; decision support for spraying/harvesting.
- Assumptions/Dependencies: Extensive domain adaptation to plant phenotypes, growth stages, and sensors (RGB/NIR).
- [Smart cities/Infrastructure] Inspection and maintenance
- Use case: โSegment corrosion,โ โmask concrete spalling,โ โisolate potholes/cracksโ from urban scans.
- Tools/workflows: City maintenance dashboards; prioritization and cost estimation pipelines.
- Assumptions/Dependencies: Cross-city generalization; clear SOPs tying masks to interventions.
- [Finance/Underwriting/Risk] Property condition assessment from imagery
- Use case: โSegment roof damage,โ โmask standing water,โ โisolate wildfire defensible space.โ
- Tools/workflows: Remote underwriting aids; catastrophe response triage.
- Assumptions/Dependencies: Reliability across jurisdictions; explainability/auditability for regulated decisions.
- [Education/Assessment] Fine-grained skill evaluation in labs and vocational training
- Use case: โSegment the solder joint,โ โmask chemical meniscus region,โ โisolate unsafe hand positionsโ in training videos.
- Tools/workflows: Feedback systems that localize errors; auto-grading augmentation.
- Assumptions/Dependencies: Privacy safeguards; domain calibration and safety review.
- [Policy/Governance] Energy- and cost-efficient AI benchmarking for vision tasks
- Use case: Establish evaluation protocols showing decoder-free segmentationโs compute/energy savings vs. decoder-based pipelines for public procurement or grants.
- Tools/workflows: Green AI reporting standards; shared testbeds for segmentation/VQA co-capabilities.
- Assumptions/Dependencies: Transparent metrology (throughput, power draw, accuracy); vendor cooperation.
- [Software ecosystems] Foundation multimodal SDKs with unified VQA+SEG capability
- Use case: Standardize a single-token segmentation interface across MLLMs, reducing model fragmentation in apps.
- Tools/workflows: Cross-model adapters; inference graphs that route VQA/SEG through shared encoders and attention masks.
- Assumptions/Dependencies: Generalization of SELF1E mechanics beyond InternVL-family models; community adoption.
- [Privacy-by-design] On-device personal media processing
- Use case: Fully local โmask faces,โ โremove address labels,โ โblur childrenโ in home videos and photos.
- Tools/workflows: Mobile frameworks (Core ML/NNAPI) hosting compact models; private photo library batch processors.
- Assumptions/Dependencies: Aggressive compression/quantization; acceptable quality at mobile speeds.
- [Scientific imaging] Open-vocabulary segmentation for microscopy and Earth science
- Use case: โSegment mitochondria,โ โisolate chlorophyll plumes,โ โmask kelp forestsโ from specialized imagery.
- Tools/workflows: Lab notebooks with language-driven ROI extraction; citizen-science platforms.
- Assumptions/Dependencies: Modality shifts (fluorescence, SAR, multispectral) require retraining and careful validation.
Cross-cutting assumptions and dependencies
- Model architecture: SELF1E relies on access to image encoder features and pixel-shuffle compression typical of InternVL-like MLLMs; porting to other MLLMs may need engineering to expose pre-/post-compression features and to implement pixel-unshuffle pathways.
- Compute and memory: Although decoder-free, retaining high-resolution features and unshuffle operations increases memory; feasibility on edge devices depends on quantization, tiling, and resolution constraints.
- Data/domain shift: Strong results were achieved with mixed segmentation/VQA training; out-of-domain performance may require fine-tuning and prompt engineering.
- Language coverage: Performance depends on the MLLMโs text understanding; multilingual deployments may need multilingual pretraining/fine-tuning.
- Safety/regulation: High-stakes uses (healthcare, autonomous systems, surveillance) require validation, monitoring, and compliance frameworks before deployment.
Glossary
- AdamW optimizer: An optimization algorithm that decouples weight decay from the gradient-based update to improve generalization. Example: "with the AdamW optimizer and a cosine scheduler."
- Attention mask (segmentation-specific): A pattern that restricts which tokens can attend to which others during attention; here redesigned to enable both image-to-image and image-to-segmentation visibility. Example: "we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation"
- Bidirectional interaction: Mutual attention flow allowing tokens (e.g., image and segmentation tokens) to attend to each other. Example: "facilitates the bidirectional interaction between [SEG] token and [IMG] tokens"
- Binary Cross-Entropy (BCE) loss: A per-pixel classification loss for binary mask prediction. Example: "pixel-level cross-entropy loss $\mathcal{L}_{\text{BCE}$ and DICE loss $\mathcal{L}_{\text{DICE}$ for optimization."
- cIoU: A variant of Intersection-over-Union used for evaluating referring segmentation masks. Example: "following the evaluation protocol using the cIoU metric."
- Causal mask: An autoregressive attention constraint that prevents tokens from attending to future positions. Example: "The causal mask and bidirectional mask denote two widely used attention masks."
- Class-agnostic mask embeddings: Mask representations that do not encode semantic category labels, enabling category-agnostic segmentation. Example: "apply multiple segmentation tokens and class-agnostic mask embeddings."
- Cosine scheduler: A learning-rate schedule that follows a cosine decay pattern during training. Example: "with the AdamW optimizer and a cosine scheduler."
- DICE loss: An overlap-based segmentation loss that maximizes similarity between predicted and ground-truth masks. Example: "pixel-level cross-entropy loss $\mathcal{L}_{\text{BCE}$ and DICE loss $\mathcal{L}_{\text{DICE}$ for optimization."
- Dot-product similarity: Using the inner product between embeddings (e.g., image and segmentation token) to score mask membership. Example: "predicting masks through dot-product similarity between [SEG] tokens and image embeddings"
- Generalized Referring Expression Segmentation (GRES): A referring segmentation setting that includes multiple targets and nonexistent targets. Example: "designed for Generalized Referring Expression Segmentation~(GRES) that contains scenarios of referring to multiple target objects as well as nonexistent objects."
- gIoU: A generalized IoU metric variant used for evaluating segmentation quality in GRES. Example: "achieves 44.4% / 37.5% (cIoU / gIoU) on the validation split"
- LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that inserts low-rank adapters into a pretrained model. Example: "The finetuning of the MLLM requires LoRA~\cite{lora22} adaptation with a rank of 128 for SELF1E-2B and 64 for SELF1E-8B."
- Mask de-tokenizer: A lightweight module that reconstructs segmentation masks directly from a small set of tokens. Example: "employs a lightweight mask de-tokenizer that decodes the prediction mask from 32 tokens"
- Matrix product: Computing the segmentation mask by multiplying image and segmentation embeddings without a specialist decoder. Example: "yet directly produces the segmentation mask via matrix product"
- mean Intersection-over-Union (mIoU): The average IoU across classes used to evaluate semantic segmentation. Example: "with mean Intersection-over-Union (mIoU) as the evaluation metric."
- Multi-modal LLM (MLLM): A LLM capable of processing and reasoning over both text and images. Example: "Recent segmentation methods leveraging Multi-modal LLMs (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception."
- Open-Vocabulary Segmentation (OVS): Segmenting categories not seen during training by leveraging language supervision. Example: "We also evaluate our model on the open-vocabulary segmentation (OVS) task"
- Pixel-shuffle downsampling: A compression technique that rearranges spatial information into channel dimensions, reducing resolution before LLM processing. Example: "incorporate pixel-shuffle downsampling with MLPs to spatially compressed visual features for efficient LLM processing"
- Pixel-unshuffle: The inverse of pixel-shuffle that restores higher-resolution spatial structure from compressed embeddings, often combined with an MLP. Example: "We introduce an MLP-with-Pixel-Unshuffle operation $f_{\text{PUS}$ to restore the compressed feature representations."
- Referring Expression Segmentation (RES): Segmenting the object(s) in an image described by a natural-language expression. Example: "Referring Expression Segmentation (RES) is a canonical benchmark for evaluating language-guided segmentation"
- Residual Features Amplifier (RFA): A module that enhances residual features by unshuffling compressed representations from with/without-LLM streams for better alignment. Example: "we revisit Eq.~\ref{eq:nr} and introduce the Residual Features Amplifier operation as follows:"
- Residual Features Refilling (RFR): A process that upsamples and fuses residuals from the LLM back into high-resolution image features to recover fine details. Example: "This Residual Features Refilling~(RFR) process effectively restores the resolution of fine-grained features"
- Segment Anything Model (SAM): A foundation segmentation model used as a specialist decoder in many pipelines. Example: "e.g., methods with SAM~\cite{sam23} exploit features with much higher resolution compared to $\bm{F}_{\text{IMG}$."
- Segmentation token ([SEG]): A special token whose embedding conditions segmentation predictions or decoders. Example: "These methods typically introduce a customized [SEG] token whose embedding prompts the mask decoder to generate precise masks from image features."
- Side-adapter: An auxiliary module used to merge multi-stage pipelines into a unified framework. Example: "SAN~\cite{san23} designs a side-adapter to unify the two stages"
- Specialist mask decoder: An external segmentation-specific decoder (e.g., SAM, Mask2Former) attached to an MLLM for mask prediction. Example: "does not require a specialist mask decoder"
- Vision Question Answering (VQA): A task where models answer questions about images; used here to ensure language-vision reasoning is preserved. Example: "while preserving VQA capabilities"
- Vision-LLM (VLM): A model jointly trained on images and text to align visual and textual representations. Example: "The emergence of Vision-LLMs(VLMs) pretraining significantly narrows the gap between visual and textual modalities."
- Zero-shot inference: Performing evaluation on unseen datasets or categories without task-specific fine-tuning. Example: "conduct the zero-shot inference without any dataset-specific fine-tuning."
Collections
Sign up for free to add this paper to one or more collections.
