Text-Based Pixel/Object Reference

• Text-Based Pixel/Object Reference is a domain combining natural language processing with fine-grained image segmentation to pinpoint pixels and objects.
• It employs various architectures—including unified vision–language models and transformer-based tokenization—to achieve accurate region localization and segmentation.
• Recent advances integrate perception priors with end-to-end training, yielding high accuracy in segmentation benchmarks and supporting interactive visual tasks.

Text-based pixel/object reference encompasses the family of computational models and systems that resolve free-form natural language expressions to specific, fine-grained visual regions—at the pixel, object, or arbitrary spatial level—in images or videos. This capability is fundamental to natural language grounding, interactive visual reasoning, image editing, segmentation, and human–machine communication for embodied agents. Recent work demonstrates integration of flexible language understanding, perception priors, and precise pixel-level localization in unified, end-to-end architectures, as well as modular and compositional pipelines, benchmarking these methods on referring segmentation, region-based question answering, and multi-turn interaction tasks.

1. Core Architectures for Text-Based Pixel/Object Reference

Recent approaches generally follow one of several high-level architectures that tightly couple language and vision signals:

• Unified Vision–LLMs: Methods such as OMG-LLaVA integrate a frozen universal segmentation encoder–decoder (e.g., ConvNeXt-L backbone, OMG-Seg), specialized perception-prior embeddings, and a LLM. Visual features, object queries, and mask predictions are fused into compact visual tokens and concatenated with tokenized text inputs. Decoding is performed by the LLM, which emits both text and segmentation tokens (Zhang et al., 2024).
• Object-Centric and Arbitrary Granularity Tokenization: Frameworks like PixelRefer introduce a Scale-Adaptive Object Tokenizer (SAOT) that adapts to free-form, user-specified regions at arbitrary granularity (object, part, fine pattern). These regions yield rich, compact object tokens, fused with global and local features, then input to the LLM for reasoning and answer generation (Yuan et al., 27 Oct 2025).
• End-to-End Mask Generation and Reasoning: UniPixel implements a multimodal transformer wherein mask prediction is triggered on demand (e.g., special <SEG> tokens in the LLM's output), feeding mask queries through a SAM-style decoder and integrating mask pointers into downstream reasoning via an explicit memory bank (Liu et al., 22 Sep 2025).
• Pixel-Aligned Language Decoders: In PixelLLM, each word generated by an LLM is jointly aligned to a regressed pixel coordinate, trained on datasets with dense pixel-word alignment (e.g., Localized Narrative), which supports dense grounding, region-conditioned captioning, and segmentation (Xu et al., 2023).
• Transformer and Cross-Attention Models: Pixel-BERT aligns raw pixel tokens and language tokens within a single-stream bidirectional transformer, using cross-attention weights for on-the-fly localization of natural language tokens to specific pixels (Huang et al., 2020).
• Modular, Compositional Pipelines: Ref-SAM3D composes an off-the-shelf text-to-mask system with a SAM3D 3D reconstructor, mapping referring expressions to foreground masks, then using these masks for object-centric 3D geometry prediction (Zhou et al., 24 Nov 2025).
• Direct Manipulation and Text Fusion: Point and Instruct serializes user-selected points/boxes into JSON snippets embedded in text prompts, allowing an LLM (e.g., GPT-3.5) to output a revised object layout that is then rendered by a layout-to-image generative model (Helbling et al., 2024).

2. Visual–Linguistic Representation and Perception Priors

A central advance in recent systems is the explicit preservation and embedding of strong visual priors into the text reasoning pathway:

• In OMG-LLaVA, the frozen segmentation module produces per-query object masks, confidences, and queries. These are fused via a perception-prior embedding:
  • Compute mask scores using $M \odot S$ with a softmax, generating per-pixel, per-query assignment.
  • Construct pixel tokens as $T_{pv} = MS \cdot Q + F$, where $F$ are image features (Zhang et al., 2024).
  • Object-centric tokens (foreground queries) $Q_{fg}$ become $T_{ov}$.
  • All tokens are projected to the LLM's embedding space.
• PixelRefer’s SAOT pools masked visual features with explicit position encoding, followed by $k$-means clustering for redundancy reduction. Object-centric Infusion further fuses global scene context into object tokens in memory-efficient variants (Yuan et al., 27 Oct 2025).
• UniPixel’s mask queries, object memory tokens, and dynamic pooling over predicted masks encode both explicit spatial memory and object-centric visual features for downstream compositional reasoning (Liu et al., 22 Sep 2025).
• PixelLLM uses a prompt feature extractor and two-way Q-former attention to tightly couple patch features with location prompts, providing dense spatial word–pixel alignment (Xu et al., 2023).

3. Instruction Processing and Output Modalities

Text-based reference is always driven by natural language prompts, whose processing determines the flexibility and controllability of the system:

• Standard tokenization is typically used, but special tokens (<Image>, <Region>, [SEG], <REF>, <SEG>, <MEM>) are inserted to signal pointer/mask/instruction points within sequences.
• In OMG-LLaVA, text instructions (e.g., “segment the red car”) attend over both pixel and object tokens in self- and cross-attention layers; the LLM emits [SEG] tokens when the referenced region is identified (Zhang et al., 2024).
• PixelRefer supports concatenation of region, object, and free-form question tokens. The system handles multi-turn dialogue and multiple regions simultaneously, maintaining arbitrary granularity (Yuan et al., 27 Oct 2025).
• User-provided points/boxes can be serialized (e.g., as JSON) and interleaved with text, as in Point and Instruct, without the need for explicit neural fusion (Helbling et al., 2024).

Output modalities include:

• Segmentation masks per region or per referred object.
• Regressed pixel/box coordinates accompanying generated words.
• Mask-informed textual or dialog responses.
• Object-centric memory bank tokens for multi-stage reasoning.
• Direct object or region selection for subsequent editing or 3D reconstruction.

4. Training Objectives and Supervision Signals

Supervision strategies balance text generation, mask prediction, and optional auxiliary tasks:

• Multi-Stage Training: OMG-LLaVA uses two stages: (1) pretrain projectors on text and regularization, (2) instruction tuning with cross-entropy for text and combined binary mask loss (L_CE + L_DICE) for segmentation prediction (Zhang et al., 2024).
• Pixel/Object Mask Losses: UniPixel jointly optimizes language modeling loss, focal loss and Dice loss for mask pixels, MAE for IoU prediction, objectness cross-entropy, and (if needed) answer classification loss (Liu et al., 22 Sep 2025).
• Localization Loss: PixelLLM regresses a coordinate for each generated token, with per-token L1 loss on aligned (word, pixel) pairs and standard label-smoothed cross-entropy for vocabulary output (Xu et al., 2023).
• Adversarial + Mask Losses: Text-as-Neural-Operator models combine GAN loss, L1 reconstruction loss, and weak mask supervision, using incidental masks inferred from input–output differences (Zhang et al., 2020).
• Proxy or No Training: Ref-SAM3D, as an off-the-shelf composition, performs no joint training and relies on existing segmenter and reconstructor weights (Zhou et al., 24 Nov 2025).

5. Benchmarking, Evaluation, and Empirical Outcomes

• Segmentation and Referring Expression Datasets: Across models, RefCOCO, RefCOCO+, RefCOCOg, COCO panoptic, VIPSeg, LVIS, PACO, Visual Genome, and VideoRefer-Bench are used for evaluation.
  • OMG-LLaVA achieves cIoU ≈ 78.0/69.1/72.9 on RefCOCO/+/g after finetuning, and maintains strong panoptic segmentation scores (Zhang et al., 2024).
  • UniPixel attains 64.0 J&F for video segmentation, up to 65.3% gIoU for image-level referring (Liu et al., 22 Sep 2025).
  • PixelLLM surpasses prior SOTA in box and mask precision on RefCOCO, e.g., 89.8% [email protected] and 76.9% mask cIoU (Xu et al., 2023).
  • PixelRefer-7B yields 91.7 SSim and 85.3 SIoU on PACO, with improved attribute and reasoning metric benchmarks (Yuan et al., 27 Oct 2025).
• Efficiency and Ablations: PixelRefer-Lite achieves ∼100× FLOP savings by omitting global tokens, with ≥90% accuracy retention (Yuan et al., 27 Oct 2025). Ablations confirm that joint training, memory mechanisms, and perception prior fusion are critical to performance in all systems.
• Qualitative Outcomes: Examples include “segment the red car” producing precise masks, region captioning on arbitrary selected regions, and multi-object editing guided by text plus pointers.

6. Extensions: Editing, 3D, and Unified Pixel Representation

• Image Editing: I2E proposes a full decompose–then–action pipeline, segmenting images into explicit object layers, maintaining a structured environment state, and parsing natural language instructions into atomic manipulation actions, governing both edit plausibility and compositional complexity (Yu et al., 7 Jan 2026).
• 3D Object Reference: Ref-SAM3D demonstrates text-based reference for 3D reconstruction by integrating a text-to-mask segmenter with SAM3D, allowing the selection and reconstruction of referential objects in 3D space from single RGB images (Zhou et al., 24 Nov 2025).
• Unified Pixel Space: PixelWorld (PEAP) removes the distinction between text and image entirely by rasterizing all inputs as images and processing using only ViT-style architectures, highlighting the potential for completely unified visual–linguistic reasoning in the pixel domain (Lyu et al., 31 Jan 2025).
• Calibration in Diffusion Models: ELBO-T2IAlign calibrates pixel-level class alignment using per-class likelihood lower bounds (ELBO) directly from the diffusion training objective, specifically improving referencing of small, rare, or occluded objects in zero-shot segmentation (Zhou et al., 11 Jun 2025).

7. Open Challenges and Future Directions

Open issues remain concerning the dependency on high-quality mask proposals (Yuan et al., 27 Oct 2025, Liu et al., 22 Sep 2025), integration of proposal networks into main inference, maintaining global scene context with aggressive token reduction (Yuan et al., 27 Oct 2025), handling of spatial relationships and complex references (Zhou et al., 24 Nov 2025), and extending models beyond 2D to 3D or spatio-temporal domains. Exploration of end-to-end referable region discovery, unsupervised pointer learning, interactive dialogue for pixel-grounded visual QA, and multi-modal iterative editing with physical logic are considered promising extensions (Yuan et al., 27 Oct 2025, Yu et al., 7 Jan 2026, Liu et al., 22 Sep 2025). The field trends toward general-purpose, reasoning-augmented, pointer-based models that unify pixel, object, and text semantics for both perception and action.