UniPixel: Unified Pixel-Level Reasoning

• UniPixel is a unified multi-modal model that integrates pixel-level segmentation, object referring, and reasoning using mask-grounded processing.
• It leverages a Qwen2.5-VL backbone, SAM2.1 mask decoder, and an object memory bank to dynamically handle visual prompts and propagate masks across frames.
• The term also applies to a nanoscale OLED design, unifying charge-injection, plasmonic enhancement, and vertical stacking for high-density pixel arrays.

UniPixel denotes a large multi-modal model for pixel-level visual reasoning that integrates object referring, segmentation, and subsequent reasoning within a single Qwen2.5-VL–based system. It processes visual prompt inputs, generates relevant masks on demand, and performs subsequent reasoning conditioning on those intermediate pointers during inference, thereby coupling language semantics to pixel-accurate visual representations in both image and video settings (Liu et al., 22 Sep 2025). In later work, UniPixel also serves as a video-MLLM substrate for SPARROW, which augments it with mechanisms for spatial precision and temporal referential consistency in video grounding (Alansari et al., 12 Mar 2026).

1. Definition and problem setting

UniPixel was proposed to address a specific limitation of prior large multi-modal models: earlier systems had demonstrated strong image- and video-language understanding, yet had given less attention to fine-grained pixel-level understanding, especially when referring and segmentation had to be integrated into downstream visual reasoning (Liu et al., 22 Sep 2025). The central claim of the model is not merely that it can segment or localize, but that it can flexibly comprehend visual prompt inputs and generate mask-grounded responses, then reuse those masks as intermediate pointers during inference.

The task scope is correspondingly broad. UniPixel is evaluated on pixel-level referring and segmentation, object-centric understanding in images and videos, referred video description, referred video QA, and a dedicated PixelQA benchmark that jointly requires referring, segmentation, and question answering. This task formulation makes mask prediction an internal computational primitive rather than only a terminal output.

A useful way to characterize the model is as a unified referring-segmentation-reasoning system. In the paper’s formulation, “referring” identifies which object is intended, “segmentation” extracts its mask, and “reasoning” conditions subsequent language generation on those mask-grounded object representations. This suggests that UniPixel is designed to collapse a pipeline of separately trained referring, segmentation, and QA modules into a single end-to-end multi-modal architecture.

2. Architectural organization

UniPixel is built on top of a pre-trained multi-modal backbone, Qwen2.5-VL, and comprises three principal components: a prompt encoder, an object memory bank, and a mask decoder (Liu et al., 22 Sep 2025). The visual encoder is ViT-based with dynamic resolution support and tokenizes each video frame into patch embeddings. The text encoder/decoder handles instruction and response generation. The mask decoder is inherited from SAM 2.1 and produces pixel-accurate masks when triggered.

The prompt encoder transforms sparse visual prompts such as points and boxes, as well as dense prompts such as masks, into high-dimensional tokens consumable by the LLM. For a 2D box $(x_1,y_1,x_2,y_2,t)$, the paper specifies embedding via 2D Fourier positional embeddings plus a learnable “corner-type” embedding, followed by projection into the LLM’s hidden space. This prompt pathway is essential because UniPixel is intended to accept visual interactions rather than only text instructions.

The object memory bank is a hashmap that stores object ID to spatio-temporal mask pairs as memory slots. Operationally, when the LLM encounters a <REF> token during referring, it emits a sequence of <SEG> tokens that request mask predictions. The mask decoder uses each <SEG> token’s hidden state, downsampled into two tokens, to produce a mask on the key frame, which is then propagated across frames. These masks are written into the memory bank. Later, when the LLM sees a <MEM> placeholder, that placeholder is replaced by a mask-pooled feature token from the corresponding bank entry.

This architecture turns segmentation outputs into reusable symbolic-visual memory. Rather than reasoning over only global frame tokens, the LLM can attend to object-centric features injected at the locations where <MEM> tokens appear. The result is a pixel-grounded conversational state that supports multi-object and multi-round interaction.

3. Training objectives and pixel-grounded reasoning

UniPixel is trained in three stages, with the final joint loss in stage 3 summarized as

$$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$

Here, $L_{LM}$ is the autoregressive token cross-entropy loss for text output; $L_{\text{focal}}$ and $L_{\text{dice}}$ supervise binary mask prediction; $L_{\text{IoU}}$ is an MAE loss on the predicted IoU score versus ground-truth IoU; and $L_{\text{obj}}$ is the binary cross-entropy loss for the mask-decoder’s objectness head (Liu et al., 22 Sep 2025). During stage 2, an auxiliary referring loss is added:

$$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$

The focal and dice terms are given explicitly in the paper summary. The focal loss is

$$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$

and the dice loss is

$$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$

These objectives enforce alignment between language-side special tokens such as <REF> and <SEG> and the corresponding pixel regions. The paper notes that segmentation and referring losses are weighted heavily, with the example $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$0, to strengthen this coupling.

The model’s reasoning mechanism is centered on the object memory bank. On a <REF> query, UniPixel predicts masks and stores them as memory entries. When the same objects are referenced later through <MEM> tokens, each <MEM> is replaced by a pooled feature extracted from the stored mask via masked-average pooling over the visual encoder’s features, then projected into the LLM. No external tracker or sampler is needed; the mask decoder handles temporal propagation, and the memory bank handles conditioning. The consequence is that self-attention operates over object-centric visual cues rather than only frame-level embeddings.

4. Empirical performance, PixelQA, and ablations

UniPixel is evaluated on 10 benchmarks spanning Reasoning Video Object Segmentation, Referring Video Object Segmentation, Motion-Grounded Video QA, image referring expression segmentation, reasoning segmentation, referring expression comprehension, referred video description, referred video QA, and PixelQA (Liu et al., 22 Sep 2025). Representative results are summarized below.

Benchmark	UniPixel result	Comparator or note
ReVOS (val), Overall JF	62.1	VISA (13B): 50.9
MeViS (valᵘ) JF	53.1	best prior: 45.2
Ref-YouTube JF	70.5	best prior: 63.0
Ref-DAVIS17 JF	74.2	best prior: 70.4
GroundMoRe Acc	37.4	best prior: 27.2
VideoRefer-Bench^Q avg MCQ accuracy	76.3%	VideoRefer: 72.1%

PixelQA is introduced as a benchmark built on the VideoRefer-Bench^Q splits, but with ground-truth masks replaced by single-frame point or box clicks, or a random mix. The model must identify which object was clicked, segment it across all frames, and answer a multi-choice question. For the 3B UniPixel model, the reported results are: point prompts, $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$1, Acc $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$2; box prompts, $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$3, Acc $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$4; mixed (50/50), $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$5, Acc $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$6. The paper also reports that InternVL2 26B and Qwen2-VL 72B achieve approximately $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$7 segmentation because they have no mask head, while their QA scores are approximately $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$8–$$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$9.

The ablations identify which subsystems are responsible for the gains. On PixelQA, referring-only pre-train gives $L_{LM}$0 JF, segmentation-only gives $L_{LM}$1, joint referring+segmentation gives $L_{LM}$2, and adding memory pre-filling/injection yields $L_{LM}$3 JF and $L_{LM}$4 Acc. For the memory mechanism specifically, simple <REF> yields $L_{LM}$5 JF; <REF>+<SEG> auxiliary supervision yields $L_{LM}$6; adding masked pooling gives $L_{LM}$7; and the full memory bank reaches $L_{LM}$8 JF and $L_{LM}$9 Acc. Removing temporal embedding costs approximately $L_{\text{focal}}$0 JF points, while predicting independent frame masks without propagation loses approximately $L_{\text{focal}}$1 JF. Table 11 reports that two hidden tokens per object are optimal, with more tokens giving negligible gains. Multi-stage training also matters: no pre-training yields JF $L_{\text{focal}}$2 on ReVOS, pre-training the prompt encoder or L→M projector gives approximately $L_{\text{focal}}$3, and using both plus stage 3 gives JF $L_{\text{focal}}$4.

The paper’s stated limitations are equally specific. UniPixel is trained on approximately $L_{\text{focal}}$5 M fine-grained samples but does not yet include tasks such as grounded caption generation or extremely long videos. The mask decoder always segments on the first frame and simply propagates forward, so better key-frame selection or bidirectional propagation may improve noisy or occluded scenarios. Inference speed is also bounded by SAM’s mask head, which may not meet real-time VR/AR requirements.

5. SPARROW as a video-grounding extension on UniPixel

SPARROW extends pixel-grounded video MLLMs by addressing spatial precision and temporal consistency, and it is explicitly integrated into UniPixel, GLUS, and VideoGLaMM (Alansari et al., 12 Mar 2026). In the UniPixel video setting summarized in the SPARROW paper, the base system uses a CLIP-based ViT for spatial features, an InternVideo2 stream for temporal context, Qwen2.5-VL with V→L adapters to inject visual tokens and L→V adapters to emit grounding tokens, and a SAM2.1 pixel decoder triggered by a single [SEG] token together with an object-memory bank keyed by <REF>/<MEM> tokens.

SPARROW introduces two components. The first is Target-Specific Tracked Features (TSF). During training only, the method extracts $L_{\text{focal}}$6 temporally varied target crops per referent, encodes them via the spatial encoder $L_{\text{focal}}$7, projects them to LLM token space, and appends them as extra tokens $L_{\text{focal}}$8. The multimodal input is written as

$L_{\text{focal}}$9

TSF construction is described as

$L_{\text{dice}}$0

The second component is a dual-prompt grounding design. The LLM emits parallel [BOX] and [SEG] tokens in a single coarse-to-fine pass. [BOX] drives a Dense-DETR proposal-filtering head that produces coarse boxes $L_{\text{dice}}$1, while [SEG] provides a SAM2 prompt embedding that refines each $L_{\text{dice}}$2 into a mask $L_{\text{dice}}$3. The box branch is defined by ROIAlign and pooling, cross-attention with the [BOX] embedding, and score plus box refinement:

$L_{\text{dice}}$4

$L_{\text{dice}}$5

$L_{\text{dice}}$6

The segmentation branch is

$L_{\text{dice}}$7

Training uses a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs, sourced from HC-STVG, Vid-Sentence, A2D Sentences, LaSOT, MeViS, GOT-10k, and Ref-SAV. The offline pipeline is GroundingDINO → CLDTracker for dense tracks, crop-and-encode spatial features plus K-means with $L_{\text{dice}}$8, and optional SAM2 masks for supervision. Stage 1 tunes only V→L adapters $L_{\text{dice}}$9, the L→V [SEG] adapter $L_{\text{IoU}}$0, and LoRA in the LLM, while keeping backbones and SAM2 frozen, using

$L_{\text{IoU}}$1

Stage 2 pretrains a class-agnostic Deformable-DETR on COCO, Objects365, OpenImages, and V3Det with

$L_{\text{IoU}}$2

then freezes proposer and backbone and fine-tunes only the [BOX] adapter $L_{\text{IoU}}$3 and filtration head with

$L_{\text{IoU}}$4

At inference, no external detectors are used by default; TSF is off, and only the SAM2-based proposer plus filter head run.

On UniPixel, the measured improvements over the released UniPixel$L_{\text{IoU}}$5 follow the original protocols for RVOS, VidSTG, and VideoGCG. On MeViS, J&F improves from $L_{\text{IoU}}$6 to $L_{\text{IoU}}$7, and MeViS$L_{\text{IoU}}$8 J&F improves from $L_{\text{IoU}}$9 to $L_{\text{obj}}$0. On Ref-YTVOS, J&F rises from $L_{\text{obj}}$1 to $L_{\text{obj}}$2; on Ref-DAVIS17, from $L_{\text{obj}}$3 to $L_{\text{obj}}$4. On VidSTG, I-mIoU increases from $L_{\text{obj}}$5 to $L_{\text{obj}}$6. On VideoGCG, mIoU goes from $L_{\text{obj}}$7 to $L_{\text{obj}}$8, Recall from $L_{\text{obj}}$9 to $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$0, METEOR from $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$1 to $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$2, CIDEr from $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$3 to $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$4, and CLAIR from $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$5 to $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$6.

The analysis attributes these gains to complementary effects. TSF supplies explicit, object-centric embeddings drawn from multiple time points, and an ablation reports that train-only TSF yields $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$7 J&F without runtime cost. The [BOX] token provides a coarse geometric prior, while [SEG] adds class-conditioned semantics for fine boundary refinement. The paper further states that optional inference-time TSF can be enabled for small, heavily occluded, or rapidly moving targets at approximately $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$8 ms/target/frame overhead.

6. Separate usage of “UniPixel” in nanoscale OLED research

In a distinct context, the summary of “Individually Addressable Nanoscale OLEDs” uses “UniPixel” to denote a display-architecture concept that unifies nanoscale charge-injection, plasmonic enhancement, and vertical stacking into a single building-block pixel (Zhang et al., 2024). This usage is unrelated to the multi-modal model, but it is technically precise and therefore relevant to the term’s broader scholarly usage.

The device context is an individually addressable subwavelength OLED pixel based on a nanoscale electrode capable of supporting plasmonic modes. The reported proof of principle includes a complete vertical OLED pixel with an individually addressable nanoelectrode of $$L_{\text{stage2}} = L_{LM} + \alpha L_{\text{ref}} + \beta \left(L_{\text{focal}} + L_{\text{dice}} + L_{\text{IoU}} + L_{\text{obj}}\right).$$9 nm$$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$0, as well as hole-only devices with above $$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$1 pixel yield and longtime operation stability. The architecture employs an Au bottom electrode patch overcoated with a conformal insulating layer of hydrogen silsesquioxane, except for a central nano-aperture, thereby suppressing high-field hot spots at edges and corners and forcing charge injection through the planar center of the patch.

Charge transport in the operating range $$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$2 V is described by a Poole–Frenkel–modified space-charge-limited-current model,

$$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$3

with fitted values $$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$4 cm$$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$5V$$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$6s$$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$7 and $$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$8 for the macrojunction, and $$L_{\text{focal}}(p,y) = - \alpha (1-p)^\gamma y \log p - (1-\alpha) p^\gamma (1-y)\log(1-p),$$9 cm$$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$0V$$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$1s$$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$2 and $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$3 for the nanojunction. Empirically, more than $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$4 of nanojunctions, specifically $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$5 of $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$6, showed no filamentation over repeated cycling.

For full OLED nanopixels with a $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$7 nm aperture, the reported turn-on voltage is $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$8 V, the operating range is $$L_{\text{dice}}(\hat M, M^*) = 1 - \frac{2 \cdot |\hat M \cap M^*|}{|\hat M| + |M^*|}.$$9 V to $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$00 V with no electrical failure, the electroluminescence peak is $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$01 nm, the spatial emission FWHM is less than $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$02 nm, and peak EQE reaches up to $$L_{\text{total}} = L_{LM} + \lambda_{\text{focal}} L_{\text{focal}} + \lambda_{\text{dice}} L_{\text{dice}} + \lambda_{\text{IoU}} L_{\text{IoU}} + \lambda_{\text{obj}} L_{\text{obj}}.$$03. The paper’s “UniPixel” concept is thus a nanoscale emitter architecture rather than a language-vision model. A plausible implication is that the shared term reflects a common emphasis on unification, but the two usages belong to separate technical domains: one in pixel-grounded multi-modal reasoning, the other in ultrahigh-density nano-OLED display engineering.