REF-VLM: Unified Visual Decoding Paradigm

• REF-VLM is a unified multimodal framework that employs a triplet-based referring paradigm to generate compositional and interpretable visual outputs.
• Its architecture integrates dual vision encoders and a visual prompt encoder to fuse global and local image features for dense prediction tasks.
• Staged multi-task training with symbolic triplet outputs enables robust zero-shot generalization across diverse vision-language applications.

REF-VLM (Triplet-Based Referring Paradigm for Unified Visual Decoding) is an end-to-end multimodal LLM (MLLM) framework designed for unified, compositional, and interpretable visual decoding. By leveraging a triplet-based referring structure and multimodal instruction tuning at scale, REF-VLM achieves state-of-the-art performance across diverse tasks, including object detection, semantic segmentation, keypoint estimation, depth prediction, and grounded vision-language tasks. Its scalable architecture, symbolic output representation, and dense visual prompt capabilities enable robust zero-shot and multi-task generalization without reliance on specialized per-task decoders or vocabulary.

1. Framework and Model Architecture

REF-VLM accepts as input a natural language prompt and a visual input (image, plus optional structured visual prompts), producing a hybrid output that can include both conventional text and dense predictions (e.g., segmentation masks, bounding boxes, keypoint heatmaps, depth maps). The model framework comprises the following major components (Tai et al., 10 Mar 2025):

• Dual Vision Encoders: CLIP-ViT-L (336×336 input) provides strong global image–text alignment, while CLIP-ConvNeXt-L (512×512 input) supplies multi-scale local features. Outputs from both branches are concatenated along the channel axis to form a unified feature pyramid.
• Visual Prompt (VPT) Encoder: Mask-Guided Aggregation fuses user-provided visual prompts (points, boxes, scribbles, masks) with image features by patch-wise multiplication and summation, supporting fine-grained spatial conditioning. Explicit additive cosine positional encoding injects further spatial context. No parameters are introduced in this module.
• Text Decoder (LLM): Vicuna-1.5-7B serves as the backbone LLM, with a projector mapping the vision and VPT outputs into its embedding space. Input sequences include text tokens with ⟨image⟩ and ⟨VPT⟩ placeholders, replaced during training/inference by the corresponding embeddings.
• Visual Unit Decoders: Specialized transformer-heads decode latent “reference” (⟨REF⟩) tokens into boxes (DETR-style), masks (MaskFormer-style), or keypoints (DETR-style with OKS/L2 losses). Latent Embeddings Routing ensures one-to-one mapping between LLM outputs and visual predictions. Parallel Group Hungarian matching associates predictions and ground-truth for dense outputs.
• Stepwise Processing:

1. Image processed by both vision encoders → feature pyramid
2. User prompt and visual prompt mapped → text tokens and VPT embeddings
3. Fusion into text sequence; LLM generates text intermixed with special tokens (⟨REF⟩, ⟨Phrase⟩, ⟨Unit⟩)
4. Each target instance referenced by a ⟨REF⟩ token is routed to the relevant dense decoder for output

2. Triplet-Based Referring Paradigm (TRP)

The central innovation is the Triplet-Based Referring Paradigm (TRP), which structures dense predictions as compositional symbolic triplets, explicitly decoupling:

• Concept: the semantic phrase (e.g., “dog,” “left tire”)
• DecodingType: output modality (box, mask, keypoint, depth)
• TargetRefs: ordered list of ⟨REF⟩ tokens, each linked to a specific visual instance

Triplet outputs use symbolic delimiters for parsing. For example:

<Phrase>dogs</Phrase>(<Unit>box</Unit>[0]<REF>[1]<REF>)

This format provides unambiguous mapping between context and output, supports multi-instance and hierarchical labeling, and is extensible (e.g., allows new unit types such as <Unit>depth</Unit> to be incorporated without retraining the vocabulary). The TRP design directly supports multi-task and multi-granular reasoning within a unified autoregressive sequence (Tai et al., 10 Mar 2025).

3. VT-Instruct: Multi-Task Instruction Following Dataset

REF-VLM is trained on the VT-Instruct dataset, a large-scale corpus (>100M multimodal dialogues) tailored for multi-task visual decoding and instruction following. VT-Instruct covers 25 task types, including:

• Dense Tasks: keypoint detection, segmentation, depth estimation
• Referring Tasks: REC, RES, REG
• Visual QA and Captioning: general understanding, scene graph, conversation
• Interactive Grounding: IG-Box, IG-Mask, IG-Keypoint
• Open-Vocabulary Segmentation and Detection: OVS, OVD, FOVS, FOVD

Visual input prompts include points, boxes, scribbles, and masks; visual output units comprise boxes, keypoints, masks, and depth. Multimodal dialogues are generated using reasoning steps (“VD-CoT”), and outputs are always expressed as triplets with reference tokens, facilitating direct supervision for both text and structured outputs.

Example GCG-Mask dialogue:

<Phrase>Two men</Phrase>(<Unit>box</Unit>[0]<REF>[1]<REF>) wearing …

This design supports grounded open-ended conversation, descriptive vision-language output, and fine-grained dense prediction in a single output sequence.

4. Training Objectives and Optimization

The model is optimized using a combined multi-task objective (Tai et al., 10 Mar 2025):

$$\mathcal{L}_{\rm REF-VLM} = \lambda_0\,\mathcal{L}_{\rm LLM} \;+\;\sum_{i=1}^n \lambda_i\,\mathcal{L}_{\rm Decoder_i}$$

• LLM Loss: Cross-entropy next-token loss over the generated text and all special tokens
• Decoder Losses:
  • Box decoder: L1 regression plus Generalized IoU
  • Mask decoder: binary cross-entropy plus Dice loss
  • Keypoint decoder: L2 OKS loss plus auxiliary classification

Optimal performance is achieved via a staged training schedule:

• Stage 1: Freeze LLM and vision encoders, train only the projection layer on captioning/VQA tasks.
• Stage 2: Unfreeze all modules except the keypoint decoder, enable full multi-task loss.
• Stage 3: Add and train keypoint decoder and optional external plugins (SAM, Grounding DINO, UniPose).

Curriculum weighting (e.g., λ{L1}=5, λ{GIoU}=2) ensures balancing of distinct objectives.

5. Experimental Results and Comparative Evaluation

REF-VLM demonstrates state-of-the-art performance across multiple standard benchmarks (Tai et al., 10 Mar 2025):

Task	REF-VLM Result	Notable Prior
Flickr30k (Caption CIDEr)	96.0	N/A
NoCaps (Caption CIDEr)	122.4	N/A
VQAv2 (Acc)	81.6%	N/A
OKVQA (Acc)	62.4%	N/A
RefCOCOg REG (CIDEr/Meteor)	119.1/21.6	118.5/21.2
GCG-Mask (GranD_f Val CIDEr)	56.9	<56.9
REC ([email protected], RefCOCO Test-A/B)	93.7/89.1%	N/A
COCO-Interactive (cIoU, best variant)	~84%	<84%
ADE20k Zero-Shot (mAP_S)	16.7	<16.7
COCO Detection Zero-Shot (mAP_S)	26.7	<26.7

Ablation studies show:

• Group Hungarian matching improves RES cIoU by 0.66 points.
• Visual-Prompt Encoder design influences REG CIDEr (mask tokening can degrade performance).
• Chain-of-Thought reasoning within VD-CoT boosts freeform segmentation and grounded GCG tasks.

REF-VLM meta and external plugin variants outperform specialist and generalist MLLMs (e.g., GLaMM, SAM, ODISE, VisionLLMv2-Chat) across all reporting metrics, confirming the effectiveness of symbolic triplet output and multi-modal prompt fusion at scale.

6. Implementation and Scalability

Model size is ∼248M parameters (excluding the Vicuna-7B backbone LLM), with 199M parameters in frozen dual visual encoders and a 4,096-dim, 2-layer learned projector. Task-specific meta-decoders are compact (box: ∼15M, mask: ∼20M, keypoint: ∼5M). Training employs 64×A800 (80GB) GPUs in stages with AdamW optimizer and precision scheduling. All operations are fully end-to-end, supporting large-batch scalable learning of text-driven visual decoding with strong generalization to new tasks and visual structures (Tai et al., 10 Mar 2025).

7. Significance and Impact

REF-VLM establishes a general-purpose paradigm for unified visual task decoding, fusing dense output reasoning, explicit symbolic structure, and instruction following within a single transformer architecture. The TRP format ensures compositionality, extensibility, and interpretability, directly bridging vision-language understanding and downstream dense prediction. The model’s robust zero-shot, open-vocabulary, and multi-prompt capabilities render it suitable for broad deployment in research and real-world evaluation pipelines, including image captioning, interactive segmentation, multi-object detection, and grounded vision-language dialogue (Tai et al., 10 Mar 2025).