VIRTUE: Visual-Interactive Text-Image Embedder

• The paper introduces a unified embedding framework that integrates explicit visual-interactive region grounding with conventional global text-image representations to enhance tasks like retrieval and VQA.
• The model employs a novel two-stream architecture combining SAM-2’s segmentation features and Qwen2-VL’s global vision-language embeddings, fused through a Conv2D-to-MLP connector.
• Contrastive training over mixed modality inputs, evaluated on benchmarks like SCaR, significantly improves entity-level and scene-level alignment.

VIRTUE, short for Visual-InteRactive Text-Image Universal Embedder, is a multimodal representation learning framework that extends text-image embedding with explicit visual-interactive grounding. It is designed to ingest not only images and text but also user-provided visual prompts such as points, bounding boxes, and masks, thereby enabling region-sensitive retrieval and alignment in addition to conventional global embedding tasks. The model combines a frozen segmentation model, SAM-2, with a frozen vision-LLM, Qwen2-VL, and trains the resulting system with contrastive learning to produce a unified embedding vector that is simultaneously context-aware and entity-sensitive (Wang et al., 1 Oct 2025).

1. Problem Setting and Motivation

Multimodal representation learning models have shown successful operation across complex tasks, and the integration of vision-LLMs has enabled embedding models with instruction-following capabilities. VIRTUE is motivated by a limitation in existing embedding models: they generally lack visual-interactive capabilities for specifying regions of interest through user inputs such as a point, bounding box, or mask. The underlying claim is that such capabilities, while explored in generative models, had remained unexplored for embedding models prior to VIRTUE (Wang et al., 1 Oct 2025).

The framework is intended to address two related objectives. First, it enables localized grounding of user intent, allowing an embedding query to refer to a specific entity within an image rather than only to the image as a whole. Second, it allows the model to learn entity-level information within images to complement global representations for conventional embedding tasks. In the formulation presented for VIRTUE, region sensitivity is not treated as a separate downstream module, but as part of the representation itself.

A common misconception in multimodal retrieval is that region selection can be approximated adequately by cropping alone. The reported ablations do not support that view: only the SAM2-based segmentation stream yields strong SCaR performance, whereas naïve cropping is substantially weaker. This indicates that VIRTUE treats visual interaction as a structured conditioning signal rather than merely a spatial preprocessing heuristic (Wang et al., 1 Oct 2025).

2. System Architecture

VIRTUE is described as a novel two-stream framework. At its core, it combines three components: a frozen segmentation model, a frozen VLM vision stack, and the VLM’s LLM backbone. The segmentation pathway is responsible for grounding user prompts to specific image regions, while the VLM pathway provides global visual and textual representations.

The first component is a frozen SAM-2 segmentation model that, given an image and a visual prompt, produces a spatial feature map encoding the indicated region. The second is the frozen vision encoder plus vision-language connector from Qwen2-VL, which ingests the raw image to yield global visual embeddings $H_v$. The third is the VLM’s LLM backbone, which ingests a concatenated sequence of segmentation embeddings $H_s$, global vision embeddings $H_v$, and text embeddings $H_t$, and outputs a unified embedding vector.

The data flow is explicitly specified. The image $I$ is processed both by SAM-2’s image encoder and by the VLM’s vision encoder. A visual prompt $P$ is tokenized by SAM-2’s prompt encoder, and SAM-2’s mask decoder combines image and prompt features to produce a $64\times64\times d_s$ feature map $F_s$. A lightweight segmentation-language connector, implemented as Conv2D $\rightarrow$ MLP, reduces and projects this map into a sequence $H_s \in \mathbb R^{|S|\times d}$. In parallel, the VLM vision-language connector produces $H_s$0, and the tokenizer produces $H_s$1. The model then concatenates $H_s$2 and feeds the result through the LLM to produce a single vector $H_s$3, defined as the final hidden state of the last token (Wang et al., 1 Oct 2025).

This architecture places region-grounded and global representations in the same sequence space. A plausible implication is that the model can express scene-level and entity-level cues with a shared embedding interface, rather than requiring separate retrievers or late-fusion mechanisms.

3. Visual Prompt Encoding and Region-Grounded Fusion

VIRTUE supports three prompt types: point clicks, bounding boxes, and free-form masks. Point clicks correspond to one or more user-selected pixels. Bounding boxes are represented as rectangles $H_s$4. Free-form masks allow arbitrary region selection. These prompt modalities are processed by SAM-2’s prompt encoder and combined with image features in the mask decoder (Wang et al., 1 Oct 2025).

The prompt-conditioned segmentation feature map is defined as

$H_s$5

This feature map is then transformed into a token sequence through a small 2D convolution and two MLP layers:

$H_s$6

The resulting sequence $H_s$7 is prepended to the global vision embeddings $H_s$8 and the text embeddings $H_s$9 before the full sequence is processed by the LLM. The connector is therefore not merely a projection head; it is the mechanism by which segmentation-derived region information enters the language-conditioned embedding space.

For standard image-text tasks where no explicit prompt is available, VIRTUE samples $H_v$0 points uniformly over the image as pseudo-prompts. This preserves a consistent input structure across prompted and unprompted training instances. The design suggests that visual prompting is integrated not as a special-case inference feature but as a routine part of the model’s representation-learning pipeline (Wang et al., 1 Oct 2025).

4. Contrastive Training Formulation

VIRTUE is trained with contrastive learning over mixed mini-batches of matched pairs. Each side of a pair may contain any subset of $H_v$1, allowing both global alignment and region-grounded alignment to be optimized within the same framework. In training, a sample is either an $H_v$2 pair or an $H_v$3 pair.

Let $H_v$4 denote the embedding of a query and $H_v$5 the embedding of its matched caption. The training loss is standard InfoNCE with in-batch negatives:

$H_v$6

where $H_v$7 is cosine similarity and the temperature $H_v$8 is set to $H_v$9 (Wang et al., 1 Oct 2025).

Optimization uses GradCache to enlarge the effective batch size up to $H_t$0 despite GPU memory limitations. Within the scope of the reported design, this is significant because embedding quality in contrastive setups is often sensitive to negative diversity. The paper does not claim a novel loss; the contribution lies in the input structure and fusion pathway that allow the same contrastive objective to supervise both global and interactive retrieval.

5. SCaR Benchmark

To evaluate visual-interactive embedding, the work introduces SCaR, short for Segmentation-and-Scene Caption Retrieval, a large-scale image-to-text retrieval benchmark with 1M samples (Wang et al., 1 Oct 2025). SCaR is intended to test whether a model can retrieve a caption by jointly considering a specified entity and the surrounding scene, rather than recognizing either in isolation.

SCaR is built from five public datasets: RefCOCO+, RefCOCOg, VisualGenome, COCO-Stuff, and ADE20K. Each dataset contributes images, bounding boxes or masks, and captions of the form \<object> <relation> <scene>. For each sample, the query consists of an image $H_t$1 and a region $H_t$2, while the candidate set contains one ground-truth caption and nine hard negatives. These negatives are generated by prompting GPT-4V with three swap strategies—global-context swap, relation swap, and object swap—applied three times each, with the constraint that each candidate remains plausible under a naïve crop while becoming incorrect when the full context is considered.

The dataset is split into approximately 958 K training queries and 47 K evaluation queries. Retrieval is defined as

$H_t$3

and performance is measured by precision@1. SCaR operationalizes a distinction that standard image-text benchmarks often obscure: the correct caption must depend on both the region-selected object and the broader scene relations. This suggests that SCaR is designed to penalize models that overfit to local crops or purely global context.

6. Empirical Performance and Ablation Findings

On the 36 tasks of MMEB—comprising 20 in-distribution and 16 out-of-distribution tasks—VIRTUE-2B with a Qwen2-VL-2B backbone achieves an overall score of 64.8, compared with 59.7 for the previous best VLM-based model, VLM2Vec-2B, for a +5.1 point gain (Wang et al., 1 Oct 2025). The reported meta-task breakdown is:

• Classification: $H_t$4 vs. $H_t$5 $H_t$6
• VQA: $H_t$7 vs. $H_t$8 $H_t$9
• Retrieval: $I$0 vs. $I$1 $I$2
• Grounding: $I$3 vs. $I$4 $I$5

VIRTUE-7B also outperforms all 7B-parameter baselines, improving the overall score from 66.6 to 68.6 $I$6. The abstract further summarizes these improvements as 3.1\%-8.5\% across the 36 universal MMEB tasks.

On SCaR, the visual-interactive benchmark introduced alongside the model, off-the-shelf VIRTUE-2B achieves 30.4\% prec@1, compared with 24.1\% for VLM2Vec-2B, a +6.3 improvement. After a brief +1 000-step fine-tuning on SCaR-train, VIRTUE-2B reaches 56.2\% versus 46.7\% for VLM2Vec-2B, a +9.5 gain. VIRTUE-7B shows similar behavior, including 56.9\% versus 49.4\% $I$7 when SCaR-tuned. The abstract characterizes the SCaR gains as 15.2\%-20.3\% across five visual-interactive tasks.

The ablation suite on VIRTUE-2B isolates several factors. First, the SAM2-based segmentation stream is materially stronger than naïve cropping or a learned bounding-box prompter; on SCaR, segmentation yields 30.4 versus 25.9 for cropping. Second, image resolution matters: 1 344×1 344 outperforms 672×672 by approximately 5 points. Third, the segmentation-language connector performs best with two MLP layers. Fourth, the number of segmentation tokens $I$8 exhibits a trade-off, with 256 providing the best balance between MMEB (64.8) and SCaR (30.4). Fifth, among SAM2 scales, the “base+” variant works best, balancing speed and quality. The reported conclusion is that region-grounded features materially improve both global and interactive tasks (Wang et al., 1 Oct 2025).

7. Applications, Limitations, and Prospective Directions

By allowing users to click, box, or mask an object at inference time, VIRTUE enables several applications. The paper identifies visual-interactive image-to-image retrieval, where the goal is to retrieve images containing a similar object in a similar scene; on-the-fly correction of VQA or retrieval predictions through a quick bounding box; and fine-grained “entity-first” search in large galleries without retraining (Wang et al., 1 Oct 2025). These use cases are direct consequences of exposing region grounding as part of the embedding query.

The stated limitations are primarily data- and benchmark-related. The model relies on only two training sources—MMEB and SCaR—due to compute constraints. In addition, there is no large-scale interactive image-to-image benchmark at present. These constraints are important for interpreting the reported results: the gains are substantial within the available evaluation regime, but the paper does not claim exhaustive coverage of interactive multimodal retrieval settings.

The proposed future directions are explicit. They include incorporating more diverse sources, such as webly-supervised pairs and other interactive datasets; extending the framework to multi-round or chain-of-thought visual prompting; and exploring more efficient fusion of region and global cues in streaming or video settings. Taken together, these directions frame VIRTUE as a representation-learning architecture that unifies global and localized embeddings within a single VLM by leveraging SAM-2’s promptable segmentation features, while leaving open the broader question of how interactive grounding should scale across richer data regimes and temporal modalities (Wang et al., 1 Oct 2025).