---
title: 'VIRTUE: Visual-Interactive Text-Image Embedder'
url: https://www.emergentmind.com/topics/visual-interactive-text-image-universal-embedder-virtue
type: topic
---

# VIRTUE: Visual-Interactive Text-Image Embedder

VIRTUE, short for **Visual-InteRactive Text-Image Universal Embedder**, is a multimodal representation learning framework that extends text-image embedding with explicit visual-interactive grounding. It is designed to ingest not only images and text but also user-provided visual prompts such as points, bounding boxes, and masks, thereby enabling region-sensitive retrieval and alignment in addition to conventional global embedding tasks. The model combines a frozen segmentation model, **SAM-2**, with a frozen vision-language model, **Qwen2-VL**, and trains the resulting system with contrastive learning to produce a unified embedding vector that is simultaneously context-aware and entity-sensitive [2510.00523].

## 1. Problem Setting and Motivation

Multimodal representation learning models have shown successful operation across complex tasks, and the integration of vision-language models has enabled embedding models with instruction-following capabilities. VIRTUE is motivated by a limitation in existing embedding models: they generally lack visual-interactive capabilities for specifying regions of interest through user inputs such as a point, bounding box, or mask. The underlying claim is that such capabilities, while explored in generative models, had remained unexplored for embedding models prior to VIRTUE [2510.00523].

The framework is intended to address two related objectives. First, it enables **localized grounding of user intent**, allowing an embedding query to refer to a specific entity within an image rather than only to the image as a whole. Second, it allows the model to learn **entity-level information within images** to complement global representations for conventional embedding tasks. In the formulation presented for VIRTUE, region sensitivity is not treated as a separate downstream module, but as part of the representation itself.

A common misconception in multimodal retrieval is that region selection can be approximated adequately by cropping alone. The reported ablations do not support that view: only the SAM2-based segmentation stream yields strong SCaR performance, whereas naïve cropping is substantially weaker. This indicates that VIRTUE treats visual interaction as a structured conditioning signal rather than merely a spatial preprocessing heuristic [2510.00523].

## 2. System Architecture

VIRTUE is described as a **novel two-stream framework**. At its core, it combines three components: a frozen segmentation model, a frozen VLM vision stack, and the VLM’s LLM backbone. The segmentation pathway is responsible for grounding user prompts to specific image regions, while the VLM pathway provides global visual and textual representations.

The first component is a frozen **SAM-2** segmentation model that, given an image and a visual prompt, produces a spatial feature map encoding the indicated region. The second is the frozen vision encoder plus vision-language connector from **Qwen2-VL**, which ingests the raw image to yield global visual embeddings \(H_v\). The third is the VLM’s LLM backbone, which ingests a concatenated sequence of segmentation embeddings \(H_s\), global vision embeddings \(H_v\), and text embeddings \(H_t\), and outputs a unified embedding vector.

The data flow is explicitly specified. The image \(I\) is processed both by SAM-2’s image encoder and by the VLM’s vision encoder. A visual prompt \(P\) is tokenized by SAM-2’s prompt encoder, and SAM-2’s mask decoder combines image and prompt features to produce a \(64\times64\times d_s\) feature map \(F_s\). A lightweight segmentation-language connector, implemented as **Conv2D \(\rightarrow\) MLP**, reduces and projects this map into a sequence \(H_s \in \mathbb R^{|S|\times d}\). In parallel, the VLM vision-language connector produces \(H_v \in \mathbb R^{|v|\times d}\), and the tokenizer produces \(H_t \in \mathbb R^{|t|\times d}\). The model then concatenates \(\bigl[H_s;H_v;H_t\bigr]\) and feeds the result through the LLM to produce a single vector \(\mathbf z\), defined as the final hidden state of the last token [2510.00523].

This architecture places region-grounded and global representations in the same sequence space. A plausible implication is that the model can express scene-level and entity-level cues with a shared embedding interface, rather than requiring separate retrievers or late-fusion mechanisms.

## 3. Visual Prompt Encoding and Region-Grounded Fusion

VIRTUE supports three prompt types: **point clicks**, **bounding boxes**, and **free-form masks**. Point clicks correspond to one or more user-selected pixels. Bounding boxes are represented as rectangles \([x_{\min}, y_{\min}, x_w, y_h]\). Free-form masks allow arbitrary region selection. These prompt modalities are processed by SAM-2’s prompt encoder and combined with image features in the mask decoder [2510.00523].

The prompt-conditioned segmentation feature map is defined as

$$
F_s \;=\; f\bigl(E_i(I),\,E_p(P)\bigr)\;\in\mathbb R^{64\times64\times d_s}.
$$

This feature map is then transformed into a token sequence through a small 2D convolution and two MLP layers:

$$
H_s \;=\;\mathrm{MLP}\Bigl(\,\mathrm{Conv2D}(F_s)\Bigr)\;\in\mathbb R^{|S|\times d}.
$$

The resulting sequence \(H_s\) is prepended to the global vision embeddings \(H_v\) and the text embeddings \(H_t\) before the full sequence is processed by the LLM. The connector is therefore not merely a projection head; it is the mechanism by which segmentation-derived region information enters the language-conditioned embedding space.

For standard image-text tasks where no explicit prompt is available, VIRTUE samples \(N=9\) points uniformly over the image as **pseudo-prompts**. This preserves a consistent input structure across prompted and unprompted training instances. The design suggests that visual prompting is integrated not as a special-case inference feature but as a routine part of the model’s representation-learning pipeline [2510.00523].

## 4. Contrastive Training Formulation

VIRTUE is trained with contrastive learning over mixed mini-batches of matched pairs. Each side of a pair may contain any subset of \(\{H_s, H_v, H_t\}\), allowing both global alignment and region-grounded alignment to be optimized within the same framework. In training, a sample is either an \((\text{Image}, \text{Caption})\) pair or an \((\text{Image} + \text{Prompt}, \text{Caption})\) pair.

Let \(\mathbf z_q\) denote the embedding of a query and \(\mathbf z_t\) the embedding of its matched caption. The training loss is standard InfoNCE with in-batch negatives:

$$
\mathcal L \;=\;
-\,\log\frac{\exp\bigl(\mathrm{sim}(\mathbf z_q,\mathbf z_t)/\tau\bigr)}
{\exp\bigl(\mathrm{sim}(\mathbf z_q,\mathbf z_t)/\tau\bigr)\;+\;\sum_{j\ne t}\exp\bigl(\mathrm{sim}(\mathbf z_q,\mathbf z_j)/\tau\bigr)}\,,
$$

where \(\mathrm{sim}(u,v)\) is cosine similarity and the temperature \(\tau\) is set to \(0.02\) [2510.00523].

Optimization uses **GradCache** to enlarge the effective batch size up to \(1\,024\) despite GPU memory limitations. Within the scope of the reported design, this is significant because embedding quality in contrastive setups is often sensitive to negative diversity. The paper does not claim a novel loss; the contribution lies in the input structure and fusion pathway that allow the same contrastive objective to supervise both global and interactive retrieval.

## 5. SCaR Benchmark

To evaluate visual-interactive embedding, the work introduces **SCaR**, short for **Segmentation-and-Scene Caption Retrieval**, a large-scale image-to-text retrieval benchmark with **1M samples** [2510.00523]. SCaR is intended to test whether a model can retrieve a caption by jointly considering a specified entity and the surrounding scene, rather than recognizing either in isolation.

SCaR is built from five public datasets: **RefCOCO+**, **RefCOCOg**, **VisualGenome**, **COCO-Stuff**, and **ADE20K**. Each dataset contributes images, bounding boxes or masks, and captions of the form `\<object> <relation> <scene>`. For each sample, the query consists of an image \(I\) and a region \(P\), while the candidate set contains one ground-truth caption and nine hard negatives. These negatives are generated by prompting **GPT-4V** with three swap strategies—**global-context swap**, **relation swap**, and **object swap**—applied three times each, with the constraint that each candidate remains plausible under a naïve crop while becoming incorrect when the full context is considered.

The dataset is split into approximately **958 K training** queries and **47 K evaluation** queries. Retrieval is defined as

$$
\hat t
= \arg\max_{c_i\in\mathrm{cands}}
\mathrm{sim}\!\bigl(\phi(I,P),\,\phi(c_i)\bigr),
$$

and performance is measured by **precision@1**. SCaR operationalizes a distinction that standard image-text benchmarks often obscure: the correct caption must depend on both the region-selected object and the broader scene relations. This suggests that SCaR is designed to penalize models that overfit to local crops or purely global context.

## 6. Empirical Performance and Ablation Findings

On the **36 tasks of MMEB**—comprising **20 in-distribution** and **16 out-of-distribution** tasks—**VIRTUE-2B** with a **Qwen2-VL-2B** backbone achieves an overall score of **64.8**, compared with **59.7** for the previous best VLM-based model, **VLM2Vec-2B**, for a **+5.1** point gain [2510.00523]. The reported meta-task breakdown is:

- **Classification**: \(64.1\) vs. \(58.7\) \((+5.4)\)
- **VQA**: \(55.7\) vs. \(49.3\) \((+6.4)\)
- **Retrieval**: \(68.4\) vs. \(65.0\) \((+3.4)\)
- **Grounding**: \(78.7\) vs. \(72.9\) \((+5.8)\)

**VIRTUE-7B** also outperforms all 7B-parameter baselines, improving the overall score from **66.6** to **68.6** \((+2.0)\). The abstract further summarizes these improvements as **3.1\%-8.5\%** across the 36 universal MMEB tasks.

On **SCaR**, the visual-interactive benchmark introduced alongside the model, off-the-shelf **VIRTUE-2B** achieves **30.4\% prec@1**, compared with **24.1\%** for **VLM2Vec-2B**, a **+6.3** improvement. After a brief **+1 000-step** fine-tuning on **SCaR-train**, VIRTUE-2B reaches **56.2\%** versus **46.7\%** for VLM2Vec-2B, a **+9.5** gain. **VIRTUE-7B** shows similar behavior, including **56.9\%** versus **49.4\%** \((+7.5)\) when SCaR-tuned. The abstract characterizes the SCaR gains as **15.2\%-20.3\%** across five visual-interactive tasks.

The ablation suite on **VIRTUE-2B** isolates several factors. First, the **SAM2-based segmentation stream** is materially stronger than naïve cropping or a learned bounding-box prompter; on SCaR, segmentation yields **30.4** versus **25.9** for cropping. Second, image resolution matters: **1 344×1 344** outperforms **672×672** by approximately **5 points**. Third, the segmentation-language connector performs best with **two MLP layers**. Fourth, the number of segmentation tokens \(|S|\) exhibits a trade-off, with **256** providing the best balance between **MMEB (64.8)** and **SCaR (30.4)**. Fifth, among SAM2 scales, the **“base+”** variant works best, balancing speed and quality. The reported conclusion is that region-grounded features materially improve both global and interactive tasks [2510.00523].

## 7. Applications, Limitations, and Prospective Directions

By allowing users to click, box, or mask an object at inference time, VIRTUE enables several applications. The paper identifies **visual-interactive image-to-image retrieval**, where the goal is to retrieve images containing a similar object in a similar scene; **on-the-fly correction of VQA or retrieval predictions** through a quick bounding box; and **fine-grained “entity-first” search** in large galleries without retraining [2510.00523]. These use cases are direct consequences of exposing region grounding as part of the embedding query.

The stated limitations are primarily data- and benchmark-related. The model relies on only **two training sources—MMEB and SCaR—due to compute constraints**. In addition, there is **no large-scale interactive image-to-image benchmark** at present. These constraints are important for interpreting the reported results: the gains are substantial within the available evaluation regime, but the paper does not claim exhaustive coverage of interactive multimodal retrieval settings.

The proposed future directions are explicit. They include incorporating **more diverse sources**, such as **webly-supervised pairs** and **other interactive datasets**; extending the framework to **multi-round or chain-of-thought visual prompting**; and exploring more efficient fusion of region and global cues in **streaming or video settings**. Taken together, these directions frame VIRTUE as a representation-learning architecture that unifies global and localized embeddings within a single VLM by leveraging SAM-2’s promptable segmentation features, while leaving open the broader question of how interactive grounding should scale across richer data regimes and temporal modalities [2510.00523].

Source: https://www.emergentmind.com/topics/visual-interactive-text-image-universal-embedder-virtue