---
title: 'UniSE: Universal Screenshot Embeddings'
url: https://www.emergentmind.com/topics/unise
type: topic
---

# UniSE: Universal Screenshot Embeddings

UniSE is an overloaded research acronym whose meaning depends on domain. In the most fully specified sense, it denotes **Universal Screenshot Embeddings**, a family of retrieval models introduced within **Visualized Information Retrieval (Vis-IR)**, where text, images, tables, charts, webpages, PDF pages, and repository READMEs are uniformly represented as screenshots and embedded into a shared retrieval space [2502.11431]. In 2025 literature, the same or closely related label is also used for a unified segmentation engine in multiple sclerosis lesion segmentation, for unified speech enhancement formulations, and as shorthand for a unified search embedding or unified soft-effects formulation in multimodal retrieval and image restoration [2508.03982].

## 1. Terminological scope and principal meaning

Within Vis-IR, UniSE is the operational model family that learns dense embeddings for screenshots and for modalities that interact with screenshots, primarily text and composed text conditioned on screenshots [2502.11431]. The underlying premise is that “any information is worth one single screenshot”: instead of maintaining modality-specific parsers and retrievers, heterogeneous inputs are reduced to one visual format and retrieved by similarity in a single embedding space.

This formulation is coupled to two additional artifacts. **VIRA** is the large-scale training corpus of screenshot–caption and question-answering data, and **MVRB** is the evaluation benchmark spanning screenshot retrieval, composed screenshot retrieval, screenshot question answering, and open-vocabulary classification [2502.11431]. In this usage, UniSE is therefore not an isolated encoder but one component of a broader retrieval paradigm.

The acronym is reused elsewhere. In multiple sclerosis lesion segmentation, **UNISELF** explicitly states that its method is “termed ‘UniSE’ for a unified segmentation engine,” referring to a single model and inference pipeline with test-time instance normalization and self-ensembled lesion fusion [2508.03982]. Other papers use “UniSE” more descriptively: UniECS presents a **unified search embedding** for e-commerce, UniSER treats “UniSE” as shorthand for **Unified Soft Effects**, and several speech papers use the term to denote unified speech enhancement or unified enhancement-and-separation formulations rather than a single shared canonical model [2508.13843].

## 2. UniSE in Visualized Information Retrieval

Vis-IR defines screenshots as a unified visual entity that preserves both content and layout, thereby avoiding preprocessing steps such as parsing, OCR-based text extraction, table detection, and figure linking [2502.11431]. UniSE is the embedding model family that realizes this idea.

Two variants are defined.

**UniSE-CLIP** is a dual encoder built on OpenAI CLIP ViT-L/14. A screenshot $s$ is encoded by the visual encoder $\phi_v$ and a text query $t$ by the text transformer $\phi_t$, yielding
$$
e_s \leftarrow \phi_v(s), \qquad e_t \leftarrow \phi_t(t).
$$
For composed queries conditioned on a screenshot plus text $(s,q)$, UniSE-CLIP uses linear fusion in embedding space,
$$
e_{s,q} \leftarrow e_s + e_q.
$$
Embeddings are L2-normalized, and cosine similarity is the default scoring function.

**UniSE-MLLM** is built on Qwen2-VL-2B. Multimodal inputs are tokenized and interleaved under a uniform template:
```text
[Task]: $task, [Query]: $s-tok, $q-tok, [EOS]
```
The last hidden state at the special `[EOS]` token is used as the embedding. Screenshots alone use visual tokens with a task header, while text-only queries use only text tokens. To preserve screenshot detail, the model applies an aspect-ratio-preserving smart resize policy with token budget $M=2500$ and image tokens computed on a $28\times 28$ grid. If original dimensions exceed the budget, resizing is defined by
$$
H' = \lfloor H/\beta \rfloor \times 28,\qquad
W' = \lfloor W/\beta \rfloor \times 28,
$$
with
$$
\beta = \sqrt{\frac{W \times H}{M \times 28 \times 28}}.
$$

The retrieval scenarios enabled by UniSE include text-to-screenshot $(q \to s)$, screenshot-to-screenshot $(s \to s')$, screenshot-conditioned retrieval $(s+q \to s')$, screenshot-to-text $(s \to c)$, and image-to-screenshot retrieval by treating a natural image as a screenshot [2502.11431]. In all cases, scoring uses normalized embeddings and cosine similarity:
$$
\mathrm{score}(q,d) = \frac{f(q)\cdot f(d)}{\|f(q)\|\,\|f(d)\|}.
$$

## 3. Training objectives and data regime

UniSE aligns screenshots and text into one embedding space through contrastive learning [2502.11431]. Let $f(\cdot)$ be the screenshot encoder and $g(\cdot)$ the text or composed-query encoder, with cosine similarity
$$
s(x,y)=\frac{f(x)\cdot g(y)}{\|f(x)\|\,\|g(y)\|}.
$$
Given a minibatch of paired instances $\{(x_i,y_i)\}_{i=1}^N$, the source-to-target InfoNCE term is
$$
L_{x\to y} = - \frac{1}{N}\sum_{i=1}^N
\log \frac{\exp(s(x_i,y_i)/\tau)}
{\sum_{j=1}^N \exp(s(x_i,y_j)/\tau)}.
$$
Pretraining uses bidirectional screenshot–caption matching,
$$
L_{s1}=L_{s\to c}+L_{c\to s},
$$
whereas fine-tuning on retrieval-style question-answering data uses one-way query-to-screenshot contrast,
$$
L_{s2}=L_{q\to s}.
$$

The training corpus, **VIRA**, contains approximately **20 million instances**: **12.96M screenshot–caption pairs** and **7.11M question-answering items**, consisting of **5.97M q2s tuples** and **1.14M sq2s triplets** [2502.11431]. Its seven domains are news webpages, Amazon product pages, arXiv paper pages, GitHub project homepages, general PDF documents, charts from ArxivCap, and Wikipedia screenshots. Quality control discards aspect ratios greater than 9 and captions shorter than 100 characters.

Hard negatives are incorporated in addition to in-batch negatives. They are mined by off-the-shelf text and visual embedders, specifically **BGE** and **EVA-CLIP** [2502.11431]. OCR is used during data construction to create captions, but not at inference time; retrieval is learned directly from screenshot pixels and paired text.

The training schedule is two-stage. Pretraining uses approximately 13M screenshot–caption pairs, and instruction fine-tuning uses approximately 6M question-answering instances. For both UniSE-CLIP and UniSE-MLLM, the initial learning rate is $5\times 10^{-6}$. UniSE-CLIP uses batch sizes 8192 for pretraining and 4096 for fine-tuning, while UniSE-MLLM uses 2048 and 1024, respectively. In UniSE-MLLM, the language model component of Qwen2-VL is fine-tuned with LoRA of rank 32 while other layers remain frozen [2502.11431].

## 4. Benchmarking and empirical performance

Evaluation is conducted on **MVRB (Massive Visualized IR Benchmark)**, which organizes tasks into four families: **Screenshot Retrieval (SR)**, **Composed Screenshot Retrieval (CSR)**, **Screenshot Question Answering (SQA)**, and **Open-Vocab Classification (OVC)** [2502.11431]. Candidate corpora contain approximately 5,000 items with curated hard negatives, and the primary metric is **Recall@1**.

UniSE-MLLM attains an **overall Recall@1 of 55.72**, exceeding the strongest screenshot document retriever **GME** at **48.14** by **+7.6 points on average** [2502.11431]. Its task-family results are **SR 69.64**, **CSR 54.49**, **SQA 43.20**, and **OVC 48.26**. UniSE-CLIP, though much smaller at **428M parameters**, achieves **34.99 overall**, compared with **MM-Embed 34.48** and **VLM2Vec 32.19**.

The paper also reports that conventional OCR-plus-text pipelines remain weaker despite strong text embedders: **BGE** achieves **41.99 overall** and **E5-Mistral** achieves **45.51** [2502.11431]. The authors interpret this as evidence that screenshots retain layout and visual semantics that are lost in text-only conversion.

Several ablations identify the principal contributors. Caption pretraining alone already yields strong Vis-IR performance, surpassing ColPali by **+2.5%** overall; question-answer fine-tuning alone is **+7.8%** over caption-only; combining the two stages yields the best result, **+9.6%** over caption-only and **+1.8%** over QA-only. Both q2s and sq2s fine-tuning matter, with sq2s adding **+1.8%** over q2s alone. Diverse-domain training adds **+2.2%** on average relative to training on less diverse data, and hard negatives consistently improve performance for both model variants [2502.11431].

This suggests that UniSE’s gains derive not from a single architectural choice but from the combination of screenshot-native representation, multi-stage alignment, composed-query supervision, and heterogeneous-domain coverage.

## 5. Serving, efficiency, and operational limits

Both UniSE variants produce **single-vector embeddings** suitable for ANN retrieval with **FAISS** or similar indices [2502.11431]. Candidate screenshots or label texts are embedded offline, queries are embedded online, and top-$K$ results are ranked by cosine similarity. The work notes that standard ANN methods such as IVF, HNSW, and PQ can be used in deployment, although compression and quantization are not explicitly explored.

The efficiency trade-off between the two variants is explicit. **UniSE-CLIP** is a **428M-parameter** dual encoder with **224×224** inputs and is positioned for high-throughput embedding and large-scale indexing. **UniSE-MLLM**, at **2.21B parameters**, is more expressive, especially for composed inputs, but is heavier. On MVRB with approximately 5K candidates, UniSE-MLLM completes evaluation in **around 3.5 hours on a single A800-80G GPU** [2502.11431].

The reported failure modes follow directly from the screenshot representation and encoder design. Fixed-resolution encoders such as UniSE-CLIP may underperform on **tiny text**, **very dense tables**, or **intricate charts** where content occupies only a small part of the image. **Heavy occlusion**, **highly stylized layouts**, and **noisy renderings** can also degrade recognition. The current release has **limited multilingual coverage**, and screenshot question answering that requires external world knowledge rather than screenshot-grounded retrieval is outside the scope of Vis-IR itself [2502.11431].

## 6. Other 2025 uses of the acronym

The term **UniSE** also appears in several unrelated research contexts, usually to denote a unified architecture rather than the screenshot-retrieval model.

In medical image analysis, **UNISELF** uses “UniSE” to denote a **unified segmentation engine** for multiple sclerosis lesion segmentation from multicontrast brain MRI [2508.03982]. It combines **test-time instance normalization (TTIN)** with **self-ensembled lesion fusion (SE-LF)** in a 2.5D U-Net. Trained on the ISBI 2015 challenge data, its finalized **CondIN+TTIN** configuration with $\tau_1=16$ and $\tau_2=7$ reaches **Score 93.28 ± 7.40** and **DSC 0.664 ± 0.141** on the ISBI 2015 test set, while also outperforming benchmark methods across MICCAI 2016, UMCL, and a private multisite dataset under domain shift and missing-contrast conditions [2508.03982].

In multimodal commerce retrieval, UniECS describes **UniSE** as a **unified search embedding** that supports all retrieval directions across text, image, and multimodal inputs [2508.13843]. Its shared-space model with gated cross-modal fusion uses ViT-B/16 and a 12-layer BERT, produces **256-dimensional** normalized embeddings, and reaches **R@10 = 0.85** on text-to-image retrieval and **0.87** on image-to-text retrieval on M-BEER, while operating at approximately **0.2B parameters** [2508.13843].

In speech processing, the acronym is used in at least three distinct ways. A decoder-only autoregressive LM-based framework titled **UniSE** unifies **speech restoration**, **target speaker extraction**, and **speech separation** by conditioning a LLaMA-style transformer on WavLM features and generating BiCodec tokens [2510.20441]. A separate work presents **USEF-PNet** and **UDSEF-PNet** as practical unified speech-enhancement systems that handle both conventional SE and personalized SE in one speaker-embedding-free architecture [2505.12288]. Another paper defines **unified speech enhancement and separation (UniSE)** as the task achieved by **UniVoiceLite**, a lightweight audio-visual Wasserstein autoencoder of approximately **2.3M parameters** [2512.06689].

A further neighboring usage appears in image restoration: **UniSER** states that “UniSE” is best understood as shorthand for **Unified Soft Effects**, meaning the unified treatment of haze, reflections, shadows, and lens flare as semi-transparent occlusions, while **UniSER** is the actual trained Diffusion-Transformer model [2511.14183].

Taken together, these usages show that **UniSE** is not a globally unique model name. It most specifically designates **Universal Screenshot Embeddings** in Vis-IR [2502.11431], but in 2025 it also functions as a recurring shorthand for **unified** retrieval, segmentation, enhancement, or restoration systems across multiple subfields.

Source: https://www.emergentmind.com/topics/unise