UniSE: Universal Screenshot Embeddings

• UniSE is a research concept that defines a unified embedding approach converting screenshots and diverse digital inputs into a single retrieval space.
• It employs dual encoder and multimodal MLLM variants trained via contrastive learning to achieve high recall on retrieval benchmarks.
• Its applications span visualized information retrieval, medical segmentation, multimodal search, and unified speech enhancement.

UniSE is an overloaded research acronym whose meaning depends on domain. In the most fully specified sense, it denotes Universal Screenshot Embeddings, a family of retrieval models introduced within Visualized Information Retrieval (Vis-IR), where text, images, tables, charts, webpages, PDF pages, and repository READMEs are uniformly represented as screenshots and embedded into a shared retrieval space (Liu et al., 17 Feb 2025). In 2025 literature, the same or closely related label is also used for a unified segmentation engine in multiple sclerosis lesion segmentation, for unified speech enhancement formulations, and as shorthand for a unified search embedding or unified soft-effects formulation in multimodal retrieval and image restoration (Zhang et al., 6 Aug 2025).

1. Terminological scope and principal meaning

Within Vis-IR, UniSE is the operational model family that learns dense embeddings for screenshots and for modalities that interact with screenshots, primarily text and composed text conditioned on screenshots (Liu et al., 17 Feb 2025). The underlying premise is that “any information is worth one single screenshot”: instead of maintaining modality-specific parsers and retrievers, heterogeneous inputs are reduced to one visual format and retrieved by similarity in a single embedding space.

This formulation is coupled to two additional artifacts. VIRA is the large-scale training corpus of screenshot–caption and question-answering data, and MVRB is the evaluation benchmark spanning screenshot retrieval, composed screenshot retrieval, screenshot question answering, and open-vocabulary classification (Liu et al., 17 Feb 2025). In this usage, UniSE is therefore not an isolated encoder but one component of a broader retrieval paradigm.

The acronym is reused elsewhere. In multiple sclerosis lesion segmentation, UNISELF explicitly states that its method is “termed ‘UniSE’ for a unified segmentation engine,” referring to a single model and inference pipeline with test-time instance normalization and self-ensembled lesion fusion (Zhang et al., 6 Aug 2025). Other papers use “UniSE” more descriptively: UniECS presents a unified search embedding for e-commerce, UniSER treats “UniSE” as shorthand for Unified Soft Effects, and several speech papers use the term to denote unified speech enhancement or unified enhancement-and-separation formulations rather than a single shared canonical model (Liang et al., 19 Aug 2025).

2. UniSE in Visualized Information Retrieval

Vis-IR defines screenshots as a unified visual entity that preserves both content and layout, thereby avoiding preprocessing steps such as parsing, OCR-based text extraction, table detection, and figure linking (Liu et al., 17 Feb 2025). UniSE is the embedding model family that realizes this idea.

Two variants are defined.

UniSE-CLIP is a dual encoder built on OpenAI CLIP ViT-L/14. A screenshot $s$ is encoded by the visual encoder $\phi_v$ and a text query $t$ by the text transformer $\phi_t$, yielding

$$e_s \leftarrow \phi_v(s), \qquad e_t \leftarrow \phi_t(t).$$

For composed queries conditioned on a screenshot plus text $(s,q)$, UniSE-CLIP uses linear fusion in embedding space,

$e_{s,q} \leftarrow e_s + e_q.$

Embeddings are L2-normalized, and cosine similarity is the default scoring function.

UniSE-MLLM is built on Qwen2-VL-2B. Multimodal inputs are tokenized and interleaved under a uniform template: $t$7 The last hidden state at the special [[EOS](https://www.emergentmind.com/topics/electro-optic-sampling-eos)] token is used as the embedding. Screenshots alone use visual tokens with a task header, while text-only queries use only text tokens. To preserve screenshot detail, the model applies an aspect-ratio-preserving smart resize policy with token budget $M=2500$ and image tokens computed on a $28\times 28$ grid. If original dimensions exceed the budget, resizing is defined by

$$H' = \lfloor H/\beta \rfloor \times 28,\qquad W' = \lfloor W/\beta \rfloor \times 28,$$

with

$\phi_v$0

The retrieval scenarios enabled by UniSE include text-to-screenshot $\phi_v$1, screenshot-to-screenshot $\phi_v$2, screenshot-conditioned retrieval $\phi_v$3, screenshot-to-text $\phi_v$4, and image-to-screenshot retrieval by treating a natural image as a screenshot (Liu et al., 17 Feb 2025). In all cases, scoring uses normalized embeddings and cosine similarity:

$\phi_v$5

3. Training objectives and data regime

UniSE aligns screenshots and text into one embedding space through contrastive learning (Liu et al., 17 Feb 2025). Let $\phi_v$6 be the screenshot encoder and $\phi_v$7 the text or composed-query encoder, with cosine similarity

$\phi_v$8

Given a minibatch of paired instances $\phi_v$9, the source-to-target InfoNCE term is

$t$0

Pretraining uses bidirectional screenshot–caption matching,

$t$1

whereas fine-tuning on retrieval-style question-answering data uses one-way query-to-screenshot contrast,

$t$2

The training corpus, VIRA, contains approximately 20 million instances: 12.96M screenshot–caption pairs and 7.11M question-answering items, consisting of 5.97M q2s tuples and 1.14M sq2s triplets (Liu et al., 17 Feb 2025). Its seven domains are news webpages, Amazon product pages, arXiv paper pages, GitHub project homepages, general PDF documents, charts from ArxivCap, and Wikipedia screenshots. Quality control discards aspect ratios greater than 9 and captions shorter than 100 characters.

Hard negatives are incorporated in addition to in-batch negatives. They are mined by off-the-shelf text and visual embedders, specifically BGE and EVA-CLIP (Liu et al., 17 Feb 2025). OCR is used during data construction to create captions, but not at inference time; retrieval is learned directly from screenshot pixels and paired text.

The training schedule is two-stage. Pretraining uses approximately 13M screenshot–caption pairs, and instruction fine-tuning uses approximately 6M question-answering instances. For both UniSE-CLIP and UniSE-MLLM, the initial learning rate is $t$3. UniSE-CLIP uses batch sizes 8192 for pretraining and 4096 for fine-tuning, while UniSE-MLLM uses 2048 and 1024, respectively. In UniSE-MLLM, the LLM component of Qwen2-VL is fine-tuned with LoRA of rank 32 while other layers remain frozen (Liu et al., 17 Feb 2025).

4. Benchmarking and empirical performance

Evaluation is conducted on MVRB (Massive Visualized IR Benchmark), which organizes tasks into four families: Screenshot Retrieval (SR), Composed Screenshot Retrieval (CSR), Screenshot Question Answering (SQA), and Open-Vocab Classification (OVC) (Liu et al., 17 Feb 2025). Candidate corpora contain approximately 5,000 items with curated hard negatives, and the primary metric is Recall@1.

UniSE-MLLM attains an overall Recall@1 of 55.72, exceeding the strongest screenshot document retriever GME at 48.14 by +7.6 points on average (Liu et al., 17 Feb 2025). Its task-family results are SR 69.64, CSR 54.49, SQA 43.20, and OVC 48.26. UniSE-CLIP, though much smaller at 428M parameters, achieves 34.99 overall, compared with MM-Embed 34.48 and VLM2Vec 32.19.

The paper also reports that conventional OCR-plus-text pipelines remain weaker despite strong text embedders: BGE achieves 41.99 overall and E5-Mistral achieves 45.51 (Liu et al., 17 Feb 2025). The authors interpret this as evidence that screenshots retain layout and visual semantics that are lost in text-only conversion.

Several ablations identify the principal contributors. Caption pretraining alone already yields strong Vis-IR performance, surpassing ColPali by +2.5% overall; question-answer fine-tuning alone is +7.8% over caption-only; combining the two stages yields the best result, +9.6% over caption-only and +1.8% over QA-only. Both q2s and sq2s fine-tuning matter, with sq2s adding +1.8% over q2s alone. Diverse-domain training adds +2.2% on average relative to training on less diverse data, and hard negatives consistently improve performance for both model variants (Liu et al., 17 Feb 2025).

This suggests that UniSE’s gains derive not from a single architectural choice but from the combination of screenshot-native representation, multi-stage alignment, composed-query supervision, and heterogeneous-domain coverage.

5. Serving, efficiency, and operational limits

Both UniSE variants produce single-vector embeddings suitable for ANN retrieval with FAISS or similar indices (Liu et al., 17 Feb 2025). Candidate screenshots or label texts are embedded offline, queries are embedded online, and top-$t$4 results are ranked by cosine similarity. The work notes that standard ANN methods such as IVF, HNSW, and PQ can be used in deployment, although compression and quantization are not explicitly explored.

The efficiency trade-off between the two variants is explicit. UniSE-CLIP is a 428M-parameter dual encoder with 224×224 inputs and is positioned for high-throughput embedding and large-scale indexing. UniSE-MLLM, at 2.21B parameters, is more expressive, especially for composed inputs, but is heavier. On MVRB with approximately 5K candidates, UniSE-MLLM completes evaluation in around 3.5 hours on a single A800-80G GPU (Liu et al., 17 Feb 2025).

The reported failure modes follow directly from the screenshot representation and encoder design. Fixed-resolution encoders such as UniSE-CLIP may underperform on tiny text, very dense tables, or intricate charts where content occupies only a small part of the image. Heavy occlusion, highly stylized layouts, and noisy renderings can also degrade recognition. The current release has limited multilingual coverage, and screenshot question answering that requires external world knowledge rather than screenshot-grounded retrieval is outside the scope of Vis-IR itself (Liu et al., 17 Feb 2025).

6. Other 2025 uses of the acronym

The term UniSE also appears in several unrelated research contexts, usually to denote a unified architecture rather than the screenshot-retrieval model.

In medical image analysis, UNISELF uses “UniSE” to denote a unified segmentation engine for multiple sclerosis lesion segmentation from multicontrast brain MRI (Zhang et al., 6 Aug 2025). It combines test-time instance normalization (TTIN) with self-ensembled lesion fusion (SE-LF) in a 2.5D U-Net. Trained on the ISBI 2015 challenge data, its finalized CondIN+TTIN configuration with $t$5 and $t$6 reaches Score 93.28 ± 7.40 and DSC 0.664 ± 0.141 on the ISBI 2015 test set, while also outperforming benchmark methods across MICCAI 2016, UMCL, and a private multisite dataset under domain shift and missing-contrast conditions (Zhang et al., 6 Aug 2025).

In multimodal commerce retrieval, UniECS describes UniSE as a unified search embedding that supports all retrieval directions across text, image, and multimodal inputs (Liang et al., 19 Aug 2025). Its shared-space model with gated cross-modal fusion uses ViT-B/16 and a 12-layer BERT, produces 256-dimensional normalized embeddings, and reaches R@10 = 0.85 on text-to-image retrieval and 0.87 on image-to-text retrieval on M-BEER, while operating at approximately 0.2B parameters (Liang et al., 19 Aug 2025).

In speech processing, the acronym is used in at least three distinct ways. A decoder-only autoregressive LM-based framework titled UniSE unifies speech restoration, target speaker extraction, and speech separation by conditioning a LLaMA-style transformer on WavLM features and generating BiCodec tokens (Yan et al., 23 Oct 2025). A separate work presents USEF-PNet and UDSEF-PNet as practical unified speech-enhancement systems that handle both conventional SE and personalized SE in one speaker-embedding-free architecture (Huang et al., 18 May 2025). Another paper defines unified speech enhancement and separation (UniSE) as the task achieved by UniVoiceLite, a lightweight audio-visual Wasserstein autoencoder of approximately 2.3M parameters (Park et al., 7 Dec 2025).

A further neighboring usage appears in image restoration: UniSER states that “UniSE” is best understood as shorthand for Unified Soft Effects, meaning the unified treatment of haze, reflections, shadows, and lens flare as semi-transparent occlusions, while UniSER is the actual trained Diffusion-Transformer model (Zhang et al., 18 Nov 2025).

Taken together, these usages show that UniSE is not a globally unique model name. It most specifically designates Universal Screenshot Embeddings in Vis-IR (Liu et al., 17 Feb 2025), but in 2025 it also functions as a recurring shorthand for unified retrieval, segmentation, enhancement, or restoration systems across multiple subfields.