Pailitao-VL: High-Precision Multi-Modal Retrieval

• Pailitao-VL is a comprehensive multi-modal retrieval system that integrates a unified vision-language embedding model with a generative listwise reranker for industrial search.
• It replaces traditional contrastive learning with absolute ID classification against billions of semantic prototypes, improving instance-level SKU discrimination under noisy conditions.
• The system delivers practical improvements with a 2.4× speedup in throughput and significant GMV uplifts, demonstrating its efficacy in real-time e-commerce search.

Pailitao-VL is a comprehensive multi-modal retrieval system engineered for high-precision, real-time industrial search in Alibaba’s Pailitao e-commerce platform. It is organized as a two-stage stack with a unified vision-language embedding model, Pailitao-VL-Embedding, and a generative Multi-Modal LLM reranker, Pailitao-VL-Reranker. The system is defined by two explicit paradigm shifts: replacing traditional contrastive embedding learning with absolute ID-recognition against billions of semantic prototypes, and replacing isolated pointwise reranking with a listwise compare-and-calibrate policy that combines chunkwise comparison with calibrated absolute relevance scoring. The design is motivated by three industrial constraints identified by the paper: insufficient retrieval granularity, vulnerability to environmental noise, and a prohibitive efficiency-performance gap in large-scale reranking (Chen et al., 14 Feb 2026).

1. Industrial search setting and problem formulation

Pailitao-VL is designed for billion-scale e-commerce retrieval in which each document is multi-modal, $d = (d_{\text{img}}, d_{\text{txt}})$, and each query may be pure image, pure text, or joint image+text. The corpus is denoted $\mathcal{D} = \{d_i\}_{i=1}^L$ with $L \approx 10^9$. The operating environment is explicitly non-curated: images may contain low lighting, cluttered backgrounds, occlusion, low resolution, or watermarks and OCR overlays, while text may include typos, abbreviations such as “BBQ,” informal spellings, and incomplete attribute mentions. At the same time, the platform imposes thousands of QPS and end-to-end latency budgets in the tens of milliseconds, with hard constraints on embedding dimension, index size, and MLLM inference cost (Chen et al., 14 Feb 2026).

The system follows a standard two-stage retrieval pipeline, but redefines both stages. Coarse retrieval computes a query embedding $h_q = \Phi_{\text{embd}}(q)$, searches an ANN index over precomputed document embeddings $h_d = \Phi_{\text{embd}}(d)$, and returns a hundred-scale candidate set:

$$\mathcal{D}_{\text{cand}} = \mathop{\text{arg-top-}N}_{d_i \in \mathcal{D}} \left( \text{sim}\big(\Phi_{\text{embd}}(q), \Phi_{\text{embd}}(d_i)\big)\right).$$

Fine reranking then applies a generative model with scores written in the paper as

$$s_i = P(\text{“Rel”} \mid q, d_i; \Phi_{\text{rank}}).$$

The paper argues that conventional contrastive embeddings are concept-level rather than instance-level, and that pointwise rerankers are both binary-reductive and computationally linear in the number of candidates. Pailitao-VL is therefore intended for exact or near-exact SKU discrimination under severe noise rather than generic semantic matching (Chen et al., 14 Feb 2026).

2. System architecture

The deployed architecture contains two tightly coupled components with distinct roles in recall and precision.

Component	Backbone	Primary role
Pailitao-VL-Embedding	TBStars-VL-3B-based MLLM2vec	Unified embedding for ANN retrieval
Pailitao-VL-Reranker	Qwen3-VL-2B	Generative candidate reranking

Pailitao-VL-Embedding is a 3B-parameter MLLM2vec backbone based on TBStars-VL-3B. It produces a single shared embedding space for queries and documents, supports offline item embedding precomputation, and is used online only for query-side encoding plus ANN search. Pailitao-VL-Reranker is a Qwen3-VL-2B-based model fine-tuned in both pointwise and listwise variants, with the listwise model used as the primary online reranker. Candidate documents are grouped into chunks, scored with chunkwise comparative reasoning, and then merged with an absolute relevance calibration mechanism (Chen et al., 14 Feb 2026).

The architectural separation is functional. The embedding model is optimized for high-throughput recall, with the explicit goal of restricting the search space to roughly 100 candidates while preserving instance-level coverage. The reranker uses full cross-modal attention over image and text tokens to increase precision under industrial noise. The paper emphasizes that the two models are not independent baselines but mutually supporting stages: better embeddings improve candidate purity for the reranker, while improved rerankers feed back into the prototype-construction and annotation pipeline used to train the embedding model (Chen et al., 14 Feb 2026).

3. Unified embedding through absolute ID-recognition

The central departure from standard VL retrieval is the replacement of contrastive learning with absolute ID classification. The paper identifies three problems in InfoNCE-style training for this setting: stochastic supervision from in-batch negatives and false negatives, an objective that constrains only local geometry within a batch, and insufficient high-resolution partitioning of the embedding space for near-duplicate SKUs. Pailitao-VL instead asks a classification question of the form “What is the exact item ID this (image,text) belongs to?” and anchors that classification to a global library of semantic prototypes (Chen et al., 14 Feb 2026).

A prototype is defined as a cluster representing a unique instance-ID derived from multiple views of the same item. Prototype construction is handled by an MLLM-based agent pipeline described as Plan–Propose–Organize–Review. Candidate links are proposed from deterministic metadata such as SKU tree and catalog IDs, behavioral logs such as click and buy trajectories, and generative inference by an MLLM planner. These candidates are organized via manifold structuralization and density-based expansion, then reviewed with multi-view consensus, attribute verification, and cross-view consistency checks. The resulting cluster centroids in embedding space become prototype vectors and define a billion-scale prototype library. A global ID head with weight matrix $W \in \mathbb{R}^{U \times D}$ is then initialized so that each row $w_u$ equals the centroid of prototype $u$, creating what the paper describes as a globally consistent coordinate system for the embedding space (Chen et al., 14 Feb 2026).

Training proceeds in three stages. Stage 1 uses TBStars-VL-3B as a decoder-only embedding generator with a special {EMB} token and Matryoshka Representation Learning over granularities

$\mathcal{D} = \{d_i\}_{i=1}^L$0

This stage is trained on approximately 2B image-text pairs and curated triplets, producing strong concept-level alignment. Stage 2 initializes the global ID head from purified prototype centroids. Stage 3 jointly optimizes the backbone and ID head with an ArcFace-style additive angular margin loss over normalized embeddings and normalized prototype weights, enforcing intra-ID compactness and inter-ID separation. The paper states that model/data hybrid parallelism is used for the billion-scale classifier, that the ID head is tensor-parallelized across GPUs, and that KNN-based sparse similarity considers only the most similar approximately 10% of anchors in the softmax denominator. A dynamic proxy update periodically refreshes the prototype similarity graph during training. At inference time, the ID head is dropped; only the learned embedding is retained, so serving cost remains comparable to a typical MLLM2vec model (Chen et al., 14 Feb 2026).

This training scheme is intended to solve three industrial failure modes simultaneously. Deterministic ID labels remove the randomness of negative sampling; global prototypes act as absolute semantic anchors and reduce embedding drift; and a large ID space creates a high-resolution partition of the manifold so that subtle intra-category distinctions become retrievable. The paper explicitly attributes robustness to noise to the fact that clean catalog images and noisy user photos are forced to collapse around the same prototype anchor during classification (Chen et al., 14 Feb 2026).

4. Listwise compare-and-calibrate reranking

The reranking stage begins from a pointwise baseline, Pailitao-VL-Reranker-Point, in which each query–candidate pair is processed independently with a binary “Yes/No” prompt and optimized with binary cross-entropy. The paper identifies three limitations of this formulation: binary granularity, lack of comparative context, and $\mathcal{D} = \{d_i\}_{i=1}^L$1 MLLM passes per query for $\mathcal{D} = \{d_i\}_{i=1}^L$2 candidates. The proposed replacement, Pailitao-VL-Reranker-List, uses a listwise strategy in which candidates are partitioned into chunks, compared side by side, and calibrated by absolute multi-level relevance scoring (Chen et al., 14 Feb 2026).

The listwise reranker is built around four relevance levels:

• $\mathcal{D} = \{d_i\}_{i=1}^L$3: instance-level match,
• $\mathcal{D} = \{d_i\}_{i=1}^L$4: concept-level match,
• $\mathcal{D} = \{d_i\}_{i=1}^L$5: functional-level match,
• $\mathcal{D} = \{d_i\}_{i=1}^L$6: irrelevant.

For a candidate set $\mathcal{D} = \{d_i\}_{i=1}^L$7, the system partitions the list into $\mathcal{D} = \{d_i\}_{i=1}^L$8 chunks of size $\mathcal{D} = \{d_i\}_{i=1}^L$9, with the paper giving $L \approx 10^9$0 as an example. Each chunk is fed to the MLLM together with index tokens “1”, “2”, …, “m” and an additional [NULL] index. Instead of using only the generated best index, the model extracts logits for all candidate indices and the NULL token, yielding local chunkwise confidence scores $L \approx 10^9$1 and $L \approx 10^9$2. Training uses a pairwise ranking loss over discordant pairs and a NULL-token loss that enforces $L \approx 10^9$3 for instance-level matches and $L \approx 10^9$4 for all others. These terms are averaged across chunks to form the chunkwise component of the ranking objective (Chen et al., 14 Feb 2026).

Because chunkwise logits are not directly comparable across chunks, the same forward pass is reused to obtain absolute relevance scores. The model mean-pools hidden states over visual token segments to obtain a query image embedding $L \approx 10^9$5 and document image embeddings $L \approx 10^9$6, concatenates $L \approx 10^9$7, and applies a small MLP head $L \approx 10^9$8 to produce 4-way logits $L \approx 10^9$9 and relevance distributions $h_q = \Phi_{\text{embd}}(q)$0. Cross-entropy on these 4-level labels defines the absolute scoring loss. The complete reranker objective is

$h_q = \Phi_{\text{embd}}(q)$1

where $h_q = \Phi_{\text{embd}}(q)$2 is the next-token prediction loss for the generated index token (Chen et al., 14 Feb 2026).

At inference time, the hybrid ranking policy first sorts documents within each chunk by chunkwise score, then performs a multi-way merge across chunks using the absolute relevance score as the merge key:

$h_q = \Phi_{\text{embd}}(q)$3

This preserves comparative ordering within chunks while calibrating priorities across chunks. The efficiency improvement is explicit: for 100 queries and 800 pairs on A800, the pointwise model requires 800 passes, 182.25 ms per query, and 43.9 pairs/s, whereas the chunkwise model requires 100 passes, 75.01 ms per query, and 106.65 pairs/s. The paper summarizes this as a $h_q = \Phi_{\text{embd}}(q)$4 speedup in latency and an approximately $h_q = \Phi_{\text{embd}}(q)$5 throughput increase, while also improving retrieval quality (Chen et al., 14 Feb 2026).

5. Data governance, supervision, and serving pipeline

The training pipeline depends on large-scale data curation rather than only model architecture. Embedding training uses approximately 2B image-text pairs and triplets together with the multi-source prototype corpus. Stage 1 performs contrastive MLLM2vec pretraining with MRL on in-domain data. Stage 2 constructs prototypes with the Plan–Propose–Organize–Review workflow. Stage 3 performs angular-margin ID classification against prototype labels. The process is explicitly recursive: improved embeddings and rerankers are reused by the organizer and reviewer components of the agent system, producing cleaner prototypes and labels for the next cycle. The paper describes this feedback process as a “self-loop gain” (Chen et al., 14 Feb 2026).

Reranker supervision is also multi-stage. First, Qwen3-VL-2B is industrially adapted on the prototype library using same-ID positives and different-ID negatives. Second, a reasoning-based labeling pipeline is created from a manually labeled seed set with 4-level relevance labels. A powerful MLLM such as Gemini-3.0 is used to generate chain-of-thought explanations and labels, and the resulting reasoning model is further aligned with GRPO using reward functions measuring label correctness. Third, that reasoning model annotates millions of query-candidate pairs sampled from logs, and a Qwen3-VL-32B expert judge keeps only high-confidence samples on which the reasoning model and judge agree. These filtered labels train both the pointwise and listwise rerankers (Chen et al., 14 Feb 2026).

Serving is organized around asymmetric computation. Item embeddings are computed offline and stored in vector databases such as Faiss or Milvus using ANN structures such as IVF+PQ or HNSW. Production embeddings are 512-dimensional, with the paper stating that 512-D is chosen as a sweet spot for recall versus memory and bandwidth and that Matryoshka training is intended to keep lower-dimensional prefixes strong. Online retrieval computes only the query embedding, performs sub-millisecond ANN search, and then invokes the listwise reranker with chunkwise batching and vLLM serving. The reported online latency is approximately 67 ms per query for Pailitao-VL-Embedding and approximately 76 ms per query for Pailitao-VL-Reranker-List (Chen et al., 14 Feb 2026).

A common misunderstanding would be to interpret the billion-scale prototype head as an online inference dependency. The paper states the opposite: the massive ID head is used only during training and is removed at inference time. Another possible simplification is to view the system as a conventional dual-encoder plus generative cross-encoder. The paper instead frames both stages as redefinitions of standard practice: the embedding is trained as prototype-anchored ID-recognition rather than relative contrastive alignment, and the reranker is designed as listwise compare-and-calibrate rather than isolated pointwise classification (Chen et al., 14 Feb 2026).

6. Evaluation, business impact, and limitations

The offline and online results are reported on industrial data and are the primary basis for the paper’s claim that Pailitao-VL achieves state-of-the-art performance in real-time industrial search.

Evaluation	Metric	Result
Embedding offline	I-HR@1 / C-HR@1 / Average	64.52 / 82.01 / 73.73
Reranker offline (List)	I-HR@1 / C-HR@1 / Average	57.92 / 94.14 / 73.79
Reranker classification (List)	Instance-level F1 / Concept-level F1	74.10 / 91.33
Online A/B	GMV uplift	+2% embedding alone; +6% reranker in standardized product categories; up to +20% in SKU-price comparison

The embedding benchmark uses 5K e-commerce queries against 1.5M documents with instance and concept labels. Adaptive hit-rate metrics are defined as

$h_q = \Phi_{\text{embd}}(q)$6

where $h_q = \Phi_{\text{embd}}(q)$7 is the number of relevant documents for the query. On this benchmark, Pailitao-VL-Embedding improves over TBStars-VL-Embedding from 60.40 to 64.52 on I-HR@1 and from a best baseline 80.38 to 82.01 on C-HR@1. The paper interprets this as evidence that in-domain adaptation alone is not sufficient: CLIP-IDClass-Embedding already outperforms contrastive MLLM2vec baselines, and the best result comes from combining MLLM2vec capacity with ID classification (Chen et al., 14 Feb 2026).

The reranker benchmark uses 2K queries with approximately 50 candidates per query and 4-level labels. At instance level, the best baseline I-HR@1 is 54.48 for CLIP-IDClass-Embedding used as a reranker, while Pailitao-VL-Reranker-Point reaches 56.45 and Pailitao-VL-Reranker-List reaches 57.92. At concept level, the best baseline @1 is 92.08 for Qwen3-VL-Reranker, compared with 92.93 for the pointwise industrial model and 94.14 for the listwise model. On classification metrics, instance-level F1 rises from 60.68 for Qwen3-VL-Reranker to 72.35 for Pailitao-VL-Reranker-Point and 74.10 for Pailitao-VL-Reranker-List, while concept-level F1 rises from 86.66 to 88.51 and then 91.33. The listwise ablation further attributes gains to the cumulative addition of the NULL-token loss, absolute relevance score, and hybrid ranking merge, with the final system improving average score from 72.30 for chunkwise-only training to 73.79 (Chen et al., 14 Feb 2026).

The online A/B results connect model quality to platform outcomes. Pailitao-VL-Embedding alone yields +2% GMV across platform-wide traffic. Pailitao-VL-Reranker-List yields +6% GMV in standardized product categories. In specialized AI scenarios such as SKU-price comparison, the uplift reaches up to +20% GMV. These numbers are accompanied by reported production latencies of 67 ms for the embedding service and 76 ms for the listwise reranker, indicating that the proposed architecture is not only an offline improvement but a deployable real-time system (Chen et al., 14 Feb 2026).

The paper also states several limitations. The embedding paradigm depends heavily on high-quality prototype clusters and label purity; building and maintaining billion-scale prototypes requires substantial engineering and MLLM agents. Training with a $h_q = \Phi_{\text{embd}}(q)$8 ID head at billion-scale pushes GPU memory and communication limits even with sparse KNN optimization. The reranker’s 4-level relevance labels may not capture all business nuances, and personalization or user context is not addressed. The authors position future work around richer reasoning, direct ranking-metric optimization, more efficient serving, and extensions of prototype-based ID embeddings to other domains such as IP protection, scientific literature, and video. A plausible implication is that the paper’s main contribution is not merely a stronger Alibaba-specific stack, but a concrete systems claim: in industrial retrieval, scaling identity resolution and calibrating listwise MLLM judgments can be more important than scaling contrastive negatives or applying pointwise generative scoring alone (Chen et al., 14 Feb 2026).