Pplx-Embed: Multilingual Retrieval Embeddings

• pplx-embed is a family of multilingual embedding models that generate dense, contextual retrieval embeddings using a diffusion-pretrained bidirectional Transformer backbone.
• It employs a multi-stage contrastive curriculum and late chunking to optimize both standard and contextualized retrieval tasks with high efficiency.
• Its quantization-friendly design significantly boosts retrieval speed, achieving up to 5× acceleration on CPUs and efficient storage for large-scale applications.

pplx-embed is a family of multilingual embedding models designed for efficient, high-quality web-scale retrieval tasks. It employs a diffusion-pretrained bidirectional Transformer backbone and a multi-stage contrastive curriculum, producing quantization-friendly dense and contextual embeddings. The family includes two main variants: pplx-embed-v1, for standard retrieval, and pplx-embed-context-v1, which yields contextualized chunk-level embeddings that incorporate global document context. pplx-embed targets scalability, retrieval accuracy, and storage/computational efficiency, as demonstrated by benchmark and internal evaluations (Eslami et al., 11 Feb 2026).

1. Model Variants and Scope

The pplx-embed architecture comprises two principal retrieval model types—pplx-embed for standard dense retrieval and pplx-embed-context for contextualized chunk retrieval—each provided at two parameter scales (0.6 B and 4 B). Both variants rely on diffusion-pretrained, bidirectional Transformers, followed by a quantization-aware contrastive training pipeline. The pplx-embed-context variant is specifically designed to encode and retrieve document-level context within passage or chunk-level embeddings, enabling context-sensitive chunk retrieval. Production deployments emphasize end-to-end INT8 and binary quantization support, yielding compact and efficient embedding representations suitable for large-scale, low-latency applications across multilingual corpora.

2. Diffusion Pretraining and Bidirectional Architecture

The foundation of pplx-embed is a bidirectional Transformer derived from a decoder-only LLM (Qwen3-0.6B or Qwen3-4B) with its causal mask disabled. Pretraining is conducted via a discrete absorbing-state diffusion objective. For a sequence $x_0 = (x_0^1, \dots, x_0^L)$, tokens are noised independently over $t \in [0,1]$, decaying to a [MASK] token with probability $t$, according to

$q(x_t\,|\,x_0) = \prod_{l=1}^L \left[(1-t)\,\delta_{x_t^l,x_0^l}+t\,\delta_{x_t^l,[\textsc{MASK}]}\right].$

A bidirectional Transformer $p_\theta$ models the denoising/reverse process. The objective is the evidence lower bound (ELBO)

$$\mathcal{L}^{\rm ELBO}(x_0) = \mathbb{E}_{t\sim\mathcal{U}(0.001,1)}\Bigr[\frac{1}{t}\, \mathbb{E}_{q(x_t|x_0)}\bigl[-\sum_{l=2}^L \delta_{x_t^l,[{\rm MASK}]}\,\log p_\theta(x_0^l|x_t)\bigr]\Bigl].$$

Pretraining uses 250 B tokens (50% English from FineWeb-Edu; 29 other languages from FineWeb2/HQ), with a global batch size of 1,024, sequence length of 4,096, AdamW optimizer, and FlashAttention-2 implementation in bfloat16.

This diffusion-pretrained, non-causal backbone enables bidirectional contextualization, allowing true mean pooling over sequence tokens, and unlocks efficient processing of long documents by supporting late chunking and global context propagation.

3. Quantized Embedding and Mean Pooling

Quantization is a core design element throughout the pplx-embed pipeline. After diffusion pretraining, token hidden states $\{\mathbf{v}_l\}$ are aggregated by true mean pooling over the entire (bidirectional) context: $$\mathbf{e} = \left\lfloor 127\cdot\tanh\left(\frac{1}{L}\sum_{l=1}^L\mathbf{v}_l\right) + \frac{1}{2}\right\rfloor,\quad \mathbf{e}\in\{-127, \ldots, 127\}^d.$$ This construction produces INT8 embeddings. Training uses straight-through gradient estimation to pass through the non-differentiable quantization. Additionally, sign-based ("$\pm$1") binary quantization is supported: $$\operatorname{bin}(x) = \begin{cases} +1 & x\geq 0 \ -1 & x < 0 \end{cases}$$ Embeddings are directly produced in the quantized space, eliminating the need for post-hoc quantization and enabling efficient similarity computation at inference. The combination of mean pooling over bidirectional encodings and quantization yields both accuracy and substantial speed-ups ($t \in [0,1]$0 on GPU, $t \in [0,1]$1 on CPU for INT8 vs fp16; $t \in [0,1]$2 further for binary).

4. Multi-Stage Contrastive Training and Curriculum

After freezing the pretrained backbone, pplx-embed models train projection and quantization heads through a sequence of contrastive learning stages, all using cosine similarity with temperature $t \in [0,1]$3.

4.1 Pairwise Stage

Initial training uses query-document pairs $t \in [0,1]$4 with InfoNCE loss and aggressive in-batch negative mining, masking likely false negatives where similarity to the query exceeds $t \in [0,1]$5. Training proceeds through English-only, cross-lingual, and fully multilingual sub-phases.

4.2 Contextual Stage (for pplx-embed-context)

In the contextual stage, documents are decomposed into $t \in [0,1]$6 chunks. Both local (chunk-level) and global (document-level) contrastive losses are applied. The dual context loss interpolates between chunk discrimination and global document discrimination with a cosine annealed coefficient $t \in [0,1]$7.

4.3 Triplet Stage and Model Merging

A final triplet training phase is performed with hard negative mining. Subsequently, standard pplx-embed checkpoints are merged in embedding space (Slerp interpolation) between the contextual and triplet stages, to preserve both global document context and fine-grained discrimination.

5. Architecture, Inference, and Late Chunking

The elimination of the causal mask at all training and inference stages establishes full-sequence bidirectional attention, a necessary condition for true mean pooling and late chunking. In pplx-embed-context, at inference a long document is encoded in a single forward pass (subject to model context window), after which chunk embeddings are extracted by mean pooling over non-overlapping windows (typically 256 tokens). This late chunking strategy preserves global document context within local chunk embeddings, markedly improving chunk-level retrieval granularity and efficiency, as all chunk embeddings are distilled in one pass.

The model is sized at both 0.6 B and 4 B parameters. Both are natively calibrated to emit INT8 or binary outputs, supporting downstream embedding search implementations without additional quantization steps. ONNX and FlashAttention-2 are used for efficient, low-latency GPU and CPU deployment.

6. Empirical Evaluation and Benchmarking

pplx-embed and pplx-embed-context are extensively benchmarked on standard public evaluation suites and large-scale, real-world internal datasets. Some key results (all INT8 unless noted):

Benchmark	pplx-4B (INT8)	Reference/Comparator	pplx-0.6B (INT8)	Reference/Comparator
MTEB (Multiling. v2)	nDCG@10=69.66	Qwen3-4B: 69.60, Gemini-001: 67.71	65.41	Qwen3-0.6B: 64.65
MTEB (Code, 12 tasks)	78.73	Qwen3-4B: 80.07, Gemini-001: 76.00	75.85	Qwen3-0.6B: 75.42
MIRACL	-	Qwen3-4B better by ~1 pp	Outperforms Qwen3-0.6B	-
ConTEB (pplxcontext-4B)	81.96	Voyage-context-3: 79.45, Anthropic 72.4	76.53	3rd overall
BERGEN (RAG over 24.8M)	Best on 3/5	Outperforms Qwen3-4B on 4/5	Beats Qwen3-4B on 3 tasks	-
ToolRet (35 tasks)	nDCG@10=44.45	GritLM-7B: 41.13, NV-Embed-7B: 42.71	43.05	-

Internal web-scale results demonstrate scalability and recall gains with recall@10 up to 73.46% on 2.4 M query tasks and up to 91.66% recall@1000 on 30 M document multilingual retrieval.

Notably, binary variants lose only $t \in [0,1]$8 percentage points on 4 B and up to 5 points on 0.6 B. Storage efficiency reaches up to 3,125 docs/MB (binary) and 390 docs/MB (INT8, 4B variant), outperforming Gemini-001 (81 docs/MB) and Qwen3-4B (97 docs/MB).

7. Deployment and Real-World Usage

pplx-embed models are engineered for production-scale retrieval. Direct quantization during training enables up to $t \in [0,1]$9 CPU and $t$0 GPU acceleration (INT8) compared to fp16, with binary providing a further $t$1 gain. Storage efficiency supports resource-constrained environments. Empirically, the diffusion-pretrained backbone achieves $t$2 percentage point higher recall than autoregressive alternatives. Real-world search deployments have been validated at corpora up to 1 B pages and 24.8 M passages.

The use of late chunking, global context distillation in chunk embeddings, and highly parallelizable inference pipelines enables efficient retrieval with high accuracy, supporting large-scale search, RAG, and specialized retrieval use cases spanning 146 languages. The native quantization-aware design retains more than $t$3 of full-precision accuracy in the 4 B model.

---

pplx-embed exemplifies the integration of diffusion-derived bidirectional Transformers, quantization-first mean pooling, and staged contrastive curriculum, delivering scalable, production-oriented retrieval embeddings across multilingual and multimodal domains (Eslami et al., 11 Feb 2026).