pplx-embed-v1: Multilingual Retrieval Embeddings

• pplx-embed-v1 is a multilingual embedding model that leverages a diffusion-pretrained, bidirectional transformer for web-scale retrieval tasks.
• It employs a multi-stage contrastive learning pipeline—including pair, contextual, and triplet training—to fine-tune dense representations.
• The model supports INT8 and binary quantization for exceptional storage efficiency while delivering competitive performance on key benchmarks.

The pplx-embed-v1 model is a member of the pplx-embed family, a suite of multilingual embedding architectures optimized for web-scale retrieval tasks across text and code. Distinguished by a diffusion-pretrained, bidirectional backbone and a multi-phase contrastive curriculum, pplx-embed-v1 is designed for standard retrieval scenarios, providing dense representations that are both high-performing and storage-efficient. Two main versions–pplx-embed-v1 for passage retrieval and pplx-embed-context-v1 for contextualized, document-aware chunk embeddings–address diverse retrieval settings requiring either global context or efficiency at scale. Their quantitative efficacy is demonstrated on leading retrieval and QA benchmarks as well as in production-scale internal evaluations (Eslami et al., 11 Feb 2026).

1. Diffusion-Based Pretraining Backbone

pplx-embed-v1 initiates from a Qwen3 autoregressive decoder-only LLM (0.6B or 4B parameters) where causal masking is disabled to realize a bidirectional transformer. Pretraining employs a diffusion-style objective in which, at each time step $t\in[0,1]$, each token in sequence $x_0$ is independently replaced by an absorbing [MASK] with probability $t$:

$$q(x_t \mid x_0) = \prod_{l=1}^L \bigl((1-t)\,\delta_{x_t^l,x_0^l} + t\,\delta_{x_t^l,[\text{MASK}]}\bigr)$$

A bidirectional transformer $p_\theta(x_0\mid x_t)$ is trained to reconstruct the masked sequence, retaining the left-shift from the original decoder and beginning predictions at position 2. The ELBO training loss is:

$$\mathcal{L}^{\mathrm{ELBO}}(x_0) = \mathbb{E}_{t\sim \mathcal{U}(0.001,1)}\left[ \frac{1}{t} \mathbb{E}_{q(x_t\mid x_0)}\left[ -\sum_{l=2}^{L} \delta_{x_t^l,[\mathrm{MASK}]}\log p_\theta(x_0^l\mid x_t) \right] \right]$$

Training is conducted for 60k steps with global batch size 1024, sequence length 4096, AdamW optimizer ($\beta_1=0.9$, $\beta_2=0.98$, weight decay 0.01), and a learning rate of $5\times10^{-4}$ (0.6B), $3.16\times10^{-4}$ (4B), with cosine decay scheduled after a 6k-step warmup.

2. Multi-Stage Contrastive Learning Pipeline

Subsequent to diffusion pretraining, the architectural backbone remains, but parameters are fine-tuned through a structured contrastive curriculum.

Stage 1: Pair Training

Web-scale query–document pairs (65.6% English, remainder cross-lingual and code) inform initial training. Embeddings are mean-pooled and quantized to INT8:

$x_0$0

InfoNCE loss with in-batch document and query negatives and false-negative masking is used:

$x_0$1

Three sub-phases transition from English-only to full multilingual training. Training is done for 50k steps (batch size 16,384, sequence length 256, $x_0$2).

Stage 2: Contextual Training

For pplx-embed-context-v1, documents are split into $x_0$3 chunks (256 tokens each), enabling fine-grained, chunk-aware retrieval. Local and global InfoNCE losses are combined:

• Local chunk-level loss ($x_0$4) is a mixture ($x_0$5) of sequence-based and batch-based measures across chunk embeddings.
• Global document-level loss incorporates duplicate-document/similarity threshold masking:

$x_0$6

with $x_0$7 annealed from 0.2 to 0.5 over 7k steps.

Stage 3: Triplet Training

Utilizing triplets with $x_0$8 mined hard negatives per query, a triplet loss is optimized:

$x_0$9

Merging of checkpoints from pair+triplet and contextual phases is performed using Spherical Linear Interpolation (SLERP) to yield the final pplx-embed-v1 model.

3. Model Architecture and Embedding Strategy

Both pplx-embed-v1 and pplx-embed-context-v1 utilize a diffusion-pretrained, bidirectional transformer backbone (Qwen3-0.6B or 4B). No special instruction tokens are introduced. For output, mean-pooling is applied over token representations, enabling efficient global aggregation. Quantization to INT8 is realized using $t$0 and rounding; a straight-through estimator permits quantization-aware fine-tuning. Post-hoc binary quantization is optionally supported ($t$1).

The bidirectional design allows comprehensive context modeling and facilitates mean pooling over token representations, rather than relying on truncated endpoints. For extremely long documents, pplx-embed-context-v1 employs "late chunking": encoding the entire document in a single forward pass (with sliding windows as required), then extracting chunk-wise pooled embeddings, thereby preserving global context across chunks.

Model Variant	Embedding Dimension	INT8 Storage (docs/MB)	Binary Storage (docs/MB)
pplx-0.6B	1024	976	7812
pplx-4B	2560	390	3125

A plausible implication is that binary quantization is particularly attractive at scale, with high storage efficiency and minimal additional loss at the 4B parameter regime.

4. Evaluation Benchmarks and Empirical Results

pplx-embed-v1 and pplx-embed-context-v1 are evaluated on a comprehensive battery of retrieval and question-answering (QA) benchmarks:

• MTEB Multilingual v2 (18 tasks, 146 languages): nDCG@10.
• MTEB Code (12 code-to-code tasks): nDCG@10.
• MIRACL (retrieval across 18 languages): nDCG@10.
• ConTEB (chunk retrieval, 8 datasets): nDCG@10.
• BERGEN (QA, 5 tasks): match@top-5 in a RAG pipeline with Qwen2.5-32B-Instruct.
• ToolRet (35 tool-retrieval tasks): nDCG@10, Precision@10, Recall@10, Comprehensiveness@10.
• Internal web-scale evaluations:
  • PPLXQuery2Query: 100k queries vs. up to 2.4M pseudo-docs (Recall@10/20/100)
  • PPLXQuery2Doc: 15k queries vs. up to 30M real docs (Recall@10/20/100/1000)

Representative scores for pplx-embed-v1 4B INT8:

Benchmark	pplx-4B INT8	Comparator(s)
MTEB (Multilingual)	69.66%	Qwen3-4B 69.60%, Gemini 67.71%
MTEB (Code)	78.73%	Qwen3-4B 80.07%
ConTEB (context-4B)	81.96%	Voyage-context-3 79.45%, Anthropic 72.4%
BERGEN (QA)	Best match on 3/5 tasks	Outperforms Qwen3-4B on 4/5
ToolRet	44.45%	2nd overall, best among INT8 models
PPLXQ2Q (2.4M)	73.46% R@10	Qwen3-4B: 67.90%
PPLXQ2D (30M, Eng)	12.18% R@10, 88.23% R@1000	Qwen3-4B: 8.46%/83.13%

On MIRACL, pplx-embed-4B is slightly below Qwen3-4B on some metrics but exhibits greater storage efficiency.

5. Production Deployment, Efficiency, and Scalability

The pplx-embed models are explicitly designed for high-throughput, large-scale deployment. Both 0.6B and 4B parameter models produce INT8 embeddings by default (with optional binary quantization for maximal efficiency). Storage requirements are substantially reduced: INT8 quantization allows up to 100M+ documents to reside in RAM on commodity servers; binary quantization multiplies practical scale by an order of magnitude.

Embeddings are computed in a single forward pass, achieving throughput of thousands of queries per second per GPU when using FlashAttention-2 and bfloat16. In integration with standard ANN search systems, INT8 encoding reduces memory usage by 75% relative to FP16. Quantization-aware training preserves retrieval accuracy, incurring <1.6 percentage points loss at 4B INT8 and <4.4 at 0.6B INT8. Mean pooling in conjunction with the bidirectional backbone is critical for long-document retrieval and enables the late chunking employed in pplx-embed-context-v1.

Observed in real-world deployments, binary quantization is especially effective at the 4B scale. These characteristics underpin the suitability of pplx-embed-v1 and its variants for industrial-scale retrieval systems requiring both retrieval quality and computational/storage efficiency (Eslami et al., 11 Feb 2026).

6. Context, Implications, and Related Methodologies

The pplx-embed framework demonstrates the effectiveness of combining diffusion-based pretraining with a structured, multi-stage contrastive learning strategy for dense and contextual embeddings. The use of bidirectional attention, mean pooling, quantization, and chunk-aware architecture positions these models at the forefront of retrieval embeddings for both multilingual text and code. The integration of InfoNCE-based objectives (pairwise, contextual, and triplet losses), quantization-aware fine-tuning, and SLERP checkpoint merging reflects an overview of recent advances in representation learning and production-oriented model design.

This suggests that continued pretraining with diffusion processes, particularly when paired with refined contrastive curricula and quantization strategies, can realize new state-of-the-art performance in scalable search settings while maintaining stringent storage and computational constraints. The pplx-embed models provide empirical evidence supporting this approach across a variety of benchmarks and real-world use cases.