Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems

Abstract: We benchmark Google Embeddings (GE2), a Vertex-AI-hosted bi-encoder with 2,048-token context and explicit task-type conditioning, against five open-source alternatives: BGE-M3, E5-large, Multilingual-E5-large (mE5-L), LaBSE, and Paraphrase-Multilingual-MPNet (mMPNet). Evaluation covers four BEIR subsets, a synthetic Italian RAG corpus, a chunking ablation considering 5 sizes of tokens with three strategies, and per-query latency on commodity CPU hardware. GE2 ranks first on every task, achieving BEIR avg.nDCG@10 = 0.638 and IT-RAG-Bench nDCG@10 = 0.282, but at 231.6 ms median latency, it is roughly 14x slower than the fastest local models. mE5-L reaches within 0.003 nDCG of GE2 on Italian at 31 ms, making it the preferred option when sub-100 ms SLAs matter. A more striking finding concerns LaBSE, which, despite widespread multilingual deployment scores 0.188 average nDCG@10 on BEIR, below every dedicated retrieval model including mMPNet. Chunking experiments show that all six models saturate at 32-token chunks on our corpus, with semantic chunking providing measurable gains only at 16 tokens.

• The paper demonstrates that GE2, with its extended context capacity and asymmetric task-type conditioning, delivers the highest retrieval quality across BEIR benchmarks.
• Experimental results reveal GE2’s significant advantage in heterogeneous domains despite higher latency, while mE5-L offers near-equivalent performance in short, multilingual settings.
• The study underscores the misalignment of sentence-similarity models like LaBSE for RAG tasks, emphasizing the need for task-specific architectural choices and effective chunking strategies.

Comparative Evaluation of Google Embeddings 2 and Open-Source Models for Multilingual Dense Retrieval and RAG

Introduction

The paper "Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems" (2605.23618) delivers a systematic assessment of Google Embeddings 2 (GE2), a Vertex-AI-hosted bi-encoder incorporating explicit task-type conditioning and extended context capacity, juxtaposed with five open-source alternatives (BGE-M3, E5-large, Multilingual-E5-large [mE5-L], LaBSE, and Paraphrase-Multilingual-MPNet [mMPNet]). The evaluation spans four BEIR subsets, a synthetic Italian RAG corpus (IT-RAG-Bench), chunking ablation at five token granularities across three strategies, and per-query latency measured on commodity CPUs. The analysis foregrounds the impact of model architectural choices, training objectives, and operational deployment constraints (latency, language coverage, input granularity) on retrieval-augmented generation (RAG) pipeline effectiveness.

Model and Architectural Analysis

GE2 distinguishes itself architecturally via a 2,048-token context window and asymmetric task-type conditioning (RETRIEVAL_QUERY vs. RETRIEVAL_DOCUMENT). The open-source models are clustered into three paradigms: retrieval-focused (E5-large, mE5-L), multi-task/multi-granularity (BGE-M3), and sentence similarity (LaBSE, mMPNet). Only GE2 implements asymmetric encoding at the representation level, conferring an inductive bias beneficial where query-document surfaces diverge. E5 variants leverage contrastive objectives with instruction prefixes to emulate task variance; BGE-M3 optimizes for dense/sparse/multi-vector retrieval but demonstrates suboptimal zero-shot performance. Sentence similarity models, notably LaBSE, are shown to exhibit inductive biases misaligned for passage retrieval, as evidenced by systematically degraded ranking quality.

Experimental Results and Latency-Quality Trade-Off

The experiments reveal GE2 as the highest-performing model across BEIR subsets (average nDCG@10 = 0.638), with a pronounced advantage on long-document and heterogeneous-query tasks.

Figure 1: The latency–quality Pareto frontier for BEIR retrieval; GE2 is Pareto-optimal in quality but a latency outlier (231.6 ms), while all local models cluster below 32 ms.

The quality gap is especially prominent in TREC-COVID (GE2: 0.799, mE5-L: 0.702), due to GE2's capacity to encode full biomedical passages without truncation, and in FiQA (GE2: 0.582, mE5-L: 0.438), where financial queries and passages demonstrate substantial surface divergence. BGE-M3 underperforms mE5-L (+0.109 nDCG delta), underscoring the zero-shot penalty incurred by multi-task, multi-head training. LaBSE's performance is consistently and markedly inferior (average nDCG@10: 0.188), confirming the methodological misalignment when sentence similarity models are applied to passage retrieval.

For Italian dense retrieval (IT-RAG-Bench), mE5-L achieves a near-equivalent nDCG@10 (0.279 vs. GE2: 0.282) at a fraction of the latency (31 ms vs. GE2's 231.6 ms). E5-large remains competitive due to corpus template design but should not be interpreted as generally robust to Italian domains.

Latency results exhibit two clusters among local models (16.6 ms for compact architectures, 31 ms for larger ones), comfortably sub-100 ms and therefore SLA-compatible for interactive RAG deployments. GE2, by contrast, incurs an API-induced latency overhead and higher variance, driven primarily by network and autoscaling factors rather than model depth.

Chunking Ablation and Corpus-Driven Effects

Chunking strategy ablation in IT-RAG-Bench exposes two dominant effects:

Figure 2: nDCG@10 versus chunk size and strategy; all models saturate at $\geq$32 tokens, with quality collapse below 16; semantic chunking outperforms fixed chunking at 16 tokens, except for LaBSE.

All models reach $\geq$95\% of saturation at 32 tokens, reflecting passage-length alignment in IT-RAG-Bench. Below 16 tokens, nDCG collapses across all strategies due to semantic fragmentation. Semantic chunking yields measurable quality gains at 16 tokens for retrieval-oriented architectures (GE2 +0.09, mE5-L +0.075), preserving clause boundaries and local coherence. LaBSE's performance is invariant above 16 tokens, reflecting a hard ceiling imposed by its symmetric similarity objective and its inability to capture asymmetric query-document relevance patterns.

Analysis of Architectural and Training Objective Implications

The paper elucidates the critical dependence of retrieval effectiveness on both architectural and training-objective alignment. GE2's task-type conditioning instantiates an advantageous asymmetry, especially when queries and passages are lexically or structurally divergent. mE5-L, trained contrastively with multilingual web data and instruction prefixing, reliably matches GE2 on short, language-homogeneous corpora. Multi-task architectures (BGE-M3) do not confer zero-shot retrieval advantages, substantiating the trade-off between retrieval and ancillary task performance observed in prior literature.

The pronounced underperformance of LaBSE anchors a bold claim: sentence-similarity models, while robust for alignment and bitext mining, are fundamentally ill-suited for asymmetric passage retrieval tasks in RAG pipelines, regardless of multilingual capacity. This contradicts common practitioner assumptions and should prompt migration to contrastive retrieval models.

Practical Recommendations and Theoretical Implications

For practitioners, mE5-L is recommended as the default backbone for multilingual RAG scenarios with short passages and homogeneous domains, given its near-optimal retrieval quality and superior latency profile. GE2 is justified only when extended context or asymmetric query-document domains are present and latency constraints are relaxed. The empirical evidence decisively contraindicates LaBSE and mMPNet for retrieval use.

From a theoretical standpoint, the results reinforce an information-theoretic bottleneck in RAG: irrecoverable retrieval errors delimit downstream generative capacity, and the importance of task-aligned, asymmetric encoding becomes apparent as document and query distributions diverge. The findings advocate for explicit conditioning (or dual-tower architectures) and extended context handling in retrieval models, especially as RAG pipelines enter specialized, heterogeneous, or high-latency production environments.

Future research directions outlined include evaluation expansion to diverse, longer Italian corpora, systematic GPU latency benchmarking, chunking strategy ablation for 256–1024 tokens, and bootstrap-based confidence interval estimation for nDCG differences.

Conclusion

The study demonstrates that GE2 delivers maximal retrieval quality, primarily due to its extended context window and task-type asymmetry, but at a latency and cost premium that is rarely justified for short, monolingual retrieval scenarios. mE5-L closes the quality gap entirely in the latter case and should be operationalized as the default model. The results decisively refute the appropriateness of sentence-similarity models (LaBSE, mMPNet) for retrieval tasks in RAG, identifying a widespread methodological error. The implications for both research and production model selection are clear: prioritize architectural and training objective congruence with retrieval-specific requirements, and carefully balance SLA and deployment constraints against empirical quality and latency measurements.