Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

Abstract: We introduce Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving. Its largest member is a sparse 10B-parameter Mixture-of-Experts encoder with approximately 1.8B active parameters per token. Across English, Russian, multilingual, and code MTEB benchmarks, this model achieves the strongest aggregate performance within the family on all four evaluated suites. In our vLLM benchmark with 1024-token inputs, it processes 114.5k tokens per second, providing 25 percent higher throughput than the dense 3B model and 1.56-2.65x the throughput of the evaluated external systems. The family also includes a dense 3B encoder and a distilled 480M encoder for tighter compute and memory budgets. We train the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions. The resulting 480M model scores 70.98 on Russian MTEB, surpassing FRIDA while using 42 percent fewer parameters. We release all three model checkpoints.

• The Giga-Embeddings family of models includes three variants with varying capacities and dimensions, with the 10B-A1 =8B model yielding the best performance with 114.5 thousand tokens per second throughput, partially due to activating only approximately 1.8B parameters per token.
• The compact 480M model obtained a comparable performance to FRIDA, requiring 42% fewer parameters at 70.98
• Similarity-distribution distillation facilitates a 480M model's compact yet competitive performance and stands out due to its agnostic training proceses
• Models excels in multilingual and code tasks over the rest

Motivation and contribution

"Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings" (2608.23806) addresses a deployment tension in general-purpose text embedding: increasing encoder capacity tends to improve retrieval representations, but dense billion-parameter models impose substantial storage and serving costs. The paper proposes two complementary solutions. The first is a sparse Mixture-of-Experts (MoE) encoder that retains a 10B-parameter resident model while activating approximately 1.8B parameters per token. The second is a compact 480M encoder trained with similarity-distribution distillation from a larger teacher.

The resulting family contains three models:

Model	Architecture	Total parameters	Active parameters per token	Embedding dimension
Giga-Embeddings-480M	Dense	480M	480M	1024
Giga-Embeddings-3B	Dense	3B	3B	2048
Giga-Embeddings-10B-A1.8B	MoE	10B	Approximately 1.8B	1536

The principal claims are empirical. The 10B-A1.8B model obtains the best score within the proposed family on English, Russian, multilingual, and code MTEB suites. At an input length of 1024 tokens, it reaches 114.5 thousand tokens per second under the reported vLLM configuration, exceeding the dense 3B model by 25%. The compact 480M model reaches 70.98 on Russian MTEB, narrowly exceeding FRIDA's reported 70.95 despite using 42% fewer parameters.

Encoder architecture and training pipeline

The models are obtained by adapting decoder-only language-model backbones into bidirectional encoders. The causal attention mask is replaced with a fully visible mask, allowing every token representation to attend to both preceding and succeeding tokens. Mean pooling converts token states into a fixed-dimensional representation, which is then normalized for cosine-similarity scoring. Instruction templates distinguish asymmetric retrieval from symmetric semantic-similarity inputs.

The 480M and 3B models use dense Qwen3-based encoders. The largest model uses a DeepSeekMoE-style architecture with 64 routed experts, one shared expert, and top-4 routing. Its 10B parameters therefore represent model capacity rather than per-token computation. The distinction is central to the paper's efficiency argument: sparse activation reduces token-level computation, although it does not eliminate the memory required to host the expert weights.

Training follows a three-stage contrastive recipe: broad pre-training, retrieval fine-tuning with hard negatives, and multitask fine-tuning. The multitask stage includes retrieval, classification, clustering, and semantic textual similarity objectives. Pre-training uses in-batch InfoNCE with query-query and document-document contrastive terms, together with false-negative masking. Fine-tuning and multitask training instead use explicit candidate groups containing one positive and seven negatives. Each global batch consequently contains 8,192 candidate texts.

Figure 1: Three-stage embedding training pipeline; similarity-distribution distillation is applied only to the 480M student during retrieval and multitask fine-tuning.

The optimization schedule is shared across the family: the learning rate begins at $3\times10^{-5}$, uses 1% warmup, follows cosine decay to $10^{-7}$, and applies Adam with weight decay of 0.01. Retrieval fine-tuning lasts one epoch, while multitask training lasts three epochs. The training mixture combines publicly available data with non-public contractual data. This combination may improve task coverage, but it prevents complete reproduction and limits external auditing of the training corpus.

Similarity-distribution distillation

Distillation is applied only to the 480M model, and only during retrieval fine-tuning and multitask fine-tuning. The teacher supplies cosine similarities over the same eight-candidate support used by the student. These similarities are converted into temperature-scaled probability distributions, and the student minimizes the KL divergence from the teacher distribution in addition to the contrastive loss.

The combined objective uses a distillation weight of $\lambda=0.3$, a distillation temperature of $\tau_{\mathrm{KD}}=0.05$, and a contrastive temperature of 0.02. In contrast to representation-level mean-squared-error or cosine alignment, this formulation does not require equal embedding dimensions. It also avoids matching hidden states, expert assignments, or routing decisions. Consequently, a dense student can receive supervision from a sparse MoE teacher through the teacher's final relative similarity structure.

This design has a specific methodological consequence: the student is encouraged to preserve the teacher's ranking geometry on local candidate sets rather than reproduce its embedding coordinates. The method is therefore architecture-agnostic at the output level, but its effectiveness depends on the quality and informativeness of the candidate supports. The reported study does not evaluate alternative candidate-set sizes, distillation temperatures, or distillation weights, so the robustness of this objective outside the selected eight-candidate setting remains unresolved.

Benchmark performance

Evaluation uses task-macro aggregation across four suites: 41 English tasks, 23 Russian tasks, 131 multilingual tasks, and 12 code tasks. Scores are first averaged within tasks and then uniformly across tasks, preventing suites with more subsets or languages from receiving disproportionate weight.

Model	English	Russian	Multilingual	Code
480M	69.52	70.98	56.97	72.87
3B	71.93	74.56	63.89	76.93
10B-A1.8B	72.23	74.98	65.64	78.41

The MoE model is the best member of the family in every suite. Its gains over the dense 3B model are highly uneven: 0.30 points on English, 0.42 on Russian, 1.75 on multilingual tasks, and 1.48 on code tasks. The larger multilingual and code improvements suggest that additional conditional capacity is more useful for heterogeneous distributions than for the English and Russian aggregates, although the paper does not provide per-task analyses to establish which linguistic or task properties account for the differences.

The sub-one-point English and Russian improvements should not be treated as definitive because each model is evaluated in a single reported run without uncertainty estimates. The claim that the MoE model preserves quality while improving throughput is therefore strongest for the family-level aggregate and for multilingual and code performance, and weaker for the small English and Russian margins.

The 480M model exhibits a substantially larger compression trade-off. Relative to the 3B model, its deficits are 2.41 English points, 3.58 Russian points, 6.92 multilingual points, and 4.06 code points. The multilingual gap is particularly pronounced, indicating that parameter reduction is not uniform across evaluation regimes. Nevertheless, the compact model is competitive with several external models and exceeds FRIDA on Russian MTEB by 0.03 points while using 480M rather than 823M parameters.

Distillation ablation

The paper reports a matched ablation for the 480M model:

480M variant	English	Russian	Code
Without distillation	69.43	70.86	72.65
With distillation	69.52	70.98	72.87
Gain	+0.09	+0.12	+0.22

Similarity-distribution distillation improves all three evaluated suites, with the largest gain on code. The consistency of the gains supports the claim that teacher similarity distributions provide useful supervision beyond the contrastive objective. However, the absolute improvements are small. They establish a positive contribution under the selected recipe, but not that distillation is the primary source of the 480M model's performance, nor that it would outperform representation-level or token-level distillation under controlled comparisons.

The paper's stronger architectural claim is that the method remains valid when teacher and student dimensions differ. That claim follows directly from the loss construction: it transfers distributions over candidate similarities rather than vectors in a shared coordinate system. The experiments, however, do not isolate dimension mismatch from other differences between the teacher and student, leaving the relative value of dimension-agnostic supervision empirically underdetermined.

Throughput and storage efficiency

The reported throughput measurements use vLLM at sequence lengths of 512, 1024, and 2048 tokens.

Model	512 tokens	1024 tokens	2048 tokens
Qwen3 Embedding 4B	70.1k tok/s	73.2k tok/s	71.2k tok/s
F2LLM-v2-8B	43.2k tok/s	43.4k tok/s	42.6k tok/s
Nemotron 8B	42.6k tok/s	43.2k tok/s	41.7k tok/s
Giga-Embeddings-3B	87.9k tok/s	91.5k tok/s	90.4k tok/s
Giga-Embeddings-10B-A1.8B	112.6k tok/s	114.5k tok/s	102.3k tok/s

The MoE model has the highest measured throughput at every sequence length. Relative to the dense 3B model, its speedups are 1.28x at 512 tokens, 1.25x at 1024 tokens, and 1.13x at 2048 tokens. At 1024 tokens, it is 1.56x faster than Qwen3 Embedding 4B and 2.65x faster than the evaluated F2LLM-v2-8B and Nemotron 8B systems.

The declining advantage at 2048 tokens is operationally important. It indicates that sparse activation does not produce a fixed throughput multiplier across sequence lengths; memory bandwidth, kernel efficiency, routing overhead, and sequence-dependent execution effects may become increasingly relevant. The paper does not provide hardware specifications, latency distributions, batch-size sweeps, peak memory, or energy measurements. Moreover, the comparison changes both architecture and implementation across systems, so the results demonstrate an end-to-end serving advantage in the reported environment rather than isolating the causal contribution of MoE sparsity.

The 480M model's efficiency is characterized only by weight storage. Under fp16 assumptions, it requires approximately 0.96 GB, compared with approximately 6 GB for the 3B model and 20 GB for the 10B MoE model. Thus, the compact model uses about 4.8% of the MoE model's weight storage. This is a storage result, not a runtime result: activations, KV-related runtime state, framework overhead, batching behavior, and actual latency are excluded, and the 480M model is absent from the throughput experiment.

Comparison with external systems

Against compact public baselines, the 480M model scores 70.98 on Russian MTEB, surpassing FRIDA's 70.95 and outperforming the smaller BERTA, embeddinggemma-300m, Qwen3-Embedding-0.6B, and F2LLM-v2-0.6B on Russian evaluation. Its English score of 69.52 is below Qwen3-Embedding-0.6B's 70.47 and F2LLM-v2-0.6B's 69.97, while its multilingual score of 56.97 is below the listed compact baselines. Its code score of 72.87 is also below F2LLM-v2-0.6B's 77.41 and Qwen3-Embedding-0.6B's 75.42.

Among larger systems, the 10B-A1.8B model leads the reported Russian comparison with 74.98. It does not lead English, multilingual, or code: Qwen3-Embedding-8B reaches 75.23, 70.58, and 80.69 on those suites, respectively, while F2LLM-v2-14B reaches 80.75 on code. The paper therefore supports a targeted, not universal, superiority claim. The MoE model combines strong Russian performance with favorable measured throughput, but it is not the highest-scoring external model on three of the four benchmark suites.

These comparisons are explicitly contextual rather than controlled same-run experiments. The public baselines were taken from leaderboard results, and differences in prompts, preprocessing, checkpoints, hardware, and evaluation configurations can affect the ranking. The external results consequently establish competitiveness and a Russian-leading result, but not a strict causal or statistically controlled advantage.

Limitations and open questions

The evaluation has several limitations that constrain interpretation. Each model is represented by a single run, so the small score differences lack confidence intervals or run-to-run variance. The throughput study uses one serving environment and does not include the 480M model; it also does not compare a dense and sparse encoder matched for backbone, parameterization, implementation, and hardware. Accordingly, the paper cannot determine how much of the throughput advantage is attributable specifically to routing sparsity.

The undisclosed portion of the training mixture limits reproducibility and prevents independent contamination audits. Aggregate MTEB scores also conceal task- and language-level variance, which is especially relevant given the large gap between the model's Russian performance and its multilingual performance. Finally, the study omits peak memory, latency percentiles, energy consumption, and throughput as a function of batch size. The main open empirical question is therefore whether the reported quality-throughput trade-off persists under matched dense-sparse controls and across hardware and serving regimes that expose different MoE routing and memory costs.

Conclusion

Giga-Embeddings presents a three-model embedding family that combines conventional dense scaling, sparse conditional capacity, and output-level similarity distillation. The 10B-A1.8B encoder achieves the family's best benchmark scores and the highest measured throughput, reaching 114.5 thousand tokens per second at 1024 tokens and exceeding the dense 3B model by 25%. The 480M student provides a substantially smaller alternative, with 0.96 GB of estimated fp16 weight storage and a small but consistent distillation benefit. The evidence supports MoE encoders as a promising high-throughput design for text embeddings, while the absence of matched controls, uncertainty estimates, and comprehensive systems measurements limits the strength of broader efficiency claims.