---
title: 'F2LLM-v2: Open Multilingual Embeddings'
url: https://www.emergentmind.com/topics/f2llm-v2
type: topic
---

# F2LLM-v2: Open Multilingual Embeddings

Searching arXiv for the specified paper and closely related entries to ground the article in current metadata.
F2LLM-v2 is a family of open, multilingual text-embedding models designed as general-purpose encoders for retrieval, classification, clustering, semantic textual similarity, reranking, and bitext mining. It is described as a set of decoder-only LLM-based embedding models built on Qwen3, spanning 8 distinct sizes from 80M to 14B, trained on a newly curated composite of 60 million publicly available high-quality data samples, and supporting more than 200 languages with particular emphasis on previously underserved mid- and low-resource languages [2603.19223]. The designation “F2LLM-v2” refers to this multilingual embedding family rather than to the unrelated “Fast-dLLM v2” generative block-diffusion system [2509.26328]. In the authors’ framing, F2LLM-v2 combines a two-stage LLM-based embedding training pipeline with matryoshka learning, model pruning, and knowledge distillation, and is positioned as a transparent, fully open release including models, data, code, and intermediate checkpoints [2603.19223].

## 1. Definition and scope

F2LLM-v2 is presented as a multilingual successor to the earlier F2LLM embedding line. The earlier F2LLM report introduced a family of open-source text embedding models in three sizes—0.6B, 1.7B, and 4B—directly finetuned from foundation models on 6 million query-document-negative tuples curated from open-source, non-synthetic datasets [2510.02294]. F2LLM-v2 extends that line in three principal ways: scale, multilinguality, and efficiency techniques. Concretely, it expands the model range to 8 sizes from 80M to 14B, increases the data mixture to 60M samples from 157 public sources, and introduces matryoshka representation learning, pruning, and distillation as first-class components of the training recipe [2603.19223].

The family is explicitly defined as multilingual and general-purpose. The paper states support for 282 natural languages and 40+ programming languages, with evaluation over 430 tasks across 17 MTEB/MMTEB benchmarks [2603.19223]. This places F2LLM-v2 within the LLM-based embedding literature rather than the encoder-only multilingual embedding literature, and within that space it is described as fully open about data and training methodology, in contrast to several recent high-performing but partially undocumented systems [2603.19223].

A recurring source of confusion is nomenclature. “F2LLM-v2” in the embedding literature refers to “Inclusive, Performant, and Efficient Embeddings for a Multilingual World” [2603.19223]. By contrast, “Fast-dLLM v2” is a block diffusion language model for parallel text generation that adapts pretrained autoregressive LLMs for faster generation; despite occasional abbreviation overlap, it is a distinct generative architecture rather than an embedding model [2509.26328].

## 2. Architecture and model family

F2LLM-v2 is built on the Qwen3 dense Transformer decoder architecture. The paper describes the backbone as a decoder-only Transformer and states that the 0.6B–14B models are directly Qwen3 LLMs adapted for embeddings, while the 80M, 160M, and 330M variants are obtained by pruning the 0.6B Qwen3-based embedding model [2603.19223]. The family therefore combines direct large-model adaptation with post hoc compression into smaller students.

The model sizes and architectural scales are given explicitly.

| Model | Parameters | Layers |
|---|---:|---:|
| F2LLM-v2-80M | 80M | 8 |
| F2LLM-v2-160M | 159M | 9 |
| F2LLM-v2-330M | 334M | 16 |
| F2LLM-v2-0.6B | 596M | 28 |
| F2LLM-v2-1.7B | 1.7B | 28 |
| F2LLM-v2-4B | 4.0B | 36 |
| F2LLM-v2-8B | 7.6B | 36 |
| F2LLM-v2-14B | 14.0B | 40 |

The paper further specifies hidden dimensions ranging from 320 at 80M to 5120 at 14B, and notes that all sizes use `KV heads = 8` [2603.19223]. Positional encodings are inherited from Qwen3. The architecture is therefore not encoder-only in the BERT/XLM-R sense; instead, it follows the now common “LLM embedding” paradigm of repurposing decoder LLMs as embedding extractors.

Embedding extraction is defined using the EOS-token representation rather than mean pooling or a dedicated CLS token. For an input sequence $x = (w_1, \dots, w_T)$, the embedding is the final hidden state of the EOS token after the last layer, written as $f_\theta(x) \in \mathbb{R}^d := h_T^{(L)}$ [2603.19223]. The paper explicitly states “No mean pooling, no CLS token,” which is an important architectural distinction relative to many encoder-based retrieval systems [2603.19223].

All models support matryoshka embeddings, meaning that smaller deployment dimensions are formed by taking nested prefix subvectors. The paper writes this as $f_\theta^{(k)}(x) = f_\theta(x)_{1:k} \in \mathbb{R}^k$ for $k \in \{8, 16, 32, \dots, d\}$ [2603.19223]. This creates a single checkpoint that can be evaluated or deployed at multiple effective dimensions without retraining.

## 3. Training pipeline and objectives

F2LLM-v2 adopts a two-stage pipeline similar to NV-Embed and Qwen3-Embedding [2603.19223]. Stage 1 is retrieval-centric and is intended to establish dense retrieval performance and cross-lingual alignment. The paper states that this stage uses 7 large retrieval datasets totaling 27M samples, including CodeSearchNet, CodeSearchNet-CCR, OpenCodeGeneticInstruct, WebFAQ, mMARCO, CLIRMatrix, and ParaCrawl [2603.19223]. These are formatted as retrieval tuples with hard negatives mined using Qwen3-Embedding-8B, alongside in-batch negatives [2603.19223]. The models trained in this stage are 0.6B, 1.7B, 4B, 8B, and 14B, and after stage 1 the 0.6B model is pruned to 80M, 160M, and 330M [2603.19223].

Stage 2 is multi-task and instruction-aware. Its stated goal is to sharpen performance across classification, clustering, STS, paraphrase, reranking, and related tasks while aligning the models with instruction-style prompts [2603.19223]. The paper reports 18M samples drawn from 157 public sources, with at most 80k queries per source to avoid over-fitting to any single dataset [2603.19223]. The stage-2 mixture is unified into three canonical formats: retrieval, clustering, and two-way classification. For clustering and classification, only hard negatives are used in order to avoid false negatives from in-batch sampling [2603.19223].

Instruction injection is a notable design choice. The paper states that task-specific instructions are added to all queries, and that for symmetric tasks such as clustering, STS, bitext, and paraphrase, instructions are randomly applied to 30% of documents and negatives as well [2603.19223]. This suggests a compromise between instruction-conditioned semantics and invariance across text roles.

The training objective is described conceptually as a standard contrastive objective unified across anchor-positive-negative tuples, using cosine similarity with temperature scaling [2603.19223]. The paper does not spell out the exact final implementation equation in the same level of granularity as the earlier F2LLM report, but it explicitly states that negatives include in-batch and hard negatives, and that knowledge distillation adds an MSE term between student and teacher embeddings [2603.19223]. A plausible implication is that the training recipe prioritizes consistency with existing LLM-based embedding practice while focusing innovation on multilingual data design and efficiency mechanisms.

The optimizer is AdamW. Matryoshka representation learning is used in both stages, with minimum matryoshka dimension equal to 8 [2603.19223]. The per-size training schedule is specified as follows.

| Size | Learning rate | Epochs |
|---|---:|---:|
| 80M | 4e-5 | 4 |
| 160M | 3e-5 | 3 |
| 330M | 2e-5 | 3 |
| 0.6B | 1e-5 | 2 |
| 1.7B | 9e-6 | 2 |
| 4B | 7e-6 | 2 |
| 8B | 6e-6 | 2 |
| 14B | 5e-6 | 2 |

Teacher assignment is also explicit: 0.6B teaches 80M, 160M, and 330M; 1.7B teaches 0.6B; 4B teaches 1.7B; and 4B, 8B, and 14B are trained without explicit distillation due to resource constraints [2603.19223].

## 4. Data composition and multilingual coverage

The training corpus is described as 60M training samples from 157 public sources, covering 282 natural languages and approximately 40 programming languages [2603.19223]. The paper gives concrete language counts, including 16.1M English samples, 4.28M Chinese, 3.43M Russian, 2.77M Spanish, 2.43M French, 1.75M German, 1.40M Arabic, 1.40M Dutch, 1.16M Vietnamese, 1.11M Hindi, and 1.08M each for Korean and Japanese [2603.19223]. Italian, Indonesian, and Portuguese are reported at roughly 0.9–1.0M each, while Scandinavian, Indic, and African languages extend into a long tail down to a few hundred examples [2603.19223].

Programming-language coverage is also quantified. The largest code categories are Python at 1.97M, PHP at 0.55M, Java at 0.48M, C++ at 0.39M, Go at 0.35M, and JavaScript at 0.25M, with additional coverage for C#, Rust, Kotlin, SQL, Scala, and others [2603.19223]. This is consistent with the paper’s positioning of F2LLM-v2 as both a multilingual natural-language embedder and a code-aware embedding family.

The task mixture includes bitext mining and parallel corpora such as UNPC, ParaCrawl, Europarl, and Bactrian-X; multilingual and monolingual QA and retrieval such as SQuAD, Natural Questions, TriviaQA, HotpotQA, MKQA, MQA, WebFAQ, and mMARCO; code retrieval sets such as CodeSearchNet and xCodeEval; instruction-following corpora such as Aya, MURI, OASST2, MultiAlpaca, WildChat, M2Lingual, NaturalReasoning, Infinity Instruct, OpenOrca, and MEDI2; clustering and topic datasets including Arxiv/BioRxiv/MedRxiv clustering, MLSUM, THUCNews, and SIB200; paraphrase and duplicate-question corpora such as QQP and PAWS-X; intent and domain classification datasets such as Banking77, MTOP, and MASSIVE; acceptability tasks including MELA, ScaLA, and DaLA; claim verification datasets such as FEVER and SciFact; and citation prediction resources such as S2ORC and SPECTER [2603.19223]. The breadth of this list is central to the paper’s claim of task generality.

The multilingual training design relies heavily on mixed-language batches and explicit parallel data. The paper states that cross-lingual alignment is supported by UNPC, ParaCrawl, Europarl, CLIRMatrix, and Bactrian-X, while multilingual QA and instruction datasets such as WebFAQ, MKQA, Aya, MURI, MultiAlpaca, and M2Lingual contribute supervision across dozens to 194+ languages [2603.19223]. Because all languages use a single multilingual tokenizer inherited from Qwen3, the authors state that all languages share the same embedding space [2603.19223]. This suggests that alignment is not imposed by separate language-specific heads but learned within a unified representation space.

The paper emphasizes support for mid- and low-resource languages through both curation and labeling. It explicitly notes that training data is annotated with specific languages, unlike some baselines with only coarse labels, and that the design emphasizes task variety per language rather than retrieval-only supervision [2603.19223]. This is significant because many multilingual embedding systems are strong on aggregate benchmarks while remaining disproportionately optimized for English-heavy evaluation mixes.

## 5. Efficiency mechanisms: matryoshka learning, pruning, and distillation

A defining feature of F2LLM-v2 is that efficiency is built into the training pipeline rather than treated as an afterthought. The paper’s first mechanism is Matryoshka Representation Learning (MRL), which enforces useful semantics in early embedding dimensions [2603.19223]. Training evaluates multiple prefix sizes of the same embedding and optimizes contrastive losses across those prefixes, with minimum dimension equal to 8 [2603.19223]. The reported empirical pattern is “steep gains from 8 → 128D” followed by a plateau as dimension approaches full size [2603.19223]. The authors give the concrete observation that the 330M model at 896D is approximately comparable to the 8B and 14B models at 32D, illustrating a deployment trade-off between model size and stored vector dimensionality [2603.19223].

The second mechanism is model pruning. After stage 1, the 0.6B model is pruned into 80M, 160M, and 330M variants by reducing hidden size and MLP size based on activation norms on a calibration set, and by keeping the first $n$ layers; the paper notes that norm-based layer selection was tested but “keep first n” worked better [2603.19223]. This is a concrete design choice rather than a general hypothesis. The smaller models are therefore not independently pretrained from scratch but structurally derived from a larger embedding teacher.

The third mechanism is embedding-space knowledge distillation. The paper defines teacher-student mappings explicitly and adds an MSE term over teacher and student embeddings for queries, positives, and negatives [2603.19223]. The ablation table reports the following average scores over approximately 350 tasks.

| Size | w/ distill | w/o distill |
|---|---:|---:|
| 80M | 58.04 | 53.37 |
| 160M | 60.53 | 56.27 |
| 330M | 64.55 | 62.77 |
| 0.6B | 66.72 | 65.87 |
| 1.7B | 69.13 | 68.58 |

The paper interprets this as a 2–5 point improvement, especially important after pruning [2603.19223]. The strongest effect is on the smallest models, which is consistent with pruning-induced information loss. A plausible implication is that F2LLM-v2’s small-model competitiveness depends materially on teacher-guided recovery rather than pruning alone.

These techniques jointly produce what the paper calls a spectrum of models suitable from mobile and edge settings up to large servers [2603.19223]. Unlike approaches that require separate low-dimensional projection heads or post-training compression pipelines, F2LLM-v2 treats variable-dimensional deployment and small-model derivation as intrinsic parts of the model family.

## 6. Evaluation, rankings, and comparative performance

The primary evaluation framework is MTEB/MMTEB. The paper reports 17 benchmarks and 430 tasks in total, spanning multilingual, English, European, Scandinavian, Chinese, Japanese, Korean, German, French, Dutch, Russian, Persian, Indic, Vietnamese, Polish, Code, and Medical subsets [2603.19223]. Task types include retrieval, reranking, classification, multilabel classification, clustering, pair classification, STS, instruction-reranking, bitext mining, and summarization [2603.19223].

The flagship model, F2LLM-v2-14B, is reported to rank first on 11 of 17 MTEB language/domain benchmarks at evaluation time [2603.19223]. The paper gives the following benchmark scores and ranks for the 14B model: Multilingual 68.74 (rank 6), English 73.08 (10), Code 80.75 (1), Medical 65.20 (2), European 69.89 (1), Scandinavian 71.10 (1), Indic 78.85 (1), German 67.02 (1), French 72.62 (2), Korean 74.85 (3), Polish 75.13 (1), Japanese 79.32 (1), Dutch 66.39 (1), Persian 73.55 (1), and Vietnamese 63.56 (1), with average across all evaluated tasks equal to 71.72 [2603.19223].

Scaling is reported to be smooth across the family: 14B at 71.72, 8B at 71.23, 4B at 70.27, 1.7B at 68.88, 0.6B at 66.45, 330M at 64.38, 160M at 60.16, and 80M at 57.62 [2603.19223]. This monotonic pattern suggests that the family is internally well-calibrated, with gains persisting across multiple orders of magnitude in parameter count.

The paper also provides direct comparisons against Qwen3-Embedding and EmbeddingGemma. F2LLM-v2-330M is reported to outperform EmbeddingGemma-0.3B on most language-specific benchmarks and on overall average, 64.41 versus 59.55 [2603.19223]. F2LLM-v2-0.6B is reported to outperform Qwen3-Embedding-0.6B overall, 66.47 versus 64.02, with wins on European, Scandinavian, Indic, German, French, Korean, Polish, Dutch, Russian, Persian, and Vietnamese, while Qwen3-Embedding is slightly better on Chinese, 66.71 versus 65.23 [2603.19223]. This comparison is important because it situates F2LLM-v2 not merely as a multilingual expansion of earlier F2LLM, but as a competitive alternative to major contemporaneous LLM embedders.

The earlier F2LLM report provides useful historical context. That paper reported F2LLM-4B at 73.67 average on 41 English MTEB tasks, ranking 7th overall and 2nd among approximately 4B-parameter models, with particularly strong clustering and classification performance [2510.02294]. F2LLM-v2 changes the evaluation focus from English-centric MTEB standings to a broader multilingual and domain-diverse benchmark suite [2603.19223]. This suggests a shift in research emphasis from proving that open, modest-scale embedding training can match English SOTA to demonstrating language breadth, small-model efficiency, and cross-domain balance.

## 7. Position in the literature, practical use, and limitations

The paper situates F2LLM-v2 among encoder-based multilingual embedders such as XLM-R, mDeBERTa-v3, mBART, and mT5, and among LLM-based embedders such as E5, GTE, E5-Mistral, NV-Embed, Qwen3-Embedding, Gemini-Embedding, EmbeddingGemma, BGE-M3, Gecko, LLM2Vec, GritLM, and KaLM-Embedding-v2 [2603.19223]. Its claimed differentiators are transparency, explicit linguistic inclusivity, and efficiency techniques integrated into the core training design [2603.19223].

Transparency is emphasized repeatedly. The paper states that models, full training recipe, data list, and intermediate checkpoints are released, with GitHub and Hugging Face collections provided [2603.19223]. This follows the earlier F2LLM release, which also emphasized released models, training dataset, and code as a reproducible baseline [2510.02294]. In the present context, openness matters not only for reproducibility but also for multilingual data auditing, since language balance and long-tail inclusion are central research claims.

For deployment, the paper states that pooling should always use the EOS hidden state and that the default full dimensions range from 320 to 5120 depending on model size [2603.19223]. It recommends matryoshka truncation to 32–256 dimensions when storage or latency is constrained, noting empirical steep gains up to 128 dimensions and diminishing returns thereafter [2603.19223]. It characterizes 14B and 8B as best for multilingual accuracy, code, and long-tail languages; 4B and 1.7B as strong trade-offs for large-scale services; 0.6B and 330M as efficient open alternatives that outperform comparable baselines; and 160M and 80M as suitable for highly constrained environments, including mobile and on-device settings [2603.19223].

The paper lists multilingual semantic search and RAG, clustering and topic discovery, code search and code-aware RAG, and domain-specific retrieval in medical, legal, and scientific settings as suitable use cases [2603.19223]. These use cases follow directly from the training mixture and benchmark coverage rather than from a separate qualitative application section.

Limitations are also stated. Despite 282-language coverage, some very low-resource languages have only a few hundred to a few thousand examples, and performance on those languages is not directly reported [2603.19223]. Domain mismatch remains possible for highly specialized or proprietary corpora [2603.19223]. The paper does not deeply address fairness or safety, noting that biases and harmful content may be present in embeddings derived from many public sources [2603.19223]. It also acknowledges that training the 14B model with MRL and large datasets is expensive, even if smaller-model finetuning and subset reproduction are more accessible [2603.19223]. Finally, some hard negatives are mined using Qwen3-Embedding-8B, so while the pipeline is open, part of its supervision depends on a teacher whose exact training lies outside the paper’s scope [2603.19223].

Taken together, F2LLM-v2 can be understood as an embedding research program centered on multilingual breadth, reproducible training, and parameter-to-dimension efficiency. Relative to the original F2LLM family, it replaces a primarily English, 3-size, 6M-sample baseline with an 8-size multilingual system trained on 60M samples and evaluated across 430 tasks [2510.02294][2603.19223]. Relative to contemporary LLM embedders, it is distinguished by explicitly targeting underrepresented language benchmarks and by combining large-model adaptation with pruning, distillation, and elastic embedding dimensionality in a single open framework [2603.19223].

Source: https://www.emergentmind.com/topics/f2llm-v2