F2LLM-v2: Open Multilingual Embeddings

• The paper introduces F2LLM-v2, a multilingual embedding family spanning 80M to 14B parameters, designed for tasks like retrieval, classification, clustering, and bitext mining.
• F2LLM-v2 leverages a decoder-only Transformer architecture with matryoshka representation learning, pruning, and teacher-student distillation to optimize efficiency.
• The models are trained on 60M high-quality samples across 282 natural and 40+ programming languages, achieving competitive performance on 430 diverse benchmarks.

Searching arXiv for the specified paper and closely related entries to ground the article in current metadata. F2LLM-v2 is a family of open, multilingual text-embedding models designed as general-purpose encoders for retrieval, classification, clustering, semantic textual similarity, reranking, and bitext mining. It is described as a set of decoder-only LLM-based embedding models built on Qwen3, spanning 8 distinct sizes from 80M to 14B, trained on a newly curated composite of 60 million publicly available high-quality data samples, and supporting more than 200 languages with particular emphasis on previously underserved mid- and low-resource languages (Zhang et al., 19 Mar 2026). The designation “F2LLM-v2” refers to this multilingual embedding family rather than to the unrelated “Fast-dLLM v2” generative block-diffusion system (Wu et al., 30 Sep 2025). In the authors’ framing, F2LLM-v2 combines a two-stage LLM-based embedding training pipeline with matryoshka learning, model pruning, and knowledge distillation, and is positioned as a transparent, fully open release including models, data, code, and intermediate checkpoints (Zhang et al., 19 Mar 2026).

1. Definition and scope

F2LLM-v2 is presented as a multilingual successor to the earlier F2LLM embedding line. The earlier F2LLM report introduced a family of open-source text embedding models in three sizes—0.6B, 1.7B, and 4B—directly finetuned from foundation models on 6 million query-document-negative tuples curated from open-source, non-synthetic datasets (Zhang et al., 2 Oct 2025). F2LLM-v2 extends that line in three principal ways: scale, multilinguality, and efficiency techniques. Concretely, it expands the model range to 8 sizes from 80M to 14B, increases the data mixture to 60M samples from 157 public sources, and introduces matryoshka representation learning, pruning, and distillation as first-class components of the training recipe (Zhang et al., 19 Mar 2026).

The family is explicitly defined as multilingual and general-purpose. The paper states support for 282 natural languages and 40+ programming languages, with evaluation over 430 tasks across 17 MTEB/MMTEB benchmarks (Zhang et al., 19 Mar 2026). This places F2LLM-v2 within the LLM-based embedding literature rather than the encoder-only multilingual embedding literature, and within that space it is described as fully open about data and training methodology, in contrast to several recent high-performing but partially undocumented systems (Zhang et al., 19 Mar 2026).

A recurring source of confusion is nomenclature. “F2LLM-v2” in the embedding literature refers to “Inclusive, Performant, and Efficient Embeddings for a Multilingual World” (Zhang et al., 19 Mar 2026). By contrast, “Fast-dLLM v2” is a block diffusion LLM for parallel text generation that adapts pretrained autoregressive LLMs for faster generation; despite occasional abbreviation overlap, it is a distinct generative architecture rather than an embedding model (Wu et al., 30 Sep 2025).

2. Architecture and model family

F2LLM-v2 is built on the Qwen3 dense Transformer decoder architecture. The paper describes the backbone as a decoder-only Transformer and states that the 0.6B–14B models are directly Qwen3 LLMs adapted for embeddings, while the 80M, 160M, and 330M variants are obtained by pruning the 0.6B Qwen3-based embedding model (Zhang et al., 19 Mar 2026). The family therefore combines direct large-model adaptation with post hoc compression into smaller students.

The model sizes and architectural scales are given explicitly.

Model	Parameters	Layers
F2LLM-v2-80M	80M	8
F2LLM-v2-160M	159M	9
F2LLM-v2-330M	334M	16
F2LLM-v2-0.6B	596M	28
F2LLM-v2-1.7B	1.7B	28
F2LLM-v2-4B	4.0B	36
F2LLM-v2-8B	7.6B	36
F2LLM-v2-14B	14.0B	40

The paper further specifies hidden dimensions ranging from 320 at 80M to 5120 at 14B, and notes that all sizes use KV heads = 8 (Zhang et al., 19 Mar 2026). Positional encodings are inherited from Qwen3. The architecture is therefore not encoder-only in the BERT/XLM-R sense; instead, it follows the now common “LLM embedding” paradigm of repurposing decoder LLMs as embedding extractors.

Embedding extraction is defined using the EOS-token representation rather than mean pooling or a dedicated CLS token. For an input sequence $x = (w_1, \dots, w_T)$, the embedding is the final hidden state of the EOS token after the last layer, written as $f_\theta(x) \in \mathbb{R}^d := h_T^{(L)}$ (Zhang et al., 19 Mar 2026). The paper explicitly states “No mean pooling, no CLS token,” which is an important architectural distinction relative to many encoder-based retrieval systems (Zhang et al., 19 Mar 2026).

All models support matryoshka embeddings, meaning that smaller deployment dimensions are formed by taking nested prefix subvectors. The paper writes this as $$f_\theta^{(k)}(x) = f_\theta(x)_{1:k} \in \mathbb{R}^k$$ for $k \in \{8, 16, 32, \dots, d\}$ (Zhang et al., 19 Mar 2026). This creates a single checkpoint that can be evaluated or deployed at multiple effective dimensions without retraining.

3. Training pipeline and objectives

F2LLM-v2 adopts a two-stage pipeline similar to NV-Embed and Qwen3-Embedding (Zhang et al., 19 Mar 2026). Stage 1 is retrieval-centric and is intended to establish dense retrieval performance and cross-lingual alignment. The paper states that this stage uses 7 large retrieval datasets totaling 27M samples, including CodeSearchNet, CodeSearchNet-CCR, OpenCodeGeneticInstruct, WebFAQ, mMARCO, CLIRMatrix, and ParaCrawl (Zhang et al., 19 Mar 2026). These are formatted as retrieval tuples with hard negatives mined using Qwen3-Embedding-8B, alongside in-batch negatives (Zhang et al., 19 Mar 2026). The models trained in this stage are 0.6B, 1.7B, 4B, 8B, and 14B, and after stage 1 the 0.6B model is pruned to 80M, 160M, and 330M (Zhang et al., 19 Mar 2026).

Stage 2 is multi-task and instruction-aware. Its stated goal is to sharpen performance across classification, clustering, STS, paraphrase, reranking, and related tasks while aligning the models with instruction-style prompts (Zhang et al., 19 Mar 2026). The paper reports 18M samples drawn from 157 public sources, with at most 80k queries per source to avoid over-fitting to any single dataset (Zhang et al., 19 Mar 2026). The stage-2 mixture is unified into three canonical formats: retrieval, clustering, and two-way classification. For clustering and classification, only hard negatives are used in order to avoid false negatives from in-batch sampling (Zhang et al., 19 Mar 2026).

Instruction injection is a notable design choice. The paper states that task-specific instructions are added to all queries, and that for symmetric tasks such as clustering, STS, bitext, and paraphrase, instructions are randomly applied to 30% of documents and negatives as well (Zhang et al., 19 Mar 2026). This suggests a compromise between instruction-conditioned semantics and invariance across text roles.

The training objective is described conceptually as a standard contrastive objective unified across anchor-positive-negative tuples, using cosine similarity with temperature scaling (Zhang et al., 19 Mar 2026). The paper does not spell out the exact final implementation equation in the same level of granularity as the earlier F2LLM report, but it explicitly states that negatives include in-batch and hard negatives, and that knowledge distillation adds an MSE term between student and teacher embeddings (Zhang et al., 19 Mar 2026). A plausible implication is that the training recipe prioritizes consistency with existing LLM-based embedding practice while focusing innovation on multilingual data design and efficiency mechanisms.

The optimizer is AdamW. Matryoshka representation learning is used in both stages, with minimum matryoshka dimension equal to 8 (Zhang et al., 19 Mar 2026). The per-size training schedule is specified as follows.

Size	Learning rate	Epochs
80M	4e-5	4
160M	3e-5	3
330M	2e-5	3
0.6B	1e-5	2
1.7B	9e-6	2
4B	7e-6	2
8B	6e-6	2
14B	5e-6	2

Teacher assignment is also explicit: 0.6B teaches 80M, 160M, and 330M; 1.7B teaches 0.6B; 4B teaches 1.7B; and 4B, 8B, and 14B are trained without explicit distillation due to resource constraints (Zhang et al., 19 Mar 2026).

4. Data composition and multilingual coverage

The training corpus is described as 60M training samples from 157 public sources, covering 282 natural languages and approximately 40 programming languages (Zhang et al., 19 Mar 2026). The paper gives concrete language counts, including 16.1M English samples, 4.28M Chinese, 3.43M Russian, 2.77M Spanish, 2.43M French, 1.75M German, 1.40M Arabic, 1.40M Dutch, 1.16M Vietnamese, 1.11M Hindi, and 1.08M each for Korean and Japanese (Zhang et al., 19 Mar 2026). Italian, Indonesian, and Portuguese are reported at roughly 0.9–1.0M each, while Scandinavian, Indic, and African languages extend into a long tail down to a few hundred examples (Zhang et al., 19 Mar 2026).

Programming-language coverage is also quantified. The largest code categories are Python at 1.97M, PHP at 0.55M, Java at 0.48M, C++ at 0.39M, Go at 0.35M, and JavaScript at 0.25M, with additional coverage for C#, Rust, Kotlin, SQL, Scala, and others (Zhang et al., 19 Mar 2026). This is consistent with the paper’s positioning of F2LLM-v2 as both a multilingual natural-language embedder and a code-aware embedding family.

The task mixture includes bitext mining and parallel corpora such as UNPC, ParaCrawl, Europarl, and Bactrian-X; multilingual and monolingual QA and retrieval such as SQuAD, Natural Questions, TriviaQA, HotpotQA, MKQA, MQA, WebFAQ, and mMARCO; code retrieval sets such as CodeSearchNet and xCodeEval; instruction-following corpora such as Aya, MURI, OASST2, MultiAlpaca, WildChat, M2Lingual, NaturalReasoning, Infinity Instruct, OpenOrca, and MEDI2; clustering and topic datasets including Arxiv/BioRxiv/MedRxiv clustering, MLSUM, THUCNews, and SIB200; paraphrase and duplicate-question corpora such as QQP and PAWS-X; intent and domain classification datasets such as Banking77, MTOP, and MASSIVE; acceptability tasks including MELA, ScaLA, and DaLA; claim verification datasets such as FEVER and SciFact; and citation prediction resources such as S2ORC and SPECTER (Zhang et al., 19 Mar 2026). The breadth of this list is central to the paper’s claim of task generality.

The multilingual training design relies heavily on mixed-language batches and explicit parallel data. The paper states that cross-lingual alignment is supported by UNPC, ParaCrawl, Europarl, CLIRMatrix, and Bactrian-X, while multilingual QA and instruction datasets such as WebFAQ, MKQA, Aya, MURI, MultiAlpaca, and M2Lingual contribute supervision across dozens to 194+ languages (Zhang et al., 19 Mar 2026). Because all languages use a single multilingual tokenizer inherited from Qwen3, the authors state that all languages share the same embedding space (Zhang et al., 19 Mar 2026). This suggests that alignment is not imposed by separate language-specific heads but learned within a unified representation space.

The paper emphasizes support for mid- and low-resource languages through both curation and labeling. It explicitly notes that training data is annotated with specific languages, unlike some baselines with only coarse labels, and that the design emphasizes task variety per language rather than retrieval-only supervision (Zhang et al., 19 Mar 2026). This is significant because many multilingual embedding systems are strong on aggregate benchmarks while remaining disproportionately optimized for English-heavy evaluation mixes.

5. Efficiency mechanisms: matryoshka learning, pruning, and distillation

A defining feature of F2LLM-v2 is that efficiency is built into the training pipeline rather than treated as an afterthought. The paper’s first mechanism is Matryoshka Representation Learning (MRL), which enforces useful semantics in early embedding dimensions (Zhang et al., 19 Mar 2026). Training evaluates multiple prefix sizes of the same embedding and optimizes contrastive losses across those prefixes, with minimum dimension equal to 8 (Zhang et al., 19 Mar 2026). The reported empirical pattern is “steep gains from 8 → 128D” followed by a plateau as dimension approaches full size (Zhang et al., 19 Mar 2026). The authors give the concrete observation that the 330M model at 896D is approximately comparable to the 8B and 14B models at 32D, illustrating a deployment trade-off between model size and stored vector dimensionality (Zhang et al., 19 Mar 2026).

The second mechanism is model pruning. After stage 1, the 0.6B model is pruned into 80M, 160M, and 330M variants by reducing hidden size and MLP size based on activation norms on a calibration set, and by keeping the first $n$ layers; the paper notes that norm-based layer selection was tested but “keep first n” worked better (Zhang et al., 19 Mar 2026). This is a concrete design choice rather than a general hypothesis. The smaller models are therefore not independently pretrained from scratch but structurally derived from a larger embedding teacher.

The third mechanism is embedding-space knowledge distillation. The paper defines teacher-student mappings explicitly and adds an MSE term over teacher and student embeddings for queries, positives, and negatives (Zhang et al., 19 Mar 2026). The ablation table reports the following average scores over approximately 350 tasks.

Size	w/ distill	w/o distill
80M	58.04	53.37
160M	60.53	56.27
330M	64.55	62.77
0.6B	66.72	65.87
1.7B	69.13	68.58

The paper interprets this as a 2–5 point improvement, especially important after pruning (Zhang et al., 19 Mar 2026). The strongest effect is on the smallest models, which is consistent with pruning-induced information loss. A plausible implication is that F2LLM-v2’s small-model competitiveness depends materially on teacher-guided recovery rather than pruning alone.

These techniques jointly produce what the paper calls a spectrum of models suitable from mobile and edge settings up to large servers (Zhang et al., 19 Mar 2026). Unlike approaches that require separate low-dimensional projection heads or post-training compression pipelines, F2LLM-v2 treats variable-dimensional deployment and small-model derivation as intrinsic parts of the model family.

6. Evaluation, rankings, and comparative performance

The primary evaluation framework is MTEB/MMTEB. The paper reports 17 benchmarks and 430 tasks in total, spanning multilingual, English, European, Scandinavian, Chinese, Japanese, Korean, German, French, Dutch, Russian, Persian, Indic, Vietnamese, Polish, Code, and Medical subsets (Zhang et al., 19 Mar 2026). Task types include retrieval, reranking, classification, multilabel classification, clustering, pair classification, STS, instruction-reranking, bitext mining, and summarization (Zhang et al., 19 Mar 2026).

The flagship model, F2LLM-v2-14B, is reported to rank first on 11 of 17 MTEB language/domain benchmarks at evaluation time (Zhang et al., 19 Mar 2026). The paper gives the following benchmark scores and ranks for the 14B model: Multilingual 68.74 (rank 6), English 73.08 (10), Code 80.75 (1), Medical 65.20 (2), European 69.89 (1), Scandinavian 71.10 (1), Indic 78.85 (1), German 67.02 (1), French 72.62 (2), Korean 74.85 (3), Polish 75.13 (1), Japanese 79.32 (1), Dutch 66.39 (1), Persian 73.55 (1), and Vietnamese 63.56 (1), with average across all evaluated tasks equal to 71.72 (Zhang et al., 19 Mar 2026).

Scaling is reported to be smooth across the family: 14B at 71.72, 8B at 71.23, 4B at 70.27, 1.7B at 68.88, 0.6B at 66.45, 330M at 64.38, 160M at 60.16, and 80M at 57.62 (Zhang et al., 19 Mar 2026). This monotonic pattern suggests that the family is internally well-calibrated, with gains persisting across multiple orders of magnitude in parameter count.

The paper also provides direct comparisons against Qwen3-Embedding and EmbeddingGemma. F2LLM-v2-330M is reported to outperform EmbeddingGemma-0.3B on most language-specific benchmarks and on overall average, 64.41 versus 59.55 (Zhang et al., 19 Mar 2026). F2LLM-v2-0.6B is reported to outperform Qwen3-Embedding-0.6B overall, 66.47 versus 64.02, with wins on European, Scandinavian, Indic, German, French, Korean, Polish, Dutch, Russian, Persian, and Vietnamese, while Qwen3-Embedding is slightly better on Chinese, 66.71 versus 65.23 (Zhang et al., 19 Mar 2026). This comparison is important because it situates F2LLM-v2 not merely as a multilingual expansion of earlier F2LLM, but as a competitive alternative to major contemporaneous LLM embedders.

The earlier F2LLM report provides useful historical context. That paper reported F2LLM-4B at 73.67 average on 41 English MTEB tasks, ranking 7th overall and 2nd among approximately 4B-parameter models, with particularly strong clustering and classification performance (Zhang et al., 2 Oct 2025). F2LLM-v2 changes the evaluation focus from English-centric MTEB standings to a broader multilingual and domain-diverse benchmark suite (Zhang et al., 19 Mar 2026). This suggests a shift in research emphasis from proving that open, modest-scale embedding training can match English SOTA to demonstrating language breadth, small-model efficiency, and cross-domain balance.

7. Position in the literature, practical use, and limitations

The paper situates F2LLM-v2 among encoder-based multilingual embedders such as XLM-R, mDeBERTa-v3, mBART, and mT5, and among LLM-based embedders such as E5, GTE, E5-Mistral, NV-Embed, Qwen3-Embedding, Gemini-Embedding, EmbeddingGemma, BGE-M3, Gecko, LLM2Vec, GritLM, and KaLM-Embedding-v2 (Zhang et al., 19 Mar 2026). Its claimed differentiators are transparency, explicit linguistic inclusivity, and efficiency techniques integrated into the core training design (Zhang et al., 19 Mar 2026).

Transparency is emphasized repeatedly. The paper states that models, full training recipe, data list, and intermediate checkpoints are released, with GitHub and Hugging Face collections provided (Zhang et al., 19 Mar 2026). This follows the earlier F2LLM release, which also emphasized released models, training dataset, and code as a reproducible baseline (Zhang et al., 2 Oct 2025). In the present context, openness matters not only for reproducibility but also for multilingual data auditing, since language balance and long-tail inclusion are central research claims.

For deployment, the paper states that pooling should always use the EOS hidden state and that the default full dimensions range from 320 to 5120 depending on model size (Zhang et al., 19 Mar 2026). It recommends matryoshka truncation to 32–256 dimensions when storage or latency is constrained, noting empirical steep gains up to 128 dimensions and diminishing returns thereafter (Zhang et al., 19 Mar 2026). It characterizes 14B and 8B as best for multilingual accuracy, code, and long-tail languages; 4B and 1.7B as strong trade-offs for large-scale services; 0.6B and 330M as efficient open alternatives that outperform comparable baselines; and 160M and 80M as suitable for highly constrained environments, including mobile and on-device settings (Zhang et al., 19 Mar 2026).

The paper lists multilingual semantic search and RAG, clustering and topic discovery, code search and code-aware RAG, and domain-specific retrieval in medical, legal, and scientific settings as suitable use cases (Zhang et al., 19 Mar 2026). These use cases follow directly from the training mixture and benchmark coverage rather than from a separate qualitative application section.

Limitations are also stated. Despite 282-language coverage, some very low-resource languages have only a few hundred to a few thousand examples, and performance on those languages is not directly reported (Zhang et al., 19 Mar 2026). Domain mismatch remains possible for highly specialized or proprietary corpora (Zhang et al., 19 Mar 2026). The paper does not deeply address fairness or safety, noting that biases and harmful content may be present in embeddings derived from many public sources (Zhang et al., 19 Mar 2026). It also acknowledges that training the 14B model with MRL and large datasets is expensive, even if smaller-model finetuning and subset reproduction are more accessible (Zhang et al., 19 Mar 2026). Finally, some hard negatives are mined using Qwen3-Embedding-8B, so while the pipeline is open, part of its supervision depends on a teacher whose exact training lies outside the paper’s scope (Zhang et al., 19 Mar 2026).

Taken together, F2LLM-v2 can be understood as an embedding research program centered on multilingual breadth, reproducible training, and parameter-to-dimension efficiency. Relative to the original F2LLM family, it replaces a primarily English, 3-size, 6M-sample baseline with an 8-size multilingual system trained on 60M samples and evaluated across 430 tasks (Zhang et al., 2 Oct 2025, Zhang et al., 19 Mar 2026). Relative to contemporary LLM embedders, it is distinguished by explicitly targeting underrepresented language benchmarks and by combining large-model adaptation with pruning, distillation, and elastic embedding dimensionality in a single open framework (Zhang et al., 19 Mar 2026).