Conan-embedding-v2: Multilingual Text Embedder

• Conan-embedding-v2 is a 1.4B parameter multilingual LLM designed to bridge data and training gaps for effective sentence-level embedding.
• It innovatively integrates a time-varying soft mask, cross-lingual retrieval datasets, and dynamic hard negative mining to enhance performance.
• The model achieves SOTA results on benchmarks like MTEB and Chinese MTEB, supporting applications in search, recommendation, and semantic clustering.

Searching arXiv for the specified paper and closely related embedding work. I’m going to look up the arXiv entry for (Li et al., 16 Sep 2025) and related embedding papers. Conan-embedding-v2 is a 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder. It is presented as a response to two stated limitations in prior LLM-based embedding work: a data gap between LLM pretraining and embedding objectives, and a training gap between causal-mask, token-level language modeling and bidirectional-mask, sentence-level embedding learning. The model combines modified pretraining data, a cross-lingual retrieval dataset, a soft-masking mechanism, and dynamic hard negative mining, and it is reported to achieve SOTA performance on both the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB as of May 19, 2025 (Li et al., 16 Sep 2025).

1. Architectural profile

Conan-embedding-v2 contains 1.4 B parameters. Its main configuration is specified as 8 Transformer layers, hidden size 3,584, 32 attention heads with 8 key–value heads via GQA optimization, feed-forward intermediate size 8,192, a maximum context window of 32,768 tokens, and a vocabulary of 150 k tokens (Li et al., 16 Sep 2025).

The underlying block is expressed with a standard Transformer forward pass:

$$h^{(l)} = \mathrm{LayerNorm}\big(h^{(l-1)} + \mathrm{MultiHeadSelfAttention}(M^{(l)}, h^{(l-1)})\big)$$

$$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$

with

$\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$

and

$$\mathrm{head}_i = \mathrm{softmax}\!\left(\frac{(QW_i^Q)(KW_i^K)^T}{\sqrt{d_k}}\right)VW_i^V,$$

$$\mathrm{MultiHead} = \mathrm{Concat}(\mathrm{head}_1 \ldots \mathrm{head}_H)W^O.$$

The reported architecture is notable less for architectural novelty in the block equations than for the way masking, data composition, and embedding-stage objectives are integrated into training. A plausible implication is that the model’s embedding performance is attributed primarily to training recipe design rather than to a nonstandard backbone.

Component	Value
Parameters	1.4 B
Transformer layers	8
Hidden size	3,584
Attention heads	32
Key–value heads	8 via GQA optimization
Feed-forward intermediate size	8,192
Max context window	32,768 tokens
Vocabulary	150 k tokens

2. Data composition and objective transitions

The pretraining corpus is described as approximately 3 T tokens drawn from multilingual web crawl (mC4), CC-News, Wikipedia, Chinese Internet corpora (BAAI-CCI), and additional news, QA pairs, and web pages filtered via Data-Juicer (Li et al., 16 Sep 2025). The model also uses 600 M instruction–query–response triples for LLM SFT, formulated as query↔positive passage pairs.

The LLM pretraining objective is causal next-token cross-entropy:

$$L_{LM} = -\sum_{t=1}^{T}\log P(w_t \mid w_{<t}; \theta),$$

implemented with a causal self-attention mask. During embedding weakly supervised training, using the same data as LLM SFT, the objective switches to contrastive InfoNCE:

$$L_{InfoNCE} = -\sum_{i=1}^{N}\log \frac{\exp(\cos(x_i, y_i^+))}{\sum_{j=1}^{N}\exp(\cos(x_i, y_j))},$$

where $(x_i, y_i^+)$ is the positive query–passage pair and the other $y_j$ act as in-batch negatives.

For STS tasks in supervised embedding fine-tuning, the model uses CoSENT loss:

$$L_{cos} = \log\left(1 + \sum_{(k,l)\,:\,\text{sim}(k,l)<\text{sim}(i,j)} \exp\left(\frac{\cos(x_k,x_l)-\cos(x_i,x_j)}{\tau}\right)\right).$$

This sequence of objectives formalizes the transition from token prediction to sentence-level representation learning. The paper’s framing is that the mismatch between these stages is not incidental; it is a central obstacle for converting an LLM into an effective embedding model. This suggests that Conan-embedding-v2 should be understood as a staged training system rather than simply a pretrained LLM with a pooling head.

3. Training-gap formulation and soft-masking

A central claim is that LLM pretraining uses a full-rank causal mask, with upper-triangle zeros, whereas embedding models require a bidirectional mask, described as all-ones and rank-1. The paper argues that directly jumping from one regime to the other can stall learning, and that this training gap makes full fine-tuning less effective than LoRA (Li et al., 16 Sep 2025).

To address this, Conan-embedding-v2 introduces a time-varying soft mask $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$0 that interpolates between causal and bidirectional masking. With schedule $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$1, where $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$2 is the step and $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$3 the total number of steps, an example linear schedule is

$$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$4

The mask at step $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$5 is defined as

$$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$6

where $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$7 is the training sequence length.

The reported effect is that early columns, corresponding to small $$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$8, of the upper triangle fill in first, gradually reducing mask rank and activating attention to future tokens. In the paper’s interpretation, this allows the model to move continuously from causal language modeling toward the bidirectional attention pattern needed for embedding extraction. The training-gap analysis in Table 9 reports that Soft-Mask closes the gap between LoRA and full fine-tuning: with SM, full fine-tuning reaches 73.52, versus 71.50 without SM (Li et al., 16 Sep 2025).

A common misconception in this area is that full fine-tuning should automatically dominate parameter-efficient adaptation because it optimizes more parameters. The reported result explicitly argues against that generalization in this setting: without addressing the mask/objective transition, more complete adaptation is not necessarily more effective.

4. Cross-lingual retrieval and dynamic hard negatives

The cross-lingual retrieval component begins from existing English retrieval sets, including MSMARCO and T2Retrieval, whose queries were machine-translated via Qwen2.5-7B into 26 languages, including French, Chinese, and Arabic, yielding approximately 10 M cross-lingual pairs (Li et al., 16 Sep 2025). This CLR dataset enables bidirectional retrieval, including Chinese→English and German→English, during supervised embedding training.

The paper states that the effect of the CLR dataset is that embeddings for diverse languages collapse into a shared space, visualized via t-SNE, and that cross-lingual alignment improves dramatically. A plausible implication is that the model’s multilingual behavior depends not only on multilingual pretraining tokens but also on explicit retrieval supervision that forces cross-language neighborhood structure.

Conan-embedding-v2 also introduces dynamic hard negative mining (DHNM). Instead of using frozen hard negatives mined offline, DHNM continuously checks each hard negative’s current difficulty during training. The difficulty score is

$$z^{(l)} = \mathrm{LayerNorm}\big(h^{(l)} + \mathrm{FFN}(h^{(l)})\big)$$9

Let $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$0 denote the initial score. The in-loop replacement rule is

$\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$1

and otherwise $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$2 is kept.

The implementation is described as requiring no extra forward passes: at each loss computation, the model caches $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$3, marks old negatives that become “too easy,” and swaps them out from a prebuilt negative pool in the next step. In contrast to static negative sets, this procedure is designed to maintain adversarial pressure as the encoder improves.

5. Training procedure and optimization regime

All stages use mixed-precision and DeepSpeed ZERO-1 (Li et al., 16 Sep 2025). The training pipeline comprises LLM pre-training, LLM finetuning, embedding weakly-supervised training, and embedding supervised training.

For LLM pre-training, the setup uses 64× Ascend 910B GPUs for 219 h with AdamW, learning rate $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$4, warmup 5%, weight decay $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$5, and batch size 256. LLM finetuning uses 16 GPUs for 38 h with AdamW, learning rate $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$6, warmup 2%, weight decay $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$7, and batch size 64. Embedding weakly-supervised training uses 16 GPUs for 97 h with AdamW, learning rate $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$8, and batch size 64. Embedding supervised training uses 16 GPUs for 13 h; retrieval tasks use batch size 4 with 7 negatives per query, STS tasks use batch size 32, MRL dimensions are $\mathrm{FFN}(x)=W_2\,\mathrm{ReLU}(W_1x+b_1)+b_2,$9, and AdamW uses learning rate $$\mathrm{head}_i = \mathrm{softmax}\!\left(\frac{(QW_i^Q)(KW_i^K)^T}{\sqrt{d_k}}\right)VW_i^V,$$0 with warmup as above.

Stage	Resources and duration	Optimization
LLM pre-training	64× Ascend 910B GPUs, 219 h	AdamW, lr=1e-4, warmup=5%, wd=1e-3, batch=256
LLM finetuning (SFT)	16 GPUs, 38 h	AdamW, lr=2e-5, warmup=2%, wd=1e-3, batch=64
Embedding weakly-supervised	16 GPUs, 97 h	AdamW, lr=1e-4, batch=64
Embedding supervised	16 GPUs, 13 h	Retrieval: batch=4, 7 negatives/query; STS: batch=32; MRL={256,…,3584}; AdamW, lr=1e-4

The inclusion of MRL dimensions indicates explicit support for variable embedding dimensionality. The paper also reports fast inference, specifically 5.14 min on a 910B GPU for 10 k queries, alongside long-context support of 32 K tokens (Li et al., 16 Sep 2025).

6. Reported performance, ablations, and boundaries

On Universal MTEB, described as English classic with 7 tasks plus summarization, Conan-v2 achieves an average of 73.52 versus a previous best of approximately 73.30 for Gemini-Embedding, with gains in classification from 90.98 to 91.11 and retrieval from 66.24 to 66.4 (Li et al., 16 Sep 2025). On Chinese MTEB, the reported average is 74.24 versus a prior best of approximately 72.71 for retrieve-zh-v1, with retrieval MAP reaching 78.31 and classification 76.8. On zero-shot MTEB-EN, trained only on MSMARCO, NQ, FEVER, and related data, the reported average is 71.43 versus 69.80 for Linq-Embed-Mistral, a +1.6 point margin with a 1.4 B model versus a 6.8 B model. On MKQA cross-lingual retrieval with 25 languages and English passages, the model reports Recall@20 = 72.5, R@100 = 80.2, and nDCG@10 = 59.1, with Recall@20 improving by 3.7 over M3-Embedding.

The ablation results are structured around the three main techniques. Soft-Mask only yields 61.73% multilingual MTEB average; CLR only yields 62.69%; DHNM only raises monosetting English and Chinese scores to 71.50 and 72.09; and the full combination of SM+CLR+DHNM yields 65.17% multilingual, English 73.52, and Chinese 74.24 (Li et al., 16 Sep 2025). These figures support the paper’s argument that the model’s reported performance is not reducible to scale alone.

The stated use cases are multilingual search and retrieval, recommendation systems, semantic clustering, zero-shot classification, reranking, STS, and soon cross-modal embedding. The stated limitations are equally explicit. First, there is cross-lingual bias: high-resource Indo-European languages outperform typologically distant languages such as Arabic and Korean. Second, the model shows weak numerical understanding, failing to capture quantitative relationships such as “3 vs. 5.” Future directions are given as balanced sampling for low-resource languages, improved numeric reasoning via external knowledge, extension to cross-modal retrieval, and continual updates of cross-lingual corpora.

These limitations constrain interpretation of the benchmark results. Strong aggregate MTEB and MKQA scores do not imply uniform multilingual quality across language families, nor do they imply robust quantitative semantics. The paper therefore positions Conan-embedding-v2 as a high-performing multilingual text embedder with clearly defined failure modes rather than as a universal semantic encoder.