ChEmbed: Chemistry-Specific Text Embeddings

• ChEmbed is a chemistry-specialized text embedding model that leverages domain-specific data and contrastive fine-tuning to enhance literature retrieval.
• It fine-tunes the nomic-embed-text-v1 backbone on 1.7 million synthetic query–passage pairs and augments its tokenizer with 900 chemistry-specific tokens.
• The model supports long-context inquiries up to 8192 tokens and raises nDCG@10 from 0.82 to 0.91, significantly improving retrieval quality in chemistry RAG systems.

Searching arXiv for the ChEmbed paper and closely related chemistry-retrieval embedding work. Using arXiv search tool with query: "ChEmbed Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings chemistry retrieval embeddings". ChEmbed is a chemistry-specialized family of text embedding models for chemical literature retrieval. It was introduced to improve the retrieval stage of retrieval-augmented generation systems in chemistry, where answer quality depends on retrieving passages that accurately reflect chemical intent. ChEmbed is a domain-adapted encoder initialized from the nomic embedding family and fine-tuned on chemistry-specific query–passage data drawn from PubChem, the Semantic Scholar Open Research Corpus, and ChemRxiv. Its defining features are contrastive retrieval training on about 1.7 million synthetic chemistry query–passage pairs, tokenizer augmentation with chemistry-specific vocabulary, and support for sequences up to 8192 tokens (Kasmaee et al., 3 Aug 2025).

1. Retrieval problem and motivation

ChEmbed is motivated by a structural mismatch between chemical literature and general-purpose information retrieval systems. Chemistry text contains dense specialized nomenclature, including IUPAC names, compound identifiers, reaction terminology, formulas, and long scientific passages. The paper argues that general-purpose embedding models, even strong ones, are not ideal for this setting because they are trained on broad web-like corpora and tend to mishandle precisely those linguistic phenomena that dominate chemical writing (Kasmaee et al., 3 Aug 2025).

Two bottlenecks are emphasized. The first is data scarcity for retrieval training: contrastive embedding models require query–passage pairs or triplets, but such supervision is expensive and scarce in chemistry. The second is benchmark mismatch: generic retrieval benchmarks rely on encyclopedic or broad-domain text and therefore do not reflect real chemical literature search. The tokenizer itself is also treated as a bottleneck, because general tokenizers fragment long chemical strings into many subwords. In chemistry, that fragmentation is consequential, since names often encode substantial semantic content.

In that formulation, ChEmbed is not presented as a general semantic search model applied to chemistry, but as a retrieval encoder specialized for chemical literature. The paper’s central claim is that chemistry retrieval requires both chemistry-specific training data and chemistry-specific tokenization, rather than only a larger generic model (Kasmaee et al., 3 Aug 2025).

2. Model family and domain adaptation

ChEmbed is a family of models built by fine-tuning the nomic-embed-text-v1 backbone. The backbone is characterized in the paper as open and reproducible, already capable of long-sequence processing, and based on architectural choices including rotary positional embeddings, FlashAttention, and SwiGLU activations (Kasmaee et al., 3 Aug 2025).

Domain adaptation proceeds through two mechanisms. The first is contrastive fine-tuning on chemistry query–passage pairs. The second is tokenizer augmentation with chemistry-specific vocabulary. This dual adaptation is the core of the model family: the encoder is trained to behave as a chemistry retriever, while the tokenizer is modified so that chemical entities are encoded more faithfully.

The paper emphasizes that ChEmbed preserves long-context behavior, supporting up to 8192 tokens. That design choice is important because chemical retrieval is often passage-centric rather than abstract-centric. Relevant evidence may be distributed across methods, experimental descriptions, conditions, and caveats, and a short-context encoder may truncate the very material that should determine ranking (Kasmaee et al., 3 Aug 2025).

3. Chemistry-specific training corpus and synthetic supervision

A major component of ChEmbed is the construction of retrieval training data from chemistry-related corpora. The paper uses PubChem, the S2ORC chemistry subset, and ChemRxiv, and it reports the following sample counts (Kasmaee et al., 3 Aug 2025):

Source	Role	Count
PubChem compounds	tokenizer work	2,087,164
PubChem descriptions	training samples	393,321
S2ORC chemistry paragraphs	training samples	1,187,726
ChemRxiv paragraphs (CC-BY)	training samples	139,057
ChemRxiv paragraphs (CC-BY-NC)	held out for evaluation	69,457
ChemRxiv metadata	training samples	30,378

The S2ORC and ChemRxiv texts were paragraphized and filtered to remove short, uninformative, or structurally noisy material, including conclusions, figure captions, and very short passages. This preprocessing constrains the training set toward passage-level content that can plausibly answer a substantive chemistry question.

Because retrieval training requires query–passage pairs, the authors used LLMs to synthesize queries from passages. For each chemistry paragraph, the prompt asked for exactly one meaningful chemistry question that the paragraph could answer, while discouraging shallow or yes/no formulations and banning references such as “according to this paragraph.” The paper uses different models for different corpora: gpt-4o-mini for PubChem descriptions, gpt-4.1-nano for S2ORC chemistry paragraphs, o3-mini for ChemRxiv training paragraphs, and Claude Sonnet 3.7 Thinking for held-out evaluation queries (Kasmaee et al., 3 Aug 2025).

The resulting dataset contains about 1.7 million synthetic query–passage pairs. The paper also notes that some paragraphs were rejected by the LLM as unsuitable, so the generation procedure functioned as a quality filter in addition to a source of supervision (Kasmaee et al., 3 Aug 2025).

4. Tokenizer augmentation and long-context behavior

One of the most concrete technical contributions of ChEmbed is its tokenizer patch. The starting tokenizer is the bert-base-uncased WordPiece tokenizer used by the nomic baseline, which contains 994 [UNUSED] tokens. ChEmbed exploits those unused slots by inserting 900 chemistry-specific tokens without altering the model architecture or retraining the tokenizer from scratch (Kasmaee et al., 3 Aug 2025).

To construct the added vocabulary, the authors trained a WordPiece tokenizer on 2,083,502 unique IUPAC names from PubChem, removed tokens already present in the base vocabulary, and retained the top 900 new chemistry terms. Their embeddings were initialized from a normal distribution,

$\mathcal{N}(0, 0.2).$

The purpose of this modification is to reduce fragmentation of chemical entities, especially long IUPAC names. In the paper’s framing, preserving a chemistry term as a more coherent token sequence improves semantic stability at the encoder level and is therefore directly relevant to retrieval quality (Kasmaee et al., 3 Aug 2025).

The model family also maintains an 8192-token context length. The paper contrasts this with many open-source embedding models that typically support 512 or 2048 tokens. For chemical literature search, the longer context window is presented as a practical advantage because it supports retrieval over long scientific passages rather than only short snippets or aggressively truncated text (Kasmaee et al., 3 Aug 2025).

5. Training objective, optimization, and model variants

ChEmbed is trained with a contrastive InfoNCE objective:

$$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{e^{\,s(q_i,d_i^+)/\tau\,}}{e^{\,s(q_i,d_i^+)/\tau\,} +\displaystyle\sum_{d^-\in\mathcal{N}(q_i)}e^{\,s(q_i,d^-)/\tau\,}}$$

where $s(q,d)$ is cosine similarity scaled by temperature $\tau$, $d_i^+$ is the positive passage for query $q_i$, and $\mathcal{N}(q_i)$ is the negative set. In the pair-based setup, negatives are other passages in the batch; in the triplet setup, they are pre-mined negatives. The paper reports that pure in-batch negatives worked best for its synthetic chemistry data, and it suggests that mined hard negatives were less effective because they were model-generated rather than human-curated (Kasmaee et al., 3 Aug 2025).

The reported training configuration uses a maximum training context of 2048 tokens, scalable inference up to 8192 tokens via dynamic NTK scaling / YaRN-style extension, 4 NVIDIA A100 40GB GPUs, GradCache, mixed precision, and a total batch size of 16,384.

The paper compares several tokenizer-adaptation variants:

• vanilla: original tokenizer, no vocabulary augmentation
• full: ChemVocab tokenizer, all embedding parameters trainable
• plug: new token embeddings trainable, original embeddings frozen
• prog: progressive schedule, where new tokens are trained first, then the whole model is unfrozen

The best result is obtained by the progressive strategy. The paper interprets this as evidence that chemistry-specific vocabulary should first be anchored and only then integrated into the fully trainable encoder (Kasmaee et al., 3 Aug 2025).

6. Benchmark, evaluation protocol, and reported performance

To evaluate domain-specific retrieval, the paper introduces the ChemRxiv Retrieval benchmark. It consists of a corpus of 69,457 ChemRxiv paragraphs and 5,000 synthetic queries. The evaluation queries were generated by Claude Sonnet 3.7, which was different from the training LLMs in order to reduce generation bias (Kasmaee et al., 3 Aug 2025).

The primary metric is nDCG@10. The reported baseline and ChEmbed results are:

• nomic-embed-text-v1 baseline: 0.821
• ChEmbed_vanilla: 0.902
• ChEmbed_progressive: 0.911

The headline summary in the paper is that ChEmbed raises nDCG@10 from 0.82 to 0.91, a 9 percentage point absolute improvement. The paper also states that ChEmbed outperforms much larger models, including proprietary systems and large open models such as Qwen3 embeddings, while remaining faster and smaller (Kasmaee et al., 3 Aug 2025).

The practical implications are framed in retrieval terms. ChEmbed is intended to provide more accurate document retrieval for chemistry queries, better handling of chemical nomenclature, and less truncation loss for long passages. The paper further argues that, in a chemistry RAG pipeline, these retrieval gains improve answer quality and reduce hallucination risk because the generator is conditioned on stronger evidence (Kasmaee et al., 3 Aug 2025).

7. Scope, significance, and terminological distinctions

Within the chemistry retrieval literature, ChEmbed is significant because it combines three interventions that are often treated separately: chemistry-specific supervision, chemistry-specific tokenization, and long-context passage retrieval. Its contribution is not the invention of a new general embedding architecture, but the specialization of an existing long-context encoder into a chemistry retriever through corpus construction, synthetic supervision, and tokenizer repair (Kasmaee et al., 3 Aug 2025).

The name should be distinguished from similarly named embedding systems in other areas. It is unrelated to Chimera, a shared matrix factorization model for dynamic attributed networks that jointly models links, content, and temporal continuity for community detection and prediction (Appel et al., 2018). It is also distinct from the graph-stream change-detection framework on constant-curvature manifolds that the literature summary explicitly characterizes as a form of “change detection via embeddings” and informally relates to the term “ChEmbed” in a different sense (Grattarola et al., 2018).

In current usage, however, ChEmbed most specifically denotes the chemistry-specialized text embedding model family for chemical literature retrieval introduced in 2025. In that sense, it occupies a domain-adapted retrieval niche: a lightweight and reproducible embedding solution designed for chemical literature search rather than a general-purpose semantic encoder (Kasmaee et al., 3 Aug 2025).